Machine & Deep Learning Compendium

Searchβ¦

Information Theory

- 3.

**i.e., the distribution of examples in each node is so that it mostly contains examples of a single class****In other words: We want a measure that prefers attributes that have a high degree of βorderβ:****Maximum order: All examples are of the same class****Minimum order: All classes are equally likely β Entropy is a measure for (un-)orderedness Another interpretation:****Entropy is the amount of information that is contained****all examples of the same class β no information**

**Maximal value when the equal class distribution****Minimal value when only one class is in S**

**Overfitting****fragmentation**

- 2.
- 3.
- 4.βMastery on entropy, kullback leibler divergence (asymmetry), jensen-shannon divergence (symmetry) (has code)
- 6.
- 7.

SOF**TMAX**

- 1.
- 3.β
**Softmax vs cross entropy****- Softmax loss and cross-entropy loss terms are used interchangeably in industry. Technically, there is no term as such Softmax loss. people use the term "softmax loss" when referring to "cross-entropy loss". The softmax classifier is a linear classifier that uses the cross-entropy loss function. In other words, the gradient of the above function tells a softmax classifier how exactly to update its weights using some optimization like****gradient descent****.**

- 1.β
**entroPY****- EntroPy is a Python 3 package providing several time-efficient algorithms for computing the complexity of one-dimensional time-series. It can be used for example to extract features from EEG signals.**

- 1.

Last modified 8mo ago