Entropy and information (divergence) measures

Abstract

The extension of notion for the measure of information with application in communication theory<br><br>back to the experiences of the C. E. Shannon during the second World War II. In 1948, he introduced that<br><br>the entropy is a real number associated with a random variable which is equal to the expected value of the<br><br>surprise that we would receive upon getting a realization of the random variable. Let, Sa(p) = −loga p<br><br>(base-2 is often used) be the measure of surprise we feel when an event with the probability p is occurring<br><br>actually occurs. Then, entropy for a random variable is calculated using the probability mass (or pdf)<br><br>function of the random variable via Ha(X) = E[Sa(p(X))]. Some of the properties and characterizations<br><br>of the Shannon entropy and its extension versions are mentioned here.<br><br>Also, finding expressions for the multivariate distributions (discrete or continuous) and information measure<br><br>such as mutual information with some of their properties and discussing in view of copula are<br><br>reviewed.<br><br>The principle of maximum entropy provides a method to select the unknown pdf (or pmf) compatible<br><br>to entropy under a specified constraint. This idea was introduced by Jaynes 1957 and obtained via a<br><br>theorem by Kagan et al. (1973). Applying to maximum Renyi or Tsallis entropy and also φ−entropy, as<br><br>a general format subsume many special cases. Similar arguments are applicable to a multivariate set-up.<br><br>In probability theory and information theory, the Kullback Leibler divergence (also information divergence,<br><br>information gain, relative entropy) is a non-symmetric measure of the difference between two<br><br>probability distributions. Specifically, the Kullback Leibler divergence is typically represents the ”true”<br><br>distribution of data and a theoretical model for approximation of the true distribution. Although it<br><br>is often intuited as a metric or distance, the KL divergence is not a true metric. Various applications<br><br>in statistics and properties of it is one of our aim in here. The link between maximum likelihood and<br><br>maximum entropy and Kullback Leibler information is important for a discussion which is coming in<br><br>this note. There are several types of information divergence measure that are studied in literature as<br><br>extensions of the Shannon entropy and Kullback Leibler information. Some of them can be collected in<br><br>Csiszar φ−divergence as special cases. So, minimization of them is important and finding these optimal<br><br>measures is the other direction that is discussed in this paper with the related special states such as Kullback<br><br>Leibler information, χ2-divergence, total variation, squared perimeter distance, Renyi divergence,<br><br>Hellinger distance, directed divergence and so on.