Open AccessArticle

On the Measurement of Randomness (Uncertainty): A More Informative Entropy

Tarald O. Kvålseth

^1,2

Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN 55455, USA

Department of Industrial & Systems Engineering, University of Minnesota, Minneapolis, MN 55455, USA

Entropy 2016, 18(5), 159; https://doi.org/10.3390/e18050159

Submission received: 11 February 2016 / Revised: 7 April 2016 / Accepted: 15 April 2016 / Published: 26 April 2016

(This article belongs to the Section Information Theory, Probability and Statistics)

Download Versions Notes

Abstract

As a measure of randomness or uncertainty, the Boltzmann–Shannon entropy H has become one of the most widely used summary measures of a variety of attributes (characteristics) in different disciplines. This paper points out an often overlooked limitation of H: comparisons between differences in H-values are not valid. An alternative entropy

H_{K}

is introduced as a preferred member of a new family of entropies for which difference comparisons are proved to be valid by satisfying a given value-validity condition. The

H_{K}

is shown to have the appropriate properties for a randomness (uncertainty) measure, including a close linear relationship to a measurement criterion based on the Euclidean distance between probability distributions. This last point is demonstrated by means of computer generated random distributions. The results are also compared with those of another member of the entropy family. A statistical inference procedure for the entropy

H_{K}

is formulated.

Keywords:

randomness; uncertainty; entropy; value validity

1. Introduction

For some probability distribution

P_{n} = (p_{1}, ..., p_{n})

, with

p_{i} \geq 0

for i = 1, …, n and

\sum_{i = 1}^{n} p_{i} = 1

, the entropy

H (P_{n})

, or simply H, is defined by:

H = - \sum_{i = 1}^{n} p_{i} log p_{i} \in [0, log n]

(1)

where the logarithm is the natural (base-e) logarithm. The probabilities

p_{i} (i = 1, ..., n)

may be associated with a set of quantum states of a physical system in statistical mechanics or physics, a set of symbols or messages in a communication system, or, most generally, a set of mutually exclusive and exhaustive events. First used by Boltzmann [1] in statistical mechanics (as kH with k being the so-called Boltzmann constant) and later introduced by Shannon [2] as the basis for information theory (with base-2 logarithm and bits as the unit of measurement), this entropy H can appropriately be called the Boltzmann–Shannon entropy.

Although interpreted in a number of different ways, the most common and general interpretation of H is as a measure of randomness or uncertainty of a set of random events (e.g., [3] (pp. 67–97), [4], [5] (Chapter 2), [6] (pp. 12, 13, 90)). A number of alternative entropy formulations have been proposed as parameterized generalizations of H in Equation (1) (see, e.g., [7,8,9]), but with limited success or impact. The most notable exception is the following one-parameter family of entropies by Rényi [10]:

H_{R} = \frac{1}{1 - α} {log}_{2} \sum_{i = 1}^{n} p_{i}^{α}, α > 0, α \neq 1

which reduces to Equation (1), with base-2 logarithm, when

α \to 1

. This entropy family has, for instance, been used as a fractal dimension [11] (pp. 686–688). Another such family of entropies is that of Tsallis [12] defined as:

H_{T} = \frac{1}{α - 1} (1 - \sum_{i = 1}^{n} p_{i}^{α}), - \infty < α < \infty

which includes the H in Equation (1) as the limiting case when

α \to 1

Since its origins in statistical physics and information theory, the entropy H in Equation (1) has proved to be remarkably versatile as a summary measure of a wide variety of attributes (characteristics) within diverse fields of study, ranging from psychology (e.g., [13]) to fractal geometry [11] (pp. 678–687). However, such widespread use of H has led to misuse, improper applications, and misleading results due to the fact that, although H has a number of desirable properties [14] (Chapter 1), it does suffer from one serious limitation: comparisons between differences in H-values are not valid. The basis for this limitation is explained in the next section of this paper.

As a clarification of such comparisons in general, consider some summary measure M and probability distributions

P_{n}

Q_{m}

R_{t}

S_{u}

. The various types of potential comparisons can then be defined as follows:

S i z e (o r d e r) c o m p a r i s o n s : M (P_{n}) > M (Q_{m})

(2a)

D i f f e r e n c e c o m p a r i s o n s : M (P_{n}) - M (Q_{m}) > M (R_{t}) - M (S_{u})

(2b)

P r o p o r t i o n a l d i f f e r e n c e c o m p a r i s o n s : M (P_{n}) - M (Q_{m}) = c [M (R_{t}) - M (S_{u})]

(2c)

where c is a constant. While, because of the properties of H in Equation (1), there is no particular reason to doubt the validity of the size comparison in Equation (2a) involving H, the difference comparisons in Equations (2b) and (2c) are not valid for H as discussed below.

In this paper, an alternative and equally simple entropy is introduced as:

H_{K} = {(\sum_{\underset{i \neq m}{i = 1}}^{n} \sqrt{p_{i}})}^{2} + \sum_{\underset{i \neq m}{i = 1}}^{n} p_{i}, p_{m} = max_{i} {p_{i}}

(3)

The term entropy is used for this measure of randomness or uncertainty since (a) it has many of the same properties as H in Equation (1) and (b) the entropy term has been used in such a variety of measurement situations for which

H_{K}

can similarly be used. As is established in this paper,

H_{K}

has the important advantage of being more informative than H in the sense that

H_{K}

meets the conditions for valid difference comparisons as in Equations (2b) and (2c). It will also be argued that the

H_{K}

is the preferred member of a family of entropies with similar properties. Statistical inference procedure for

H_{K}

will also be outlined.

2. Conditions for Valid Difference Comparisons

Consider that M is a measure of randomness (uncertainty) such that its value

M (P_{n})

for any probability distribution

P_{n} = (p_{1}, ..., p_{n})

is bounded as:

M (P_{n}^{0}) \leq M (P_{n}) \leq M (P_{n}^{1})

(4)

where

P_{n}^{0}

and

P_{n}^{1}

are the degenerate and uniform distributions

P_{n}^{0} = (1, 0, ..., 0), P_{n}^{1} = (1 / n, ..., 1 / n)

(5)

and where one can set

M (P_{n}^{0}) = 0

. In order for the difference comparisons in Equations (2b) and (2c) to be permissible or valid, some condition needs to be imposed on M (see [15]). Specifically, all intermediate values

M (P_{n})

in Equation (4) have to provide numerical representations of the extent of randomness (uncertainty) that are true or realistic with respect to some acceptable criterion. While different types of validity are used in measurement theory [16] (pp. 129–134), value validity will be used here for this required property of M. In order to establish specific requirements for M to have value validity, a particular probability distribution proves useful and Euclidean distances will be used as a criterion.

Therefore, consider the recently introduced lambda distribution:

P_{n}^{λ} = (1 - λ + \frac{λ}{n}, \frac{λ}{n}, ..., \frac{λ}{n}), λ \in [0, 1]

(6)

where

λ

is a uniformity (evenness) parameter and with

P_{n}^{0}

and

P_{n}^{1}

in Equation (5) being special (extreme) cases [17]. This

P_{n}^{λ}

is simply the following weighted mean of

P_{n}^{0}

and

P_{n}^{1}

P_{n}^{λ} = λ P_{n}^{1} + (1 - λ) P_{n}^{0}

(7)

From Equations (4) and (6), it follows that, for any given

P_{n}

M (P_{n}) = M (P_{n}^{λ}) for one unique λ

(8)

so that validity conditions on

M (P_{n})

can equivalently be determined in terms of

M (P_{n}^{λ})

. With probability distributions considered as points (vectors) in n-dimensional space and with the Euclidean distance function d being the basis for a validity criterion, the following requirement seems most natural and obvious [17]:

\frac{M (P_{n}^{1}) - M (P_{n}^{λ})}{M (P_{n}^{1}) - M (P_{n}^{0})} = \frac{d (P_{n}^{λ}, P_{n}^{1})}{d (P_{n}^{0}, P_{n}^{1})} = 1 - λ

(9)

Since

M (P_{n}^{0}) = 0

, i.e., there is no randomness when one

p_{i} = 1

, it follows from Equation (9) that:

M (P_{n}^{λ}) = λ M (P_{n}^{1})

(10)

as a value-validity condition. This condition also follows immediately from Equation (7) as:

M (P_{n}^{λ}) = M [λ P_{n}^{1} + (1 - λ) P_{n}^{0}] = λ M (P_{n}^{1}) + (1 - λ) M (P_{n}^{0})

(11)

which equals Equation (10) for

M (P_{n}^{0}) = 0

For the case when

λ = 0.5

P_{n}^{0.5}

is the midpoint of

P_{n}^{0}

and

P_{n}^{1}

with coordinates

(1 / 2 + 1 / 2 n, 1 / 2 n, ..., 1 / 2 n)

. Then:

M (P_{n}^{0.5}) = M (\frac{P_{n}^{0} + P_{n}^{1}}{2}) = \frac{M (P_{n}^{0}) + M (P_{n}^{1})}{2}

(12)

= \frac{M (P_{n}^{1})}{2} for M (P_{n}^{0}) = 0

(13)

which is exactly as stated in Equations (10) and (11) with

λ = 0.5

. Of course, Equations (12) and (13) represent a weaker value-validity condition than Equations (10) and (11). Note also that it is not assumed a priori that M is a linear function of

λ

. This linearity is a consequence of Equations (7)–(9).

The entropy H in Equation (1) with

H (P_{n}^{0}) = 0

and

H (P_{n}^{1}) = log n

does not meet these validity conditions. For example, for n = 2 and

λ = 0.5

H (P_{2}^{0.5}) = H (0.75, 0.25) = 0.56

, which far exceeds the requirement log2/2 = 0.35 in Equation (13). Similarly,

H (P_{4}^{0.5}) = H (0.625, 0.125, 0.125, 0.125) = 1.07 > > log 4 / 2 = 0.69

and

H (P_{20}^{0.5}) = 2.09 > > log 20 / 2 = 1.5 0

. It can similarly be verified that

H (P_{n}^{λ})

for all n and

0 < λ < 1

, and hence

H (P_{n})

for all

P_{n}

from Equation (8), overstates the true or realistic extent of the randomness (uncertainty) that H is supposed to measure. Consequently, difference comparisons as in Equations (2b) and (2c) based on H are invalid. An alternative measure that meets the validity conditions for such comparisons will be introduced next.

3. The New Entropy

3.1. Derivation of $H_{K}$

The logic or reasoning behind the

H_{K}

in Equation (3) as a measure of randomness or uncertainty may be outlined as follows:

(1): As a matter of convenience and as used throughout the rest of the paper, all individual probabilities will be considered ordered such that:

$p_{1} \geq p_{2} \geq \dots \geq p_{n}$

(14)
(2): Due to the constraint that $\sum_{i = 1}^{n} p_{i} = 1$ , there is no loss of generality or information by focusing on $p_{2}, ..., p_{n}$ .
(3): Instead of considering $\sum_{i = 2}^{n} f (p_{i})$ or a weighted mean or sum of $f (p_{2}), ..., f (p_{n})$ for some function f of the individual $p_{i}$ ’s, one could consider the sum of the means of all pairs of the $p_{i} (i = 2, ..., n)$ . Since an entropy measure needs to be zero-indifferent (expansible), i.e., unaffected by the addition of events with zero probabilities (e.g., [14] (Chapter 1)), a logical choice of pairwise means would be the geometric means $\sqrt{p_{i} p_{j}}$ for all i, j = 2, …, n (since obviously $\sqrt{p_{i} 0} = 0$ ). Therefore, the measure consisting of the means $\sqrt{p_{i} p_{j}}$ , including those for i = j, can be expressed as:

$H_{K} = 2 \sum_{2 \leq i} \sum_{\leq j \leq n} \sqrt{p_{i} p_{j}} = \sum_{i = 2}^{n} \sum_{j = 2}^{n} \sqrt{p_{i} p_{j}} + \sum_{i = 2}^{n} p_{i}$

(15)

where the multiplication factor 2 is included so that $H_{K} (P_{n}^{1})$ = H(1/n,…,1/n) = n − 1 instead of (n−1)/2. With $p_{1}$ being the modal (largest) probability, this $H_{K}$ in Equation (15) is twice the sum of the pairwise geometric means of all the non-modal probabilities. Furthermore, from the fact that, for a set of numbers ${x_{i}}, {(\sum_{i = 2}^{n} x_{i})}^{2} = \sum_{i = 2}^{n} \sum_{j = 2}^{n} x_{i} x_{j}$ and then setting $x_{i} = \sqrt{p_{i}}$ , it follows from the second expression in Equation (15) that:

$H_{K} = {(\sum_{i = 2}^{n} \sqrt{p_{i}})}^{2} + \sum_{i = 2}^{n} p_{i}$

(16)

which is the same as the formula in Equation (3).

As an alternative approach, one could begin by considering the power sum, or sum of order

α

S_{α} = {(\sum_{i} p_{i}^{α})}^{1 / α}

(e.g., [18] (pp. 138–139). Strict Schur-concavity, which is discussed below as an important property of an entropy and one that H in Equation (1) has, requires that the parameter

α < 1

[18] (pp. 138–139). Since

p_{i} \geq 0

(i = 1, …, n), a further restriction is that

α

be positive and hence

0 < α < 1

for the power sum

S_{α}

. In order for

S_{α}

to comply with the value-validity condition in Equation (11), it is clear that

S_{α}

can only be the power sum of the non-modal probabilities so that:

S_{α} (p_{2}, ..., p_{n}) = {(\sum_{i = 2}^{n} p_{i}^{α})}^{1 / α}, 0 < α < 1

(17)

with

S_{α} (P_{n}^{0}) = 0

and

S_{α} (P_{n}^{1}) = {(n - 1)}^{1 / α} / n

for the probability distributions in Equation (5). A reasonable upper bound would be

S_{α} (p_{n}^{1}) = n - 1

. This requirement is met for

α = 1 / 2

in Equation (17) and by the addition of

\sum_{i = 2}^{n} p_{i}

, resulting in:

S_{1 / 2} (p_{2}, ..., p_{n}) + S_{1} (p_{2}, ..., p_{n}) = {(\sum_{i = 2}^{n} \sqrt{p_{i}})}^{2} + \sum_{i = 2}^{n} p_{i}

(18)

which is the same as Equation (16) and for which

H_{K} (P_{n}^{1}) = n - 1

3.2. Properties of $H_{K}$

The properties of

H_{K}

in Equation (16), some of which are readily apparent from its definition, can be outlined as follows:

Property 1.

H_{K}

is a continuous function of all its individual arguments

p_{1}, ..., p_{n}

Property 2.

H_{K}

is (permutation) symmetric in the

p_{i} (i = 1, ..., n)

Property 3.

H_{K}

is zero-indifferent (expansible), i.e.,

H_{K} (p_{1}, ..., p_{n}, 0, ..., 0) = H_{K} (p_{1}, ..., p_{n})

Property 4.

For any given

P_{n} = (p_{1}, ..., p_{n})

and the

P_{n}^{0}

and

P_{n}^{1}

in Equation (5):

H_{K} (P_{n}^{0}) \leq H_{K} (P_{n}) \leq H_{K} (P_{n}^{1}); H_{K} (P_{n}^{0}) = 0, H_{K} (P_{n}^{1}) = n - 1

(19)

Property 5.

From Equation (19),

H_{K} (P_{n}^{1})

is strictly increasing in n.

Property 6.

H_{K}

is strictly Schur-concave so that, if

P_{n} = (p_{1}, ..., p_{n})

is majorized by

Q_{n} = (q_{1}, ..., q_{n})

(denoted by

≺

P_{n} ≺ Q_{n} \Rightarrow H_{K} (P_{n}) \geq H_{K} (Q_{n})

(20)

with strict inequality unless

P_{n}

is simply a permutation of

Q_{n}

Property 7.

H_{K}

is concave, but not strictly concave.

Property 8.

H_{K}

meets the value-validity condition in Equation (10) with

H_{K} (P_{n}^{λ}) = λ (n - 1)

Proof of Property 6.

The strict Schur-concavity of

H_{K} = {(\sum_{i = 2}^{n} \sqrt{p_{i}})}^{2} + 1 - p_{1}

follows immediately from the partial derivatives:

\frac{\partial H_{K}}{\partial p_{1}} = - 1; \frac{\partial H_{K}}{\partial p_{i}} = p_{i}^{- 1 / 2} \sum_{j = 2}^{n} \sqrt{p_{j}}, i = 2, ..., n

and the fact that

\partial H_{K} / \partial p_{i}

is strictly increasing in i = 1, …, n (unless

p_{i} = p_{i + 1}

) ([18] (p. 84)). The majorization

P_{n} ≺ Q_{n}

in Equation (20) is a more precise statement than the vague notion that the components of

P_{n}

are “more nearly equal” or “less spread out” than are those of

Q_{n}

. By definition, if

\sum_{i = 1}^{n} p_{i} = \sum_{i = 1}^{n} q_{i}

(and with the ordering in Equation (14) for

P_{n}

and

Q_{n}

P_{n} ≺ Q_{n} if \sum_{i = 1}^{j} p_{i} \leq \sum_{i = 1}^{j} q_{i}, j = 1, ..., n - 1

([18] (p. 8)). ☐

Proof of Property 7.

From Equation (16) and for the probability distributions

P_{n}

and

Q_{n}

and all

λ \in [0, 1]

H_{K} [λ P_{n} + (1 - λ) Q_{n}] = {[\sum_{i = 2}^{n} {(λ p_{i} + (1 - λ) q_{i})}^{1 / 2}]}^{2} + λ \sum_{i = 2}^{n} p_{i} + (1 - λ) \sum_{i = 1}^{n} q_{i}

(21)

From Minkowski’s inequality (e.g., [19] (p. 175)):

{[\sum_{i = 2}^{n} {(λ p_{i} + (1 - λ) q_{i})}^{1 / 2}]}^{2} \geq {[\sum_{i = 2}^{n} {(λ p_{i})}^{1 / 2}]}^{2} + {[\sum_{i = 2}^{n} {((1 - λ) q_{i})}^{1 / 2}]}^{2}

(22)

so that, from Equations (21) and (22):

\begin{matrix} H_{K} [λ P_{n} + (1 - λ) Q_{n}] & \geq λ [{(\sum_{i = 2}^{n} \sqrt{p_{i}})}^{2} + \sum_{i = 2}^{n} p_{i}] + (1 - λ) [{(\sum_{i = 2}^{n} \sqrt{q_{i}})}^{2} + \sum_{i = 2}^{n} q_{i}] = \\ = λ H_{K} (P_{n}) + (1 - λ) H_{K} (Q_{n}) \end{matrix}

(23)

proving that

H_{K}

is concave. However, and importantly,

H_{K}

is not strictly concave since the inequality in Equation (23) is not strict for all

P_{n} and Q_{n}

such as for

P_{n} = P_{n}^{1}

and

Q_{n} = P_{n}^{0}

in Equation (5) when

H_{K} (P_{n}^{λ}) = H_{K} [λ P_{n}^{1} + (1 - λ) P_{n}^{0}] = λ H_{K} (P_{n}^{1}) + (1 - λ) H_{K} (P_{n}^{0}) = λ (n - 1)

(24)

as required by the value-validity conditions in Equations (10) and (11). ☐

Note 1: If a measure (function) M is strictly concave so that, instead of Equation (23), the inequality $M [λ P_{n} + (1 - λ) Q_{n}] > λ M (P_{n}) + (1 - λ) M (Q_{n})$ is strict for all $P_{n}, Q_{n}, and λ \in (0, 1)$ , then the condition in Equation (11) cannot be met. The H in Equation (1) is one such measure.
Note 2: The extremal values $H_{K} (P_{n}^{1}) = H_{K} (1 / n, \dots, 1 / n) = n - 1$ for a measure of randomness or uncertainty is also a logical requirement for valid difference comparisons. As a particular case of the proportional difference comparisons in Equation (2c), and for any integer m < n:

$H_{K} (P_{n + m}^{1}) - H_{K} (P_{n}^{1}) = H_{K} (P_{n}^{1}) - H_{K} (P_{n - m}^{1})$

(25)

i.e., adding an amount m to n results in the same absolute change in the value of $H_{K}$ as does subtracting m from n in the equiprobable case. Or, in terms of the function f where $H_{K} (P_{n}^{1}) = f (n)$ , Equation (25) can be expressed more conveniently as:

$f (n + m) - f (n) = f (n) - f (n - m)$

(26)

The general solution of the functional equation in Equation (26) is $f (n) = a n + b$ with real constants a and b [20] (p. 82), which equals $H_{K} (P_{n}^{1})$ for a = 1 and b = −1.
Note 3: For the binary case of n = 2, $H_{K} (P_{2}^{1}) = H_{K} (0.5, 0.5) = 1$ , which equals H(0.5, 0.5) in Equation (1) if the base-2 logarithm is used. In fact, H(0.5, 0.5) = 1 is an axiom or required property, the normalization axiom, frequently used in information theory to justify the use of the base-2 logarithm in Equation (1) and bits as the unit of measurement [14] (Chapter 1). The binary entropy $H_{K} (1 - p, p) = 2 p for p \leq 0.5$ or:

$H_{K} (1 - p, p) = 1 - | 1 - 2 p | for all p \in [0, 1]$

4. Generalization of $H_{K}$

Instead of the pairwise geometric means in Equation (15), one could consider power means, or arithmetic means of order

α

, and hence the following parameterized family of entropies:

H_{K α} = \sum_{i = 2}^{n} \sum_{j = 2}^{n} M_{α} (p_{i}, p_{j}) + \sum_{i = 2}^{n} p_{i}; M_{α} (p_{i}, p_{j}) = {(\frac{p_{i}^{α} + p_{j}^{α}}{2})}^{1 / α}, - \infty < α \leq 0

(27)

of which

H_{K}

in Equations (15) and (16) is the particular member

H_{K 0} as α \to 0

. Since a measure of randomness (uncertainty) should be zero-indifferent (see Property 3 of

H_{K}

), it is clear from the formula in Equation (27) that

α

cannot be positive, i.e.,

α \leq 0

where

α = 0

means the limit when

α \to 0

. If

p_{i} = 0, p_{j} = 0, or p_{i} = p_{j} = 0, M_{α} (p_{i}, p_{j})

is taken to be 0 for

α \leq 0

(see, e.g., [21] (Chapter 2) for the properties of such power means). One of the important properties of

M_{α} (p_{i}, p_{j})

is that it is a non-decreasing function of

α (for - \infty < α < \infty)

and is strictly increasing unless

p_{i} = p_{j}

. Besides this

M_{α}

, there are other types of means that could be considered (e.g., [18] (pp. 139–145), [22]).

Since

M_{α} (p_{i}, p_{j})

is strictly increasing in

α (if p_{i} \neq p_{j})

, it follows from Equation (27) that, for any probability distribution

P_{n} = (p_{1}, ..., p_{n})

H_{L} (P_{n}) \leq H_{K α} (P_{n}) \leq H_{K} (P_{n}) for all α \in (- \infty, 0)

(28)

where the lower limit

H_{L} (P_{n}) = H_{K (- \infty)} (P_{n})

is the limit of

H_{K α} as α \to - \infty and H_{K} (P_{n})

is defined in Equations (15) and (16) and is the limit of

H_{K α} (P_{n}) as α \to 0

. The inequalities in Equation (28) are strict unless

P_{n} equals P_{n}^{0} or P_{n}^{1}

in Equation (5).

Each member of

H_{K α}

has the same types of properties as those of

H_{K}

discussed above. The strict Schur-concavity of

H_{K α}

follows from the fact that (a)

H_{K α}

is (permutation) symmetric in the

p_{i} (i = 1, ..., n)

and (b) the partial derivatives, after setting

\sum_{i = 2}^{n} p_{i} = 1 - p_{1}

in Equation (27):

\frac{\partial H_{K α}}{\partial p_{1}} = - 1, \frac{\partial H_{K α}}{\partial p_{i}} = {\sum_{j = 2}^{n} (\frac{p_{j}^{α} p_{i}^{- α} + 1}{2})}^{(1 - α) / α}, i = 2, ..., n

are clearly strictly increasing in i = 1, …, n (for

p_{i} > p_{i + 1}

) for all

α \in (- \infty, 1)

. The case when

α \to 0

was proved in the preceding subsection.

As with any reasonable measure of randomness or uncertainty, each member of

H_{K α}

in Equation (27) is a compound measure consisting of two components: the dimension of the distribution or vector

P_{n}

and the uniformity (evenness) with which the elements of

P_{n}

are distributed. For any probability distribution

P_{n} = (p_{1}, ..., p_{n})

, this fact can be most simply represented by:

H_{K α} (P_{n}) = H_{K α} (P_{n}^{1}) H_{K α}^{*} (P_{n}), H_{K α}^{*} (P_{n}) \in [0, 1]

(29)

where

H_{K α} (P_{n}^{1}) = n - 1

for the uniform distribution

P_{n}^{1}

in Equation (5) and where

H_{K α}^{*} (P_{n}) = H_{K α} (P_{n}) / (n - 1)

reflects the uniformity (evenness) of

P_{n}

. The

H_{K α}^{*}

basically controls for n. For the distribution in Equation (6),

H_{K α}^{*} (P_{n}^{λ}) = λ

The limiting member of

H_{L}

in Equation (28) as

α \to - \infty

is defined by:

H_{L} = \sum_{i = 2}^{n} \sum_{j = 2}^{n} min {p_{i}, p_{j}} + \sum_{i = 2}^{n} p_{i} = \sum_{i = 1}^{n} \sum_{j = 1}^{n} min {p_{i}, p_{j}} - 1

(30a)

= 2 \sum_{i = 2}^{n} (i - 1) p_{i}

(30b)

where the expression in Equation (30b) can easily be seen to follow directly from Equation (30a) (remembering again the order in Equation (14)). The second expression in Equation (30a) has been briefly mentioned by Morales et al. [23] and the form in Equation (30b), divided by 2, has been suggested by Patil and Taillie [24] as one potential measure of diversity.

5. Comparative Analysis

5.1. Why the Preference for $H_{K}$

From a practical point of view, what sets

H_{K}

in Equation (3) or Equation (16) apart from any other member of the family

H_{K α}

in Equation (27) is its ease of computation. Its values can easily be computed on a hand calculator for any probability distribution

P_{n}

even when the dimension n is quite large. For other members of

H_{K α}

{(n - 1)}^{2}

pairwise means have to be computed, which becomes practically impossible without the use of a computer program even when n is not large. The computational effort for the member

H_{L}

(when

α \to - \infty

) is somewhat less than for other members. Nevertheless, the apparently simpler formula for

H_{L}

in Equation (30b) requires that all

p_{i}

be ordered as in Equation (14), which can be very tedious if done manually and nearly impossible if n is large.

The

H_{K}

is also favored over other members of

H_{K α}

when considering the agreement with some other measure based on Euclidean distance and the familiar standard deviation. Specifically, for any probability distribution

P_{n} = (p_{1}, ..., p_{n})

and

P_{n}^{1} = (1 / n, ..., 1 / n)

, consider the following linearly decreasing function of the distance

d (P_{n}, P_{n}^{1})

D (P_{n}) = (n - 1) [1 - \sqrt{\frac{n}{n - 1}} d (P_{n}, P_{n}^{1})] = (n - 1) (1 - \frac{n}{\sqrt{n - 1}} s_{n}) = (n - 1) C N V

(31)

where

s_{n}

is the standard deviation of

p_{1}, ..., p_{n}

(with devisor n rather than n – 1) and CNV is the coefficient of nominal variation [25,26]. It is clear from Equation (31) that, for

P_{n}^{0} and P_{n}^{1}

in Equation (5),

D (P_{n}^{0}) = 0

and

D (P_{n}^{1}) = n - 1

. Also, for the lambda distribution in Equation (6),

D (P_{n}^{λ}) = λ D (P_{n}^{1}) = λ (n - 1)

, so that D satisfies the value-validity condition in Equation (10). Of course, D is not zero-indifferent (see Property 3 for

H_{K}

Since the Euclidean distance and the standard deviation are such universally used measures, it is to be expected that an acceptable measure of randomness (uncertainty) should not differ substantially from D in Equation (31). From numerical examples, it is seen that values of

H_{K}

in Equation (16) tend to be closer to those of D in Equation (31) than are the values of any other member of the

H_{K α}

family in Equation (27). In order to demonstrate this fact, a computer simulation was used to generate a number of random distributions using the following algorithm. For each randomly generated probability distribution

P_{n} = (p_{1}, ..., p_{n})

, n was first generated as a random integer between 3 and 20, inclusive. Then, with the ordering in Equation (14), each

p_{i}

was generated as a random number (to 5 decimal places) within the following intervals:

\frac{1}{n} \leq p_{i} \leq 1

\frac{1 - \sum_{j = 1}^{i - 1} p_{j}}{n - (i - 1)} \leq p_{i} \leq min {p_{i - 1}, 1 - \sum_{j = 1}^{i - 1} p_{j}}, i = 2, \dots, n - 1

and finally

p_{n} = 1 - \sum_{i = 1}^{n - 1} p_{i}

For each such generated distribution, the values of

H_{K}

H_{L}

, and D were computed according to the formulas in Equations (16), (30b) and (31) as were their corresponding uniformity (evenness) indices from Equation (29). After excluding some (five) distributions

P_{n}

that were nearly equal to

P_{n}^{0} or P_{n}^{1}

in Equation (5), the results for 30 different distributions are summarized in Table 1.

It is apparent from the data in Table 1 that

H_{K}

agrees quite closely with D and clearly more so than does

H_{L}

. Exceptions are Data Sets 1, 11, and 26 when the

H_{K}

-values differ considerably from those of D, but still less so than do the

H_{L}

-values. If D is used to predict

H_{K}

(i.e., for the fitted model

{\hat{H}}_{K} = D

), it is found for the 30 data sets in Table 1 that the coefficient of determination, when properly computed as

R^{2} = 1 - \sum {(H_{K} - D)}^{2} / \sum {(H_{K} - {\bar{H}}_{K})}^{2}

[27], becomes

R^{2} = 0.98

(i.e., 98% of the variation of

H_{K}

is explained by the fitted model

{\hat{H}}_{K} = D

) as compared to

R^{2} = 0.91

in the case of

H_{L}

. Also, the root mean square (RMS) of the differences between the values of

H_{K}

and D is found to be 0.64 as compared to 1.33 for

H_{L}

and

D

. Similarly, when comparing the values of the indices

H_{K}^{*}, H_{L}^{*}, and D^{*}

, the

H_{K}^{*}

-values are considerably closer to the

D^{*}

-values than are the

H_{L}^{*}

-values, with

R M S (H_{K}^{*}, D^{*}) = 0.05

and

R M S (H_{L}^{*}, D^{*}) = 0.10.

No other member of

H_{K α}

in Equation (27) is generally in as close agreement with D as is

H_{K}

, but more so than

H_{L}

. This can be explained by the fact that (a) whenever there is a notable difference between the values of

H_{K}

and D, those of

H_{K}

tend to be less than those of D as seen from Table 1; and (b)

H_{K α} (P_{n})

is a strictly increasing function of

α

for any given

P_{n}

other than

P_{n}^{0} and P_{n}^{1}

in Equation (5).

5.2. Comparative Weights on $p_{1}, ..., p_{n}$

The difference between values of

H_{K}

and

H_{L}

as demonstrated in Table 1, or between any of the members of the

H_{K α}

family, is due to the fact that

H_{K α}

places different weights or emphases on the

p_{i}

(i = 1, …, n) depending upon

α

. When considering each pairwise mean

M_{α} (p_{i}, p_{j})

in Equation (27),

p_{i} and p_{j}

are weighted equally only when

α = 1

. Then, since (a)

M_{α} (p_{i}, p_{j})

is strictly increasing in

α (for p_{i} \neq p_{j})

and (b)

H_{K α}

is zero-indifferent (Property 3 of

H_{K}

) only for

α \leq 0

, the

H_{K 0} = H_{K}

in Equations (15) and (16) is the zero-indifferent member of

H_{K α}

that is always closest in value to

H_{K 1}

and whose pairwise means

M_{0} (p_{i}, p_{j})

are always closest to

M_{1} (p_{i}, p_{j})

for all i and j.

Besides the weights placed on each component of all pairs

(p_{i}, p_{j})

, the weights given to each individual

p_{i} (i = 2, ..., n)

can also be examined by expressing the

H_{K α}

in Equation (27) as the following weighted sum:

H_{K α} = \sum_{i = 2}^{n} [\sum_{j = 2}^{n} {(\frac{p_{i}^{α} + p_{j}^{α}}{2})}^{1 / α} (\frac{1}{p_{i}}) + 1] p_{i} = \sum_{i = 2}^{n} w_{α i} p_{i}

(32)

which shows that the weights

w_{α i} (i = 2, ..., n)

are increasing in both

α

and i. In the case of

H_{K}

, with

α \to 0

in Equation (32),

w_{0 i} = \sum_{j = 2}^{n} \sqrt{p_{j} / p_{i}} + 1

for i = 2, …, n whereas, for

H_{L}

when

α \to - \infty, w_{- \infty i} = \sum_{j = 2}^{n} min {p_{i}, p_{j}} p_{i}^{- 1} + 1

for i = 2, …, n. These weights for

H_{K}

are basically a compromise between the weights for

H_{L}

and those for

H_{K 1}

. Note also that these weights for

H_{L}

from Equation (32) can differ substantially from those in Equation (30b) as can the weights for

H_{K 1}

from Equation (32) when compared with the weights from the expression

H_{K 1} = n \sum_{i = 2}^{n} p_{i}

When comparing

H_{K}

and

H_{L}

, small

p_{i}

’s are given more weight by

H_{K}

than by

H_{L}

and the addition of low probability components to a set of events has more effect on

H_{K}

than on

H_{L}

. However, when weighting the pros and cons of such relative sensitivity to small

p_{i}

’s, it is important to keep in mind the relationship in Equation (29) and not jump to conclusion. For example, when going from

P_{4} = (0.40, 0.30, 0.20, 0.10)

Q_{6} = (0.40, 0.30, 0.20, 0.05, 0.04, 0.01)

H_{K}

increases from

H_{K} (P_{4}) = 2.32 to H_{K} (Q_{6}) = 2.91

, a 25% increase, while

H_{L} (P_{4}) = 2.00

and

H_{L} (Q_{6}) = 2.12

, a 6% increase. However, from Equation (29), the dimensional component of both

H_{K}

and

H_{L}

increased by 67% (from n − 1 = 3 to n − 1 = 5) whereas the uniformity (evenness) components decreased by 25% in the case of

H_{K}

(from 2.32/3 to 2.91/5) and 37% for

H_{L}

(from 2.00/3 to 2.12/5). In this regard the 25% increase in randomness (uncertainty) as measured by

H_{K}

does not appear unreasonable.

5.3. Inconsistent Orderings

Although all members of the family

H_{K α}

in Equation (27) have the same types of properties, including the value-validity property in Equation (10), this does not necessarily imply that different members will always produce the same results for the comparisons in Equation (2). Such lack of consistency is inevitable whenever measures are used to summarize data sets into a single number. However, as stated by Patil and Taillie [24] (p. 551), “Inconsistent measures…are a familiar problem and should not be a cause for undue pessimism”, pointing out the fact that, for instance, the arithmetic mean and the median are not consistent measures of average (central tendency) and the standard deviation and the mean absolute deviation are inconsistent measures of variability (spread). One type of consistent results for all members of

H_{K α}

is the size (order) comparison

H_{K α} (P_{n}) > H_{K α} (Q_{n})

in Equation (2a) whenever

P_{n}

is majorized by

Q_{n}

and

P_{n}

is not a permutation of

Q_{n}

. This is the result of Equation (20) and the fact, as proved above, that

H_{K α}

is strictly Schur-concave for all

α \in (- \infty, 1)

It is only when two measures

M_{1}

and

M_{2}

have a perfect linear relationship that (a) the comparison results from Equation (2) will always be consistent and (b) the compliance by

M_{1}

with the value-validity conditions in Equations (10) and (11) also implies compliance by

M_{2}

. In the case of

H_{K}

and

H_{L}

, and from the simulation results in Table 1, Pearson’s correlation coefficient between

H_{K}

and

H_{L}

is found to be r = 0.993, indicating a near perfect linear relationship between

H_{K}

and

H_{L}

. However, since the linearity is not truly perfect or exact,

H_{K}

and

H_{L}

will not always give the same results for the comparisons in Equation (2) as is evident from some of the data in Table 1.

6. Discussion

The value-validity condition in Equation (10) as a necessary requirement for valid difference comparisons as in Equation (2) is based on Euclidean distances. Such distances are also being used as a basis for the preference of

H_{K}

over other potential members of the family of entropies in Equation (27). This distance metric is the standard one in engineering and science. The use of any other “distance” measures, such as directed divergencies discussed below, would seem to require particular justification in the context of value-validity assessment.

As a simple numerical example illustrating the reasoning behind the value-validity arguments in Equations (6)–(13) and the use of Euclidean distances, consider the following probability distributions based on

P_{n}^{λ}

in Equation (6):

P_{5}^{0} = {p_{i}^{(0)}} = (1, 0, 0, 0, 0)

P_{5}^{0.5} = {p_{i}^{(0.5)}} = (0.6, 0.1, 0.1, 0.1, 0.1)

P_{5}^{1} = {p_{i}^{(1)}} = (0.2, 0.2, 0.2, 0.2, 0.2)

The Euclidean distances

d (P_{5}^{0.5}, P_{5}^{0}) = d (P_{5}^{0.5}, P_{5}^{1})

and

| p_{i}^{(0.5)} - p_{i}^{(0)} | = | p_{i}^{(0.5)} - p_{i}^{(1)} |

for

i = 1, ..., 5.

A measure of uncertainty (randomness) M that takes on reasonable numerical values within the general bounds

M (P_{n}^{0})

and

M (P_{n}^{1})

should in this example satisfy the equality

| M (P_{5}^{0.5}) - M (P_{5}^{0}) | = | M (P_{5}^{0.5}) - M (P_{5}^{1}) |

so that, with

M (P_{5}^{0}) = 0

M (P_{5}^{0.5}) = M (P_{5}^{1}) / 2

. That is, since

P_{5}^{0.5}

is the same distance from

P_{5}^{0}

as it is from

P_{5}^{1}

and each element of

P_{5}^{0.5}

is the same distance from the corresponding element of

P_{n}^{0}

as it is from that of

P_{5}^{1}

, M would reflect this fact by taking on the value

M (P_{5}^{0.5}) = M (P_{5}^{1}) / 2

. The

H_{K}

in Equation (3) or Equation (16) meets this requirement with

H_{K} (P_{5}^{0.5}) = H_{K} (P_{5}^{1}) / 2 = (5 - 1) / 2 = 2

. However, in the case of H in Equation (1),

H (P_{5}^{0.5}) = 1.23 > > H (P_{5}^{1}) / 2 = 0.80

, a substantial overstatement of the extent of the uncertainty or randomness.

A similar comparison between

H_{K}

and

H

for

P_{5}^{λ}

with

λ = 0.25

and

λ = 0.75

is given in Table 2 together with the results from some other probability distributions. The results are also given in terms of the normalized measures

H_{K}^{*} (P_{n}) = H_{K} (P_{n}) / (n - 1)

and

H^{*} (P_{n}) = H (P_{n}) / log n

as well as

D^{*} (P_{n}) = D (P_{n}) / (n - 1)

for D in Equation (31). As seen from Table 2, while

H_{K}^{*} (P_{5}^{λ}) = D^{*} (P_{5}^{λ}) = λ

H^{*} (P_{5}^{λ}) > > λ

for both

λ

-values. For all distributions in Table 2, the values of

H_{K}^{*}

are quite comparable to those of

D^{*}

, but those of

H^{*}

are all considerably greater.

The distributions

P_{10}^{(3)}

–

P_{5}^{(8)}

are included in Table 2 to exemplify the types of contradictory results that may be obtained when making the difference comparisons in Equation (2) based on

H_{K}

versus

H

. The distributions

P_{10}^{(3)}

and

P_{14}^{(4)}

in Table 2 are real data for the market shares (proportions) of the carbonated soft drinks industry in the U.S. and the world-wide market shares of cell phones, respectively (obtained by Googling “market shares” by industries). Some of the smaller market shares are not given in Table 2 because of space limitation, but were included in the computations. The

H

, which has been used as a measure of market concentration or rather of its converse, deconcentration (e.g., [28]), would indicate that these two industries have nearly the same market deconcentration. By contrast, when considered in terms of

H_{K}

for which such comparison is valid because of the value-validity property of

H_{K}

, the results in Table 2 show that the cell-phone industry is about 20% more deconcentrated than the soft-drink industry. Similarly, for the fictitious distributions

P_{4}^{(5)}

–

P_{5}^{(8)}

in Table 2, the type of difference comparison in Equation (2b) shows that

H_{K} (P_{5}^{(8)}) - H_{K} (P_{4}^{(5)}) = 1.13 > H_{K} (P_{5}^{(6)}) - H_{K} (P_{5}^{(7)}) = 0.98

whereas the result would have been the reverse had

H

been used for this comparison, with

H (P_{5}^{(8)}) - H (P_{4}^{(5)}) = 0.28 < H (P_{5}^{(6)}) - H (P_{5}^{(7)}) = 0.35

Instead of using the Euclidean metric to formulate the value-validity conditions in Section 2, one could perhaps consider other potential “distance” measures such as divergencies, also referred to as “statistical distances”. The best known such measure of the divergence of the distribution

P_{n} = (p_{1}, ..., p_{n})

from the distribution

Q_{n} = (q_{1}, ..., q_{n})

is the Kullback-Leibler divergence [29] defined as:

K L D (P_{n} : Q_{n}) = \sum_{i = 1}^{n} p_{i} log (\frac{p_{i}}{q_{i}})

This measure is directional or asymmetric in

P_{n}

and

Q_{n}

. A symmetric measure is the so-called Jensen–Shannon divergence (JSD) (e.g., [30,31,32,33]), which can be expressed in terms of the Kullback–Leibler divergence (KLD) as:

J S D (P_{n}, Q_{n}) = \frac{1}{2} K L D (P_{n} : M_{n}) + \frac{1}{2} K L D (Q_{n} : M_{n})

where

M_{n} = (P_{n} + Q_{n}) / 2

. Neither

K L D

nor

J S D

are metrics, but

\sqrt{J S D}

is [34].

Consider now the family of distributions

P_{n}^{λ}

in Equation (6) and the extreme members of

P_{n}^{0}

and

P_{n}^{1}

in Equation (5). For the case of n = 5, for example, it is found that

K L D (P_{5}^{0} : P_{5}^{0.5}) = 0.51

(

K L D (P_{5}^{0.5} : P_{5}^{0})

is undefined) and

K L D (P_{5}^{1} : P_{5}^{0.5}) = 0.33

. In the case of

J S D

M_{5} (P_{5}^{0.5}, P_{5}^{0}) =

(0.8, 0.05, ..., 0.05)

so that

J S D (P_{5}^{0.5}, P_{5}^{0}) = 0.16

. Similarly,

M_{5} (P_{5}^{0.5}, P_{5}^{1}) = (0.40, 0.15, ..., 0.15)

and

J S D (P_{5}^{0.5}, P_{5}^{1}) = 0.09

. These results differ greatly from those based on Euclidean distances for which

d (P_{5}^{0.5}, P_{5}^{0}) = d (P_{5}^{0.5}, P_{5}^{1})

. The fact that

d (P_{n}^{0.5}, P_{n}^{0}) = d (P_{n}^{0.5}, P_{n}^{1})

for all n, which is also reflected by the normalized

H_{K}^{*} (P_{n}^{0.5}) = 0.5

, corresponds to the fact that each component of

P_{n}^{0.5}

is of equal distance from the corresponding components of

P_{n}^{0}

and

P_{n}^{1}

. However, no such correspondence exists for the divergence measures

K L D

and

J S D

The derivation of

H_{K}

in Equation (3) or Equation (16) is based on the exclusion of the modal probability

p_{1}

. Of course,

p_{1}

can enter the expression for

H_{K}

since

\sum_{i = 2}^{n} p_{i} = 1 - p_{1}

. One may wonder what the result would be if a different

p_{i}

were to be excluded. If the smallest

p_{i}

is excluded, the measure would not be zero-indifferent (expansible). If any

p_{i}

other than

p_{1}

is excluded, then the measure would not be strictly Schur-concave as can be verified from the proof of Property 6 of

H_{K}

. This property is essential for any measure of uncertainty (randomness). In fact, the exclusion of

p_{1}

makes

H_{K}

unique in this regard.

It should also be emphasized that even though the entropy H in Equation (1) lacks the value-validity property, it has many of the same properties as

H_{K}

and has undoubtedly numerous useful and appropriate applications as demonstrated in the extensive published literature. The problems with H arise when it is used uncritically and indiscriminately in fields far from its origin: a statistical concept of communication theory. Both Shannon [35] and Wiener [36] cautioned against such uncritical applications. It is when H or its normalized form

H^{*}

is used as a summary measure (statistic) of various attributes (characteristics) and when its values are interpreted and compared that its lack of the value-validity property can lead to incorrect and misleading results and conclusions. This is the motivation for introducing the new entropy

H_{K}

as a measure that overcomes the lack of value validity by H.

7. Statistical Inference about $H_{K}$

Consider now the case when each

p_{i} = n_{i} / N

is the multinomial sample estimate of the unknown population probability

π_{i}

for i = 1, …, n and based on sample size

N = \sum_{i = 1}^{n} n_{i}

. It may then be of interest to investigate the potential statistical bias of

H_{K} (P_{n})

and to construct confidence intervals for the population measure

H_{K} (Π_{n})

for the population probability distribution

Π_{n} = (π_{1}, ..., π_{n})

7.1. Bias

Using bold letters to distinguish random variables from their sample values and expanding

H_{K} (P_{n})

in Equation (16) into a Taylor series about

p_{i} = π_{i}

for

i = 2, ..., n

, the following result is obtained:

\begin{matrix} H_{K} (P_{n}) = & H_{K} (Π_{n}) + \sum_{i = 2}^{n} (\frac{1}{\sqrt{π_{i}}} \sum_{i = 2}^{n} \sqrt{π_{i}} + 1) (p_{i} - π_{i}) \\ + \frac{1}{4} \sum_{i = 2}^{n} (- π_{i}^{- 3 / 2} \sum_{i = 2}^{n} \sqrt{π_{i}} + π_{i}^{- 1}) {(p_{i} - π_{i})}^{2} \\ + \frac{1}{4} \underset{i \neq j}{\sum_{i = 2}^{n} \sum_{j = 2}^{n}} (\frac{1}{\sqrt{π_{i} π_{j}}}) (p_{i} - π_{i}) (p_{j} - π_{j}) + ... \end{matrix}

(33)

Taking the expected value of each side of Equation (33) and using the well-known expectations

E (p_{i} - π_{i}) = 0

E {(p_{i} - π_{i})}^{2} = π_{i} (1 - π_{i}) / N

, and

E [(p_{i} - π_{i}) (p_{j} - π_{j})] = - π_{i} π_{j} / N

for i,

j = 2, ..., n

, it is found that:

E [H_{K} (P_{n})] = H_{K} (Π_{n}) + \frac{1}{4 N} [- (\sum_{i = 2}^{n} \sqrt{π_{i}}) \sum_{i = 2}^{n} \frac{1}{\sqrt{π_{i}}} + n - 1] + O (N^{- 3 / 2})

(34)

Equation (34) shows that the estimator

H_{K} (P_{n})

, while asymptotically unbiased, does have a small bias for finite sample size N. However, unless N is small, this bias can effectively be ignored for all practical purposes.

7.2. Confidence Interval Construction

Under multinomial sampling and based on the delta method of large sample theory (e.g., [37] (Chapter 14)), the following convergence to the normal distribution holds:

\sqrt{N} [H_{K} (P_{n}) - H_{K} (Π_{n})] \overset{d}{\to} N o r m a l (0, σ^{2})

(35)

where, in terms of partial derivatives:

σ^{2} = \sum_{i = 1}^{n} π_{i} {(\frac{\partial H_{K} (Π_{n})}{\partial π_{i}})}^{2} - {[\sum_{i = 1}^{n} π_{i} (\frac{\partial H_{K} (Π_{n})}{\partial π_{i}})]}^{2}

(36)

That is, for large N,

H_{K} (P_{n})

is approximately normally distributed with mean

H_{K} (Π_{n})

and variance Var

[H_{K} (P_{n})] = σ^{2} / N

or standard error

S E = σ / \sqrt{N}

. The limiting normal distribution in Equation (35) also holds when, as is necessary in practice, the

σ^{2}

is replaced with the estimated variance

{\hat{σ}}^{2}

by substituting the sample estimates (proportions)

p_{i}

for the population probabilities

π_{i}

for i = 1, …, n, resulting in the estimated standard error

\overset{\land}{S E} = \hat{σ} / \sqrt{N}

. It then readily follows from Equations (16) and (36) that:

\overset{\land}{S E} = N^{- 1 / 2} {n [H_{K} (P_{n}) - \sum_{i = 2}^{n} p_{i}] + H_{K} (P_{n}) [1 - H_{K} (P_{n})]}^{1 / 2}

(37)

As a simple numerical example, consider the sample distribution

P_{4} = (0.4, 0.3, 0.25, 0.05)

based on sample size N = 100. From Equation (16),

H_{K} (P_{4}) = 2.22

and, from Equation (37),

\overset{\land}{S E} = 0.19

. Therefore, because of Equation (35), an approximate 95% confidence interval for the population measure

H_{K} (Π_{n})

becomes

2.22 \pm 1.96 (0.19)

, or [1.85, 2.59]. Statistical hypotheses such as

H_{0} : H_{K} (Π_{n}) = H_{K} (Π_{m})

versus

H_{1} : H_{K} (Π_{n}) > H_{K} (Π_{m})

can also be tested based on Equations (35) and (37).

8. Conclusions

Since the ubiquitous Boltzmann–Shannon entropy H is only valid for making size (order or “larger than”) comparisons, the entropy

H_{K}

is being introduced as an alternative measure of randomness (uncertainty) that is more informative than H in the sense that

H_{K}

can also be used for making valid difference comparisons as in Equations (2b) and (2c). The

H_{K}

, which is a particular member of the family of entropies

H_{K α}

and is basically a compromise between the members

H_{K 1}

and

H_{L} = H_{K (- \infty)}

, has the types of desirable properties one would reasonably expect of a randomness (uncertainty) measure. One of the differences between

H_{K}

and

H_{L}

is that small probabilities have a greater influence on

H_{K}

than on

H_{L}

. The addition of some small probability events causes a larger increase in

H_{K}

than in

H_{L}

, but causes a smaller decrease in the uniformity (evenness) index

H_{K}^{*}

than in

H_{L}^{*}

as defined in Equation (29).

Besides being computationally most simple, which is certainly a practical advantage,

H_{K}

is also that member of

H_{K α}

that appears to be most nearly linearly (and decreasingly) related to the Euclidean distance between the points

P_{n} = (p_{1}, ..., p_{n})

and

P_{n}^{1} = (1 / n, ..., 1 / n)

or to the standard deviation

s_{n}

p_{1}, ..., p_{n}

. The

s_{n}

is the usual measure of variability (spread) for a set of data, although it is not resistant against “outliers” (extreme and suspect data points). However, “outliers” are not a concern when dealing with probabilities

p_{i} (i = 1, ..., n)

. Therefore,

s_{n}

cannot justifiably be criticized for being excessively influenced by large or small

p_{i}

’s, with the same argument extending to

H_{K}

Acknowledgments

The author would like to thank the reviewers for constructive and helpful comments.

Conflicts of Interest

The author declares no conflict of interest.

References

Von Boltzmann, L. Weitere Studien über das Wärmegleichgewicht unter Gasmolekülen. In Sitzungsberichte der Kaiserliche Akademie der Wissenschaften, II Abteil; (Vol. 66, Pt. 2); K.-K. Hof- und Staatsdruckerei in Commission bei C. Gerold’s Sohn: Wien, Austria, 1872; pp. 275–370. (In German) [Google Scholar]
Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423, 623–656. [Google Scholar] [CrossRef]
Klir, G.J. Uncertainty and Information: Foundations of a Generalized Information Theory; Wiley: Hoboken, NJ, USA, 2006. [Google Scholar]
Ruelle, D. Chance and Chaos; Princeton University Press: Princeton, NJ, USA, 1991. [Google Scholar]
Han, T.S.; Kobayashi, K. Mathematics of Information and Coding; American Mathematical Society: Providence, RI, USA, 2002. [Google Scholar]
Shannon, C.E.; Weaver, W. The Mathematical Theory of Communication; University of Illinois Press: Urbana, IL, USA, 1949. [Google Scholar]
Arndt, C. Information Measures: Information and Its Description in Science and Engineering; Springer: Berlin/Heidelberg, Germany, 2004. [Google Scholar]
Kapur, J.N. Measures of Information and Their Applications; Wiley: New Delhi, India, 1994. [Google Scholar]
Kvålseth, T.O. Entropy. In International Encyclopedia of Statistical Science; Lovric, M., Ed.; Springer: Berlin/Heidelberg, Germany, 2011; Part 5; pp. 436–439. [Google Scholar]
Rényi, A. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA, 20 June–30 July 1961; University of California Press: Berkeley, CA, USA, 1961; Volume 1, pp. 547–561. [Google Scholar]
Peitgen, H.-O.; Jürgens, H.; Saupe, D. Chaos and Fractals: New Frontiers of Science, 2nd ed.; Springer: New York, NY, USA, 2004. [Google Scholar]
Tsallis, C. Possible generalization of Boltzmann–Gibbs statistics. J. Stat. Phys. 1988, 52, 479–487. [Google Scholar] [CrossRef]
Norwich, K.H. Information, Sensation, and Perception; Academic Press: San Diego, CA, USA, 1993. [Google Scholar]
Aczél, J.; Daróczy, Z. On Measures of Information and Their Characterizations; Academic Press: New York, NY, USA, 1975. [Google Scholar]
Kvålseth, T.O. Entropy evaluation based on value validity. Entropy 2014, 16, 4855–4873. [Google Scholar] [CrossRef]
Hand, D.J. Measurement Theory and Practice; Wiley: London, UK, 2004. [Google Scholar]
Kvålseth, T.O. The Lambda distribution and its applications to categorical summary measures. Adv. Appl. Stat. 2011, 24, 83–106. [Google Scholar]
Marshall, A.W.; Olkin, I.; Arnold, B.C. Inequalities: Theory of Majorization and Its Applications, 2nd ed.; Springer: New York, NY, USA, 2011. [Google Scholar]
Bullen, P.S. A Dictionary of Inequalities; Addison Wesley Longman: Essex, UK, 1998. [Google Scholar]
Aczél, J. Lectures on Functional Equations and Their Applications; Academic Press: New York, NY, USA, 1966. [Google Scholar]
Hardy, G.H.; Littlewood, J.E.; Pólya, G. Inequalities; Cambridge University Press: Cambridge, UK, 1934. [Google Scholar]
Ebanks, B. Looking for a few good means. Am. Math. Mon. 2012, 119, 658–669. [Google Scholar]
Morales, D.; Pardo, L.; Vajda, I. Uncertainty of discrete stochastic systems: General theory and statistical inference. IEEE Trans. Syst. Man Cyber Part A 1996, 26, 681–697. [Google Scholar] [CrossRef]
Patil, G.P.; Taillie, C. Diversity as a concept and its measurement. J. Am. Stat. Assoc. 1982, 77, 548–567. [Google Scholar] [CrossRef]
Kvålseth, T.O. Coefficients of variation for nominal and ordinal categorical data. Percept. Mot. Skills 1995, 80, 843–847. [Google Scholar] [CrossRef]
Kvålseth, T.O. Variation for categorical variables. In International Encyclopedia of Statistical Science; Lovric, M., Ed.; Springer: Berlin/Heidelberg, Germany, 2011; Part 22; pp. 1642–1645. [Google Scholar]
Kvålseth, T.O. Cautionary note about R². Am. Stat. 1985, 39, 279–285. [Google Scholar]
Nawrocki, D.; Carter, W. Industry competitiveness using Herfindahl and entropy concentration indices with firm market capitalization data. Appl. Econ. 2010, 42, 2855–2863. [Google Scholar] [CrossRef]
Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Match. Stat. 1951, 22, 79–86. [Google Scholar] [CrossRef]
Lin, J. Divergence measures based on Shannon entropy. IEEE Trans. Inf. Theory 1991, 37, 145–151. [Google Scholar] [CrossRef]
Wong, A.K.C.; You, M. Entropy and distance of random graphs with application to structural pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 1985, PAMI-7, 599–609. [Google Scholar] [CrossRef]
Sagar, R.P.; Laguna, H.G.; Guevara, N.L. Electron pair density information measures in atomic systems. Int. J. Quantum Chem. 2011, 111, 3497–3504. [Google Scholar] [CrossRef]
Antolin, J.; Angulo, J.C.; López-Rosa, S. Fisher and Jensen–Shannon divergences: Quantitative comparisons among distributions. Application to position and momentum atomic densities. J. Chem. Phys. 2009, 130, 074110. [Google Scholar] [CrossRef] [PubMed]
Endres, D.M.; Schindelin, J.E. A new metric for probability distributions. IEEE Trans. Inf. Theory 2003, 49, 1858–1860. [Google Scholar] [CrossRef] [Green Version]
Shannon, C.E. The bandwagon. IRE Trans. Inf. Theory 1956, 2, 3. [Google Scholar] [CrossRef]
Weiner, N. What is information theory? IRE Trans. Inf. Theory 1956, 2, 48. [Google Scholar] [CrossRef]
Bishop, Y.M.M.; Fienberg, S.E.; Holland, P.W. Discrete Multivariate Analysis: Theory and Practice; MIT Press: Cambridge, MA, USA, 1975. [Google Scholar]

Table 1. Values of

H_{K}

H_{L}

, and D in Equations (16), (30b), and (31) and of their corresponding uniformity (evenness) indices from Equation (29) based on 30 randomly generated probability distributions.

**Table 1.** Values of $H_{K}$ , $H_{L}$ , and D in Equations (16), (30b), and (31) and of their corresponding uniformity (evenness) indices from Equation (29) based on 30 randomly generated probability distributions.
Data Set	n	$H_{K}$	$H_{L}$	D	$H_{K}^{*}$	$H_{L}^{*}$	$D^{*}$
1	16	2.17	1.28	3.89	0.14	0.09	0.26
2	19	11.14	10.32	11.22	0.62	0.57	0.62
3	3	0.96	0.82	0.97	0.48	0.41	0.49
4	18	4.28	3.21	5.05	0.25	0.19	0.15
5	15	1.84	1.40	2.08	0.13	0.10	0.15
6	15	12.65	11.42	12.57	0.90	0.82	0.90
7	13	10.37	9.93	10.34	0.86	0.83	0.86
8	15	4.13	3.01	3.76	0.30	0.22	0.27
9	7	5.38	4.83	5.25	0.90	0.81	0.88
10	4	0.78	0.65	0.86	0.26	0.22	0.29
11	12	1.67	1.23	3.83	0.15	0.11	0.35
12	17	14.71	14.56	14.71	0.92	0.91	0.92
13	14	4.09	3.88	4.10	0.31	0.30	0.32
14	8	6.04	5.39	5.95	0.86	0.77	0.85
15	17	10.01	9.03	10.13	0.63	0.56	0.63
16	5	0.57	0.41	0.74	0.14	0.10	0.19
17	10	4.85	3.82	5.10	0.54	0.42	0.57
18	5	1.00	0.73	1.27	0.25	0.18	0.32
19	5	2.05	1.16	2.09	0.51	0.40	0.52
20	19	12.00	11.05	12.06	0.67	0.61	0.67
21	19	13.78	13.64	13.78	0.77	0.76	0.77
22	17	6.93	6.33	7.40	0.43	0.40	0.46
23	12	8.16	7.71	8.17	0.74	0.70	0.74
24	20	13.56	11.98	13.83	0.71	0.63	0.73
25	17	3.47	3.07	3.58	0.22	0.19	0.22
26	13	1.15	0.72	2.29	0.10	0.06	0.19
27	14	10.69	9.66	10.65	0.82	0.74	0.82
28	12	1.68	1.66	1.68	0.15	0.15	0.15
29	20	9.72	6.88	11.09	0.51	0.36	0.58
30	9	5.23	4.20	5.36	0.65	0.53	0.67

Table 2. Comparative results for

H

in Equation (1) and

H_{K}

in Equation (16) and their normalized forms as well as

D^{*}

from Equation (31) for various probability distributions.

**Table 2.** Comparative results for $H$ in Equation (1) and $H_{K}$ in Equation (16) and their normalized forms as well as $D^{*}$ from Equation (31) for various probability distributions.
$P_{n}^{(i)}$	$H_{K}$	$H_{K}^{*}$	$H$	$H^{*}$	$D^{*}$
$P_{5}^{(1)} = P_{5}^{0.75} = (0.40, 0.15, 0.15, 0.15, 0.15)$	3.00	0.75	1.50	0.93	0.75
$P_{5}^{(2)} = P_{5}^{0.25} = (0.80, 0.05, 0.05, 0.05, 0.05)$	1.00	0.25	0.78	0.48	0.25
$P_{10}^{(3)} = (0.26, 0.15, 0.15, 0.10, ..., 0.03)$	6.87	0.76	2.09	0.91	0.77
$P_{14}^{(4)} = (0.27, 0.25, 0.15, 0.07, ..., 0.01)$	8.22	0.63	2.10	0.80	0.67
$P_{4}^{(5)} = (0.40, 0.30, 0.20, 0.10)$	2.32	0.77	1.28	0.92	0.74
$P_{5}^{(6)} = (0.50, 0.20, 0.10, 0.10, 0.10)$	2.45	0.61	1.36	0.84	0.61
$P_{5}^{(7)} = (0.70, 0.10, 0.10, 0.05, 0.05)$	1.47	0.37	1.01	0.63	0.37
$P_{5}^{(8)} = (0.30, 0.20, 0.20, 0.20, 0.10)$	3.45	0.86	1.56	0.97	0.84

© 2016 by the author; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Kvålseth, T.O. On the Measurement of Randomness (Uncertainty): A More Informative Entropy. Entropy 2016, 18, 159. https://doi.org/10.3390/e18050159

AMA Style

Kvålseth TO. On the Measurement of Randomness (Uncertainty): A More Informative Entropy. Entropy. 2016; 18(5):159. https://doi.org/10.3390/e18050159

Chicago/Turabian Style

Kvålseth, Tarald O. 2016. "On the Measurement of Randomness (Uncertainty): A More Informative Entropy" Entropy 18, no. 5: 159. https://doi.org/10.3390/e18050159

APA Style

Kvålseth, T. O. (2016). On the Measurement of Randomness (Uncertainty): A More Informative Entropy. Entropy, 18(5), 159. https://doi.org/10.3390/e18050159

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

On the Measurement of Randomness (Uncertainty): A More Informative Entropy

Abstract

1. Introduction

2. Conditions for Valid Difference Comparisons

3. The New Entropy

3.1. Derivation of $H_{K}$

3.2. Properties of $H_{K}$

4. Generalization of $H_{K}$

5. Comparative Analysis

5.1. Why the Preference for $H_{K}$

5.2. Comparative Weights on $p_{1}, ..., p_{n}$

5.3. Inconsistent Orderings

6. Discussion

7. Statistical Inference about $H_{K}$

7.1. Bias

7.2. Confidence Interval Construction

8. Conclusions

Acknowledgments

Conflicts of Interest

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI

Article Menu

On the Measurement of Randomness (Uncertainty): A More Informative Entropy

Abstract

1. Introduction

2. Conditions for Valid Difference Comparisons

3. The New Entropy

3.1. Derivation of H K

3.2. Properties of H K

4. Generalization of H K

5. Comparative Analysis

5.1. Why the Preference for H K

5.2. Comparative Weights on p 1 , ... , p n

5.3. Inconsistent Orderings

6. Discussion

7. Statistical Inference about H K

7.1. Bias

7.2. Confidence Interval Construction

8. Conclusions

Acknowledgments

Conflicts of Interest

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI

3.1. Derivation of $H_{K}$

3.2. Properties of $H_{K}$

4. Generalization of $H_{K}$

5.1. Why the Preference for $H_{K}$

5.2. Comparative Weights on $p_{1}, ..., p_{n}$

7. Statistical Inference about $H_{K}$