Double Contingency of Communications in Bayesian Learning

We work in the $C^{\infty }$-smooth category. Take a possibly non-compact and possibly disconnected manifold X equipped with a volume form $dvo{l}_X$. Note that a discrete set is a 0-dimensional manifold on which a positive function is a volume form. Suppose that each point x of the manifold X presents a possible action of a person. A positive function $f:X\to {\mathbb{R}}_{>0}$ on X is called a density. If its integral ${\left|f\right|}_{dvo{l}_X}:={\int }_{X}fdvo{l}_X$ is finite, it defines the probability $f/{\left|f\right|}_{dvolX}$ on X. Suppose that the selection of an action x is weighted by a density $f_0$ on X. In our story, the person believes that a density ${\rho }_x:Y\to {\mathbb{R}}_{>0}$ on another manifold $(Y,dvo{l}_Y)$ depends on his action x. That is why the person perceives a given point $y_0\in Y$ by multiplying the density $f_0$ by the function which is called the likelihood of the datum $y_0\in Y$. The perception updates the prior density $f_0$ to the posterior density $f_1\left(x\right):=l\left(x\right)f_0\left(x\right)={\rho }_x\left(y_0\right)f_0\left(x\right)$. Indeed $f_1/{\left|f_1\right|}_{dvo{l}_X}$ is the Bayesian posterior probability provided that $f_0/{\left|f_0\right|}_{dvo{l}_X}$ is the prior probability. The only change from the description in [1] is the aim of the learning, i.e., prediction is replaced with action. Although the word action has an active meaning, an activity consisting of countless actions would be a chain of automatic adaptations to the environment.

It is natural to symmetrize the above setting by altering the roles of X and Y. Specifically, we further suppose that a point y of the second manifold Y parameterizes a density ${\rho }_y^{\prime }:X\to {\mathbb{R}}_{>0}$ of the first manifold X, and the perception of a datum $x_0\in X$ by the second person updates a prior density $g_0:Y\to {\mathbb{R}}_{>0}$ on the second manifold to the posterior density $g_1\left(y\right)={\rho }_y^{\prime }\left(x_0\right)g_0\left(y\right)$. This models the double contingency of Parsons [2]. We further modify it as follows. Fix volume forms $dvo{l}_X$, $dvo{l}_Y$, and $dvo{l}_{X\times Y}$ on X, Y, and $X\times Y$, respectively. Take densities $f_0:X\to {\mathbb{R}}_{>0}$, $g_0:Y\to {\mathbb{R}}_{>0}$, and $\lambda :X\times Y\to {\mathbb{R}}_{>0}$. Suppose that prior densities $f_0$ and $g_0$, respectively, changes to the posterior densities This models the double contingency of Luhmann [3]. We say that $f_0$ is coupled with $g_0$ in the mutual learning through Luhmann’s potential $\lambda$ on the product $X\times Y$. Since the potential $\lambda$ is also a density, it can be coupled with a density ${\sigma }_0$ on another manifold Z. Specifically, if there is a datum $({(x,y)}_0,z_0)$ and a density ${\tau }_0:(X\times Y)\times Z\to {\mathbb{R}}_{>0}$, the pair of two persons can change the tendency of its action selection. This mathematics enables us to consider the double contingency not only between persons but also between systems. Here we suppose that the points ${(x,y)}_0$ and $(x_0,y_0)$ are given as the same point. We emphasize that what we are discussing is not how the datum appears objectively, but how we perceive it or how we learn from it subjectively. We discuss in Section 4 the discordance between ${(x,y)}_0$ and $(x_0,y_0)$ to understand a proposition in Luhmann’s systems theory saying that no system is a subsystem.

Shannon [6] introduced the notion of entropy in information theory. As for continuous distributions, Jaynes [7] pointed out that the notion of relative entropy is rather foundational to the notion of entropy Indeed, the entropy takes all real values even for normal distributions, whereas the relative entropy is non-negative for any pair of distributions, where the non-negativity is obvious from $log(1/t)\ge 1-t$ and is called the Gibbs inequality. Further, if we multiply the volume form by a positive constant, the entropy changes while the relatrive entropy does not. In any case, putting $f=f_1/f_0$ and using the volume form $f_{0}dvo{l}_X$, we have Note that we cannot put $f_0=1$ unless $dvo{l}_X$ is finite. If we multiply the volume form by a non-constant density, the relative entropy varies in general. We notice that the choice of the volume forms in the above mutual learning does not affect the result of the learning.

The information geometry [4], as well as its partial globalization by the author [1,5], starts with a family of probability distributions. Slightly more generally, we consider a manifold W equipped with a volume form $dvo{l}_W$ and a family ${\left\{h_x\right\}}_{x\in X}$ of densities with finite total masses on it. We regard the parameter space X as a manifold, and define the function $\phi :X\times X\to {\mathbb{R}}_{\ge 0}$ on its square by The information geometry focuses on the 3-jet of $\phi$ at the diagonal set $\Delta \subset X\times X$. From the Gibbs inequality, the symmetric quadratic tensor defined by the 2-jet of $\phi$ is positive semi-definite. If it is positive definite, it defines a Riemannian metric called the Fisher–Rao metric. Then the symmetric cubic tensor defined by the 3-jet of the anti-symmetrization $\phi (x,y)-\phi (y,x)$ directs a line of torsion-free affine connections passing through the Levi-Civita connection of the Fisher–Rao metric. This line of connections is the main subject of the information geometry. On the other hand, developing the global geometry in [1,5], we define Luhmann’s potential for mutual learning between two copies of X as We couple a prior density $f_0:X\to {\mathbb{R}}_{>0}$ on the first factor with a prior density $g_0:X\to {\mathbb{R}}_{>0}$ on the second factor through the potential $\lambda$ on the product $X\times X$. Here, the mutual learning updates $f_0$ and $g_0$, respectively, to the posterior densities Note that the function $\phi$ changes if we multiply the volume form $dvo{l}_W$ by a non-constant density in general. Thus, the choice of the volume form is crucial. The volume form $dvo{l}_X$ might be related to the Fisher–Rao metric, although the choice of $dvo{l}_X$ is indeed irrelevant to the mutual learning. We can also imagine that the other volume forms $dvo{l}_{X\times X}$ and $dvo{l}_W$ have been determined in earlier mutual learnings “connected” to the current one.

Let W be a 0-dimensional manifold with $N+1$ unit components, i.e., $W=\{0,\dots ,N\}$ with volume form ${\displaystyle dvo{l}_W=1}$. A point x of the open N-simplex with the standard volume form $dvo{l}_X$ presents a categorical distribution (i.e., a finite distribution) on W. We take the product manifold $X\times X$ with Luhmann’s potential Suppose that the prior densities are the constants $f_0\left(x\right)\equiv 1$ and $g_0\left(y\right)\equiv 1$ on the first and second factors of $X\times X$. Then, the iteration of mutual Bayesian learning yields where the overlines denote arithmetic means ${\displaystyle \overline{x^0}=\frac{x_0^0+\cdots +x_{n-1}^0}{n}}$ etc.

We notice that the probability $g_n/{\left|g_n\right|}_{dvo{l}_X}$ for the posterior density $g_n$ on the second factor of $X\times X$ is known as the Dirichlet distribution.

We identify the set W with the 0-skeleton of the closure $\mathrm{cl}\left(X\right)$ of the open N-simplex $X\subset {\mathbb{R}}^{N+1}$. If the prior is the flat Dirichlet distribution $\mathrm{Dir}(1,\cdots ,1)$, the Bayesian learning from categorical data $x_0^{\prime },\cdots ,x_{n-1}^{\prime }\in W$ yields the posterior $\mathrm{Dir}((1,\cdots ,1)+x_0^{\prime }+\cdots +x_N^{\prime })$. This is the conventional Bayesian learning from categorical data. On the other hand, the above probability $g_n/{\left|g_n\right|}_{dvolX}$ is the Dirichlet distribution $\mathrm{Dir}((1,\cdots ,1)+x_0+\cdots +x_N)$. Here we believe that the data $x_k\in X$ obey the probability $\lambda (x,y_k)/{\left|\lambda (x,y_k)\right|}_{dvo{l}_X}$, which we can consider as a continuous version of the categorical distribution. Imagine that a coarse graining of the data $x_k$ on X yields data $x_k^{\prime }$ obeying a categorical distribution on the 0-skeleton W of the closure of X. Then the probability $g_n/{\left|g_n\right|}_{dvolX}$ for the new data $x_k^{\prime }$ reaches the posterior probability of the conventional Bayesian learning.

The following is the summary of the above.

Thus, the answer to Problem 1 is affirmative in this case.

In the case where X is the space of normal distributions, we would like to change the coordinates of the second factor of the product $X\times X$ to make the expression simpler, although one can reach the same result through a straightforward calculation.

A von Mises distribution $M_k\left(m\right)$ with a fixed large concentration $k(\gg 1)$ is a circular analogue of a normal distribution with a fixed small variance that is parametrized by a point m of $X=\mathbb{R}/2\pi \mathbb{Z}$. Its density is proportional to the restriction of the function $exp(kcos(m)x+ksin(m\left)y\right)$ to the circle $W=\left\{\right(x,y)\mid x=cosw,y=sinw,w\in \mathbb{R}/2\pi \mathbb{Z}\}$ with $dvo{l}_W=dw$. Then, using the easy formula ${\displaystyle {\int }_0^{2\pi }exp(kcosx)sinxdx=0}$, we obtain the following expression of the relative entropy: where c is a positive constant. (When $k\in \mathbb{Z}$, using modified Bessel, we have ${\displaystyle c=\frac{k{I}_1\left(k\right)}{I_0\left(k\right)}}$.) Thus, Luhmann’s potential is $\lambda (m,m^{\prime })=exp(-c(1-cos(m-m^{\prime }))$. We put $f_0\left(m\right)\equiv 1$ and $g_0\left(m^{\prime }\right)\equiv 1$. Then, the iteration of mutual Bayesian learning on the torus $X\times X$ yields On the other hand, the conventional Bayesian learning on W yields the posterior probability density proportional to $exp\left(nk\overline{cos}\left(m\right)cosw+nk\overline{sin}\left(m\right)sinw\right)$, which looks like $g_n\left(w\right)$. This suggests the affirmative answer to Problem 1.

We have observed that the answer to Problem 1 is affirmative in some cases. Specifically, the mutual Bayesian learning covers at least a non-empty area of parametric statistics. The author expects that it could cover the whole from some consistent perspective.

In our setup of mutual learning, a system must be organized as the product of two manifolds with Luhmann’s potential before each member learns. Further, the potential is the result of an earlier mutual learning in which the system was a member. In Luhmann’s description [3], the unit of society is not the agent of an action but a communication or rather a chain of communications. In mathematics, a manifold is locally a product of manifolds and is characterized as the algebraic system of functions on it. By analogy, Luhmann’s society seems to be a system of relations between certain systems of functions. Some authors criticize his theory for failing to acquire individual identity, but an individual is a relation between identities that are already represented by manifolds.

As a matter of course, reality cannot be explained by theories. Instead, a theory which can better explain something on reality is chosen. In Section 2.2, we have assumed that ${(x,y)}_0$ and $(x_0,y_0)$ are given as the same point in reality. Then there are two possibilities: (1) The potential $\lambda$ is updated by using ${(x,y)}_0$ as a component of a datum, or (2) the mutual learning of $(x_0,y_0)$ is performed under the undated potential $\lambda$. The discordance between (1) and (2) does not affect the reality. Further, there is no consistent hierarchy among Luhmann’s systems that choose either (1) or (2), and therefore there is no system that is a proper subsystem of another system. Perhaps, the social system chooses either (1) or (2), which can better explain the “fact” in relation to other “facts” in a story on reality. Undertaking all of the above, the notion of autopoiesis that Maturana and Varela [8] found in living organisms can be the foundation of Luhmann’s socio-cybernetics.

In objective probability theory, one considers a continuous probability distribution as the limit of a family of finite distributions presented by relative frequency histograms and the entropy of the limit as the limit of the entropies. Since the entropy of a finite distribution whose support is not a singleton is positive, a distribution with negative entropy, e.g., a normal distribution with small variance, does not appear. On the other hand, we take the position of subjective probability theory, and regard a positive function on a manifold that has the unit mass with respect to a fixed volume form as a probability density. From our point of view, the relative entropy between two probability densities is essential as it is non-negative; it presents the information gain; and it does not change (while even the sign of entropy does change) by multiplying the volume form by any positive constant. We notice that an objective probability is a subjective probability, and not vice-versa.

We know that the lowest entropy at the beginning of the universe must be relative to higher entropy in the future. In this regard, the total amount of information decreases as the order of time. However, it is still possible that the amount of consumable information increases, and perhaps that is how this world works. Here we would like to distinguish the world from the universe, even though they concern the same reality and therefore communicate with each other. The world consists of human affairs, including the possible variations of knowledge on facts in the universe—there is no love in the universe, but love is the most important consumable thing in the world. We consider that the notion of complexity in Luhmann’s systems theory concerns such consumability as it relates to coupling of systems. Now the problem is not the total reserve of information, but how to strike it and refine it like oil. At present, autopoiesis is gaining ground against mechanistic cybernetics. Our research goes against this stream: Its goal is to invent a learning machine to exploit information resources to be consumed by humans and machines.

In this paper, we have quickly gone from the general definition of mutual learning to a discussion of the special mutual learning via relative entropy. However, it may be worthwhile to stop and study various types of learning according to purely geometric interests. For example, the result of previous work [1] is apparently related to the geometry of dual numbers, and fortunately this special issue includes a study [9] on a certain pair of dual number manifolds. Considering mutual learning for pairs of related manifolds such as this is something to be investigated in the future.

In addition, in proceeding to the case of the mutual learning via relative entropy, one basic problem was left unaddressed: Given a non-negative function $\phi$ on a squared manifold $M\times M$ that takes zero on the diagonal set, can we take a family of probability densities with parameter space M so that the relative entropy induces the function $\phi$?