Jarzyski’s Equality and Crooks’ Fluctuation Theorem for General Markov Chains with Application to Decision-Making Systems

Over the last 20 years, several advances in thermodynamics have led to the development of results relating equilibrium quantities to nonequilibrium trajectories. Those advances have crystallized in a new area of research, nonequilibrium thermodynamics, where these relations play a major role [1,2,3]. Among them, two of the most remarkable are Jarzynski’s equality [4,5,6] and Crooks’ fluctuation theorem [7,8], for which experimental evidence has been reported in several contexts: unfolding and refolding processes involving RNA [9,10], electronic transitions between electrodes manipulating a charge parameter [11], the rotation of a macroscopic object inside a fluid surrounded by magnets where the current of a wire attached to the macroscopic object is manipulated [12], and a trapped-ion system [13,14].

These two results have been derived under several assumptions in the context of nonequilibrium thermodynamics, including both deterministic [5,6] and stochastic dynamics [4,7,8,15]. Moreover, it has been argued that both results can be obtained as a consequence of Bayesian retrodiction in a physical context [16]. Here, we derive both of them using only the concepts from the theory of Markov chains. This allows us to both distinguish the mathematical from the physical assumptions underlying them and, thus, to make them available for application in other areas where the framework of thermodynamics may be useful. The distinction between mathematical and physical assumptions will be of particular importance for the definition of work, as we will see, since the usual definition based on physical considerations leads to an asymmetry of the definition in processes that run either forward or backward in time—see Figure 1 for a simple example. This is relevant, for instance, when analysing trajectories in terms of their work value, if we do not know whether they were recorded in the forward direction or whether they have been generated by playing them backwards. Ideally, we would like to be able to ascribe work values directly to trajectories without any additional information.

One of the application areas where the framework of thermodynamics has recently been investigated outside the realm of physics is the analysis of simple learning systems [1,18,19,20,21,22,23,24,25,26] and, in particular, the problem of decision-making under uncertainty and resource constraints [22,27,28,29,30,31]. The basic analogy follows from the idea that decision-making involves two opposing forces: (i) the tendency of the decision-maker toward better options (equivalently, to maximize a function called utility) and (ii) the restrictions on this tendency given by the limited information-processing capabilities of the decision-maker, which prevents him/her from always picking the best option and is usually modelled by a bound on the entropy of the probability distribution that describes the decision-maker’s behaviour. Thermodynamic systems are also explained in terms of two opposing forces, the first being the energy, which the system tries to minimize, and the second being the entropy, which prevents the minimization of the energy to its full extent. Thus, in both cases, we formally deal with optimization problems under information constraints, and thus, we can conceptualize both decision-making and thermodynamics in terms of information theory. In particular, we can consider the environment in which a decision is being made or a thermodynamic system is immersed as a source of information (in the form of either utility or energy), which, due to the noise (modelled by entropy), reaches the decision-making or thermodynamic system with some error. This results in an imperfect response by the system.

The analogy between thermodynamics and decision-making is not restricted to the equilibrium case [22,27,28,29,30,31], but can be taken further from equilibrium to nonequilibrium systems. In particular, the aforementioned fluctuation theorems of Jarzynski and Crooks have been previously suggested to apply to decision-makers that adapt to changing environments [32]. In this previous work, hysteresis and adaptation were investigated in decision-makers, however, based on the physical convention of defining work differently for forward and backward processes. Here, we improve on the work there by replacing this convention with a different energy protocol that naturally entails a symmetric definition of work and by weakening the assumptions that are actually needed in order for the fluctuation theorems to hold in the context of general Markov chains. Given the fact that the literature on this topic belongs for the most part to thermodynamics, we adopt the thermodynamic notation here. In particular, we consider energy functions instead of utilities and take Markov chains as the starting point.

Our manuscript is organized as follows. In Section 2, we introduce the notions of work and other thermodynamic concepts that are inherent to Markov chains, that is, in contrast to the formalism in physics [7,8,15], we start with the assumption of a Markov chain and deduce all other concepts from that without the need to presuppose the existence of an external energy function. We discuss under what conditions these concepts are uniquely specified from the Markov chain. In Section 3, we use this framework to weaken the derivation of Jarzynski’s equality (Theorem 1) in the context of decision-making that was presented in [32]. In Section 4, we prove Crooks’ fluctuation theorem (Theorem 2 and Corollary 2) within the same setup. In particular, we use an additional assumption that is not mandatory in Crooks’ work [7,8,15], but here is needed given the inherent nature of our definition of work. In fact, we provide an example in which the new requirement is violated and, as a consequence, Crooks’ theorem is false everywhere (Proposition 3). In Section 5, we discuss how the concepts we have developed can be applied to decision-making systems.

Notice, for simplicity, we develop the results for discrete-time Markov chains with finite state spaces. However, the ideas can be translated, for example, to continuous state spaces by assuming for densities of Markov kernels the properties we use here for transition matrices. We include a few more details regarding this scenario in the discussion (Section 5), where we also briefly address the case of continuous-time Markov chains.

Jarzynski’s equation was originally derived for deterministic dynamics [5,6] (see also [35]) and later extended to stochastic dynamics [4] using a Master equation approach. Shortly after that, it was shown in the non-deterministic Markov chain context relying on assumptions about the time reversal of the dynamics [8]. In Theorem 1 below, we see that, in the context of Markov chains, Jarzynski’s equality is a straightforward consequence of the definitions of work and free energy. Importantly, it does not require any assumptions regarding time reversal, in contrast to the requirements in a previous decision-theoretic approach to fluctuation theorems in [32].

For the proof of Jarzynski’s equality for Markov chains, the basic observation is that we start with the expected value of a quantity closely related to the equilibrium distributions of our initial Markov chain $\mathit{X}$, namely $e^{-\beta W\left(\mathit{X}\right)}$. With this in mind, we define a new Markov chain $\mathit{Y}$ using the transition matrices of $\mathit{X}$ and the equilibrium distributions of the individual steps. In particular, we define it in a way such that we cancel the dependency of $e^{-\beta W\left(\mathit{X}\right)}$ on $\mathit{X}$ and end up with a constant, whose expected value over $\mathit{Y}$ is the constant itself. We include the details in the following theorem.

The main purpose of proving Theorem 1 is to show that Jarzynski’s equality can be obtained under milder conditions than the ones that were considered before in decision-making. It should be noted that, in contrast to the usual approaches such as [8], where $\mathit{Y}$ is assumed to be the time reversal of $\mathit{X}$, here, it is just a convenient mathematical object for the proof. A similar approach to Jarzynski’s equality in the context of continuous-time Markov chains can be found in [36]. Our approach to the discrete-time case in Theorem 1 is much simpler, since we do not need measure-theoretic concepts nor smoothness assumptions.

The original derivation of Crooks’ fluctuation theorem for Markovian dynamics [7,8,15] was carried out using a definition of work different from the one in (4). In this section, we derive the theorem for Markov chains using (4) and comment on the difference between these approaches in the discussion. As discussed in the Introduction, an additional hypothesis is needed for the result to hold in our setup. We derive Crooks’ fluctuation theorem using this additional assumption in Theorem 2 and Corollary 2 below, and then, in Proposition 3, we provide an example where this requirement is violated and the theorem is false everywhere.

Before proving the intermediate results and, finally, Crooks’ fluctuation theorem, let us briefly sketch the procedure we follow throughout this section. We start with Proposition 1, where we use the same Markov chain $\mathit{Y}$ that we defined in the proof of Theorem 1 and obtain, along similar lines, a more precise relation between $\mathit{X}$ and $\mathit{Y}$, in particular between the probability of some realization of $\mathit{X}$ and that of the same realization (with the events taking place in reversed order) of $\mathit{Y}$. As a matter of fact, we show that, for a given $\mathit{X}$, $\mathit{Y}$ is (roughly) the only Markov chain fulfilling such a relation. Then, in Proposition 2, we show how the driving signals of $\mathit{X}$ and $\mathit{Y}$ are related. In order to do so, we exploit the relation between the equilibrium distributions of $\mathit{X}$ and $\mathit{Y}$, which comes from the fact the both Markov chains share the same equilibrium distributions. By combining these two propositions, we reach a relation between the probability distributions of the driving signals of $\mathit{X}$ and $\mathit{Y}$ in Theorem 2. Lastly, in Corollary 2, we impose an extra condition on the equilibrium distributions of $\mathit{X}$ in order to obtain Crooks’ fluctuation theorem in its usual form. We proceed now to show the details involved in the argument we just presented. We start by proving Proposition 1.

We say $\mathit{X}$ is microscopically reversible [15] if (13) is satisfied for $\mathit{Y}$ being the time reversal of $\mathit{X}$, i.e., if the unique Markov chain $\mathit{Y}$ that exists by Proposition 1 has initial distribution $p_N$ and transition matrices ${\left(M_{N-n+1)}\right)}_{n=1}^N$. Notice, if this property holds, then (14) relates the probability of observing a realization $\mathit{x}$ of a Markov chain $\mathit{X}$ with that of observing the reversed realization ${\mathit{x}}^R$ in the time reversal $\mathit{Y}$ of $\mathit{X}$, that is when starting with the equilibrium distribution of the last environment and choosing according to the same conditional probabilities, but in reversed order. This is the case if and only if the transition matrices of $\mathit{X}$ satisfy detailed balance, as we show in the following lemma.

In particular, this means that, if $\mathit{X}$ satisfies detailed balance, then the time reversal $\mathit{Y}$ of $\mathit{X}$ satisfies (14), that is for any family of energies $\mathit{E}={\left(E_n\right)}_{n=0}^N$ of $\mathit{X}$, where $W^d≔W_{\mathit{X}}^d$ is the dissipated work of $\mathit{X}$. Thus, in this case, dissipated work is an unambiguous measure of the discrepancy between the probability of observing a realization of $\mathit{X}$ and the probability of observing the same trajectory in reversed order in the time reversal of $\mathit{X}$. We have, hence, an unambiguous measure of hysteresis (see Section 5).

Before showing Theorem 2 and Crooks’ fluctuation theorem, we relate the work of $\mathit{X}$ with that of its time reversal.

If detailed balance holds, then (16) and (17) relate the work along a realization $\mathit{x}$ of $\mathit{X}$ with the reversed realization ${\mathit{x}}^R$ of its time reversal $\mathit{Y}$. More precisely, we obtain the following corollary.

The non-constant term $E_1\left(x_0\right)-E_0\left(x_0\right)$ in (16), which remains even when $\mathit{X}$ satisfies detailed balance, follows from an asymmetry between $\mathit{X}$ and $\mathit{Y}$. In particular, $\mathit{X}$ goes from $E_0$ to $E_N$, whereas $\mathit{Y}$ goes from $E_N$ to $E_1$, because while $X_0\sim p_0\propto e^{-\beta E_0}$ and the stationary distribution of $M_N$ is $p_N\propto e^{-\beta E_N}$, which is also the initial distribution of $\mathit{Y}$, the stationary distribution of the final transition matrix ${\hat{M}}_N$ of $\mathit{Y}$ is $p_1\propto e^{-\beta E_1}$ (and not $p_0$). Furthermore, while $\mathit{X}$ may begin with a change in the energy function, since $p_1\ne p_0$ is allowed, $\mathit{Y}$ does not, as $p_N$ is both the initial distribution of $\mathit{Y}$ and the stationary distribution of ${\hat{M}}_1$. An example can be found in Figure 2. This asymmetry is erased if we assume that $p_0$ is the stationary distribution of $M_1$, in which case, for Markov chains $\mathit{X}$ that satisfy detailed balance, the work along any realization of $\mathit{X}$ has the opposite sign of the work along the reversed realization of $\mathit{Y}$. That is, thermodynamic work becomes odd under time reversal.

The following theorem contains Crooks’ fluctuation theorem as the special case when $\mathit{X}$ satisfies detailed balance (see Corollary 2 below).

As the special case when each transition matrix of $\mathit{X}$ in Theorem 2 satisfies detailed balance, we obtain Crooks’ fluctuation theorem modified by the additional assumption of $p_0=p_1$.

Notice, in most of the literature on Crooks’ fluctuation theorem, one writes ${\mathbb{P}}^F\left(W\right)$ for the probability $\mathbb{P}\left(W_{\mathit{X},\mathit{E}}\right)$ of the work along the so-called forward process $\mathit{X}$ and ${\mathbb{P}}^B\left(W\right)$ for the probability $\mathbb{P}\left(W_{\mathit{Y},\hat{\mathit{E}}}\right)$ of the work along the so-called backward process $\mathit{Y}$ (the time reversal of $\mathit{X}$), so that, by Corollary 2, under detailed balance, Equation (18) reads

The condition that $p_0$ is the stationary distribution of $M_1$ is not necessary in Crooks’ original work [7,8,15,17]. Nonetheless, it is fundamental in our approach: Crooks’ fluctuation theorem can even be false for every work value if $p_0$ is not the stationary distribution of $M_1$, as we show in Proposition 3.

The bridge connecting thermodynamics and decision-making is an optimization principle directly inspired by the maximum entropy principle [27,30,37,38]. In particular, given a finite set of possible choices S, the optimal behaviour is given by the distribution $p\in {\mathbb{P}}_S$ that optimally trades utility and uncertainty according to the following optimization principle: where ${\mathbb{P}}_S$ is the set of probability distributions over S, H is the Shannon entropy, ${\mathbb{E}}_q[·]$ denotes the expected value over $q\in {\mathbb{P}}_S$, and $U:S\to \mathbb{R}$ is a utility function, that is a function that assigns larger values to options in S that are preferred by the decision-maker. Note that the main difference between (20) and the maximum entropy principle is the the substitution of an energy function $E:S\to \mathbb{R}$ by a utility U, which behaves as a negative energy function $U=-E$ (in the sense that it is the force that opposes uncertainty). As a result of their similarity, both principles yield the same result, namely the Boltzmann distribution: where $\beta$ is a trade-off parameter between uncertainty and utility/energy and Z is a normalization constant.