Derivative Formulas and Gradient of Functions with Non-Independent Variables

We used to work with models defined through functions that include non-independent variables such as correlated input variables. This is also the case for models defined via functions with independent input variables and equations or inequations connecting such inputs; functions subject to constraints involving input variables and/or the model output. Knowing the key role of partial derivatives at a given point in (i) the mathematical analysis of functions and convergence, (ii) Poincaré inequalities ([1,2]) and equalities ([3,4]), (iii) optimization and active subspaces ([5,6]), (iv) implicit functions ([7,8]) and (v) differential geometry (see, e.g., [9,10,11]), it is interesting and relevant to have formulas that enable the calculations of partial derivatives of functions in the presence of non-independent variables, including the gradients. Of course, such formulas must account for the dependency structures among the model inputs, including the constraints imposed on such inputs.

Actual partial derivatives aim to calculate the partial derivatives of functions, taking into account the relationships between the input variables ([12]). For instance, let us consider the function $f:{\mathbb{R}}^3\to \mathbb{R}$ given by $f(x,y,z)=2x{y}^2{z}^3$ under the constrained equation $h(x,y,z)=0$, where h is any smooth function. Using $f_x^{\prime }$ to represent the formal partial derivative of f with respect to. x, which is the partial derivative of f when considering other inputs as constant or independent, the chain rule yields the following partial derivative:

In probability and statistics, the point $(x,y,z)$ represents a realization or a sample value of the implicit random vector $(X,Y,Z)$. The quantities $\frac{\partial y}{\partial x},\frac{\partial z}{\partial x}$, and $\frac{\partial f}{\partial x}$ do not have a definite meaning when the variables $(X,Y,Z)$ are npn-independent. At this point, determining $\frac{\partial y}{\partial x},\frac{\partial z}{\partial x}$ and $\frac{\partial f}{\partial x}$ is often challenging without supplementary assumptions such as the choice of directions or paths. When $(X,Z)$ or $(X,Y)$ are independent and using the equation $h(x,y,z)=x+2y-z=0$, we can write ([12]):

It is clear that the actual partial derivatives are not unique, as such derivatives rely on two different paths or assumptions. While each supplementary assumption can make sense in some cases, it cannot always be guaranteed by the constrained equation $h(x,y,z)=0$ in general. Indeed, when all the initial input variables are dependent or correlated, the above partial derivatives are no longer valid, even for a linear function h, and it is worth finding the correct relationship between the input variables and the partial derivatives, including the gradient.

In differential geometry (see, e.g., [9,10,11]), using the differential of the function f, that is, the gradient of f is defined as the dual of $df$ with respect to a given tensor metric. Obviously, different tensor metrics will yield different gradients of the same function. While the Euclidean metric, given by $\gamma :=d{x}^2+d{y}^2+d{z}^2$, is more appropriate for independent inputs, finding the appropriate metrics is challenging in general. Indeed, the first fundamental form (in differential geometry) requires the Jacobian matrix of the inputs to define the associated tensor metric.

For non-independent variables with F as the joint cumulative distribution function (CDF), the bivariate dependency models ([13]) and multivariate dependency models ([3,14,15]), including the conditional and inverse Rosenblatt transformation ([16,17]), establish formal and analytical relationships among such variables using either CDFs or the corresponding copulas or new distributions that look like and behave like a copula ([18]). A dependency function characterizes the probabilistic dependency structures among these variables. For a d-dimensional random vector of non-independent variables, the dependency models express a subset of $d-1$ variables as a function of independent variables, consisting of the remaining input and new independent variables.

In this paper, we propose a new approach for calculating the partial derivatives of functions, considering the dependency structures among the input variables. Our approach relies on dependency models. By providing known relationships between the dependent inputs (including constraints imposed on inputs or outputs), dependency models can be regarded as global and probabilistic implicit functions (see Section 2.2). Such dependency models are used to determine the dependent Jacobian and the tensor metric for non-independent variables. The contributions of this paper are threefold:

In Section 2, we first review the dependency models of dependent input variables, including correlated variables. Next, we derive interesting properties of these models regarding the calculus of partial derivatives and probabilistic implicit functions. By coupling dependency functions with the function of interest, we extend the actual partial derivatives of functions with non-independent variables in Section 3. To avoid their drawbacks, the dependent partial derivatives of functions with non-independent variables are provided in Section 4. The gradient and Hessian matrix of these functions are derived in Section 5 using the framework of differential geometry. We provide an application in Section 6 and conclude this work in Section 7.

In probability theory, it is common to treat input variables as random vectors with associated CDFs. For instance, for the inputs that take their values within a known domain or space, the Bayesian framework allows for assigning a joint distribution, known as a prior distribution, to such inputs. Without additional information about the inputs, it is common to use non-informative prior distributions, such as uniform distributions or Gaussian distributions associated with higher values of the variances (see e.g., [19]).

Functions with non-independent variables include many types of models encountered in practice. An example is the models defined via a given function and equations or inequations connecting its inputs. The resulting inputs that comply with such constraints are often dependent or correlated. In the subsequent discussion, we use probability theory to characterize non-independent variables (see Definition 1).

Using $\mathcal{N}\left(\mathbf{0},\Sigma \right)$ for the multivariate normal distribution, we can check that a function that includes $\mathbf{X}\sim \mathcal{N}\left(\mathbf{0},\Sigma \right)$ as inputs, with $\Sigma$ being a non-diagonal covariance matrix, is a member of the class of functions defined by

This section discusses the calculus of partial derivatives of functions with non-independent variables using only one relationship among the inputs, such as the DM given by Equation (5). The usual assumptions made are:

(A2) The joint (resp. marginal) CDF is continuous and has a density function $\rho >0$ on its open support;

(A3) Each component of the dependency function $r_{{\pi }_{j,k}}$ is differentiable with respect to $X_{{\pi }_{j,k}}$;

(A4) Each component of the function f, that is, $f_{\ell }$ with $\ell \in \{1,\dots ,n\}$, is differentiable with respect to each input.

Without loss of generality and for the sake of simplicity, here, we suppose that $n=1$. Namely, we use ${\mathcal{I}}_{d\times d}\in {\mathbb{R}}^{d\times d}$ for the identity matrix, and ${\mathsf{O}}_{d\times d_1}\in {\mathbb{R}}^{d\times d_1}$ for the null matrix. It is common to use $\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_k}}$ for the formal partial derivatives of f with respect to each input of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ (i.e., the derivatives obtained by considering inputs as independent) with $k=1,\dots ,K$. Thus, the formal gradient of f (i.e., the gradient with respect to the Euclidean metric) is given by

Keeping in mind the function $c_{{\pi }_{j,k}}(·)$, the partial derivatives of each component of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{j,k}}$ are given by We use $J_i^{\left({\pi }_{j,k}\right)}$ for the $i^{th}$ element of $J^{\left({\pi }_{j,k}\right)}$. For instance, $J_j^{\left({\pi }_{j,k}\right)}=1$ and $J_{d_k}^{\left({\pi }_{j,k}\right)}$ represent the partial derivative of $X_{{\pi }_{d_k,k}}$ with respect to $X_{{\pi }_{j,k}}$. It is worth recalling that $J^{\left({\pi }_{j,k}\right)}$ is a vector-valued function of $(X_{{\pi }_{j,k}},{\mathbf{Z}}_k)$, and Lemma 1 expresses $J^{\left({\pi }_{j,k}\right)}$ as a function of ${\mathbf{x}}_{{\mathit{\pi }}_k}$ only.

Again, $J_{\ell }^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$ with $\ell \in \{1,\dots ,d_k\}$ represents the ${\ell }^{th}$ component of $J^{\left({\pi }_{j,k}\right)}\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$, as provided in Lemma 1. Using these components and the chain rule, Theorem 2 provides the actual partial derivatives of functions with non-independent variables (i.e., $\frac{{\partial }_{a}f}{\partial \mathbf{x}}=:J_{{\mathit{\pi }}_k}^a\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right))$, which are the derivatives obtained by utilizing only one dependency function given by Equation (5).

The results from Theorem 2 are based on only one dependency function, which uses $X_{{\pi }_{j,k}}$ as the explanatory input. Thus, the actual Jacobian $J^a\left(\mathbf{x}\right)$ and the actual partial derivatives of f provided in (10)–(12) will change with the choice of the explanatory input $X_{{\pi }_{j,k}}$ for every $j\in \{1,\dots ,d_k\}$. All these possibilities are not surprising. Indeed, while no additional explicit assumption is necessary for calculating the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{j,k}}$ (i.e., $J^{\left({\pi }_{j,k}\right)}$), we implicitly keep the other variables fixed when calculating the partial derivative of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$, that is, $\frac{J^{\left({\pi }_{j,k}\right)}}{J_i^{\left({\pi }_{j,k}\right)}}$ for each $i\ne j$. Such an implicit assumption is due to the reciprocal rule used to derive the results (see Appendix C). In general, the components of ${\mathbf{X}}_{\sim {\pi }_{j,k}}$, such as $X_{{\pi }_{i_1,k}}$ and $X_{i_2,k}$, are both functions of $X_{{\pi }_{j,k}}$ and $Z_{{\pi }_{1,k}}$ at the very least. Thus, different possibilities of the actual Jacobians are based on different implicit assumptions, making it challenging to use the actual partial derivatives. Further drawbacks of the actual partial derivatives of f are illustrated in Section 3

This section aims to derive the first- and second-order partial derivatives of functions with non-independent variables, without relying on any additional assumption, whether explicit or implicit. Essentially, we calculate or compute the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$ using only the dependency function that includes $X_{{\pi }_{i,k}}$ as an explanatory input, which can be expressed as follows: By using the above dependency function, the partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$ are given as follows (see (9)): It should be noted that $J^{\left({\pi }_{i,k}\right)}$ does not require any supplementary assumption, as $X_{{\pi }_{i,k}}$ and ${\mathbf{Z}}_{{\mathit{\pi }}_{i,k}}$ are independent. Thus, $d_k$ different DMs are necessary to derive the dependent Jacobian and partial derivatives of f (see Theorem 3).

Although the results from Theorem 3 require different DMs, these results are more comprehensive than the actual partial derivatives because no supplementary assumption is available for each non-independent variable.

To derive the second-order partial derivatives of f, we use $f_{x_i{x}_j}^{{}^{″}}$ to denote the formal cross-partial derivative of f with respect to $x_i$ and $x_j$, and $H_{{\mathit{\pi }}_k}:={\left(f_{x_{j_1}{x}_{j_2}}^{{}^{″}}\right)}_{j_1,j_2\in {\mathit{\pi }}_k}$ to denote the formal or ordinary Hessian matrices of f restricted to ${\mathbf{X}}_{{\mathit{\pi }}_k}$ for $k=1,\dots ,K$. In the same sense, we use $H_{{\mathit{\pi }}_{k_1},{\mathit{\pi }}_{k_2}}:={\left(f_{x_{j_1}{x}_{j_2}}^{{}^{″}}\right)}_{j_1\in {\mathit{\pi }}_{k_1},j_2\in {\mathit{\pi }}_{k_2}}$ to denote the formal cross-Hessian matrix of f restricted to $({\mathbf{X}}_{{\mathit{\pi }}_{k_1}},{\mathbf{X}}_{{\mathit{\pi }}_{k_2}})$ for every pair $k_1,k_2\in \{1,\dots K\}$ with $k_1\ne k_2$. To ensure the existence of the second-order partial derivatives, we assume that:

(A5) The function f is twice (formal) differentiable with respect to each input;

(A6) Every dependency function $r_{{\pi }_{j,k}}$ is twice differentiable with respect to $X_{{\pi }_{j,k}}$.

By considering the $d_k$ DMs of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ (i.e., ${\mathbf{X}}_{\sim {\pi }_{i,k}}=r_{{\pi }_{i,k}}\left(X_{{\pi }_{i,k}},{\mathbf{Z}}_{{\pi }_{i,k}}\right)$, with $i=1,\dots ,d_k$) used to derive the dependent Jacobian, we can write for the second partial derivatives of ${\mathbf{X}}_{{\mathit{\pi }}_k}$ with respect to $X_{{\pi }_{i,k}}$. By using $diag\left(\mathbf{x}\right)\in {\mathbb{R}}^{d\times d}$ to represent a diagonal matrix with $\mathbf{x}$ as its diagonal elements and for all $k\in \{2,\dots ,K\}$, Theorem 4 provides the dependent second-order partial derivatives (i.e., $\frac{{\partial }^{2}f}{{\partial }^2\mathbf{x}}$).

Although Section 4 provided the partial derivatives and cross-partial derivatives of f, it is misleading to think that the infinitesimal increment of f, given by $f({\mathbf{X}}_{{\mathit{\pi }}_k}+ϵ{\mathbf{e}}_{{\mathit{\pi }}_{j,k}})-f\left({\mathbf{X}}_{{\mathit{\pi }}_k}\right)$, should result in the individual effect quantified by $\frac{\partial f\left({\mathbf{X}}_{{\mathit{\pi }}_k}\right)}{\partial x_{{\pi }_{j,k}}}ϵ$ with ${\mathbf{e}}_{{\pi }_{j,k}}:={\left[0,\dots ,0,\underset{{\pi }_{j,k}^{\mathrm{th}}\mathrm{position}}{\underbrace{1}},0,\dots ,0\right]}^T$ and $ϵ>0$. Indeed, when moving $X_{{\pi }_{j,k}}$, it leads to partial movements of the other variables, and the effects we observe (i.e., $\nabla f_{{\mathbf{x}}_{{\mathit{\pi }}_k}}^T{J}^{\left({\pi }_{j,k}\right)}$) can also be attributed to other variables. The dependency structures of these effects are described by the dependent Jacobian matrix $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$ (see Equation (13)). Therefore, the definition of the gradient and Hessian of f with non-independent variables requires the introduction of a tensor metric or a Riemannian tensor.

In differential geometry, the function of the form for every ${\pi }_{i,k}\in {\mathit{\pi }}_k$ can be seen as a parametrization of a manifold ${\mathcal{M}}_k$ in ${\mathbb{R}}^{d_k}$. The $d_k$ column entries of the dependent Jacobian matrix $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)\in {\mathbb{R}}^{d_k\times d_k}$ span a local $m_k$-dimensional vector space, also known as the tangent space at ${\mathbf{x}}_{{\mathit{\pi }}_k}$, where $m_k$ is the rank of $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$, indicating the number of linearly independent columns in $J_{{\mathit{\pi }}_k}^d\left({\mathbf{x}}_{{\mathit{\pi }}_k}\right)$.