An Entropy-Based Tool to Help the Interpretation of Common-Factor Spaces in Factor Analysis

In factor analysis, extracting interpretable factors is important for practical data analysis. In order to carry it out, methods for factor rotation have been studied, e.g., varimax [1] and orthomax [2] for orthogonal rotations and oblimin [3] and orthoblique [4] for oblique rotations. The basic idea for factor rotation in factor analysis is owed to the criteria of simple structures of factor analysis models by Thurstone [5], and the methods of factor rotation are constructed with respect to maximizations of variations of the squared factor loadings in order to derive simple structures of factor analysis models. Let $X_i$ be manifest variables, let ${\xi }_j$ be latent variables (common factors), let ${\epsilon }_i$ be unique factors related to $X_i$, and finally, let ${\lambda }_{ij}$ be factor loadings that are weights of common factors ${\xi }_j$ to explain $X_i$. Then, the factor analysis model is given as follows: where To derive simple structures of factor analysis models, for example, in the varimax method, the following variation function of squared factor loadings is maximized with respect to factor loadings: where $\overline{{\lambda }^2}=\frac{1}{pm}{\sum }_{i=1}^p{\sum }_{j=1}^m{\lambda }_{ij}^2$. In this sense, the basic factor rotation methods can be viewed as those for exploratively analyzing multidimensional common-factor spaces. The interpretation of factors is made according to manifest variables with large weights in common factors. As far as we know, novel methods for factor rotation have not been investigated except for rotation methods similar to the above basic ones. In real data analyses, manifest variables are usually classified into some groups of variables in advance that may have common factors and concepts for themselves. For example, suppose we have a test battery including the following five subjects: Japanese, English, Social Science, Mathematics, and Natural Science. It is then reasonable to classify the five subjects into two groups, {Japanese, English, Social Science} and {Mathematics, Natural Science}. In such cases, it is meaningful to determine common factors related to the two manifest variable groups. For this objective, it is useful to develop a novel method to derive the common factors based on a factor contribution measure. In conventional methods of factor rotation, for example, as mentioned above, variation function (2) for the varimax method is not related to factor contribution.

An entropy-based method for measuring factor contribution was proposed by [6], and the method can measure factor contributions to manifest variables vectors and can decompose the factor contributions into those of manifest subvectors and individual manifest variables. By using the method, we can derive important common factors related to the manifest subvectors and the manifest variables. The aim of the present paper is to propose a new method for deriving simple structures based on entropy, that is, extracting common factors easy to interpret. In Section 2, an entropy-based method for measuring factor contribution [6] is reviewed to apply its properties for deriving simple structures in factor analysis models. Section 3 discusses canonical correlation analysis between common factors and manifest variables, and the contributions of common factors to the manifest variables are decomposed into components related to the extracted pairs of canonical variables. A numerical example is given to demonstrate the approach. In Section 4, canonical correlation analysis is applied to obtain common factors easy to interpret, and the contributions of the extracted factors are measured. Numerical examples are given to illustrate the present approach, and finally, Section 5 provides discussions and conclusions to summarize the present approach.

First, in order to derive factor contributions, factor analysis model (1) with error terms ${\epsilon }_i, i=1,2,\dots ,p$, which are normally distributed, can be discussed in the framework of generalized linear models (GLMs) [7]. A general path diagram among manifest variables $X_i, i=1,2,\dots ,p$ and common factors ${\xi }_j, j=1,2,\dots ,m$ in the factor analysis model is illustrated in Figure 1. The conditional density functions of manifest variables of $X_i, i=1,2,\dots ,p$, given the factors ${\xi }_j, j=1,2,\dots ,m$, are expressed as follows: Let ${\theta }_i={\sum }_{j=1}^m{\lambda }_{ij}{\xi }_j$ and $d\left(x_i,{\omega }_i^2\right)=-\frac{x_i^2}{2{\omega }_i^2}-\log \sqrt{2\pi {\omega }_i^2}$. Then, the above density function is described in a GLM framework as According to the local independence of the manifest variables in factor analysis model (1), the conditional density function of $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$ given $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ is expressed as

Let $g\left(\mathit{\xi }\right)$ be the joint density function of common-factor vector $={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$; let $f_i\left(x_i\right)$ be the marginal density functions of $X_i, i=1,2,\dots ,p$; and let us set where “KL” stands for “Kullback–Leibler information” [8]. From (3) and (4), we have The above quantities (7) and (8) are interpreted as the signal-to-noise ratios for dependent variables $X_i$ and predictors ${\theta }_i$; and the signal-to-noise ratio for dependent-variable vectors $\mathit{X}$ and common-factor vector $\mathit{\xi }$, respectively.

From (7) and (8), the following theorem can be derived [6]:

Consistently, the following theorem, which is actually an extended version of Corollary 1 in [6], can be also obtained:

Following Eshima et al. [6], the contribution of factor vector $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ to manifest variable vector $\mathit{X}={\left(X_1, X_2,\dots ,X_p\right)}^T$ is thus defined as so that, in Theorem 2, the contributions of factor vector $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ to manifest variable vectors ${\mathit{X}}^{\left(a\right)}, a=1,2,\dots ,A$ are defined by Let ${\mathit{\xi }}^{\backslash \mathit{j}}$ be subvectors of all variables ${\xi }_i$ except ${\xi }_j$ from $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$, i.e., and let $\mathrm{KL}\left(\mathit{X},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)$ and $\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},{\mathit{\xi }}^{\backslash \mathit{j}}|{\xi }_j\right)$ be the conditional Kullback–Leibler information as defined in (5) and (6). The contributions of common factors ${\xi }_j$ are defined by

Finally, the following decomposition of $\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)$ holds for orthogonal factors ([6], Theorem 3):

The entropy coefficient of determination (ECD) [9] between $\mathit{\xi }$ and $\mathit{X}$ is defined by so that the total relative contribution of factor vector $\mathit{\xi }$ to manifest variable vector $\mathit{X}$ in entropy can be defined as while, for a single factor ${\xi }_j$, two relative contribution ratios can be defined: (see [6] for details).

Second, factor analysis model (1) in a general case is discussed. Let $\Sigma$ be the variance–covariance matrix of manifest variable vector $\mathit{X}={\left(X_1,X_2,\dots ,X_p\right)}^T$; let $\mathsf{\Omega }$ be the $p\times p$ variance–covariance matrix of unique factor vector $\mathsf{\epsilon }={\left({\epsilon }_1,{\epsilon }_2,\dots ,{\epsilon }_p\right)}^T$; let $\mathsf{\Lambda }$ be the $p\times m$ factor loading matrix of ${\lambda }_{ij}$; and let $\mathsf{\Phi }$ be the correlation matrix of common-factor vector $\mathit{\xi }={\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right)}^T$. Then, model (1) can be expressed as and we have Now, the above discussion is extended in a general factor analysis model (1) with the following variance–covariance matrix of $\mathit{X}$ and $\mathsf{\epsilon }$: Let $\mathit{\theta }=\mathsf{\Lambda }\mathit{\xi }$ be the predictor vector of manifest variable vector ${\mathit{X}}^T=\left(X_1,X_2,\dots ,X_p\right)$. Then, the contribution of common-factor vector $\mathit{\xi }$ to manifest variable vector $X$ is defined by the following generalized signal-to-noise ratio: where $\tilde{\mathsf{\Omega }}$ is the cofactor matrix of $\mathsf{\Omega }$. The signal is $\mathrm{tr}\tilde{\mathsf{\Omega }}\mathsf{\Lambda }\mathsf{\Phi }{\mathsf{\Lambda }}^T$ and the noise $\left|\mathsf{\Omega }\right|$, and both are positive. Hence, the above quantity is defined as the explained entropy with the factor analysis model, and the same notation $KL\left(\mathit{X}, \mathit{\xi }\right)$ as above is used, having to do with the Kullback–Leibler information for the factor analysis model with normal distribution errors (4). Similarly, in the general model, as in (9), signal-to-noise ratio (11) is decomposed into so the above theorems hold true as well. Thus, the results mentioned above are applicable to factor analysis models with error terms with non-normal distributions.

In order to derive interpretable factors from the common-factor space, we propose taking advantage of the results of canonical correlation analysis applied to manifest variables and common factors. This approach can be referred to as “canonical factor analysis” [10]. In the factor analysis model (1), the variance–covariance matrix of $\mathit{X}={\left(X_1, X_2,\dots ,X_p\right)}^T$ and $\mathit{\xi }={\left({\xi }_1, {\xi }_2,\dots ,{\xi }_m\right)}^T$ is given by (10). Then, we have the following theorem:

In the proof of the above theorem, we have It implies that Theorem 4 shows that the contribution of common-factor vector $\mathit{\xi }$ to manifest variable vector $\mathit{X}$ is decomposed into those of canonical common factors ${\eta }_j$, i.e., Let us assume According to the entropy-based criterion in Theorem 4, the order of importance of canonical common factors is that of canonical correlation coefficients. The interpretation of factors ${\eta }_j$ can be made with the corresponding manifest canonical variables $V_j$ and the factor loading matrix of canonical common factors $\mathit{\eta }=\mathit{F}\mathit{\xi }$. For the canonical common factors, the factor loading matrix can be obtained as ${\mathsf{\Lambda }}^{\ast }=\mathsf{\Lambda }{\mathit{F}}^{-1}$. We refer to the canonical correlation analysis in Theorem 4 as canonical factor analysis [10].

Notice that we also have From the above theorem, the results of the canonical factor analysis do not depend on the initial common factors ${\xi }_j$ in factor analysis model (1). For factor analysis model (1), it follows that implying that where $R_i$ are the multiple correlation coefficients between manifest variables $X_i$ and factor vector $\mathit{\xi }=\left({\xi }_1,{\xi }_2,\dots ,{\xi }_m\right), i=1,2,\dots ,p$.

From (9) in Theorem 2, $\mathrm{KL}\left(\mathit{X},\mathit{\xi }\right)$ is decomposed into those for manifest variable subvectors ${\mathit{X}}^{\left(a\right)}$, $\mathrm{KL}\left({\mathit{X}}^{\left(a\right)},\mathit{\xi }\right), a=1,2,\dots ,A$. Thus, we have the following theorem: From Theorem 2, the theorem follows. □

In this sense,

To derive important common factors, the above theorem can be used. In many of the data in factor analysis, manifest variables can be classified into subsets that have common concepts (factors) to be measured. For example, in the data used for Table 1, it is meaningful to classify the five variables into two subsets ${\mathit{X}}^{\left(1\right)}=\left(X_1,X_2,X_3\right)$ and ${\mathit{X}}^{\left(2\right)}=\left(X_4,X_5\right)$, where the first subset is related to the liberal arts and the second one is related to the sciences. In $\left({\mathit{X}}^{\left(1\right)},\mathit{\xi }\right)$ and $\left({\mathit{X}}^{\left(2\right)},\mathit{\xi }\right)$, it is possible to derive the latent ability for the liberal arts and that for the sciences, respectively.