Measuring instruments

The Self-Compassion Scale (SCS; Neff, 2003) measures six aspects of self-compassion in situations of a perceived difficult time. The scale includes 26 items rated on a 5-point Likert-type Scale of frequency (1 = almost never; 5 = almost always). The subscale Self-Kindness (SK) represents the ability of taking care of oneself and being warm towards oneself when encountering failure situations. Common Humanity (CH) reflects the personal understanding that suffering is part of the shared human experience. Mindfulness (MI) is a non-judgmental state of mind in which individuals observe their thoughts and feelings as they are, without over-identification or without trying to suppress or deny them. They are seen as either “negative” or “positive”. The scale measures the degree to which individuals display self-kindness against self-judgment, common humanity versus isolation, and mindfulness versus over-identification. The Over-identification (OI), Isolation (IS) and Self-Judgment (SJ) subscales are therefore scored negatively. The total score of the scale is calculated by the average of individual subscales, while a negatively scored item must be transformed. In Slovak language, we did back translation of the scale and the discrepancies were discussed and decided by consensus. The items of the English and Slovak versions of SCS are in Appendix 1.

The Forms of Self-criticism/Attacking & Self-Reassuring Scale (FSCRS; Gilbert et al., 2004) is a 22-item instrument which was developed to determine the level self-criticism and the ability of self-reassurance. On a 5-point Likert Scale, participants rated the extent to which various statements are true about themselves (1 = not at all like me; 5 = extremely like me). The questionnaire consists of 22 items, which measure how a person feels and thinks in severe, adverse life situations. The scale comprises three scales: Inadequate Self (IS) which focuses on feelings of personal inadequacy, Hated Self (HS) measuring the desire to hurt or punish oneself, and Reassured Self (RS) which is an ability of self-affirmation.

The Levels of Self-Criticism Scale (LOSC; Thompson & Zuroff, 2004) was developed to measure two dysfunctional forms of negative self-evaluation: Comparative Self-Criticism (CSC) and Internalized Self-Criticism (ISC). The scale contains 22 items and measures Comparative Self-Criticism (CSC), which is defined as the negative view of the one’s self, acquired by comparison with other people. Internalized Self-Criticism (ISC) is the negative view of the self, which is formed by comparing oneself with one’s own personal standards and objectives. It consists of 12 items measuring the Comparative Self-Criticism subscale and 10 items measuring the Internalized Self-Criticism subscale. Participants answered the items on a Likert scale from 1 = not at all to 7 =very well.

The Self-Compassion and Self-Criticism Scales (SCCS; Falconer, King, & Brewin, 2015) measures two dimensions: Self-criticism (SCR) and Self-compassion (SCO). It consists of five self-threatening scenarios describing various situations that have the potential to induce people to varying degrees of self-criticism or self-compassion. On a 7-point scale (1-not at all 7- to highly), respondents indicated the extent to which they would respond reassuringly, soothingly, contemptuously, compassionately, critically and harshly to these situations.

The Research Sample

The research sample included 1,181 participants of whom 402 were males (34%) and 779 females (66%). The mean age was 30.30 years (SD = 12.40), and ranged from 18 to 82 years. 667 male and female respondents were single, and 514 were in relationship. With regard to education, 152 respondents (13%) had completed primary education, 572 (48%) had completed secondary school education and 457 (39%) had a university degree.

An independent research sample was used only for validating our factor analysis. This sample included 676 participants out of which 15 % were male and 85% were female. Their mean age was 29.90 years (SD = 11.21).

Data Collection

Data was collected gradually over two and half years within a research grant focused on self-criticism and self-compassion. Data was obtained by convenience sampling; questionnaires were distributed on paper and also in a digital form via social networks. The authors declare that there are no conflicts of interest and confirm complying with APA ethical principles in the treatment of individuals participating in the research. The research has been carried out in accordance with The Code of Ethics of the corresponding University.

Data Analysis

For data recording, we used the program SPSS Statistics-20 and for statistical processing the software R (version 3. 1. 3, R Core Team, 2015) packages psych (Revelle, 2015), mirt (Chalmers, 2012), and mokken (Van der Ark, 2012) were used. The procedure was as follows: (1) Descriptive analysis: standard distributive properties of items, as well as testing univariate normal distributions of items and multivariate normal distribution of scale (with respect to the ordinal nature of the data, we do not assume a normal distribution); (2) Analysis of the overall reliability of the instrument and reliability of each dimension; (3) Verification of convergent validity; (4) IRT confirmatory factor analysis with three models: six-factor correlated model, bifactor model, two-tier model which was also validated on the independent sample; (5) Mokken’s nonparametric IRT analysis to verify the scalability; (6) Analysis of DTF (i.e. the differential test functioning) across gender and relationship status; (7) In the case of absence of DTF, we made the comparison of responses of men and women as well as between singles and people in relationship through the extension of the non-parametric Mann-Whitney test for multivariate data.

Descriptive Analysis

Descriptive statistical analysis of the items can be found in Table 1. The distribution characteristics of each item were verified testing skewness and kurtosis. Since items are ordinal, the non-normal distribution was assumed. In line with this expectation, 7 of 26 items were significantly skewed (p < 0.01). Furthermore, 26 of the 26 items had a significant kurtosis (p < 0.01). Given the results of the robust Jarque-Bera tests, all items except three were very far from normal. These results provide clear statistical evidence that the items do not have a normal distribution, and any analysis based on this assumption (Pearson correlation, principal component analysis, linear factor analysis with maximum likelihood estimation, etc.) cannot provide accurate results. This conclusion is confirmed even more by the test of multivariate normal distribution: Mardia’s test (Mardia, 1970) showed that the data do not have multivariate normal distribution (g² = 602, z.kurtosis = 39, p < 0.001). In addition, the adjusted projection test to detect multivariate outliers (Filzmoser, Maronna, & Werner, 2008) revealed the presence of 11 of these outlying values in the data.

Analysis of Reliability

The most commonly used test of reliability is Cronbach’s α which can, however, be very inaccurate when used for ordinal scales (Dunn et al., 2014; Zumbo et al., 2007). This uncertainty can be partially corrected, when the Cronbach α is not calculated from the Pearson product-moment correlation matrix but from the polychoric correlation matrix, which takes into account the ordinal nature of the variables (Zumbo et al., 2007). Furthermore, an even better alternative is to use the McDonald ω test (Dunn et al., 2014). Hence, for the analysis of reliability we use the McDonald ω as an indicator, although for reasons of comparability we list the values of the classical Cronbach α, (calculated from the Pearson product-moment correlation matrix) and also the Cronbach α, which is calculated from the polychoric correlation matrix. Another highly desirable feature of the ω index is the possibility to validate the assumption that the instrument measures a sufficiently general construct behind all dimensions (which can be determined from the value of the hierarchical ω). Table 2 shows the values of the reliability tests for the whole range of the SCS and its individual subscales (dimensions), as well as the value of McDonald’s total and hierarchical ω. As shown in Table 2, all reliability values are relatively high. However, the value of the hierarchical McDonald ω (0.61) reveals that there is only a weak general latent factor behind the six dimensions of the SCS, which would not explain sufficient amount of the variance. Therefore, the use of a total score would be inappropriate. However, the value of the hierarchical McDonald ω for two general dimensions (0.82) means that two general factors account for 82 % of variance, so the use of two scores (13 positive and 13 negative items) should be recommended.

Analysis of Validity

Construct validity of the SCS was measured using Spearman’s correlations between the SCS and other instruments which measure related constructs, i.e. FSCRS, LOSC, SCCS and their respective dimensions. Correlations were in agreement to the theoretical expectations, which indicate that the SCS and its subscales show good construct validity, see Table 3.

IRT Factor Analysis

As already stated, there is no hope that the ordinal variables that make up the items of the questionnaire can meet the assumption of multivariate normal distribution, which is essential for the correct functioning of classical linear factor analysis (based on the maximum likelihood method of estimation). For the factor analysis, therefore, methods of IRT (item-response theory) will be used, that are much more relevant and accurate for analysing ordinal variables, given the logistic and not the linear method of their estimation.

The analysis will start with the confirmatory six-factor correlated IRT model, estimated in the “mirt” package (Chalmers, 2012), method of estimation is the Samejima graded response model, the algorithm is the Metropolis-Hastings Robbins-Monro algorithm which is more appropriate for highly dimensional constructs. The second model will be the bifactor IRT model (Reise, 2012; Reise et al., 2010); the method of estimation is again the Samejima graded response model. This IRT model allows testing the loadings of items for the general factor and thus estimating the proportion of variance explained by a common factor. The last model will be the two-tier IRT model (Bonifay, 2016; Cai, 2010; Cai, 2016), method of estimation is again the Samejima graded response model (see Figure 1). For verification of the fit of these models with the data we used standard indices of fit (CFI, RMSEA, SRMSR), which have their recommended thresholds CFI (>0.90 acceptable fit >0.95 excellent fit), RMSEA (<0.08 acceptable fit; <0.05 excellent fit) SRMSR (<0.08 acceptable fit; <0.05 excellent fit). For robust linear models, we also used the WRMR index which has as recommended thresholds <1.50 acceptable fit, and <1.00 excellent fit. Furthermore, we compared these models by means of the likelihood-ratio tests to determine which of them had the best absolute fit with the data. We also inspected correlations among the latent factors in the six-factor correlated model and the factor loadings of the general factor in the bifactor model to detect possible differences between positive and negative dimensions.

The 6-dimensional IRT model showed an suboptimal fit with the data (CFI = 0.80, RMSEA = 0.076, SRMSR = 0.191). Correlations within the positive (0.70, 0.62, 0.72) and negative dimensions (0.83, 0.78, 0.92) were far stronger than between positive and negative dimensions (0.28, 0.18, 0.24, 0.20, 0.22, 0.25, 0.37, 0.36, 0.42), which suggests that there are two general factors rather than a single general one. The bifactor IRT model revealed significantly better fit (CFI = 0.91, RMSEA = 0.053, SRMSR = 0.098), but a common factor did not explain a sufficient proportion of variance (hierarchical ω = 0.61), and therefore the total score can not be used. Moreover, the mean loadings for negative items (M = 0.587) are significantly higher than the mean loadings for positive items (M = 0.264) suggesting that a single general factor does not sufficiently explain the variance associated with the positive items. The two-tier model revealed the best fit (CFI = 0.95, RMSEA = 0.042, SRMSR = 0.089). Likelihood-ratio tests showed that the two-tier model has better fit than both the six-factor correlated model (χ²_diff = 795, df = 12, p < 0.001) and the bifactor model (χ²_diff = 972, df = 1, p < 0.001). In this two-tier model, the mean loadings for negative items (M = 0.589) were pretty the same than the mean loadings for the positive items (M = 0.544) suggesting that two general factors capture a sufficient amount of variance. Standardized factor loadings and explained variance of the two-tier model are shown in Table 4.

To compare the results with a more traditional linear method of estimation, we fitted all the models with robust linear estimator WLSMV. The 6-dimensional robust linear model showed an suboptimal fit with the data (CFI = 0.88, RMSEA = 0.075, WRMR = 2.211). Correlations within positive dimensions (0.64, 0.66, 0.68) and negative dimensions (0.82, 0.77, 0.92) were far stronger than between the positive and negative dimensions (0.29, 0.18, 0.24, 0.18, 0.36, 0.19, 0.21, 0.35, 0.42). This result suggests that there are two general factors rather than a single general one. The bifactor robust linear model revealed significantly better fit (CFI = 0.93, RMSEA = 0.068, WRMR = 1.342), but a common factor does not explain a sufficient proportion of variance (hierarchical ω = 0.67), and therefore the total score can not be used. The two-tier robust linear model revealed the best fit (CFI = 0.96, RMSEA = 0.049, WRMR = 1.121). Likelihood-ratio tests showed that the two-tier model has a better fit than both the six-factor correlated model (χ²_diff = 381, df = 12, p < 0.001) and the bifactor model (χ²_diff = 199, df = 1, p < 0.001). We can therefore conclude that the more traditional linear method of estimation confirmed the results of the IRT method.

The 6-dimensional IRT model in the validation sample again showed an suboptimal fit with the data (CFI = 0.58, RMSEA = 0.116, SRMSR = 0.253). Correlations within the positive (0.77, 0.68, 0.92) and the negative dimensions (0.80, 0.78, 0.94) were far stronger than between positive and negative dimensions (0.76, 0.43, 0.46, 0.61, 0.63, 0.52, 0.46, 0.63, 0.59), which also suggests that there are two general factors rather than a single general factor. The bifactor IRT model in the validation sample revealed significantly better fit (CFI = 0.87, RMSEA = 0.066, SRMSR = 0.080), but a common factor again does not explain a sufficient proportion of variance (hierarchical ω = 0.68), and therefore the total score can not be used. The two-tier model again revealed the best fit (CFI = 0.93, RMSEA = 0.056, SRMSR = 0.071). Likelihood-ratio tests show that the two-tier model has a better fit than both the six-factor correlated model (χ²_diff = 488, df = 12, p < 0.001) and the bifactor model (χ²_diff = 333, df = 1, p < 0.001).

Mokken’s Analysis of Scalability

Mokken’s analysis (Sijtsma & Molenaar, 2002) allows checking whether the items are scalable into a single scale. It uses covariances between pairs of items to test the monotonic model – if a model satisfies the test of scalability and monotonicity, it is safe to use the total score: items are scalable into a single scale. Unlike the Rasch’s model, it does not assume any parametric shape of function response of items, so it is a nonparametric IRT model. The Self-compassionate responding scale (SK + CH + MI) is scalable after discarding of items 9 and 22, its coefficient H = 0.339 (SE = 0.013), which is above the recommended threshold 0.300. The Self-uncompassionate responding scale (SJ+IS +OI) is scalable after the discarding of item 8, value of coefficient H = 0.348 (SE = 0.012). Obviously, it makes no sense to sum up the total score of the whole questionnaire of the SCS (26 items), since its scalability is very low (H = 0.207, SE = 0.009).

Analysis of Differential Test Functioning

Concerning the invariance of the test to different demographic groups, differential item functioning (DIF) is the most commonly used analysis in the context of IRT. However, it is more appropriate to verify differential test functioning (DTF) (Chalmers et al., 2016): Recent research showed that even if one or several items displayed a significant DIF, this does not necessarily imply that the test would display the DTF as a whole, for it might happen that the DIF of one item (i.e. a different probability of responses of members of one group as compared to members of the second group, while the value of the latent ability is the same) is compensated by another item. We are particularly interested in the DTF across latent ability (signed DTF), because this may create systematic distortion of the total score at the disadvantage of one group (for the sake of completeness, let us also add that the DTF for a particular part of the latent ability can be verified – unsigned DTF).

Concerning gender, values of the signed DTF (i.e. the average distorted score, which is in this case in the advantage/disadvantage of men) are 0.18 for the subscale Self-compassionate responding (SK + CH + MI), which is 0.34% and represents a non-significant difference (p = 0.65), and –0.05 for the subscale Self-uncompassionate responding (SJ + IS + OI), which is –0.10%, and also represents a non-significant difference (p = 0.88). In the case of the difference between the singles and people in relationship, values of the DTF (the advantage/disadvantage of singles) are –0.47 for the subscale Self-compassionate responding (SK +CH+M), which is –0.90% and represents a non-significant difference (p = 0.18), and 0.35 for the subscale Self-uncompassionate responding (SJ + IS + OI), which is 0.68%, and also represents a non-significant difference (p = 0.27).

Note that these values represent a systematic distortion of the test and they cannot be confused with the differences in the total scores or latent ability scores between the groups. Invariance of the test is just a prerequisite for an accurate comparison of groups and is a real, although unfortunately extremely widespread problem of how to perform the comparison of groups (e.g., t-test, Mann-Whitney nonparametric test, etc.) without ascertaining whether the scores, which are to be compared, are or are not systematically distorted in the advantage and disadvantage of any of the groups.

Because the DTF did not show any significant systematic distortion, we can test possible differences in scoring responses. Due to the multivariate non-normal distribution and the presence of outliers we use an extension of (projection type) the non-parametric Mann-Whitney test for multivariate data (Wilcox, 2005). In this test, it is verified if there is a probability of ranking significantly deflected for one of the groups – therefore the null hypothesis is that the value of η is 0.5 (i.e. between the groups there is no difference) and the test verifies if the estimate of this value is significantly different from 0.5. The testing of the responses between gender resulted in an estimated value of η = 0.51 which is obviously a very insignificant amount (95% CI 0.23 – 0.80). The same goes for testing differences in the responses between singles and people in relationship, which reveals an estimated value of η = 0.56 (95% CI 0.14 – 0.84). In conclusion, we can say that there is no difference in the responses between men and women and between people in relationship and single people.

Development of Norms

Because all subscales are scalable, and the total score is invariant in respect to its items, we also provide here the norms calculated for the total score in each subscale, i.e., for the subscales of Self-compassionate responding and Self-uncompassionate responding (Table 5). Based on Mokken’s analysis (see above), however, we had to exclude three items: 9 and 22 in the first scale, and 8 in the second scale. Thus, the first scale (Self-compassionate responding) contains 11 items, and the range of scores is from 11 to 55, and the second scale (Self-uncompassionate responding) contains 12 items, and its range score is between 12 and 60. These norms can serve to approximately provide the differentiation degree of Self-compassionate responding and Self-uncompassionate responding within a selected population.