Computer-Based Neuropsychological Theory of Mind Assessment: A Validation Study

1.1 Theory of Mind

Theory of mind (ToM) is a term proposed by Premack and Woodruff in 1978 [1] to refer to a psychological function that allows making inferences about the other’s oneself mental states such as thoughts, feelings, etc. Further, this ability is considered an executive function as it is involved in the comprehension, prediction and explanation of the conduct of others, their intentions, beliefs and their knowledge about a situation [2,3,4,5]. The importance of this mental function is that it allows establishing functional social interactions since we can evaluate and adjust our behavior to specific social conditions [6]. A better development of theory of mind makes it more likely to have successful social interactions, since it is possible to understand not only the intentions of the others, but the way in which the other people represents the world and it can be different from the own way [2, 7].

Assessing cognitive processes is a remarkable need in clinical neuropsychology, but it represents a constant improvement and precision of instruments development. According to Björngrim [8], during 80’s and 90’s decades, the analysis carried out around the subject favored to the traditional test that were made on paper; afterwards, at the beginning of the year 2000, it is possible to observe a significant reduction in the differences between the traditional methods versus the digital ones. The instruments that require a computer are now more frequently used in clinical practice; therefore, the research field has been paying more attention to it. In general terms, the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology emphasize and point out the benefits of computerized tests: they allow to evaluate in a quicker way, a greater number of patients; measure time reaction more precisely; notably reduces the costs and time in evaluations; exports data automatically; and they are capable of integrate and automate interpretative algorithms, as well as the rules which determine the disability or changes in a trustworthy, statistic manner [9]. In the clinical work, computerized tests have been so useful to detect and diagnose a wide variety of conditions or neuropsychologist alterations that have been exhaustive studied in different populations. Evaluating cognitive process through digital mechanism allows getting more information, including sensitive and multidimensional information. In the case of adults with Alzheimer’s disease or other neurodegenerative disorders, besides reaching the status information about cognitive deterioration and functional changes, it is possible to detect subtle changes and patterns of movement. On the other hand, in patients with deficit in working memory, and visual integration alterations, commonly observed in people with attention-deficit disorder, autism disorder, and Down’s syndrome, the use of digital tools makes possible to stimulate and measure cognitive and motor skills [10].

In clinical neuropsychology, it can be found that ToM can be assessed since early stages of life. From the classical Piagetian perspective, children develop ToM from its fundamentals of egocentric preoperational perspective that progressively moves towards a world’s decentered and independent understanding [11]. Recently, neurocognitivist perspective, suggest that the social cognition (that includes ToM) develops as a brain network that becomes increasingly fined tuned to relevant stimuli and events in activity-dependent manner, including abilities like the perception and processing of faces, joint attention, and empathy [12, 13].

Participants.

54 participants were included in the sample; two were excluded due to incomplete data on paper-based assessment. The final two related-sample groups consisted of 52 children 10–12 years old, 10.78 mean age 0.74 SD, 57% were male, while 43% were female. The protocol was successfully approved by the Ethics Committee, indicating no need for informed consent, the application took place in elementary school’s facilities, and parents and school’s authorities gave proper authorization for participation.

Instruments and Procedures.

The paper-based assessment consisted of the 28 item version for children [6]. Each item was presented in a single paper sheet, displaying the visual black and white colored stimuli; all stimuli presented the eyes section of a human face, 58% male, 42% female. At each corner of the paper sheet four options were displayed in times new roman 14 pt. text defining one correct response for the displayed visual stimuli, showing the closest description to the eyes intentions representation, and three incorrect options. Neither time limit nor feedback was given for each stimuli or test in general, once the participant made a selection the next paper-sheet stimuli was administered.

The computer-based RMET test consisted on a 28 trial of visual stimuli based on the Baron–Cohen, et al. original version [6], containing eyes section from the face (Fig. 1) of male 58% and female 42% displayed in black and white over a gray background screen. Four options were presented simultaneously to the eyes picture in text boxes under it. The application was developed in Java, following a modular, iterative and incremental software development process based on requirements gathering, analysis, design, construction and testing. All logic and interaction flow were directly programmed in the application modules. The modules included are: Patient registration, where the patient captures his/her full name and age, and based on these a profile is created. Instructions, as its name implies, gives the patient information about what the test consists of and how they should respond. Test questions, present each of the questions conforming the test, each of which include the image to be identified and the options that the user may choose to respond; the order of the questions and the answers is random in each instance of the test. Transition between questions, patients have 10 s to answer each question, in case it is not answered at that time, the question is omitted and the next question is presented. At the end of each question, the patient is presented with a distracting stimulus (black cross on a white background) for a short time, at the end of which the next test question is presented. Save results, the results of each test (i.e. question number, answer and score) are stored temporarily and at the end of the test they are exported to a file that is identified with a name based on the patient’s name, date and time of test application.

Sample stimuli from the computer-based version of the theory of mind test.

Full size image

Each child was assessed individually in a quiet room, 50% of the participants received the paper-based application first, followed by the computer-based assessment, while the other 50% of the participants received the inverse sequence of administration of the instruments.

Data Analysis.

Analyses were performed in three moments. First we employed related-samples T-test to compare the total score which compiles a sum of correct responses from the 28 items of each application format. Secondly, percentages of selection for each one of the four options for each stimulus were calculated and mean percentages were compared. Finally a Pearson’s coefficient was obtained between the percentages of selections and KR20 inter-item coefficient for correct and error responses on the computer-based RMET data.

For each application format we first calculated the sum of correct responses from each item. Afterwards related-samples two-tailed T-test was calculated, which showed no difference between the correct responses means (t = −.036, DF = .51, p = .971).The linear regression showed non-proportional bias coefficient (B = −.135, p = .243) among the samples. Results from errors and correct responses for each item were used to estimate inter-item internal consistency reliability test with Kuder-Richardson 20 (KR20) test for dichotomic variables from the computer-based version. A strong internal consistency coefficient was found in this analysis (KR20 = .805). Differences between selections for each expression are listed in Table 1. Selections in the target column indicate the number of correct responses made by each participant and are represented in percentage in the table. However, proportion of errors when making any of the other selections was also analyzed to compare if significant differences were presented.

The objective of this study was to present the first validation results of a computer-based neuropsychological instrument for assessing ToM, towards its comparison with the original paper-based version in a sample of children. The use of computer-based versions of clinical assessment instruments offers several advantages, although they require to be evaluated to determine their equivalence with traditional (paper and pencil) versions [14], they result attractive because computer-based versions have features that are not possible in traditional versions of the instruments [15].

Evaluating the comparability of paper-based and computer-based test is crucial before introducing computer-aided assessment [16]. In the scientific literature, there are many comparability studies that have examined the impact of transferring a test from paper to computer among which are testsfor assessing cognitive processes such as language [17], memory [18, 19] and working memory [20]. However, this is the first work which applies and validates a social cognition assessment task for Mexican children in a computer-based format.

Several studies have described that major advances in ToM understanding occur between 3 and 6 years old, and they are linked to improvements in both, cognitive and social functioning in neurotypical population. Nevertheless, ToM is a developmental phenomenon with important advances over early to middle childhood [5, 21]. In atypical development, ToM abilities fail to fully develop, such as neurodevelopmental disorders (like autistic spectrum disorders) or in acquired brain damage [6]. For this reason it is important to have validated assessment instruments that can be used both in typical and atypical development populations.

Our study demonstrates no significant differences between the paper version and the digital one, as the comparative analysis shows, suggesting that the test properties are very similar. However, it has been reported that the use of digital tools for the assessment could have some advantages over the traditional ones, such as: to capture and engage the person’s interest, to assess a greater number of individuals quickly, reduced cost because the administration and the scoring are carried out automatically, automated data exporting for research purposes, likewise it is more suitable for repeated evaluations as it allows for the randomization of stimuli producing alternative forms of the test, increased accessibility to patients in areas in which neuropsychological services are scarcer and more accurate measurement of time indicators such as the reaction time [22,23,24].

In clinical populations, such as people diagnosed with autism spectrum disorders (ASD), researchers have suggested that technological tools may be particularly promising as clinical mechanisms for some children with this disorder given that many of them exhibit greater strengths in understanding the “physical” world compared with the social one, and they respond well to technological feedback and show interest in technology [25]. Gillespie-Lynch, Kapp, Shane-Simpson, Shane &Hutman [26] carried out a study whose aim was to evaluate whether computer-mediated communication or use of the internet to interact with others, benefits to ASD participants. The sample was conformed by 291 participants with ASD diagnosis and 311 without it. The perception of communicative benefits of internet use was explored through a digital survey. Results showed that the ASD participants reported significantly less face-to-face interactions compared with the control group. The perceived benefits of computer mediated communication that showed greater difference between ASD and non-ASD groups, were: express true self, practice interaction, time to think, choose who talk to, find people like you and written down. This findings are particularly important in the case of our instrument because children with autism spectrum disorders present as one of their main characteristically symptoms, alterations in the development of social cognition and therefore in the theory of mind.

The computerized tests have been used in other fields of child neuropsychology, such as the motor disabilities populations [27], patients with sequelae of sport’s concussion [15] or from treatments for serious diseases such as cancer. Heitzer, Ashford, Harel and collaborators [28] reported the results of their study in 73 children diagnosed with medulloblastoma and treated with radiotherapy, who completed a neuropsychological assessment using the Cogstate test before and three months after of the radiotherapy. At the baseline, standard neuropsychological measures of similar cognitive domains were administered. Pearson´s correlation coefficient showed moderate correlations between Cogstate measures (as reaction time) and well-validated neuropsychological measures. Further, analysis revealed that the performance at the baseline was within age expectations but following radiation therapy, there was a decline in performance, suggesting that the test is sensitive to acute cognitive effects of medulloblastoma treated with radiation therapy.

Future validation results of this neuropsychological assessment instrument proposal, should include exploratory and confirmatory factor analysis, and mean difference comparison based on confidence intervals ranks, finally normative data with regression methods including a comprehensive sample controlling variables, such as family, socio-cultural, and clinical features.