Keywords

1 Introduction

Social interaction plays a fundamental role in our daily lives. Simultaneously measuring multiple brains, cross-brain neural synchronization has been found between subjects during various social interactions [1, 18, 19] such as face-to-face communications [4] and musical improvisation on the guitar [5]. These findings suggest that cross-brain synchronization may play an important role in human social interaction. For groups with social barriers, cross-brain interactions in social interactions are abnormal [6]. The study of cross-brain interaction plays an important role to reveal the neural mechanism of human social interaction, and may be helpful to find a new therapy for social disorder such as autism.

According to the existing research, there are mainly two approaches for cross-brain interactions. The first typical approach is to communicate directly using brain signal. Rajesh has applied EEG to send information from one subject, and used transcranial magnetic stimulation (TMS) to let another subject receive the message [7]. Another approach for cross-brain interaction is to rely on the third-party tools such as collaborate BCI and feedback. Wang has proposed a collaborative paradigm to improve overall BCI performance to response to the direction cue as fast as possible by integrating information from multiple users [8]. Duan has built an experimental platform on the basis of functional near infrared spectroscopy (fNIRS) which allows the two subjects to interact with each other through competing or collaborative tasks such as the tug-of-war game [3].

Neurofeedback is an approach to investigate the relationship between brain activity and behavior, which extracts the neural characteristics of the brain signals and feedbacks them to the subjects in visual or auditory ways [2]. It feeds back the neural signatures of a participant to allow him to modulate his own brain activity. Unlike traditional brain imaging studies which uses the “behavioral manipulation – brain observation” paradigms, neurofeedback enables researchers to manipulate the brain activity as an independent variable, which can provide more causal insights into the relationship between brain and behavior. It has been shown that there is converging evidence that a single participant’s brain activity can be self-regulated with neurofeedback technology [9, 10]. In 2004, Goebel for the first time extend neurofeedback from single-person context to multi-person situation and this pioneering work allows multiple subjects simultaneously self-regulate their own neural activities in a social interaction environment [9, 10]. Duan has gone a step further and applied the neurofeedback to two subjects using fNIRS aiming to explore the relationship between the cross-brain neural synchronization and social behavior [3].

Cross-brain synchronization has widely been found during interactive activities. It has been used to provide evidence for the involvement of the brain regions across the subjects for processing the information during the interaction and reveal the neural mechanism behind the social contact. To analyze the synchronization, methods such as phase locking value (PLV), wavelet coherence are widely used. Because human brain is a nonlinear, chaotic and nonstationary system, phase locking is an appropriate approach to quantifying interactions. The most commonly used phase interaction measure is the phase locking value which has been used to measure brain synchronization [22]. Using PLV to measure synchronization, Szymanski has reported increased inter-brain phase synchronization in joint attention relative to individual attention during a visual search task and interpret the findings as neural substrates of social facilitation [21]. Jarhwan has used PLV to analyze the synchronization across the brains and reveals that the right temporal-parietal cortical region, might play an important role in the social interactions of autism spectrum disorder patients [11]. Joy Hirsch has used wavelet analysis and cross-brain coherence analysis to explore the mechanism behind eye-to-eye contact [12]. They found the cross-brain coherence increased for signals originating within left superior temporal, middle temporal and supplementary motor cortices of both interacting brains. The findings reveal a network that mediates neural responses during eye-to-eye contact between dyads.

In this study, a cross-brain interaction platform was constructed based on EEG and neurofeedback technology. The platform allows the participants to achieve cross-brain interaction directly with the medium of neurofeedback instead of other participants’ body languages or sounds. To test the platform, we conducted an experiment using a “tug-of-war” game. Wavelet coherence analysis was applied to analyze the synchronization between the specific electrodes across the brains, and reveals the synchronization between the subjects during the game.

2 Materials and Method

2.1 Cross-Brain Interaction Platform

The whole platform of the cross-brain interaction based on EEG is shown in Fig. 1. We used 32 channels g.Nautilus wireless EEG acquisition equipment (g.tec medical engineering GmbH, Austria) to simultaneously measure multiple brains’ signals. The raw EEG signals are recorded by the scalps and transmitted to PC stations using Bluetooth protocol. The signals will be merged into a core PC computer in real time using UDP protocol, which are prepared for the subsequent calculation including online analysis module and feedback module. An online analysis module is used to extract and calculate the feedback signal based on EEG (Fig. 4A). The module is developed using SDK provided by g.tec company with MATLAB (R2014a, The MathWorks Corporation). The feedback module is used to give the subjects a more directly and intuitive form of the current interactive situation across the brains to allow the subjects to do self-regulating and is built using Java swing framework (Java version “1.8.0_144”, Java Runtime Environment build 1.8.0_144-b01).

Fig. 1.
figure 1

The EEG-based interaction platform cross two subjects in the hardware level

Full size image

2.2 The Experiment Design

To test the platform, we conducted an experiment using the “tug-of-war” game which let the two subjects fought a cross-brain “tug-of-war” against each other [3]. An overview of the experimental scenario is shown in the Fig. 2B. Two male right-handed volunteers (age 24 years) without brain diseases or bad habits are recruited from Beijing Normal University participant in this experiment, and gave their written informed consent prior to their inclusion in the study. Two subjects were required to engage through motor imagination in the event of physical inactivity. A line with a cursor was displayed on the screen as is shown in Fig. 2A. Initially, the cursor was positioned at the midpoint of the line. Each subject was allowed to imagine only left or right and use a strategy to pull the cursor on the screen to their side. The difference between the amplitudes of their brain activities at each time point corresponded to the amount of the cursor’s shift. The entire experiment consists of a preparation process lasting 30 s, and 5 rounds to conduct the game. The preparation process was used for the subjects to adjust the state of their minds while facilitating the follow-up baseline correction. Each round consists of a resting and a tasking phase, where the duration of the rest period is 40 s and the duration of the task is also 40 s.

Fig. 2.
figure 2

The feedback screen for participants (A) and an overview of the experimental scenario (B).

Full size image

2.3 Online Analysis

The online data processing was programmed with MATLAB and performed on a PC station (Microsoft Windows 7 operating system). The data was filtered by a FIR band-pass filter at 0.01–40 Hz and baseline corrected. Based on event related synchronization and desynchronization principle, for one subject, the difference of band power between the C3, C4 at beta band was calculated as BP_A for subject A and BP_B for subject B in real time to determine directions the subject was thinking [13, 14, 17]. The difference between the calculation of bandpower between subjects, which is BP_A – BP_B, was used as the feedback information to shift the cursor on the screen. The feedback calculation was programmed as the MATLAB m-files to embed into modules. The experimenter could monitor the subjects’ brain activity online which is represented as the trajectory of cursor, as shown in Fig. 1. The trajectory of the cursor was recorded for subsequent analysis. Because of the limitation of performance in the simulation system, the feedback module is built independently which is connected to the feedback calculation module using UDP communication protocol.

2.4 Offline Analysis

In order to explore the neural synchronization between participants during the interactions, an offline analysis is performed with a wavelet coherence analysis [12, 15, 20]. Wavelet analysis decomposes a time varying signal into frequency components. Cross brain coherence is measured as a correlation between two corresponding frequency components, and is represented as a function of the period of the frequency components. It is a great tool to track the correlation between the band pass filtered components of time series signals and the equations are as follows:

$$ {\text{WC}}\left( {{\text{t}},{\text{f}}} \right) = \frac{{\left| {SW_{XY} \left( {t,f} \right)} \right|}}{{\sqrt {\left| {SW_{XX} \left( {t,f} \right)} \right|\left| {SW_{YY} \left( {t,f} \right)} \right|} }} $$
(1)

Where

$$ {\text{SW}}_{XY} \left( {t,f} \right) = \int_{{t - \frac{\delta }{2}}}^{{t + \frac{\delta }{2}}} {W_{X} \left( {\tau ,f} \right)W_{Y}^{ *} \left( {\tau ,f} \right)d\tau } $$
(2)
$$ {\text{SW}}_{XX} \left( {t,f} \right) = \int_{{t - \frac{\delta }{2}}}^{{t + \frac{\delta }{2}}} {W_{X} \left( {\tau ,f} \right)W_{X}^{ *} \left( {\tau ,f} \right)d\tau } $$
(3)
$$ {\text{SW}}_{YY} \left( {t,f} \right) = \int_{{t - \frac{\delta }{2}}}^{{t + \frac{\delta }{2}}} {W_{Y} \left( {\tau ,f} \right)W_{Y}^{ *} \left( {\tau ,f} \right)d\tau } $$
(4)

And \( W_{X} \left( {\tau ,f} \right) \) is the complex Morlet wavelet transform, the definition is as follows:

$$ W_{X} \left( {\tau ,f} \right) = \int x\left( u \right)\varphi_{\tau ,f}^{ *} \left( u \right)du $$
(5)

Where

$$ {\upvarphi }_{\tau ,f} \left( u \right) = \sqrt f \cdot e^{{j2\pi f\left( {u - \tau } \right)}} \cdot e^{{ - \frac{{\left( {u - \tau } \right)^{2} }}{{\delta^{2} }}}} $$
(6)

and \( W_{X}^{*} \left( {\tau ,f} \right) \) is the conjugate of the \( W_{X} \left( {\tau ,f} \right) \).

4-layer wavelet packet decomposition is performed with db4 wavelet base to extract beta rhythm waves from C3 and C4 electrodes for all participants [16]. As shown in Fig. 3, the cross coherence was computed across the C3, C4 channel on the same side and different side between the subjects. Namely the coherence was computed between C3_A – C3_B, C4_A – C4_B, C3_A – C4_B, C4_A – C3_B.

Fig. 3.
figure 3

The coherence computation between subject A and B in offline analysis.

Full size image

3 Result

All the participants reported that playing the tug-of-war game with another person was very interesting and they can perform the motor imagery well. They reported that when their attentions were highly concentrated, they can pull the cursor back to their side easily. On the contrary, if they lost their attention, the cursor would be pulled to the opposite side.

As shown in Fig. 4B, the top subgraph indicates the track of cursor’s movement controlled by subjects through online analysis. It expands the trajectory of the movement to the time dimension and represents that as time goes by, the cursor moves left to right. The blue line represents the movement of the cursor. If the cursor has been pulled back over three-fifths of the half-length of the line by one subject, we determine him the winner at that time [3]. The red dash lines represent the boundaries. Outside the boundaries of the red lines indicates the winner while inside the boundaries of the red lines cannot indicate who is winner. It can be found that the blue line is mainly outside the boundaries during 0 s–10 s and 28–31 s. The down subgraph represents the cross-coherence between specific channels across the subjects by offline analysis.

Fig. 4.
figure 4

The structure of the cross-brain interaction platform (A) and the results of online and offline analysis in the tug-of-war experiment (B). (Color figure online)

Full size image

Through offline analysis, it can be found that the coherence increased between the electrodes across the brains, such as C3 from subject A and C4 from subject B during 0 s–10 s, 18–25 s and 28–31 s. However, the coherence’s polarity of the same side electrodes is opposite to that of the different side. For example, the polarity of the coherence between C3_A – C3_B and C4_A – C3_B is opposite.

4 Discussion

We successfully build a cross-brain interaction platform based on neurofeedback and conduct a tug-of-war game to test the platform. Through the platform, the neural activity of the two subjects was observed and calculated online based on EEG. According to the trajectory recorded during the online game, most of the time two subjects fought against each other equivalently which indicates that the amplitudes of the brain activities across the subjects are roughly similar. Through the questionnaire, we found that when the subjects were distracted, the cursor would move to the other side while the subjects were concentrated, the cursor would move to their side. The subjects expressed that the game was very interesting and novel and they could interact with each other simply using EEG signals through the imagination of left or right to control the cursor to move.

Through the offline analysis, we found that there exists a relationship between the synchronous state of the brains and the movement of the cursor during the game. The higher the synchrony of the beta band of the C3, C4 electrodes’ signals across the brains, the better the results to distinguish between the subjects. During the period between 0 s–10 s, we can easily determine the winner in Fig. 4A while the coherence between the two subjects is relatively high. We can’t determine the winner during the 10 s–20 s while the coherence between the two is relatively low. In Duan’s research, the Pearson correlation coefficient (r) was used to measure the cross-brain relationship between the two participants and they found the neural synchronization during the game [3]. In this study, the synchronization between specific electrodes at beta bands demonstrates that EEG based neurofeedback can be used to explore the cross-brain interaction.

According to the existing research, Jaehwan has found the cross-brain synchronization when the subjects tend to choose different strategies such as cooperation or defection strategy in the prisoner’s dilemma game experiment [11]. The PLV between FZ-P8, FCZ-P8, C3-P6 across the brains is higher than that when subjects choose the defection strategy compared with cooperation strategy and the right temporal-parietal cortical region may play an important role in the social interactions of autism spectrum disorder patients. In this study, the synchronization across the brains may reflects the strategies which subjects has chosen to compete with each other during the game. Because C3, C4 electrodes are relevant to motor imagery [17], we only choose these two electrodes to conduct the experiment and analysis for simplicity and convenience at initial stage of the study. In the following phases of the study, individual channels should be grouped into anatomical regions based on shared anatomy, which is served to optimize signal-to-noise ratios. Multiple channels electrodes should be taken into consideration of cross-brain coherence analysis not just the specific ones.

5 Conclusions

In summary, the study established a cross-brain interaction platform based on neurofeedback using EEG signals. A validation experiment with a tug-of-war game has demonstrated that the platform can record and integrate two participants’ EEG signals to calculate and feedback the cross-brain neural interaction. Cross-brain synchronization has been found during the offline analysis which reflects the interaction state across the brains and may reflect the strategy that the participants choose. This study is still a preliminary work and needs further work to do. On the basis of the present work, next step we plan to investigate the neural features in the cross-brain synchronization which reflects the cooperation or competition strategies.