An organic artificial soma for spatio-temporal pattern recognition via dendritic integration

The capability of recognizing and reacting to stimulation patterns coming from multiple sources, integrating them in a coordinated, low-dimensionality response, is one of the main features of neural computing, which enables extremely low-power parallel computation, concurring in making the human brain a more robust, plastic and fault-tolerant computing system with respect to digital architectures [1]. In each neuron, the soma constantly integrates inputs coming from thousands of dendritic synapses, which are located at the connection knots between axons of pre-synaptic neurons and its (post-synaptic) dendrites. This process is known as 'dendritic integration' [2] and it is critical in neural signal transmission and computing. Indeed, it has been demonstrated that, since action potentials are initiated near the soma in the axon initial segment [3, 4], synaptic inputs can influence action potentials of post-synaptic neurons by effectively modulating the membrane potential at this location [5]. In addition, this modulation is influenced by geometrical and electrochemical properties of dendrites, thus enabling a wide variety of non-linear operations in neurons [6].

In recent years, there has been a growing endeavor in trying to implement brain computing features on a hardware level [7]. This ambitious goal is pursued either by building massive networks of artificial neurons and synapses achieved via conventional silicon-based electronics, aiming at building neuromorphic supercomputers (e.g. the recently announced Deep South, aiming at >10¹⁴ synaptic operations per second [8]), or by emulating synaptic functions in neuromorphic device units, exploiting the inherent signal response properties of unconventional materials, such as organic mixed ionic electronic conductors—OMIECs [9]. The latter resulted in the newly developed field of Organic Neuromorphic Electronics, which succeeded in the demonstration of a wide variety of neural processing functions, concerning plasticity in response to input signals [10, 11] and spike generation [12]. Importantly, such architectures can be promptly interfaced to the biological environment, and their neuromorphic behavior is directly influenced by chemo-physical features of the environment itself, making them ideal candidates for biointerfacing and sensing applications [13–17]. Lately, both reversible [18–20] and irreversible [21] means for achieving tunability of the device response have been explored, especially in three-terminal architectures [22], and this enabled the demonstration of higher functions, such as pattern recognition [23] and reservoir computing for image recognition [24] in OMIEC-based neuromorphic devices.

The proneness of OMIECs towards such applications directly stems from their capability of establishing large electroactive surface area interfaces with electrolytic solutions [25] and from their strong interaction with ionic species in solution, which ultimately results in a kinetic unbalance between the processes of ion adsorption and desorption upon an external driving force [26] (i.e. typically, a pulsed bias in the electrolyte). The consequent, relatively 'slow', relaxation time of OMIEC/electrolyte interfaces can serve as a design tool for the fabrication of neuromorphic units exhibiting an inherent representation of time [27] as well as real-time filtering capabilities [28]. Such features make them ideal building blocks for in situ spatio-temporal pattern recognition, which is one of the main desiderata when imagining compact, low-power circuitry for on board data treatment and classification.

In this work, a model OMIEC-based neuromorphic architecture for spatio-temporal pattern classification is proposed, namely a dendritic integrator made by artificial post-synaptic soma with three dendrites interfaced to three pre-synaptic neurons, and its classification capabilities are assessed both on model patterns and in the context of speech recognition.

Figure 1 shows the proposed approach and the device fabrication. In particular, figures 1(a) and (b) provide a direct comparison between the biological concept we are proposing to mimick and the corresponding artificial dendritic integrator. The three presynaptic neurons in figure 1(a) are three input terminals carrying the voltage time series V_in,1, V_in,2 and V_in,3, respectively, in figure 1(b). The three synapses are identified by their three weight coefficients k₁, k₂ and k₃. The post synaptic neurons receives the three inputs and outputs a current, I_out, which corresponds to a weighted combination of the input signals, thus reducing the dimensionality of the input by a factor 3 (i.e. three bi-dimensional V vs t inputs are reduced to a one bi-dimensional I vs t output). Figure 1(c) shows the steps of the device fabrication, which are detailed in the experimental section. At first, three linear electrodes and a three-branched one are patterned via direct laser ablation on a metalized poly-imide layer, then a second layer of polyimide allows spatial confinement of the three synaptic cleft, each constituted by two terminals, namely one axon terminal from a pre-synaptic neuron and one dendrite from the post-synaptic soma. Synaptic terminals are then coated with PEDOT/PSS via potentiostatic electrodeposition, and the device is completed with three working electrolytes, one for each synaptic cleft. Figures 1(d) and (e) show the schematic and the actual device characterization layout.

Zoom In Zoom Out Reset image size

Due to the chosen device layout, the administration of a voltage pulse elicits, in each synaptic cleft, a currents which can be collected at the presynaptic terminal, , and has the usual form:

where R_E and C are resistance of the electrolyte and equivalent series capacitance of both electrode/electrolyte interfaces, respectively. This current is mirrored on the other terminal of each synapse, resulting in an excitatory post synaptic current, EPSC, in the soma. The overall somatic current I_out will simply be the linear combination of the EPSCs, as follows:

where indexes 1, 2, 3 indicate the synaptic clefts.

Since both R_E and C, in the absence of any other difference concerning electrode area or thickness of the PEDOT/PSS layers, are solely influenced by the molar concentration of the electrolyte, it is possible to change the weight of each synaptic connection by changing the molar concentration in the corresponding electrolyte compartment. For this reason, in the present work, weight coefficients k₁, k₂ and k₃ are expressed in mol l⁻¹.

At first, we assessed the device classification performances benchmarking them against the discrimination of two model patterns, the results are summarized in figure 2. Figure 2(a) shows the selected 3 × 3 binary patterns, termed 'A' and 'B', and the corresponding input voltammograms. Each of the three input voltages codifies for a row of the pattern, each column is encoded in a Δt time interval, V_in = 1 V is assigned to a white pixel, while V_in = 0 V is assigned to a black one. A 'black pixel', namely a Δt time interval with V_in,1 = V_in,2 = V_in,3 = 0 V is added before and after every row, to ensure equal contour conditions. In this experiment, k₁ = k₂ = k₃ = 1 M.

Zoom In Zoom Out Reset image size

These patterns have been chosen since, although spatially different, they are temporally equal in terms of transitions between ON and OFF pixels. Such transitions are of critical importance since the cause spikes and current decays in I_out, as from equation (2). In particular, in both cases there is one voltage turning on at t = Δt, one voltage turning ON and one turning OFF at t = 2Δt and at t = 3Δt, and one turning OFF at t = 4Δt. Comparison between figures 2(b) and (c), showing the discrimination between Pattern A and Pattern B built on a time basis of 20 ms and 3 ms, respectively, unveils the relevance of Δt in determining the classification efficiency or the artificial soma. The characteristic RC relaxation time of our synapses with k = 1 M in response to a single pulse is τ_RC = 3.84 ± 0.62 ms (see figure S1 in the supporting information) When Δt = 20 ms (i.e. longer than τ_RC, figure 2(b)), all the presynaptic currents return to steady state before any further perturbation of the system equilibrium occurs. The resulting somatic currents, top right panel in figure 2(b), are hence poorly distinguishable. As a consequence, also the absolute exchanged somatic charges, |Q| (bottom right panel, figure 2(b)), calculated as the integrals of current traces in time, show comparable profiles between Pattern A and Pattern B.

It is worth noticing that, due to minimum differences between the individual synapses, the proposed architecture still manages to discriminate between the patterns, meaning—as expected—that spatial information can be encoded in the difference of synaptic weights, be them intentionally variated or adventitiously different. This effect is magnified when Δt is lower than τ_RC (figure 2(c)), since incomplete relaxation of each synapse (the phenomenon at the origin of STP, STDP and paired-pulse plasticity in organic neuromorphic devices) brings in additional charge contributions.

We choose to regard the absolute value of the integral of current in time, |Q|, and in particular its value at the end of the stimulation protocol, |Q|_end, as the final quantitative output of the proposed classifier because, on the one hand, it magnifies even transient differences between currents and, on the other, in view of future technological development it could be directly measured with a simple integrator circuit acting as a pseudo-additional classification layer, yielding a further dimensionality reduction (as commonly done in artificial neural networks) by condensating the information of a pattern of three bidimensional time series in a single charge value. Upon these premises, a performance coefficient of our binary classifier, χ, can be expressed as the difference between the total somatic charge exchanged upon Pattern A and Pattern B stimulations, normalized over the sum of the two charges. In this example, when Δt goes from 20 ms to 3 ms, χ increases from 0.047 (data from figure 2(b)) to 0.081 (data from figure 2(c)). As said, another way to improve classification performance is to diversify the weights of the synaptic connections. Even keeping Δt = 20 ms, if k₁ = 0.8 M, k₂ = 1 M and k₃ = 0.6 M, χ is increased to a value of 0.059.

In absolute terms, ||Q|_end,A − |Q|_end,B| = 1.67 µC when Δt = 20 ms and ||Q|_end,A − |Q|_end,B| = 1.36 µC when Δt = 3 ms, and both charge differences largely exceed the sensitivity of state of the art charge integrators (≈pC).

Aiming at the translation of the proposed architecture and of its classification capability to more complex and significant scenarios, we devised a proof-of-concept application of the optimized classifier architecture (Δt = 3 ms, k₁ = 0.8 M, k₂ = 1 M and k₃ = 0.6 M) to the problem of speech recognition. We used a set of 14 audio recordings, each containing an individual Italian word. In particular, seven of them (i.e. AMORE—CASA—CIAO—FAME—FELICE—GRAZIE—TRISTE) were collected in the framework of the present study, while the remaining seven (i.e. GIORNATA—INFANZIA—INVENTATO—MASCHERA—ONDOSO—PRIVILEGIO—TIMONE) were extracted from the Multi-SPeaKing-style Articulatory corpus (MSPKA) [29], to rule out influences from the sampling conditions and to compare classification efficiency starting either from audio traces or from kinematic data.

From each audio, we compute the mel-frequency cepstral coefficients (MFCCs) [30] to derive three dimensional features to serve as input signals. MFCC extraction—a standard pre-processing step in speech recognition tasks—facilitates a compact parameterization of speech signals capable of capturing phonetically relevant aspects [31]. The algorithm to generate input sequences for the artificial soma starting from audio tracks is schematically shown in figure 3, using the word 'AMORE' as an example. As commonly known, it is possible to apply a moving time window to an audio track (figure 3(a)) and to compute the discrete Fourier transform for each segment, resulting in a spectrogram (figure 3(b)). Since the proposed artificial soma features three input synapses, it is necessary to reduce to 3 bins one of the dimensions of the spectrograms. Precisely, we discretized the spectrogram in three time windows (figure 3(c)), as detailed in the experimental section, and applied the discrete cosine transform to the Mel spectrogram to extract 13 cepstral coefficients (MFCCs, figure 3(d)):

Zoom In Zoom Out Reset image size

where S_k is the power of the kth frequency band. We, then, excluded the first cepstral coefficient c₀, accounting for the overall energy of the signal, and binarized the result using a median-split approach:

Binarization produces a black and white pattern representing the audio file (figure 3(e)). All the patterns are reported in figure S2, in the supporting information.

A 90° counterclockwise rotation of the pattern (figure 3(f)) and its conversion in a square voltage input on a time basis Δt = 3 ms, with V_in = 0.5 V for white rectangles and V_in = 0 V for black ones, yields the input sequences for the artificial soma (figure 3(g)). As for the model patterns of figure 2, for each stimulation pattern it is possible to collect the individual pre-synaptic currents I₁, I₂ and I₃ (figure 3(h)) and the resulting somatic current, I_out (figure 3(i)).

The performances of the classifier are summarized in figure 4. Figure 4(a) shows the profiles of I_out in response to the 14 V input patterns derived from audio recordings, as described in figure 3. Conversely to what happened in the cases of the model patterns in figure 2, here the current outputs are clearly distinguishable also from a qualitative examination of the tracks. To provide a quantitative estimate of the pattern recognition efficiency, it is possible to refer to the cross-correlation table in figure 4(b), which reports Pearson's correlation coefficients computed between current traces.

Zoom In Zoom Out Reset image size

For this dataset, the average cross correlation coefficient is as low as 0.39 ± 0.23, with a maximum value of 0.84 (between MASCHERA and INFANZIA), hinting at easy discrimination amongst all the selected words. As in the case of model patterns, it is possible to further reduce the dimensionality of the problem by integrating I_out in time and acquiring charge vs time profiles, reported in figure 4(c). As predictable by the current traces in figure 4(a), every word yields an unambiguously attributable charge profile and an individual |Q|_end value (figure 4(d)).

To quantitatively express the classification efficiency of a given classifier architecture (i.e. a given time basis for patter encoding and a given kernel of k values) it is not useful to refer to the parameter χ, which was devised for one-to-one comparisons, but it is possible to refer to the average difference between two subsequent |Q|_end values, S.

Data in figure 4 yield an S as high as 1.38 µC while, in a control experiment with k₁ = k₂ = k₃ = 1 M (figure S3 in the supporting information), S = 0.43 µC. This means that, by properly tuning the set of synaptic weights, it is possible to increase the classification performance by 320%.

As a final test for our classifier, as introduced above, we chose a less pre-treated but more clinically relevant input dataset, still in the field of speech recognition: namely, kinematic data of the speaker's articulatory trait, collected by electromagnetic articulography (EMA), on the subset of words extracted by the MSPKA. Results are shown in figure 5. MSPKA dataset provides, along with the audio, the corresponding speaker's articulatory kinematics data (specifically lips, jaws, and tongue movements).

Zoom In Zoom Out Reset image size

From the high-dimensional EMA data (comprising 7 sensors × 3 dimensions = 21 time-series), only the three most relevant recordings [32] were chosen: namely tongue movement towards and away-from the lips (TB.x) and the palate (TB.z), and finally opening and closing of the mouth (lower lip moving towards and away from the upper lip: LL.z) (figure 5(a)). Figure S4 in the supporting information shows the entire dataset. To derive voltage input sequences we binned the time-series with a moving average approach on windows of 10 ms and cut all the selected words at the duration of the shortest one (figure 5(b)). The time basis is then converted to 3 ms in accordance with previous experiments, resulting in the three input sequences (figure 5(c)). I_out is collected as already described. Figure 5(e) shows the profiles of I_out in response to the seven voltage input patterns derived from EMA recordings. Figures 5(f) and (e) show more efficient classification with respect to the corresponding data in figures 4(c) and (d). In particular, restricting the analysis to the seven-word subset, S is increased from 2.06 µC to 3.43 µC, with a 66% increase and here a charge resolution of 1 µC is sufficient for unambiguous classification. This is further evidenced by the comparison between cross correlation tables in figure 5(h) and in figure 5(i), built on I_out profiles resulting from MFCCs-derived and EMA-derived input sequences, respectively, which show much higher correlation (i.e. poorer distinction) between I_out profiles resulting from audio classification with respect to those coming from classification of kinematic data, well in accordance with previous literature on speech recognition [32].

3.1. Device fabrication

3.2. Electrical characterization

3.3. Audio spectrogram computation

3.4. Analysis, graphing and presentation

In this work we develop and demonstrate an organic neuromorphic spatiotemporal pattern classifier based on the concept of dendritic integration between three two-terminal artificial synapses whose weight factors can be arbitrarily set by simply changing the working electrolyte. The device is planar and fabricated through laser prototyping, making it prone to miniaturization and integration with planar, flexible and eventually bio-compatible/degradable electrode arrays for physiological applications. Furthermore, it is virtually passive, and performs classification without the necessity of any driving voltage, with the exception of the input ones. The proposed architecture can efficiently discriminate between fourteen 13 × 3 spatiotemporal patterns derived from audio files (figures 3 and 4) and shows a first proof of principle classification of quasi-continuous physiologically relevant time series (EMA data in figure 5), with even better performances with respect to strongly pre-treated model patterns. The results herein presented, coupled with the high integration possibilities offered by the technological platform of organic electronic techniques—which allows the envisioning of networks of multiple artificial somas—have the potential to foster novel online classification strategies of signaling patterns based on supervised learning strategies with organic neuromorphic electronics.

Research work leading to this publication was funded by IIT—Istituto Italiano di Tecnologia, University of Ferrara and University of Modena and Reggio Emilia (FAR 2018 Project e-MAP). This work has received funding from the European Union's Horizon Europe research and Innovation program under Grant Agreement No. 10109859, Project Piezo4Spine.

Luciano Fadiga and Fabio Biscarini equally contributed to this work.

All data that support the findings of this study are included within the article (and any supplementary files).