Hardware software co-design for leveraging STDP in a memristive neuroprocessor

For the simulation, the clock frequency is kept at 1 MHz. All the capacitors have 626fF capacitance. The values of the reference voltages VForm, VReset, VSet, and VRead are kept at 3.3 V, 2 V, 2 V and 1.2 V respectively. We also used readref (0.6 V), (0.6 V), and (0.1 V).

Figure 7 shows the functional verification of the STDP circuit. From the Res subplot, we see that after a RESET, the synaptic resistance is set at 18.09 kΩ using a of 0.6 V. At 3 µs, a pre-synaptic spike (Pre) is observed, succeeded by a post-synaptic spike (Post) at 6 µs. For this work, STDP time window is 11 µs. Since the time gap falls within the STDP time window for potentiation, the memristor's resistance decreases to 12.81  kΩ, thereby increasing the weight. This weight augmentation involves a two-step process: initially resetting the memristor and subsequently configuring it to a low-resistance state during the clock cycle following Post. Likewise, a depression operation is demonstrated when a post-synaptic spike occurs at 18 µs, promptly followed by a pre-synaptic spike. This sequence leads to a resistance increase to 17.16 kΩ.

Zoom In Zoom Out Reset image size

Figure 8 shows the layout of the test structure containing the STDP circuitry with a 2 × 12 pin-pad structure. The circuits take up an area of 1436  without the pin-pads, and 321 933.64  with the pin-pads. The potentiation and depression operations consume an average power of 115.9 µW and 121 µW respectively without the pin-pads. With pin-pads, the test structure takes 164.4 µW power on average for depression operation, and for potentiation operation, the average power consumption is 156.9 µW.

Zoom In Zoom Out Reset image size

The data reported in the paper is for a single memristor synapse circuit with STDP control. The number of memristors needed for an application depends on the optimum network generated by the evolutionary optimization algorithm (EONS), as discussed in section 3. As each synapse has one memristor, depending on the number of synapses in the generated network, the number of memristors needed can be calculated. The hardware cost consists of the area and power of the synapses and neurons required by the network. The average power consumption of the neuron is 166.70 µW [25], and it takes up an area of 2350  [32].

Figure 9 illustrates how the weight change is dependent on the time interval between neuron spikes for different initial resistances. The initiation time difference between the pre-synaptic and post-synaptic spikes determines whether the change is potentiation (for negative time differences) or depression (for positive differences). The weight is inversely proportional to the memristor's resistance. For different initial resistance values programmed using different in figure 5, extensive simulations were performed using a 65 nm CMOS process in Cadence Spectre, and the change in resistance was recorded. This change was then converted to the percentage of the initial weight of the memristor.

Zoom In Zoom Out Reset image size

The most significant resistance change occurs when the time difference is low, gradually diminishing as the time gap increases. The figures are consistent with (1) and a typical STDP graph shown in figure 1. However, biological STDP might vary from the ideal STDP curve shown in 1 depending on the initial weight. For our circuit, depression and potentiation are defined by the charging and discharging of capacitors C₂ and C₃ in figure 5. The charging and discharging follow an RC circuit topology where the resistors are implemented by MOSFET devices to save the area needed by large resistors. Due to that, certain non-linearity is induced in the charging and discharging process, leading to a slightly non-ideal STDP behavior. This is a trade-off between chip area and non-linearity. Using large resistors instead of MOSFETs will result in a more ideal STDP behavior.

In our circuit, the time difference between spikes is multiple of the number of clock cycles. However, the way that the STDP circuit works in figure 5 is completely analog in nature, meaning the value of Prog depends on the absolute time difference between pre- and post-synaptic spikes, thus leading to a continuous decrease in weight change as the time difference increases. So, although we are currently sampling the spikes every microsecond, if we change the clock frequency, it could be sampled with a higher or lower time period. But it will not change the amount of weight change for the absolute time differences between the spikes.

The four subplots also show the dependence of weight change on the value of initial resistance. At the lower resistance level shown in subplot (a), depression operation is stronger, and potentiation is weak. It is because the initial resistance is 6.99 kΩ, where the range of the synaptic resistance is between 5 kΩ and 18 kΩ. As a result, the resistance has more room to go upwards, causing stronger depression. As we increase the initial resistance, the STDP curve becomes more symmetric, denoting similar potentiation and depression rates, as shown in subplot (c). As we further increase the resistance to 18.1 kΩ, the potentiation operation becomes stronger, and depression rates become negligible (shown in subplot (d)).

Evolutionary approaches have proven highly effective in designing and generating SNNs, thus in this work we employ an evolutionary training algorithm called Evolutionary Optimization for Neuromorphic systems (EONS) [33]. EONS optimizes parameters of the network like number of neurons, number of synapses, connection between neurons and parameters of the neurons and synapses over time. Schuman et al [34] EONS uses the traditional evolutionary approach, it randomly initializes a parent network of a given population size, and then evaluates that network using a fitness score. It uses tournament selection and mutation to produce a child population. In the next epoch, this child population becomes the parent and the process is repeated for a number of epochs. The fitness function is usually specific to the application being solved, in this work for the classification tasks, accuracy is used as the fitness, for the control tasks, the fitness score is the reward associated with completing the task.

A RC systems consist of a highly recurrent neural network known as a reservoir that maps input into a high-dimensional space and a readout layer that transforms the high-dimensional states within the reservoir. The reservoir layer's operation relies on the dynamics of spiking neurons and their interconnections, rather than memristors. We leverage sparse recurrent connections with synaptic delays between spiking neurons, and synaptic plasticity to enable the reservoir to capture and retain information over longer time periods. The readout layer is typically trained with a simple linear regression method [35]. Several studies have shown RC as an effective method for temporal/sequential data processing [36, 37]. In this work, we use two approaches to generate the reservoir architecture; the first is a fixed, randomly initialized SNN as the reservoir which acts as the feature extractor for the input data. The reservoir has 100 hidden neurons and 20 reservoir output neurons that feed into the readout layer, we use a 10% connectivity within the reservoir. We use a simple linear classifier [38] to train the readout layer. For the second reservoir, we use EONS to optimize the internal structure of the reservoir [the number of neurons and synapses, and their connectivity], we retain 20 neurons and a linear classifier for the reservoir readout layer.

For the classification problems, the primary objective was to maximize the accuracy on the training set. Each network begins its simulation by processing a data sample, and ultimately producing a classification label. This label is determined either through a network trained by EONS or via the readout layer of a reservoir. Each dataset is split into training and testing samples, with the training dataset used for network training and the test dataset employed for subsequent evaluation. We evaluate three simple datasets from the UCI machine learning repository [43], and one timeseries dataset [42]. Table 4 provides detailed features of each dataset.

Control applications involve the implementation of algorithms or strategies aimed at regulating and manipulating system behavior to achieve predefined fitness scores or optimize specific performance criteria. Within these applications, each action is assigned a reward value, determining its contribution to the overall control task objective. The cumulative reward obtained from executing multiple actions to achieve the task is referred to as the fitness score. These applications commonly utilize feedback mechanisms and decision-making processes to steer the system towards desired outcomes, thereby contributing to the overall control and optimization of system behavior. We evaluate four control applications, which are outlined below, and detail their respective observations and actions in table 5.

• TENNLab Control Applications: these applications were internally developed for TENNLab control baseline evaluations. They were all developed using C++.
  • Pole Balance: in this application, imagine a virtual scenario with a cart positioned on a track, supporting a standing pole. Your choices are limited to pushing the cart either to the left or the right. The aim is to navigate the cart strategically, preventing collisions with the environment's walls and maintaining the pole's upright position. Your ultimate goal is to extend the time the cart avoids hitting the wall while ensuring the pole remains stable, for a cumulative duration of up to five minutes.
  • Bowman: in this application, envision a bowman situated at the bottom of the screen, tasked with moving his bow across the playing field in both clockwise and counterclockwise directions to target birds in flight. Your actions involve determining the direction to move the bow and when to shoot arrows. The primary goal is to optimize the number of birds successfully shot while concurrently minimizing the usage of arrows.
• OpenAI Gym Applications: these applications were developed and maintained by OpenAI Gym [44].
  • CartPole-v1: this is the OpenAI version of the pole balance application discussed earlier. In this application, there's a cart accompanied by a standing pole in an upright position. The challenge is to navigate the cart horizontally while maintaining the pole's upright stance. The available actions involve striking the pole either from the left or the right.
  • Lunar-Lander-v2: the aim of this application is to oversee the descent of a spaceship toward the surface. The actions include activating the left, right, or main engine, or refraining from firing any engine.

• (i)  
  EONS: figure 10 shows the performance of the EONS-trained network on the classification tasks. Three static and one temporal dataset are considered to evaluate the impact of STDP in neuroprocessor. The accuracy is scaled from 0 to 1, where the highest value denotes the better performance of the processor. Figure 10(a) shows the testing performance of the Iris dataset with three different test cases such as (i) blue: neuroprocessor with memristive-Mem-STDP, (ii) orange: neuroprocessor with default or digital-Def-STDP, and (iii) green: neuroprocessor without-No-STDP. According to figure 10(a), the MemStdp shows optimized accuracy distributions compared to DefStdp. However, the NoStdp configuration shows better performance than others. Figure 10(b) shows the testing performance of the Wine dataset, where DefStdp shows optimized data distributions compared to other test cases. In addition, MemStdp shows better performance distribution compared to NoStdp. Figure 10(c) shows the testing performance on the breast cancer dataset. DefStdp performed better than other test cases. Finally, the testing performance on EEG Emotion figure 10(d) shows, that the performance of all test cases is similar with a similar data distribution. From the results, different datasets exhibit enhanced performance with specific design configurations, emphasizing the need for tailored solutions. The memristive device consistently achieves comparable accuracies across all datasets, showcasing its efficacy in comparison to other processor types.
• (ii)  
  Evolved Reservoir: in figure 11, we thoroughly examined how EONS evolved reservoir networks performed across various classification datasets. We evaluated their accuracy on three static datasets (Iris, Wine and breast cancer), both in training and testing scenarios, providing a detailed understanding of their capabilities. We exclude the temporal dataset from this experiment due to resource constraints. Looking specifically at the Iris dataset figure 11(a), each neuroprocessor shows commendable performance. MemSTDP stands out with an impressive mean accuracy of around 98%, slightly surpassing DefSTDP and NoSTDP, which achieve approximately 97% and 96%, respectively. While these differences are subtle, it is important to highlight the overall consistency across the three neuroprocessors. Discussing the Wine and breast cancer datasets (figures 11(b) and (c)), we notice a similar trend. Each neuroprocessor demonstrates comparable accuracy distributions with similar maximum and minimum accuracies. This consistency emphasizes the adaptability of STDP implementations in the evolved reservoirs, showcasing their reliability across different static datasets in classification tasks.
• (iii)  
  Randomly Initialized Reservoir: the outcomes for the randomly initialized reservoir are illustrated in figure 12. We conducted tests on three static datasets and the EEG Emotion temporal datasets. Again, the performance of each neuroprocessor varies across these datasets. In the case of both the Iris and Wine datasets, we observe that NoSTDP slightly outperforms Default or Digital-Def-STDP and Memrisitive-Mem-STDP. Specifically, in figure 12(a), all neuroprocessors exhibit mean accuracies of approximately 85%. For the Wine dataset (figure 12(b)), NoSTDP outperforms the other processors with a mean accuracy of 63%, DefSTDP achieves a mean accuracy of 57%, while MemSTDP has an accuracy of 55%. For the breast cancer dataset shown in figure 12(c), networks generated for MemSTDP shows better accuracy distributions, when compared to the DefSTDP and NoSTDP Neuroprocessors. MemSTDP networks show a maximum accuracy of 90%, a minimum accuracy of 65% and a mean accuracy of 75%. Both DefSTDP and NoSTDP show maximum and minimum accuracies of ≈90% and 58% respectively. All networks across each neuroprocessor achieve a mean accuracy of ≈75%. Finally, the EEG dataset figure 12(d) shows a wide accuracy distribution across each processor, with NoSTDP slightly outperforming the others with a max accuracy of 70%, a minimum accuracy of 35% and a mean accuracy of 75%.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

In summary, looking at how different training algorithms performed on four different datasets for our classification task, we noticed some common patterns. The best performing STDP neuroprocessor appears to be contingent upon the specific dataset. Notably, the memristive STDP consistently demonstrates comparable accuracies across all scenarios and, in certain instances, even outperforms alternative STDP implementations, including some cases where it produces networks with more consistent performances as indicated by a narrower distribution in figures 10(b) and (d). These results underscore the nuanced relationship between STDP implementations and dataset characteristics, emphasizing the importance of tailoring the approach to the specific requirements of the task at hand.

We utilized the Evolutionary Optimization (EONS) training algorithm to evaluate the performance of three distinct neuroprocessors in control applications. The results, illustrated in figure 13, showcase the performance of networks generated by EONS. For each application, the fitness score refers to a reward associated with control actions, as defined in section 3.4.2. The final fitness score is the overall reward obtained from performing a series of actions to achieve the control task objective. Similar to the classification tasks, we conducted three different test cases: (i) blue represents the neuroprocessor with memristive-Mem-STDP, (ii) orange for the neuroprocessor with default or digital-Def-STDP, and (iii) green for the neuroprocessor without -No- STDP.

Zoom In Zoom Out Reset image size

The outcomes of the control applications present an intriguing and diverse narrative, with each neuroprocessor exhibiting unique behaviors based on the specific application. In figures 13(a) and (d), we observe that the inclusion of STDP negatively impacts the performance of the application. However, for NoSTDP, there is a widely varied distribution for both the Polebalance and Lunar Lander applications. In contrast, MemSTDP and DefSTDP demonstrate a narrower distribution of fitness scores across all networks. Concerning CartPole (figure 13(c)), each neuroprocessor performs relatively well, with DefSTDP showing less variation across the fitness of other networks. MemSTDP achieves a maximum fitness of 45, outperforming DefSTDP, and the mean fitness score across each neuroprocessor is approximately 35. For Bowman (figure 13(b)), the inclusion or exclusion of STDP results in similar fitness levels. Notably, networks without STDP exhibit a less varied distribution, indicating that more networks in this category have fitness scores within the Interquartile Range (IQR) of 0.40 to 0.55. In contrast, other processors show a more diverse distribution, with an IQR of 0.26 to 0.55 for MemSTDP and an IQR of 0.35 to 0.58.

As observed in section 3.5.1, the results suggest that various implementations of STDP in neuromorphic hardware may offer superior performance in specific applications compared to others.