Annotation of the results obtained in 2022
In the course of scientific research for 2022, we have developed new neural network technologies based on reservoir computing (RC), including entropy analysis methods, and have shown the application of these technologies in important industries such as medicine and the Internet of Things. The work, in particular, was based on the results of the previous RSF project No. 16-19-00135 (2019-2020), where we proposed the architecture of the LogNNet reservoir neural network and a number of analog neuromorphic circuits. Main results: 1. A methodology has been developed for determining the most effective routine blood values(RBV) in the diagnosis/prediction of COVID-19 for the reservoir neural network LogNNet. The most important RBVs that affect disease diagnosis from the first dataset were mean corpuscular hemoglobin concentration (MCHC), mean corpuscular hemoglobin (MCH), and activated partial prothrombin time (aPTT). The LogNNet model has demonstrated high accuracy in diagnosing COVID-19 disease without knowledge of symptoms or patient history. The model is suitable for devices with limited hardware resources (3-14 KB of RAM), such as the Arduino software and hardware platform, and is promising for creating mobile health monitoring systems in the Internet of Things. The proposed method will reduce the burden on the healthcare sector, improve understanding of the pathogenesis of COVID-19 through the analysis of key blood parameters, and may contribute to the treatment process. 2. Identified predictors of COVID-19 mortality based on blood counts using machine learning models. The square of the metric F1^2 = F1(class 1)*F1 (class 2) calculated based on the values for both classes was used as a classification criterion. It was found that the most successful classification algorithm was Histogram-based Gradient Boosting (HGB) using 34 blood parameters. The pairs of procalcitonin with D-dimer, erythrocyte sedimentation rate, direct bilirubin, and ferritin turned out to be the most effective pairs of traits. The HGB model, working with these feature pairs, correctly identified almost all surviving and deceased patients (Precision > 0.98, Recall > 0.98, F1^2 > 0.98). Taking into account all the results, we assume that procalcitonin and ferritin are the main indicators of blood in combination with which other indicators give high classification accuracy. The HGB model can be successfully used to identify the risk of death from COVID-19. To understand how the HGB model works in classifying patients with COVID-19, 1D and 2D masks of the HGB model were built. 3. A method that improves the classification accuracy of the LogNNet network has been implemented. By increasing the length of the input vector using the squares of the principal components, there is a slight increase in the classification accuracy. When using the SARS-CoV-2-RBV3 base in the case of a single-layer network, an increase in accuracy from 95% to 97% is observed. For the SARS-CoV-2-RBV1 baseline, the increase in accuracy with doubling of the input data is less noticeable, the observed change is from 99.5% to 99.7%. 4. Based on the leaky integrate-and-fire (LIF) oscillator neuron with frequency coding, which we proposed earlier, we developed a chaotic LIF oscillator neuron model. To implement a chaotic oscillator, the spikes are frequency-coded in a resistive feedback circuit with a second-order filter. We have developed an electronic circuit of a chaotic LIF neuron, in which we adhered to the principle of the absence of inductive connections and elements, which makes it possible to implement the circuit in a fully integrated design in the future. A bifurcation and entropy analysis of the chaotic oscillator scheme was carried out, which showed that the mechanisms of chaos development correspond to the period-doubling scenario. We have demonstrated that the developed chaotic integrate-and-fire oscillator with frequency coding in the feedback loop can be used in reservoir calculations, for example, in the LogNNet network. 5. A model of a physical reservoir based on relaxation oscillators (digital oscillators) with a high-order synchronization effect has been developed. As the interaction between the generators, a digital circuit was chosen, which mathematically closely corresponds to the thermal coupling between the oscillators. 6. A compact LogNNet algorithm consisting of 77 lines of code has been created, operating on the Arduino platform, designed to diagnose and predict COVID-19 disease by routine blood values. The advantage of using a chaotic map in the reservoir can be considered as a reduction in the consumption of RAM for the operation of a neural network, due to the effect of the determinism of chaos. Implemented LogNNet on an Arduino Nano 33 IoT board with limited computing resources. During testing, it was revealed that the developed algorithm allows obtaining the same accuracy as the original computer model: 99.7%. Such devices and algorithms are easily integrated into the Internet of Things. 7. For the first time it is shown that the entropy of short time series can be approximated by machine learning (ML) regression methods. The trained models recognized regions of chaos and order in satellite images by approximating the permutation entropy (PermEn), pattern entropy (SampEn), singular value entropy (SvdEn), and neural network entropy (NNetEn) algorithms. To implement the task, we used 200 images of the Earth's surface with a size of 256x256 pixels, obtained from the Sentinel-2 satellite in four different bands. With the help of a circular kernel, the pixels of the image were transformed into series of numbers, for which the entropy was calculated. The coefficient of determination R2 was used as a target metric in the selection of models. The most accurate results of SvdEn approximation were shown by the gradient boosting algorithm, for which the average value R2 = 0.996 according to the results of cross-validation. Reducing the length of the series by reducing the radius of the core increased the accuracy of the model: R2 increased from 0.947 to 0.997 when the length of the series decreased from 113 to 5 elements. The ML_SvdEn model was trained and tested using synthetic data. High results were obtained for the approximation of the logistic map (Pearson correlation coefficient 0.968) by the model trained on the Planck map. The speed of the machine learning models for ML_NNetEn, ML_PermEn, ML_SampEn turned out to be higher than that of the original algorithm, although the ML_PermEn, ML_ SampEn models did not show the highest accuracy. The greatest acceleration was achieved for the ML_NNetEn model (several orders of magnitude). Thus, as a result of this work, the possibility of a universal approximation of the entropy calculated by various methods using machine learning methods for series of numbers up to 113 elements long has been shown. 8. A technique for transforming a 2D image into a one-dimensional time series is proposed. Circular kernels have rotational symmetry and allow you to get the result of the 2D entropy distribution that is resistant to image rotation. 9. A technique has been developed using the reservoir neural network LogNNet and several standard classification algorithms for automatic labeling of Github reports.

 

Publications

---

Annotation of the results obtained in 2023
In the course of scientific research for 2023, we developed new neural network technologies based on reservoir computing (RC), and demonstrated their application in important industries such as medicine and the Internet of Things (IoT). The results are published in ranking journals cited by Scopus (' Nonlinear Dynamics ' (1 article), ' Sensors ' (2 articles), ' Algorithms ' (1 article), ' AIP conference proceedings ' (2 articles)), 2 conferences, 1 intellectual property rights and 26 media sources are presented. https://petrsu.ru/persons/1385/pages/15384/smi-o-laboratorii 1) A review of current trends in the development of artificial intelligence applications in the field of the Internet of Things, including intelligent healthcare services, intelligent object recognition, intelligent environmental monitoring and smart the rescue at spontaneous disasters. Completed thematic special issue in journal ‘Sensors’ titled “Artificial Neural Networks for IoT -Enabled Smart Applications”. 2) Optimized neural network LogNNet, based on the technology of "reservoir computing with auto-generation of weights". The main advantage of LogNNet is the low RAM consumption of NMC memory (Number of Memory Cells). Automatic generation of reservoir matrix weights through a congruent generator allows not to remember the entire matrix, but to generate weights as they are used in the calculation process. For example, the memory for the LogNNet 100:50:20:3 model is reduced to RAM (NMC 2)~ 5.12 kB. 3) An effective algorithm for selecting significant features for LogNNet through optimization of the hyperparameters of the congruent generator has been developed. 4) A new entropy measure NNetEn is proposed and a library in Python for its calculation is created. The entropy value is equivalent to one of the classification metrics of the standard MNIST-10 database. In the case of an ordered series, the MNIST-10 classification metrics are low, and vice versa, a chaotic series leads to increased classification metrics. The results show that NNetEn is an effective time series feature. As a practical application, the classification of electroencephalographic (EEG) signals from patients with Alzheimer's disease (36 people) and a control group (29 healthy people) is illustrated. The synergistic effect of increasing classification accuracy with the combined use of traditional entropy measures and NNetEn has been demonstrated. 5) Methods for using machine learning (classification algorithms) on various medical databases, such as determination of inflammation of appendicitis, data on mild appendicitis, data on coronary heart disease using blood markers, were studied. The LogNNet neural network was successfully used on these bases and showed the following target parameters, respectively: 1) MCC = 1, LogNNet 12:12:7:2, RAM (NMC 2)~ 0.6 kB ; 2) MCC = 0.93, LogNNet 12:11:6:2, RAM (NMC 2)~0.52 kB ; 3) MCC = 0.65, LogNNet 13:15:10:2, RAM (NMC 2)~1 kB. MCC - Matthew's correlation coefficient. 6) Github reports using LogNNet has been studied. LogNNet classification accuracy was 70%, that is, 6% less than the highest accuracy of standard algorithms shown by Logistic Regression. However, the memory footprint for the LogNNet 100:50:20:3 model was only RAM (NMC 2)~ 5.1 kB. 7) EmoSurv database was studied, which contains information about a person’s emotional state based on his typing style on the keyboard. The gradation of emotions has 5 states: anger, happiness, calm, sadness and neutral state. The best performance for the LogNNet 15:50:10:5 model showed an accuracy of MCC = 0.37 and RAM (NMC 2) ~ 2.7 kB. 8) A new discrete map (DM) is presented to describe the dynamics of a chaotic pulse position modulation oscillator. The model circuit has a voltage-controlled oscillator (PVCO) and a feedback loop (FB) with pulse-frequency encoding threshold, which is performed by a monostable multivibrator. The model scheme and DM demonstrate dynamic chaos in a wide range of control parameters. The transition to chaos in DM occurs abruptly either from a fixed point (tangent bifurcation) or from a limit cycle. An experimental hybrid (digital-analog) chaotic pulse generator circuit has been implemented, in which the PVCO and multivibrator are built on the basis of an IC NE 555 integrated circuit and with Arduino Uno microcontroller in FB circuit. The constructed DM can be considered a sigmoid modification of the well-known Bernoulli map. Unlike the Bernoulli map, our map has flexible settings with five parameters, as well as access to a chaotic attractor from any starting point. 9) Transfer learning of reservoir networks is studied using the example of predicting chaotic time series using echo networks (echo state networks). Two time series were generated using the Mackie- Glass equation with different parameter values and two models were trained (Model 1 and Model 2). Using a multi-task transfer learning approach, a third model (Model 3) was created whose output weights were calculated based on the output weights of the first two models. Model 3 predicted both time series with higher accuracy (RMSE~0.2). 10) A bioinspired model of a chaos sensor has been developed to approximate the entropy of a sequence of impulses in neurodynamic systems. The operating principle of the sensor is based on assessing the irregularity of time series using a perceptron. Short series of long NL = 50 elements containing time intervals between spikes generated by the Hindmarsh-Rose model were fed to the input of a single-layer perceptron, and the entropy of this sequence was estimated at the output. To assess the accuracy of entropy approximation, a base of 20000 short time series with different dynamics was used, and the coefficient of determination (R2) between perceptron values (Sensor on Perceptron, (SPE)) and ideal calculation using the FuzzyEn algorithm (Sensor on FuzzyEn, (SFU)). As a result, after training, the agreement between the readings of the perceptron and FuzzyEn was R2 ~ 0.88. Thus, we were the first to apply the approach of approximating the entropy of a time series using machine learning. 11) The results of the research on the chaos sensor model were used to record action potential impulses from the L5 dorsal root of the rat, where it was shown that the chaos sensor records different types of spike activity in the stimulation and rest modes of the root (experimental data were taken from open sources). 12) The possibility of assessing the synchronization of two pulse oscillators through the Fuzzy calculation of the entropy of mutual oscillations is shown. The degree of synchronization has the strongest negative cross-correlation with Fuzzy entropy. 13) We applied HVG coding (horizontal visibility graph) to series of audio information and showed that the resulting integer series take up about 4 times less disk space, and at the same time, the approach of generating features using mel spectrograms with subsequent classification of audio samples works with them. 14) The database of audio files in wav format has been studied soerenab / AudioMNIST from cloud platform for hosting IT projects. The initial data set consisted of audio samples (0.56 s) with a length of 12000 and sampling 22050 Hz, in which 50 different variants of each digit from 0 to 9 are pronounced. The neural network LogNNet 20:38:19:10 using only 20 features extracted from Mel-spectrograms classified the base with an MCC accuracy of ~0.92. The memory occupied by LogNNet is only RAM (NMC2)~4.18 kB. 15) A method for recognizing audio commands from a Google's Speech Commands dataset has been studied. For this purpose, 2212 files with 13 commands were selected from the original dataset. For each file, the Mel-spectral coefficients were calculated, as well as the 1st and 2nd derivatives of these coefficients using the Savitsky-Golay differential filter. The classification accuracy of the reservoir network LogNNet 1000:30:26:13, using only 1000 features, was MCC ~0.27, while the memory occupied by the model was only ~8.8 kB.

 

Publications

---