Correlation Analysis of In-Vehicle Sensors Data and Driver Signals in Identifying Driving and Driver Behaviors
<p>Methodological procedure flowchart.</p> "> Figure 2
<p>Route applied in the tests. The drivers started and ended the route at the same place.</p> "> Figure 3
<p>Data source (vehicle sensor information from the CAN network and vehicle instrumented sensors).</p> "> Figure 4
<p>Signal processing steps.</p> "> Figure 5
<p>Example of signals obtained from the vehicle and driver for each sensor on (<b>I</b>) less cautious and (<b>II</b>) cautious state: CAN bus (accelerator, RPM, velocity data), sEMG signals, brake pedal sensor, and IMU data (accelerometer and gyroscope for the three axes). The values of the CAN bus are normalized for the acquisition. Some moments are highlighted for some conditions, such as step on brake pedal (a), pressing the accelerator (b), use of brake on the road (b,c), straight paths on the road (d), and end of the route (e).</p> "> Figure 6
<p>Distributions for each class based on the features extracted from <a href="#sensors-23-00263-t003" class="html-table">Table 3</a>. The data were separated for driver behavior for Cautious (C) and Less Cautious (L) conditions. The features of five drivers were also explored. All of the features were normalized between 0 and 1.</p> "> Figure 7
<p>The accuracies and sensibilities obtained from each feature set for (<b>I</b>) drivers’ behavior and (<b>II</b>) their identity in the following classifiers: (<b>a</b>) k-NN, (<b>b</b>) SVM, and (<b>c</b>) RF.</p> "> Figure 8
<p>Distributions for the Tukey post-hoc from Friedman statistical test to evaluate the distribution of results with a confidence interval of 95% (<span class="html-italic">p</span>-value < 0.05) for (<b>I</b>) driver’s behavior and (<b>II</b>) driver’s identification. (<b>a</b>) presents the analysis for the classifiers and (<b>b</b>) the analysis for each sensor set.</p> ">
Abstract
:1. Introduction
- Could the driver’s leg movement information (electromyography and inertial modules) provide studies of driving patterns?
- Could this driver’s information be somehow correlated with data from the vehicle’s internal sensors?
- Could these data jointly be used to feed computational intelligence tools to assess driver behavior?
- Since there is a correlation between driver data and vehicle data, would data from the vehicle alone be sufficient to assess driver behavior? And how accurately?
- What would be the minimum set of sensors inside the vehicle that could provide enough information to correctly describe the driver’s driving mode and still identify him? And how accurately?
2. Materials and Methods
2.1. Experimental Protocol
2.2. Instrumentation and Data Acquisition
2.3. Signal Processing and Pattern Recognition
3. Results and Discussion
Discussion about the State-of-the-Art
- (a)
- Using only vehicle data: accuracy of 0.82 and 0.66 from behavior and driver identification, respectively;
- (b)
- Only driver data: accuracy of 0.91 and 0.96 from behavior and driver identification, respectively;
- (c)
- Vehicle and driver data: (accuracy of 0.94 and 0.96) from behavior and driver identification, respectively.
4. Practical Implications and Limitation of the Study
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
ABS | Anti-lock braking systems |
ADAC | Automotive data acquisition center |
ADAS | Advanced driver assistance systems |
CAAE | Ethical Committee for Research on Humans |
CAN | Controlled area network |
ESC | Electronic stability control |
ECG | Electrocardiography |
EEG | Electroencephalography |
IMU | Inertial measurement unit |
k-NN | k-nearest neighbors |
MAV | Mean absolute value |
NHTSA | National Highway Traffic Safety Administration |
ODB2 | On board diagnostics, version 2 |
RF | Random forest |
RPM | Revolutions per minute |
sEMG | Surface electromyography |
SVM | Support vector machine |
TCS | Traction control system |
References
- NHTSA. Distracted Driving in Fatal Crashes, 2017; NHTSA: Washington, DC, USA, 2019.
- AAA Foundation for Traffic Safety. Prevalence of Self-Reported Aggressive Driving Behavior: United States, 2014; AAA Foundation for Traffic Safety: Washington, DC, USA, 2016. [Google Scholar]
- NHTSA. Fatalities in Frontal Crashes Despite Seat Belts and Air Bags; NHTSA: Washington, DC, USA, 2009.
- NHTSA. The Long-Term Effect of ABS in Passenger Cars and LTVs; NHTSA: Washington, DC, USA, 2009.
- Høye, A. The effects of Electronic Stability Control (ESC) on crashes—An update. Accid. Anal. Prev. 2011, 43, 1148–1159. [Google Scholar] [CrossRef]
- Ziebinski, A.; Cupek, R.; Grzechca, D.; Chruszczyk, L. Review of Advanced Driver Assistance Systems (ADAS). In AIP Conference Proceedings; AIP Publishing LLC: Melville, NY, USA, 2017; p. 120002. [Google Scholar] [CrossRef]
- Mühlbacher-Karrer, S.; Faller, L.M.; Hamid, R.; Zangl, H. A wireless steering wheel gripping sensor for hands on/off detection. In Proceedings of the 2016 IEEE Sensors Applications Symposium (SAS), Catania, Italy, 20–22 April 2016; pp. 1–5. [Google Scholar] [CrossRef]
- Ferreira, J.; Carvalho, E.; Ferreira, B.V.; de Souza, C.; Suhara, Y.; Pentland, A.; Pessin, G. Driver behavior profiling: An investigation with different smartphone sensors and machine learning. PLoS ONE 2017, 12, e0174959. [Google Scholar] [CrossRef]
- Farinelli, F.A. Sistema de Aquisicao E Classificacao de Dados Para Reconhecimento da Presenca das Maos Do Motorista No Volante Por Redes Neurais Artificiais. Ph.D. Thesis, UTFPR, Ponta Grossa, Brazil, 2019. [Google Scholar]
- Liu, L.; Wang, Z.; Qiu, S. Driving Behavior Tracking and Recognition Based on Multisensors Data Fusion. IEEE Sens. J. 2020, 20, 10811–10823. [Google Scholar] [CrossRef]
- Alshaqaqi, B.; Baquhaizel, A.S.; El Amine Ouis, M.; Boumehed, M.; Ouamri, A.; Keche, M. Vision based system for driver drowsiness detection. In Proceedings of the 2013 11th International Symposium on Programming and Systems (ISPS), Algiers, Algeria, 22–24 April 2013; IEEE: Algiers, Algeria, 2013; pp. 103–108. [Google Scholar] [CrossRef]
- Li, N.; Jain, J.J.; Busso, C. Modeling of Driver Behavior in Real World Scenarios Using Multiple Noninvasive Sensors. IEEE Trans. Multimed. 2013, 15, 1213–1225. [Google Scholar] [CrossRef]
- Rodrigues, J.G.P.; Vieira, F.; Vinhoza, T.T.V.; Barros, J.; Cunha, J.P.S. A non-intrusive multi-sensor system for characterizing driver behavior. In Proceedings of the 13th International IEEE Conference on Intelligent Transportation Systems, Funchal, Madeira Island, Portugal, 19–22 September 2010; IEEE: Madeira Island, Portugal, 2010; pp. 1620–1624. [Google Scholar] [CrossRef]
- Rebolledo-Mendez, G.; Reyes, A.; Paszkowicz, S.; Domingo, M.C.; Skrypchuk, L. Developing a Body Sensor Network to Detect Emotions During Driving. IEEE Trans. Intell. Transp. Syst. 2014, 15, 1850–1854. [Google Scholar] [CrossRef]
- McCall, J.C.; Trivedi, M.M. Driver Behavior and Situation Aware Brake Assistance for Intelligent Vehicles. Proc. IEEE 2007, 95, 374–387. [Google Scholar] [CrossRef]
- Kim, B.; Baek, Y. NETTED: Non-Standard CAN ID Extraction System Based on Deep Neural Networks. In Proceedings of the 2018 IEEE 3rd International Conference on Communication and Information Systems (ICCIS), Singapore, 28–30 December 2018; IEEE: Singapore, 2018; pp. 198–202. [Google Scholar] [CrossRef]
- Martinelli, F.; Mercaldo, F.; Orlando, A.; Nardone, V.; Santone, A.; Sangaiah, A.K. Human behavior characterization for driving style recognition in vehicle system. Comput. Electr. Eng. 2020, 83, 102504. [Google Scholar] [CrossRef]
- Lu, J.; Filev, D.; Prakah-Asante, K.; Tseng, F.; Kolmanovsky, I.V. From vehicle stability control to intelligent personal minder: Real-time vehicle handling limit warning and driver style characterization. In Proceedings of the 2009 IEEE Workshop on Computational Intelligence in Vehicles and Vehicular Systems, Nashville, TN, USA, 30 March–2 April 2009; IEEE: Nashville, TN, USA, 2009; pp. 43–50. [Google Scholar] [CrossRef]
- Filev, D.; Lu, J.; Tseng, F.; Prakah-Asante, K. Real-time driver characterization during car following using stochastic evolving models. In Proceedings of the 2011 IEEE International Conference on Systems, Man, and Cybernetics, Anchorage, AK, USA, 9–12 October 2011; IEEE: Anchorage, AK, USA, 2011; pp. 1031–1036. [Google Scholar] [CrossRef]
- Raz, O.; Fleishman, H.; Mulchadsky, I. System and Method for Vehicle Driver Behavior Analysis and Evaluation. U.S. Patent US7389178B2, 17 June 2008. [Google Scholar]
- Obuhuma, J.; Okoyo, H.; Mcoyowo, S. Real-time Driver Advisory Model: Intelligent Transportation Systems. In Proceedings of the 2018 IST-Africa Week Conference (IST-Africa), Gaborone, Botswana, 9–11 May 2018; p. 11. [Google Scholar]
- Khan, M.Q.; Lee, S. A Comprehensive Survey of Driving Monitoring and Assistance Systems. Sensors 2019, 19, 2574. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- NHTSA. Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey. Traffic Safety Facts Crash Stats 2018. Number: DOT HS 812 506. Available online: https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812506 (accessed on 10 November 2022).
- Petridou, E.; Moustaki, M. Human Factors in the Causation of Road Traffic Crashes. Eur. J. Epidemiol. 2000, 16, 819–826. [Google Scholar] [CrossRef] [PubMed]
- Lattanzi, E.; Freschi, V. Machine Learning Techniques to Identify Unsafe Driving Behavior by Means of In-Vehicle Sensor Data. Expert Syst. Appl. 2021, 176, 114818. [Google Scholar] [CrossRef]
- Tadano, Y.S.; Potgieter-Vermaak, S.; Kachba, Y.R.; Chiroli, D.M.; Casacio, L.; Santos-Silva, J.C.; Moreira, C.A.; Machado, V.; Alves, T.A.; Siqueira, H.; et al. Dynamic model to predict the association between air quality, COVID-19 cases, and level of lockdown. Environ. Pollut. 2021, 268, 115920. [Google Scholar] [CrossRef] [PubMed]
- de Souza Tadano, Y.; Siqueira, H.V.; Alves, T.A. Unorganized machines to predict hospital admissions for respiratory diseases. In Proceedings of the 2016 IEEE Latin American Conference on Computational Intelligence (LA-CCI), Cartagena, Colombia, 2–4 November 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 1–6. [Google Scholar]
- Hallac, D.; Sharang, A.; Stahlmann, R.; Lamprecht, A.; Huber, M.; Roehder, M.; Sosic, R.; Leskovec, J. Driver Identification Using Automobile Sensor Data from a Single Turn. arXiv 2017, arXiv:1708.04636. [Google Scholar]
- Zhang, J.; Wu, Z.; Li, F.; Xie, C.; Ren, T.; Jie, C.; Liu, L. A Deep Learning Framework for Driving Behavior Identification on In-Vehicle CAN bus Sensor Data. Sensors 2019, 19, 1356. [Google Scholar] [CrossRef] [Green Version]
- Li, F.; Zhang, H.; Che, H.; Qiu, X. Dangerous Driving Behavior Detection Using Smartphone Sensors. In Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil, 1–4 November 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 1902–1907. [Google Scholar] [CrossRef]
- Petraki, V.; Ziakopoulos, A.; Yannis, G. Combined impact of road and traffic characteristic on driver behavior using smartphone sensor data. Accid. Anal. Prev. 2020, 144, 105657. [Google Scholar] [CrossRef] [PubMed]
- Saleh, K.; Hossny, M.; Nahavandi, S. Driving behavior classification based on sensor data fusion using LSTM recurrent neural networks. In Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan, 16–19 October 2017; pp. 1–6. [Google Scholar] [CrossRef]
- Hamdy, A.; Atia, A.; Mostafa, M.S. Recognizing Driving Behavior and Road Anomaly using Smartphone Sensors. Int. J. Ambient. Comput. Intell. 2017, 8, 22–37. [Google Scholar] [CrossRef] [Green Version]
- Khodairy, M.; Abo Samra, G.A.A. Driving Behavior Classification Based on Oversampled Signals of Smartphone Embedded Sensors Using an Optimized Stacked-LSTM Neural Networks. IEEE Access 2021, 9, 4957–4972. [Google Scholar] [CrossRef]
- Lindow, F.; Kashevnik, A. Driver Behavior Monitoring Based on Smartphone Sensor Data and Machine Learning Methods. In Proceedings of the 2019 25th Conference of Open Innovations Association (FRUCT), Helsinki, Finland, 5–8 November 2019; pp. 196–203. [Google Scholar] [CrossRef]
- Phinyomark, A.; Quaine, F.; Charbonnier, S.; Serviere, C.; Tarpin-Bernard, F.; Laurillau, Y. EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Syst. Appl. 2013, 40, 4832–4840. [Google Scholar] [CrossRef]
- Phinyomark, A.; N. Khushaba, R.; Scheme, E. Feature Extraction and Selection for Myoelectric Control Based on Wearable EMG Sensors. Sensors 2018, 18, 1615. [Google Scholar] [CrossRef] [PubMed]
REF | Objective | Sensor/Data | Main Results | Study Limitation |
---|---|---|---|---|
[25] | Recognize safe and unsafe driver behaviors using machine learning models | Data from CAN bus (51 signals, focus on vehicle speed, engine speed, engine load, throttle position, steering wheel angle, brake pedal pressure)/10 drivers | Classifiers presented accuracies above 90 | Signals from the driver were not considered |
[28] | Identify the driver using a shorter segment of road | CAN bus signals (12 signals, including steering wheel angle, velocity, and acceleration; vehicle velocity and heading, engine RPM, gas pedal position, brake pedal position, forward acceleration, lateral acceleration, torque, throttle position)/10 cars and 64 drivers | System obtained accuracy of 76% between 2 drivers and 50% for 5 drivers | Signals from the driver were not inserted on the analysis. |
[29] | Driving behavior identification by deep learning, with automatic activity recognition on real-time-based CAN bus data and without feature selection | 51 features from CAN bus sensor data/Use an Ocslab driving dataset, with 94401 records, ten drivers, two rounds trips | The proposed model reached above 95% without feature extraction | Used a database and did not correlate with sensors without CAN bus. |
[30] | Detect abnormal behavior (such as speed changes and steering) during driving using a smartphone | Accelerometer and gyroscope from smartphone/4 smartphones in a taxi | Four typical dangerous driving behaviors were detected: abnormal speeding up or down, steering, weaving, and operating smartphone, with 90% of accuracy. A novel method to compute the yaw angle was proposed. | Use only smartphone to acquire the data, without correlation with variable on a car. Difficulty in weaving detection. |
[31] | Examine driver behavior using smartphone sensors by harsh accelerations and harsh decelerations across junctions and road segments | Sensors on smartphone (accelerometer, gyroscope, magnetometer, and GPS) and traffic characteristics by inductive loops/data from 303 drivers in two urban expressways | Authors realized that there were harsh increases on road segments if average traffic flow per lane also increases | Authors did not recognize patterns, they used prediction models. |
[32] | Recognize driver behavior (normal, aggressive, and drowsy driving) using an LSTM and sensors on a smartphone | Accelerometer (x, y, and z acceleration), gyroscope (roll, pitch, and yaw angles), GPS (vehicle speed), and camera sensor (distance ahead and number of detected vehicles)/UAH-DriveSet (two types of road, six drivers and vehicles, and three driving behaviors) | LSTM provides a mean f1-score of 0.91 for all sets without feature extraction | The authors used a database and did not explore the influence of data obtained from a car. |
[33] | Recognize driving events using DTW and KNN using a smartphone, such as right and left turns, right of left lane change and road anomalies. The movements were analyzed along the three axes. Drivers were classified with abnormal (aggressive) and normal driving behavior. | Accelerometer, gyroscope, and GPS from smartphone on the center of vehicle dashboard/two drivers in two different vehicles (540 driving events) | About 99% for driving detection using KNN and about 97% for driving behavior with DTW | Authors only used smartphone data, without correlation with variables on a car. |
[34] | Two approaches to recognize driver behavior (normal, drowsy, and aggressive, and aggressive and non-aggressive) using an LSTM model. | Accelerometer, gyroscope, GPS, preprocessed vehicle detection (video recordings)/UAH-DriveSet (two types of road, six drivers and vehicles, and three driving behaviors) | Winner model with f1-score of 99% | The authors used a database. |
[35] | Detection of dangerous states for accident prevention by driver behavior monitoring. | Smartphone (accelerometer, gyroscope, GPS, and microphone) and camera/Data acquired from 10 volunteers | States such as distraction and drowsiness were recognize | Authors did not provide overall accuracy and data from the bus were not applied. |
Action | Driving Type | |
---|---|---|
Calm, Cautious | Hurried, Less Cautious | |
Initial acceleration | Slow, gradual | Abrupt, fast |
Breaking | Slow, gradual | Abrupt |
Unforeseen, simulation | Slow braking | Abrupt braking |
Regaining steering control | Slow, gradual | Abrupt, fast |
Lane change | Slow, gradual | Abrupt, steering wheel movement |
Curve | Moderate speed, reduction | Minimum speed, reduction |
Sensor | Features |
---|---|
sEMG | Mean absolute value (MAV) |
Accelerometer | Mode in each axis (X, Y, Z) |
Gyroscope | Mode in each axis (X, Y, Z) |
Break | Median |
Velocity | Mean |
RPM | Mean |
Accelerator | Median |
Sensor Set | Data Origin | Selected Features |
---|---|---|
1 | Vehicle | Velocity |
2 | Vehicle | RPM |
3 | Vehicle | Velocity + RPM |
4 | Vehicle | Accelerator |
5 | Vehicle | Velocity + RPM + Accelerator |
6 | Vehicle | Brake Pedal |
7 | Vehicle | Accelerator + Brake Pedal |
8 | Vehicle | Velocity + RPM + Brake Pedal |
9 | Vehicle | Velocity + RPM + Accelerator + Brake Pedal |
10 | Driver | Inertial |
11 | Driver | sEMG |
12 | Driver | Inertial + sEMG |
13 | Vehicle + Driver | Velocity + RPM + Accelerator + Inertial + sEMG |
14 | Vehicle + Driver | Velocity + RPM + Accelerator + Brake Pedal + Inertial + sEMG |
Behavior Identification | Driver Identification | ||||||
---|---|---|---|---|---|---|---|
Feature Set | k-NN | SVM | RF | k-NN | SVM | RF | |
In-Vehicle Sensor Data only | 5 | 0.80 | 0.76 | 0.79 | |||
8 | 0.81 | 0.76 | 0.80 | 0.66 | 0.64 | 0.66 | |
9 | 0.82 | 0.76 | 0.80 | 0.52 | 0.51 | 0.56 | |
Driver Data only | 12 | 0.81 | 0.82 | 0.91 | 0.91 | 0.91 | 0.96 |
Vehicle and Driver Data | 13 | 0.89 | 0.86 | 0.93 | 0.93 | 0.91 | 0.96 |
14 | 0.89 | 0.86 | 0.94 | 0.93 | 0.91 | 0.96 |
REF | Objective | Sensor/Data | Main Results | Study Limitation |
---|---|---|---|---|
[25] | Recognize safe and unsafe driver behaviors using machine learning models | Data from CAN bus (51 signals, focus on vehicle speed, engine speed, engine load, throttle position, steering wheel angle, brake pedal pressure)/10 drivers, resulting in 26 h | Classifiers presented accuracies above 90 | Signals from the driver were not considered |
[28] | Identify the driver using a shorter segment of road | CAN bus signals (12 signals, including steering wheel angle, velocity, and acceleration; vehicle velocity and heading, engine RPM, gas pedal position, brake pedal position, forward acceleration, lateral acceleration, torque, throttle position)/10 cars and 64 drivers | System obtained accuracy of 76% between two drivers and 50 for five drivers | Signals from the driver were not inserted on the analysis. |
[29] | Driving behavior identification by deep learning, with automatic activity recognition on real-time-based CAN bus data and without feature selection | 51 features from CAN bus sensor data/Used a Ocslab driving dataset, with 94401 records, ten drivers, two rounds trips | The proposed model reached above 95% without feature extraction | Used a database and did not correlate with sensors without CAN bus. |
This work | Analyze if there is a correlation between signals from the vehicle (can) with data instrumented on the driver’s leg/foot to propose a system for the detection of dangerous states for accident prevention by driver behavior monitoring, based on two standards of driver behaviors and identifying of the driver. | An acquisition board developed by the authors was used, and only three CAN data were captured (velocity + RPM + accelerator), in addition to externally instrumented data (break pedal) and driver leg/foot movement signals (sEMG and inertial movements) | The results presented a good correlation between driver and vehicle signals. About the accuracies: (a) Using only vehicle data: accuracy of 0.82 and 0.66 from behavior and driver identification, respectively; (b) Only drive data: accuracy of 0.91 and 0.96 from behavior and driver identification, respectively; (c) Vehicle and driver data: accuracy of 0.94 and 0.96 from behavior and driver identification, respectively. | Number of users, a limited number of parameters, but enough to analyze and prove the correlation between driver and vehicle data, for future study without driver data. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bonfati, L.V.; Mendes Junior, J.J.A.; Siqueira, H.V.; Stevan, S.L., Jr. Correlation Analysis of In-Vehicle Sensors Data and Driver Signals in Identifying Driving and Driver Behaviors. Sensors 2023, 23, 263. https://doi.org/10.3390/s23010263
Bonfati LV, Mendes Junior JJA, Siqueira HV, Stevan SL Jr. Correlation Analysis of In-Vehicle Sensors Data and Driver Signals in Identifying Driving and Driver Behaviors. Sensors. 2023; 23(1):263. https://doi.org/10.3390/s23010263
Chicago/Turabian StyleBonfati, Lucas V., José J. A. Mendes Junior, Hugo Valadares Siqueira, and Sergio L. Stevan, Jr. 2023. "Correlation Analysis of In-Vehicle Sensors Data and Driver Signals in Identifying Driving and Driver Behaviors" Sensors 23, no. 1: 263. https://doi.org/10.3390/s23010263
APA StyleBonfati, L. V., Mendes Junior, J. J. A., Siqueira, H. V., & Stevan, S. L., Jr. (2023). Correlation Analysis of In-Vehicle Sensors Data and Driver Signals in Identifying Driving and Driver Behaviors. Sensors, 23(1), 263. https://doi.org/10.3390/s23010263