Efficient and Robust Driver Fatigue Detection Framework Based on the Visual Analysis of Eye States

Yancheng Ling; Xiaoxiong Weng

doi:10.7307/ptt.v35i4.223

Authors

Yancheng Ling School of Civil Engineering and Transportation, South China University of Technology
Xiaoxiong Weng School of Civil Engineering and Transportation, South China University of Technology

DOI:

https://doi.org/10.7307/ptt.v35i4.223

Keywords:

fatigue detection, visual-based, fusion, PERCLOS, PLCDB

Abstract

Fatigue detection based on vision is widely employed in vehicles due to its real-time and reliable detection results. With the coronavirus disease (COVID-19) outbreak, many proposed detection systems based on facial characteristics would be unreliable due to the face covering with the mask. In this paper, we propose a robust visual-based fatigue detection system for monitoring drivers, which is robust regarding the coverings of masks, changing illumination and head movement of drivers. Our system has three main modules: face key point alignment, fatigue feature extraction and fatigue measurement based on fused features. The innovative core techniques are described as follows: (1) a robust key point alignment algorithm by fusing global face information and regional eye information, (2) dynamic threshold methods to extract fatigue characteristics and (3) a stable fatigue measurement based on fusing percentage of eyelid closure (PERCLOS) and proportion of long closure duration blink (PLCDB). The excellent performance of our proposed algorithm and methods are verified in experiments. The experimental results show that our key point alignment algorithm is robust to different scenes, and the performance of our proposed fatigue measurement is more reliable due to the fusion of PERCLOS and PLCDB.

References

Toroyan T, Peden MM, Iaych K. WHO launches second global status report on road safety. Injury Prevention. 2013;19(2):150. DOI: 10.1136/injuryprev-2013-040775.

Zhang G, Yau KKW, Zhang X, Li Y. Traffic accidents involving fatigue driving and their extent of casualties. Accident Analysis and Prevention. 2016;87:34–42. DOI: 10.1016/j.aap.2015.10.033.

Williamson A, et al. The link between fatigue and safety. Accident Analysis and Prevention. 2016;43(2):498–515. DOI: 10.1016/j.aap.2009.11.011.

Dawson D, Searle AK, Paterson JL. Look before you (s)leep: Evaluating the use of fatigue detection technologies within a fatigue risk management system for the road transport industry. Sleep Medicine Reviews. 2016;18(2):141–152. DOI: 10.1016/j.smrv.2013.03.003.

National Highway Traffic Safety Administration. Drowsy Driving 2015: A Brief Statistical Summary. Oct. 2017. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812446.

Amodio A, et al. Automatic detection of driver impairment based on pupillary light reflex. IEEE Transactions on Intelligent Transportation Systems. 2019;20(8):3038–3048. DOI: 10.1109/TITS.2018.2871262.

Seen KS, Mohd Tamrin SB, Meng GY. Driving fatigue and performance among occupational drivers in simulated prolonged driving. Global Journal of Health Science. 2010;2(1). DOI: 10.5539/gjhs.v2n1p167.

Kokonozi AK, Michail EM, Chouvarda IC, Maglaveras NM. A study of heart rate and brain system complexity and their interaction in sleep-deprived subjects. Computers in Cardiology. 2008;35:969–971. DOI: 10.1109/CIC.2008.4749205.

Kar S, Bhagat M, Routray A. EEG signal analysis for the assessment and quantification of driver’s fatigue. Transportation Research Part F: Traffic Psychology and Behaviour. 2010;13(5):297–306. DOI: 10.1016/j.trf.2010.06.006.

Galley N. Blink Parameter as indicator of drivers sleepiness. Nursing Standard. 2003;23(35):26–27. http://www.ncbi.nlm.nih.gov/pubmed/19749128.

Patel M, Lal SKL, Kavanagh D, Rossiter P. Applying neural network analysis on heart rate variability data to assess driver fatigue. Expert Systems with Applications. 2011;38(6):7235–7242. DOI: 10.1016/j.eswa.2010.12.028.

Ahmed J, Li JP, Khan SA, Shaikh RA. Eye behaviour based drowsiness Detection System. 2015 12th International Computer Conference on Wavelet Active Media Technology and Information Processing, ICCWAMTIP. 2015. p. 268–272. DOI: 10.1109/ICCWAMTIP.2015.7493990.

Chau L, Yap K. Driver fatigue detection.

Li K, Gong Y, Ren Z. A fatigue driving detection algorithm based on facial multi-feature fusion. IEEE Access. 2020;8:101244–101259. DOI: 10.1109/ACCESS.2020.2998363.

Luo XQ, Hu R, Fan TE. The driver fatigue monitoring system based on face recognition technology. Proceedings of the 2013 International Conference on Intelligent Control and Information Processing, ICICIP. 2013. p. 384–388. DOI: 10.1109/ICICIP.2013.6568102.

Deng W, Wu R. Real-time driver-drowsiness detection system using facial features. IEEE Access. 2019;7:118727–118738. DOI: 10.1109/access.2019.2936663.

Coetzer RC, Hancke GP. Eye detection for a real-time vehicle driver fatigue monitoring system. IEEE Intelligent Vehicles Symposium, Proceedings. 2011. p. 66–71. DOI: 10.1109/IVS.2011.5940406.

Savaş BK, Becerikli Y. Real time driver fatigue detection system based on multi-task ConNN. IEEE Access. 2020;8:12491–12498. DOI: 10.1109/ACCESS.2020.2963960.

Smith P, Shah M, da Vitoria Lobo N. Determining driver visual attention with one camera. IEEE Transactions on Intelligent Transportation Systems. 2003;4(4):205–218. DOI: 10.1109/TITS.2003.821342.

Picot A, Charbonnier S, Caplier A. On-line detection of drowsiness using brain and visual information. IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans. 2012. DOI: 10.1109/TSMCA.2011.2164242.

Horng WB, Chen CY, Chang Y, Fan CH. Driver fatigue detection based on eye tracking and dynamic template matching. IEEE International Conference on Networking, Sensing and Control. 2004;1:7-12. DOI: 10.1109/ICNSC.2004.1297595.

King DE. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research. 2009;10:1755–1758.

Cheng Q, et al. Assessment of driver mental fatigue using facial landmarks. IEEE Access. 2019;7:150423–150434. DOI: 10.1109/ACCESS.2019.2947692.

You F, et al. A real-time driving drowsiness detection algorithm with individual differences consideration. IEEE Access.2019;7:179396–179408. DOI: 10.1109/ACCESS.2019.2958667.

Sun Y, et al. Driver fatigue detection system based on colored and infrared eye features fusion. Computers, Materials and Continua. 2020;63(3):1563–1574. DOI: 10.32604/CMC.2020.09763.

Devi MS, Bajaj PR. Fuzzy based driver fatigue detection. Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics. 2010. p. 3139–3144. DOI: 10.1109/ICSMC.2010.5641788.

Savas BK, Becerikli Y. Real time driver fatigue detection based on SVM Algorithm. 2018 6th International Conference on Control Engineering and Information Technology, CEIT 2018. 2018. DOI: 10.1109/CEIT.2018.8751886.

Wierwille WW, et al. Research on vehicle-based driver status/performance monitoring; Development, validation, and refinement of algorithms for detection of driver drowsiness. NTIS. Report No. DOT HS 808 247, 2019.

Caffier PP, Erdmann U, Ullsperger P. Experimental evaluation of eye-blink parameters as a drowsiness measure. European Journal of Applied Physiology. 2003;89(3–4):319–325. DOI: 10.1007/s00421-003-0807-5.

Bandara IB. Driver drowsiness detection based on eye blink. 2009. p. 1–4. http://collections.crest.ac.uk/9782/1/Bandara Indrachapa - Thesis pdf.pdf.

Huang R, Wang Y. P-FDCN based eye state analysis for fatigue detection. 2018 IEEE 18th International Conference on Communication Technology (ICCT). 2018. p. 1174–1178.

Devi MS, Bajaj PR. Driver fatigue detection based on eye tracking. Proceedings - 1st International Conference on Emerging Trends in Engineering and Technology, ICETET 2008. 2008. p. 649–652. DOI: 10.1109/ICETET.2008.17.

Tabrizi PR, Zoroofi RA. Open/closed eye analysis for drowsiness detection. 2008 1st International Workshops on Image Processing Theory, Tools and Applications, IPTA 2008. 2008. DOI: 10.1109/IPTA.2008.4743785.

Mandal B, Li L, Wang GS. Lin J. Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Transactions on Intelligent Transportation Systems. 2017;18(3):545–557. DOI: 10.1109/TITS.2016.2582900.

Of O, Carriers M. PERCLOS: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance. 1998;31(5):1237–1252. http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:PERCLOS+:+A+Valid+Psychophysiological+Measure+of+Alertness+As+Assessed+by+Psychomotor+Vigilance#0.

Johns MW. The amplitude velocity ratio of blinks: A new method for monitoring drowsiness. Sleep. 2003;26:A51-52.

Trutschel U, et al. PERCLOS: An alertness measure of the past. Driving Assessment Conference. 2011. p. 172–179. DOI: 10.17077/drivingassessment.1394.

Guo X, et al. PFLD: A practical facial landmark detector. arXiv:1902.10859 [cs.CV]. DOI: 10.48550/arXiv.1902.10859.

Geng L, Liang X, Xiao Z, Li Y. Real-time driver fatigue detection based on morphology infrared features and deep learning. Hongwai Yu Jiguang Gongcheng/Infrared and Laser Engineering. 2018;47(2). DOI: 10.3788/IRLA201847.0203009.

Wang H, et al. A novel real-time driving fatigue detection system based on wireless dry EEG. Cognitive Neurodynamics. 2018;12(4):365–376. DOI: 10.1007/s11571-018-9481-5.

Jap BT, Lal S, Fischer P, Bekiaris E. Using EEG spectral components to assess algorithms for detecting fatigue. Expert Systems with Applications. 2009;36(2 PART 1):2352–2359. DOI: 10.1016/j.eswa.2007.12.043.

Söylemez ÖF, Ergen B. Eye location and eye state detection in facial images using circular Hough transform. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8104 LNCS. 2013. p. 141–147. DOI: 10.1007/978-3-642-40925-7_14.

Cheng E, Kong B, Hu R, Zheng F. Eye state detection in facial image based on linear prediction error of wavelet coefficients. 2008 IEEE International Conference on Robotics and Biomimetics, ROBIO 2008. 2009. p. 1388–1392. DOI: 10.1109/ROBIO.2009.4913203.

Miah AA, Ahmad M, Mim KZ. Drowsiness detection using eye-blink pattern and mean eye landmarks’ distance. Proceedings of International Joint Conference on Computational Intelligence. Algorithms for Intelligent Systems. 2020. p. 111–121. DOI: 10.1007/978-981-13-7564-4_10.

Liu H, Soto RAR, Xiao F, Lee YJ. YolactEdge: Real-time instance segmentation on the edge (Jetson AGX Xavier: 30 FPS, RTX 2080 Ti: 170 FPS). 2020. http://arxiv.org/abs/2012.12259.

Tsuyama M, et al. Embedded implementation of human detection using only color features on the NVIDIA Xavier. Proceedings - 2019 International Symposium on Intelligent Signal Processing and Communication Systems, ISPACS 2019. 2019. DOI: 10.1109/ISPACS48206.2019.8986281.