Research on Multimodal Human-Machine Interface for Takeover Request of Automated Vehicles
Downloads
In L3 automated driving, the driver performing the non-driving related tasks (NDRT) is easy to miss the takeover request and cause safety hazards. The takeover prompt strategy has a great impact on this situation. In this paper, four multi-modal takeover interfaces for automatic driving are designed to address the typical takeover scenarios in which the driver is under medium and high-level task loads. The driving simulator is used to conduct experiments, and each scheme’s takeover success rate, takeover time and takeover quality are selected as the evaluation criteria to study the effect of different interfaces on the driver’s takeover performance. The results show that the multimodal takeover interface can shorten takeover time, and the visual-auditory-tactile prompt has the shortest takeover time; and the visual-auditory prompt and auditory-tactile prompt have nearly the same takeover time, but the latter increases the longitudinal deceleration of the vehicle; the visual-tactile prompt has the worst takeover performance. These results provide practical implications for developing suitable interfaces to remind drivers to take over the automated vehicles.
Downloads
SAE International. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles - SAE international [Internet]. www.sae.org. 2021. Available from: https://www.sae.org/standards/content/j3016_202104/
Melnicuk V, Thompson S, Jennings P, Birrell S. Effect of cognitive load on drivers’ state and task performance during automated driving: Introducing a novel method for determining stabilisation time following take-over of control. Accident Analysis & Prevention. 2021;151:105967, DOI: 10.1016/j.aap.2020.105967.
Li Q, et al. Drivers’ visual-distracted take-over performance model and its application on adaptive adjustment of time budget. Accident Analysis & Prevention. 2021;154:106099–9, DOI: 10.1016/j.aap.2021.106099.
Hadi AM, et al. Influence of passive fatigue and take-over request lead time on drivers’ take-over performance. Proceedings of the AHFE 2020 Virtual Conference on Human Aspects of Transportation, 16-20 July 2020, USA, Springer, 2020, p. 253–259. DOI: 10.1007/978-3-030-50943-9_32.
Pepin G, et al. Impact of mind‐wandering on visual information processing while driving: An electrophysiological study. Applied Cognitive Psychology. 2020;35(2):508–516. DOI: 10.1002/acp.3773.
Gold C, Körber M, Lechner D, Bengler K. Taking over control from highly automated vehicles in complex traffic situations. Human Factors: The Journal of the Human Factors and Ergonomics Society. 2016;58(4):642–652. DOI: 10.1177/0018720816634226.
Körber M, Gold C, Lechner D, Bengler K. The influence of age on the take-over of vehicle control in highly automated driving. Transportation Research Part F: Traffic Psychology and Behaviour. 2016;39:19–32. DOI: 10.1016/j.trf.2016.03.002.
Du N, et al. Examining the effects of emotional valence and arousal on takeover performance in conditionally automated driving. Transportation Research Part C: Emerging Technologies. 2020;112:78–87. DOI: 10.1016/j.trc.2020.01.006.
Wan J, Wu C. The effects of lead time of take-over request and nondriving tasks on taking-over control of automated vehicles. IEEE Transactions on Human-Machine Systems. 2018;48(6):582–591. DOI: 10.1109/THMS.2018.2844251.
Eriksson A, Stanton NA. Takeover time in highly automated vehicles: noncritical transitions to and from manual control. Human Factors: The Journal of the Human Factors and Ergonomics Society. 2017;59(4):689–705. DOI:10.1177/0018720816685832.
Gold C, Damböck D, Lorenz L, Bengler K.Take over!” How long does it take to get the driver back into the loop? Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57, p.1938 - 1942. DOI: 10.1177/1541931213571433.
Wang HC, Guo Z, Rau PLP. The shorter takeover request time the better? Car-driver handover control in highly automated vehicles. Automation, Collaboration, & E-Services. 2022; pp.3–17. DOI: 10.1007/978-3-031-10784-9_1.
Endsley MR. Toward a theory of situation awareness in dynamic systems. Human Factors: the Journal of the Human Factors and Ergonomics Society [Internet]. 1995;37(1):32–64. Available from: https://journals.sagepub.com/doi/10.1518/001872095779049543.
Gilson RD, Human factors and ergonomics society. Situation Awareness. Santa Monica, Ca.: Human Factors And Ergonomics Society; 1995.
Kim H, et al. A study on the effects of providing situation awareness information for the control authority transition of automated vehicle. In: 2019 International Conference on Information and Communication Technology Convergence (ICTC); 2019 Oct; Jeju Island, Korea. IEEE; 2019. p. 1394–1396. DOI: 10.1109/ICTC46691.2019.8939867.
Liu W, et al. A literature review on additional semantic information conveyed from driving automation systems to drivers through advanced in-vehicle HMI just before, during, and right after takeover request. International Journal of Human–Computer Interaction. 2023;39(10):1995–2015. DOI: 10.1080/10447318.2022.2074669.
Atakishiyev S, Salameh M, Goebel R. Incorporating explanations into human-machine interfaces for trust and situation awareness in autonomous vehicles. In: 2024 IEEE Intelligent Vehicles Symposium (IV); 2024 Jun; Jeju Island, Korea. IEEE; 2024. p. 2948–2955. DOI: 10.1109/IV55156.2024.10588812.
De Winter JCF, Happee R, Martens MH, Stanton NA. Effects of adaptive cruise control and highly automated driving on workload and situation awareness: A review of the empirical evidence. Transportation Research Part F Traffic Psychology and Behavior. 2014;27:196–217. DOI: 10.1016/j.trf.2014.06.016.
Weaver BW, DeLucia PR. A systematic review and meta-analysis of takeover performance during conditionally automated driving. Human Factors: The Journal of the Human Factors and Ergonomics Society. 2020;001872082097647. DOI: 10.1177/0018720820976476.
Pfleging B, Rang M, Broy N. Investigating user needs for non-driving-related activities during automated driving. In: Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia; 2016 Dec; Rovaniemi, Finland. ACM; 2016. pp. 91–99. DOI: 10.1145/3012709.3012735.
Zhang N, et al. Influence of non-driving related tasks on driving performance after takeover transition in conditionally automated driving. Transportation research Part F, Traffic psychology and behaviour. 2023;96:248–264. DOI: 10.1016/j.trf.2023.05.009.
Naujoks F, Purucker C, Neukum A. Secondary task engagement and vehicle automation – Comparing the effects of different automation levels in an on-road experiment. Transportation Research Part F: Traffic Psychology and Behaviour. 2016;38:67–82. DOI: 10.1016/j.trf.2016.01.011.
Chen W, Sawaragi T, Hiraoka T. Adaptive multi-modal interface model concerning mental workload in take-over request during semi-autonomous driving. SICE Journal of Control, Measurement, and System Integration. 2021;1–12.DOI: 10.1080/18824889.2021.1894023.
Zhang B, et al. Determinants of take-over time from automated driving: A meta-analysis of 129 studies. Transportation Research Part F: Traffic Psychology and Behaviour. 2019:285–307. DOI: 10.1016/j.trf.2019.04.020.
Petermeijer S, Doubek F, de Winter J. Driver response times to auditory, visual, and tactile take-over requests: a simulator study with 101 participants. In: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC); 2017 Oct; Banff, Canada. IEEE; 2017. pp. 1505–1510. DOI: 10.1109/SMC.2017.8122827.
Houtenbos M, et al. Concurrent audio-visual feedback for supporting drivers at intersections: A study using two linked driving simulators. Applied Ergonomics. 2017 Apr. 60:30–42. DOI: 10.1016/j.apergo.2016.10.010.
Talamonti W, et al. Mirage events & driver haptic steering alerts in a motion-base driving simulator: A method for selecting an optimal HMI. Applied Ergonomics. 2017;65:90–104. DOI: 10.1016/j.apergo.2017.05.009.
Sanghavi H, Zhang Y, Jeon M. Effects of anger and display urgency on takeover performance in semi-automated vehicles. In: 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; 2020 Sep; Virtual Event, DC, USA. ACM; 2020. pp. 48–56. DOI: 10.1145/3409120.3410664.
Li S, et al. Evaluation of the effects of age-friendly human-machine interfaces on the driver’s takeover performance in highly automated vehicles. Transportation Research Part F: Traffic Psychology and Behaviour. 2019;67:78–100. DOI: 10.1016/j.trf.2019.10.009.
Petermeijer SM, Cieler S, de Winter JCF. Comparing spatially static and dynamic vibrotactile take-over requests in the driver seat. Accident Analysis & Prevention. 2017;99:218–227. DOI: 10.1016/j.aap.2016.12.001.
SELCON SJ, Taylor RM, Mckenna FP. Integrating multiple information sources: using redundancy in the design of warnings. Ergonomics. 1995;38(11):2362–2370. DOI: 10.1080/00140139508925273.
Petermeijer S, Bazilinskyy P, Bengler K, de Winter J. Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop. Applied Ergonomics. 2017;62:204–215. DOI: 10.1016/j.apergo.2017.02.023.
Bazilinskyy P, et al. Take-over requests in highly automated driving: A crowdsourcing survey on auditory, vibrotactile, and visual displays. Transportation Research Part F: Traffic Psychology and Behaviour. 2018;56:82–98. DOI: 10.1016/j.trf.2018.04.001.
Diaconescu AO, Alain C, McIntosh AR. The co-occurrence of multisensory facilitation and cross-modal conflict in the human brain. Journal of Neurophysiology. 2011;106(6):2896–909. DOI: 10.1152/jn.00303.2011.
Talsma D, Senkowski D, Soto-Faraco S, Woldorff MG. The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences. 2010;14(9):400–10. DOI: 10.1016/j.tics.2010.06.008.
Tan X, Zhang Y. Analyzing driver strategy for scheduled takeovers under the effect of multi-stage takeover request design. Proceedings of the Human Factors and Ergonomics Society. 2023;67(1):1197-1203. DOI: 10.1177/21695067231192422. DOI: 10.1177/21695067231192422.
Borowsky A, Zangi N, Oron-Gilad T. Interruption management in the context of take-over-requests in conditional driving automation. IEEE Transactions on Human-Machine Systems. 2022;52(5):1015–1024. DOI: 10.1109/THMS.2022.3194006.
Jiang L, Wang X, Li Z, Zhang Y. Research on design model of human-machine interface of automatic driving takeover system based on user experience. Lecture notes in computer science. 2019;41–60. DOI: 10.1007/978-3-030-23538-3_4.
Bazilinskyy P, de Winter JCF. Analyzing crowdsourced ratings of speech-based take-over requests for automated driving. Applied Ergonomics. 2017;64:56–64. DOI: 10.1016/j.apergo.2017.05.001.
Forster Y, Naujoks F, Neukum A, Huestegge L. Driver compliance to take-over requests with different auditory outputs in conditional automation. Accident Analysis & Prevention. 2017;109:18–28. DOI: 10.1016/j.aap.2017.09.019.
Kim S, van Egmond R, Happee R. Effects of user interfaces on take-over performance: A review of the empirical evidence. Information. 2021;12(4):162. DOI: 10.3390/info12040162.
Schwarz F, Fastenmeier W. Augmented reality warnings in vehicles: Effects of modality and specificity on effectiveness. Accident Analysis & Prevention. 2017;101:55–66. DOI: 10.1016/j.aap.2017.01.019.
Zhou H, Keita K, Itoh M, Satoshi K. Effects of explanation-based knowledge regarding system functions and driver’s roles on driver takeover during conditionally automated driving: A test track study. Transportation research Part F, Traffic psychology and behaviour. 2021;77:1–9. DOI: 10.1016/j.trf.2020.11.015.
Lu Z, et al. Human factors of transitions in automated driving: A general framework and literature survey. Transportation Research Part F: Traffic Psychology and Behaviour. 2016;43:183–198. DOI: 10.1016/j.trf.2016.10.007.
Happee R, et al. Take-over performance in evasive manoeuvres. Accident Analysis & Prevention. 2017;106:211–222.DOI: 10.1016/j.aap.2017.04.017.
Young W, Sobhani A, Lenné MG, Sarvi M. Simulation of safety: A review of the state of the art in road safety simulation modelling. Accident Analysis & Prevention. 2014;66:89–103. DOI: 10.1016/j.aap.2014.01.008.
Hecht D, Reiner M, Karni A. Multisensory enhancement: Gains in choice and in simple response times. Experimental Brain Research. 2008;189(2):133–143. DOI: 10.1007/s00221-008-1410-0.
Politis I, Brewster S, Pollick F. Speech tactons improve speech warnings for drivers. In: Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; 2014 Sep; Seattle, WA, USA. ACM; 2014. pp. 1–8. DOI: 10.1145/2667317.2667318.
Politis I, Brewster S, Pollick F. Language-based multimodal displays for the handover of control in autonomous cars. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications; 2015 Sep; Nottingham, United Kingdom. ACM; 2015. pp. 3–10. DOI: 10.1145/2799250.2799262.
Huang G, Steele C, Zhang X, Pitts BJ. Multimodal cue combinations: a possible approach to designing in-vehicle takeover requests for semi-autonomous driving. Proc Hum Factors Ergon Soc Annu Meet. 2019;63(1):1739–43. pp. 1739–1743. DOI: 10.1177/1071181319631053.
Hancock PA, et al. Tactile cuing to augment multisensory human-machine interaction. Ergonomics in Design The Quarterly of Human Factors Applications. 2015;23(2):4–9. DOI: 10.1177/1064804615572623.
Meng F, Spence C. Tactile warning signals for in-vehicle systems. Accident Analysis & Prevention. 2015 Feb 1;75:333–346. DOI: 10.1016/j.aap.2014.12.013.
Faltaous S, Schönherr C, Detjen H, Schneegass S. Exploring proprioceptive take-over requests for highly automated vehicles. In: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia; 2019 Nov; Pisa, Italy. ACM; 2019. p. 1–6. DOI: 10.1145/3365610.3365644.
Zeeb K, Buchner A, Schrauf M. What determines the take-over time? An integrated model approach of driver take-over after automated driving. Accident Analysis & Prevention. 2015;78:212–221. DOI: 10.1016/j.aap.2015.02.023.
Mahajan K, Large DR, Burnett G, Velaga NR. Exploring the benefits of conversing with a digital voice assistant during automated driving: A parametric duration model of takeover time. Transportation Research Part F: Traffic Psychology and Behaviour. 2021;80:104–126. DOI: 10.1016/j.trf.2021.03.012.
Sarter NB. Multimodal information presentation: Design guidance and research challenges. International Journal of Industrial Ergonomics. 2006;36(5):439–445. DOI: 10.1016/j.ergon.2006.01.007.
Zeeb K, Härtel M, Buchner A, Schrauf M. Why is steering not the same as braking? The impact of non-driving related tasks on lateral and longitudinal driver interventions during conditionally automated driving. Transportation Research Part F: Traffic Psychology and Behaviour. 2017;50:65–79. DOI: 10.1016/j.trf.2017.07.008.
Abbas-Zadeh M, Hossein-Zadeh GA, Vaziri-Pashkam M. Dual-task interference in a simulated driving environment: Serial or parallel processing? Frontiers in Psychology. 2021;11. DOI: 10.3389/fpsyg.2020.579876.
Copyright (c) 2025 Junfeng WANG, Yue WANG, Yin CUI
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
