A Computer Vision Approach for Non-contact Psychophysiological Assessment: Speaking-Aware Video-Based Stress Detection Using Extended TSST Protocol
Article Sidebar
Main Article Content
Abstract
The assessment of psychological stress through non-contact computer vision methods faces significant challenges when facial expressions are affected by motions caused by speech. In this paper, we introduce a novel video-based stress detection system with speaking awareness that dynamically processes facial features according to online detection of speech activities. The system employs an extended 41-minute Trier Social Stress Test protocol with a controlled baseline, moderate stress induction, peak stress increase, and recovery stages to comprehensively record the dynamics of stress development. Facial landmark detection is handled by MediaPipe Face Mesh and provides 468 three-dimensional landmarks with sub-pixel accuracy, while the speaking-aware processing strategy dynamically selects the most appropriate feature sets: seven robust ones during speech-containing intervals like face movement, eye aspect ratio, and forehead wrinkles, and nine full-featured ones during silent intervals with additional mouth and jaw metrics. The adaptive strategy addresses the intrinsic limitation of traditional facial analysis, which treats speaking and non-speaking states homogeneously. Deployment on the Raspberry Pi CM4 edge computing device enables real-time operation with privacy preservation via localized processing. Empirical testing with 71 participants illustrates strong performance with an F1-score of 83.16%, outperforming conventional methods by 4.24%. Cross-population validation affirms good generalization potential across various populations for the entire 41-minute protocol, thereby demonstrating the system's applicability to real-world applications in stress tracking across healthcare, educational, and workplace well-being contexts.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright
Submission of a manuscript implies that the submitted work has not been published before (except as part of a thesis or report, or abstract); that it is not under consideration for publication elsewhere; that its publication has been approved by all co-authors. If and when the manuscript is accepted for publication, the author(s) still hold the copyright and retain publishing rights without restrictions. Authors or others are allowed to multiply article as long as not for commercial purposes. For the new invention, authors are suggested to manage its patent before published. The license type is CC-BY-NC 4.0.
Disclaimer
No responsibility is assumed by publisher and co-publishers, nor by the editors for any injury and/or damage to persons or property as a result of any actual or alleged libelous statements, infringement of intellectual property or privacy rights, or products liability, whether resulting from negligence or otherwise, or from any use or operation of any ideas, instructions, procedures, products or methods contained in the material therein.
References
American Institute of Stress, "Workplace stress statistics and facts," 2023. [Online]. Available: https://www.stress.org/workplace-stress/
C. Kirschbaum, K. M. Pirke, and D. H. Hellhammer, “The ‘Trier social stress test’ - A tool for investigating psychobiological stress responses in a laboratory setting,” Neuropsychobiology, vol. 28, no. 1–2, pp. 76–81, 1993, doi: 10.1159/000119004.
P. F. Lovibond and S. H. Lovibond, "The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories," Behaviour Research and Therapy, vol. 33, no. 3, pp. 335-343, 1995.
J. A. Healey and R. W. Picard, “Detecting stress during real-world driving tasks using physiological sensors,” IEEE Trans. Intell. Transp. Syst., vol. 6, no. 2, pp. 156–166, 2005, doi: 10.1109/TITS.2005.848368.
P. Schmidt, A. Reiss, R. Duerichen, and K. Van Laerhoven, “Introducing WeSAD, a multimodal dataset for wearable stress and affect detection,” ICMI 2018 - Proc. 2018 Int. Conf. Multimodal Interact., pp. 400–408, 2018, doi: 10.1145/3242969.3242985.
C. Lugaresi et al., “MediaPipe: A Framework for Building Perception Pipelines,” 2019, [Online]. Available: http://arxiv.org/abs/1906.08172.
G. Giannakakis, D. Grigoriadis, K. Giannakaki, O. Simantiraki, A. Roniotis, and M. Tsiknakis, “Review on Psychological Stress Detection Using Biosignals,” IEEE Trans. Affect. Comput., vol. 13, no. 1, pp. 440–460, 2022, doi: 10.1109/TAFFC.2019.2927337.
G. Giannakakis, A. Roussos, C. Andreou, S. Borgwardt, and A. I. Korda, “Stress recognition identifying relevant facial action units through explainable artificial intelligence and machine learning,” Comput. Methods Programs Biomed., vol. 259, no. November 2024, p. 108507, 2025, doi: 10.1016/j.cmpb.2024.108507.
D. Canedo and A. J. R. Neves, “Facial expression recognition using computer vision: A systematic review,” Appl. Sci., vol. 9, no. 21, pp. 1–31, 2019, doi: 10.3390/app9214678.
A. F. Abate, L. Cimmino, B. C. Mocanu, F. Narducci, and F. Pop, “The limitations for expression recognition in computer vision introduced by facial masks,” Multimed. Tools Appl., vol. 82, no. 8, pp. 11305–11319, 2023, doi: 10.1007/s11042-022-13559-8.
B. M. Kudielka, A. Buske-Kirschbaum, D. H. Hellhammer, and C. Kirschbaum, “HPA axis responses to laboratory psychosocial stress in healthy elderly adults, younger adults, and children: Impact of age and gender,” Psychoneuroendocrinology, vol. 29, no. 1, pp. 83–98, 2004, doi: 10.1016/S0306-4530(02)00146-4.
F. X. Lesage and S. Berjot, “Validity of occupational stress assessment using a visual analogue scale,” no. April, pp. 434–436, 2011, doi: 10.1093/occmed/kqr037.