CU Medicine-developed app proves effective in assessing depression
CU Medicine has developed a mobile depression assessment app that deploys multimodal artificial intelligence (AI) analysis, which means using AI to analyse the user’s multimodal data, including facial expressions, voice, language and subjective mood state, as well as rest-activity statistics, measured using actigraphy. CUHK researchers proved the new technology is effective in diagnosing depression. (From left) Professor Wing Yun-kwok, Choh-Ming Li Professor of Psychiatry and Department Chairman; Dr Watson Chen Jie, Postdoctoral Fellow ; and Dr Tim Li Man-ho, Assistant Professor, from the Department of Psychiatry at CU Medicine.
Depression is the leading health-related burden globally, affecting an estimated 300 million people. In Hong Kong, the prevalence of depression was about 8.3% according to the 2015 Hong Kong Mental Morbidity Survey. The current gold standard for the diagnosis of depression relies on clinical evaluation, but mental health services are insufficient to meet the demand in a lot of regions, including Hong Kong. The Chinese University of Hong Kong’s (CUHK) Faculty of Medicine (CU Medicine) has developed a mobile depression assessment app that deploys multimodal artificial intelligence (AI) analysis, which means using AI to analyse the user’s multimodal data, including facial expressions, voice, language and subjective mood state, as well as rest-activity statistics, measured using actigraphy. CUHK researchers proved the new technology is effective in diagnosing depression. The findings were recently published in Translational Psychiatry.
Over 70% of people with common mental disorders have not sought help from mental health services
The 2015 Hong Kong Mental Morbidity Survey showed that over 70% of people with common mental disorders, including depressive disorder and anxiety disorder, had not sought help from mental health services, due to lack of awareness, stigma or inaccessibility. Digital phenotyping has emerged to provide mental status assessments without face-to-face clinical consultations, increasing people’s willingness to seek help.
Professor Wing Yun-kwok says that depression is more than simply sadness but encompasses a series of physiological, cognitive, mood, language and rest-activity changes, which can be measured in a scientific way for assessment.
Professor Wing Yun-kwok, Choh-Ming Li Professor of Psychiatry and Chairman of Department of Psychiatry at CU Medicine, said, “Depression is more than simply sadness but encompasses a series of physiological, cognitive, mood, language and rest-activity changes. Therefore, other than conventional assessment by clinicians, digital phenotyping, which measures a series of parameters, has great potential to help with the assessment and monitoring of depression.
App records facial expression, voice and subjective mood state
Researchers from the Department of Psychiatry at CU Medicine conducted a case-control study to evaluate the performance of multimodal digital assessment for depression in Chinese. They recruited 101 subjects with major depressive disorder (MDD) and 82 controls free from psychiatric disorders between June 2021 and March 2023, who received actigraphy measurement and/or app assessment for one week. The assessment involved the measurement of rest-activity patterns via actigraphy, and also used a mobile app developed by the team to rate subjective mood state and record facial expression, voice during description of feelings in video clips.
The study’s key findings include:
- Rest-activity measurement results showed that MDD subjects were mobile for less time and suffered from circadian rhythm problems such as delayed sleep midpoint.
- Facial expression analysis showed MDD subjects had more eyebrow narrowing and less lip corner pulling.
- MDD subjects also appeared to refer to themselves more during conversations and use more negative emotion words. MDD subjects exhibited a lower articulation rate and increased variation in pause duration during speech.
- The average happiness level in remitted and non-remitted MDD subjects was lower than control.
(Left) Professor Wing explains major depressive disorder subjects show more eyebrow narrowing, a melancholic facial movement also known as the omega sign. (Right) Dr Chen mimics eyebrow narrowing, a common facial expression of patients of depression.
Professor Wing elaborated, “Our findings from facial expression analysis are concurrent with our previous hypothesis, in which MDD subjects showed more eyebrow narrowing, a melancholic facial movement also known as the omega sign. In line with other studies, on Caucasians, we found that Chinese MDD subjects exhibited a lower articulation rate and tended to use more first-person-singular pronouns, which reflects a universal cross-cultural phenomenon of self-focused attention in depressive patients. These are some markers we can use to identify people at a higher risk of depressive emotions, and the digital-AI platform might further aid in medical education and training.”
The mobile app developed by the CU Medicine team rates subjective mood state and records facial expression and voice during description of feelings in video clips. The whole process takes about two minutes.
Dr Watson Chen Jie, Postdoctoral Fellow in the Department of Psychiatry at CU Medicine, said, “With further AI analysis, our team found that the fusion of all digital modalities can achieve a decent performance, with an F1 score[1] of 0.81, in predicting lifetime diagnosis of MDD. Furthermore, by combining the conventional self-reporting scale with multimodal assessment, performance in predicting the non-remission status of MDD can reach an F1 score of 0.70, better than an isolated modality or self-report scale.”
Dr Tim Li Man-ho mentions the mobile app is only used in research of depression patients for the time being. Research team will need a more diverse and bigger sample size to develop the app for depression assessment in community in the future.
Dr Tim Li Man-ho, Assistant Professor in the Department of Psychiatry, said, “We plan to develop an automated AI system, with digital biomarkers for depression obtained from this research, that specialises in detecting and monitoring depression in the Chinese community. Depression is very common and the demand for its health services is huge. The new technology has the potential to assist diagnosis and monitoring of this health issue, helping to alleviate the pressure on healthcare providers.”
This study was funded by the Health Bureau’s Health and Medical Research Fund.
[1] F1 score is a measure of a test’s accuracy, ranging from 0 to 1, with a value of 1 represents perfect performance.
