Melola Discovery

← Melola Discovery

Open access Jurnal SINTA 2

Optimization of Multimodal Deep Learning for Depression Detection

IJCCS (Indonesian Journal of Computing and Cybernetics Systems) · Vol. 19 No. 4 (2025) · pp. 383–394

DOI
10.22146/ijccs.111407
ISSN
2460-7258, 1978-1520
Published
2025-10-31
Publisher
IndoCEISS in colaboration with Universitas Gadjah Mada, Indonesia.
Language
eng
License
creativecommons.org/licenses/by-sa/4.0

Abstract

Depression is a complex and often underdiagnosed mental health condition that manifests through subtle verbal, acoustic, and behavioral cues. Traditional unimodal detection systems struggle to capture the full spectrum of depressive symptoms, often leading to inaccurate or incomplete assessments. This study proposes a multimodal deep learning framework that integrates textual, audio, and visual modalities to improve the robustness and reliability of automatic depression detection, achieving an overall classification accuracy of 74%. The approach prioritizes privacy and interpretability by using facial keypoints and gaze direction rather than raw video frames, and applies attention mechanisms to align and fuse features across modalities. Each modality is processed through dedicated neural architectures tailored to its data type, and their outputs are combined within a fusion model that learns to capture cross-modal emotional patterns. Experimental results demonstrate that the proposed multimodal system significantly outperforms its unimodal counterparts in terms of classification performance. The visual modality was found to contribute most strongly to detection accuracy, as confirmed by ablation analysis. These findings highlight the value of multimodal integration in capturing complex psychological signals and support the development of intelligent, non-invasive screening tools for use in digital mental health applications.

Keywords

Collaboration network of the authors

18 authors · 55 co-authorship links · 2 across different organisations

Hover or tap a name to highlight its co-authors. Double-click to open the author's page.

  • Universitas Buddhi Dharma
  • Satya Wacana Christian University
  • Affiliation unknown
  • Different organisations
  • Same organisation
  • Affiliation unknown
  • Thicker line = more joint articles · bigger circle = more articles
Show as table
Co-authorOrganisationJoint articles
Hendra HendraUniversitas Buddhi Dharma1
Edy EdyUniversitas Buddhi Dharma1
Junaedi JunaediUniversitas Buddhi Dharma1
Aditiya HermawanUniversitas Buddhi Dharma1
Yusuf KurniaUniversitas Buddhi Dharma1
Ardiane Rossi Kurniawan MarantoUniversitas Buddhi Dharma1
Irwan SembiringSatya Wacana Christian University1
Danny ManonggaSatya Wacana Christian University1
Sutarto WijonoAffiliation unknown1
Krismiyati KrismiyatiAffiliation unknown1
Theophilus WellemAffiliation unknown1
Sutarto WijonoAffiliation unknown1
Krismiyati KrismiyatiAffiliation unknown1
Theophilus WellemAffiliation unknown1

Affiliations come from OpenAlex; journal OAI-PMH metadata carries none. Authors without an OpenAlex ID are matched by name, which can merge or split people.

Metadata source: OAI-PMH (metadata jurnal) https://journal.ugm.ac.id/ijccs/oai, first harvested 26 Okt 2026, last seen 26 Okt 2026.
Original metadata (oai_dc) 
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en-US">Optimization of Multimodal Deep Learning for Depression Detection</dc:title>
	<dc:creator>Hermawan, Aditiya</dc:creator>
	<dc:creator>Daniawan, Benny</dc:creator>
	<dc:creator>Edy, Edy</dc:creator>
	<dc:creator>Nathaniel, Joese</dc:creator>
	<dc:subject xml:lang="en-US">Computer Science;Machine Learning; Deep Learning</dc:subject>
	<dc:subject xml:lang="en-US">Attention Mechanism; Depression Detection; Facial Keypoints; Mental Health Screening; Multimodal Deep Learning</dc:subject>
	<dc:description xml:lang="en-US">Depression is a complex and often underdiagnosed mental health condition that manifests through subtle verbal, acoustic, and behavioral cues. Traditional unimodal detection systems struggle to capture the full spectrum of depressive symptoms, often leading to inaccurate or incomplete assessments. This study proposes a multimodal deep learning framework that integrates textual, audio, and visual modalities to improve the robustness and reliability of automatic depression detection, achieving an overall classification accuracy of 74%. The approach prioritizes privacy and interpretability by using facial keypoints and gaze direction rather than raw video frames, and applies attention mechanisms to align and fuse features across modalities. Each modality is processed through dedicated neural architectures tailored to its data type, and their outputs are combined within a fusion model that learns to capture cross-modal emotional patterns. Experimental results demonstrate that the proposed multimodal system significantly outperforms its unimodal counterparts in terms of classification performance. The visual modality was found to contribute most strongly to detection accuracy, as confirmed by ablation analysis. These findings highlight the value of multimodal integration in capturing complex psychological signals and support the development of intelligent, non-invasive screening tools for use in digital mental health applications.</dc:description>
	<dc:publisher xml:lang="en-US">IndoCEISS in colaboration with Universitas Gadjah Mada, Indonesia.</dc:publisher>
	<dc:contributor xml:lang="en-US"></dc:contributor>
	<dc:date>2025-10-31</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en-US"></dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://journal.ugm.ac.id/ijccs/article/view/111407</dc:identifier>
	<dc:identifier>10.22146/ijccs.111407</dc:identifier>
	<dc:source xml:lang="en-US">IJCCS (Indonesian Journal of Computing and Cybernetics Systems); Vol 19, No 4 (2025): October; 383-394</dc:source>
	<dc:source>2460-7258</dc:source>
	<dc:source>1978-1520</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://journal.ugm.ac.id/ijccs/article/view/111407/42781</dc:relation>
	<dc:rights xml:lang="en-US">Copyright (c) 2025 IJCCS (Indonesian Journal of Computing and Cybernetics Systems)</dc:rights>
	<dc:rights xml:lang="en-US">http://creativecommons.org/licenses/by-sa/4.0</dc:rights>
</oai_dc:dc>