Melola Discovery

← Melola Discovery

Akses terbuka Jurnal SINTA 1

EmoVibe: AI-driven multimodal emotion analysis for mental health via social media dashboards

IAES International Journal of Artificial Intelligence · Vol. 14 No. 6 (2025) · hlm. 4565–4578

DOI
10.11591/ijai.v14.i6.pp4565-4578
ISSN
2252-8938, 2089-4872
Terbit
2025-12-01
Penerbit
Institute of Advanced Engineering and Science
Bahasa
eng
Lisensi
creativecommons.org/licenses/by-sa/4.0

Abstrak

Monitoring mental health via social media often utilizes unimodal approaches, such as sentiment analysis on text or single-staged image categorization, or executes early feature fusion. However, in real-world contexts where emotions are conveyed via text, emojis, and images, unimodal approach leads to obscured decision-making pathways and overall diminished performance. To overcome these limitations, we propose EmoVibe, a hybrid multimodal AI framework for emotive analysis. EmoVibe uses attention-based late fusion strategy, where text embeddings are generated from bidirectional encoder representations from transformers (BERT) and visual features are extracted by vision transformer. Subsequently, emoticon vectors linked to avatars are processed independently. Later, these independent data features are integrated at higher levels, enhancing interpretability and performance. In contrast to early fusion methods and integrated multimodal large language models (LLMs) like CLIP, Flamingo, GPT-4V, MentaLLaMA, and domain-adapted models like EmoBERTa, EmoVibe preserves modality-specific contexts without premature fusion. This architecture saves processing cost, allowing for clearer, unambiguous rationalization and explanations. EmoVibe outperforms unimodal baselines and early fusion models, obtaining 89.7% accuracy on GoEmotions, FER, and AffectNet, compared to BERT's 87.4% and ResNet-50's 84.2%, respectively. Furthermore, a customizable, real time, privacy-aware dashboard is created, supporting physicians and end users. This technology enables scalable and proactive intervention options and fosters user self-awareness of mental health.

Kata kunci

Jaringan kolaborasi para penulis

9 penulis · 21 tautan kepenulisan bersama · 4 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Symbiosis International University
  • Bharati Vidyapeeth (Deemed to be University)
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Ketan V. KotechaSymbiosis International University1
Nikita ChaudhariSymbiosis International University1
Payal S. KadamBharati Vidyapeeth (Deemed to be University)1
Vaishali D. KhairnarAfiliasi tidak diketahui1
Shruti PatilSymbiosis International University1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) http://ijai.iaescore.com/index.php/IJAI/oai, pertama di-harvest 8 Nov 2026, terakhir terlihat 16 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en-US">EmoVibe: AI-driven multimodal emotion analysis for mental health via social media dashboards</dc:title>
	<dc:creator>Vora, Deepali</dc:creator>
	<dc:creator>Sharma, Aryan</dc:creator>
	<dc:creator>Garg, Mudit</dc:creator>
	<dc:creator>Fransis, Steve</dc:creator>
	<dc:subject xml:lang="en-US"></dc:subject>
	<dc:subject xml:lang="en-US">Affective state estimation;  Emotion dashboard;   Hybrid transfer learning;  Late-fusion;  Multimodal emotion recognition;  Real-time monitoring</dc:subject>
	<dc:description xml:lang="en-US">Monitoring mental health via social media often utilizes unimodal approaches, such as sentiment analysis on text or single-staged image categorization, or executes early feature fusion. However, in real-world contexts where emotions are conveyed via text, emojis, and images, unimodal approach leads to obscured decision-making pathways and overall diminished performance. To overcome these limitations, we propose EmoVibe, a hybrid multimodal AI framework for emotive analysis. EmoVibe uses attention-based late fusion strategy, where text embeddings are generated from bidirectional encoder representations from transformers (BERT) and visual features are extracted by vision transformer. Subsequently, emoticon vectors linked to avatars are processed independently. Later, these independent data features are integrated at higher levels, enhancing interpretability and performance. In contrast to early fusion methods and integrated multimodal large language models (LLMs) like CLIP, Flamingo, GPT-4V, MentaLLaMA, and domain-adapted models like EmoBERTa, EmoVibe preserves modality-specific contexts without premature fusion. This architecture saves processing cost, allowing for clearer, unambiguous rationalization and explanations. EmoVibe outperforms unimodal baselines and early fusion models, obtaining 89.7% accuracy on GoEmotions, FER, and AffectNet, compared to BERT's 87.4% and ResNet-50's 84.2%, respectively. Furthermore, a customizable, real time, privacy-aware dashboard is created, supporting physicians and end users. This technology enables scalable and proactive intervention options and fosters user self-awareness of mental health.</dc:description>
	<dc:publisher xml:lang="en-US">Institute of Advanced Engineering and Science</dc:publisher>
	<dc:contributor xml:lang="en-US"></dc:contributor>
	<dc:date>2025-12-01</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en-US"></dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://ijai.iaescore.com/index.php/IJAI/article/view/28398</dc:identifier>
	<dc:identifier>10.11591/ijai.v14.i6.pp4565-4578</dc:identifier>
	<dc:source xml:lang="en-US">IAES International Journal of Artificial Intelligence (IJ-AI); Vol 14, No 6: December 2025; 4565-4578</dc:source>
	<dc:source>2252-8938</dc:source>
	<dc:source>2089-4872</dc:source>
	<dc:source>10.11591/ijai.v14.i6</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://ijai.iaescore.com/index.php/IJAI/article/view/28398/14835</dc:relation>
	<dc:rights xml:lang="en-US">Copyright (c) 2025 Deepali Vora, Aryan Sharma, Mudit Garg, Steve Francis</dc:rights>
	<dc:rights xml:lang="en-US">http://creativecommons.org/licenses/by-sa/4.0</dc:rights>
</oai_dc:dc>