Melola Discovery

← Melola Discovery

Jurnal SINTA 1

Advanced Multimodal Emotion Recognition for Javanese Language Using Deep Learning

Indonesian Journal of Electical Engineering and Informatics (IJEEI) · Vol. 12 No. 3 (2024) · hlm. 503–515

DOI
10.52549/ijeei.v12i3.5662
ISSN
2089-3272, 2089-3272
Terbit
2024-08-15
Penerbit
IAES Indonesian Section
Bahasa
en

Abstrak

This research develops a robust emotion recognition system for the Javanese language using multimodal audio and video datasets, addressing the limited advancements in emotion recognition specific to this language. Three models were explored to enhance emotional feature extraction: the SpectrogramImage Model (Model 1), which converts audio inputs into spectrogram images and integrates them with facial images for emotion labeling; the Convolutional-MFCC Model (Model 2), which leverages convolutional techniques for image processing and Mel-frequency cepstral coefficients for audio; and the Multimodal Feature-Extraction Model (Model 3), which independently processes video and audio features before integrating them for emotion recognition. Comparative analysis shows that the Multimodal Feature-Extraction Model achieves the highest accuracy of 93%, surpassing the Convolutional-MFCC Model at 85% and the Spectrogram-Image Model at 71%. These findings demonstrate that effective multimodal integration, mainly through separate feature extraction, significantly enhances emotion recognition accuracy. This research improves communication systems and offers deeper insights into Javanese emotional expressions, with potential applications in human-computer interaction, healthcare, and cultural studies. Additionally, it contributes to the advancement of sophisticated emotion recognition technologies.

Kata kunci

Jaringan kolaborasi para penulis

40 penulis · 112 tautan kepenulisan bersama · 36 lintas organisasi · menampilkan 40 yang paling banyak terhubung

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • International Islamic University Malaysia
  • Yogyakarta State University
  • University of Kuala Lumpur
  • University of Indonesia
  • International Centre for Education in Islamic Finance
  • Politeknik Negeri Cilacap
  • Lac Hong University
  • Organisasi lain
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Mira KartiwiInternational Islamic University Malaysia17
Ali SophianInternational Islamic University Malaysia6
Arselan AshrafInternational Islamic University Malaysia6
Mohamed Hadi HabaebiInternational Islamic University Malaysia5
Muslikhin MuslikhinYogyakarta State University4
Siti Hajar YusoffInternational Islamic University Malaysia3
Rashidah Funke OlanrewajuInternational Islamic University Malaysia3
Malik Arman MorshidiInternational Islamic University Malaysia3
Suprapto SupraptoYogyakarta State University3
Hasmah MansorInternational Islamic University Malaysia2
Noreha Abdul MalikInternational Islamic University Malaysia2
Suriza Ahmad ZabidiInternational Islamic University Malaysia2
Mir Suhail AlamInternational Islamic University Malaysia2
Faisal Ahmed AssaigInternational Islamic University Malaysia2
Amart SulongInternational Islamic University Malaysia1
Othman Omran KhalifaInternational Islamic University Malaysia1
Hassan DaoUniversity of Kuala Lumpur1
Abdul MutholibInternational Islamic University Malaysia1
Intan Rahmithul Husna YaldiInternational Islamic University Malaysia1
Yasmin Mahira Saiful MunirInternational Islamic University Malaysia1
Tika ArundinaUniversity of Indonesia1
Mohd Azmi OmarInternational Centre for Education in Islamic Finance1
Nur Atikah Muhamat SalehInternational Islamic University Malaysia1
W. IdrisInternational Islamic University Malaysia1
Siti Noorjannah IbrahimInternational Islamic University Malaysia1
Didik HariyantoYogyakarta State University1
Arif Ainur RafiqPoliteknik Negeri Cilacap1
Nguyen Vu QuynhLac Hong University1
Herjuna ArtantoYogyakarta State University1
Farah Diyana Abdul RahmanInternational Islamic University Malaysia1
Muhammad Hamdan Hasan GaniInternational Islamic University Malaysia1
Muhammad Fahreza AlghifariInternational Islamic University Malaysia1
Amir Akramin ShafieInternational Islamic University Malaysia1
Norfarah Nadia IsmailUniversiti Teknologi MARA1
Ali Aryo BawonoTechnical University of Munich1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) http://section.iaesonline.com/index.php/IJEEI/oai, pertama di-harvest 8 Nov 2026, terakhir terlihat 8 Nov 2026.
Metadata asli (oai_dc) 
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">Advanced Multimodal Emotion Recognition for Javanese Language Using Deep Learning</dc:title>
	<dc:creator xml:lang="en">Arifin, Fatchul</dc:creator>
	<dc:creator xml:lang="en">Nasuha, Aris</dc:creator>
	<dc:creator xml:lang="en">Priambodo, Ardy Seto</dc:creator>
	<dc:creator xml:lang="en">Winursito, Anggun</dc:creator>
	<dc:creator xml:lang="en">Gunawan, Teddy Surya</dc:creator>
	<dc:subject xml:lang="en">Javanese emotion recognition</dc:subject>
	<dc:subject xml:lang="en">multimodal deep learning</dc:subject>
	<dc:subject xml:lang="en">audio-visual integration</dc:subject>
	<dc:subject xml:lang="en">emotion detection models</dc:subject>
	<dc:subject xml:lang="en">cultural emotion analysis</dc:subject>
	<dc:subject xml:lang="en">human-computer interaction</dc:subject>
	<dc:description xml:lang="en">This research develops a robust emotion recognition system for the Javanese language using multimodal audio and video datasets, addressing the limited advancements in emotion recognition specific to this language. Three models were explored to enhance emotional feature extraction: the SpectrogramImage Model (Model 1), which converts audio inputs into spectrogram images and integrates them with facial images for emotion labeling; the Convolutional-MFCC Model (Model 2), which leverages convolutional techniques for image processing and Mel-frequency cepstral coefficients for audio; and the Multimodal Feature-Extraction Model (Model 3), which independently processes video and audio features before integrating them for emotion recognition. Comparative analysis shows that the Multimodal Feature-Extraction Model achieves the highest accuracy of 93%, surpassing the Convolutional-MFCC Model at 85% and the Spectrogram-Image Model at 71%. These findings demonstrate that effective multimodal integration, mainly through separate feature extraction, significantly enhances emotion recognition accuracy. This research improves communication systems and offers deeper insights into Javanese emotional expressions, with potential applications in human-computer interaction, healthcare, and cultural studies. Additionally, it contributes to the advancement of sophisticated emotion recognition technologies.</dc:description>
	<dc:publisher xml:lang="en">IAES Indonesian Section</dc:publisher>
	<dc:date>2024-08-15</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://section.iaesonline.com/index.php/IJEEI/article/view/5662</dc:identifier>
	<dc:identifier>10.52549/ijeei.v12i3.5662</dc:identifier>
	<dc:source xml:lang="en">Indonesian Journal of Electrical Engineering and Informatics (IJEEI); Vol. 12 No. 3: September 2024; 503-515</dc:source>
	<dc:source>2089-3272</dc:source>
	<dc:source>2089-3272</dc:source>
	<dc:language>en</dc:language>
	<dc:relation>https://section.iaesonline.com/index.php/IJEEI/article/view/5662/928</dc:relation>
</oai_dc:dc>