Melola Discovery

← Melola Discovery

Akses terbuka Jurnal SINTA 1

Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers

Bulletin of Electrical Engineering and Informatics · Vol. 12 No. 1 (2022) · hlm. 355–364

DOI
10.11591/eei.v12i1.4231
ISSN
2089-3191, 2302-9285
Terbit
2022-11-07
Penerbit
Institute of Advanced Engineering and Science (IAES)
Bahasa
en
Lisensi
creativecommons.org/licenses/by-sa/4.0/

Abstrak

Music carries emotional information and allows the listener to feel the emotions contained in the music. This study proposes a multimodal music emotion recognition (MER) system using Indonesian song and lyrics data. In the proposed multimodal system, the audio data will use the mel spectrogram feature, and the lyrics feature will be extracted by going through the tokenizing process from XLNet. Convolutional long short term memory network (CNN-LSTM) performs the audio classification task, while XLNet transformers performs the lyrics classification task. The outputs of the two classification tasks are probability weight and actual prediction with the value of positive, neutral, and negative emotions, which are then combined using the stacking ensemble method. The combined output will be trained into an artificial neural network (ANN) model to get the best probability weight output. The multimodal system achieves the best performance with an accuracy of 80.56%. The results showed that the multimodal method of recognizing musical emotions gave better performance than the single modal method. In addition, hyperparameter tuning can affect the performance of multimodal systems.

Kata kunci

Jaringan kolaborasi para penulis

31 penulis · 50 tautan kepenulisan bersama · 12 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Binus University
  • Universitas Muhammadiyah Kalimantan Timur
  • Syarif Hidayatullah State Islamic University Jakarta
  • Andalas University
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Muhammad Farhan FadhilBinus University2
Maimunah MaskurBinus University1
Hendra KurniawanBinus University1
Suma'inna Suma'innaAfiliasi tidak diketahui1
Muhammad ManaqibSyarif Hidayatullah State Islamic University Jakarta1
Bhakti Prabandyo WicaksonoBinus University1
Hendra SaputraAndalas University1
Faldi FaldiUniversitas Muhammadiyah Kalimantan Timur1
Ferry Fadzlul RahmanUniversitas Muhammadiyah Kalimantan Timur1
Sayekti Harits SuryawanUniversitas Muhammadiyah Kalimantan Timur1
Wawan Joko PranotoUniversitas Muhammadiyah Kalimantan Timur1
Fathur RahmanUniversitas Muhammadiyah Kalimantan Timur1
Satrio SadewoBinus University1
Yasinta Roesmiatun PurnamadewiAfiliasi tidak diketahui1
Ahmad Iqbal AbdurrahmanBinus University1
Panji WijonarkoBinus University1
Yoga TanokoBinus University1
Jonathan Christian SetyonoBinus University1
Gede Putra KusumaBinus University1
Francisco Calvin Arnel FeranoBinus University1
Khasyi Al MukarramBinus University1
M. Anang MukhlasBinus University1
Stefanus MichaelBinus University1
Ebenhaiser Jonathan CaprisianoBinus University1
Muhammad Hafizh RamadhansyahBinus University1
Juan HebertBinus University1
Michael J. HenryBinus University1
Wiryanata ChandraBinus University1
Mhd TaufikBinus University1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OpenAlex API (jurnal SINTA, ADR 0048) https://openalex.org/S2765016208, pertama di-harvest 19 Nov 2026, terakhir terlihat 19 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"><dc:title xml:lang="en">Multimodal music emotion recognition in Indonesian songs based on CNN-LSTM, XLNet transformers</dc:title><dc:creator>Andrew Steven Sams</dc:creator><dc:creator>Amalia Zahra</dc:creator><dc:subject>music emotion recognition</dc:subject><dc:subject>Indonesian songs</dc:subject><dc:subject>lyrics</dc:subject><dc:subject>multimodal fusion</dc:subject><dc:subject>CNN-LSTM</dc:subject><dc:subject>XLNet</dc:subject><dc:subject>Mel spectrogram</dc:subject><dc:subject>stacking ensemble</dc:subject><dc:subject>audio classification</dc:subject><dc:subject>text classification</dc:subject><dc:subject>hyperparameter optimization</dc:subject><dc:description xml:lang="en">Music carries emotional information and allows the listener to feel the emotions contained in the music. This study proposes a multimodal music emotion recognition (MER) system using Indonesian song and lyrics data. In the proposed multimodal system, the audio data will use the mel spectrogram feature, and the lyrics feature will be extracted by going through the tokenizing process from XLNet. Convolutional long short term memory network (CNN-LSTM) performs the audio classification task, while XLNet transformers performs the lyrics classification task. The outputs of the two classification tasks are probability weight and actual prediction with the value of positive, neutral, and negative emotions, which are then combined using the stacking ensemble method. The combined output will be trained into an artificial neural network (ANN) model to get the best probability weight output. The multimodal system achieves the best performance with an accuracy of 80.56%. The results showed that the multimodal method of recognizing musical emotions gave better performance than the single modal method. In addition, hyperparameter tuning can affect the performance of multimodal systems.</dc:description><dc:publisher>Institute of Advanced Engineering and Science (IAES)</dc:publisher><dc:date>2022-11-07</dc:date><dc:type>info:eu-repo/semantics/article</dc:type><dc:format>application/pdf</dc:format><dc:identifier>https://doi.org/10.11591/eei.v12i1.4231</dc:identifier><dc:identifier>10.11591/eei.v12i1.4231</dc:identifier><dc:source>Bulletin of Electrical Engineering and Informatics; Vol 12 No 1 (2022); 355-364</dc:source><dc:source>2089-3191</dc:source><dc:source>2302-9285</dc:source><dc:language>en</dc:language><dc:relation>https://beei.org/index.php/EEI/article/download/4231/3083</dc:relation><dc:rights>https://creativecommons.org/licenses/by-sa/4.0/</dc:rights></oai_dc:dc>