Melola Discovery

← Melola Discovery

Jurnal SINTA 1

On the Audio-Visual Emotion Recognition using Convolutional Neural Networks and Extreme Learning Machine

Indonesian Journal of Electical Engineering and Informatics (IJEEI) · Vol. 10 No. 3 (2022) · hlm. 684–697

DOI
10.52549/ijeei.v10i3.3879
ISSN
2089-3272, 2089-3272
Terbit
2022-09-25
Penerbit
IAES Indonesian Section
Bahasa
en

Abstrak

The advances in artificial intelligence and machine learning concerning emotion recognition have been enormous and in previously inconceivable ways. Inspired by the promising evolution in human-computer interaction, this paper is based on developing a multimodal emotion recognition system. This research encompasses two modalities as input, namely speech and video. In the proposed model, the input video samples are subjected to image pre-processing and image frames are obtained. The signal is pre-processed and transformed into the frequency domain for the audio input. The aim is to obtain Mel-spectrogram, which is processed further as images. Convolutional neural networks are used for training and feature extraction for both audio and video with different configurations. The fusion of outputs from two CNNs is done using two extreme learning machines. For classification, the proposed system incorporates a support vector machine. The model is evaluated using three databases, namely eNTERFACE, RML, and SAVEE. For the eNTERFACE dataset, the accuracy obtained without and with augmentation was 87.2% and 94.91%, respectively. The RML dataset yielded an accuracy of 98.5%, and for the SAVEE dataset, the accuracy reached 97.77%. Results achieved from this research are an illustration of the fruitful exploration and effectiveness of the proposed system.

Kata kunci

Jaringan kolaborasi para penulis

40 penulis · 106 tautan kepenulisan bersama · 30 lintas organisasi · menampilkan 40 yang paling banyak terhubung

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • International Islamic University Malaysia
  • Yogyakarta State University
  • Universitas Gadjah Mada
  • University of Kuala Lumpur
  • University of Indonesia
  • International Centre for Education in Islamic Finance
  • University of Sousse
  • Organisasi lain
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Aris NasuhaYogyakarta State University5
Anggun WinursitoYogyakarta State University5
Siti Hajar YusoffInternational Islamic University Malaysia5
Hasmah MansorInternational Islamic University Malaysia4
Malik Arman MorshidiInternational Islamic University Malaysia4
Ardy Seto PriambodoYogyakarta State University4
Huda Adibah Mohd RamliInternational Islamic University Malaysia3
Rashidah Funke OlanrewajuInternational Islamic University Malaysia3
Suriza Ahmad ZabidiInternational Islamic University Malaysia3
Md. Rafiqul IslamInternational Islamic University Malaysia2
Noreha Abdul MalikInternational Islamic University Malaysia2
Elsheikh Mohamed Ahmed ElsheikhInternational Islamic University Malaysia2
Mohammed H. A. ArafaInternational Islamic University Malaysia2
Kazi Istiaque AhmedInternational Islamic University Malaysia2
Diyah PutrianiUniversitas Gadjah Mada2
Gairuzazmi Mat GhaniInternational Islamic University Malaysia2
Amart SulongInternational Islamic University Malaysia2
Othman Omran KhalifaInternational Islamic University Malaysia2
Hassan DaoUniversity of Kuala Lumpur2
Abdul MutholibInternational Islamic University Malaysia2
Intan Rahmithul Husna YaldiInternational Islamic University Malaysia2
Yasmin Mahira Saiful MunirInternational Islamic University Malaysia2
Tika ArundinaUniversity of Indonesia2
Mohd Azmi OmarInternational Centre for Education in Islamic Finance2
Nur Atikah Muhamat SalehInternational Islamic University Malaysia2
Israth Jahan ChowdhuryInternational Islamic University Malaysia1
Nur Aishah ZainalInternational Islamic University Malaysia1
W. IdrisInternational Islamic University Malaysia1
Siti Noorjannah IbrahimInternational Islamic University Malaysia1
Islam Md. RafiqulInternational Islamic University Malaysia1
Ali Kadhim LwasInternational Islamic University Malaysia1
Md Moktarul AlamInternational Islamic University Malaysia1
Jalel ChebilUniversity of Sousse1
Jit Singh MandeepNational University of Malaysia1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) http://section.iaesonline.com/index.php/IJEEI/oai, pertama di-harvest 8 Nov 2026, terakhir terlihat 8 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">On the Audio-Visual Emotion Recognition using Convolutional Neural Networks and Extreme Learning Machine</dc:title>
	<dc:creator xml:lang="en">Ashraf, Arselan</dc:creator>
	<dc:creator xml:lang="en">Gunawan, Teddy Surya</dc:creator>
	<dc:creator xml:lang="en">Arifin, Fatchul</dc:creator>
	<dc:creator xml:lang="en">Kartiwi, Mira</dc:creator>
	<dc:creator xml:lang="en">Sophian, Ali</dc:creator>
	<dc:creator xml:lang="en">Habaebi, Mohamed Hadi</dc:creator>
	<dc:subject xml:lang="en">artificial intelligence</dc:subject>
	<dc:subject xml:lang="en">convolutional neural networks</dc:subject>
	<dc:subject xml:lang="en">emotion recognition</dc:subject>
	<dc:subject xml:lang="en">human-computer interaction</dc:subject>
	<dc:subject xml:lang="en">machine learning</dc:subject>
	<dc:description xml:lang="en">The advances in artificial intelligence and machine learning concerning emotion recognition have been enormous and in previously inconceivable ways. Inspired by the promising evolution in human-computer interaction, this paper is based on developing a multimodal emotion recognition system. This research encompasses two modalities as input, namely speech and video. In the proposed model, the input video samples are subjected to image pre-processing and image frames are obtained. The signal is pre-processed and transformed into the frequency domain for the audio input. The aim is to obtain Mel-spectrogram, which is processed further as images. Convolutional neural networks are used for training and feature extraction for both audio and video with different configurations. The fusion of outputs from two CNNs is done using two extreme learning machines. For classification, the proposed system incorporates a support vector machine. The model is evaluated using three databases, namely eNTERFACE, RML, and SAVEE. For the eNTERFACE dataset, the accuracy obtained without and with augmentation was 87.2% and 94.91%, respectively. The RML dataset yielded an accuracy of 98.5%, and for the SAVEE dataset, the accuracy reached 97.77%. Results achieved from this research are an illustration of the fruitful exploration and effectiveness of the proposed system.</dc:description>
	<dc:publisher xml:lang="en">IAES Indonesian Section</dc:publisher>
	<dc:date>2022-09-25</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://section.iaesonline.com/index.php/IJEEI/article/view/3879</dc:identifier>
	<dc:identifier>10.52549/ijeei.v10i3.3879</dc:identifier>
	<dc:source xml:lang="en">Indonesian Journal of Electrical Engineering and Informatics (IJEEI); Vol. 10 No. 3: September 2022; 684-697</dc:source>
	<dc:source>2089-3272</dc:source>
	<dc:source>2089-3272</dc:source>
	<dc:language>en</dc:language>
	<dc:relation>https://section.iaesonline.com/index.php/IJEEI/article/view/3879/771</dc:relation>
</oai_dc:dc>