Melola Discovery

← Melola Discovery

Jurnal SINTA 1

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Indonesian Journal of Electical Engineering and Informatics (IJEEI) · Vol. 10 No. 2 (2022) · hlm. 385–397

DOI
10.52549/ijeei.v10i2.3683
ISSN
2089-3272, 2089-3272
Terbit
2022-05-30
Penerbit
IAES Indonesian Section
Bahasa
en

Abstrak

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages. This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems.

Kata kunci

Jaringan kolaborasi para penulis

2 penulis · 1 tautan kepenulisan bersama

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Central University of Kerala
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) http://section.iaesonline.com/index.php/IJEEI/oai, pertama di-harvest 8 Nov 2026, terakhir terlihat 8 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges</dc:title>
	<dc:creator xml:lang="en">Bhaskar, Shabina</dc:creator>
	<dc:creator xml:lang="en">Madathil, Thasleema Thazhath</dc:creator>
	<dc:subject xml:lang="en">Automatic speech recognition</dc:subject>
	<dc:subject xml:lang="en">Sign language</dc:subject>
	<dc:subject xml:lang="en">Audio-visual systems</dc:subject>
	<dc:subject xml:lang="en">Speech recognition</dc:subject>
	<dc:subject xml:lang="en">Neural networks.</dc:subject>
	<dc:description xml:lang="en">Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures- facial, lip- hand position, etc., are mainly used sensory modalities for the development of hearing-impaired multimodal systems. This paper encapsulates an overview of multimodal systems available within literature towards hearing impaired studies. This paper also discusses some of the studies related to hearing-impaired acoustic analysis. It is observed that very less algorithms have been developed for hearing impaired AVSR as compared to normal hearing. Thus, the study of audio-visual based speech recognition systems for the hearing impaired is highly demanded for the people who are trying to communicate with natively speaking languages.  This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers for the development of AVSR systems.</dc:description>
	<dc:publisher xml:lang="en">IAES Indonesian Section</dc:publisher>
	<dc:date>2022-05-30</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://section.iaesonline.com/index.php/IJEEI/article/view/3683</dc:identifier>
	<dc:identifier>10.52549/ijeei.v10i2.3683</dc:identifier>
	<dc:source xml:lang="en">Indonesian Journal of Electrical Engineering and Informatics (IJEEI); Vol. 10 No. 2: June 2022; 385-397</dc:source>
	<dc:source>2089-3272</dc:source>
	<dc:source>2089-3272</dc:source>
	<dc:language>en</dc:language>
	<dc:relation>https://section.iaesonline.com/index.php/IJEEI/article/view/3683/698</dc:relation>
</oai_dc:dc>