Melola Discovery

← Melola Discovery

Jurnal Tanpa peringkat SINTA

Combining Multiple Text Representations for Improved Automatic Evaluation of Indonesian Essay Answers

Scientific Journal of Informatics · Vol. 11 No. 3 (2024) · hlm. 681–692

DOI
10.15294/sji.v11i3.9440
ISSN
2407-7658, 2460-0040
Terbit
2024-08-30
Penerbit
Universitas Negeri Semarang
Bahasa
eng

Abstrak

multiple-choices, regarding students’ learning achievement. When the number of students in a class is huge, however, examinations using essay questions become hard to conduct and take a long evaluation time. Automatic essay evaluation has, therefore, become a potential approach in this situation. Various methods have been proposed, however, optimal solutions for such evaluation in the Indonesian language are less known. Furthermore, with the rapid development of machine learning approaches, in particular deep learning approaches, the investigation of such optimal solutions becomes more necessary. Method: To address the aforementioned issue, this study proposed the investigation of text representation approaches for optimal automatic evaluation of Indonesian essay answers. The investigation compared pre-trained word embedding methods such as Word2vec, GloVe, FastText, and RoBERTa, as well as compared text encoding methods such as long short-term memories (LSTMs) and transformers. LSTMs are able to capture temporal semantics by employing state variables, while transformers are able to capture long-term dependency between parts of their input sequences. Additionally, we investigated classification-based and similarity-based training to build text encoders. We expected that these training approaches allowed encoders to extract different views of information. We compared classification results produced by different text encoders and combinations of text encoders. Result: We evaluated various text representation approaches using the UKARA dataset. Our experiments showed that the FastText word embedding method outperformed the Word2vec, GloVe, and RoBERTa methods. The FastText method achieved an F1-score of 75.43% on validation sets, while the Word2vec, GloVe, and RoBERTa methods achieved F1-scores of 69.56%, 74.53%, and 72.87%, respectively. In addition, the experiments showed that combinations of text encoders outperformed individual encoders. The combination of the LSTM encoder, the transformer encoder, and the TF-IDF encoder obtained an F1-score of 77.22% in the best case, which is better than the best F1-scores of the individual LSTM encoders (75.35%), the best combination of transformer encoders (71.49%), and the individual TF-IDF encoder (76.69%). We observed that LSTM encoders produced better performance when they were built using classification-based training. Meanwhile, the transformer encoders obtained better performance when built using similarity-based training. Novelty: The novelty proposed in this research is the optimal combination of text encoders specifically constructed for the evaluation of essay answers in the Indonesian language. Our experiments showed that the combination of three encoders - namely the LSTM encoder built using classification-based training, the transformer encoder built using classification-based and similarity-based training, and the TF-IDF encoder - obtained the best classification performance.

Kata kunci

Jaringan kolaborasi para penulis

39 penulis · 56 tautan kepenulisan bersama · 6 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Universitas Gadjah Mada
  • Institut Teknologi Del
  • Universitas Teknokrat Indonesia
  • Jenderal Soedirman University
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Sigit PriyantaUniversitas Gadjah Mada2
Retantyo WardoyoUniversitas Gadjah Mada2
Anifudin AzisUniversitas Gadjah Mada1
Praretno WibowoAfiliasi tidak diketahui1
Juwita MaharantiAfiliasi tidak diketahui1
Iqnatius Dimas NugrohoUniversitas Gadjah Mada1
Amelia NursantiUniversitas Gadjah Mada1
Tri Kuntoro PriyambodoUniversitas Gadjah Mada1
Zaied ShouranUniversitas Gadjah Mada1
Nur Azizul HaqimiUniversitas Gadjah Mada1
Friska Ayu ListyaUniversitas Gadjah Mada1
Wahyono WahyonoUniversitas Gadjah Mada1
Riska Amalia PraptiwiUniversitas Gadjah Mada1
Dayan Ramly RamadhanUniversitas Gadjah Mada1
Hendri Kurniawan PrakosaUniversitas Gadjah Mada1
Anindhita NisitasariUniversitas Gadjah Mada1
Raymond LuisUniversitas Gadjah Mada1
Muhammad Zha'farudin Pudya WardanaUniversitas Gadjah Mada1
Muhammad AnsharUniversitas Gadjah Mada1
Raden SumihartoUniversitas Gadjah Mada1
Sarah Rosdiana TambunanInstitut Teknologi Del1
Luthfia Nurma HapsariUniversitas Gadjah Mada1
Hizbullah HamdaUniversitas Gadjah Mada1
Arie KurniawanUniversitas Gadjah Mada1
Sri MulyanaUniversitas Gadjah Mada1
Reka AlamsyahUniversitas Gadjah Mada1
Doni SetyawanUniversitas Gadjah Mada1
E. Elsa Herdiana MurhandarwatiUniversitas Gadjah Mada1
Debby AlitaUniversitas Teknokrat Indonesia1
Muslikhah MuslikhahAfiliasi tidak diketahui1
Elih SudiapermanaAfiliasi tidak diketahui1
Arief Kelik NugrohoJenderal Soedirman University1
Hardyanto SoebonoUniversitas Gadjah Mada1
Dina Fitriana RosyadaUniversitas Gadjah Mada1
Krida Tri WahyuliUniversitas Gadjah Mada1
Muhammad HasaniUniversitas Gadjah Mada1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) https://journal.unnes.ac.id/journals/sji/oai, pertama di-harvest 6 Nov 2026, terakhir terlihat 8 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">Combining Multiple Text Representations for Improved Automatic Evaluation of Indonesian Essay Answers</dc:title>
	<dc:creator xml:lang="en">Wibowo, Moh Edi</dc:creator>
	<dc:creator xml:lang="en">Rokhman, Nur</dc:creator>
	<dc:creator xml:lang="en">Sihabudin, Agus </dc:creator>
	<dc:subject xml:lang="en">Automatic essay evaluation</dc:subject>
	<dc:subject xml:lang="en">LSTM</dc:subject>
	<dc:subject xml:lang="en">Transformer</dc:subject>
	<dc:subject xml:lang="en">Text similarity</dc:subject>
	<dc:subject xml:lang="en">Siamese neural network</dc:subject>
	<dc:description xml:lang="en">multiple-choices, regarding students’ learning achievement. When the number of students in a class is huge, however, examinations using essay questions become hard to conduct and take a long evaluation time. Automatic essay evaluation has, therefore, become a potential approach in this situation. Various methods have been proposed, however, optimal solutions for such evaluation in the Indonesian language are less known. Furthermore, with the rapid development of machine learning approaches, in particular deep learning approaches, the investigation of such optimal solutions becomes more necessary.
Method: To address the aforementioned issue, this study proposed the investigation of text representation approaches for optimal automatic evaluation of Indonesian essay answers. The investigation compared pre-trained word embedding methods such as Word2vec, GloVe, FastText, and RoBERTa, as well as compared text encoding methods such as long short-term memories (LSTMs) and transformers. LSTMs are able to capture temporal semantics by employing state variables, while transformers are able to capture long-term dependency between parts of their input sequences. Additionally, we investigated classification-based and similarity-based training to build text encoders. We expected that these training approaches allowed encoders to extract different views of information. We compared classification results produced by different text encoders and combinations of text encoders.
Result: We evaluated various text representation approaches using the UKARA dataset. Our experiments showed that the FastText word embedding method outperformed the Word2vec, GloVe, and RoBERTa methods. The FastText method achieved an F1-score of 75.43% on validation sets, while the Word2vec, GloVe, and RoBERTa methods achieved F1-scores of 69.56%, 74.53%, and 72.87%, respectively. In addition, the experiments showed that combinations of text encoders outperformed individual encoders. The combination of the LSTM encoder, the transformer encoder, and the TF-IDF encoder obtained an F1-score of 77.22% in the best case, which is better than the best F1-scores of the individual LSTM encoders (75.35%), the best combination of transformer encoders (71.49%), and the individual TF-IDF encoder (76.69%). We observed that LSTM encoders produced better performance when they were built using classification-based training. Meanwhile, the transformer encoders obtained better performance when built using similarity-based training.
Novelty: The novelty proposed in this research is the optimal combination of text encoders specifically constructed for the evaluation of essay answers in the Indonesian language. Our experiments showed that the combination of three encoders - namely the LSTM encoder built using classification-based training, the transformer encoder built using classification-based and similarity-based training, and the TF-IDF encoder - obtained the best classification performance.</dc:description>
	<dc:publisher xml:lang="en">Universitas Negeri Semarang</dc:publisher>
	<dc:date>2024-08-30</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en">Peer-reviewed Article</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://journal.unnes.ac.id/journals/sji/article/view/9440</dc:identifier>
	<dc:identifier>10.15294/sji.v11i3.9440</dc:identifier>
	<dc:source xml:lang="en">Scientific Journal of Informatics; Vol. 11 No. 3: Special Issue; 681-692</dc:source>
	<dc:source>2407-7658</dc:source>
	<dc:source>2460-0040</dc:source>
	<dc:source>10.15294/sji.v11i3</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://journal.unnes.ac.id/journals/sji/article/view/9440/1339</dc:relation>
	<dc:rights xml:lang="en">Copyright (c) 2024 Scientific Journal of Informatics</dc:rights>
</oai_dc:dc>