Melola Discovery

← Melola Discovery

Akses terbuka Jurnal SINTA 1

Evaluating the consistency of automated CEFR analyzers: a study of English language text classification

International Journal of Evaluation and Research in Education (IJERE) · Vol. 14 No. 4 (2025) · hlm. 3283–3294

DOI
10.11591/ijere.v14i4.33528
ISSN
2620-5440, 2252-8822
Terbit
2025-08-01
Penerbit
Institute of Advanced Engineering and Science
Bahasa
eng
Lisensi
creativecommons.org/licenses/by-nc/4.0

Abstrak

With the increasing use of web-based tools for text analysis, there is a growing reliance on automated systems to assess text difficulty and classify texts to the Common European Framework of Reference for Languages (CEFR). However, inconsistencies in these tools’ outputs could undermine their effectiveness for language learners and researchers. This study investigates the consistency of five widely used automated CEFR analyzer tools, including ChatGPT, by analyzing 20 English descriptive texts at CEFR levels B1 and B2. A quantitative approach was employed to compare the CEFR classifications generated by these tools. The results reveal significant inconsistencies across the tools, raising concerns about the reliability of automated CEFR alignment. Additionally, the content and genre of texts appeared to influence the CEFR classification, suggesting that certain factors beyond the tools’ algorithms may affect their accuracy. These findings have important implications for language educators, curriculum designers, and researchers who rely on automated CEFR tools for text selection, grading, and analysis. The study highlights the limitations of automated CEFR classification systems and calls for a more qualitative approach to text difficulty alignment analysis. Future research recommendation is discussed and call for more focus on refining these tools and exploring additional factors that may impact their effectiveness in text difficulty measurement and CEFR alignment.

Kata kunci

Jaringan kolaborasi para penulis

5 penulis · 7 tautan kepenulisan bersama · 1 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Mahidol University
  • Suranaree University of Technology
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Jeffrey Dawala WilangSuranaree University of Technology1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) http://ijere.iaescore.com/index.php/IJERE/oai, pertama di-harvest 8 Nov 2026, terakhir terlihat 16 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en-US">Evaluating the consistency of automated CEFR analyzers:  a study of English language text classification</dc:title>
	<dc:creator>Siripol, Piyathat</dc:creator>
	<dc:creator>Rhee, Seongha</dc:creator>
	<dc:creator>Thirakunkovit, Suthathip</dc:creator>
	<dc:creator>Liang-Itsara, Aphiwit</dc:creator>
	<dc:subject xml:lang="en-US">Analyzer tools; Automated text analysis; CEFR; Text difficulty classification; Web based analysis</dc:subject>
	<dc:description xml:lang="en-US">With the increasing use of web-based tools for text analysis, there is a growing reliance on automated systems to assess text difficulty and classify texts to the Common European Framework of Reference for Languages (CEFR). However, inconsistencies in these tools’ outputs could undermine their effectiveness for language learners and researchers. This study investigates the consistency of five widely used automated CEFR analyzer tools, including ChatGPT, by analyzing 20 English descriptive texts at CEFR levels B1 and B2. A quantitative approach was employed to compare the CEFR classifications generated by these tools. The results reveal significant inconsistencies across the tools, raising concerns about the reliability of automated CEFR alignment. Additionally, the content and genre of texts appeared to influence the CEFR classification, suggesting that certain factors beyond the tools’ algorithms may affect their accuracy. These findings have important implications for language educators, curriculum designers, and researchers who rely on automated CEFR tools for text selection, grading, and analysis. The study highlights the limitations of automated CEFR classification systems and calls for a more qualitative approach to text difficulty alignment analysis. Future research recommendation is discussed and call for more focus on refining these tools and exploring additional factors that may impact their effectiveness in text difficulty measurement and CEFR alignment.</dc:description>
	<dc:publisher xml:lang="en-US">Institute of Advanced Engineering and Science</dc:publisher>
	<dc:contributor xml:lang="en-US"></dc:contributor>
	<dc:date>2025-08-01</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en-US"></dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://ijere.iaescore.com/index.php/IJERE/article/view/33528</dc:identifier>
	<dc:identifier>10.11591/ijere.v14i4.33528</dc:identifier>
	<dc:source xml:lang="en-US">International Journal of Evaluation and Research in Education (IJERE); Vol 14, No 4: August 2025; 3283-3294</dc:source>
	<dc:source>2620-5440</dc:source>
	<dc:source>2252-8822</dc:source>
	<dc:source>10.11591/ijere.v14i4</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://ijere.iaescore.com/index.php/IJERE/article/view/33528/14619</dc:relation>
	<dc:rights xml:lang="en-US">Copyright (c) 2025 Institute of Advanced Engineering and Science</dc:rights>
	<dc:rights xml:lang="en-US">http://creativecommons.org/licenses/by-nc/4.0</dc:rights>
</oai_dc:dc>