Melola Discovery

← Melola Discovery

Jurnal

Can We Trust AI to Assess Writing? An Analysis of Scoring Reliability and Feedback Consistency

Jurnal Profesi Keguruan · Vol. 11 No. 1 (2025) · hlm. 66–79

DOI
10.15294/jpk.v11i1.25849
ISSN
2528-7214, 2460-4399
Terbit
2025-06-30
Penerbit
Universitas Negeri Semarang
Bahasa
eng

Abstrak

This study analyzes AI-generated writing assessments' scoring reliability and feedback consistency using ChatGPT. Adopting a mixed-methods approach, 23 student descriptive texts were evaluated across three assessment rounds. Quantitative findings showed high scoring reliability, with an Intraclass Correlation Coefficient (ICC) of 0.93, indicating excellent consistency across repeated evaluations. Qualitative analysis revealed that ChatGPT consistently addressed five core writing criteria—content, organization, vocabulary, language use, and mechanics. However, the feedback varied in focus and detail across rounds, and the absence of reference to prior feedback limited its support for revision as a recursive process. The findings suggest that although ChatGPT demonstrates reliable scoring and generally stable feedback themes, it lacks the continuity to facilitate sustained writing development. To enhance its pedagogical value, AI-based feedback systems should be designed to build upon previous responses, thereby enabling more effective support for students' progressive improvement in writing.

Kata kunci

Jaringan kolaborasi para penulis

17 penulis · 30 tautan kepenulisan bersama · 24 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Universitas Halu Oleo
  • Institut Agama Islam Negeri Bone
  • Universitas Islam Negeri Alauddin Makassar
  • Sunan Gunung Djati State Islamic University Bandung
  • Institut Agama Islam Negeri Langsa
  • Institut Agama Islam Negeri Kendari
  • Universitas Islam Negeri Raden Fatah Palembang
  • Organisasi lain
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Esya HeryanaUniversitas Islam Negeri Alauddin Makassar2
Raihan RaihanInstitut Agama Islam Negeri Langsa2
Samrin SamrinInstitut Agama Islam Negeri Kendari1
Yoo Eka Yana KansilUniversitas Halu Oleo1
Karsadi KarsadiUniversitas Halu Oleo1
Hamuni HamuniUniversitas Halu Oleo1
Muhamad Safiuddin SarananiUniversitas Halu Oleo1
Muhamad UyunUniversitas Islam Negeri Raden Fatah Palembang1
Yuli BahriahAfiliasi tidak diketahui1
Sri WinarniUniversitas Jambi1
Ade IsmayaniUniversitas Islam Negeri Ar-Raniry1
Ruslan SangajiInstitut Agama Islam Negeri Bone1
Winona LutfiahInstitut Agama Islam Negeri Bone1
Muhsin MahfudzUniversitas Islam Negeri Alauddin Makassar1
Winona LuthfiahInstitut Agama Islam Negeri Bone1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) https://journal.unnes.ac.id/journals/jpk/oai, pertama di-harvest 6 Nov 2026, terakhir terlihat 8 Nov 2026.
Metadata asli (oai_dc) 
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">Can We Trust AI to Assess Writing? An Analysis of Scoring Reliability and Feedback Consistency</dc:title>
	<dc:creator xml:lang="en">Fitriani, Fitriani</dc:creator>
	<dc:creator xml:lang="en">Eko Rini, Puput Zuli</dc:creator>
	<dc:subject xml:lang="en">AI-generated writing assessment; scoring reliability; feedback consistency   </dc:subject>
	<dc:description xml:lang="en">This study analyzes AI-generated writing assessments' scoring reliability and feedback consistency using ChatGPT. Adopting a mixed-methods approach, 23 student descriptive texts were evaluated across three assessment rounds. Quantitative findings showed high scoring reliability, with an Intraclass Correlation Coefficient (ICC) of 0.93, indicating excellent consistency across repeated evaluations. Qualitative analysis revealed that ChatGPT consistently addressed five core writing criteria—content, organization, vocabulary, language use, and mechanics. However, the feedback varied in focus and detail across rounds, and the absence of reference to prior feedback limited its support for revision as a recursive process. The findings suggest that although ChatGPT demonstrates reliable scoring and generally stable feedback themes, it lacks the continuity to facilitate sustained writing development. To enhance its pedagogical value, AI-based feedback systems should be designed to build upon previous responses, thereby enabling more effective support for students' progressive improvement in writing.</dc:description>
	<dc:publisher xml:lang="en">Universitas Negeri Semarang </dc:publisher>
	<dc:date>2025-06-30</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en">Peer-reviewed Article</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://journal.unnes.ac.id/journals/jpk/article/view/25849</dc:identifier>
	<dc:identifier>10.15294/jpk.v11i1.25849</dc:identifier>
	<dc:source xml:lang="en">Jurnal Profesi Keguruan; Vol. 11 No. 1 (2025); 66-79</dc:source>
	<dc:source>2528-7214</dc:source>
	<dc:source>2460-4399</dc:source>
	<dc:source>10.15294/jpk.v11i1</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://journal.unnes.ac.id/journals/jpk/article/view/25849/4814</dc:relation>
	<dc:rights xml:lang="en">Copyright (c) 2025 Jurnal Profesi Keguruan</dc:rights>
</oai_dc:dc>