Melola Discovery

← Melola Discovery

Jurnal

Assessing Generative AI with Context-Augmented Zero-Shot Prompting for HOTS Question Generation Aligned with Bloom’s Taxonomy

Edu Komputika Journal · Vol. 12 No. 2 (2025) · hlm. 145–156

DOI
10.15294/edukom.v12i2.34437
ISSN
2599-297X, 2252-6811
Terbit
2025-12-31
Penerbit
Universitas Negeri Semarang
Bahasa
eng

Abstrak

This study investigates the use of HOTS-GenAI, a generative AI system employing context-augmented zero-shot prompting, to automatically generate multiple-choice questions aligned with higher-order thinking skills (HOTS) in Bloom’s taxonomy. A dataset of 200 items for vocational high schools was validated by three experts. The ground truth data demonstrated good quality with an inter-rater reliability of 0.75 (Gregory’s Index). System performance across analysis (C4), evaluation (C5), and creation (C6) levels was evaluated using Content Validity Index (CVI), gap analysis, and confusion-matrix-based metrics. The findings revealed that HOTS-GenAI performed relatively well at the analytical level, where 70% of items met the HOTS threshold, supported by higher expert consensus. However, only 10% of items achieved the threshold for evaluation, and none for creation. CVI results indicated moderate validity overall, with stronger agreement for C4 than for C5 or C6. Confusion matrix analysis further confirmed this imbalance: accuracy and F1-scores were highest for analysis items but dropped sharply for evaluation and creation, where recall and precision were near zero. These results suggest that while HOTS-GenAI has potential in generating analytical questions, its capacity to model evaluative and creative tasks remains underdeveloped. Future research should involve larger datasets, refined prompt design, and more operational rubrics to enhance both validity and reliability in AI-generated HOTS assessments.

Kata kunci

Jaringan kolaborasi para penulis

40 penulis · 85 tautan kepenulisan bersama · 11 lintas organisasi · menampilkan 40 yang paling banyak terhubung

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • State University of Semarang
  • STMIK Bumigora Mataram
  • Universitas Bumigora
  • Universitas Airlangga
  • Universiti Sains Malaysia
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Amin RetnoningsihAfiliasi tidak diketahui5
Joko WidodoAfiliasi tidak diketahui4
Indri MurniawatyAfiliasi tidak diketahui3
Ani RusilowatiState University of Semarang3
Ade Novi Nurul IhsaniAfiliasi tidak diketahui3
Sri NgabektiState University of Semarang2
Risanda Alirastra BudiantoroAfiliasi tidak diketahui2
Anik MaghfirohAfiliasi tidak diketahui2
Mia Hafizah TumanggerAfiliasi tidak diketahui2
Yustinus Ulung AnggraitoAfiliasi tidak diketahui2
Widiyanto WidiyantoAfiliasi tidak diketahui2
Kartono KartonoUniversitas Airlangga2
Wiwi IsnaeniState University of Semarang2
Wiyanto WiyantoState University of Semarang2
Siti Harnina BintariAfiliasi tidak diketahui2
Dwi PurwantiAfiliasi tidak diketahui2
Mohammad AkhlasAfiliasi tidak diketahui2
Yustinus Ulung AnggraitoAfiliasi tidak diketahui2
Dwita TrianaAfiliasi tidak diketahui2
Suharto LinuwihmoAfiliasi tidak diketahui2
Abdurahim AbdurahimSTMIK Bumigora Mataram1
Andi Sofyan AnasUniversitas Bumigora1
Siti Agrippina Alodia YusufAfiliasi tidak diketahui1
Muhammad TajuddinUniversitas Bumigora1
Muhammad Tajuddin AnwarSTMIK Bumigora Mataram1
Ahmat AdilUniversitas Bumigora1
Sony ZulfikasariState University of Semarang1
Devi Ajeng EfriliandaState University of Semarang1
RodiyahState University of Semarang1
Kien Tsong ChauUniversiti Sains Malaysia1
Lutfia Nur HadiyantiState University of Semarang1
H. MarinaState University of Semarang1
Dian SapitriState University of Semarang1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) https://journal.unnes.ac.id/journals/edukom/oai, pertama di-harvest 5 Nov 2026, terakhir terlihat 5 Nov 2026.
Metadata asli (oai_dc) 
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">Assessing Generative AI with Context-Augmented Zero-Shot Prompting for HOTS Question Generation Aligned with Bloom’s Taxonomy</dc:title>
	<dc:creator xml:lang="en">Ridlo, Saiful</dc:creator>
	<dc:creator xml:lang="en">Sehabuddin, Ahmad </dc:creator>
	<dc:creator xml:lang="en">Hidayat, Syahroni </dc:creator>
	<dc:creator xml:lang="en">Achmadi, Taofan Ali </dc:creator>
	<dc:creator xml:lang="en">Hasanah, Uswatun</dc:creator>
	<dc:creator xml:lang="en">Afifah, Indah Indi </dc:creator>
	<dc:creator xml:lang="en">Abror, Haikal</dc:creator>
	<dc:subject xml:lang="en">Automatic Question Generation</dc:subject>
	<dc:subject xml:lang="en">Bloom’s Taxonomy</dc:subject>
	<dc:subject xml:lang="en">Context-Augmented Zero-Shot Prompting</dc:subject>
	<dc:subject xml:lang="en">Generative AI Higher Order Thinking Skills (HOTS)</dc:subject>
	<dc:description xml:lang="en">This study investigates the use of HOTS-GenAI, a generative AI system employing context-augmented zero-shot prompting, to automatically generate multiple-choice questions aligned with higher-order thinking skills (HOTS) in Bloom’s taxonomy.  A dataset of 200 items for vocational high schools was validated by three experts. The ground truth data demonstrated good quality with an inter-rater reliability of 0.75 (Gregory’s Index). System performance across analysis (C4), evaluation (C5), and creation (C6) levels was evaluated using Content Validity Index (CVI), gap analysis, and confusion-matrix-based metrics. The findings revealed that HOTS-GenAI performed relatively well at the analytical level, where 70% of items met the HOTS threshold, supported by higher expert consensus. However, only 10% of items achieved the threshold for evaluation, and none for creation. CVI results indicated moderate validity overall, with stronger agreement for C4 than for C5 or C6. Confusion matrix analysis further confirmed this imbalance: accuracy and F1-scores were highest for analysis items but dropped sharply for evaluation and creation, where recall and precision were near zero. These results suggest that while HOTS-GenAI has potential in generating analytical questions, its capacity to model evaluative and creative tasks remains underdeveloped. Future research should involve larger datasets, refined prompt design, and more operational rubrics to enhance both validity and reliability in AI-generated HOTS assessments.</dc:description>
	<dc:publisher xml:lang="en">Universitas Negeri Semarang</dc:publisher>
	<dc:date>2025-12-31</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en">Peer-reviewed Article</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://journal.unnes.ac.id/journals/edukom/article/view/34437</dc:identifier>
	<dc:identifier>10.15294/edukom.v12i2.34437</dc:identifier>
	<dc:source xml:lang="en">Edu Komputika Journal; Vol. 12 No. 2 (2025): Edu Komputika Journal; 145-156</dc:source>
	<dc:source>2599-297X</dc:source>
	<dc:source>2252-6811</dc:source>
	<dc:source>10.15294/edukom.v12i2</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://journal.unnes.ac.id/journals/edukom/article/view/34437/9292</dc:relation>
	<dc:rights xml:lang="en">Copyright (c) 2026 Edu Komputika Journal</dc:rights>
</oai_dc:dc>