Melola Discovery

← Melola Discovery

Akses terbuka Jurnal SINTA 1

Evaluating large language models in Calculus learning: Performance, prompting strategies, and conceptual understanding

Journal on Mathematics Education · Vol. 17 No. 2 (2026) · hlm. 629–654

DOI
10.22342/jme.v17i2.pp629-654
ISSN
2407-0610, 2087-8885
Terbit
2026-05-25
Penerbit
Indonesian Mathematical Society
Lisensi
creativecommons.org/licenses/by/4.0

Abstrak

This study investigates the pedagogical effectiveness of Large Language Models (LLMs) in calculus education through a mixed-methods quasi-experimental design comprising three sequential stages and four interconnected sub-studies examining mathematical problem solving, prompting strategies, and conceptual understanding. Although artificial intelligence (AI)-powered conversational systems are increasingly integrated into STEM education, robust empirical evidence regarding their instructional effectiveness in higher mathematics remains limited. To address this gap, the study benchmarked ChatGPT-4o, Gemini 2.0, and LLaMA 4 using 90 calculus problems across multiple conceptual domains, examined the effects of Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting on model accuracy and explanatory clarity, and conducted a comparative classroom experiment involving 75 undergraduate students to evaluate conceptual understanding of the epsilon–delta definition of limits. The findings demonstrate that prompting strategy is a critical determinant of LLM performance, with CoT prompting consistently producing the greatest reasoning coherence and explanatory clarity, although performance varied across calculus topics. Integral problems were solved most consistently, whereas limit problems posed the greatest challenge. Statistical analyses further showed that increasing mathematical complexity frequently reduced explanation clarity despite correct numerical solutions. Students receiving ChatGPT-assisted instruction demonstrated significantly different conceptual understanding outcomes than those receiving conventional lecture-based instruction (χ² = 6.726, df = 2, p = 0.035, Cramér’s V = 0.299, 95% CI [0.111, 0.505]), with fewer students remaining at minimal comprehension and more achieving partial understanding. Moreover, the logistic mixed-effects model indicated that Few-Shot prompting significantly reduced explanation clarity relative to CoT prompting (β = −0.222, 95% CI [−0.312, −0.132]). Interpreted through Cognitive Load Theory, these findings suggest that CoT prompting reduces extraneous cognitive load by making intermediate reasoning explicit. Overall, this study provides empirical evidence supporting the responsible integration of generative AI into undergraduate calculus instruction by emphasizing prompt literacy and reaffirming the indispensable role of human guidance in fostering conceptual understanding, proof-oriented reasoning, and higher-order mathematical thinking.

Jaringan kolaborasi para penulis

16 penulis · 48 tautan kepenulisan bersama

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Telkom University
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Z. K. A. BaizalTelkom University1
Aniq A. RahmawatiAfiliasi tidak diketahui1
Mizanul KiromTelkom University1
Zidni MubarokTelkom University1
Prasti Eko YunantoTelkom University1
Untari Novia WisestyTelkom University1
Febryanti SthevanieTelkom University1
Gamma KosalaTelkom University1
Arfive GandhiTelkom University1
Mahmud Dwi SulistiyoTelkom University1
Donni RichasdyTelkom University1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Data ini berasal dari Crossref (yang disetorkan penerbit), karena endpoint OAI-PMH milik jurnal tidak dapat di-harvest.

Sumber metadata: Crossref API (cadangan, berdasarkan ISSN) https://api.crossref.org/journals/2407-0610/works, pertama di-harvest 28 Okt 2026, terakhir terlihat 15 Nov 2026.
Metadata sebagai oai_dc, dikonversi dari Crossref
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"><dc:title>Evaluating large language models in Calculus learning: Performance, prompting strategies, and conceptual understanding</dc:title><dc:creator>Ihsan, Aditya Firman</dc:creator><dc:creator>Lhaksmana, Kemas Muslim</dc:creator><dc:creator>Saepudin, Deni</dc:creator><dc:creator>Zulkarnaen, Argi Putra</dc:creator><dc:creator>Putra, Cakra Budiman</dc:creator><dc:description>&lt;jats:p&gt;This study investigates the pedagogical effectiveness of Large Language Models (LLMs) in calculus education through a mixed-methods quasi-experimental design comprising three sequential stages and four interconnected sub-studies examining mathematical problem solving, prompting strategies, and conceptual understanding. Although artificial intelligence (AI)-powered conversational systems are increasingly integrated into STEM education, robust empirical evidence regarding their instructional effectiveness in higher mathematics remains limited. To address this gap, the study benchmarked ChatGPT-4o, Gemini 2.0, and LLaMA 4 using 90 calculus problems across multiple conceptual domains, examined the effects of Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting on model accuracy and explanatory clarity, and conducted a comparative classroom experiment involving 75 undergraduate students to evaluate conceptual understanding of the epsilon–delta definition of limits. The findings demonstrate that prompting strategy is a critical determinant of LLM performance, with CoT prompting consistently producing the greatest reasoning coherence and explanatory clarity, although performance varied across calculus topics. Integral problems were solved most consistently, whereas limit problems posed the greatest challenge. Statistical analyses further showed that increasing mathematical complexity frequently reduced explanation clarity despite correct numerical solutions. Students receiving ChatGPT-assisted instruction demonstrated significantly different conceptual understanding outcomes than those receiving conventional lecture-based instruction (χ² = 6.726, df = 2, p = 0.035, Cramér’s V = 0.299, 95% CI [0.111, 0.505]), with fewer students remaining at minimal comprehension and more achieving partial understanding. Moreover, the logistic mixed-effects model indicated that Few-Shot prompting significantly reduced explanation clarity relative to CoT prompting (β = −0.222, 95% CI [−0.312, −0.132]). Interpreted through Cognitive Load Theory, these findings suggest that CoT prompting reduces extraneous cognitive load by making intermediate reasoning explicit. Overall, this study provides empirical evidence supporting the responsible integration of generative AI into undergraduate calculus instruction by emphasizing prompt literacy and reaffirming the indispensable role of human guidance in fostering conceptual understanding, proof-oriented reasoning, and higher-order mathematical thinking.&lt;/jats:p&gt;</dc:description><dc:publisher>Indonesian Mathematical Society</dc:publisher><dc:date>2026-05-25</dc:date><dc:type>info:eu-repo/semantics/article</dc:type><dc:format>application/pdf</dc:format><dc:identifier>https://doi.org/10.22342/jme.v17i2.pp629-654</dc:identifier><dc:identifier>10.22342/jme.v17i2.pp629-654</dc:identifier><dc:source>Journal on Mathematics Education; Vol 17 No 2 (2026); 629-654</dc:source><dc:source>2407-0610</dc:source><dc:source>2087-8885</dc:source><dc:relation>https://jme.ejournal.unsri.ac.id/index.php/jme/article/download/4056/421</dc:relation><dc:rights>https://creativecommons.org/licenses/by/4.0</dc:rights></oai_dc:dc>