Melola Discovery

← Melola Discovery

Akses terbuka Jurnal SINTA 1

Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods

Jurnal Online Informatika · Vol. 11 No. 1 (2026) · hlm. 243–255

DOI
10.15575/join.v11i1.1733
ISSN
2527-9165, 2528-1682
Terbit
2026-05-14
Penerbit
Sunan Gunung Djati State Islamic University Bandung
Bahasa
en
Lisensi
creativecommons.org/licenses/by-nd/4.0/

Abstrak

Retrieval-Augmented Generation (RAG) combines generative capabilities of language models with external document retrieval to answer questions grounded in reference texts. However, deploying RAG on low-resource devices like Android smartphones is challenging because SLMs have limited computational capacity and depend heavily on efficient chunking and retrieval. Although interest in on-device processing is growing, research on RAG configurations for SLMs under strict resource constraints especially for domain-specific tasks remains limited. This study therefore investigates which combinations of chunking technique, chunk size, overlap, and retrieval strategy best balance accuracy and speed on low-resource devices. The evaluation uses 148 Indonesian questions sourced from an official Hajj guidebook. The study consists of two phases retrieval and generation. Retrieval is evaluated using BLEU, ROUGE-L, MRR, MAP, and Hit@k, while answer quality is measured with BERTScore. The experiments compare different chunking methods (fixed-size or semantic), chunk sizes (128 or 256 tokens), overlaps (25, 50 and 100 tokens), and retrieval methods (dense, sparse, or hybrid). Results show that sparse retrieval with 256-token chunks and 100-token overlap yields the best answer quality (F1 = 0.726). However, 128-token chunks with the same overlap provide the fastest generation time (69.737 seconds). The main contribution of this study is a systematic evaluation of RAG configurations for fully on-device SLMs using a domain-specific Hajj and Umrah dataset not explored in prior research. The findings provide practical guidance for designing efficient and accurate RAG-based question-answering systems on low-resource devices.

Kata kunci

Jaringan kolaborasi para penulis

21 penulis · 65 tautan kepenulisan bersama · 6 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Politeknik Negeri Bandung
  • Pukyong National University
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Nurjannah SyakraniPoliteknik Negeri Bandung2
Dimas KurniawanPoliteknik Negeri Bandung1
Wili Akbar NugrahaAfiliasi tidak diketahui1
Priyanto HidayatullahPoliteknik Negeri Bandung1
Lukmannul Hakim FirdausAfiliasi tidak diketahui1
Yayang Setia BudiAfiliasi tidak diketahui1
Fardan Al JihadAfiliasi tidak diketahui1
Trisna GelarAfiliasi tidak diketahui1
Djoko Cahyo Utomo LieharyaniAfiliasi tidak diketahui1
Maisevli HarikaPoliteknik Negeri Bandung1
Iwan AwaludinPoliteknik Negeri Bandung1
Yunita Citra DewiPoliteknik Negeri Bandung1
Fachri Dhia FauzanPoliteknik Negeri Bandung1
Bima Putra SudimulyaPoliteknik Negeri Bandung1
Vandha Pradiyasma WidartaPukyong National University1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OpenAlex API (jurnal SINTA, ADR 0048) https://openalex.org/S4210216628, pertama di-harvest 21 Nov 2026, terakhir terlihat 21 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"><dc:title xml:lang="en">Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods</dc:title><dc:creator>Amelia Dewi Agustiani</dc:creator><dc:creator>Salsabila Maharani Putri</dc:creator><dc:creator>Jonner Hutahaean</dc:creator><dc:creator>Muhammad Rizqi Sholahuddin</dc:creator><dc:creator>Muhammad Riza Alifi</dc:creator><dc:creator>Ade Hodijah</dc:creator><dc:subject>retrieval-augmented generation</dc:subject><dc:subject>small language models</dc:subject><dc:subject>on-device processing</dc:subject><dc:subject>chunking</dc:subject><dc:subject>sparse retrieval</dc:subject><dc:subject>dense retrieval</dc:subject><dc:subject>hybrid retrieval</dc:subject><dc:subject>chunk size</dc:subject><dc:subject>resource-constrained devices</dc:subject><dc:subject>question answering</dc:subject><dc:subject>Indonesian language</dc:subject><dc:subject>Android smartphones</dc:subject><dc:subject>generation speed</dc:subject><dc:description xml:lang="en">Retrieval-Augmented Generation (RAG) combines generative capabilities of language models with external document retrieval to answer questions grounded in reference texts. However, deploying RAG on low-resource devices like Android smartphones is challenging because SLMs have limited computational capacity and depend heavily on efficient chunking and retrieval. Although interest in on-device processing is growing, research on RAG configurations for SLMs under strict resource constraints especially for domain-specific tasks remains limited. This study therefore investigates which combinations of chunking technique, chunk size, overlap, and retrieval strategy best balance accuracy and speed on low-resource devices. The evaluation uses 148 Indonesian questions sourced from an official Hajj guidebook. The study consists of two phases retrieval and generation. Retrieval is evaluated using BLEU, ROUGE-L, MRR, MAP, and Hit@k, while answer quality is measured with BERTScore. The experiments compare different chunking methods (fixed-size or semantic), chunk sizes (128 or 256 tokens), overlaps (25, 50 and 100 tokens), and retrieval methods (dense, sparse, or hybrid). Results show that sparse retrieval with 256-token chunks and 100-token overlap yields the best answer quality (F1 = 0.726). However, 128-token chunks with the same overlap provide the fastest generation time (69.737 seconds). The main contribution of this study is a systematic evaluation of RAG configurations for fully on-device SLMs using a domain-specific Hajj and Umrah dataset not explored in prior research. The findings provide practical guidance for designing efficient and accurate RAG-based question-answering systems on low-resource devices.</dc:description><dc:publisher>Sunan Gunung Djati State Islamic University Bandung</dc:publisher><dc:date>2026-05-14</dc:date><dc:type>info:eu-repo/semantics/article</dc:type><dc:format>application/pdf</dc:format><dc:identifier>https://doi.org/10.15575/join.v11i1.1733</dc:identifier><dc:identifier>10.15575/join.v11i1.1733</dc:identifier><dc:source>Jurnal Online Informatika; Vol 11 No 1 (2026); 243-255</dc:source><dc:source>2527-9165</dc:source><dc:source>2528-1682</dc:source><dc:language>en</dc:language><dc:relation>https://join.if.uinsgd.ac.id/index.php/join/article/download/1733/399</dc:relation><dc:rights>https://creativecommons.org/licenses/by-nd/4.0/</dc:rights></oai_dc:dc>