Akses terbuka Jurnal SINTA 1
Evaluating RAG Performance on Small Language Models for Low-Resource Devices through Chunking and Retrieval Methods
Jurnal Online Informatika · Vol. 11 No. 1 (2026) · hlm. 243–255
- ISSN
- 2527-9165, 2528-1682
- Terbit
- 2026-05-14
- Penerbit
- Sunan Gunung Djati State Islamic University Bandung
- Bahasa
- en
Abstrak
Retrieval-Augmented Generation (RAG) combines generative capabilities of language models with external document retrieval to answer questions grounded in reference texts. However, deploying RAG on low-resource devices like Android smartphones is challenging because SLMs have limited computational capacity and depend heavily on efficient chunking and retrieval. Although interest in on-device processing is growing, research on RAG configurations for SLMs under strict resource constraints especially for domain-specific tasks remains limited. This study therefore investigates which combinations of chunking technique, chunk size, overlap, and retrieval strategy best balance accuracy and speed on low-resource devices. The evaluation uses 148 Indonesian questions sourced from an official Hajj guidebook. The study consists of two phases retrieval and generation. Retrieval is evaluated using BLEU, ROUGE-L, MRR, MAP, and Hit@k, while answer quality is measured with BERTScore. The experiments compare different chunking methods (fixed-size or semantic), chunk sizes (128 or 256 tokens), overlaps (25, 50 and 100 tokens), and retrieval methods (dense, sparse, or hybrid). Results show that sparse retrieval with 256-token chunks and 100-token overlap yields the best answer quality (F1 = 0.726). However, 128-token chunks with the same overlap provide the fastest generation time (69.737 seconds). The main contribution of this study is a systematic evaluation of RAG configurations for fully on-device SLMs using a domain-specific Hajj and Umrah dataset not explored in prior research. The findings provide practical guidance for designing efficient and accurate RAG-based question-answering systems on low-resource devices.
Kata kunci
Jaringan kolaborasi para penulis
21 penulis · 65 tautan kepenulisan bersama · 6 lintas organisasi
Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.
- Politeknik Negeri Bandung
- Pukyong National University
- Afiliasi tidak diketahui
- Organisasi berbeda
- Organisasi sama
- Afiliasi tidak diketahui
- Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
| Rekan penulis | Organisasi | Artikel bersama |
|---|---|---|
| Nurjannah Syakrani | Politeknik Negeri Bandung | 2 |
| Dimas Kurniawan | Politeknik Negeri Bandung | 1 |
| Wili Akbar Nugraha | Afiliasi tidak diketahui | 1 |
| Priyanto Hidayatullah | Politeknik Negeri Bandung | 1 |
| Lukmannul Hakim Firdaus | Afiliasi tidak diketahui | 1 |
| Yayang Setia Budi | Afiliasi tidak diketahui | 1 |
| Fardan Al Jihad | Afiliasi tidak diketahui | 1 |
| Trisna Gelar | Afiliasi tidak diketahui | 1 |
| Djoko Cahyo Utomo Lieharyani | Afiliasi tidak diketahui | 1 |
| Maisevli Harika | Politeknik Negeri Bandung | 1 |
| Iwan Awaludin | Politeknik Negeri Bandung | 1 |
| Yunita Citra Dewi | Politeknik Negeri Bandung | 1 |
| Fachri Dhia Fauzan | Politeknik Negeri Bandung | 1 |
| Bima Putra Sudimulya | Politeknik Negeri Bandung | 1 |
| Vandha Pradiyasma Widarta | Pukyong National University | 1 |
Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.