Zero-Shot Per-Criteria Prompting for Systematic Review Acceleration: A Case Study of Title and Abstract Filtering Process

Abstract

Background: Systematic Literature Review (SLR) is a widely adopted method for obtaining a comprehensive understanding of a specific research area. It involves a structured, transparent, and research question-driven process consisting of three main stages: planning, conducting, and reporting. Although SLR provides objectivity and accuracy, the document selection phase remains labor-intensive and time-consuming due to the need to manually screen large volumes of literature. Objective: This study investigates the potential of large language models (LLMs) to automate the document selection phase in the SLR process by proposing a per-criteria prompting approach that naturally mimics the human annotation process, thereby improving screening efficiency and accuracy. Methods: We propose a per-criteria prompting approach to classify paper metadata using LLM in the zero-shot paradigm. This is done by classifying whether the given paper metadata (title and abstract) is relevant to each inclusion exclusion criteria and then aggregating them. As the baseline, we compare our proposed method to the existing global prompting approach, which directly classifies whether the given paper metadata is relevant to all inclusion exclusion criteria in single steps. Results: The proposed per-criteria prompting approach generally outperforms the global prompting approach. Especially when applied using the Llama model, our proposed approach outperforms the existing global prompting approach in all 10 dataset benchmarks we use. Conclusion: The finding suggests that prompting each criterion significantly boosts the LLM performance because it makes the LLM more focused in classifying the paper metadata into the given inclusion and exclusion criteria. Keywords: Systematic Literature Review, Text Classification, Large Language Model, Per-Criteria Prompting, Zero-Shot Prompting