Integrating Prompt-Guided Pipeline of YOLOv8 and Segment Anything Model for Cervical Precancerous Lesion Segmentation

Abstract

Background: Segmenting precancerous cervical lesions from cervicography images remains challenging due to irregular lesion morphology, blurred boundaries, low contrast, and visual similarity between abnormal and normal cervical tissue. Automated segmentation methods are needed to support more objective and efficient cervical cancer screening. Objective: This study aimed to develop and evaluate a prompt-guided integration framework that integrates variants of YOLOv8 with the Segment Anything Model (SAM) for the detection and segmentation of precancerous cervical lesions. Methods: A total of 1,350 cervicography images were used, consisting of 300 primary clinical images (TeleOTIVA Dataset) and 1,050 public images from the International Agency for Research on Cancer (IARC). The dataset was divided into 80% training data and 20% validation data. Five variants of YOLOv8—namely YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l, and YOLOv8x—were evaluated for detecting the cervical region, columnar region, and lesion classes. The best YOLOv8 model was then used to generate bounding box prompts for SAM-based segmentation using the ViT-B, ViT-L, and ViT-H encoders. Results: YOLOv8m achieved the best overall detection performance, with an average mAP of 65.7%. In the segmentation stage, ViT-L SAM achieved the best performance. The optimal model configuration depends on the object being segmented. YOLOv8m–ViT-L SAM delivered the best performance in lesion segmentation, with an IoU of 0.591 and a Dice coefficient of 0.705. Statistical analysis shows that YOLOv8m–ViT-L SAM achieved the best average score for lesion segmentation and significantly outperformed the evaluated comparison models. Conclusion: The proposed YOLOv8m-SAM pipeline demonstrates the feasibility of detection-guided segmentation using prompts generated by YOLOv8. However, more detailed lesion segmentation remains challenging and requires further improvement through prompt refinement, domain adaptation, and validation on larger, multicenter datasets. Keywords: Cervicography, Precancerous Cervical Lesions, Segmentation, Segment Anything Model, YOLOv8