Automated Assignment of Community Reports With Early Fusion Multimodal Transformer

Abstract

In the current digital era, city governments require effective and responsive platforms to handle public reports and feedback. One such example is Cepat Respon Masyarakat (CRM) in Jakarta, Indonesia, which allows residents to report various issues to the city government, such as infrastructure damage, traffic accidents, and environmental problems. However, after a report is created, it must be assigned to the appropriate agency. Currently, this assignment process is a challenge, taking an average of nearly two hours. To improve the efficiency and responsiveness of handling public reports through the CRM platform, this research proposes an innovative, multimodal solution for classifying public report data, using both text and images to automatically assign community reports. The proposed method was trained and evaluated using a dataset built from real CRM data. Experiments showed that the multimodal model, using a fusion of the DINOv2 transformer and Multilingual E5 with the Early Fusion method, achieved 80.73% accuracy, an increase from the 68.9% achieved by BERT and ResNet. The results of this research are expected to expedite the issue reporting process and enhance the effectiveness of public services, ultimately contributing to the prosperity of all Indonesian citizens in this era of technological advancement.