COMPARING STATISTICAL AND DEEP LEARNING METHODS FOR INSURANCE CLAIM ESTIMATION: A CASE STUDY OF HIDDEN MARKOV MODEL (HMM) AND CONVOLUTIONAL NEURAL NETWORK - LONG SHORTTERM MEMORY (CNN-LSTM)

Abstract

The insurance industry relies heavily on accurate claim prediction to support risk management, reserve allocation, and financial stability. However, motor vehicle insurance claim data are typically characterized by temporal dependency, highly skewed distributions, and fluctuating claim severity, making accurate prediction a challenging task. While deep learning approaches have recently gained attention for time-series forecasting, their effectiveness on moderate-scale insurance claim datasets remains uncertain. This study aims to compare the predictive performance of the Hidden Markov Model (HMM) and CNN-LSTM in modelling temporal patterns and predicting daily motor vehicle insurance claims. In addition, an Attention-LSTM + XGBoost ensemble model is included as a supplementary deep learning benchmark. This study utilizes historical motor vehicle insurance claim data collected from 2017 to 2021, consisting of 11,679 claim observations. The data preprocessing stage included data cleaning, missing value handling, outlier detection, and claim severity categorization for HMM modelling. The HMM parameters were estimated using the Baum–Welch algorithm, while the deep learning models were trained using sequential claim data. Model performance was evaluated using Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and coefficient of determination (R^2) on the testing dataset to ensure objective comparison. The experimental results show that the HMM model achieved the best predictive performance, outperforming both the CNN-LSTM and Attention-LSTM + XGBoost models. The findings indicate that the probabilistic structure of HMM is more suitable for modelling the temporal risk patterns and fluctuating claim behavior observed in the motor vehicle insurance dataset. Furthermore, the study demonstrates that classical probabilistic models can remain competitive and even outperform more complex deep learning approaches when applied to moderately sized insurance claim datasets with limited hidden complexity.