OPTIMIZATION OF LIE DETECTION WITH DEEP LEARNING APPROACH USING FUSION METHOD

Abstract

In lie detection, early fusion methods that combine information from multiple modalities, such as images and sounds, are used. To improve performance, a lie detection system is designed using mean fusion techniques. The feature extraction method, which uses Optical Flow (OF) and GaussianBlur, uses image data as input. This process generates facial feature change data as numeric values, enabling more efficient processing and allowing the model to be trained quickly and effectively. Evaluation of the model with accuracy, precision, recall, and F1 score using 10 (Fold) cross-validation using a Convolutional Neural Network (CNN) architecture to find features associated with lying in visual content. At the same time, voice signals are studied through voice signal processing and voice feature extraction methods using Mel Frequency Cepstral Coefficients (MFCC) feature extraction and classification using Mel Frequency Cepstral Coefficients (LSTM). The purpose of this process is to discover lying patterns through the audio module. The mean fusion model combines the decisions of multiple lie detection models for each modality, enabling the system to leverage the strengths of each modality to create a broader feature representation. The dataset used contains images, and voice is used for performance evaluation. This dataset can show various lying situations and contexts. The experimental results show that the fusion method using the mean fusion model achieves a lie detection accuracy of 99% and an F1-Score of 0.99. In the context of lying, this research helps develop a more comprehensive and reliable lie-detection system model. The main contribution of this work is a measurable multimodal fusion strategy that integrates pupil-based facial landmarks and temporal voice features, yielding an accuracy improvement of over 14% compared to unimodal baselines.