A hybrid convolutional neural network-recurrent neural network approach for breast cancer detection through Mask R-CNN and ARI-TFMOA optimization

Abstract

This paper presents a novel hybrid deep learning-based approach for breast cancer detection, addressing critical challenges such as overfitting and performance degradation in varying data conditions. Unlike traditional methods that struggle with detection accuracy, this work integrates a unique combination of advanced segmentation and classification techniques. The segmentation phase leverages Mask region-based convolutional neural network (R-CNN), enhanced by the adaptive random increment-based tomtit flock metaheuristic optimization algorithm (ARI-TFMOA), a novel algorithm inspired by natural flocking behavior. ARI-TFMOA fine-tunes Mask R-CNN parameters, achieving improved feature extraction and segmentation precision while ensuring adaptability to diverse datasets. For classification, a hybrid convolutional neural network-recurrent neural network (CNN-RNN) model is introduced, combining spatial feature extraction by CNNs with temporal pattern recognition by RNNs, resulting in a more nuanced and comprehensive analysis of breast cancer images. The proposed framework achieved significant advancements over existing methods, demonstrating improved performance. This hybrid integration of ARI-TFMOA and Hybrid CNN-RNN models represents a unique contribution, enabling robust, accurate, and efficient breast cancer detection.