A Hybrid Deep Texture Feature Approach Based on GLCM and ResNet with Autoencoder for Histopathological Breast Cancer Classification

Abstract

Background: Breast cancer is one of the largest contributors to cancer cases in the world. Accurate diagnosis and effective treatment can increase the chances of recovery and improve patients’ life expectancy. To address these challenges, various approaches have been developed, including the use of machine learning and deep learning. Despite the promising performance of deep learning approaches, constructing classification models using full deep learning architectures requires substantial computational resources and longer training times compared to traditional machine learning methods. Objective: This study aims to develop both binary and multiclass classification models for breast cancer subtypes by integrating handcrafted texture features with deep features extracted from Residual Network (ResNet). Methods: This study uses the BreakHis dataset, which includes images at 40×, 100×, 200×, and 400× magnifications. The dataset was divided into training and testing sets using an 80:20 ratio. Texture features derived from Gray Level Co-occurrence Matrix (GLCM) were utilized to capture spatial relationships across multiple distances and orientations. In addition, deep features were extracted from augmented images using transfer learning with a pre-trained ResNet50 model. Feature optimization was conducted using an autoencoder to reduce dimensionality and eliminate feature redundancy. Results: Among the four machine learning algorithms evaluated using the GLCM–ResNet features, Support Vector Machine (SVM) and Artificial Neural Network (ANN) consistently outperformed Random Forest and k-Nearest Neighbors (k-NN), confirming their suitability for handling high-dimensional hybrid feature representations. Moreover, the best performance was achieved using SVM at 40× magnification, with an accuracy of 98.43% for binary classification and 94.17% for multiclass classification. Conclusion: A comparative analysis with existing state-of-the-art methods further validates the effectiveness of the proposed approach, as it achieves competitive performance without relying on excessively complex model architectures. Nevertheless, the relatively lower classification performance observed for the lobular carcinoma subtype highlights the inherent difficulty of distinguishing this class. Keywords: Breast Cancer, ResNet, GLCM, Histopathological Images, Classification