Melola Discovery

← Melola Discovery

Jurnal Tanpa peringkat SINTA

Which Features Matter Most? Evaluating Numerical and Textual Features for Helpfulness Classification in Imbalance Dataset using XGBoost

Scientific Journal of Informatics · Vol. 12 No. 4 (2025) · hlm. 731–742

DOI
10.15294/sji.v12i4.33443
ISSN
2407-7658, 2460-0040
Terbit
2025-11-19
Penerbit
Universitas Negeri Semarang
Bahasa
eng

Abstrak

Purpose: This study aims to develop and realistically evaluate a reliable model for identifying helpful online reviews, particularly in the context of Indonesian-language texts, which are often informal and challenging. Methods: This study addresses several key challenges in predicting review helpfulness: the relative effectiveness of numerical features from metadata compared with traditional text representations (TF-IDF, FastText) on noisy data; the impact of severe class imbalance; and the limitations of standard validation compared with time-based validation. To address these challenges, we built an XGBoost model and evaluated various feature combinations. A hybrid approach combining SMOTE and scale_pos_weight was applied to handle class imbalance, and the best configuration was further assessed using time-based validation to better simulate real-world conditions. Result: The results show that the model based on numerical features consistently outperformed the text-based model, achieving a peak macro F1-score of 0.7214. Compared to the IndoBERT baseline (F1-score = 0.6400) and the RCNN FastText baseline (F1-score = 0.5317), this indicates that simpler feature-driven models can provide more reliable predictions under noisy review data. Time-based validation further revealed a performance decline of up to 8.06%, confirming the presence of concept drift and highlighting that standard validation tends to yield overly optimistic estimates. Novelty: The main contribution of this research lies in offering a robust methodology while demonstrating the superiority of metadata-based approaches in this context. By quantifying performance degradation through temporal validation, this study provides a more realistic benchmark for real-world applications and highlights the critical importance of regular model retraining.

Kata kunci

Jaringan kolaborasi para penulis

10 penulis · 21 tautan kepenulisan bersama · 2 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Sebelas Maret University
  • University of Indonesia
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Wiharto WihartoSebelas Maret University1
Firdaus Ashiddqi NoorSebelas Maret University1
Gita Bangun PrakosoSebelas Maret University1
Dewi Wisnu WardaniSebelas Maret University1
Hasan Dwi CahyonoSebelas Maret University1
Endar Suprih WihidayatSebelas Maret University1
Bambang WidoyonoUniversity of Indonesia1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) https://journal.unnes.ac.id/journals/sji/oai, pertama di-harvest 6 Nov 2026, terakhir terlihat 8 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en">Which Features Matter Most? Evaluating Numerical and Textual Features for Helpfulness Classification in Imbalance Dataset using XGBoost</dc:title>
	<dc:creator xml:lang="en">Kirani, Anindita Putri</dc:creator>
	<dc:creator xml:lang="en">Saptono, Ristu</dc:creator>
	<dc:creator xml:lang="en">Anggrainingsih, Rini</dc:creator>
	<dc:subject xml:lang="en">Review helpfulness</dc:subject>
	<dc:subject xml:lang="en">Helpful vote</dc:subject>
	<dc:subject xml:lang="en">Time-based evaluation</dc:subject>
	<dc:subject xml:lang="en">Imbalanced data handling </dc:subject>
	<dc:description xml:lang="en">Purpose: This study aims to develop and realistically evaluate a reliable model for identifying helpful online reviews, particularly in the context of Indonesian-language texts, which are often informal and challenging.
Methods: This study addresses several key challenges in predicting review helpfulness: the relative effectiveness of numerical features from metadata compared with traditional text representations (TF-IDF, FastText) on noisy data; the impact of severe class imbalance; and the limitations of standard validation compared with time-based validation. To address these challenges, we built an XGBoost model and evaluated various feature combinations. A hybrid approach combining SMOTE and scale_pos_weight was applied to handle class imbalance, and the best configuration was further assessed using time-based validation to better simulate real-world conditions.
Result: The results show that the model based on numerical features consistently outperformed the text-based model, achieving a peak macro F1-score of 0.7214. Compared to the IndoBERT baseline (F1-score = 0.6400) and the RCNN FastText baseline (F1-score = 0.5317), this indicates that simpler feature-driven models can provide more reliable predictions under noisy review data. Time-based validation further revealed a performance decline of up to 8.06%, confirming the presence of concept drift and highlighting that standard validation tends to yield overly optimistic estimates.
Novelty: The main contribution of this research lies in offering a robust methodology while demonstrating the superiority of metadata-based approaches in this context. By quantifying performance degradation through temporal validation, this study provides a more realistic benchmark for real-world applications and highlights the critical importance of regular model retraining. </dc:description>
	<dc:publisher xml:lang="en">Universitas Negeri Semarang</dc:publisher>
	<dc:date>2025-11-19</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en">Peer-reviewed Article</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://journal.unnes.ac.id/journals/sji/article/view/33443</dc:identifier>
	<dc:identifier>10.15294/sji.v12i4.33443</dc:identifier>
	<dc:source xml:lang="en">Scientific Journal of Informatics; Vol. 12 No. 4: Special Issue; 731-742</dc:source>
	<dc:source>2407-7658</dc:source>
	<dc:source>2460-0040</dc:source>
	<dc:source>10.15294/sji.v12i4</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://journal.unnes.ac.id/journals/sji/article/view/33443/8435</dc:relation>
	<dc:rights xml:lang="en">Copyright (c) 2025 Scientific Journal of Informatics</dc:rights>
</oai_dc:dc>