Melola Discovery

← Melola Discovery

Akses terbuka Jurnal SINTA 1

Indonesian-English Textual Similarity Detection Using Universal Sentence Encoder (USE) and Facebook AI Similarity Search (FAISS)

CommIT (Communication and Information Technology) Journal · Vol. 18 No. 2 (2024) · hlm. 183–195

DOI
10.21512/commit.v18i2.11274
ISSN
2460-7010, 1979-2484
Terbit
2024-09-10
Penerbit
Bina Nusantara University
Bahasa
eng
Lisensi
creativecommons.org/licenses/by-sa/4.0

Abstrak

The tremendous development in Natural Language Processing (NLP) has enabled the detection of bilingual and multilingual textual similarity. One of the main challenges of the Textual Similarity Detection (TSD) system lies in learning effective text representation. The research focuses on identifying similar texts between Indonesian and English across a broad range of semantic similarity spectrums. The primary challenge is generating English and Indonesian dense vector representation, a.k.a. embeddings that share a single vector space. Through trial and error, the research proposes using the Universal Sentence Encoder (USE) model to construct bilingual embeddings and FAISS to index the bilingual dataset. The comparison between query vectors and index vectors is done using two approaches: the heuristic comparison with Euclidian distance and a clustering algorithm, Approximate Nearest Neighbors (ANN). The system is tested with four different semantic granularities, two text granularities, and evaluation metrics with a cutoff value of k={2,10}. Four semantic granularities used are highly similar or near duplicate, Semantic Entailment (SE), Topically Related (TR), and Out of Topic (OOT), while the text granularities take on the sentence and paragraph levels. The experimental results demonstrate that the proposed system successfully ranks similar texts in different languages within the top ten. It has been proven by the highest F1@2 score of 0.96 for the near duplicate category on the sentence level. Unlike the near-duplicate category, the highest F1 scores of 0.77 and 0.89 are shown by the SE and TR categories, respectively. The experiment results also show a high correlation between text and semantic granularity.

Kata kunci

Jaringan kolaborasi para penulis

25 penulis · 53 tautan kepenulisan bersama · 15 lintas organisasi

Arahkan kursor atau ketuk sebuah nama untuk menyorot rekan penulisnya. Klik dua kali untuk membuka halaman penulis.

  • Multimedia University
  • Duta Wacana Christian University
  • Telkom University
  • SRM Institute of Science and Technology
  • Jining Normal University
  • Afiliasi tidak diketahui
  • Organisasi berbeda
  • Organisasi sama
  • Afiliasi tidak diketahui
  • Garis lebih tebal = lebih banyak artikel bersama · lingkaran lebih besar = lebih banyak artikel
Tampilkan sebagai tabel
Rekan penulisOrganisasiArtikel bersama
Naveen PalanichamyMultimedia University4
Lit-Jie ChewMultimedia University2
Dana Sulistyo KusumoTelkom University2
Mandy SiaMultimedia University2
J. JayapradhaSRM Institute of Science and Technology2
Elham Abdulwahab AnaamMultimedia University2
Gee-Kok TongMultimedia University2
Shi-Kuo ChangMultimedia University2
Yih-Jian YoongMultimedia University2
Sri SuwarnoDuta Wacana Christian University1
Samini SubramaniamAfiliasi tidak diketahui1
Poo Kuan HoongMultimedia University1
Gloria VirginiaDuta Wacana Christian University1
Bill Edward Butar-ButarDuta Wacana Christian University1
Khye SpongeMultimedia University1
Choo‐Yee TingMultimedia University1
Ian ChaiMultimedia University1
Emyliana SongMultimedia University1
Li-xian JiaoJining Normal University1
Hau Lee TongMultimedia University1

Afiliasi berasal dari OpenAlex; metadata OAI-PMH jurnal tidak memuatnya. Penulis tanpa ID OpenAlex dicocokkan berdasarkan nama, sehingga dua orang bisa tergabung atau satu orang bisa terpisah.

Sumber metadata: OAI-PMH (metadata jurnal) https://journal.binus.ac.id/index.php/commit/oai, pertama di-harvest 8 Nov 2026, terakhir terlihat 16 Nov 2026.
Metadata asli (oai_dc)
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en-US">Indonesian-English Textual Similarity Detection Using Universal Sentence Encoder (USE) and Facebook AI Similarity Search (FAISS)</dc:title>
	<dc:creator>Krisnawati, Lucia D.</dc:creator>
	<dc:creator>Mahastama, Aditya W.</dc:creator>
	<dc:creator>Haw, Su-Cheng</dc:creator>
	<dc:creator>Ng, Kok-Why</dc:creator>
	<dc:creator>Naveen, Palanichamy</dc:creator>
	<dc:subject xml:lang="en-US">Textual Similarity Detection</dc:subject>
	<dc:subject xml:lang="en-US">Universal Sentence Encoder (USE)</dc:subject>
	<dc:subject xml:lang="en-US">Facebook AI Similarity Search (FAISS)</dc:subject>
	<dc:description xml:lang="en-US">The tremendous development in Natural Language Processing (NLP) has enabled the detection of bilingual and multilingual textual similarity. One of the main challenges of the Textual Similarity Detection (TSD) system lies in learning effective text representation. The research focuses on identifying similar texts between Indonesian and English across a broad range of semantic similarity spectrums. The primary challenge is generating English and Indonesian dense vector representation, a.k.a. embeddings that share a single vector space. Through trial and error, the research proposes using the Universal Sentence Encoder (USE) model to construct bilingual embeddings and FAISS to index the bilingual dataset. The comparison between query vectors and index vectors is done using two approaches: the heuristic comparison with Euclidian distance and a clustering algorithm, Approximate Nearest Neighbors (ANN). The system is tested with four different semantic granularities, two text granularities, and evaluation metrics with a cutoff value of k={2,10}. Four semantic granularities used are highly similar or near duplicate, Semantic Entailment (SE), Topically Related (TR), and Out of Topic (OOT), while the text granularities take on the sentence and paragraph levels. The experimental results demonstrate that the proposed system successfully ranks similar texts in different languages within the top ten. It has been proven by the highest F1@2 score of 0.96 for the near duplicate category on the sentence level. Unlike the near-duplicate category, the highest F1 scores of 0.77 and 0.89 are shown by the SE and TR categories, respectively. The experiment results also show a high correlation between text and semantic granularity.</dc:description>
	<dc:publisher xml:lang="en-US">Bina Nusantara University</dc:publisher>
	<dc:date>2024-09-10</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:type xml:lang="en-US">Peer-reviewed Article</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:identifier>https://journal.binus.ac.id/index.php/commit/article/view/11274</dc:identifier>
	<dc:identifier>10.21512/commit.v18i2.11274</dc:identifier>
	<dc:source xml:lang="en-US">CommIT (Communication and Information Technology) Journal; Vol. 18 No. 2 (2024): CommIT Journal ; 183-195</dc:source>
	<dc:source>2460-7010</dc:source>
	<dc:source>1979-2484</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://journal.binus.ac.id/index.php/commit/article/view/11274/5179</dc:relation>
	<dc:rights xml:lang="en-US">Copyright (c) 2024 Lucia D. Krisnawati, Aditya W. Mahastama, Su-Cheng Haw, Kok-Why Ng, Palanichamy Naveen</dc:rights>
	<dc:rights xml:lang="en-US">https://creativecommons.org/licenses/by-sa/4.0</dc:rights>
</oai_dc:dc>