Melola Discovery

← Melola Discovery

Jurnal SINTA 2

Visual Recognition Of Graphical User Interface Components Using Deep Learning Technique

Jurnal Ilmu Komputer dan Informasi (JIKI) · Vol. 13 No. 1 (2020) · pp. 35–45

DOI
10.21609/jiki.v13i1.845
ISSN
2502-9274, 2088-7051
Published
2020-03-14
Publisher
Faculty of Computer Science - Universitas Indonesia
Language
eng

Abstract

Graphical User Interface (GUI) building in software development is a process which ideally need to go through several steps. Those steps in the process start from idea or rough sketch of the GUI, then refined into visual design, implemented in coding or prototype, and finally evaluated for its function and usability to discover design problem and to get feedback from users. Those steps repeated until the GUI considered satisfactory or acceptable by the user. Computer vision technique has been researched and developed to make the process faster and easier; for example generating code for implementation, or automatic GUI testing using component images. But among those techniques, there are still few for usability testing purpose. This preliminary research attempted to make the foundation for usability testing using computer vision technique by built minimalist dataset which has images of various GUI components and used the dataset in deep learning experiment for GUI components visual recognition. The experiment results showed deep learning technique suitable for the intended task, with accuracy of 95% for recognition of two different types of components, and accuracy of 72% for six different types of component.

Keywords

Collaboration network of the authors

2 authors · 1 co-authorship link

Hover or tap a name to highlight its co-authors. Double-click to open the author's page.

  • Universitas 17 Agustus 1945 Surabaya
  • Different organisations
  • Same organisation
  • Affiliation unknown
  • Thicker line = more joint articles · bigger circle = more articles
Show as table
Co-authorOrganisationJoint articles

Affiliations come from OpenAlex; journal OAI-PMH metadata carries none. Authors without an OpenAlex ID are matched by name, which can merge or split people.

Metadata source: OAI-PMH (metadata jurnal) https://jiki.cs.ui.ac.id/index.php/jiki/oai, first harvested 26 Okt 2026, last seen 26 Okt 2026.
Original metadata (oai_dc) 
<oai_dc:dc xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/  http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
	<dc:title xml:lang="en-US">Visual Recognition Of Graphical User Interface Components  Using Deep Learning Technique</dc:title>
	<dc:creator>Rahmadi, Agyl Ardi</dc:creator>
	<dc:creator>Sudaryanto, Aris</dc:creator>
	<dc:subject xml:lang="en-US">User Interface</dc:subject>
	<dc:subject xml:lang="en-US">Usability Testing</dc:subject>
	<dc:subject xml:lang="en-US">GUI</dc:subject>
	<dc:subject xml:lang="en-US">Computer Vision</dc:subject>
	<dc:subject xml:lang="en-US">Deep Learning</dc:subject>
	<dc:description xml:lang="en-US">Graphical User Interface (GUI) building in software development is a process which ideally need to go through several steps. Those steps in the process start from idea or rough sketch of the GUI, then refined into visual design, implemented in coding or prototype, and finally evaluated for its function and usability to discover design problem and to get feedback from users. Those steps repeated until the GUI considered satisfactory or acceptable by the user. Computer vision technique has been researched and developed to make the process faster and easier; for example generating code for implementation, or automatic GUI testing using component images. But among those techniques, there are still few for usability testing purpose. This preliminary research attempted to make the foundation for usability testing using computer vision technique by built minimalist dataset which has images of various GUI components and used the dataset in deep learning experiment for GUI components visual recognition. The experiment results showed deep learning technique suitable for the intended task, with accuracy of 95% for recognition of two different types of components, and accuracy of 72% for six different types of component.</dc:description>
	<dc:publisher xml:lang="en-US">Faculty of Computer Science  - Universitas Indonesia</dc:publisher>
	<dc:date>2020-03-14</dc:date>
	<dc:type>info:eu-repo/semantics/article</dc:type>
	<dc:type>info:eu-repo/semantics/publishedVersion</dc:type>
	<dc:format>application/pdf</dc:format>
	<dc:format>application/zip</dc:format>
	<dc:identifier>https://jiki.cs.ui.ac.id/index.php/jiki/article/view/845</dc:identifier>
	<dc:identifier>10.21609/jiki.v13i1.845</dc:identifier>
	<dc:source xml:lang="en-US">Jurnal Ilmu Komputer dan Informasi; Vol. 13 No. 1 (2020): Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information); 35-45</dc:source>
	<dc:source>2502-9274</dc:source>
	<dc:source>2088-7051</dc:source>
	<dc:source>10.21609/jiki.v13i1</dc:source>
	<dc:language>eng</dc:language>
	<dc:relation>https://jiki.cs.ui.ac.id/index.php/jiki/article/view/845/423</dc:relation>
	<dc:relation>https://jiki.cs.ui.ac.id/index.php/jiki/article/view/845/475</dc:relation>
	<dc:rights xml:lang="en-US">Copyright (c) 2020 Jurnal Ilmu Komputer dan Informasi</dc:rights>
</oai_dc:dc>