Work in Progress

SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors

Authors:

Vinoth Pandian Sermuga Pandian,

Sarah Suleri,

Matthias JarkeAuthors Info & Claims

IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

Pages 67 - 69

https://doi.org/10.1145/3397482.3450725

Published: 14 April 2021 Publication History

Get Access

Abstract

User Interface (UI) prototyping is an iterative process where designers initially sketch UIs before transforming them into interactive digital designs. Recent research applies Deep Neural Networks (DNNs) to identify the constituent UI elements of these UI sketches and transform these sketches into front-end code. Training such DNN models requires a large-scale dataset of UI sketches, which is time-consuming and expensive to collect. Therefore, we earlier proposed Syn to generate UI sketches synthetically by random allocation of UI element sketches. However, these UI sketches are not statistically similar to real-life UI screens. To bridge this gap, in this paper, we introduce the SynZ dataset, which contains 175,377 synthetically generated UI sketches statistically similar to real-life UI screens. To generate SynZ, we analyzed, enhanced, and extracted annotations from the RICO dataset and used 17,979 hand-drawn UI element sketches from the UISketch dataset. Further, we fine-tuned a UI element detector with SynZ and observed that it doubles the mean Average Precision of UI element detection compared to the Syn dataset.

Supplementary Material

p67-pandian-supplement (p67-pandian-supplement.png)

Poster

Download
539.12 KB

References

[1]

Georgia Albuquerque, Thomas Löwe, and Marcus Magnor. 2011. Synthetic Generation of High-Dimensional Datasets. IEEE transactions on visualization and computer graphics 17 (12 2011), 2317–24. https://doi.org/10.1109/tvcg.2011.237

Crossref

Google Scholar

[2]

Kumar Ashwin. 2018. sketch-code. https://github.com/ashnkumar/sketch-code.

Google Scholar

[3]

Wilkins Benjamin. 2017. Sketching Interfaces - Generating code from low fidelity wireframes. https://airbnb.design/sketching-interfaces/.

Google Scholar

[4]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 845–854. https://doi.org/10.1145/3126594.3126651

Digital Library

Google Scholar

[5]

Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019. Swire: Sketch-based User Interface Retrieval(CHI ’19). ACM, New York, NY, USA, Article 104, 10 pages. https://doi.org/10.1145/3290605.3300334

Digital Library

Google Scholar

[6]

Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning Design Semantics for Mobile Apps. In The 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). ACM, New York, NY, USA, 569–579. https://doi.org/10.1145/3242587.3242650

Digital Library

Google Scholar

[7]

Microsoft. 2018. Sketch2Code - Microsoft/ailab. https://github.com/Microsoft/ailab/tree/master/Sketch2Code.

Google Scholar

[8]

Vinoth Pandian Sermuga Pandian, Sarah Suleri, Christian Beecks, and Matthias Jarke. 2020. MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities. In Proceedings of the 32nd Australian Conference on HCI (Sydney, Australia) (ozCHI’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3441000.3441030

Digital Library

Google Scholar

[9]

Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2020. Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion(Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 79–80. https://doi.org/10.1145/3379336.3381498

Digital Library

Google Scholar

[10]

Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021. UISketch: A Large-Scale Dataset of UI Element Sketches (forthcoming). In CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445784

Digital Library

Google Scholar

[11]

N. Patki, R. Wedge, and K. Veeramachaneni. 2016. The Synthetic Data Vault. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA). 399–410. https://doi.org/10.1109/DSAA.2016.49

Crossref

Google Scholar

[12]

Sarah Suleri, Vinoth Pandian Sermuga Pandian, Svetlana Shishkovets, and Matthias Jarke. 2019. Eve: A Sketch-based Software Prototyping Workbench. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI EA ’19). ACM, New York, NY, USA, Article Lbw1410, 6 pages. https://doi.org/10.1145/3290607.3312994

Digital Library

Google Scholar

[13]

Beltramelli Tony. 2017. Teaching Machines to Understand User Interfaces. https://hackernoon.com/teaching-machines-to-understand-user-interfaces-5a0cdeb4d579.

Google Scholar

Cited By

View all

Wu JWang SShen SPeng YNichols JBigham J(2023)WebUI: A Dataset for Enhancing Visual UI Understanding with Web SemanticsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581158(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581158
Sermuga Pandian VShams ASuleri SJarke P(2022)LoFi Sketch: A Large Scale Dataset of Smartphone Low Fidelity SketchesExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519624(1-5)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519624

Index Terms

SynZ: Enhanced Synthetic Dataset for Training UI Element Detectors
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
  2. Interaction design
2. Software and its engineering

Index terms have been assigned to the content through auto-classification.

Recommendations

Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches
IUI '20 Companion: Companion Proceedings of the 25th International Conference on Intelligent User Interfaces

User Interface design is an iterative process that progresses through low-, medium-, and high-fidelity prototypes. A few research projects use deep learning to automate this process by transforming low fidelity (lo-fi) sketches into front-end code. ...
UISketch: A Large-Scale Dataset of UI Element Sketches
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

This paper contributes the first large-scale dataset of 17,979 hand-drawn sketches of 21 UI element categories collected from 967 participants, including UI/UX designers, front-end developers, HCI, and CS grad students, from 10 different countries. We ...
MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities
OzCHI '20: Proceedings of the 32nd Australian Conference on Human-Computer Interaction

Transforming lo-fi UI sketches to higher-fidelities is an expensive, time-consuming process that requires significant rework. In this paper, we systematically research utilizing AI to assist the transformation of lo-fi sketches to higher fidelities. To ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

April 2021

101 pages

ISBN:9781450380188

DOI:10.1145/3397482

Co-chairs:
Katrien Verbert
Katholieke Universiteit Leuven
,
Dennis Parra
Pontificia Universidad Católica de Chile
,
General Chair:
Tracy Hammond
Texas A&M University
,
Program Chairs:
Bart Knijnenburg
Clemson University
,
John O'Donovan
UC Santa Barbara
,
Paul Taele
Texas A&M University

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2021

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Conference

IUI '21

Sponsor:

IUI '21: 26th International Conference on Intelligent User Interfaces

April 14 - 17, 2021

TX, College Station, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
201
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu JWang SShen SPeng YNichols JBigham J(2023)WebUI: A Dataset for Enhancing Visual UI Understanding with Web SemanticsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581158(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581158
Sermuga Pandian VShams ASuleri SJarke P(2022)LoFi Sketch: A Large Scale Dataset of Smartphone Low Fidelity SketchesExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519624(1-5)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519624

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplementary Material

References

Cited By

Index Terms

Recommendations

Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches

UISketch: A Large-Scale Dataset of UI Element Sketches

MetaMorph: AI Assistance to Transform Lo-Fi Sketches to Higher Fidelities

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations