More Web Proxy on the site http://driver.im/

research-article

Vajra: step-by-step programming with natural language

Authors:

Viktor Schlegel,

Siegfried Handschuh,

André FreitasAuthors Info & Claims

IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces

Pages 30 - 39

https://doi.org/10.1145/3301275.3302267

Published: 17 March 2019 Publication History

Abstract

Building natural language programming systems that are geared towards end-users requires the abstraction of formalisms inherently introduced by programming languages, capturing the intent of natural language inputs and mapping it to existing programming language constructs.

We present a novel end-user programming paradigm for Python, which maps natural language commands into Python code. The proposed semantic parsing model aims to reduce the barriers for producing well-formed code (syntactic gap) and for exploring third-party APIs (lexico-semantic gap). The proposed method was implemented in a supporting system and evaluated in a usability study involving programmers as well as non-programmers. The results show that both groups are able to produce code with or without prior programming experience.

Supplementary Material

MP4 File (p30-schlegel.mp4)

Download
177.29 MB

References

[1]

Imran Sarwar Bajwa, Ali Samad, and Shahzad Mumtaz. 2009. Object oriented software modeling using NLP based knowledge extraction. European Journal of Scientific Research 35, 01 (2009), 22--33.

[2]

Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Studies 4, 3 (May 2009), 114--123. http://dl.acm.org/citation.cfm?id=2835587.2835589

Digital Library

[3]

John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.

[4]

Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, Subhajit Roy, et al. 2016. Program synthesis using natural language. In Proceedings of the 38th International Conference on Software Engineering. ACM, 345--356.

Digital Library

[5]

James P Diprose, Bruce A MacDonald, and John G Hosking. 2011. Ruru: A spatial and interactive visual programming language for novice robot programming. In Visual Languages and Human-Centric Computing (VL/HCC), 2011 IEEE Symposium on. IEEE, 25--32.

[6]

Manfred Eppe, Sean Trott, and Jerome Feldman. 2016. Exploiting deep semantics and compositionality of natural language for Human-Robot-Interaction. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on. IEEE, 731--738.

[7]

Eclipse Foundation. 2007. Eclipse IDE.

[8]

George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais. 1987. The vocabulary problem in human-system communication. Commun. ACM 30, 11 (1987), 964--971.

Digital Library

[9]

Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. arXiv preprint arXiv:1704.07926 (2017).

[10]

IDEA IntelliJ. 2011. the most intelligent Java IDE. JetBrains{online}.{cit. 2016-02-23}. Dostupné z: https://www.jetbrains.com/idea/#chooseYourEdition (2011).

[11]

Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The State of the Art in End-user Software Engineering. ACM Comput. Surv. 43, 3, Article 21 (April 2011), 44 pages.

Digital Library

[12]

Andrew Jensen Ko, Brad A. Myers, and Htet Htet Aung. 2004. Six Learning Barriers in End-User Programming Systems. In 2004 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2004), 26--29 September 2004, Rome, Italy. 199--206.

Digital Library

[13]

Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, and Ewan Klein. 2002. Mobile robot programming using natural language. Robotics and Autonomous Systems 38, 3 (2002), 171--181.

[14]

V. Levenshtein. 1965. Binary codes capable of correcting spurious insertions and deletions of ones. Problems of Information Transmission. (1965).

[15]

Hang Li. 2011. A short introduction to learning to rank. IEICE TRANSACTIONS on Information and Systems 94, 10 (2011), 1854--1862.

[16]

Henry Lieberman, Fabio Paternò, Markus Klann, and Volker Wulf. 2006. End-user development: An emerging paradigm. In End user development. Springer, 1--8.

[17]

Greg Little and Robert C Miller. 2009. Keyword programming in Java. Automated Software Engineering 16, 1 (2009), 37.

Digital Library

[18]

Hugo Liu and Henry Lieberman. 2005. Metafor: visualizing stories as code. In Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 305--307.

Digital Library

[19]

Val Marcovic. 2015. YouCompleteMe. http://doc.aldebaran.com/1-14/software/choregraphe/index.html.

[20]

Microsoft. 2018. Microsoft Visual Programming Language. https://msdn.microsoft.com/en-us/library/bb483088.aspx.

[21]

Rada Mihalcea, Hugo Liu, and Henry Lieberman. 2006. NLP (natural language processing) for NLP (natural language programming). In CICLing. Springer, 319--330.

Digital Library

[22]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. CoRR abs/1301.3781 (2013). arXiv:1301.3781 http://arxiv.org/abs/1301.3781

[23]

Alon Mishne, Sharon Shoham, and Eran Yahav. 2012. Typestate-based semantic code search over partial programs. In Acm Sigplan Notices, Vol. 47. ACM, 997--1016.

Digital Library

[24]

Gonzalo Navarro. 2001. A Guided Tour to Approximate String Matching. ACM Comput. Surv. 33, 1 (March 2001), 31--88.

Digital Library

[25]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research 12, Oct (2011), 2825--2830.

Digital Library

[26]

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532--1543.

[27]

Aldebaran Robotics. 2017. Choreographe. http://doc.aldebaran.com/1-14/software/choregraphe/index.html. Behaviour Creation Tool for the NAO robot.

[28]

Xin Rong, Shiyan Yan, Stephen Oney, Mira Dontcheva, and Eytan Adar. 2016. CodeMend: Assisting Interactive Programming with Bimodal Embedding. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 247--258.

Digital Library

[29]

Juliano Efson Sales, Siegfried Handschuh, and André Freitas. 2017. SemEval-2017 Task 11: End-User Development using Natural Language. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3--4, 2017. 556--564.

[30]

Miguel Gomez Simon and Lara Lorna Jimenez. 2015. Analysis of End-User programming platforms. http://www.diva-portal.se/smash/get/diva2:995396/FULLTEXT01.pdf

[31]

Yunwen Ye and Gerhard Fischer. 2005. Reuse-conducive development environments. Automated Software Engineering 12, 2 (2005), 199--235.

Digital Library

Cited By

Hatipoğlu ABilgin T(2024)DOĞAL DİL METİNLERİNDEN PROGRAMLAMA DİLİ KODU OLUŞTURMA ÇALIŞMALARI: BİR DERLEME ÇALIŞMASIİstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi10.55071/ticaretfbd.1354040Online publication date: 21-Mar-2024
https://doi.org/10.55071/ticaretfbd.1354040
Do HBrachman MDugan CJohnson JLauer JRai PPan Q(2024)Grounding with Structure: Exploring Design Variations of Grounded Human-AI Collaboration in a Natural Language InterfaceProceedings of the ACM on Human-Computer Interaction10.1145/36869028:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686902
Le KAndrzejak A(2024)Rethinking AI code generation: a one-shot correction approach based on user feedbackAutomated Software Engineering10.1007/s10515-024-00451-y31:2Online publication date: 12-Jul-2024
https://doi.org/10.1007/s10515-024-00451-y
Show More Cited By

Index Terms

Vajra: step-by-step programming with natural language
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

A verified packrat parser interpreter for parsing expression grammars
CPP 2020: Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs

Parsing expression grammars (PEGs) offer a natural opportunity for building verified parser interpreters based on higher-order parsing combinators. PEGs are expressive, unambiguous, and efficient to parse in a top-down recursive descent style. We use ...
Improving Semantic Parsing with Enriched Synchronous Context-Free Grammars in Statistical Machine Translation
TALLIP Notes and Regular Papers

Semantic parsing maps a sentence in natural language into a structured meaning representation. Previous studies show that semantic parsing with synchronous context-free grammars (SCFGs) achieves favorable performance over most other alternatives. ...
Empirical study of novice programming with plans and objects

Empirical studies conducted at two colleges (traditional and online) indicate that students have difficulty with plan integration, understanding of the object-oriented paradigm, and incorporating OOP concepts into problem solving. When novices are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces

March 2019

713 pages

ISBN:9781450362726

DOI:10.1145/3301275

General Chairs:
Wai-Tat Fu,
Shimei Pan,
Program Chairs:
Oliver Brdiczka,
Polo Chau,
Gaelle Calvary

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI '19

Sponsor:

IUI '19: 24th International Conference on Intelligent User Interfaces

March 17 - 20, 2019

California, Marina del Ray

Acceptance Rates

IUI '19 Paper Acceptance Rate 71 of 282 submissions, 25%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
475
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hatipoğlu ABilgin T(2024)DOĞAL DİL METİNLERİNDEN PROGRAMLAMA DİLİ KODU OLUŞTURMA ÇALIŞMALARI: BİR DERLEME ÇALIŞMASIİstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi10.55071/ticaretfbd.1354040Online publication date: 21-Mar-2024
https://doi.org/10.55071/ticaretfbd.1354040
Do HBrachman MDugan CJohnson JLauer JRai PPan Q(2024)Grounding with Structure: Exploring Design Variations of Grounded Human-AI Collaboration in a Natural Language InterfaceProceedings of the ACM on Human-Computer Interaction10.1145/36869028:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686902
Le KAndrzejak A(2024)Rethinking AI code generation: a one-shot correction approach based on user feedbackAutomated Software Engineering10.1007/s10515-024-00451-y31:2Online publication date: 12-Jul-2024
https://doi.org/10.1007/s10515-024-00451-y
Kazemitabaar MChow JMa CEricson BWeintrop DGrossman T(2023)Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580919(1-23)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580919
Liu MSarkar ANegreanu CZorn BWilliams JToronto NGordon A(2023)“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language ModelsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580817(1-31)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580817
Mehdi Pourhashem Kallehbasti MGhafari M(2023)Naturalistic Static Program Analysis2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00083(743-747)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00083
Kim TChoi DChoi YKim J(2022)Stylette: Styling the Web with Natural LanguageProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501931(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501931
Chen TXu LZhu K(2021)FritzBotInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2021.102699155:COnline publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1016/j.ijhcs.2021.102699
Sridhar SSanagavarapu S(2020)A Compiler-based Approach for Natural Language to Code Conversion2020 3rd International Conference on Computer and Informatics Engineering (IC2IE)10.1109/IC2IE50715.2020.9274674(1-6)Online publication date: 15-Sep-2020
https://doi.org/10.1109/IC2IE50715.2020.9274674
Sales JFreitas AOliveira DKoumpis AHandschuh S(2020)Revisiting Principles and Challenges in Natural Language ProgrammingKnowledge-Based Software Engineering: 202010.1007/978-3-030-53949-8_2(7-19)Online publication date: 25-Jul-2020
https://doi.org/10.1007/978-3-030-53949-8_2

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents