Article

Automated evaluation of syntax error recovery

Authors:

Maartje de Jonge,

Eelco VisserAuthors Info & Claims

ASE '12: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering

Pages 322 - 325

https://doi.org/10.1145/2351676.2351736

Published: 03 September 2012 Publication History

Get Access

Abstract

Evaluation of parse error recovery techniques is an open problem. The community lacks objective standards and methods to measure the quality of recovery results. This paper proposes an automated technique for recovery evaluation that offers a solution for two main problems in this area. First, a representative testset is generated by a mutation based fuzzing technique that applies knowledge about common syntax errors. Secondly, the quality of the recovery results is automatically measured using an oracle-based evaluation technique. We evaluate the validity of our approach by comparing results obtained by automated evaluation with results obtained by manual inspection. The evaluation shows a clear correspondence between our quality metric and human judgement.

References

[1]

M. Bravenboer, K. T. Kalleberg, R. Vermaas, and E. Visser. Stratego/XT 0.17. A language and toolset for program transformation. Science of Computer Programming, 72(1-2):52–70, June 2008.

Digital Library

Google Scholar

[2]

S. S. Chawathe, A. Rajaraman, H. Garcia-Molina, and J. Widom. Change detection in hierarchically structured information. SIGMOD Rec., 25:493–504, June 1996.

Digital Library

Google Scholar

[3]

M. de Jonge, E. Nilsson-Nyman, L. C. L. Kats, and E. Visser. Natural and ﬂexible error recovery for generated parsers. In M. van den Brand, D. Gasevic, and J. Gray, editors, SLE, volume 5969 of LNCS, pages 204–223. Springer, 2009.

Digital Library

Google Scholar

[4]

M. de Jonge and E. Visser. An algorithm for layout preservation in refactoring transformations. In U. Assmann and T. Sloane, editors, SLE, volume 6940 of LNCS, pages 40–59. Springer, 2012.

Digital Library

Google Scholar

[5]

J. Harm and R. Lämmel. Two-dimensional approximation coverage. Informatica (Slovenia), 24(3), 2000.

Google Scholar

[6]

T. Jiang, L. Wang, and K. Zhang. Alignment of trees - an alternative to tree edit. In CPM ’94, volume 807 of LNCS, pages 75–86, London, 1994. Springer-Verlag.

Digital Library

Google Scholar

[7]

B. Luttik and E. Visser. Speciﬁcation of rewriting strategies. In M. P. A. Sellink, editor, ASF+SDF 1997, Electronic Workshops in Computing, Berlin, 1997. Springer-Verlag.

Digital Library

Google Scholar

[8]

E. Nilsson-Nyman, T. Ekman, and G. Hedin. Practical scope recovery using bridge parsing. In D. Gasevic, R. Lämmel, and E. V. Wyk, editors, SLE, volume 5452 of LNCS, pages 95–113. Springer, 2009.

Digital Library

Google Scholar

[9]

T. J. Pennello and F. DeRemer. A forward move algorithm for LR error recovery. In POPL, pages 241–254. ACM, 1978.

Digital Library

Google Scholar

[10]

G. D. Ripley and F. C. Druseikis. A statistical analysis of syntax errors. Computer Languages, Systems & Structures, 3(4):227––240, 1978.

Digital Library

Google Scholar

[11]

E. Visser. WebDSL: A case study in domain-speciﬁc language engineering. In R. Lämmel, J. Visser, and J. Saraiva, editors, GTTSE, volume 5235 of LNCS, pages 291–373, Heidelberg, 2008. Springer.

Digital Library

Google Scholar

[12]

S. V. Zelenov and S. A. Zelenova. Generation of positive and negative tests for parsers. Programming and Computer Software, 31(6):310–320, 2005.

Digital Library

Google Scholar

Cited By

View all

de Souza Amorim LVisser E(2020)Multi-purpose Syntax Definition with SDF3Software Engineering and Formal Methods10.1007/978-3-030-58768-0_1(1-23)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-58768-0_1
Koo JSaumya CKulkarni MBagchi S(2019)PySE: Automatic Worst-Case Test Generation by Reinforcement Learning2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST.2019.00023(136-147)Online publication date: Apr-2019
https://doi.org/10.1109/ICST.2019.00023
Saumya CKoo JKulkarni MBagchi S(2019)XSTRESSOR : Automatic Generation of Large-Scale Worst-Case Test Inputs by Inferring Path Conditions2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST.2019.00011(1-12)Online publication date: Apr-2019
https://doi.org/10.1109/ICST.2019.00011
Show More Cited By

Index Terms

Automated evaluation of syntax error recovery
1. Software and its engineering
  1. Software notations and tools
    1. Compilers

Recommendations

Syntax error recovery in parsing expression grammars
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied Computing

Parsing Expression Grammars (PEGs) are a formalism used to describe top-down parsers with backtracking. As PEGs do not provide a good error recovery mechanism, PEG-based parsers usually do not recover from syntax errors in the input, or recover from ...
Practical syntactic error recovery

This paper describes a recovery scheme for syntax errors which provides automatically-generated high quality recovery with good diagnostic information at relatively low cost. Previous recovery techniques are summarized and empirical comparisons are ...
Towards automatic error recovery in parsing expression grammars
SBLP '18: Proceedings of the XXII Brazilian Symposium on Programming Languages

Error recovery is an essential feature for a parser that should be plugged in Integrated Development Environments (IDEs), which must build Abstract Syntax Trees (ASTs) even for syntactically invalid programs in order to offer features such as automated ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ASE '12: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering

September 2012

409 pages

ISBN:9781450312042

DOI:10.1145/2351676

General Chair:
Michael Goedicke,
Program Chairs:
Tim Menzies,
Motoshi Saeki

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SIGAI: ACM Special Interest Group on Artificial Intelligence
Universität Duisburg Essen: Universität Duisburg Essen
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ASE'12

Sponsor:

SIGSOFT

ASE'12: IEEE/ACM International Conference on Automated Software Engineering

September 3 - 7, 2012

Essen, Germany

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
188
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

de Souza Amorim LVisser E(2020)Multi-purpose Syntax Definition with SDF3Software Engineering and Formal Methods10.1007/978-3-030-58768-0_1(1-23)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.1007/978-3-030-58768-0_1
Koo JSaumya CKulkarni MBagchi S(2019)PySE: Automatic Worst-Case Test Generation by Reinforcement Learning2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST.2019.00023(136-147)Online publication date: Apr-2019
https://doi.org/10.1109/ICST.2019.00023
Saumya CKoo JKulkarni MBagchi S(2019)XSTRESSOR : Automatic Generation of Large-Scale Worst-Case Test Inputs by Inferring Path Conditions2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)10.1109/ICST.2019.00011(1-12)Online publication date: Apr-2019
https://doi.org/10.1109/ICST.2019.00011
Liang HPei XJia XShen WZhang J(2018)Fuzzing: State of the ArtIEEE Transactions on Reliability10.1109/TR.2018.283447667:3(1199-1218)Online publication date: Sep-2018
https://doi.org/10.1109/TR.2018.2834476
Di Nardo DPastore FBriand L(2017)Augmenting Field Data for Testing Systems Subject to Incremental Requirements ChangesACM Transactions on Software Engineering and Methodology10.1145/305343026:1(1-40)Online publication date: 10-May-2017
https://dl.acm.org/doi/10.1145/3053430
Di Nardo DPastore FBriand L(2015)Generating Complex and Faulty Test Data through Model-Based Mutation Analysis2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2015.7102589(1-10)Online publication date: Apr-2015
https://doi.org/10.1109/ICST.2015.7102589
Di Nardo DPastore FArcuri ABriand LCohen MGrunske LWhalen M(2015)Evolutionary robustness testing of data processing systems using models and data mutationProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2015.13(126-137)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1109/ASE.2015.13
de Jonge MKats LVisser ESöderberg E(2012)Natural and Flexible Error Recovery for Generated Modular Language EnvironmentsACM Transactions on Programming Languages and Systems10.1145/2400676.240067834:4(1-50)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1145/2400676.2400678

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Syntax error recovery in parsing expression grammars

Practical syntactic error recovery

Towards automatic error recovery in parsing expression grammars