[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3324884.3416556acmconferencesArticle/Chapter ViewAbstractPublication PagesaseConference Proceedingsconference-collections
research-article
Public Access

FlashRegex: deducing anti-ReDoS regexes from examples

Published: 27 January 2021 Publication History

Abstract

Regular expressions (regexes) are widely used in different fields of computer science such as programming languages, string processing and databases. However, existing tools for synthesizing or repairing regexes were not designed to be resilient to Regex Denial of Service (ReDoS) attacks. Specifically, if a regex has super-linear (SL) worst-case complexity, an attacker could provide carefully-crafted inputs to launch ReDoS attacks. Therefore, in this paper, we propose a programming-by-example framework, FlashRegex, for generating anti-ReDoS regexes by either synthesizing or repairing from given examples. It is the first framework that integrates regex synthesis and repair with the awareness of ReDoS-vulnerabilities. We present novel algorithms to deduce anti-ReDoS regexes by reducing the ambiguity of these regexes and by using Boolean Satisfiability (SAT) or Neighborhood Search (NS) techniques. We evaluate FlashRegex with five related state-of-the-art tools. The evaluation results show that our work can effectively and efficiently generate anti-ReDoS regexes from given examples, and also reveal that existing synthesis and repair tools have neglected ReDoS-vulnerabilities of regexes. Specifically, the existing synthesis and repair tools generated up to 394 ReDoS-vulnerable regex within few seconds to more than one hour, while FlashRegex generated no SL regex within around five seconds. Furthermore, the evaluation results on ReDoS-vulnerable regex repair also show that FlashRegex has better capability than existing repair tools and even human experts, achieving 4 more ReDoS-invulnerable regex after repair without trimming and resorting, highlighting the usefulness of FlashRegex in terms of the generality, automation and user-friendliness.

References

[1]
Maaz Bin Safeer Ahmad and Alvin Cheung. 2016. Leveraging Parallel Data Processing Frameworks With Verified Lifting. In Proceedings Fifth Workshop on Synthesis, SYNT@CAV 2016, Toronto, Canada, July 17--18, 2016. 67--83.
[2]
Adam Baldwin. 2016. Regular Expression Denial Of Service Affecting Express.js. https://medium.com/node-security/
[3]
Alberto Bartoli, Giorgio Davanzo, Andrea De Lorenzo, Eric Medvet, and Enrico Sorio. 2014. Automatic Synthesis Of Regular Expressions From Examples. IEEE Computer 47, 12 (2014), 72--80.
[4]
Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. 2014. Playing Regex Golf With Genetic Programming. In Genetic and Evolutionary Computation Conference, GECCO '14, Vancouver, BC, Canada, July 12--16, 2014. 1063--1070.
[5]
Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. 2016. Inference Of Regular Expressions For Text Extraction From Examples. IEEE Trans. Knowl. Data Eng. 28, 5 (2016), 1217--1230.
[6]
Michela Becchi and Srihari Cadambi. 2007. Memory-Efficient Regular Expression Search Using State Merging. In INFOCOM 2007. 26th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, 6--12 May 2007, Anchorage, Alaska, USA. IEEE, 1064--1072.
[7]
Geert Jan Bex, Wouter Gelade, Frank Neven, and Stijn Vansummeren. 2010. Learning Deterministic Regular Expressions For The Inference Of Schemas From XML Data. TWEB 4, 4 (2010), 14:1--14:32.
[8]
Geert Jan Bex, Frank Neven, Thomas Schwentick, and Stijn Vansummeren. 2010. Inference Of Concise Regular Expressions And DTDs. ACM Trans. Database Syst. 35, 2 (2010), 11:1--11:47.
[9]
Anne Brüggemann-Klein. 1993. Unambiguity Of Extended Regular Expressions In SGML Document Grammars. In Algorithms - ESA '93, First Annual European Symposium, Bad Honnef, Germany, September 30 - October 2, 1993, Proceedings. 73--84.
[10]
Carl Chapman, Peipei Wang, and Kathryn T. Stolee. 2017. Exploring Regular Expression Comprehension. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017. 405--416.
[11]
Brendan Cody-Kenny, Michael Fenton, Adrian Ronayne, Eoghan Considine, Thomas McGuire, and Michael O'Neill. 2017. A Search For Improved Performance In Regular Expressions. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, Berlin, Germany, July 15--19, 2017. 1280--1287.
[12]
Russ Cox. 2007. Regular Expression Matching Can Be Simple And Fast (But Is Slow In Java, Perl, PHP, Python, Ruby, ...). https://swtch.com/~rsc/regexp/regexp1.html
[13]
Loris D'Antoni, Rishabh Singh, and Michael Vaughn. 2017. NoFAQ: Synthesizing Command Repairs From Examples. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4--8, 2017. 582--592.
[14]
James C. Davis, Christy A. Coghlan, Francisco Servant, and Dongyoon Lee. 2018. The Impact Of Regular Expression Denial Of Service (ReDoS) In Practice: An Empirical Study At The Ecosystem Scale. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04--09, 2018. 246--256.
[15]
James C. Davis, Louis G. Michael IV, Christy A. Coghlan, Francisco Servant, and Dongyoon Lee. 2019. Why Aren't Regular Expressions A Lingua Franca? An Empirical Study On The Re-Use And Portability Of Regular Expressions. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26--30, 2019. 443--454.
[16]
Leonardo Mendonça de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings. 337--340.
[17]
Erling Ellingsen. 2019. Regex Golf. https://alf.nu/RegexGolf
[18]
Kevin Ellis and Sumit Gulwani. 2017. Learning To Learn Programs From Examples: Going Beyond Program Structure. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19--25, 2017. 1638--1645.
[19]
Stack Exchange. 2016. Outage Postmortem. http://stackstatus.net/post/147710624694/outage-postmortem-july-20-2016
[20]
Yu Feng, Ruben Martins, Jacob Van Geffen, Isil Dillig, and Swarat Chaudhuri. 2017. Component-Based Synthesis Of Table Consolidation And Transformation Tasks From Examples. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2017, Barcelona, Spain, June 18--23, 2017. 422--436.
[21]
Henning Fernau. 2009. Algorithms For Learning Regular Expressions From Positive Data. Inf. Comput. 207, 4 (2009), 521--541.
[22]
John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations From Input-Output Examples. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, Portland, OR, USA, June 15--17, 2015. 229--239.
[23]
Python Software Foundation. 2018. PyPI-The Python Package Index. https://pypi.org/
[24]
Dominik D. Freydenberger and Timo Kötzing. 2015. Fast Learning Of Restricted Regular Expressions And DTDs. Theory Comput. Syst. 57, 4 (2015), 1114--1158.
[25]
E. Mark Gold. 1978. Complexity Of Automaton Identification From Given Data. Information and Control 37, 3 (1978), 302--320.
[26]
Benoît Groz and Sebastian Maneth. 2017. Efficient Testing And Matching Of Deterministic Regular Expressions. J. Comput. Syst. Sci. 89 (2017), 372--399.
[27]
Sumit Gulwani. 2011. Automating String Processing In Spreadsheets Using Input-Output Examples. In Proceedings of the 38th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2011, Austin, TX, USA, January 26--28, 2011. 317--330.
[28]
William R. Harris and Sumit Gulwani. 2011. Spreadsheet Table Transformations From Examples. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, San Jose, CA, USA, June 4--8, 2011. 317--328.
[29]
Louis G. Michael IV, James Donohue, James C. Davis, Dongyoon Lee, and Francisco Servant. 2019. Regexes Are Hard: Decision-Making, Difficulties, And Risks In Programming Regular Expressions. In 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11--15, 2019. 415--426.
[30]
James Kirrage, Asiri Rathnayake, and Hayo Thielecke. 2013. Static Analysis For Regular Expression Denial-of-Service Attacks. In Network and System Security - 7th International Conference, NSS 2013, Madrid, Spain, June 3--4, 2013. Proceedings. 135--148.
[31]
Nate Kushman and Regina Barzilay. 2013. Using Semantic Unification To Generate Regular Expressions From Natural Language. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9--14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA. 826--836.
[32]
Mina Lee, Sunbeom So, and Hakjoo Oh. 2016. Synthesizing Regular Expressions From Examples For Introductory Automata Assignments. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences, GPCE 2016, Amsterdam, The Netherlands, October 31 - November 1, 2016. 70--80.
[33]
Yunyao Li, Rajasekar Krishnamurthy, Sriram Raghavan, Shivakumar Vaithyanathan, and H. V. Jagadish. 2008. Regular Expression Learning For Information Extraction. In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25--27 October 2008, Honolulu, Hawaii, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. 21--30.
[34]
Yeting Li, Xiaolan Zhang, Jialun Cao, Haiming Chen, and Chong Gao. 2019. Learning K-Occurrence Regular Expressions With Interleaving. In Database Systems for Advanced Applications - 24th International Conference, DASFAA 2019, Chiang Mai, Thailand, April 22--25, 2019, Proceedings, Part II. 70--85.
[35]
Cheng-Hung Lin, Chen-Hsiung Liu, and Shih-Chieh Chang. 2011. Accelerating Regular Expression Matching Using Hierarchical Parallel Machines On GPU. In Proceedings of the Global Communications Conference, GLOBECOM 2011, 5--9 December 2011, Houston, Texas, USA. IEEE, 1--5.
[36]
Nicholas Locascio, Karthik Narasimhan, Eduardo DeLeon, Nate Kushman, and Regina Barzilay. 2016. Neural Generation Of Regular Expressions From Natural Language With Minimal Domain Knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1--4, 2016. 1918--1923.
[37]
Anders Møller. 2017. dk.brics.automaton - Finite-State Automata and Regular Expressions for Java. https://www.brics.dk/automaton/
[38]
Inc. NPM. 2018. NPM. https://www.npmjs.com/
[39]
Rong Pan, Qinheping Hu, Gaowei Xu, and Loris D'Antoni. 2019. Automatic Repair Of Regular Expressions. PACMPL 3, OOPSLA (2019), 139:1--139:29.
[40]
Jun-U. Park, Sang-Ki Ko, Marco Cognetta, and Yo-Sub Han. 2019. SoftRegex: Generating Regex From Natural Language Descriptions Using Softened Regex Equivalence. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, HongKong, China, November 3--7, 2019. 6424--6430.
[41]
Theofilos Petsios, Jason Zhao, Angelos D. Keromytis, and Suman Jana. 2017. Slow-Fuzz: Automated Domain-Independent Detection Of Algorithmic Complexity Vulnerabilities. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 - November 03, 2017. 2155--2168.
[42]
Asiri Rathnayake. 2015. Semantics, Analysis And Security Of Backtracking Regular Expression Matchers. Ph.D. Dissertation. University of Birmingham, UK.
[43]
Asiri Rathnayake and Hayo Thielecke. 2014. Static Analysis For Regular Expression Exponential Runtime Via Substructural Logics. CoRR abs/1405.7058 (2014).
[44]
Thomas Rebele, Katerina Tzompanaki, and Fabian M. Suchanek. 2018. Adding Missing Words To Regular Expressions. In Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3--6, 2018, Proceedings, Part II. 67--79.
[45]
RegExLib. 2019. Regular Expression Library. http://regexlib.com/
[46]
David E. Shaw, William R. Swartout, and C. Cordell Green. 1975. Inferring LISP Programs From Examples. In Advance Papers of the Fourth International Joint Conference on Artificial Intelligence, Tbilisi, Georgia, USSR, September 3--8, 1975. 260--267.
[47]
Yuju Shen, Yanyan Jiang, Chang Xu, Ping Yu, Xiaoxing Ma, and Jian Lu. 2018. ReScue: Crafting Regular Expression DoS Attacks. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3--7, 2018. 225--235.
[48]
Rishabh Singh. 2016. BlinkFill: Semi-Supervised Programming By Example For Syntactic String Transformations. PVLDB 9, 10 (2016), 816--827.
[49]
Rishabh Singh and Sumit Gulwani. 2012. Learning Semantic String Transformations From Examples. PVLDB 5, 8 (2012), 740--751.
[50]
Calvin Smith and Aws Albarghouthi. 2016. MapReduce Program Synthesis. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2016, Santa Barbara, CA, USA, June 13--17, 2016. 326--340.
[51]
Eric Spishak, Werner Dietl, and Michael D. Ernst. 2012. A Type System For Regular Expressions. In Proceedings of the 14th Workshop on Formal Techniques for Java-like Programs, FTfJP 2012, Beijing, China, June 12, 2012. 20--26.
[52]
Satoshi Sugiyama and Yasuhiko Minamide. 2014. Checking Time Linearity Of Regular Expression Matching Based On Backtracking. Information and Media Technologies 9, 3 (2014), 222--232.
[53]
Bryan Sullivan. 2010. New Tool: SDL Regex Fuzzer. https://cloudblogs.microsoft.com/microsoftsecure/2010/10/12/new-tool-sdl-regex-fuzzer
[54]
Ken Thompson. 1968. Regular Expression Search Algorithm. Commun. ACM 11, 6 (1968), 419--422.
[55]
Brink van der Merwe, Nicolaas Weideman, and Martin Berglund. 2017. Turning Evil Regexes Harmless. In Proceedings of the South African Institute of Computer Scientists and Information Technologists, SAICSIT 2017, Thaba Nchu, South Africa, September 26--28, 2017. 38:1--38:10.
[56]
Chenglong Wang, Alvin Cheung, and Rastislav Bodík. 2017. Synthesizing Highly Expressive SQL Queries From Input-Output Examples. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2017, Barcelona, Spain, June 18--23, 2017. 452--466.
[57]
Xinyu Wang, Sumit Gulwani, and Rishabh Singh. 2016. FIDEX: Filtering Spread-sheet Data Using Examples. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016, Amsterdam, The Netherlands, October 30 - November 4, 2016. 195--213.
[58]
Nicolaas Weideman, Brink van der Merwe, Martin Berglund, and Bruce W. Watson. 2016. Analyzing Matching Time Behavior Of Backtracking Regular Expression Matchers By Using Ambiguity Of NFA. In Implementation and Application of Automata - 21st International Conference, CIAA 2016, Seoul, South Korea, July 19--22, 2016, Proceedings. 322--334.
[59]
Valentin Wüstholz, Oswaldo Olivo, Marijn J. H. Heule, and Isil Dillig. 2017. Static Detection Of DoS Vulnerabilities In Programs That Use Regular Expressions. In Tools and Algorithms for the Construction and Analysis of Systems - 23rd International Conference, TACAS 2017, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2017, Uppsala, Sweden, April 22--29, 2017, Proceedings, Part II. 3--20.
[60]
Navid Yaghmazadeh, Christian Klinger, Isil Dillig, and Swarat Chaudhuri. 2016. Synthesizing Transformations On Hierarchically Structured Data. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2016, Santa Barbara, CA, USA, June 13--17, 2016. 508--521.
[61]
Xiaodong Yu and Michela Becchi. 2013. GPU Acceleration Of Regular Expression Matching For Large Datasets: Exploring The Implementation Space. In Computing Frontiers Conference, CF'13, Ischia, Italy, May 14 - 16, 2013, Hubertus Franke, Alexander Heinecke, Krishna V. Palem, and Eli Upfal (Eds.). ACM, 18:1--18:10.
[62]
Sai Zhang and Yuyin Sun. 2013. Automatically Synthesizing SQL Queries From Input-Output Examples. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11--15, 2013. 224--234.
[63]
Zexuan Zhong, Jiaqi Guo, Wei Yang, Jian Peng, Tao Xie, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2018. SemRegex: A Semantics-Based Approach For Generating Regular Expressions From Natural Language Specifications. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. 1608--1618.

Cited By

View all
  • (2024)Structure and design of multimodal dataset for automatic regex synthesis methods in Roman UrduInternational Journal of Data Science and Analytics10.1007/s41060-024-00612-yOnline publication date: 23-Jul-2024
  • (2024)Automatic regex synthesis methods for english: a comparative analysisKnowledge and Information Systems10.1007/s10115-024-02232-1Online publication date: 3-Oct-2024
  • (2024)Efficient Matching with Memoization for Regexes with Look-around and Atomic GroupingProgramming Languages and Systems10.1007/978-3-031-57267-8_4(90-118)Online publication date: 6-Apr-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering
December 2020
1449 pages
ISBN:9781450367684
DOI:10.1145/3324884
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Anti-ReDoS
  2. program repair
  3. program synthesis
  4. regular expression (regexes)

Qualifiers

  • Research-article

Funding Sources

Conference

ASE '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)184
  • Downloads (Last 6 weeks)28
Reflects downloads up to 11 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Structure and design of multimodal dataset for automatic regex synthesis methods in Roman UrduInternational Journal of Data Science and Analytics10.1007/s41060-024-00612-yOnline publication date: 23-Jul-2024
  • (2024)Automatic regex synthesis methods for english: a comparative analysisKnowledge and Information Systems10.1007/s10115-024-02232-1Online publication date: 3-Oct-2024
  • (2024)Efficient Matching with Memoization for Regexes with Look-around and Atomic GroupingProgramming Languages and Systems10.1007/978-3-031-57267-8_4(90-118)Online publication date: 6-Apr-2024
  • (2023)Repairing Regular Expressions for ExtractionProceedings of the ACM on Programming Languages10.1145/35912877:PLDI(1633-1656)Online publication date: 6-Jun-2023
  • (2023)Search-Based Regular Expression Inference on a GPUProceedings of the ACM on Programming Languages10.1145/35912747:PLDI(1317-1339)Online publication date: 6-Jun-2023
  • (2023)Automated Synthesis of Safe Timing Behaviors for Requirements Models Using CCSLIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems10.1109/TCAD.2023.328541242:12(5127-5140)Online publication date: Dec-2023
  • (2023)Improving Developers’ Understanding of Regex Denial of Service Tools through Anti-Patterns and Fix Strategies2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179442(1238-1255)Online publication date: May-2023
  • (2023)Effective ReDoS Detection by Principled Vulnerability Modeling and Exploit Generation2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179328(2427-2443)Online publication date: May-2023
  • (2023)InfeRE: Step-by-Step Regex Generation via Chain of Inference2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00111(1505-1515)Online publication date: 11-Sep-2023
  • (2023)Sound static analysis of regular expressions for vulnerabilities to denial of service attacksScience of Computer Programming10.1016/j.scico.2023.102960229:COnline publication date: 1-Jul-2023
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media