Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts

Assoc.-Prof. Rui Abreu (Universität Lissabon) | 19.12.2019 | 10:00 Uhr | S.1.37

Abstract:

In the past decade, research on test-suite-based automatic program repair has grown significantly. Each year, new approaches and implementations are featured in major software engineering venues. However, most of those approaches are evaluated on a single benchmark of bugs, which are also rarely reproduced by other researchers. In this paper, we present a large-scale experiment using 11 Java test-suite-based repair tools and 2,141 bugs from 5 benchmarks. Our goal is to have a better understanding of the current state of automatic program repair tools on a large diversity of benchmarks. Our investigation is guided by the hypothesis that the repairability of repair tools might not be generalized across different benchmarks. We found that the 11 tools 1) are able to generate patches for 21% of the bugs from the 5 benchmarks, and 2) have better performance on Defects4J compared to other benchmarks, by generating patches for 47% of the bugs from Defects4J compared to 10-30% of bugs from the other benchmarks. Our experiment comprises 23,551 repair attempts, which we used to find causes of non-patch generation. These causes are reported in this presentation, which can help repair tool designers to improve their approaches and tools. This work was presented at ESEC/FSE19 and was given an ACM SIGSOFT Distinguished Paper Award.

CV:

Rui Abreu holds a Ph.D. in Computer Science – Software Engineering from TU Delft, The Netherlands, and a M.Sc. in Computer and Systems Engineering from the U.Minho, Portugal. His research revolves around software quality, with emphasis on automating the testing and debugging phases of the software development life-cycle as well as self-adaptation. He is the recipient of 7 Best Paper Awards (including a 2019 FSE  Distinguished Paper Award). Before joining IST, U.Lisbon as an Associate Professor (with habilitation), he was a member of the Model-Based Reasoning group at PARC’s System and Sciences Laboratory. He has co-founded DashDash in January 2017, a platform to create web apps using only spreadsheet skills. Currently, he is enjoying a sabbatical leave as a Visiting Scientist at Google NY’s. He is also passionate about soccer and a FC Porto fan.