Generating Effective Test Suites by Combining Coverage Criteria

Friday, September 22, 2017 - 2:20pm to 3:10pm
Swearingen room 2A14

I would like to invite you to attend this week's CSCE 791 seminar. These seminars highlight research being performed in our department and across the world. All CSCE 791 seminars are open to anybody who wishes to attend - not just students registered for the course.

Speaker: Dr. Gregory Gay, University of South Carolina

Abstract: A number of criteria have been proposed to judge test suite adequacy. While search-based test generation has improved greatly at criteria coverage, the produced suites are still often ineffective at detecting faults. Efficacy may be limited by the single-minded application of one criterion at a time when generating suites - a sharp contrast to human testers, who simultaneously explore multiple testing strategies. We hypothesize that automated generation can be improved by selecting and simultaneously exploring multiple criteria. To address this hypothesis, we have generated multi-criteria test suites, measuring efficacy against the Defects4J fault database. We have found that multi-criteria suites can be up to 31.15% more effective at detecting complex, real-world faults than suites generated to satisfy a single criterion and 70.17% more effective than the default combination of all eight criteria. Given a fixed search budget, we recommend pairing a criterion focused on structural exploration - such as Branch Coverage - with targeted supplemental strategies aimed at the type of faults expected from the system under test. Our findings offer lessons to consider when selecting such combinations.

Bio: Gregory Gay is an assistant professor of Computer Science & Engineering at University of South Carolina. His research interests include automated testing and analysis, and search-based software engineering, with a focus is on the use of coverage criteria in automated test case generation, as well as the construction of effective test oracles for real-time and safety critical systems. He serves on the steering committees for the Symposium on Search-Based Software Engineering and the International Workshop on Search-Based Software Testing, as well as the organizing and program committees of a variety of conferences and workshops. He graduated with a PhD from the University of Minnesota under a NSF Graduate Research Fellowship, working with the Critical Systems research group. He received his BS and MS in Computer Science from West Virginia University. Additionally, he has previously worked at NASA's Ames Research Center and Independent Verification & Validation Center, and served as a visiting academic at the Laboratory for Internet Software Technologies at the Chinese Academy of Sciences in Beijing.