Skeletal Program Enumeration for Rigorous Compiler Testing

Qirun Zhang, Chengnian Sun and Zhendong Su

Introduction

Skeleton program enumeration (SPE) is a novel framework for systematic, rigorous compiler testing. We view a program as a syntactic structure P (syntactic skeleton) parameterized by a collection of identifiers V (variable names). Given a syntactic skeleton P and a set of variables V, the goal of SPE is to enumerate a set of programs P exhibiting all possible variable usage patterns within P. We can then use the enumerated programs in P to stress-test compilers.

The key insights behind SPE are: (1) Programs with different variable usage patterns exhibit diverse control- and data-dependence, and help exploit different compiler optimizations; (2) most real compiler bugs were revealed by small tests — this "small-scope" observation opens up SPE for practical compiler validation; and (3) SPE is exhaustive w.r.t. a given syntactic skeleton and variable set, offering a level of guarantee absent from all existing compiler testing techniques.

Reported Bugs

We continue our team efforts working with the open-source community on various compiler projects. The following table gives an overview of the bugs found by SPE. This list is not exhaustive as it does not include some of the GCC/LLVM bugs reported by Chengnian and Zhendong (link1, link2, link3, link4) using SPE.


Compiler Bug Counts
GCC 87
LLVM 96
Scala 15
Dotty 53
Rust 13
Rust Bindgen 11
CompCert 30


Last Updated: 2017-11.