Skeleton program enumeration (SPE) is a novel framework for systematic, rigorous compiler testing. We view a program as a syntactic structure P (syntactic skeleton) parameterized by a collection of identifiers V (variable names). Given a syntactic skeleton P and a set of variables V, the goal of SPE is to enumerate a set of programs P exhibiting all possible variable usage patterns within P. We can then use the enumerated programs in P to stress-test compilers.
The key insights behind SPE are: (1) Programs with different variable usage patterns exhibit diverse control- and data-dependence, and help exploit different compiler optimizations; (2) most real compiler bugs were revealed by small tests — this "small-scope" observation opens up SPE for practical compiler validation; and (3) SPE is exhaustive w.r.t. a given syntactic skeleton and variable set, offering a level of guarantee absent from all existing compiler testing techniques.
We continue our team efforts working with the open-source community on various compiler projects. The following table gives an overview of the bugs found by SPE. This list is not exhaustive as it does not include some of the GCC/LLVM bugs reported by Chengnian and Zhendong (link1, link2, link3, link4) using SPE.