3 Data and Analysis

3.1 Data

The data we present in the paper is in paper/data/. The .rktd files contained therein are Racket data files in a simple S-expression format. Each file contains a vector of lists of runtimes that can easily be read into Racket. Each list in the vector contains all the runtimes (in milliseconds) for a specific typed/untyped configuration. The ordering is lexicographic in the typed/untyped bitstring which is formatted as follows. For a program with n modules, each configuration is assigned a length n bitstring where the ith bit corresponds to the ith module alphabetically. 1 denotes typed and 0 denotes untyped.

As an example, a program that has three modules: a.rkt, b.rkt and c.rkt will produce 2^3=8 configurations and therefore the .rktd will have 8 lists of runtimes. The bitstrings will be 3 bits longs, ordered from 000, 001, 010, etc. to 111 in the data file. Then the 3rd runtime corresponds to the bitstring 011 which means modules b.rkt and c.rkt were using their typed versions and a.rkt was using its untyped version.

3.2 Analysis

The subdirectory paper/scripts/ contains files that may be useful for parsing the data and performing new analyses. The Racket files therein document their exports fairly well.

The tools/ directory contains more high-level scripts.

tools/view.rkt will produce the LNM-plots (Figure 4 in the paper) for any number of benchmarks given a sequence of .rktd files produced by tools/run.rkt. Here is an example invocation (assuming the working directory is ~/Desktop):

racket tools/view.rkt paper/data/kcfa-2015-06-25T13:48:52.rktd

It outputs the plot as output.png to the benchmarks directory.

tools/data-lattice.rkt will produce a picture summarizing the entire runtime lattice from a benchmark .rktd file, as in Figure 3 in the paper. For example,

racket tools/data-lattice.rkt paper/data/kcfa-2015-06-25T13:48:52.rktd

The black and white ovals correspond to the bitstrings explained in Running Benchmarks, where black denotes typed and white untyped. Each number is the ratio between the runtime of the configuration and the wholly untyped configuration.

We suggest using this last visualization tool only on the smaller benchmarks (up to 7 modules). Larger lattices do not fit on most screens and may fail to load.