Benchmarking Common Larceny

In my paper and presentation at ILC 2005, I reported the performance of the 6 June 2005 alpha version of Common Larceny relative to four other implementations of Scheme:

• Petit Larceny with Lars Hansen's native code generator for the Intel Pentium
• Petit Larceny with its default code generator, which generates portable C code
• MzScheme (PLT Scheme with debugging features turned off)
• DrScheme (PLT Scheme with debugging features turned on)

Hardware and Software

Common Larceny, MzScheme, and DrScheme were benchmarked on an IBM ThinkPad laptop computer that contains a 2.8 GHz Pentium processor and 1 gigabyte of RAM, running Windows XP. Petit Larceny was run on a Linux tower that also contained a 2.8 GHz processor and 1 gigabyte RAM. This Linux machine ran MzScheme somewhat faster than the Windows laptop, so we adjusted all of the Linux timings by multiplying them by a factor computed from an average of the MzScheme timings on both machines.

The Common Larceny benchmarks were compiled to IL under CLR version 1.1.4322.2032. This IL was then JIT-compiled and benchmarked under CLR version 1.1.4322.2032 and beta version 2.0.40607.16. The version of Common Larceny was 1.0.1987.20166. We benchmarked MzScheme and DrScheme v209, and Petit Larceny version 0.53 "Dear John" with the Gnu C compiler.

MzScheme and DrScheme were benchmarked in their default mode. The Petit and Common Larceny benchmarks were compiled and executed in fast-safe mode, which is the default for Petit Larceny. The compilation process for Common Larceny was rather complicated when we ran these benchmarks, but was simplified in v0.92.

We note that, on the 2.8 GHz Linux machine, DrScheme ran the tak benchmark about ten times as fast as a Symbolics 3600, and was about as fast as a Cray-1. The other implementations are considerably faster than DrScheme.

Benchmarks and Results

We ran each benchmark three to nine times on each system and reported the median elapsed time, including gc time. (Several of these benchmarks run too fast on modern machines, and ought to be rescaled. The graphs benchmark is big enough with its default parameter of 7, but I made the mistake of using 5 instead.) All benchmarks were timed by the run-benchmark procedure, which is predefined in Petit and Common Larceny but must be loaded into MzScheme and DrScheme.

(tak-benchmark) ;(10 iterations)	relative performance
(sort-benchmark sort1 1000000)	relative performance
(nboyer-benchmark 4)	relative performance
(sboyer-benchmark 5)	relative performance
(conform-benchmark)	relative performance
(graphs-benchmark 5)	relative performance
(nfa-benchmark)	relative performance
(nucleic2-benchmark)	relative performance

We also offer these results in numerical form.

We thank Microsoft for its permission to publish these benchmark results.

Last updated 13 November 2006.