5 Walkthrough: Creating and Analyzing a Benchmark Program

To bring it all together we provide a walkthrough of creating a benchmark program, running all configurations and analyzing the data.

5.1 The Program: Echo

First we make an untyped program and add types to it. The program is intentionally small and over-modularized for demonstration purposes. The program is a simple echo server with a client to send it some data.

In the benchmark/ directory, create a echo/ directory with untyped/ and typed/ subdirectories.

5.1.1 Untyped

We start with the untyped/ files. A file constants.rkt containing global constants:

#lang racket/base

(provide

; Natural number port number to run the echo server on

PORT

; String message to send over the tcp connection

DATA)

(define PORT 8887)

(define DATA "Hello there sailor\n")

server.rkt which contains the actual echo server:

#lang racket/base

; TCP server: read from a buffer until end of file.

(provide server)

(require "constants.rkt"

(only-in racket/tcp tcp-accept tcp-listen))

; ---------------------------------------------------------------------------------------------------

(define (server)

(define-values (in out) (tcp-accept (tcp-listen PORT 5 #t)))

(define buffer (make-string (string-length DATA)))

(file-stream-buffer-mode out 'none)

(let loop ([i (read-string! buffer in)]

[bytes 0])

(cond [(not (eof-object? i))

(display buffer out)

(loop (read-string! buffer in)

(+ bytes (string-length buffer)))]

[else (printf "server processed ~a bytes\n" bytes)])))

and finally main.rkt, which hooks the client up to the server and runs for a while. Importantly, the file includes a usage of racket’s time form, which prints out the time that the block inside it takes to execute. This is what the benchmarking script will parse as the runtime.

#lang racket/base

(require (only-in "client.rkt" client)

(only-in "server.rkt" server))

; ---------------------------------------------------------------------------------------------------

(define (main arg)

(thread (lambda () (client arg)))

(server))

(time (main 200000))

5.1.2 Typed

Fortunately, it is not difficult to add types to the code above. Put all of the following files in the benchmarks/echo/typed/ directory. The only changes we need to make are to use typed racket and add a few annotations.

5.2 Running the Benchmark

As before we can use ./run.sh benchmarks/echo to run our benchmark. The raw data is now in benchmarks/echo.rktd, if you read it you see the numbers are basically all the same, so there’s very little overhead from typing.

You can see the LNM graph at benchmarks/echo.png. You’ll just see a blue line at the top since all configurations work here.

For another analysis, we can run racket tools/data-lattice.rkt benchmarks/echo.rktd, which will show that no configurations have high overhead (all the numbers should be close to 1).