Benchmarks
Every release, timed
Each release runs the same workloads on the same machine: a one-line run, where boot is the
whole cost, programs that compute, start processes or write memory, GNU hello's full Nix
build, and the commands that replay, seek and explore a run afterwards. Every round measures
every release once, in a fresh REWIND_HOME and a shuffled order, and the charts
draw the median of the rounds with their range.
One release against another
A change is marked when the medians differ by more than 3% and the two releases' ranges do not overlap. Anything inside the noise reads as the same.
Over releases
Short runs and long runs
A short run pays for boot and teardown; a long one pays per step and per keyframe. These are worked out round by round from the samples above, so each has a range of its own.
What moved the numbers
Every record
The medians behind the charts, for the metric chosen above.
How it is measured
tools/bench.py downloads each release's asset, the same tarball people
install, and runs it against inputs fixed by the benchmark rather than by the release:
static busybox, a ring of threads, the coreutils source and GNU hello's derivation,
nix build .#bench-inputs. Every run gets the options below, the only CPU
model every release accepts on a host without ERMS and the clock every release can be told
to use. Each round's commands are pinned to the idlest NUMA node, and GNU time reports
wall, user and system time and peak RSS. The workloads with no Rewind in them run natively
in the same round, for the slowdowns.
To measure a build of your own against the newest release, on your machine:
nix build .#release
nix run .#bench -- run --history /tmp/wip.json --release v0.10.0 \
--rewind wip=result/rewind-x86_64-linux.tar.gz --rounds 3
nix run .#bench -- compare --history /tmp/wip.json v0.10.0 wip