Time inside the VM

The short version

• Intel: nothing to do. Rewind uses counter time. On a hybrid Intel CPU, with performance and efficient cores, it counts on every kind of core and adds the counts up, and the self-test runs on each kind: they must all count the same for runs to use counter time.
• AMD Ryzen or EPYC: run sudo rewind pmu enable once after each boot, or turn on the NixOS setting below. Until then Rewind records with exit time and prints the warning below. Runs still work and still replay exactly. What changes is how the VM's clock moves.
• rewind pmu status says which clock your runs will use.

rewind: recording with exit time: this AMD CPU's branch counter is not exact until rr's workaround is set. ...

What the two clocks do

Rewind sees the VM only at the points where the VM stops and hands control to Rewind: a system call that reports a file or a process, a read of the clock, going idle. Each of those points is a step. Steps happen at the same places in every run, so a clock that moves only at steps is the same in every run.

Exit time moves the clock 5 microseconds per step. When nothing is runnable, it jumps straight to the next timer, so sleep 10 takes ten seconds on the VM's clock and almost none of yours. Computation between steps takes no time at all: a program that computes for a second of real time sees a few microseconds pass.

Counter time also adds the work done since the last step. Your CPU's performance counter counts the conditional branches the VM's programs run, and each one adds a nanosecond. Rewind reads the count only at steps, so it is the same in every run, as long as the counter counts exactly.

Building GNU hello in the VM shows the difference:

With exit time, timing in code that computes is off. A check that some computation finishes within a second always passes, and a timeout that should fire during a long computation never does. Counter time gives such code a clock close to the real one.

With either clock, a thread gives up the CPU only at a step. A loop that spins waiting for another thread, without making a system call, keeps the CPU and the run hangs. Locks, sleeps and I/O all make system calls, so they are fine. Go's garbage collector is the common exception: at GOMAXPROCS above 1, which --cores above 1 sets by default, it spins this way waiting for another thread. Run Go programs with GOGC=off if they fit in the VM's memory.

Why AMD needs a setting

Counter time needs the branch count to be the same on every run. On AMD Zen CPUs it is not, out of the box: a speculation feature around locked instructions makes the counter count a few extra branches, a few in tens of millions. A second of work is billions of branches, so two runs disagree almost at once and a recording would not replay. rr, the record and replay debugger, has the same problem and documents the same fix: set bit 54 of the model-specific register 0xc0011020 on every CPU, which turns that speculation off.

sudo rewind pmu enable makes that change:

$ sudo rewind pmu enable
set the branch counter workaround on 16 CPUs, until reboot

It needs root, because it writes the register through /dev/cpu/*/msr, and it lasts until you reboot. It changes how the CPU speculates for every program on the machine; Rewind has not measured what that costs them. It also leaves a note in /run/rewind/amd-branch-workaround. Reading the register back needs root too, and the note is how Rewind, running as you, knows the setting is on.

Checking

$ rewind pmu status
cpu: AMD Zen (family 25)
amd workaround (MSR 0xc0011020 bit 54): set by `rewind pmu enable` this boot
perf_event_paranoid: 2
probe branches: 3000000 branches, as built
probe locked: 2000000 branches, as built
probe not-branches: 1000000 branches, as built
probe exits: 8192 branches, as built
self-test: 58071879 and 58071879 branches, exact at every step
runs will use counter time

The self-test runs a small program twice in the VM, threads adding to one shared number, which is the pattern the miscount comes from, and compares the branch count at every step. It also runs four probes, loops whose branch counts are known from how they are written, and the counter must read exactly those counts. A counter can agree with itself and still count the wrong branches; a run recorded with it would replay differently on another machine. Rewind uses counter time when the self-test passes and, on AMD, the setting is on. On AMD the self-test alone is not enough: without the setting, its two runs sometimes agree by chance. Rewind remembers the verdict until the next boot, and rewind pmu status runs the test again.

Every boot on NixOS

Use the flake's programs.rewind module:

inputs.rewind.url = "github:fzakaria/rewindvm";

# in your configuration, with inputs.rewind.nixosModules.default imported
programs.rewind.enable = true;
programs.rewind.amdBranchCounterWorkaround = true;

The module loads the msr kernel module and runs rewind pmu enable at boot. It also puts /run/rewind, where rewind pmu enable notes the setting, in Nix's build sandbox, so a derivation that runs Rewind in its build gets counter time too.

Choosing yourself

--clock exits or --clock branches on rewind run, nix or check overrides Rewind's choice. --clock branches refuses where the counter is not exact this boot, rather than record a run that would not replay. A run remembers its clock and replays with it. A run recorded with counter time replays exactly only on a machine whose counter is exact too; one recorded with exit time does not depend on the counter.