dolysis/benchmark
3
0
Fork 0
mirror of https://github.com/google/benchmark.git synced 2024-11-28 15:34:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
benchmark/docs/random_interleaving.md
Roman Lebedev fbc31405b2
Random interleaving of benchmark repetitions - the sequel (fixes #1051) (#1163) 
Inspired by the original implementation by Hai Huang @haih-g
from https://github.com/google/benchmark/pull/1105.

The original implementation had design deficiencies that
weren't really addressable without redesign, so it was reverted.

In essence, the original implementation consisted of two separateable parts:
* reducing the amount time each repetition is run for, and symmetrically increasing repetition count
* running the repetitions in random order

While it worked fine for the usual case, it broke down when user would specify repetitions
(it would completely ignore that request), or specified per-repetition min time (while it would
still adjust the repetition count, it would not adjust the per-repetition time,
leading to much greater run times)

Here, like i was originally suggesting in the original review, i'm separating the features,
and only dealing with a single one - running repetitions in random order.

Now that the runs/repetitions are no longer in-order, the tooling may wish to sort the output,
and indeed `compare.py` has been updated to do that: #1168.
2021-06-03 21:16:54 +03:00

647 B
Raw Blame History

Random Interleaving

Random Interleaving is a technique to lower run-to-run variance. It randomly interleaves repetitions of a microbenchmark with repetitions from other microbenchmarks in the same benchmark test. Data shows it is able to lower run-to-run variance by 40% on average.

To use, you mainly need to set --benchmark_enable_random_interleaving=true, and optionally specify non-zero repetition count --benchmark_repetitions=9 and optionally decrease the per-repetition time --benchmark_min_time=0.1.