benchmark/docs/random_interleaving.md

<a name="interleaving" />

# Random Interleaving

[Random Interleaving](https://github.com/google/benchmark/issues/1051) is a
technique to lower run-to-run variance. It randomly interleaves repetitions of a
microbenchmark with repetitions from other microbenchmarks in the same benchmark
test. Data shows it is able to lower run-to-run variance by
[40%](https://github.com/google/benchmark/issues/1051) on average.

To use, you mainly need to set `--benchmark_enable_random_interleaving=true`,
and optionally specify non-zero repetition count `--benchmark_repetitions=9`
and optionally decrease the per-repetition time `--benchmark_min_time=0.1`.
Random interleaving of benchmark repetitions - the sequel (fixes #1051) (#1163) Inspired by the original implementation by Hai Huang @haih-g from https://github.com/google/benchmark/pull/1105. The original implementation had design deficiencies that weren't really addressable without redesign, so it was reverted. In essence, the original implementation consisted of two separateable parts: * reducing the amount time each repetition is run for, and symmetrically increasing repetition count * running the repetitions in random order While it worked fine for the usual case, it broke down when user would specify repetitions (it would completely ignore that request), or specified per-repetition min time (while it would still adjust the repetition count, it would not adjust the per-repetition time, leading to much greater run times) Here, like i was originally suggesting in the original review, i'm separating the features, and only dealing with a single one - running repetitions in random order. Now that the runs/repetitions are no longer in-order, the tooling may wish to sort the output, and indeed `compare.py` has been updated to do that: #1168. 2021-06-03 18:16:54 +00:00			`<a name="interleaving" />`

			`# Random Interleaving`

			`[Random Interleaving](https://github.com/google/benchmark/issues/1051) is a`
			`technique to lower run-to-run variance. It randomly interleaves repetitions of a`
			`microbenchmark with repetitions from other microbenchmarks in the same benchmark`
			`test. Data shows it is able to lower run-to-run variance by`
			`[40%](https://github.com/google/benchmark/issues/1051) on average.`

			To use, you mainly need to set `--benchmark_enable_random_interleaving=true`,
			and optionally specify non-zero repetition count `--benchmark_repetitions=9`
			and optionally decrease the per-repetition time `--benchmark_min_time=0.1`.