benchmark/test
Roman Lebedev fbc31405b2
Random interleaving of benchmark repetitions - the sequel (fixes #1051) (#1163)
Inspired by the original implementation by Hai Huang @haih-g
from https://github.com/google/benchmark/pull/1105.

The original implementation had design deficiencies that
weren't really addressable without redesign, so it was reverted.

In essence, the original implementation consisted of two separateable parts:
* reducing the amount time each repetition is run for, and symmetrically increasing repetition count
* running the repetitions in random order

While it worked fine for the usual case, it broke down when user would specify repetitions
(it would completely ignore that request), or specified per-repetition min time (while it would
still adjust the repetition count, it would not adjust the per-repetition time,
leading to much greater run times)

Here, like i was originally suggesting in the original review, i'm separating the features,
and only dealing with a single one - running repetitions in random order.

Now that the runs/repetitions are no longer in-order, the tooling may wish to sort the output,
and indeed `compare.py` has been updated to do that: #1168.
2021-06-03 21:16:54 +03:00
..
AssemblyTests.cmake split_list is not defined for assembly tests (#595) 2018-05-14 15:02:49 +01:00
BUILD Introduce "family index" field into JSON output (#1164) 2021-06-02 18:06:45 +03:00
CMakeLists.txt Random interleaving of benchmark repetitions - the sequel (fixes #1051) (#1163) 2021-06-03 21:16:54 +03:00
args_product_test.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
basic_test.cc Use fewer ramp up repetitions when KeepRunningBatch is used (#1113) 2021-04-20 07:16:05 +01:00
benchmark_gtest.cc Fix leak in test, and provide path to remove leak from library (#1169) 2021-06-03 16:08:00 +01:00
benchmark_name_gtest.cc BENCHMARK_CAPTURE() and Complexity() - naming problem (#761) 2019-03-17 16:38:51 +03:00
benchmark_random_interleaving_gtest.cc Random interleaving of benchmark repetitions - the sequel (fixes #1051) (#1163) 2021-06-03 21:16:54 +03:00
benchmark_test.cc Allow AddRange to work with int64_t. (#548) 2018-04-03 23:12:47 +01:00
clobber_memory_assembly_test.cc Add tests to verify assembler output -- Fix DoNotOptimize. (#530) 2018-03-23 16:10:47 -06:00
commandlineflags_gtest.cc Add `benchmark_context` flag that allows per-run custom context. (#1127) 2021-05-04 14:36:11 +01:00
complexity_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00
cxx03_test.cc Iteration counts should be `uint64_t` globally. (#817) 2019-05-13 12:33:11 +03:00
diagnostics_test.cc Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) 2017-10-17 12:17:02 -06:00
display_aggregates_only_test.cc Track two more details about runs - the aggregate name, and run name. (#675) 2018-09-13 15:08:15 +03:00
donotoptimize_assembly_test.cc Add tests to verify assembler output -- Fix DoNotOptimize. (#530) 2018-03-23 16:10:47 -06:00
donotoptimize_test.cc Fix uninitialized warning (#560) 2018-03-25 20:05:31 +01:00
filter_test.cc Introduce "family index" field into JSON output (#1164) 2021-06-02 18:06:45 +03:00
fixture_test.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
internal_threading_test.cc Add process_time for better OpenMP and user-managed thread timing 2019-04-09 13:01:33 +01:00
link_main_test.cc Add benchmark_main target. (#601) 2018-05-25 11:18:58 +01:00
map_test.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
memory_manager_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00
multiple_ranges_test.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
options_test.cc Add 'seconds' time unit (#1076) 2020-12-21 20:15:58 +03:00
output_test.h Clean -Wreserved-identifier instances (#1143) 2021-05-06 20:31:14 +01:00
output_test_helper.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
perf_counters_gtest.cc [PFM] Extend perf counter support to multi-threaded cases. (#1153) 2021-05-19 09:49:05 +01:00
perf_counters_test.cc Support optional, user-directed collection of performance counters (#1114) 2021-04-28 09:25:29 +01:00
register_benchmark_test.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
repetitions_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00
report_aggregates_only_test.cc Track two more details about runs - the aggregate name, and run name. (#675) 2018-09-13 15:08:15 +03:00
reporter_output_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00
skip_with_error_test.cc Support -Wsuggest-override (#1059) 2021-05-11 14:56:00 +03:00
state_assembly_test.cc Iteration counts should be `uint64_t` globally. (#817) 2019-05-13 12:33:11 +03:00
statistics_gtest.cc Fix type conversion warnings. (#951) 2020-04-06 13:52:09 +01:00
string_util_gtest.cc Fix perf counter argument parsing (#1160) 2021-06-01 15:50:42 +01:00
templated_fixture_test.cc (clang-)format all the things (#610) 2018-06-01 11:14:19 +01:00
user_counters_tabular_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00
user_counters_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00
user_counters_thousands_test.cc Introduce per-family instance index (#1165) 2021-06-02 23:45:41 +03:00