dolysis/benchmark
3
0
Fork 0
mirror of https://github.com/google/benchmark.git synced 2024-11-28 15:34:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
benchmark/test/reporter_output_test.cc

749 lines
37 KiB
C++
Raw Normal View History

Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
#undef NDEBUG
#include <utility>
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
#include "benchmark/benchmark.h"
#include "output_test.h"
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
// ========================================================================= //
// ---------------------- Testing Prologue Output -------------------------- //
// ========================================================================= //
(clang-)format all the things (#610) * format all documents according to contributor guidelines and specifications use clang-format on/off to stop formatting when it makes excessively poor decisions * format all tests as well, and mark blocks which change too much
2018-06-01 10:14:19 +00:00
ADD_CASES(TC_ConsoleOut, {{"^[-]+$", MR_Next},
{"^Benchmark %s Time %s CPU %s Iterations$", MR_Next},
{"^[-]+$", MR_Next}});
Improve CPU Cache info reporting -- Add Windows support. (#486) * Improve CPU Cache info reporting -- Add Windows support. This patch does a couple of thing regarding CPU Cache reporting. First, it adds an implementation on Windows. Second it fixes the JSONReporter to correctly (and actually) output the CPU configuration information. And finally, third, it detects and reports the number of physical CPU's that share the same cache.
2017-11-26 20:33:01 +00:00
static int AddContextCases() {
AddCases(TC_ConsoleErr,
{
use rfc3339-formatted timestamps in output [output format change] (#965) * timestamp: use rfc3339-formatted timestamps in output Replace localized timestamps with machine-readable IETF RFC 3339 format timestamps. This is an attempt to make the output timestamps easily machine-readable. ISO8601 specifies standards for time interchange formats. IETF RFC 3339: https://tools.ietf.org/html/rfc3339 defines a subset of these for use in the internet. The general form for these timestamps is: YYYY-MM-DDTHH:mm:SS[+-]hhmm This replaces the localized time formats that are currently being used in the benchmark output to prioritize interchangeability and machine-readability. This might break existing programs that rely on the particular date-time format. This might also may make times less human readable. RFC3339 was intended to balance human readability and simplicity for machine readability, but it is primarily intended as an internal representation. * timers: remove utc string formatting We only ever need local time printing. Remove the UTC printing and cosnolidate the logic slightly. * timers: manually create rfc3339 string The C++ standard library does not output the time offset in RFC3339 format, it is missing the : between hours and minutes. VS does not appear to support timezone information by default. To avoid adding too much complexity to benchmark around timezone handling e.g. a full date library like https://github.com/HowardHinnant/date, we fall back to outputting GMT time with a -00:00 offset for those cases. * timers: use reentrant form for localtime_r & tmtime_r For non-windows, use the reentrant form for the time conversion functions. * timers: cleanup Use strtol instead of brittle moving characters around. * timers: only call strftime twice. Also size buffers to known maximum necessary size and name constants more appropriately. * timers: fix unused variable warning
2020-06-15 16:28:17 +00:00
{"^%int-%int-%intT%int:%int:%int[-+]%int:%int$", MR_Default},
Print the executable name as part of the context. (#534) * Print the executable name as part of the context. A common use case of the library is to run two different versions of a benchmark to compare them. In my experience this often means compiling a benchmark twice, renaming one of the executables, and then running the executables back-to-back. In this case the name of the executable is important contextually information. Unfortunately the benchmark does not report this information. This patch adds the executable name to the context reported by the benchmark. * attempt to fix tests on Windows * attempt to fix tests on Windows
2018-02-21 16:43:57 +00:00
{"Running .*/reporter_output_test(\\.exe)?$", MR_Next},
Make tests pass on 1-core VMs (#653) found while working on reproducible builds for openSUSE To reproduce there osc checkout openSUSE:Factory/benchmark && cd $_ osc build -j1 --vm-type=kvm
2018-08-28 14:10:14 +00:00
{"Run on \\(%int X %float MHz CPU s?\\)", MR_Next},
Improve CPU Cache info reporting -- Add Windows support. (#486) * Improve CPU Cache info reporting -- Add Windows support. This patch does a couple of thing regarding CPU Cache reporting. First, it adds an implementation on Windows. Second it fixes the JSONReporter to correctly (and actually) output the CPU configuration information. And finally, third, it detects and reports the number of physical CPU's that share the same cache.
2017-11-26 20:33:01 +00:00
});
(clang-)format all the things (#610) * format all documents according to contributor guidelines and specifications use clang-format on/off to stop formatting when it makes excessively poor decisions * format all tests as well, and mark blocks which change too much
2018-06-01 10:14:19 +00:00
AddCases(TC_JSONOut,
{{"^\\{", MR_Default},
{"\"context\":", MR_Next},
{"\"date\": \"", MR_Next},
#722 Adding Host Name in Reporting (#733) * Adding Host Name and test * Addressing Review Comments * Adding Test for JSON Reporter * Adding HOST_NAME_MAX for MacOS systems * Adding Explaination for MacOS HOST_NAME_MAX Addition * Addressing Peer Review Comments * Adding codecvt in windows header guard * Changing name SystemInfo and adding empty message incase host name fetch fails * Adding Comment on Struct SystemInfo
2018-12-11 11:23:02 +00:00
{"\"host_name\":", MR_Next},
properly escape json names (#652)
2018-08-16 16:47:09 +00:00
{"\"executable\": \".*(/|\\\\)reporter_output_test(\\.exe)?\",",
MR_Next},
(clang-)format all the things (#610) * format all documents according to contributor guidelines and specifications use clang-format on/off to stop formatting when it makes excessively poor decisions * format all tests as well, and mark blocks which change too much
2018-06-01 10:14:19 +00:00
{"\"num_cpus\": %int,$", MR_Next},
{"\"mhz_per_cpu\": %float,$", MR_Next},
{"\"cpu_scaling_enabled\": ", MR_Next},
{"\"caches\": \\[$", MR_Next}});
Include system load average in console and JSON reports High system load can skew benchmark results. By including system load averages in the library's output, we help users identify a potential issue in the quality of their measurements, and thus assist them in producing better (more reproducible) results. I got the idea for this from Brendan Gregg's checklist for benchmark accuracy (http://www.brendangregg.com/blog/2018-06-30/benchmarking-checklist.html).
2018-07-09 04:17:44 +00:00
auto const& Info = benchmark::CPUInfo::Get();
auto const& Caches = Info.caches;
Improve CPU Cache info reporting -- Add Windows support. (#486) * Improve CPU Cache info reporting -- Add Windows support. This patch does a couple of thing regarding CPU Cache reporting. First, it adds an implementation on Windows. Second it fixes the JSONReporter to correctly (and actually) output the CPU configuration information. And finally, third, it detects and reports the number of physical CPU's that share the same cache.
2017-11-26 20:33:01 +00:00
if (!Caches.empty()) {
AddCases(TC_ConsoleErr, {{"CPU Caches:$", MR_Next}});
}
for (size_t I = 0; I < Caches.size(); ++I) {
std::string num_caches_str =
Caches[I].num_sharing != 0 ? " \\(x%int\\)$" : "$";
CPU caches are binary units, not SI. (#911) As disscussed in https://github.com/google/benchmark/issues/899, it is all but certain that the multiplier should be 1024, not 1000. Fixes https://github.com/google/benchmark/issues/899
2019-12-02 09:29:16 +00:00
AddCases(TC_ConsoleErr,
{{"L%int (Data|Instruction|Unified) %int KiB" + num_caches_str,
MR_Next}});
Improve CPU Cache info reporting -- Add Windows support. (#486) * Improve CPU Cache info reporting -- Add Windows support. This patch does a couple of thing regarding CPU Cache reporting. First, it adds an implementation on Windows. Second it fixes the JSONReporter to correctly (and actually) output the CPU configuration information. And finally, third, it detects and reports the number of physical CPU's that share the same cache.
2017-11-26 20:33:01 +00:00
AddCases(TC_JSONOut, {{"\\{$", MR_Next},
{"\"type\": \"", MR_Next},
{"\"level\": %int,$", MR_Next},
{"\"size\": %int,$", MR_Next},
{"\"num_sharing\": %int$", MR_Next},
{"}[,]{0,1}$", MR_Next}});
}
AddCases(TC_JSONOut, {{"],$"}});
Include system load average in console and JSON reports High system load can skew benchmark results. By including system load averages in the library's output, we help users identify a potential issue in the quality of their measurements, and thus assist them in producing better (more reproducible) results. I got the idea for this from Brendan Gregg's checklist for benchmark accuracy (http://www.brendangregg.com/blog/2018-06-30/benchmarking-checklist.html).
2018-07-09 04:17:44 +00:00
auto const& LoadAvg = Info.load_avg;
if (!LoadAvg.empty()) {
AddCases(TC_ConsoleErr,
{{"Load Average: (%float, ){0,2}%float$", MR_Next}});
}
AddCases(TC_JSONOut, {{"\"load_avg\": \\[(%float,?){0,3}],$", MR_Next}});
Improve CPU Cache info reporting -- Add Windows support. (#486) * Improve CPU Cache info reporting -- Add Windows support. This patch does a couple of thing regarding CPU Cache reporting. First, it adds an implementation on Windows. Second it fixes the JSONReporter to correctly (and actually) output the CPU configuration information. And finally, third, it detects and reports the number of physical CPU's that share the same cache.
2017-11-26 20:33:01 +00:00
return 0;
}
int dummy_register = AddContextCases();
Reporter tests: reuse csv header.
2017-04-27 18:24:06 +00:00
ADD_CASES(TC_CSVOut, {{"%csv_header"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
// ========================================================================= //
// ------------------------ Testing Basic Output --------------------------- //
// ========================================================================= //
void BM_basic(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
}
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
}
BENCHMARK(BM_basic);
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_basic %console_report$"}});
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_basic\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_basic\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
reporter_output_test: json: iterations is int, not float (#431) May be relevant for flakiness of win builds Noted by @KindDragon
2017-08-01 01:04:02 +00:00
{"\"iterations\": %int,$", MR_Next},
Json reporter: don't cast floating-point to int; adjust tooling (#426) * Json reporter: passthrough fp, don't cast it to int; adjust tooling Json output format is generally meant for further processing using some automated tools. Thus, it makes sense not to intentionally limit the precision of the values contained in the report. As it can be seen, FormatKV() for doubles, used %.2f format, which was meant to preserve at least some of the precision. However, before that function is ever called, the doubles were already cast to the integer via RoundDouble()... This is also the case for console reporter, where it makes sense because the screen space is limited, and this reporter, however the CSV reporter does output some( decimal digits. Thus i can only conclude that the loss of the precision was not really considered, so i have decided to adjust the code of the json reporter to output the full fp precision. There can be several reasons why that is the right thing to do, the bigger the time_unit used, the greater the precision loss, so i'd say any sort of further processing (like e.g. tools/compare_bench.py does) is best done on the values with most precision. Also, that cast skewed the data away from zero, which i think may or may not result in false- positives/negatives in the output of tools/compare_bench.py * Json reporter: FormatKV(double): address review note * tools/gbench/report.py: skip benchmarks with different time units While it may be useful to teach it to operate on the measurements with different time units, which is now possible since floats are stored, and not the integers, but for now at least doing such a sanity-checking is better than providing misinformation.
2017-07-24 23:13:55 +00:00
{"\"real_time\": %float,$", MR_Next},
{"\"cpu_time\": %float,$", MR_Next},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"\"time_unit\": \"ns\"$", MR_Next},
{"}", MR_Next}});
ADD_CASES(TC_CSVOut, {{"^\"BM_basic\",%csv_report$"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
// ========================================================================= //
// ------------------------ Testing Bytes per Second Output ---------------- //
// ========================================================================= //
void BM_bytes_per_second(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Bugfix/wsl selftest fixes. Fixes #839 (#843) * Update AUTHORS and CONTRIBUTORS * Fix WSL self-test failures Some of the benchmark self-tests expect and check for a particular output format from the benchmark library. The numerical values must not be infinity or not-a-number, or the test will report an error. Some of the values are computed bytes-per-second or items-per-second values, so these require that the measured CPU time for the test to be non-zero. But the loop that is being measured was empty, so the measured CPU time for the loop was extremely small. On systems like Windows Subsystem for Linux (WSL) the timer doesn't have enough resolution to measure this, so the measured CPU time was zero. This fix just makes sure that these tests have something within the timing loop, so that the benchmark library will not decide that the loop takes zero CPU time. This makes these tests more robust, and in particular makes them pass on WSL.
2019-07-27 16:02:31 +00:00
// This test requires a non-zero CPU time to avoid divide-by-zero
benchmark::DoNotOptimize(state.iterations());
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
}
state.SetBytesProcessed(1);
}
BENCHMARK(BM_bytes_per_second);
Un-deprecate [SG]et{Item,Byte}sProcessed, re-implement as custom counters. (#676) As discussed with @dominichamon and @dbabokin, sugar is nice. Well, maybe not for the health, but it's sweet. Alright, enough puns. A special care needs to be applied not to break csv reporter. UGH. We end up shedding some code over this. We no longer specially pretty-print them, they are printed just like the rest of custom counters. Fixes #627.
2018-09-13 19:03:47 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_bytes_per_second %console_report "
"bytes_per_second=%float[kM]{0,1}/s$"}});
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_bytes_per_second\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_bytes_per_second\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
reporter_output_test: json: iterations is int, not float (#431) May be relevant for flakiness of win builds Noted by @KindDragon
2017-08-01 01:04:02 +00:00
{"\"iterations\": %int,$", MR_Next},
Json reporter: don't cast floating-point to int; adjust tooling (#426) * Json reporter: passthrough fp, don't cast it to int; adjust tooling Json output format is generally meant for further processing using some automated tools. Thus, it makes sense not to intentionally limit the precision of the values contained in the report. As it can be seen, FormatKV() for doubles, used %.2f format, which was meant to preserve at least some of the precision. However, before that function is ever called, the doubles were already cast to the integer via RoundDouble()... This is also the case for console reporter, where it makes sense because the screen space is limited, and this reporter, however the CSV reporter does output some( decimal digits. Thus i can only conclude that the loss of the precision was not really considered, so i have decided to adjust the code of the json reporter to output the full fp precision. There can be several reasons why that is the right thing to do, the bigger the time_unit used, the greater the precision loss, so i'd say any sort of further processing (like e.g. tools/compare_bench.py does) is best done on the values with most precision. Also, that cast skewed the data away from zero, which i think may or may not result in false- positives/negatives in the output of tools/compare_bench.py * Json reporter: FormatKV(double): address review note * tools/gbench/report.py: skip benchmarks with different time units While it may be useful to teach it to operate on the measurements with different time units, which is now possible since floats are stored, and not the integers, but for now at least doing such a sanity-checking is better than providing misinformation.
2017-07-24 23:13:55 +00:00
{"\"real_time\": %float,$", MR_Next},
{"\"cpu_time\": %float,$", MR_Next},
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
{"\"time_unit\": \"ns\",$", MR_Next},
Json reporter: don't cast floating-point to int; adjust tooling (#426) * Json reporter: passthrough fp, don't cast it to int; adjust tooling Json output format is generally meant for further processing using some automated tools. Thus, it makes sense not to intentionally limit the precision of the values contained in the report. As it can be seen, FormatKV() for doubles, used %.2f format, which was meant to preserve at least some of the precision. However, before that function is ever called, the doubles were already cast to the integer via RoundDouble()... This is also the case for console reporter, where it makes sense because the screen space is limited, and this reporter, however the CSV reporter does output some( decimal digits. Thus i can only conclude that the loss of the precision was not really considered, so i have decided to adjust the code of the json reporter to output the full fp precision. There can be several reasons why that is the right thing to do, the bigger the time_unit used, the greater the precision loss, so i'd say any sort of further processing (like e.g. tools/compare_bench.py does) is best done on the values with most precision. Also, that cast skewed the data away from zero, which i think may or may not result in false- positives/negatives in the output of tools/compare_bench.py * Json reporter: FormatKV(double): address review note * tools/gbench/report.py: skip benchmarks with different time units While it may be useful to teach it to operate on the measurements with different time units, which is now possible since floats are stored, and not the integers, but for now at least doing such a sanity-checking is better than providing misinformation.
2017-07-24 23:13:55 +00:00
{"\"bytes_per_second\": %float$", MR_Next},
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
{"}", MR_Next}});
ADD_CASES(TC_CSVOut, {{"^\"BM_bytes_per_second\",%csv_bytes_report$"}});
// ========================================================================= //
// ------------------------ Testing Items per Second Output ---------------- //
// ========================================================================= //
void BM_items_per_second(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Bugfix/wsl selftest fixes. Fixes #839 (#843) * Update AUTHORS and CONTRIBUTORS * Fix WSL self-test failures Some of the benchmark self-tests expect and check for a particular output format from the benchmark library. The numerical values must not be infinity or not-a-number, or the test will report an error. Some of the values are computed bytes-per-second or items-per-second values, so these require that the measured CPU time for the test to be non-zero. But the loop that is being measured was empty, so the measured CPU time for the loop was extremely small. On systems like Windows Subsystem for Linux (WSL) the timer doesn't have enough resolution to measure this, so the measured CPU time was zero. This fix just makes sure that these tests have something within the timing loop, so that the benchmark library will not decide that the loop takes zero CPU time. This makes these tests more robust, and in particular makes them pass on WSL.
2019-07-27 16:02:31 +00:00
// This test requires a non-zero CPU time to avoid divide-by-zero
benchmark::DoNotOptimize(state.iterations());
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
}
state.SetItemsProcessed(1);
}
BENCHMARK(BM_items_per_second);
Un-deprecate [SG]et{Item,Byte}sProcessed, re-implement as custom counters. (#676) As discussed with @dominichamon and @dbabokin, sugar is nice. Well, maybe not for the health, but it's sweet. Alright, enough puns. A special care needs to be applied not to break csv reporter. UGH. We end up shedding some code over this. We no longer specially pretty-print them, they are printed just like the rest of custom counters. Fixes #627.
2018-09-13 19:03:47 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_items_per_second %console_report "
"items_per_second=%float[kM]{0,1}/s$"}});
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_items_per_second\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_items_per_second\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
reporter_output_test: json: iterations is int, not float (#431) May be relevant for flakiness of win builds Noted by @KindDragon
2017-08-01 01:04:02 +00:00
{"\"iterations\": %int,$", MR_Next},
Json reporter: don't cast floating-point to int; adjust tooling (#426) * Json reporter: passthrough fp, don't cast it to int; adjust tooling Json output format is generally meant for further processing using some automated tools. Thus, it makes sense not to intentionally limit the precision of the values contained in the report. As it can be seen, FormatKV() for doubles, used %.2f format, which was meant to preserve at least some of the precision. However, before that function is ever called, the doubles were already cast to the integer via RoundDouble()... This is also the case for console reporter, where it makes sense because the screen space is limited, and this reporter, however the CSV reporter does output some( decimal digits. Thus i can only conclude that the loss of the precision was not really considered, so i have decided to adjust the code of the json reporter to output the full fp precision. There can be several reasons why that is the right thing to do, the bigger the time_unit used, the greater the precision loss, so i'd say any sort of further processing (like e.g. tools/compare_bench.py does) is best done on the values with most precision. Also, that cast skewed the data away from zero, which i think may or may not result in false- positives/negatives in the output of tools/compare_bench.py * Json reporter: FormatKV(double): address review note * tools/gbench/report.py: skip benchmarks with different time units While it may be useful to teach it to operate on the measurements with different time units, which is now possible since floats are stored, and not the integers, but for now at least doing such a sanity-checking is better than providing misinformation.
2017-07-24 23:13:55 +00:00
{"\"real_time\": %float,$", MR_Next},
{"\"cpu_time\": %float,$", MR_Next},
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
{"\"time_unit\": \"ns\",$", MR_Next},
Json reporter: don't cast floating-point to int; adjust tooling (#426) * Json reporter: passthrough fp, don't cast it to int; adjust tooling Json output format is generally meant for further processing using some automated tools. Thus, it makes sense not to intentionally limit the precision of the values contained in the report. As it can be seen, FormatKV() for doubles, used %.2f format, which was meant to preserve at least some of the precision. However, before that function is ever called, the doubles were already cast to the integer via RoundDouble()... This is also the case for console reporter, where it makes sense because the screen space is limited, and this reporter, however the CSV reporter does output some( decimal digits. Thus i can only conclude that the loss of the precision was not really considered, so i have decided to adjust the code of the json reporter to output the full fp precision. There can be several reasons why that is the right thing to do, the bigger the time_unit used, the greater the precision loss, so i'd say any sort of further processing (like e.g. tools/compare_bench.py does) is best done on the values with most precision. Also, that cast skewed the data away from zero, which i think may or may not result in false- positives/negatives in the output of tools/compare_bench.py * Json reporter: FormatKV(double): address review note * tools/gbench/report.py: skip benchmarks with different time units While it may be useful to teach it to operate on the measurements with different time units, which is now possible since floats are stored, and not the integers, but for now at least doing such a sanity-checking is better than providing misinformation.
2017-07-24 23:13:55 +00:00
{"\"items_per_second\": %float$", MR_Next},
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
{"}", MR_Next}});
ADD_CASES(TC_CSVOut, {{"^\"BM_items_per_second\",%csv_items_report$"}});
// ========================================================================= //
// ------------------------ Testing Label Output --------------------------- //
// ========================================================================= //
void BM_label(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
}
state.SetLabel("some label");
}
BENCHMARK(BM_label);
ADD_CASES(TC_ConsoleOut, {{"^BM_label %console_report some label$"}});
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_label\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_label\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
reporter_output_test: json: iterations is int, not float (#431) May be relevant for flakiness of win builds Noted by @KindDragon
2017-08-01 01:04:02 +00:00
{"\"iterations\": %int,$", MR_Next},
Json reporter: don't cast floating-point to int; adjust tooling (#426) * Json reporter: passthrough fp, don't cast it to int; adjust tooling Json output format is generally meant for further processing using some automated tools. Thus, it makes sense not to intentionally limit the precision of the values contained in the report. As it can be seen, FormatKV() for doubles, used %.2f format, which was meant to preserve at least some of the precision. However, before that function is ever called, the doubles were already cast to the integer via RoundDouble()... This is also the case for console reporter, where it makes sense because the screen space is limited, and this reporter, however the CSV reporter does output some( decimal digits. Thus i can only conclude that the loss of the precision was not really considered, so i have decided to adjust the code of the json reporter to output the full fp precision. There can be several reasons why that is the right thing to do, the bigger the time_unit used, the greater the precision loss, so i'd say any sort of further processing (like e.g. tools/compare_bench.py does) is best done on the values with most precision. Also, that cast skewed the data away from zero, which i think may or may not result in false- positives/negatives in the output of tools/compare_bench.py * Json reporter: FormatKV(double): address review note * tools/gbench/report.py: skip benchmarks with different time units While it may be useful to teach it to operate on the measurements with different time units, which is now possible since floats are stored, and not the integers, but for now at least doing such a sanity-checking is better than providing misinformation.
2017-07-24 23:13:55 +00:00
{"\"real_time\": %float,$", MR_Next},
{"\"cpu_time\": %float,$", MR_Next},
Add tests for reporters (#307) * Test bytes_per_second and items_per_second. * Test SetLabel. * Reformat. * Make State::error_occurred_ private. * Fix tests with floats. * Merge private blocks
2016-10-28 16:13:57 +00:00
{"\"time_unit\": \"ns\",$", MR_Next},
{"\"label\": \"some label\"$", MR_Next},
{"}", MR_Next}});
ADD_CASES(TC_CSVOut, {{"^\"BM_label\",%csv_label_report_begin\"some "
"label\"%csv_label_report_end$"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
// ========================================================================= //
// ------------------------ Testing Error Output --------------------------- //
// ========================================================================= //
void BM_error(benchmark::State& state) {
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
state.SkipWithError("message");
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
}
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
}
BENCHMARK(BM_error);
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_error[ ]+ERROR OCCURRED: 'message'$"}});
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_error\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_error\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"\"error_occurred\": true,$", MR_Next},
{"\"error_message\": \"message\",$", MR_Next}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_error\",,,,,,,,true,\"message\"$"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
// ========================================================================= //
// ------------------------ Testing No Arg Name Output -----------------------
// //
// ========================================================================= //
void BM_no_arg_name(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
}
}
BENCHMARK(BM_no_arg_name)->Arg(3);
ADD_CASES(TC_ConsoleOut, {{"^BM_no_arg_name/3 %console_report$"}});
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_no_arg_name/3\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_no_arg_name/3\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next}});
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_no_arg_name/3\",%csv_report$"}});
// ========================================================================= //
// ------------------------ Testing Arg Name Output ----------------------- //
// ========================================================================= //
void BM_arg_name(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
}
}
Allow calling Args and ArgNames in any order.
2016-10-26 07:36:39 +00:00
BENCHMARK(BM_arg_name)->ArgName("first")->Arg(3);
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_arg_name/first:3 %console_report$"}});
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_arg_name/first:3\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_arg_name/first:3\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next}});
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_arg_name/first:3\",%csv_report$"}});
// ========================================================================= //
// ------------------------ Testing Arg Names Output ----------------------- //
// ========================================================================= //
void BM_arg_names(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
}
}
Handle the case when the argument name is an empty string.
2016-10-25 07:45:35 +00:00
BENCHMARK(BM_arg_names)->Args({2, 5, 4})->ArgNames({"first", "", "third"});
ADD_CASES(TC_ConsoleOut,
{{"^BM_arg_names/first:2/5/third:4 %console_report$"}});
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
ADD_CASES(TC_JSONOut,
{{"\"name\": \"BM_arg_names/first:2/5/third:4\",$"},
{"\"run_name\": \"BM_arg_names/first:2/5/third:4\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next}});
Handle the case when the argument name is an empty string.
2016-10-25 07:45:35 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_arg_names/first:2/5/third:4\",%csv_report$"}});
Add ArgName() and ArgNames() methods to name arguments/ranges.
2016-10-24 07:49:36 +00:00
StrFormat() is a printf-like function, mark it as such, fix fallout. (#727) Fixes #714.
2018-11-27 00:55:05 +00:00
// ========================================================================= //
// ------------------------ Testing Big Args Output ------------------------ //
// ========================================================================= //
void BM_BigArgs(benchmark::State& state) {
for (auto _ : state) {
}
}
BENCHMARK(BM_BigArgs)->RangeMultiplier(2)->Range(1U << 30U, 1U << 31U);
ADD_CASES(TC_ConsoleOut, {{"^BM_BigArgs/1073741824 %console_report$"},
{"^BM_BigArgs/2147483648 %console_report$"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
// ========================================================================= //
// ----------------------- Testing Complexity Output ----------------------- //
// ========================================================================= //
void BM_Complexity_O1(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Bugfix/wsl selftest fixes. Fixes #839 (#843) * Update AUTHORS and CONTRIBUTORS * Fix WSL self-test failures Some of the benchmark self-tests expect and check for a particular output format from the benchmark library. The numerical values must not be infinity or not-a-number, or the test will report an error. Some of the values are computed bytes-per-second or items-per-second values, so these require that the measured CPU time for the test to be non-zero. But the loop that is being measured was empty, so the measured CPU time for the loop was extremely small. On systems like Windows Subsystem for Linux (WSL) the timer doesn't have enough resolution to measure this, so the measured CPU time was zero. This fix just makes sure that these tests have something within the timing loop, so that the benchmark library will not decide that the loop takes zero CPU time. This makes these tests more robust, and in particular makes them pass on WSL.
2019-07-27 16:02:31 +00:00
// This test requires a non-zero CPU time to avoid divide-by-zero
benchmark::DoNotOptimize(state.iterations());
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
}
Support multiple ranges in the benchmark (#257) * Support multiple ranges in the benchmark google-benchmark library allows to provide up to two ranges to the benchmark method (range_x and range_y). However, in many cases it's not sufficient. The patch introduces multi-range features, so user can easily define multiple ranges by passing a vector of integers, and access values through the method range(i). * Remove redundant API Functions State::range_x() and State::range_y() have been removed. They should be replaced by State::range(0) and State::range(1). Functions Benchmark::ArgPair() and Benchmark::RangePair() have been removed. They should be replaced by Benchmark::Args() and Benchmark::Ranges().
2016-08-04 19:30:14 +00:00
state.SetComplexityN(state.range(0));
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
}
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
BENCHMARK(BM_Complexity_O1)->Range(1, 1 << 18)->Complexity(benchmark::o1);
Use correct RE for floating point numbers in reporter_output_test.cc
2016-10-08 03:56:22 +00:00
SET_SUBSTITUTIONS({{"%bigOStr", "[ ]* %float \\([0-9]+\\)"},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"%RMS", "[ ]*[0-9]+ %"}});
ADD_CASES(TC_ConsoleOut, {{"^BM_Complexity_O1_BigO %bigOStr %bigOStr[ ]*$"},
{"^BM_Complexity_O1_RMS %RMS %RMS[ ]*$"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
// ========================================================================= //
// ----------------------- Testing Aggregate Output ------------------------ //
// ========================================================================= //
// Test that non-aggregate data is printed by default
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
void BM_Repeat(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
}
}
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
// need two repetitions min to be able to output any aggregate output
BENCHMARK(BM_Repeat)->Repetitions(2);
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(TC_ConsoleOut,
{{"^BM_Repeat/repeats:2 %console_report$"},
{"^BM_Repeat/repeats:2 %console_report$"},
{"^BM_Repeat/repeats:2_mean %console_time_only_report [ ]*2$"},
{"^BM_Repeat/repeats:2_median %console_time_only_report [ ]*2$"},
{"^BM_Repeat/repeats:2_stddev %console_time_only_report [ ]*2$"}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_Repeat/repeats:2\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:2\"", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:2\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:2\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"repetition_index\": 1,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:2_mean\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:2\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 2,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:2_median\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:2\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 2,$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"name\": \"BM_Repeat/repeats:2_stddev\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:2\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
{"\"iterations\": 2,$", MR_Next}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_Repeat/repeats:2\",%csv_report$"},
{"^\"BM_Repeat/repeats:2\",%csv_report$"},
{"^\"BM_Repeat/repeats:2_mean\",%csv_report$"},
{"^\"BM_Repeat/repeats:2_median\",%csv_report$"},
{"^\"BM_Repeat/repeats:2_stddev\",%csv_report$"}});
// but for two repetitions, mean and median is the same, so let's repeat..
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
BENCHMARK(BM_Repeat)->Repetitions(3);
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(TC_ConsoleOut,
{{"^BM_Repeat/repeats:3 %console_report$"},
{"^BM_Repeat/repeats:3 %console_report$"},
{"^BM_Repeat/repeats:3 %console_report$"},
{"^BM_Repeat/repeats:3_mean %console_time_only_report [ ]*3$"},
{"^BM_Repeat/repeats:3_median %console_time_only_report [ ]*3$"},
{"^BM_Repeat/repeats:3_stddev %console_time_only_report [ ]*3$"}});
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_Repeat/repeats:3\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:3\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"\"name\": \"BM_Repeat/repeats:3\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:3\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"repetition_index\": 1,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"\"name\": \"BM_Repeat/repeats:3\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:3\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"repetition_index\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"\"name\": \"BM_Repeat/repeats:3_mean\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:3\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:3_median\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:3\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"name\": \"BM_Repeat/repeats:3_stddev\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
{"\"iterations\": 3,$", MR_Next}});
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_Repeat/repeats:3\",%csv_report$"},
{"^\"BM_Repeat/repeats:3\",%csv_report$"},
{"^\"BM_Repeat/repeats:3\",%csv_report$"},
{"^\"BM_Repeat/repeats:3_mean\",%csv_report$"},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"^\"BM_Repeat/repeats:3_median\",%csv_report$"},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"^\"BM_Repeat/repeats:3_stddev\",%csv_report$"}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
// median differs between even/odd number of repetitions, so just to be sure
BENCHMARK(BM_Repeat)->Repetitions(4);
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(TC_ConsoleOut,
{{"^BM_Repeat/repeats:4 %console_report$"},
{"^BM_Repeat/repeats:4 %console_report$"},
{"^BM_Repeat/repeats:4 %console_report$"},
{"^BM_Repeat/repeats:4 %console_report$"},
{"^BM_Repeat/repeats:4_mean %console_time_only_report [ ]*4$"},
{"^BM_Repeat/repeats:4_median %console_time_only_report [ ]*4$"},
{"^BM_Repeat/repeats:4_stddev %console_time_only_report [ ]*4$"}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_Repeat/repeats:4\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:4\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"repetition_index\": 1,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:4\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"repetition_index\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:4\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"repetition_index\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:4_mean\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 4,$", MR_Next},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"\"name\": \"BM_Repeat/repeats:4_median\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 4,$", MR_Next},
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
{"\"name\": \"BM_Repeat/repeats:4_stddev\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_Repeat/repeats:4\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 4,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
{"\"iterations\": 4,$", MR_Next}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_Repeat/repeats:4\",%csv_report$"},
{"^\"BM_Repeat/repeats:4\",%csv_report$"},
{"^\"BM_Repeat/repeats:4\",%csv_report$"},
{"^\"BM_Repeat/repeats:4\",%csv_report$"},
{"^\"BM_Repeat/repeats:4_mean\",%csv_report$"},
{"^\"BM_Repeat/repeats:4_median\",%csv_report$"},
{"^\"BM_Repeat/repeats:4_stddev\",%csv_report$"}});
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
// Test that a non-repeated test still prints non-aggregate results even when
// only-aggregate reports have been requested
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
void BM_RepeatOnce(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
}
}
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
BENCHMARK(BM_RepeatOnce)->Repetitions(1)->ReportAggregatesOnly();
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_RepeatOnce/repeats:1 %console_report$"}});
Track 'type' of the run - is it an actual measurement, or an aggregate. (#658) This is *only* exposed in the JSON. Not in CSV, which is deprecated. This *only* supposed to track these two states. An additional field could later track which aggregate this is, specifically (statistic name, rms, bigo, ...) The motivation is that we already have ReportAggregatesOnly, but it affects the entire reports, both the display, and the reporters (json files), which isn't ideal. It would be very useful to have a 'display aggregates only' option, both in the library's console reporter, and the python tooling, This will be especially needed for the 'store separate iterations'.
2018-08-28 15:11:36 +00:00
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_RepeatOnce/repeats:1\",$"},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"run_name\": \"BM_RepeatOnce/repeats:1\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"run_type\": \"iteration\",$", MR_Next},
{"\"repetitions\": 1,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next}});
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_CSVOut, {{"^\"BM_RepeatOnce/repeats:1\",%csv_report$"}});
Refactor output test runner into standalone module. (#277) * refactor * Move default substitutions into library * Move default substitutions to the *right* place in the library * Fix init order issues that caused test failures * improve diagnostics * add missing include * general cleanup * Address review comments
2016-08-28 19:24:16 +00:00
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
// Test that non-aggregate data is not reported
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
void BM_SummaryRepeat(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
}
}
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
BENCHMARK(BM_SummaryRepeat)->Repetitions(3)->ReportAggregatesOnly();
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(
TC_ConsoleOut,
{{".*BM_SummaryRepeat/repeats:3 ", MR_Not},
{"^BM_SummaryRepeat/repeats:3_mean %console_time_only_report [ ]*3$"},
{"^BM_SummaryRepeat/repeats:3_median %console_time_only_report [ ]*3$"},
{"^BM_SummaryRepeat/repeats:3_stddev %console_time_only_report [ ]*3$"}});
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
ADD_CASES(TC_JSONOut,
{{".*BM_SummaryRepeat/repeats:3 ", MR_Not},
{"\"name\": \"BM_SummaryRepeat/repeats:3_mean\",$"},
{"\"run_name\": \"BM_SummaryRepeat/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"name\": \"BM_SummaryRepeat/repeats:3_median\",$"},
{"\"run_name\": \"BM_SummaryRepeat/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"name\": \"BM_SummaryRepeat/repeats:3_stddev\",$"},
{"\"run_name\": \"BM_SummaryRepeat/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
{"\"iterations\": 3,$", MR_Next}});
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
ADD_CASES(TC_CSVOut, {{".*BM_SummaryRepeat/repeats:3 ", MR_Not},
{"^\"BM_SummaryRepeat/repeats:3_mean\",%csv_report$"},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"^\"BM_SummaryRepeat/repeats:3_median\",%csv_report$"},
Simplify clang-format and apply to tests (#302)
2016-10-07 18:04:50 +00:00
{"^\"BM_SummaryRepeat/repeats:3_stddev\",%csv_report$"}});
Add --benchmark_report_aggregates_only={true|false} flag for better summary output. (#267)
2016-08-11 00:20:54 +00:00
*Display* aggregates only. (#665) There is a flag https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L75-L78 and a call https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/include/benchmark/benchmark.h#L837-L840 But that affects everything, every reporter, destination: https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L316 It would be quite useful to have an ability to be more picky. More specifically, i would like to be able to only see the aggregates in the on-screen output, but for the file output to still contain everything. The former is useful in case of a lot of repetition (or even more so if every iteration is reported separately), while the former is **great** for tooling. Fixes https://github.com/google/benchmark/issues/664
2018-09-12 13:26:17 +00:00
// Test that non-aggregate data is not displayed.
// NOTE: this test is kinda bad. we are only testing the display output.
// But we don't check that the file output still contains everything...
void BM_SummaryDisplay(benchmark::State& state) {
for (auto _ : state) {
}
}
BENCHMARK(BM_SummaryDisplay)->Repetitions(2)->DisplayAggregatesOnly();
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(
TC_ConsoleOut,
{{".*BM_SummaryDisplay/repeats:2 ", MR_Not},
{"^BM_SummaryDisplay/repeats:2_mean %console_time_only_report [ ]*2$"},
{"^BM_SummaryDisplay/repeats:2_median %console_time_only_report [ ]*2$"},
{"^BM_SummaryDisplay/repeats:2_stddev %console_time_only_report [ ]*2$"}});
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
ADD_CASES(TC_JSONOut,
{{".*BM_SummaryDisplay/repeats:2 ", MR_Not},
{"\"name\": \"BM_SummaryDisplay/repeats:2_mean\",$"},
{"\"run_name\": \"BM_SummaryDisplay/repeats:2\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 2,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"name\": \"BM_SummaryDisplay/repeats:2_median\",$"},
{"\"run_name\": \"BM_SummaryDisplay/repeats:2\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 2,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"name\": \"BM_SummaryDisplay/repeats:2_stddev\",$"},
{"\"run_name\": \"BM_SummaryDisplay/repeats:2\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
{"\"iterations\": 2,$", MR_Next}});
*Display* aggregates only. (#665) There is a flag https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L75-L78 and a call https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/include/benchmark/benchmark.h#L837-L840 But that affects everything, every reporter, destination: https://github.com/google/benchmark/blob/d9cab612e40017af10bddaa5b60c7067032a9e1c/src/benchmark.cc#L316 It would be quite useful to have an ability to be more picky. More specifically, i would like to be able to only see the aggregates in the on-screen output, but for the file output to still contain everything. The former is useful in case of a lot of repetition (or even more so if every iteration is reported separately), while the former is **great** for tooling. Fixes https://github.com/google/benchmark/issues/664
2018-09-12 13:26:17 +00:00
ADD_CASES(TC_CSVOut,
{{".*BM_SummaryDisplay/repeats:2 ", MR_Not},
{"^\"BM_SummaryDisplay/repeats:2_mean\",%csv_report$"},
{"^\"BM_SummaryDisplay/repeats:2_median\",%csv_report$"},
{"^\"BM_SummaryDisplay/repeats:2_stddev\",%csv_report$"}});
// Test repeats with custom time unit.
Add tests.
2016-10-21 12:59:06 +00:00
void BM_RepeatTimeUnit(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Add tests.
2016-10-21 12:59:06 +00:00
}
}
BENCHMARK(BM_RepeatTimeUnit)
->Repetitions(3)
->ReportAggregatesOnly()
->Unit(benchmark::kMicrosecond);
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(
TC_ConsoleOut,
{{".*BM_RepeatTimeUnit/repeats:3 ", MR_Not},
{"^BM_RepeatTimeUnit/repeats:3_mean %console_us_time_only_report [ ]*3$"},
{"^BM_RepeatTimeUnit/repeats:3_median %console_us_time_only_report [ "
"]*3$"},
{"^BM_RepeatTimeUnit/repeats:3_stddev %console_us_time_only_report [ "
"]*3$"}});
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
ADD_CASES(TC_JSONOut,
{{".*BM_RepeatTimeUnit/repeats:3 ", MR_Not},
{"\"name\": \"BM_RepeatTimeUnit/repeats:3_mean\",$"},
{"\"run_name\": \"BM_RepeatTimeUnit/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"time_unit\": \"us\",?$"},
{"\"name\": \"BM_RepeatTimeUnit/repeats:3_median\",$"},
{"\"run_name\": \"BM_RepeatTimeUnit/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"time_unit\": \"us\",?$"},
{"\"name\": \"BM_RepeatTimeUnit/repeats:3_stddev\",$"},
{"\"run_name\": \"BM_RepeatTimeUnit/repeats:3\",$", MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 3,$", MR_Next},
Track two more details about runs - the aggregate name, and run name. (#675) This is related to @BaaMeow's work in https://github.com/google/benchmark/pull/616 but is not based on it. Two new fields are tracked, and dumped into JSON: * If the run is an aggregate, the aggregate's name is stored. It can be RMS, BigO, mean, median, stddev, or any custom stat name. * The aggregate-name-less run name is additionally stored. I.e. not some name of the benchmark function, but the actual name, but without the 'aggregate name' suffix. This way one can group/filter all the runs, and filter by the particular aggregate type. I *might* need this for further tooling improvement. Or maybe not. But this is certainly worthwhile for custom tooling.
2018-09-13 12:08:15 +00:00
{"\"time_unit\": \"us\",?$"}});
Add tests.
2016-10-21 12:59:06 +00:00
ADD_CASES(TC_CSVOut,
{{".*BM_RepeatTimeUnit/repeats:3 ", MR_Not},
{"^\"BM_RepeatTimeUnit/repeats:3_mean\",%csv_us_report$"},
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
{"^\"BM_RepeatTimeUnit/repeats:3_median\",%csv_us_report$"},
Add tests.
2016-10-21 12:59:06 +00:00
{"^\"BM_RepeatTimeUnit/repeats:3_stddev\",%csv_us_report$"}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
// ========================================================================= //
// -------------------- Testing user-provided statistics ------------------- //
// ========================================================================= //
const auto UserStatistics = [](const std::vector<double>& v) {
return v.back();
};
void BM_UserStats(benchmark::State& state) {
Refactor most usages of KeepRunning to use the perfered ranged-for. (#459) Recently the library added a new ranged-for variant of the KeepRunning loop that is much faster. For this reason it should be preferred in all new code. Because a library, its documentation, and its tests should all embody the best practices of using the library, this patch changes all but a few usages of KeepRunning() into for (auto _ : state). The remaining usages in the tests and documentation persist only to document and test behavior that is different between the two formulations. Also note that because the range-for loop requires C++11, the KeepRunning variant has not been deprecated at this time.
2017-10-17 18:17:02 +00:00
for (auto _ : state) {
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
state.SetIterationTime(150 / 10e8);
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
}
}
(clang-)format all the things (#610) * format all documents according to contributor guidelines and specifications use clang-format on/off to stop formatting when it makes excessively poor decisions * format all tests as well, and mark blocks which change too much
2018-06-01 10:14:19 +00:00
// clang-format off
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
BENCHMARK(BM_UserStats)
(clang-)format all the things (#610) * format all documents according to contributor guidelines and specifications use clang-format on/off to stop formatting when it makes excessively poor decisions * format all tests as well, and mark blocks which change too much
2018-06-01 10:14:19 +00:00
->Repetitions(3)
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
->Iterations(5)
->UseManualTime()
(clang-)format all the things (#610) * format all documents according to contributor guidelines and specifications use clang-format on/off to stop formatting when it makes excessively poor decisions * format all tests as well, and mark blocks which change too much
2018-06-01 10:14:19 +00:00
->ComputeStatistics("", UserStatistics);
// clang-format on
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
// check that user-provided stats is calculated, and is after the default-ones
// empty string as name is intentional, it would sort before anything else
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
ADD_CASES(TC_ConsoleOut, {{"^BM_UserStats/iterations:5/repeats:3/manual_time [ "
"]* 150 ns %time [ ]*5$"},
{"^BM_UserStats/iterations:5/repeats:3/manual_time [ "
"]* 150 ns %time [ ]*5$"},
{"^BM_UserStats/iterations:5/repeats:3/manual_time [ "
"]* 150 ns %time [ ]*5$"},
{"^BM_UserStats/iterations:5/repeats:3/"
"manual_time_mean [ ]* 150 ns %time [ ]*3$"},
{"^BM_UserStats/iterations:5/repeats:3/"
"manual_time_median [ ]* 150 ns %time [ ]*3$"},
{"^BM_UserStats/iterations:5/repeats:3/"
Print at least three significant digits for times. (#701) Some benchmarks are particularly sensitive and they run in less than a nanosecond. In order for the console reporter to provide meaningful output for such benchmarks it needs to be able to display the times using more resolution than a single nanosecond. This patch changes the console reporter to print at least three significant digits for all results. Unlike the initial attempt, this patch does not align the decimal point.
2018-12-14 03:49:21 +00:00
"manual_time_stddev [ ]* 0.000 ns %time [ ]*3$"},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"^BM_UserStats/iterations:5/repeats:3/manual_time_ "
"[ ]* 150 ns %time [ ]*3$"}});
ADD_CASES(
TC_JSONOut,
{{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 5,$", MR_Next},
{"\"real_time\": 1\\.5(0)*e\\+(0)*2,$", MR_Next},
{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"repetition_index\": 1,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 5,$", MR_Next},
{"\"real_time\": 1\\.5(0)*e\\+(0)*2,$", MR_Next},
{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"iteration\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"repetition_index\": 2,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"iterations\": 5,$", MR_Next},
{"\"real_time\": 1\\.5(0)*e\\+(0)*2,$", MR_Next},
{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time_mean\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"mean\",$", MR_Next},
{"\"iterations\": 3,$", MR_Next},
{"\"real_time\": 1\\.5(0)*e\\+(0)*2,$", MR_Next},
{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time_median\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"median\",$", MR_Next},
{"\"iterations\": 3,$", MR_Next},
{"\"real_time\": 1\\.5(0)*e\\+(0)*2,$", MR_Next},
{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time_stddev\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"stddev\",$", MR_Next},
{"\"iterations\": 3,$", MR_Next},
{"\"real_time\": %float,$", MR_Next},
{"\"name\": \"BM_UserStats/iterations:5/repeats:3/manual_time_\",$"},
{"\"run_name\": \"BM_UserStats/iterations:5/repeats:3/manual_time\",$",
MR_Next},
{"\"run_type\": \"aggregate\",$", MR_Next},
[JSON] add threads and repetitions to the json output (#748) * [JSON] add threads and repetitions to the json output, for better ide… [Tests] explicitly check for thread == 1 [Tests] specifically mark all repetition checks [JSON] add repetition_index reporting, but only for non-aggregates (i… * [Formatting] Be very, very explicit about pointer alignment so clang-format can not put pointers/references on the wrong side of arguments. [Benchmark::Run] Make sure to use explanatory sentinel variable rather than a magic number. * Do not pass redundant information
2019-03-26 09:53:07 +00:00
{"\"repetitions\": 3,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
Aggregates: use non-aggregate count as iteration count. (#706) It is incorrect to say that an aggregate is computed over run's iterations, because those iterations already got averaged. Similarly, if there are N repetitions with 1 iterations each, an aggregate will be computed over N measurements, not 1. Thus it is best to simply use the count of separate reports. Fixes #586.
2018-10-18 14:17:14 +00:00
{"\"aggregate_name\": \"\",$", MR_Next},
{"\"iterations\": 3,$", MR_Next},
{"\"real_time\": 1\\.5(0)*e\\+(0)*2,$", MR_Next}});
ADD_CASES(
TC_CSVOut,
{{"^\"BM_UserStats/iterations:5/repeats:3/manual_time\",%csv_report$"},
{"^\"BM_UserStats/iterations:5/repeats:3/manual_time\",%csv_report$"},
{"^\"BM_UserStats/iterations:5/repeats:3/manual_time\",%csv_report$"},
{"^\"BM_UserStats/iterations:5/repeats:3/manual_time_mean\",%csv_report$"},
{"^\"BM_UserStats/iterations:5/repeats:3/"
"manual_time_median\",%csv_report$"},
{"^\"BM_UserStats/iterations:5/repeats:3/"
"manual_time_stddev\",%csv_report$"},
{"^\"BM_UserStats/iterations:5/repeats:3/manual_time_\",%csv_report$"}});
Drop Stat1, refactor statistics to be user-providable, add median. (#428) * Drop Stat1, refactor statistics to be user-providable, add median. My main goal was to add median statistic. Since Stat1 calculated the stats incrementally, and did not store the values themselves, it is was not possible. Thus, i have replaced Stat1 with simple std::vector<double>, containing all the values. Then, i have refactored current mean/stdev to be a function that is provided with values vector, and returns the statistic. While there, it seemed to make sense to deduplicate the code by storing all the statistics functions in a map, and then simply iterate over it. And the interface to add new statistics is intentionally exposed, so they may be added easily. The notable change is that Iterations are no longer displayed as 0 for stdev. Is could be changed, but i'm not sure how to nicely fit that into the API. Similarly, this dance about sometimes (for some fields, for some statistics) dividing by run.iterations, and then multiplying the calculated stastic back is also dropped, and if you do the math, i fail to see why it was needed there in the first place. Since that was the only use of stat.h, it is removed. * complexity.h: attempt to fix MSVC build * Update README.md * Store statistics to compute in a vector, ensures ordering. * Add a bit more tests for repetitions. * Partially address review notes. * Fix gcc build: drop extra ';' clang, why didn't you warn me? * Address review comments. * double() -> 0.0 * early return
2017-08-23 23:44:29 +00:00
escape special chars in csv and json output. (#802) * escape special chars in csv and json output. - escape \b,\f,\n,\r,\t,\," from strings before dumping them to json or csv. - also faithfully reproduce the sign of nan in json. this fixes github issue #745. * functionalize. * split string escape functions between csv and json * Update src/csv_reporter.cc Co-Authored-By: tesch1 <tesch1@gmail.com> * Update src/json_reporter.cc Co-Authored-By: tesch1 <tesch1@gmail.com>
2019-04-19 17:47:25 +00:00
// ========================================================================= //
// ------------------------- Testing StrEscape JSON ------------------------ //
// ========================================================================= //
#if 0 // enable when csv testing code correctly handles multi-line fields
void BM_JSON_Format(benchmark::State& state) {
state.SkipWithError("val\b\f\n\r\t\\\"with\"es,capes");
for (auto _ : state) {
}
}
BENCHMARK(BM_JSON_Format);
ADD_CASES(TC_JSONOut, {{"\"name\": \"BM_JSON_Format\",$"},
{"\"run_name\": \"BM_JSON_Format\",$", MR_Next},
{"\"run_type\": \"iteration\",$", MR_Next},
{"\"repetitions\": 0,$", MR_Next},
{"\"repetition_index\": 0,$", MR_Next},
{"\"threads\": 1,$", MR_Next},
{"\"error_occurred\": true,$", MR_Next},
{R"("error_message": "val\\b\\f\\n\\r\\t\\\\\\"with\\"es,capes",$)", MR_Next}});
#endif
// ========================================================================= //
// -------------------------- Testing CsvEscape ---------------------------- //
// ========================================================================= //
void BM_CSV_Format(benchmark::State& state) {
state.SkipWithError("\"freedom\"");
for (auto _ : state) {
}
}
BENCHMARK(BM_CSV_Format);
ADD_CASES(TC_CSVOut, {{"^\"BM_CSV_Format\",,,,,,,,true,\"\"\"freedom\"\"\"$"}});
Change reporters to use a specified output and error stream. Add tests for output. (#219) * Add test for reporter output. * setup err_stream tests * Fix warnings in tests * whitespace * Fix build errors caused by super pedantic compilers * Pass streams by pointer not non-const reference
2016-05-27 19:34:37 +00:00
// ========================================================================= //
// --------------------------- TEST CASES END ------------------------------ //
// ========================================================================= //
Add tests.
2016-10-21 12:59:06 +00:00
int main(int argc, char* argv[]) { RunOutputTests(argc, argv); }
Reference in a new issue Copy permalink