2016-08-09 18:33:57 +00:00
|
|
|
{
|
|
|
|
|
"context": {
|
|
|
|
|
"date": "2016-08-02 17:44:46",
|
|
|
|
|
"num_cpus": 4,
|
|
|
|
|
"mhz_per_cpu": 4228,
|
|
|
|
|
"cpu_scaling_enabled": false,
|
|
|
|
|
"library_build_type": "release"
|
|
|
|
|
},
|
|
|
|
|
"benchmarks": [
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_SameTimes",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 10,
|
|
|
|
|
"cpu_time": 10,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_2xFaster",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 25,
|
|
|
|
|
"cpu_time": 25,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_2xSlower",
|
|
|
|
|
"iterations": 20833333,
|
|
|
|
|
"real_time": 100,
|
|
|
|
|
"cpu_time": 100,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
Json reporter: don't cast floating-point to int; adjust tooling (#426)
* Json reporter: passthrough fp, don't cast it to int; adjust tooling
Json output format is generally meant for further processing
using some automated tools. Thus, it makes sense not to
intentionally limit the precision of the values contained
in the report.
As it can be seen, FormatKV() for doubles, used %.2f format,
which was meant to preserve at least some of the precision.
However, before that function is ever called, the doubles
were already cast to the integer via RoundDouble()...
This is also the case for console reporter, where it makes
sense because the screen space is limited, and this reporter,
however the CSV reporter does output some( decimal digits.
Thus i can only conclude that the loss of the precision
was not really considered, so i have decided to adjust the
code of the json reporter to output the full fp precision.
There can be several reasons why that is the right thing
to do, the bigger the time_unit used, the greater the
precision loss, so i'd say any sort of further processing
(like e.g. tools/compare_bench.py does) is best done
on the values with most precision.
Also, that cast skewed the data away from zero, which
i think may or may not result in false- positives/negatives
in the output of tools/compare_bench.py
* Json reporter: FormatKV(double): address review note
* tools/gbench/report.py: skip benchmarks with different time units
While it may be useful to teach it to operate on the
measurements with different time units, which is now
possible since floats are stored, and not the integers,
but for now at least doing such a sanity-checking
is better than providing misinformation.
2017-07-24 23:13:55 +00:00
|
|
|
{
|
|
|
|
|
"name": "BM_1PercentFaster",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 98.9999999,
|
|
|
|
|
"cpu_time": 98.9999999,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_1PercentSlower",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 100.9999999,
|
|
|
|
|
"cpu_time": 100.9999999,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
2016-08-09 18:33:57 +00:00
|
|
|
{
|
|
|
|
|
"name": "BM_10PercentFaster",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 90,
|
|
|
|
|
"cpu_time": 90,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_10PercentSlower",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 110,
|
|
|
|
|
"cpu_time": 110,
|
|
|
|
|
"time_unit": "ns"
|
2016-12-09 12:24:31 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_100xSlower",
|
|
|
|
|
"iterations": 1000,
|
Json reporter: don't cast floating-point to int; adjust tooling (#426)
* Json reporter: passthrough fp, don't cast it to int; adjust tooling
Json output format is generally meant for further processing
using some automated tools. Thus, it makes sense not to
intentionally limit the precision of the values contained
in the report.
As it can be seen, FormatKV() for doubles, used %.2f format,
which was meant to preserve at least some of the precision.
However, before that function is ever called, the doubles
were already cast to the integer via RoundDouble()...
This is also the case for console reporter, where it makes
sense because the screen space is limited, and this reporter,
however the CSV reporter does output some( decimal digits.
Thus i can only conclude that the loss of the precision
was not really considered, so i have decided to adjust the
code of the json reporter to output the full fp precision.
There can be several reasons why that is the right thing
to do, the bigger the time_unit used, the greater the
precision loss, so i'd say any sort of further processing
(like e.g. tools/compare_bench.py does) is best done
on the values with most precision.
Also, that cast skewed the data away from zero, which
i think may or may not result in false- positives/negatives
in the output of tools/compare_bench.py
* Json reporter: FormatKV(double): address review note
* tools/gbench/report.py: skip benchmarks with different time units
While it may be useful to teach it to operate on the
measurements with different time units, which is now
possible since floats are stored, and not the integers,
but for now at least doing such a sanity-checking
is better than providing misinformation.
2017-07-24 23:13:55 +00:00
|
|
|
"real_time": 1.0000e+04,
|
|
|
|
|
"cpu_time": 1.0000e+04,
|
2016-12-09 12:24:31 +00:00
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_100xFaster",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 100,
|
|
|
|
|
"cpu_time": 100,
|
|
|
|
|
"time_unit": "ns"
|
Json reporter: don't cast floating-point to int; adjust tooling (#426)
* Json reporter: passthrough fp, don't cast it to int; adjust tooling
Json output format is generally meant for further processing
using some automated tools. Thus, it makes sense not to
intentionally limit the precision of the values contained
in the report.
As it can be seen, FormatKV() for doubles, used %.2f format,
which was meant to preserve at least some of the precision.
However, before that function is ever called, the doubles
were already cast to the integer via RoundDouble()...
This is also the case for console reporter, where it makes
sense because the screen space is limited, and this reporter,
however the CSV reporter does output some( decimal digits.
Thus i can only conclude that the loss of the precision
was not really considered, so i have decided to adjust the
code of the json reporter to output the full fp precision.
There can be several reasons why that is the right thing
to do, the bigger the time_unit used, the greater the
precision loss, so i'd say any sort of further processing
(like e.g. tools/compare_bench.py does) is best done
on the values with most precision.
Also, that cast skewed the data away from zero, which
i think may or may not result in false- positives/negatives
in the output of tools/compare_bench.py
* Json reporter: FormatKV(double): address review note
* tools/gbench/report.py: skip benchmarks with different time units
While it may be useful to teach it to operate on the
measurements with different time units, which is now
possible since floats are stored, and not the integers,
but for now at least doing such a sanity-checking
is better than providing misinformation.
2017-07-24 23:13:55 +00:00
|
|
|
},
|
2017-07-25 16:09:26 +00:00
|
|
|
{
|
|
|
|
|
"name": "BM_10PercentCPUToTime",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 110,
|
|
|
|
|
"cpu_time": 90,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
[RFC] Tools: compare-bench.py: print change% with two decimal digits (#440)
* Tools: compare-bench.py: print change% with two decimal digits
Here is a comparison of before vs. after:
```diff
-Benchmark Time CPU Time Old Time New CPU Old CPU New
----------------------------------------------------------------------------------------------------------
-BM_SameTimes +0.00 +0.00 10 10 10 10
-BM_2xFaster -0.50 -0.50 50 25 50 25
-BM_2xSlower +1.00 +1.00 50 100 50 100
-BM_1PercentFaster -0.01 -0.01 100 99 100 99
-BM_1PercentSlower +0.01 +0.01 100 101 100 101
-BM_10PercentFaster -0.10 -0.10 100 90 100 90
-BM_10PercentSlower +0.10 +0.10 100 110 100 110
-BM_100xSlower +99.00 +99.00 100 10000 100 10000
-BM_100xFaster -0.99 -0.99 10000 100 10000 100
-BM_10PercentCPUToTime +0.10 -0.10 100 110 100 90
+Benchmark Time CPU Time Old Time New CPU Old CPU New
+-------------------------------------------------------------------------------------------------------------
+BM_SameTimes +0.0000 +0.0000 10 10 10 10
+BM_2xFaster -0.5000 -0.5000 50 25 50 25
+BM_2xSlower +1.0000 +1.0000 50 100 50 100
+BM_1PercentFaster -0.0100 -0.0100 100 99 100 99
+BM_1PercentSlower +0.0100 +0.0100 100 101 100 101
+BM_10PercentFaster -0.1000 -0.1000 100 90 100 90
+BM_10PercentSlower +0.1000 +0.1000 100 110 100 110
+BM_100xSlower +99.0000 +99.0000 100 10000 100 10000
+BM_100xFaster -0.9900 -0.9900 10000 100 10000 100
+BM_10PercentCPUToTime +0.1000 -0.1000 100 110 100 90
+BM_ThirdFaster -0.3333 -0.3333 100 67 100 67
```
So the first ("Time") column is exactly where it was, but with
two more decimal digits. The position of the '.' in the second
("CPU") column is shifted right by those two positions, and the
rest is unmodified, but simply shifted right by those 4 positions.
As for the reasoning, i guess it is more or less the same as
with #426. In some sad times, microbenchmarking is not applicable.
In those cases, the more precise the change report is, the better.
The current formatting prints not so much the percentages,
but the fraction i'd say. It is more useful for huge changes,
much more than 100%. That is not always the case, especially
if it is not a microbenchmark. Then, even though the change
may be good/bad, the change is small (<0.5% or so),
rounding happens, and it is no longer possible to tell.
I do acknowledge that this change does not fix that problem. Of
course, confidence intervals and such would be better, and they
would probably fix the problem. But i think this is good as-is
too, because now the you see 2 fractional percentage digits!1
The obvious downside is that the output is now even wider.
* Revisit tests, more closely documents the current behavior.
2017-08-28 23:12:18 +00:00
|
|
|
{
|
|
|
|
|
"name": "BM_ThirdFaster",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 66.665,
|
|
|
|
|
"cpu_time": 66.664,
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
2019-03-06 14:22:28 +00:00
|
|
|
{
|
|
|
|
|
"name": "MyComplexityTest_BigO",
|
|
|
|
|
"run_name": "MyComplexityTest",
|
|
|
|
|
"run_type": "aggregate",
|
|
|
|
|
"aggregate_name": "BigO",
|
|
|
|
|
"cpu_coefficient": 5.6215779594361486e+00,
|
|
|
|
|
"real_coefficient": 5.6288314793554610e+00,
|
|
|
|
|
"big_o": "N",
|
|
|
|
|
"time_unit": "ns"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "MyComplexityTest_RMS",
|
|
|
|
|
"run_name": "MyComplexityTest",
|
|
|
|
|
"run_type": "aggregate",
|
|
|
|
|
"aggregate_name": "RMS",
|
|
|
|
|
"rms": 3.3128901852342174e-03
|
2019-03-06 15:13:22 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_NotBadTimeUnit",
|
|
|
|
|
"iterations": 1000,
|
|
|
|
|
"real_time": 0.04,
|
|
|
|
|
"cpu_time": 0.6,
|
|
|
|
|
"time_unit": "s"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name": "BM_DifferentTimeUnit",
|
|
|
|
|
"iterations": 1,
|
|
|
|
|
"real_time": 1,
|
|
|
|
|
"cpu_time": 1,
|
|
|
|
|
"time_unit": "ns"
|
2016-08-09 18:33:57 +00:00
|
|
|
}
|
|
|
|
|
]
|
Json reporter: don't cast floating-point to int; adjust tooling (#426)
* Json reporter: passthrough fp, don't cast it to int; adjust tooling
Json output format is generally meant for further processing
using some automated tools. Thus, it makes sense not to
intentionally limit the precision of the values contained
in the report.
As it can be seen, FormatKV() for doubles, used %.2f format,
which was meant to preserve at least some of the precision.
However, before that function is ever called, the doubles
were already cast to the integer via RoundDouble()...
This is also the case for console reporter, where it makes
sense because the screen space is limited, and this reporter,
however the CSV reporter does output some( decimal digits.
Thus i can only conclude that the loss of the precision
was not really considered, so i have decided to adjust the
code of the json reporter to output the full fp precision.
There can be several reasons why that is the right thing
to do, the bigger the time_unit used, the greater the
precision loss, so i'd say any sort of further processing
(like e.g. tools/compare_bench.py does) is best done
on the values with most precision.
Also, that cast skewed the data away from zero, which
i think may or may not result in false- positives/negatives
in the output of tools/compare_bench.py
* Json reporter: FormatKV(double): address review note
* tools/gbench/report.py: skip benchmarks with different time units
While it may be useful to teach it to operate on the
measurements with different time units, which is now
possible since floats are stored, and not the integers,
but for now at least doing such a sanity-checking
is better than providing misinformation.
2017-07-24 23:13:55 +00:00
|
|
|
}
|