Commit graph

14 commits

Author SHA1 Message Date
Roman Lebedev 3347a20e0e reporter_output_test: json: iterations is int, not float (#431)
May be relevant for flakiness of win builds

Noted by @KindDragon
2017-07-31 19:04:02 -06:00
Roman Lebedev b9be142d1e Json reporter: don't cast floating-point to int; adjust tooling (#426)
* Json reporter: passthrough fp, don't cast it to int; adjust tooling

Json output format is generally meant for further processing
using some automated tools. Thus, it makes sense not to
intentionally limit the precision of the values contained
in the report.

As it can be seen, FormatKV() for doubles, used %.2f format,
which was meant to preserve at least some of the precision.
However, before that function is ever called, the doubles
were already cast to the integer via RoundDouble()...

This is also the case for console reporter, where it makes
sense because the screen space is limited, and this reporter,
however the CSV reporter does output some( decimal digits.

Thus i can only conclude that the loss of the precision
was not really considered, so i have decided to adjust the
code of the json reporter to output the full fp precision.

There can be several reasons why that is the right thing
to do, the bigger the time_unit used, the greater the
precision loss, so i'd say any sort of further processing
(like e.g. tools/compare_bench.py does) is best done
on the values with most precision.

Also, that cast skewed the data away from zero, which
i think may or may not result in false- positives/negatives
in the output of tools/compare_bench.py

* Json reporter: FormatKV(double): address review note

* tools/gbench/report.py: skip benchmarks with different time units

While it may be useful to teach it to operate on the
measurements with different time units, which is now
possible since floats are stored, and not the integers,
but for now at least doing such a sanity-checking
is better than providing misinformation.
2017-07-24 16:13:55 -07:00
Joao Paulo Magalhaes 21600b966f Fix VS warning. 2017-05-01 23:01:57 +01:00
Joao Paulo Magalhaes 47226ccd56 CHECK(): rename EPS to FLOAT for consistency with googletest style. 2017-05-01 23:01:10 +01:00
Joao Paulo Magalhaes 2a2eb44b30 Fix VS2013 quirk. 2017-04-29 22:27:55 +01:00
Joao Paulo Magalhaes 180719d0d6 Rename ResultsCheckerEntry to Results. 2017-04-29 20:01:18 +01:00
Joao Paulo Magalhaes 03b0655d12 Fix expected values of user counters as rates in unit tests. 2017-04-29 19:35:43 +01:00
Joao Paulo Magalhaes 78548f8c6e Add (currently failing) tests for user counters with threads. 2017-04-29 19:26:34 +01:00
Joao Paulo Magalhaes 92034a8b84 Make result checkers execute on all regex-matching benchmarks. 2017-04-29 19:02:07 +01:00
Joao Paulo Magalhaes 1ce286f632 Avoid compiler-specific pragmas in result check macros.
- Epsilon is now understood as relative to expected value.
- Improve error messages for epsilon checks.
2017-04-29 18:26:30 +01:00
Joao Paulo Magalhaes da69e5de45 User counters: add more tests. 2017-04-28 20:45:30 +01:00
Joao Paulo Magalhaes 6452883027 Unit testing: add facilities to check benchmark results.
This is needed for examining the values of user counters (needed
for #348). It is also needed for checking the values of standard
benchmark results like items_processed or complexities (for example,
checking the standard deviation is needed for unit testing #357
as discussed in #362).
2017-04-28 15:02:27 +01:00
Joao Paulo Magalhaes 693a43013d User counters: add more unit tests. ...
The tests are still missing a way to check actual validity of
numerical results; this will be done next. As they currently are,
the tests pass, but the problem detected with #378 is still
standing and the results with non-standard counters are wrong.
2017-04-27 22:11:40 +01:00
Joao Paulo Magalhaes 3c2d7f5348 User counter tests: first version. 2017-04-27 19:25:20 +01:00