mirror of
https://github.com/google/benchmark.git
synced 2024-12-04 20:04:32 +00:00
aad33aab3c
My knowledge of python is not great, so this is kinda horrible.
Two things:
1. If there were repetitions, for the RHS (i.e. the new value) we were always using the first repetition,
which naturally results in incorrect change reports for the second and following repetitions.
And what is even worse, that completely broke U test. :(
2. A better support for different repetition count for U test was missing.
It's important if we are to be able to report 'iteration as repetition',
since it is rather likely that the iteration count will mismatch.
Now, the rough idea on how this is implemented now. I think this is the right solution.
1. Get all benchmark names (in order) from the lhs benchmark.
2. While preserving the order, keep the unique names
3. Get all benchmark names (in order) from the rhs benchmark.
4. While preserving the order, keep the unique names
5. Intersect `2.` and `4.`, get the list of unique benchmark names that exist on both sides.
6. Now, we want to group (partition) all the benchmarks with the same name.
```
BM_FOO:
[lhs]: BM_FOO/repetition0 BM_FOO/repetition1
[rhs]: BM_FOO/repetition0 BM_FOO/repetition1 BM_FOO/repetition2
...
```
We also drop mismatches in `time_unit` here.
_(whose bright idea was it to store arbitrarily scaled timers in json **?!** )_
7. Iterate for each partition
7.1. Conditionally, diff the overlapping repetitions (the count of repetitions may be different.)
7.2. Conditionally, do the U test:
7.2.1. Get **all** the values of `"real_time"` field from the lhs benchmark
7.2.2. Get **all** the values of `"cpu_time"` field from the lhs benchmark
7.2.3. Get **all** the values of `"real_time"` field from the rhs benchmark
7.2.4. Get **all** the values of `"cpu_time"` field from the rhs benchmark
NOTE: the repetition count may be different, but we want *all* the values!
7.2.5. Do the rest of the u test stuff
7.2.6. Print u test
8. ???
9. **PROFIT**!
Fixes #677
66 lines
1.2 KiB
JSON
66 lines
1.2 KiB
JSON
{
|
|
"context": {
|
|
"date": "2016-08-02 17:44:46",
|
|
"num_cpus": 4,
|
|
"mhz_per_cpu": 4228,
|
|
"cpu_scaling_enabled": false,
|
|
"library_build_type": "release"
|
|
},
|
|
"benchmarks": [
|
|
{
|
|
"name": "BM_One",
|
|
"run_type": "aggregate",
|
|
"iterations": 1000,
|
|
"real_time": 10,
|
|
"cpu_time": 100,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "BM_Two",
|
|
"iterations": 1000,
|
|
"real_time": 9,
|
|
"cpu_time": 90,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "BM_Two",
|
|
"iterations": 1000,
|
|
"real_time": 8,
|
|
"cpu_time": 86,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "short",
|
|
"run_type": "aggregate",
|
|
"iterations": 1000,
|
|
"real_time": 8,
|
|
"cpu_time": 80,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "short",
|
|
"run_type": "aggregate",
|
|
"iterations": 1000,
|
|
"real_time": 8,
|
|
"cpu_time": 77,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "medium",
|
|
"run_type": "iteration",
|
|
"iterations": 1000,
|
|
"real_time": 8,
|
|
"cpu_time": 80,
|
|
"time_unit": "ns"
|
|
},
|
|
{
|
|
"name": "medium",
|
|
"run_type": "iteration",
|
|
"iterations": 1000,
|
|
"real_time": 9,
|
|
"cpu_time": 82,
|
|
"time_unit": "ns"
|
|
}
|
|
]
|
|
}
|