mirror of
https://github.com/google/benchmark.git
synced 2024-11-29 18:34:47 +00:00
5e66248b44
* [Tools] A new, more versatile benchmark output compare tool
Sometimes, there is more than one implementation of some functionality.
And the obvious use-case is to benchmark them, which is better?
Currently, there is no easy way to compare the benchmarking results
in that case:
The obvious solution is to have multiple binaries, each one
containing/running one implementation. And each binary must use
exactly the same benchmark family name, which is super bad,
because now the binary name should contain all the info about
benchmark family...
What if i tell you that is not the solution?
What if we could avoid producing one binary per benchmark family,
with the same family name used in each binary,
but instead could keep all the related families in one binary,
with their proper names, AND still be able to compare them?
There are three modes of operation:
1. Just compare two benchmarks, what `compare_bench.py` did:
```
$ ../tools/compare.py benchmarks ./a.out ./a.out
RUNNING: ./a.out --benchmark_out=/tmp/tmprBT5nW
Run on (8 X 4000 MHz CPU s)
2017-11-07 21:16:44
------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------
BM_memcpy/8 36 ns 36 ns 19101577 211.669MB/s
BM_memcpy/64 76 ns 76 ns 9412571 800.199MB/s
BM_memcpy/512 84 ns 84 ns 8249070 5.64771GB/s
BM_memcpy/1024 116 ns 116 ns 6181763 8.19505GB/s
BM_memcpy/8192 643 ns 643 ns 1062855 11.8636GB/s
BM_copy/8 222 ns 222 ns 3137987 34.3772MB/s
BM_copy/64 1608 ns 1608 ns 432758 37.9501MB/s
BM_copy/512 12589 ns 12589 ns 54806 38.7867MB/s
BM_copy/1024 25169 ns 25169 ns 27713 38.8003MB/s
BM_copy/8192 201165 ns 201112 ns 3486 38.8466MB/s
RUNNING: ./a.out --benchmark_out=/tmp/tmpt1wwG_
Run on (8 X 4000 MHz CPU s)
2017-11-07 21:16:53
------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------
BM_memcpy/8 36 ns 36 ns 19397903 211.255MB/s
BM_memcpy/64 73 ns 73 ns 9691174 839.635MB/s
BM_memcpy/512 85 ns 85 ns 8312329 5.60101GB/s
BM_memcpy/1024 118 ns 118 ns 6438774 8.11608GB/s
BM_memcpy/8192 656 ns 656 ns 1068644 11.6277GB/s
BM_copy/8 223 ns 223 ns 3146977 34.2338MB/s
BM_copy/64 1611 ns 1611 ns 435340 37.8751MB/s
BM_copy/512 12622 ns 12622 ns 54818 38.6844MB/s
BM_copy/1024 25257 ns 25239 ns 27779 38.6927MB/s
BM_copy/8192 205013 ns 205010 ns 3479 38.108MB/s
Comparing ./a.out to ./a.out
Benchmark Time CPU Time Old Time New CPU Old CPU New
------------------------------------------------------------------------------------------------------
BM_memcpy/8 +0.0020 +0.0020 36 36 36 36
BM_memcpy/64 -0.0468 -0.0470 76 73 76 73
BM_memcpy/512 +0.0081 +0.0083 84 85 84 85
BM_memcpy/1024 +0.0098 +0.0097 116 118 116 118
BM_memcpy/8192 +0.0200 +0.0203 643 656 643 656
BM_copy/8 +0.0046 +0.0042 222 223 222 223
BM_copy/64 +0.0020 +0.0020 1608 1611 1608 1611
BM_copy/512 +0.0027 +0.0026 12589 12622 12589 12622
BM_copy/1024 +0.0035 +0.0028 25169 25257 25169 25239
BM_copy/8192 +0.0191 +0.0194 201165 205013 201112 205010
```
2. Compare two different filters of one benchmark:
(for simplicity, the benchmark is executed twice)
```
$ ../tools/compare.py filters ./a.out BM_memcpy BM_copy
RUNNING: ./a.out --benchmark_filter=BM_memcpy --benchmark_out=/tmp/tmpBWKk0k
Run on (8 X 4000 MHz CPU s)
2017-11-07 21:37:28
------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------
BM_memcpy/8 36 ns 36 ns 17891491 211.215MB/s
BM_memcpy/64 74 ns 74 ns 9400999 825.646MB/s
BM_memcpy/512 87 ns 87 ns 8027453 5.46126GB/s
BM_memcpy/1024 111 ns 111 ns 6116853 8.5648GB/s
BM_memcpy/8192 657 ns 656 ns 1064679 11.6247GB/s
RUNNING: ./a.out --benchmark_filter=BM_copy --benchmark_out=/tmp/tmpAvWcOM
Run on (8 X 4000 MHz CPU s)
2017-11-07 21:37:33
----------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------
BM_copy/8 227 ns 227 ns 3038700 33.6264MB/s
BM_copy/64 1640 ns 1640 ns 426893 37.2154MB/s
BM_copy/512 12804 ns 12801 ns 55417 38.1444MB/s
BM_copy/1024 25409 ns 25407 ns 27516 38.4365MB/s
BM_copy/8192 202986 ns 202990 ns 3454 38.4871MB/s
Comparing BM_memcpy to BM_copy (from ./a.out)
Benchmark Time CPU Time Old Time New CPU Old CPU New
--------------------------------------------------------------------------------------------------------------------
[BM_memcpy vs. BM_copy]/8 +5.2829 +5.2812 36 227 36 227
[BM_memcpy vs. BM_copy]/64 +21.1719 +21.1856 74 1640 74 1640
[BM_memcpy vs. BM_copy]/512 +145.6487 +145.6097 87 12804 87 12801
[BM_memcpy vs. BM_copy]/1024 +227.1860 +227.1776 111 25409 111 25407
[BM_memcpy vs. BM_copy]/8192 +308.1664 +308.2898 657 202986 656 202990
```
3. Compare filter one from benchmark one to filter two from benchmark two:
(for simplicity, the benchmark is executed twice)
```
$ ../tools/compare.py benchmarksfiltered ./a.out BM_memcpy ./a.out BM_copy
RUNNING: ./a.out --benchmark_filter=BM_memcpy --benchmark_out=/tmp/tmp_FvbYg
Run on (8 X 4000 MHz CPU s)
2017-11-07 21:38:27
------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------
BM_memcpy/8 37 ns 37 ns 18953482 204.118MB/s
BM_memcpy/64 74 ns 74 ns 9206578 828.245MB/s
BM_memcpy/512 91 ns 91 ns 8086195 5.25476GB/s
BM_memcpy/1024 120 ns 120 ns 5804513 7.95662GB/s
BM_memcpy/8192 664 ns 664 ns 1028363 11.4948GB/s
RUNNING: ./a.out --benchmark_filter=BM_copy --benchmark_out=/tmp/tmpDfL5iE
Run on (8 X 4000 MHz CPU s)
2017-11-07 21:38:32
----------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------
BM_copy/8 230 ns 230 ns 2985909 33.1161MB/s
BM_copy/64 1654 ns 1653 ns 419408 36.9137MB/s
BM_copy/512 13122 ns 13120 ns 53403 37.2156MB/s
BM_copy/1024 26679 ns 26666 ns 26575 36.6218MB/s
BM_copy/8192 215068 ns 215053 ns 3221 36.3283MB/s
Comparing BM_memcpy (from ./a.out) to BM_copy (from ./a.out)
Benchmark Time CPU Time Old Time New CPU Old CPU New
--------------------------------------------------------------------------------------------------------------------
[BM_memcpy vs. BM_copy]/8 +5.1649 +5.1637 37 230 37 230
[BM_memcpy vs. BM_copy]/64 +21.4352 +21.4374 74 1654 74 1653
[BM_memcpy vs. BM_copy]/512 +143.6022 +143.5865 91 13122 91 13120
[BM_memcpy vs. BM_copy]/1024 +221.5903 +221.4790 120 26679 120 26666
[BM_memcpy vs. BM_copy]/8192 +322.9059 +323.0096 664 215068 664 215053
```
* [Docs] Document tools/compare.py
* [docs] Document how the change is calculated
313 lines
12 KiB
Python
Executable file
313 lines
12 KiB
Python
Executable file
#!/usr/bin/env python
|
|
|
|
"""
|
|
compare.py - versatile benchmark output compare tool
|
|
"""
|
|
|
|
import argparse
|
|
from argparse import ArgumentParser
|
|
import sys
|
|
import gbench
|
|
from gbench import util, report
|
|
from gbench.util import *
|
|
|
|
|
|
def check_inputs(in1, in2, flags):
|
|
"""
|
|
Perform checking on the user provided inputs and diagnose any abnormalities
|
|
"""
|
|
in1_kind, in1_err = classify_input_file(in1)
|
|
in2_kind, in2_err = classify_input_file(in2)
|
|
output_file = find_benchmark_flag('--benchmark_out=', flags)
|
|
output_type = find_benchmark_flag('--benchmark_out_format=', flags)
|
|
if in1_kind == IT_Executable and in2_kind == IT_Executable and output_file:
|
|
print(("WARNING: '--benchmark_out=%s' will be passed to both "
|
|
"benchmarks causing it to be overwritten") % output_file)
|
|
if in1_kind == IT_JSON and in2_kind == IT_JSON and len(flags) > 0:
|
|
print("WARNING: passing optional flags has no effect since both "
|
|
"inputs are JSON")
|
|
if output_type is not None and output_type != 'json':
|
|
print(("ERROR: passing '--benchmark_out_format=%s' to 'compare.py`"
|
|
" is not supported.") % output_type)
|
|
sys.exit(1)
|
|
|
|
|
|
def create_parser():
|
|
parser = ArgumentParser(
|
|
description='versatile benchmark output compare tool')
|
|
subparsers = parser.add_subparsers(
|
|
help='This tool has multiple modes of operation:',
|
|
dest='mode')
|
|
|
|
parser_a = subparsers.add_parser(
|
|
'benchmarks',
|
|
help='The most simple use-case, compare all the output of these two benchmarks')
|
|
baseline = parser_a.add_argument_group(
|
|
'baseline', 'The benchmark baseline')
|
|
baseline.add_argument(
|
|
'test_baseline',
|
|
metavar='test_baseline',
|
|
type=argparse.FileType('r'),
|
|
nargs=1,
|
|
help='A benchmark executable or JSON output file')
|
|
contender = parser_a.add_argument_group(
|
|
'contender', 'The benchmark that will be compared against the baseline')
|
|
contender.add_argument(
|
|
'test_contender',
|
|
metavar='test_contender',
|
|
type=argparse.FileType('r'),
|
|
nargs=1,
|
|
help='A benchmark executable or JSON output file')
|
|
parser_a.add_argument(
|
|
'benchmark_options',
|
|
metavar='benchmark_options',
|
|
nargs=argparse.REMAINDER,
|
|
help='Arguments to pass when running benchmark executables')
|
|
|
|
parser_b = subparsers.add_parser(
|
|
'filters', help='Compare filter one with the filter two of benchmark')
|
|
baseline = parser_b.add_argument_group(
|
|
'baseline', 'The benchmark baseline')
|
|
baseline.add_argument(
|
|
'test',
|
|
metavar='test',
|
|
type=argparse.FileType('r'),
|
|
nargs=1,
|
|
help='A benchmark executable or JSON output file')
|
|
baseline.add_argument(
|
|
'filter_baseline',
|
|
metavar='filter_baseline',
|
|
type=str,
|
|
nargs=1,
|
|
help='The first filter, that will be used as baseline')
|
|
contender = parser_b.add_argument_group(
|
|
'contender', 'The benchmark that will be compared against the baseline')
|
|
contender.add_argument(
|
|
'filter_contender',
|
|
metavar='filter_contender',
|
|
type=str,
|
|
nargs=1,
|
|
help='The second filter, that will be compared against the baseline')
|
|
parser_b.add_argument(
|
|
'benchmark_options',
|
|
metavar='benchmark_options',
|
|
nargs=argparse.REMAINDER,
|
|
help='Arguments to pass when running benchmark executables')
|
|
|
|
parser_c = subparsers.add_parser(
|
|
'benchmarksfiltered',
|
|
help='Compare filter one of first benchmark with filter two of the second benchmark')
|
|
baseline = parser_c.add_argument_group(
|
|
'baseline', 'The benchmark baseline')
|
|
baseline.add_argument(
|
|
'test_baseline',
|
|
metavar='test_baseline',
|
|
type=argparse.FileType('r'),
|
|
nargs=1,
|
|
help='A benchmark executable or JSON output file')
|
|
baseline.add_argument(
|
|
'filter_baseline',
|
|
metavar='filter_baseline',
|
|
type=str,
|
|
nargs=1,
|
|
help='The first filter, that will be used as baseline')
|
|
contender = parser_c.add_argument_group(
|
|
'contender', 'The benchmark that will be compared against the baseline')
|
|
contender.add_argument(
|
|
'test_contender',
|
|
metavar='test_contender',
|
|
type=argparse.FileType('r'),
|
|
nargs=1,
|
|
help='The second benchmark executable or JSON output file, that will be compared against the baseline')
|
|
contender.add_argument(
|
|
'filter_contender',
|
|
metavar='filter_contender',
|
|
type=str,
|
|
nargs=1,
|
|
help='The second filter, that will be compared against the baseline')
|
|
parser_c.add_argument(
|
|
'benchmark_options',
|
|
metavar='benchmark_options',
|
|
nargs=argparse.REMAINDER,
|
|
help='Arguments to pass when running benchmark executables')
|
|
|
|
return parser
|
|
|
|
|
|
def main():
|
|
# Parse the command line flags
|
|
parser = create_parser()
|
|
args, unknown_args = parser.parse_known_args()
|
|
assert not unknown_args
|
|
benchmark_options = args.benchmark_options
|
|
|
|
if args.mode == 'benchmarks':
|
|
test_baseline = args.test_baseline[0].name
|
|
test_contender = args.test_contender[0].name
|
|
filter_baseline = ''
|
|
filter_contender = ''
|
|
|
|
# NOTE: if test_baseline == test_contender, you are analyzing the stdev
|
|
|
|
description = 'Comparing %s to %s' % (test_baseline, test_contender)
|
|
elif args.mode == 'filters':
|
|
test_baseline = args.test[0].name
|
|
test_contender = args.test[0].name
|
|
filter_baseline = args.filter_baseline[0]
|
|
filter_contender = args.filter_contender[0]
|
|
|
|
# NOTE: if filter_baseline == filter_contender, you are analyzing the
|
|
# stdev
|
|
|
|
description = 'Comparing %s to %s (from %s)' % (
|
|
filter_baseline, filter_contender, args.test[0].name)
|
|
elif args.mode == 'benchmarksfiltered':
|
|
test_baseline = args.test_baseline[0].name
|
|
test_contender = args.test_contender[0].name
|
|
filter_baseline = args.filter_baseline[0]
|
|
filter_contender = args.filter_contender[0]
|
|
|
|
# NOTE: if test_baseline == test_contender and
|
|
# filter_baseline == filter_contender, you are analyzing the stdev
|
|
|
|
description = 'Comparing %s (from %s) to %s (from %s)' % (
|
|
filter_baseline, test_baseline, filter_contender, test_contender)
|
|
else:
|
|
# should never happen
|
|
print("Unrecognized mode of operation: '%s'" % args.mode)
|
|
exit(1)
|
|
|
|
check_inputs(test_baseline, test_contender, benchmark_options)
|
|
|
|
options_baseline = []
|
|
options_contender = []
|
|
|
|
if filter_baseline and filter_contender:
|
|
options_baseline = ['--benchmark_filter=%s' % filter_baseline]
|
|
options_contender = ['--benchmark_filter=%s' % filter_contender]
|
|
|
|
# Run the benchmarks and report the results
|
|
json1 = json1_orig = gbench.util.run_or_load_benchmark(
|
|
test_baseline, benchmark_options + options_baseline)
|
|
json2 = json2_orig = gbench.util.run_or_load_benchmark(
|
|
test_contender, benchmark_options + options_contender)
|
|
|
|
# Now, filter the benchmarks so that the difference report can work
|
|
if filter_baseline and filter_contender:
|
|
replacement = '[%s vs. %s]' % (filter_baseline, filter_contender)
|
|
json1 = gbench.report.filter_benchmark(
|
|
json1_orig, filter_baseline, replacement)
|
|
json2 = gbench.report.filter_benchmark(
|
|
json2_orig, filter_contender, replacement)
|
|
|
|
# Diff and output
|
|
output_lines = gbench.report.generate_difference_report(json1, json2)
|
|
print(description)
|
|
for ln in output_lines:
|
|
print(ln)
|
|
|
|
|
|
import unittest
|
|
|
|
|
|
class TestParser(unittest.TestCase):
|
|
def setUp(self):
|
|
self.parser = create_parser()
|
|
testInputs = os.path.join(
|
|
os.path.dirname(
|
|
os.path.realpath(__file__)),
|
|
'gbench',
|
|
'Inputs')
|
|
self.testInput0 = os.path.join(testInputs, 'test_baseline_run1.json')
|
|
self.testInput1 = os.path.join(testInputs, 'test_baseline_run2.json')
|
|
|
|
def test_benchmarks_basic(self):
|
|
parsed = self.parser.parse_args(
|
|
['benchmarks', self.testInput0, self.testInput1])
|
|
self.assertEqual(parsed.mode, 'benchmarks')
|
|
self.assertEqual(parsed.test_baseline[0].name, self.testInput0)
|
|
self.assertEqual(parsed.test_contender[0].name, self.testInput1)
|
|
self.assertFalse(parsed.benchmark_options)
|
|
|
|
def test_benchmarks_with_remainder(self):
|
|
parsed = self.parser.parse_args(
|
|
['benchmarks', self.testInput0, self.testInput1, 'd'])
|
|
self.assertEqual(parsed.mode, 'benchmarks')
|
|
self.assertEqual(parsed.test_baseline[0].name, self.testInput0)
|
|
self.assertEqual(parsed.test_contender[0].name, self.testInput1)
|
|
self.assertEqual(parsed.benchmark_options, ['d'])
|
|
|
|
def test_benchmarks_with_remainder_after_doubleminus(self):
|
|
parsed = self.parser.parse_args(
|
|
['benchmarks', self.testInput0, self.testInput1, '--', 'e'])
|
|
self.assertEqual(parsed.mode, 'benchmarks')
|
|
self.assertEqual(parsed.test_baseline[0].name, self.testInput0)
|
|
self.assertEqual(parsed.test_contender[0].name, self.testInput1)
|
|
self.assertEqual(parsed.benchmark_options, ['e'])
|
|
|
|
def test_filters_basic(self):
|
|
parsed = self.parser.parse_args(
|
|
['filters', self.testInput0, 'c', 'd'])
|
|
self.assertEqual(parsed.mode, 'filters')
|
|
self.assertEqual(parsed.test[0].name, self.testInput0)
|
|
self.assertEqual(parsed.filter_baseline[0], 'c')
|
|
self.assertEqual(parsed.filter_contender[0], 'd')
|
|
self.assertFalse(parsed.benchmark_options)
|
|
|
|
def test_filters_with_remainder(self):
|
|
parsed = self.parser.parse_args(
|
|
['filters', self.testInput0, 'c', 'd', 'e'])
|
|
self.assertEqual(parsed.mode, 'filters')
|
|
self.assertEqual(parsed.test[0].name, self.testInput0)
|
|
self.assertEqual(parsed.filter_baseline[0], 'c')
|
|
self.assertEqual(parsed.filter_contender[0], 'd')
|
|
self.assertEqual(parsed.benchmark_options, ['e'])
|
|
|
|
def test_filters_with_remainder_after_doubleminus(self):
|
|
parsed = self.parser.parse_args(
|
|
['filters', self.testInput0, 'c', 'd', '--', 'f'])
|
|
self.assertEqual(parsed.mode, 'filters')
|
|
self.assertEqual(parsed.test[0].name, self.testInput0)
|
|
self.assertEqual(parsed.filter_baseline[0], 'c')
|
|
self.assertEqual(parsed.filter_contender[0], 'd')
|
|
self.assertEqual(parsed.benchmark_options, ['f'])
|
|
|
|
def test_benchmarksfiltered_basic(self):
|
|
parsed = self.parser.parse_args(
|
|
['benchmarksfiltered', self.testInput0, 'c', self.testInput1, 'e'])
|
|
self.assertEqual(parsed.mode, 'benchmarksfiltered')
|
|
self.assertEqual(parsed.test_baseline[0].name, self.testInput0)
|
|
self.assertEqual(parsed.filter_baseline[0], 'c')
|
|
self.assertEqual(parsed.test_contender[0].name, self.testInput1)
|
|
self.assertEqual(parsed.filter_contender[0], 'e')
|
|
self.assertFalse(parsed.benchmark_options)
|
|
|
|
def test_benchmarksfiltered_with_remainder(self):
|
|
parsed = self.parser.parse_args(
|
|
['benchmarksfiltered', self.testInput0, 'c', self.testInput1, 'e', 'f'])
|
|
self.assertEqual(parsed.mode, 'benchmarksfiltered')
|
|
self.assertEqual(parsed.test_baseline[0].name, self.testInput0)
|
|
self.assertEqual(parsed.filter_baseline[0], 'c')
|
|
self.assertEqual(parsed.test_contender[0].name, self.testInput1)
|
|
self.assertEqual(parsed.filter_contender[0], 'e')
|
|
self.assertEqual(parsed.benchmark_options[0], 'f')
|
|
|
|
def test_benchmarksfiltered_with_remainder_after_doubleminus(self):
|
|
parsed = self.parser.parse_args(
|
|
['benchmarksfiltered', self.testInput0, 'c', self.testInput1, 'e', '--', 'g'])
|
|
self.assertEqual(parsed.mode, 'benchmarksfiltered')
|
|
self.assertEqual(parsed.test_baseline[0].name, self.testInput0)
|
|
self.assertEqual(parsed.filter_baseline[0], 'c')
|
|
self.assertEqual(parsed.test_contender[0].name, self.testInput1)
|
|
self.assertEqual(parsed.filter_contender[0], 'e')
|
|
self.assertEqual(parsed.benchmark_options[0], 'g')
|
|
|
|
|
|
if __name__ == '__main__':
|
|
# unittest.main()
|
|
main()
|
|
|
|
# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4
|
|
# kate: tab-width: 4; replace-tabs on; indent-width 4; tab-indents: off;
|
|
# kate: indent-mode python; remove-trailing-spaces modified;
|