benchmark/docs/AssemblyTests.md

# Assembly Tests

The Benchmark library provides a number of functions whose primary
purpose in to affect assembly generation, including `DoNotOptimize`
and `ClobberMemory`. In addition there are other functions,
such as `KeepRunning`, for which generating good assembly is paramount.

For these functions it's important to have tests that verify the
correctness and quality of the implementation. This requires testing
the code generated by the compiler.

This document describes how the Benchmark library tests compiler output,
as well as how to properly write new tests.


## Anatomy of a Test

Writing a test has two steps:

* Write the code you want to generate assembly for.
* Add `// CHECK` lines to match against the verified assembly.

Example:
```c++

// CHECK-LABEL: test_add:
extern "C" int test_add() {
    extern int ExternInt;
    return ExternInt + 1;

    // CHECK: movl ExternInt(%rip), %eax
    // CHECK: addl %eax
    // CHECK: ret
}

```

#### LLVM Filecheck

[LLVM's Filecheck](https://llvm.org/docs/CommandGuide/FileCheck.html)
is used to test the generated assembly against the `// CHECK` lines
specified in the tests source file. Please see the documentation
linked above for information on how to write `CHECK` directives.

#### Tips and Tricks:

* Tests should match the minimal amount of output required to establish
correctness. `CHECK` directives don't have to match on the exact next line
after the previous match, so tests should omit checks for unimportant
bits of assembly. ([`CHECK-NEXT`](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-next-directive)
can be used to ensure a match occurs exactly after the previous match).

* The tests are compiled with `-O3 -g0`. So we're only testing the
optimized output.

* The assembly output is further cleaned up using `tools/strip_asm.py`.
This removes comments, assembler directives, and unused labels before
the test is run.

* The generated and stripped assembly file for a test is output under
`<build-directory>/test/<test-name>.s`

* Filecheck supports using [`CHECK` prefixes](https://llvm.org/docs/CommandGuide/FileCheck.html#cmdoption-check-prefixes)
to specify lines that should only match in certain situations.
The Benchmark tests use `CHECK-CLANG` and `CHECK-GNU` for lines that
are only expected to match Clang or GCC's output respectively. Normal
`CHECK` lines match against all compilers. (Note: `CHECK-NOT` and
`CHECK-LABEL` are NOT prefixes. They are versions of non-prefixed
`CHECK` lines)

* Use `extern "C"` to disable name mangling for specific functions. This
makes them easier to name in the `CHECK` lines.


## Problems Writing Portable Tests

Writing tests which check the code generated by a compiler are
inherently non-portable. Different compilers and even different compiler
versions may generate entirely different code. The Benchmark tests
must tolerate this.

LLVM Filecheck provides a number of mechanisms to help write
"more portable" tests; including [matching using regular expressions](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-pattern-matching-syntax),
allowing the creation of [named variables](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-variables)
for later matching, and [checking non-sequential matches](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-dag-directive).

#### Capturing Variables

For example, say GCC stores a variable in a register but Clang stores
it in memory. To write a test that tolerates both cases we "capture"
the destination of the store, and then use the captured expression
to write the remainder of the test.

```c++
// CHECK-LABEL: test_div_no_op_into_shr:
extern "C" void test_div_no_op_into_shr(int value) {
    int divisor = 2;
    benchmark::DoNotOptimize(divisor); // hide the value from the optimizer
    return value / divisor;

    // CHECK: movl $2, [[DEST:.*]]
    // CHECK: idivl [[DEST]]
    // CHECK: ret
}
```

#### Using Regular Expressions to Match Differing Output

Often tests require testing assembly lines which may subtly differ
between compilers or compiler versions. A common example of this
is matching stack frame addresses. In this case regular expressions
can be used to match the differing bits of output. For example:

<!-- {% raw %} -->
```c++
int ExternInt;
struct Point { int x, y, z; };

// CHECK-LABEL: test_store_point:
extern "C" void test_store_point() {
    Point p{ExternInt, ExternInt, ExternInt};
    benchmark::DoNotOptimize(p);

    // CHECK: movl ExternInt(%rip), %eax
    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
    // CHECK: movl %eax, -{{[0-9]+}}(%rsp)
    // CHECK: ret
}
```
<!-- {% endraw %} -->

## Current Requirements and Limitations

The tests require Filecheck to be installed along the `PATH` of the
build machine. Otherwise the tests will be disabled.

Additionally, as mentioned in the previous section, codegen tests are
inherently non-portable. Currently the tests are limited to:

* x86_64 targets.
* Compiled with GCC or Clang

Further work could be done, at least on a limited basis, to extend the
tests to other architectures and compilers (using `CHECK` prefixes).

Furthermore, the tests fail for builds which specify additional flags
that modify code generation, including `--coverage` or `-fsanitize=`.