bazel-skylib/docs/unittest_doc.md

Unit testing support.

Unlike most Skylib files, this exports four modules:

• unittest contains functions to declare and define unit tests for ordinary Starlark functions;
• analysistest contains functions to declare and define tests for analysis phase behavior of a rule, such as a given target's providers or registered actions;
• loadingtest contains functions to declare and define tests for loading phase behavior, such as macros and native.*;
• asserts contains the assertions used within tests.

See https://bazel.build/extending/concepts for background about macros, rules, and the different phases of a build.

unittest_toolchain

unittest_toolchain(name, escape_chars_with, escape_other_chars_with, failure_templ, file_ext,
                   join_on, success_templ)

ATTRIBUTES

Name	Description	Type	Mandatory	Default
name	A unique name for this target.	Name	required
escape_chars_with	Dictionary of characters that need escaping in test failure message to prefix appended to escape those characters. For example, {"%": "%", ">": "^"} would replace % with %% and > with ^> in the failure message before that is included in success_templ.	Dictionary: String -> String	optional	{}
escape_other_chars_with	String to prefix every character in test failure message which is not a key in escape_chars_with before including that in success_templ. For example, "" would prefix every character in the failure message (except those in the keys of escape_chars_with) with \.	String	optional	""
failure_templ	Test script template with a single %s. That placeholder is replaced with the lines in the failure message joined with the string specified in join_with. The resulting script should print the failure message and exit with non-zero status.	String	required
file_ext	File extension for test script, including leading dot.	String	required
join_on	String used to join the lines in the failure message before including the resulting string in the script specified in failure_templ.	String	required
success_templ	Test script generated when the test passes. Should exit with status 0.	String	required

analysistest.begin

analysistest.begin(ctx)

Begins an analysis test.

This should be the first function called in an analysis test implementation function. It initializes a "test environment" that is used to collect assertion failures so that they can be reported and logged at the end of the test.

PARAMETERS

Name	Description	Default Value
ctx	The Starlark context. Pass the implementation function's ctx argument in verbatim.	none

RETURNS

A test environment struct that must be passed to assertions and finally to analysistest.end. Do not rely on internal details about the fields in this struct as it may change.

analysistest.end

analysistest.end(env)

Ends an analysis test and logs the results.

This must be called and returned at the end of an analysis test implementation function so that the results are reported.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by analysistest.begin.	none

RETURNS

A list of providers needed to automatically register the analysis test result.

analysistest.fail

analysistest.fail(env, msg)

Unconditionally causes the current test to fail.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
msg	The message to log describing the failure.	none

analysistest.make

analysistest.make(impl, expect_failure, attrs, fragments, config_settings,
                  extra_target_under_test_aspects, doc)

Creates an analysis test rule from its implementation function.

An analysis test verifies the behavior of a "real" rule target by examining and asserting on the providers given by the real target.

Each analysis test is defined in an implementation function that must then be associated with a rule so that a target can be built. This function handles the boilerplate to create and return a test rule and captures the implementation function's name so that it can be printed in test feedback.

An example of an analysis test:

def _your_test(ctx):
  env = analysistest.begin(ctx)

  # Assert statements go here

  return analysistest.end(env)

your_test = analysistest.make(_your_test)

Recall that names of test rules must end in _test.

PARAMETERS

Name	Description	Default Value
impl	The implementation function of the unit test.	none
expect_failure	If true, the analysis test will expect the target_under_test to fail. Assertions can be made on the underlying failure using asserts.expect_failure	False
attrs	An optional dictionary to supplement the attrs passed to the unit test's rule() constructor.	{}
fragments	An optional list of fragment names that can be used to give rules access to language-specific parts of configuration.	[]
config_settings	A dictionary of configuration settings to change for the target under test and its dependencies. This may be used to essentially change 'build flags' for the target under test, and may thus be utilized to test multiple targets with different flags in a single build	{}
extra_target_under_test_aspects	An optional list of aspects to apply to the target_under_test in addition to those set up by default for the test harness itself.	[]
doc	A description of the rule that can be extracted by documentation generating tools.	""

RETURNS

A rule definition that should be stored in a global whose name ends in _test.

analysistest.target_actions

analysistest.target_actions(env)

Returns a list of actions registered by the target under test.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by analysistest.begin.	none

RETURNS

A list of actions registered by the target under test

analysistest.target_bin_dir_path

analysistest.target_bin_dir_path(env)

Returns ctx.bin_dir.path for the target under test.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by analysistest.begin.	none

RETURNS

Output bin dir path string.

analysistest.target_under_test

analysistest.target_under_test(env)

Returns the target under test.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by analysistest.begin.	none

RETURNS

The target under test.

asserts.equals

asserts.equals(env, expected, actual, msg)

Asserts that the given expected and actual values are equal.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
expected	The expected value of some computation.	none
actual	The actual value returned by some computation.	none
msg	An optional message that will be printed that describes the failure. If omitted, a default will be used.	None

asserts.expect_failure

asserts.expect_failure(env, expected_failure_msg)

Asserts that the target under test has failed with a given error message.

This requires that the analysis test is created with analysistest.make() and expect_failures = True is specified.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by analysistest.begin.	none
expected_failure_msg	The error message to expect as a result of analysis failures.	""

asserts.false

asserts.false(env, condition, msg)

Asserts that the given condition is false.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
condition	A value that will be evaluated in a Boolean context.	none
msg	An optional message that will be printed that describes the failure. If omitted, a default will be used.	"Expected condition to be false, but was true."

asserts.new_set_equals

asserts.new_set_equals(env, expected, actual, msg)

Asserts that the given expected and actual sets are equal.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
expected	The expected set resulting from some computation.	none
actual	The actual set returned by some computation.	none
msg	An optional message that will be printed that describes the failure. If omitted, a default will be used.	None

asserts.set_equals

asserts.set_equals(env, expected, actual, msg)

Asserts that the given expected and actual sets are equal.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
expected	The expected set resulting from some computation.	none
actual	The actual set returned by some computation.	none
msg	An optional message that will be printed that describes the failure. If omitted, a default will be used.	None

asserts.true

asserts.true(env, condition, msg)

Asserts that the given condition is true.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
condition	A value that will be evaluated in a Boolean context.	none
msg	An optional message that will be printed that describes the failure. If omitted, a default will be used.	"Expected condition to be true, but was false."

loadingtest.equals

loadingtest.equals(env, test_case, expected, actual)

Creates a test case for asserting state at LOADING phase.

PARAMETERS

Name	Description	Default Value
env	Loading test env created from loadingtest.make	none
test_case	Name of the test case	none
expected	Expected value to test	none
actual	Actual value received.	none

RETURNS

None, creates test case

loadingtest.make

loadingtest.make(name)

Creates a loading phase test environment and test_suite.

PARAMETERS

Name	Description	Default Value
name	name of the suite of tests to create	none

RETURNS

loading phase environment passed to other loadingtest functions

register_unittest_toolchains

register_unittest_toolchains()

Registers the toolchains for unittest users.

unittest.begin

unittest.begin(ctx)

Begins a unit test.

This should be the first function called in a unit test implementation function. It initializes a "test environment" that is used to collect assertion failures so that they can be reported and logged at the end of the test.

PARAMETERS

Name	Description	Default Value
ctx	The Starlark context. Pass the implementation function's ctx argument in verbatim.	none

RETURNS

A test environment struct that must be passed to assertions and finally to unittest.end. Do not rely on internal details about the fields in this struct as it may change.

unittest.end

unittest.end(env)

Ends a unit test and logs the results.

This must be called and returned at the end of a unit test implementation function so that the results are reported.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none

RETURNS

A list of providers needed to automatically register the test result.

unittest.fail

unittest.fail(env, msg)

Unconditionally causes the current test to fail.

PARAMETERS

Name	Description	Default Value
env	The test environment returned by unittest.begin.	none
msg	The message to log describing the failure.	none

unittest.make

unittest.make(impl, attrs, doc, toolchains)

Creates a unit test rule from its implementation function.

Each unit test is defined in an implementation function that must then be associated with a rule so that a target can be built. This function handles the boilerplate to create and return a test rule and captures the implementation function's name so that it can be printed in test feedback.

The optional attrs argument can be used to define dependencies for this test, in order to form unit tests of rules.

The optional toolchains argument can be used to define toolchain dependencies for this test.

An example of a unit test:

def _your_test(ctx):
  env = unittest.begin(ctx)

  # Assert statements go here

  return unittest.end(env)

your_test = unittest.make(_your_test)

Recall that names of test rules must end in _test.

PARAMETERS

Name	Description	Default Value
impl	The implementation function of the unit test.	none
attrs	An optional dictionary to supplement the attrs passed to the unit test's rule() constructor.	{}
doc	A description of the rule that can be extracted by documentation generating tools.	""
toolchains	An optional list to supplement the toolchains passed to the unit test's rule() constructor.	[]

RETURNS

A rule definition that should be stored in a global whose name ends in _test.

unittest.suite

unittest.suite(name, test_rules)

Defines a test_suite target that contains multiple tests.

After defining your test rules in a .bzl file, you need to create targets from those rules so that blaze test can execute them. Doing this manually in a BUILD file would consist of listing each test in your load statement and then creating each target one by one. To reduce duplication, we recommend writing a macro in your .bzl file to instantiate all targets, and calling that macro from your BUILD file so you only have to load one symbol.

You can use this function to create the targets and wrap them in a single test_suite target. If a test rule requires no arguments, you can simply list it as an argument. If you wish to supply attributes explicitly, you can do so using partial.make(). For instance, in your .bzl file, you could write:

def your_test_suite():
  unittest.suite(
      "your_test_suite",
      your_test,
      your_other_test,
      partial.make(yet_another_test, timeout = "short"),
  )

Then, in your BUILD file, simply load the macro and invoke it to have all of the targets created:

load("//path/to/your/package:tests.bzl", "your_test_suite")
your_test_suite()

If you pass N unit test rules to unittest.suite, N + 1 targets will be created: a test_suite target named ${name} (where ${name} is the name argument passed in here) and targets named ${name}_test_${i}, where ${i} is the index of the test in the test_rules list, which is used to uniquely name each target.

PARAMETERS

Name	Description	Default Value
name	The name of the test_suite target, and the prefix of all the test target names.	none
test_rules	A list of test rules defines by unittest.test.	none