* perf: report unused inputs for the tar rule
The `mtree` spec passed to the `tar` rule very often selects a subset of the
inputs made available through the `srcs` attribute. In many cases, these
subsets do not break down cleanly along dependency-tree lines and there
is no simple way just pass less content to the `tar` rule.
One prominent example where this occurs is when constructing the tars
for OCI image layers. For instance when [building a Python-based
container image](https://github.com/bazel-contrib/rules_oci/blob/main/docs/python.md),
we might want to split the Python interpreter, third-party dependencies, and
application code into their own layers. This is done by [filtering the
`mtree_spec`](85cb2aaf8c/oci_python_image/py_layer.bzl (L39)).
However, in the operation to construct a `tar` from a subsetted mtree,
it is usually still an unsubsetted tree of `srcs` that gets passed. As
a result, the subset tarball is considered dependent upon a larger set
of sources than is strictly necessary.
This over-scoping runs counter to a very common objective associated with
breaking up an image into layers - isolating churn to a smaller slice of
the application. Because of the spurious relationships established in
Bazel's dependency graph, all tars get rebuilt anytime any content in
the application gets changed. Tar rebuilds can even be triggered by
changes to files that are completely filtered-out from all layers of the container.
Redundent creation of archive content is usually not too computationally
intensive, but the archives can be quite large in some cases, and
avoiding a rebuild might free up gigabytes of disk and/or network
bandwidth for
better use. In addition, eliminating the spurious dependency edges
removes erroneous constraints applied to the build action schedule;
these tend to push all Tar-building operations towards the end of a
build, even when some archive construction could be scheduled much earlier.
## Risk assessment and mitigation
The `unused_inputs_list` mechanism used to report spurious dependency
relationships is a bit difficult to use. Reporting an actually-used
input as unused can create difficult to diagnose problems down the line.
However, the behaviour of the `mtree`-based `tar` rule is sufficiently
simple and self-contained that I am fairly confident that this rule's
used/unused set can be determined accurately in a maintainable fashion.
Out of an abundance of caution I have gated this feature behind a
default-off flag. The `tar` rule will continue to operate as it had
before - typically over-reporting dependencies - unless the
`--@aspect_bazel_lib//lib:tar_compute_unused_inputs` flag is passed.
### Filter accuracy
The `vis` encoding used by the `mtree` format to resiliently handle path
names has a small amount of "play" to it - it is reversable but the
encoded representation of a string is not
unique. Two unequal encoded strings might decode to the same value; this
can happen when at least one of the encoded strings contains unnecessary
escapes that are nevertheless honoured by the decoder.
The unused-inputs set is determined using a filter that compares
`vis`-encoded strings. In the presence of non-canonically-encoded
paths, false-mismatches can lead to falsely reporting that an input is
unused.
The only `vis`-encoded path content that is under the control of callers
is the `mtree` content itself; all other `vis`-encoded strings are
constructed internally to this package, not exposed publicly, and are
all derived using the `lib/private/tar.bzl%_vis_encode` function; all of
these paths are expected to compare exactly. Additionally, it is expected that
many/most users will use this package's helpers (e.g. `mtree_spec`) when
crafting their mtree content; such content is also safe. It is only when
the user crafts their own mtree, or modifies an mtree spec's `content=`
fields' encoding in some way, that a risk of inaccurate reporting
arises. The chances for this are expected to be minor since this seems
like an inconvenient and not-particularly-useful thing for a user to go
out of their way to do.
* Also include other bsdtar toolchain files in keep set
* Add tri-state attribute to control unused-inputs behaviour
This control surface provides for granular control of the feature. The
interface is selected to mirror the common behaviour of `stamp` attributes.
* Add bzl_library level dep
* Update docs
* pre-commit
* Add reminder to change flag default on major-version bump
* Add note about how to make unused input computation exactly correct
* Add a test for unused_inputs listing
* Support alternate contents= form
This is accepted by bsdtar/libarchive. In fact `contents=` is the only of
the pair documented in `mtree(5)`; `content=` is an undocumented
alternate form supported by libarchive.
* Don't try to prune the unprunable
Bazel's interpretation of unused_inputs_list cannot accomodate certain
things in filenames. These are also likely to mess up our own
line-oriented protocol in the shellscript that produces this file.
Co-authored-by: Sahin Yort <thesayyn@gmail.com>
* Rerun docs update
---------
Co-authored-by: Sahin Yort <thesayyn@gmail.com>
* feat: support bzlmod repo name aliases in tarred runfiles
Under bzlmod, repos have aliases in addition to their canonical names;
in order for lookups using these canonical names to function properly,
a file name `_repo_mapping` is located in the root of the runfiles tree
and consulted to perform repo-name translation.
See: https://github.com/bazelbuild/proposals/blob/main/designs/2022-07-21-locating-runfiles-with-bzlmod.md
* pre-commit formatting
* Enable runfiles test under bzlmod
It works now.
* feat: add an option to not include copy_to_directory output in runfiles
* chore: docs update
* chore: don't repeat default
---------
Co-authored-by: Alex Eagle <alex@aspect.dev>
* chore: turn off bzlmod misguided warning
These are misinformed, the module resolver should be permitted to find an MVS solution
* chore: update golden
* fix: Set size to a default value as well as timeout.
Currently, we are unable to run our `write_source_files` tests in our pre-upload checks, because we have `--test_size_filter=small`, and setting `size` will attempt to set it on both the run rule and the test rule, the former being invalid.
* code review feedback
* chore: fix one more test that should use size for defaulting
---------
Co-authored-by: Alex Eagle <alex@aspect.dev>
* fix(tar): append slash to top-level directory mtree entries
bsdtar's mtree format has a quirk wherein entries without "/" in their
first word are treated as "relative" entries, and "relative" directories
will cause tar to "change directory" into the declared directory entry.
If such a directory is followed by a "relative" entry, then the file
will be created within the directory, instead of at top-level as
expected. To mitigate, we append a slash to top-level directory entries.
Fixes #851.
* chore: golden files have BINDIR placeholder
---------
Co-authored-by: Alex Eagle <alex@aspect.dev>
* Fix #667: Dir hidden files in write_source_file.
Copy and manage hidden files starting with "." in write_source_files.
Previously these files were not supported if they were in the top level
of the directory to copy.
* Add test and fix error messages from cp, chmod.
* Also fix executable dir case.
* Fix issue with copying directory rather than contents.
* refactor(release): switch release integrity to be dynamic
This matches rules_py as documented by
https://blog.aspect.build/releasing-bazel-rulesets-rust
It has the benefit that developers no longer get yelled at to vendor some updated integrity hashes into bazel-lib every time they touch the Go sources.
* refactor: echo should produce trailing newline
* chore: bump action-gh-release to avoid Node 16 warning
* chore: update test that is sensitive to compilation mode
We now only use --compilation_mode=opt when cutting a release
* Add platform_transition_test
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Set target platform constraints for tests
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Update docs
Signed-off-by: Thomas Lam <thomaslam@canva.com>
---------
Signed-off-by: Thomas Lam <thomaslam@canva.com>
This teaches the jq rule about a `data` attribute and adds conditional
support to do bazel location expansion. The expansion is disabled by
default because it likely would cause compat concerns (since jq can take
arbitrary text as input args).
In practice we've disabled this setting at some clients, because there's not a surrounding workflow to find these targets are being silently dropped from CI, nor run them on another schedule.
We can re-introduce something better when we add Test Selection to Aspect Workflows.
* Reproduction for mtree_spec behaving different in other repo
* Fix the bug (but not the test)
* Fix the test
* Rename mtree_spec e2e dir to tar
* Fix bzlmod tests
* Remove unnecessary skylib dependency from e2e repos
* Add e2e tests to matrix
* Replace e2e test with a normal test
To do this, we somewhat abuse the skylib dependency, but that's
probalby OK.
* chore: add windows binaries
* chore: fix/exclude windows brokenness
* chore: try to see why diff tests fail on windows
* fix: rm bazelisk rc again for windows
* fix: try our own diff_test
* chore: use only our own diff_test
Peter Lobsinger