fix: always include files from the same workspace as the build target in `copy_to_directory()` (#360)

* fix: always include files from the same workspace as the build target in `copy_to_directory`

Fixes #359.

This updates the `copy_to_directory` tool to accept a workspace name representing the workspace of the target it is executing under. Any files in this workspace are automatically included, regardless of the `include_external_repositories` option. This makes it support usage within an external target (such as `@wksp//:dir`).

* test: add e2e test which uses `copy_to_directory` within an external workspace

Refs #359.

This should catch regressions where no files are copied when built within an external workspace and not using `include_external_repositories`.

* ci: fix stray workspace refs

---------

Co-authored-by: Alex Eagle <alex@aspect.dev>
This commit is contained in:
Douglas Parker 2023-10-10 07:16:13 -07:00 committed by Alex Eagle
parent 472bf9b122
commit 323329f119
16 changed files with 145 additions and 3 deletions

View File

@ -13,6 +13,10 @@ workspaces:
- e2e/copy_to_directory: - e2e/copy_to_directory:
gazelle: gazelle:
without: true without: true
- e2e/external_copy_to_directory:
gazelle:
without: true
bazel: bazel:
flags: flags:
- --remote_download_minimal - --remote_download_minimal

View File

@ -88,6 +88,7 @@ jobs:
- "." - "."
- "e2e/coreutils" - "e2e/coreutils"
- "e2e/copy_to_directory" - "e2e/copy_to_directory"
- "e2e/external_copy_to_directory"
- "e2e/smoke" - "e2e/smoke"
exclude: exclude:
# Don't test MacOS with RBE to minimize MacOS minutes (billed at 10X) # Don't test MacOS with RBE to minimize MacOS minutes (billed at 10X)

View File

@ -59,7 +59,7 @@ for more information on supported globbing patterns.
| <a id="copy_to_directory-exclude_srcs_packages"></a>exclude_srcs_packages | List of Bazel packages (with glob support) to exclude from output directory.<br><br>Files in srcs are not copied to the output directory if the Bazel package of the file matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators. A first character of <code>"."</code> will be replaced by the target's package path.<br><br>Files that have do not have matching Bazel packages are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | [] | | <a id="copy_to_directory-exclude_srcs_packages"></a>exclude_srcs_packages | List of Bazel packages (with glob support) to exclude from output directory.<br><br>Files in srcs are not copied to the output directory if the Bazel package of the file matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators. A first character of <code>"."</code> will be replaced by the target's package path.<br><br>Files that have do not have matching Bazel packages are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | [] |
| <a id="copy_to_directory-exclude_srcs_patterns"></a>exclude_srcs_patterns | List of paths (with glob support) to exclude from output directory.<br><br>Files in srcs are not copied to the output directory if their output directory path, after applying <code>root_paths</code>, matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators.<br><br>Files that do not have matching output directory paths are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | [] | | <a id="copy_to_directory-exclude_srcs_patterns"></a>exclude_srcs_patterns | List of paths (with glob support) to exclude from output directory.<br><br>Files in srcs are not copied to the output directory if their output directory path, after applying <code>root_paths</code>, matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators.<br><br>Files that do not have matching output directory paths are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | [] |
| <a id="copy_to_directory-hardlink"></a>hardlink | Controls when to use hardlinks to files instead of making copies.<br><br>Creating hardlinks is much faster than making copies of files with the caveat that hardlinks share file permissions with their source.<br><br>Since Bazel removes write permissions on files in the output tree after an action completes, hardlinks to source files are not recommended since write permissions will be inadvertently removed from sources files.<br><br>- <code>auto</code>: hardlinks are used for generated files already in the output tree - <code>off</code>: all files are copied - <code>on</code>: hardlinks are used for all files (not recommended) | String | optional | "auto" | | <a id="copy_to_directory-hardlink"></a>hardlink | Controls when to use hardlinks to files instead of making copies.<br><br>Creating hardlinks is much faster than making copies of files with the caveat that hardlinks share file permissions with their source.<br><br>Since Bazel removes write permissions on files in the output tree after an action completes, hardlinks to source files are not recommended since write permissions will be inadvertently removed from sources files.<br><br>- <code>auto</code>: hardlinks are used for generated files already in the output tree - <code>off</code>: all files are copied - <code>on</code>: hardlinks are used for all files (not recommended) | String | optional | "auto" |
| <a id="copy_to_directory-include_external_repositories"></a>include_external_repositories | List of external repository names (with glob support) to include in the output directory.<br><br>Files from external repositories are only copied into the output directory if the external repository they come from matches one of the external repository patterns specified.<br><br>When copied from an external repository, the file path in the output directory defaults to the file's path within the external repository. The external repository name is _not_ included in that path.<br><br>For example, the following copies <code>@external_repo//path/to:file</code> to <code>path/to/file</code> within the output directory.<br><br><pre><code> copy_to_directory( name = "dir", include_external_repositories = ["external_*"], srcs = ["@external_repo//path/to:file"], ) </code></pre><br><br>Files that come from matching external are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be. The external repository name of the file from an external repository is not included in the output directory path and is considered in subsequent filters and transformations.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | [] | | <a id="copy_to_directory-include_external_repositories"></a>include_external_repositories | List of external repository names (with glob support) to include in the output directory.<br><br>Files from external repositories are only copied into the output directory if the external repository they come from matches one of the external repository patterns specified or if they are in the same external repository as this target.<br><br>When copied from an external repository, the file path in the output directory defaults to the file's path within the external repository. The external repository name is _not_ included in that path.<br><br>For example, the following copies <code>@external_repo//path/to:file</code> to <code>path/to/file</code> within the output directory.<br><br><pre><code> copy_to_directory( name = "dir", include_external_repositories = ["external_*"], srcs = ["@external_repo//path/to:file"], ) </code></pre><br><br>Files that come from matching external are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be. The external repository name of the file from an external repository is not included in the output directory path and is considered in subsequent filters and transformations.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | [] |
| <a id="copy_to_directory-include_srcs_packages"></a>include_srcs_packages | List of Bazel packages (with glob support) to include in output directory.<br><br>Files in srcs are only copied to the output directory if the Bazel package of the file matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators. A first character of <code>"."</code> will be replaced by the target's package path.<br><br>Defaults to <code>["**"]</code> which includes sources from all packages.<br><br>Files that have matching Bazel packages are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | ["**"] | | <a id="copy_to_directory-include_srcs_packages"></a>include_srcs_packages | List of Bazel packages (with glob support) to include in output directory.<br><br>Files in srcs are only copied to the output directory if the Bazel package of the file matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators. A first character of <code>"."</code> will be replaced by the target's package path.<br><br>Defaults to <code>["**"]</code> which includes sources from all packages.<br><br>Files that have matching Bazel packages are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | ["**"] |
| <a id="copy_to_directory-include_srcs_patterns"></a>include_srcs_patterns | List of paths (with glob support) to include in output directory.<br><br>Files in srcs are only copied to the output directory if their output directory path, after applying <code>root_paths</code>, matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators.<br><br>Defaults to <code>["**"]</code> which includes all sources.<br><br>Files that have matching output directory paths are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | ["**"] | | <a id="copy_to_directory-include_srcs_patterns"></a>include_srcs_patterns | List of paths (with glob support) to include in output directory.<br><br>Files in srcs are only copied to the output directory if their output directory path, after applying <code>root_paths</code>, matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators.<br><br>Defaults to <code>["**"]</code> which includes all sources.<br><br>Files that have matching output directory paths are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above). | List of strings | optional | ["**"] |
| <a id="copy_to_directory-out"></a>out | Path of the output directory, relative to this package.<br><br>If not set, the name of the target is used. | String | optional | "" | | <a id="copy_to_directory-out"></a>out | Path of the output directory, relative to this package.<br><br>If not set, the name of the target is used. | String | optional | "" |

View File

@ -0,0 +1 @@
app/

View File

View File

@ -0,0 +1 @@
../../.bazelversion

View File

@ -0,0 +1,13 @@
load("@bazel_skylib//rules:build_test.bzl", "build_test")
load(":directory.bzl", "directory")
directory(
name = "dir",
srcs = ["foo.txt"],
visibility = ["//visibility:public"],
)
build_test(
name = "test",
targets = [":dir"],
)

View File

@ -0,0 +1,28 @@
workspace(name = "external_copy_to_directory")
local_repository(
name = "aspect_bazel_lib",
path = "../..",
)
load("@aspect_bazel_lib//:internal_deps.bzl", "bazel_lib_internal_deps")
bazel_lib_internal_deps()
load("@aspect_bazel_lib//lib:repositories.bzl", "aspect_bazel_lib_dependencies")
aspect_bazel_lib_dependencies()
load("@aspect_bazel_lib//:deps.bzl", "go_dependencies")
go_dependencies()
load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
go_rules_dependencies()
go_register_toolchains(version = "1.18.3")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
gazelle_dependencies()

View File

@ -0,0 +1 @@
../../../.bazelversion

View File

@ -0,0 +1,6 @@
sh_test(
name = "test",
srcs = ["test.sh"],
data = ["@external_copy_to_directory//:dir"],
deps = ["@bazel_tools//tools/bash/runfiles"],
)

View File

@ -0,0 +1,31 @@
local_repository(
name = "external_copy_to_directory",
path = "..",
)
local_repository(
name = "aspect_bazel_lib",
path = "../../..",
)
load("@aspect_bazel_lib//:internal_deps.bzl", "bazel_lib_internal_deps")
bazel_lib_internal_deps()
load("@aspect_bazel_lib//lib:repositories.bzl", "aspect_bazel_lib_dependencies")
aspect_bazel_lib_dependencies()
load("@aspect_bazel_lib//:deps.bzl", "go_dependencies")
go_dependencies()
load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
go_rules_dependencies()
go_register_toolchains(version = "1.18.3")
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
gazelle_dependencies()

View File

@ -0,0 +1,19 @@
# --- begin runfiles.bash initialization v2 ---
# Copy-pasted from the Bazel Bash runfiles library v2.
set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
source "$0.runfiles/$f" 2>/dev/null || \
source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
{ echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
# --- end runfiles.bash initialization v2 ---
# Read external directory and make sure it exists with a file.
readonly DIR=$(rlocation external_copy_to_directory/dir/)
readonly FILES=$(ls "${DIR}" | wc -l)
if [[ ${FILES} != 1 ]]; then
echo "Expected exactly 1 file under external directory, but found ${FILES}:" >&2
ls "${DIR}" >&2
exit 1
fi

View File

@ -0,0 +1,36 @@
"""Test rule executing `copy_to_directory_bin_action`."""
load("@aspect_bazel_lib//lib:copy_to_directory.bzl", "copy_to_directory_bin_action")
def _directory_impl(ctx):
dst = ctx.actions.declare_directory(ctx.attr.name)
copy_to_directory_bin_action(
ctx,
name = ctx.attr.name,
copy_to_directory_bin = ctx.executable._tool,
dst = dst,
files = ctx.files.srcs,
verbose = True,
)
return DefaultInfo(files = depset([dst]))
directory = rule(
implementation = _directory_impl,
attrs = {
"srcs": attr.label_list(
mandatory = True,
allow_files = True,
),
"_tool": attr.label(
executable = True,
cfg = "exec",
default = "@aspect_bazel_lib//tools/copy_to_directory",
),
},
doc = """
Copies the given source files to a directory with
`copy_to_directory_bin_action()`.
""",
)

View File

@ -0,0 +1 @@
foo

View File

@ -79,7 +79,7 @@ Globs are supported (see rule docstring above).
Files from external repositories are only copied into the output directory if Files from external repositories are only copied into the output directory if
the external repository they come from matches one of the external repository patterns the external repository they come from matches one of the external repository patterns
specified. specified or if they are in the same external repository as this target.
When copied from an external repository, the file path in the output directory When copied from an external repository, the file path in the output directory
defaults to the file's path within the external repository. The external repository defaults to the file's path within the external repository. The external repository
@ -492,7 +492,7 @@ def copy_to_directory_bin_action(
inputs = file_inputs + [config_file], inputs = file_inputs + [config_file],
outputs = [dst], outputs = [dst],
executable = copy_to_directory_bin, executable = copy_to_directory_bin,
arguments = [config_file.path], arguments = [config_file.path, ctx.label.workspace_name],
mnemonic = "CopyToDirectory", mnemonic = "CopyToDirectory",
progress_message = "Copying files to directory %s" % _progress_path(dst), progress_message = "Copying files to directory %s" % _progress_path(dst),
execution_requirements = override_execution_requirements or execution_requirements_for_copy(ctx), execution_requirements = override_execution_requirements or execution_requirements_for_copy(ctx),