fix: always include files from the same workspace as the build target in `copy_to_directory()` (#360)

* fix: always include files from the same workspace as the build target in `copy_to_directory`

Fixes #359.

This updates the `copy_to_directory` tool to accept a workspace name representing the workspace of the target it is executing under. Any files in this workspace are automatically included, regardless of the `include_external_repositories` option. This makes it support usage within an external target (such as `@wksp//:dir`).

* test: add e2e test which uses `copy_to_directory` within an external workspace

Refs #359.

This should catch regressions where no files are copied when built within an external workspace and not using `include_external_repositories`.

* ci: fix stray workspace refs

---------

Co-authored-by: Alex Eagle <alex@aspect.dev>

.aspect/workflows/config.yaml

@@ -13,6 +13,10 @@ workspaces:
   - e2e/copy_to_directory:
       gazelle:
         without: true
+  - e2e/external_copy_to_directory:
+      gazelle:
+        without: true
+
 bazel:
   flags:
     - --remote_download_minimal

.github/workflows/ci.yaml

@@ -88,6 +88,7 @@ jobs:
           - "."
           - "e2e/coreutils"
           - "e2e/copy_to_directory"
+          - "e2e/external_copy_to_directory"
           - "e2e/smoke"
         exclude:
           # Don't test MacOS with RBE to minimize MacOS minutes (billed at 10X)

docs/copy_to_directory.md

@@ -59,7 +59,7 @@ for more information on supported globbing patterns.
 | <a id="copy_to_directory-exclude_srcs_packages"></a>exclude_srcs_packages |  List of Bazel packages (with glob support) to exclude from output directory.<br><br>Files in srcs are not copied to the output directory if the Bazel package of the file matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators. A first character of <code>"."</code> will be replaced by the target's package path.<br><br>Files that have do not have matching Bazel packages are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above).   | List of strings | optional | [] |
 | <a id="copy_to_directory-exclude_srcs_patterns"></a>exclude_srcs_patterns |  List of paths (with glob support) to exclude from output directory.<br><br>Files in srcs are not copied to the output directory if their output directory path, after applying <code>root_paths</code>, matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators.<br><br>Files that do not have matching output directory paths are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above).   | List of strings | optional | [] |
 | <a id="copy_to_directory-hardlink"></a>hardlink |  Controls when to use hardlinks to files instead of making copies.<br><br>Creating hardlinks is much faster than making copies of files with the caveat that hardlinks share file permissions with their source.<br><br>Since Bazel removes write permissions on files in the output tree after an action completes, hardlinks to source files are not recommended since write permissions will be inadvertently removed from sources files.<br><br>- <code>auto</code>: hardlinks are used for generated files already in the output tree - <code>off</code>: all files are copied - <code>on</code>: hardlinks are used for all files (not recommended)   | String | optional | "auto" |
-| <a id="copy_to_directory-include_external_repositories"></a>include_external_repositories |  List of external repository names (with glob support) to include in the output directory.<br><br>Files from external repositories are only copied into the output directory if the external repository they come from matches one of the external repository patterns specified.<br><br>When copied from an external repository, the file path in the output directory defaults to the file's path within the external repository. The external repository name is _not_ included in that path.<br><br>For example, the following copies <code>@external_repo//path/to:file</code> to <code>path/to/file</code> within the output directory.<br><br><pre><code> copy_to_directory(     name = "dir",     include_external_repositories = ["external_*"],     srcs = ["@external_repo//path/to:file"], ) </code></pre><br><br>Files that come from matching external are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be. The external repository name of the file from an external repository is not included in the output directory path and is considered in subsequent filters and transformations.<br><br>Globs are supported (see rule docstring above).   | List of strings | optional | [] |
+| <a id="copy_to_directory-include_external_repositories"></a>include_external_repositories |  List of external repository names (with glob support) to include in the output directory.<br><br>Files from external repositories are only copied into the output directory if the external repository they come from matches one of the external repository patterns specified or if they are in the same external repository as this target.<br><br>When copied from an external repository, the file path in the output directory defaults to the file's path within the external repository. The external repository name is _not_ included in that path.<br><br>For example, the following copies <code>@external_repo//path/to:file</code> to <code>path/to/file</code> within the output directory.<br><br><pre><code> copy_to_directory(     name = "dir",     include_external_repositories = ["external_*"],     srcs = ["@external_repo//path/to:file"], ) </code></pre><br><br>Files that come from matching external are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be. The external repository name of the file from an external repository is not included in the output directory path and is considered in subsequent filters and transformations.<br><br>Globs are supported (see rule docstring above).   | List of strings | optional | [] |
 | <a id="copy_to_directory-include_srcs_packages"></a>include_srcs_packages |  List of Bazel packages (with glob support) to include in output directory.<br><br>Files in srcs are only copied to the output directory if the Bazel package of the file matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators. A first character of <code>"."</code> will be replaced by the target's package path.<br><br>Defaults to <code>["**"]</code> which includes sources from all packages.<br><br>Files that have matching Bazel packages are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above).   | List of strings | optional | ["**"] |
 | <a id="copy_to_directory-include_srcs_patterns"></a>include_srcs_patterns |  List of paths (with glob support) to include in output directory.<br><br>Files in srcs are only copied to the output directory if their output directory path, after applying <code>root_paths</code>, matches one of the patterns specified.<br><br>Forward slashes (<code>/</code>) should be used as path separators.<br><br>Defaults to <code>["**"]</code> which includes all sources.<br><br>Files that have matching output directory paths are subject to subsequent filters and transformations to determine if they are copied and what their path in the output directory will be.<br><br>Globs are supported (see rule docstring above).   | List of strings | optional | ["**"] |
 | <a id="copy_to_directory-out"></a>out |  Path of the output directory, relative to this package.<br><br>If not set, the name of the target is used.   | String | optional | "" |

e2e/external_copy_to_directory/.bazelignore

@@ -0,0 +1 @@
+app/

e2e/external_copy_to_directory/.bazelversion

@@ -0,0 +1 @@
+../../.bazelversion

e2e/external_copy_to_directory/BUILD.bazel

@@ -0,0 +1,13 @@
+load("@bazel_skylib//rules:build_test.bzl", "build_test")
+load(":directory.bzl", "directory")
+
+directory(
+    name = "dir",
+    srcs = ["foo.txt"],
+    visibility = ["//visibility:public"],
+)
+
+build_test(
+    name = "test",
+    targets = [":dir"],
+)

e2e/external_copy_to_directory/WORKSPACE

@@ -0,0 +1,28 @@
+workspace(name = "external_copy_to_directory")
+
+local_repository(
+    name = "aspect_bazel_lib",
+    path = "../..",
+)
+
+load("@aspect_bazel_lib//:internal_deps.bzl", "bazel_lib_internal_deps")
+
+bazel_lib_internal_deps()
+
+load("@aspect_bazel_lib//lib:repositories.bzl", "aspect_bazel_lib_dependencies")
+
+aspect_bazel_lib_dependencies()
+
+load("@aspect_bazel_lib//:deps.bzl", "go_dependencies")
+
+go_dependencies()
+
+load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
+
+go_rules_dependencies()
+
+go_register_toolchains(version = "1.18.3")
+
+load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
+
+gazelle_dependencies()

e2e/external_copy_to_directory/app/.bazelversion

@@ -0,0 +1 @@
+../../../.bazelversion

e2e/external_copy_to_directory/app/BUILD.bazel

@@ -0,0 +1,6 @@
+sh_test(
+    name = "test",
+    srcs = ["test.sh"],
+    data = ["@external_copy_to_directory//:dir"],
+    deps = ["@bazel_tools//tools/bash/runfiles"],
+)

e2e/external_copy_to_directory/app/WORKSPACE

@@ -0,0 +1,31 @@
+local_repository(
+    name = "external_copy_to_directory",
+    path = "..",
+)
+
+local_repository(
+    name = "aspect_bazel_lib",
+    path = "../../..",
+)
+
+load("@aspect_bazel_lib//:internal_deps.bzl", "bazel_lib_internal_deps")
+
+bazel_lib_internal_deps()
+
+load("@aspect_bazel_lib//lib:repositories.bzl", "aspect_bazel_lib_dependencies")
+
+aspect_bazel_lib_dependencies()
+
+load("@aspect_bazel_lib//:deps.bzl", "go_dependencies")
+
+go_dependencies()
+
+load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
+
+go_rules_dependencies()
+
+go_register_toolchains(version = "1.18.3")
+
+load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
+
+gazelle_dependencies()

e2e/external_copy_to_directory/app/test.sh

@@ -0,0 +1,19 @@
+# --- begin runfiles.bash initialization v2 ---
+# Copy-pasted from the Bazel Bash runfiles library v2.
+set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
+source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
+  source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
+  source "$0.runfiles/$f" 2>/dev/null || \
+  source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+  source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+  { echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
+# --- end runfiles.bash initialization v2 ---
+
+# Read external directory and make sure it exists with a file.
+readonly DIR=$(rlocation external_copy_to_directory/dir/)
+readonly FILES=$(ls "${DIR}" | wc -l)
+if [[ ${FILES} != 1 ]]; then
+  echo "Expected exactly 1 file under external directory, but found ${FILES}:" >&2
+  ls "${DIR}" >&2
+  exit 1
+fi

e2e/external_copy_to_directory/directory.bzl

@@ -0,0 +1,36 @@
+"""Test rule executing `copy_to_directory_bin_action`."""
+
+load("@aspect_bazel_lib//lib:copy_to_directory.bzl", "copy_to_directory_bin_action")
+
+def _directory_impl(ctx):
+    dst = ctx.actions.declare_directory(ctx.attr.name)
+
+    copy_to_directory_bin_action(
+        ctx,
+        name = ctx.attr.name,
+        copy_to_directory_bin = ctx.executable._tool,
+        dst = dst,
+        files = ctx.files.srcs,
+        verbose = True,
+    )
+
+    return DefaultInfo(files = depset([dst]))
+
+directory = rule(
+    implementation = _directory_impl,
+    attrs = {
+        "srcs": attr.label_list(
+            mandatory = True,
+            allow_files = True,
+        ),
+        "_tool": attr.label(
+            executable = True,
+            cfg = "exec",
+            default = "@aspect_bazel_lib//tools/copy_to_directory",
+        ),
+    },
+    doc = """
+        Copies the given source files to a directory with
+        `copy_to_directory_bin_action()`.
+    """,
+)

e2e/external_copy_to_directory/foo.txt

@@ -0,0 +1 @@
+foo

lib/private/copy_to_directory.bzl

@@ -79,7 +79,7 @@ Globs are supported (see rule docstring above).
 
 Files from external repositories are only copied into the output directory if
 the external repository they come from matches one of the external repository patterns
-specified.
+specified or if they are in the same external repository as this target.
 
 When copied from an external repository, the file path in the output directory
 defaults to the file's path within the external repository. The external repository
@@ -492,7 +492,7 @@ def copy_to_directory_bin_action(
         inputs = file_inputs + [config_file],
         outputs = [dst],
         executable = copy_to_directory_bin,
-        arguments = [config_file.path],
+        arguments = [config_file.path, ctx.label.workspace_name],
         mnemonic = "CopyToDirectory",
         progress_message = "Copying files to directory %s" % _progress_path(dst),
         execution_requirements = override_execution_requirements or execution_requirements_for_copy(ctx),