* install: test uses pathlib.
Cleans up a lot of code!
* install: add --wipe_destdir option.
If specified, wipe destination directory before installing.
Fixes #893.
* install: Update doc for --wipe_destdir.
Clarify that this will delete the whole directory.
---------
Co-authored-by: HONG Yifan <elsk@google.com>
After this change, log levels are defined by the following:
* -q: only print errors (not even warnings)
* (default): print errors, warnings, infos
* -v: print above & also debug logs.
There are no errors and warnings logs at this moment.
The existing verbose logs about file operations are
categorized as debug logs, so they are only printed
when -v is set.
By default (if no flag is set), the destdir is printed
as an INFO level log.
Also improve the logging format to be similar to bazel's output, e.g.
INFO: Installing to /tmp/inst
Fixes #889
Co-authored-by: HONG Yifan <elsk@google.com>
* rpm: Refine prefix matching for sub RPMs to reduce fragility
The current prefix matching for sub RPMs can potentially fail in two
ways:
- if we have two overlapping prefixes we may match the shorter prefix
against the wrong RPM
- the shorter prefix will match repeatedly even after finding its
target
This change keeps track of which sub RPMs have already match so avoid
double matching a shorter prefix. Additionally, we sort the sub RPM
list by name length in reverse order. This ensures that we prefer to
match the longest prefixes first and avoid double matching with the
same prefix.
* Expand subrpm test for similarly named RPMs
This change expands the basic subrpm test to account for multiple
subrpms that have matching prefixes.
Implementation notes:
Relative paths are interpreted against BUILD_WORKSPACE_DIRECTORY, not
BUILD_WORKING_DIRECTORY. This is for the following reasons:
The TODO tag explicitly convey the intention of using
BUILD_WORKSPACE_DIRECTORY for relative paths.
If destdir is specified in the attribute of the pkg_install() target,
interpreting it against BUILD_WORKSPACE_DIRECTORY is much stabler.
That is, no matter where your current cwd is, the destdir attribute
always refers to a path relative to the workspace root. For example:
```
pkg_install(name = "my_pkg_install", destdir = "out/dest")
```
```
cd <workspace_root>/<some_subdir>
bazel run //:my_pkg_install
```
This clearly conveys that the default destdir is
<workspace_root>/out/dest regardless of where the user runs the command.
The cost is that the --destdir command line argument becomes trickier
to understand. For example, if one is not familiar with pkg_install,
and below the workspace root they run:
```
cd <workspace_root>/out
bazel run //:my_pkg_install -- --destdir dest
```
They may expect the destdir to be set to <workspace_root>/out/dest; but
it is in fact `<workspace_root>/dest`.
We could also interpret the target attribute & the command line argument
separately (e.g. pkg_install(destdir_against_workspace)), but honestly
I think that's even more confusing when they interpret relative paths
differently. Please let me know if this is preferred by the maintainers.
Co-authored-by: HONG Yifan <elsk@google.com>
* install: Delete unused params from internal functions.
They are so confusing; I would've thought that because they
took the parameters, they would set the user/group for me.
But in fact, _chown_chmod needs to be called afterwards.
* pkg_install: Support TreeArtifacts.
Implementation notes:
For tree artifacts, when creating directories, we mostly follow the
modes set for the whole TreeArtifact, but also +x to allow searching
the directory. This is similar to how pkg_tar etc. handles things.
Link: https://github.com/bazelbuild/rules_pkg/issues/308
* pkg_install: Also add test for modes in TreeArtifacts.
---------
Co-authored-by: HONG Yifan <elsk@google.com>
* Add support for repo mapping
Esp. when include_runfiles = True
Closes #769
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Update windows golden manifest
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Split repo_mapping manifest checking logic into its own function
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Comment
Signed-off-by: Thomas Lam <thomaslam@canva.com>
---------
Signed-off-by: Thomas Lam <thomaslam@canva.com>
AlmaLinux 9.3 appears to behave as a CentOS variant so adding it in
the same places. This change also cleans up the way we declare
debuginfo_type to make it a bit less verbose.
Duplicate path entries are made possible within tar archives as
discussed in feature request #849. This includes an interaction with
create parents, where the only logical scenario which would require
inference of a parent directory is when one does not already exist.
This is because allowance of duplicates is only useful when explicit
paths are declared.
RELNOTES: Duplicate path entries supported within tar archives
* Fix pkg_files_contents_test
The test wasn't asserting anything about the destination path because it
was missing the `env` positional parameter; the "assert_true" was tested
against the assertion message (which always evaluates to True as a
non-empty string).
* Test pkg_files runfiles destination paths
* Append workspace name in runfiles in pkg_files
Commit a811e7f44f recently fixed an issue
where the workspace name was missing from the path for runfiles added
via `pkg_tar` and some other rules.
This extends the fix to `pkg_files` as well.
* Exclude release from filename when using release_file
When we're using `release_file` in lieu of `release` we're just
pointing rpmbuild at the file containing the `Release` string and we
don't have it available to inject into the filename resulting in a
strange looking filename of the form `Foo-version-.arch.rpm`.
This change extracts the RPM name generation to a single helper,
`_make_rpm_filename` and tweaks it s.t. if we're missing the value for
`release` we'll just exclude it from the filename format instead.
* Fix test broken test
The test had was using the odd RPM name structure and this change
tweaks it so that the test passes.
* Enable creation and capture of debuginfo RPMs
This change enables the creation and capture of debuginfo RPMs on
Fedora40 and CentOS7.
See:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Debuginfo/
Fedora 40 expects the RPM contents to be located in a subdirectory
which is specified using the `buildsubdir` variable. In order to
account for this, we need to tweak some of the location details if
debuginfo is enabled.
CentOS expects `buildsubdir` to have a value like `.` instead.
In both cases, we disable debugsource packages by ensuring that we
undefine `_debugsource_packages`, otherwise we'll try to generate them
alongside the debuginfo packages and will fail.
We only want this method of producing debuginfo to apply when we're
using the system `rpmbuild` because the underlying behaviour is
controlled by a combination of the rpmbuild version, macro
definitions, find-debuginfo.sh, and debugedit. If we were to expand
this to use a hermetic debuginfo then a different approach might be
desirable.
* Add an RPM example that generates debuginfo
This provides a basic example that generates a debuginfo RPM
configured to run on CentOS7.
* Upgrade rules_python to 0.31.0
rules_python seems to fail us when we're generating debuginfo RPMs
unless we upgrade to a version more recent than 0.24.0.
* Only generate debuginfo RPM when pkg_rpm() asks for it
In lieu of enabling this behaviour by default on the supported
platforms, we add an additional argument to the pkg_rpm() rule that
will allow us to enable it for pkg_rpm() targets. This prevents us
from enabling it in cases where it's not desired.
* Add test for building debuginfo RPM
This test is modelled on the subrpm test. In lieu of using a simple
text file as an input it instead generates a binary that includes
debug symbosl from a C source file and includes that in the RPM.
The baseline comparison strips out the `.build-id` paths because the
hashes that are generated may not be stable.x
* Remove architecture and size from debuginfo test output
These values may vary depending on the platform that this is being run
on and we don't really care about them.
* Add period to docstring
* Enable debuginfo support for CentOS Stream 9
CentOS Stream 9 appears to work more or less the same for debuginfo
generation as CentOS 7. `os-release` describes it as os == `centos`
and version == `9`. This change creates an extra token for `centos9`
and sticks it in the places where we currently have controls for
`centos7`.
The runfiles directory is broken down by workspace name. This means that
files belonging to the main workspace are placed at
"${name}.runfiles/${ctx.workspace_name}/${short_path}". Files belonging
to externals have a short_path starting with "../${external_name}".
Because we currently don't append ctx.workspace_name to the computed
runfiles path, files belonging to the main workspace are placed at the
top level of the runfiles directory, which is incorrect. Even worse is
that files belonging to external repositories end up alongside the
runfiles directory, instead of being contained within.
* Enable pkg_rpm and pkg_subrpm to create empty RPMs
At present we fail in two ways if we try to create empty RPMs:
- we expect srcs to be both non-empty and will fail if it is
empty and we have no spec file
- we don't emit anything for the `%files` block in the RPM if
there are no actual files and rpmbuild doesn't like this
This change tweaks the former condition so that srcs has to be
non-None or we have to have a specfile, but will allow us to have an
empty (`[]`) value for srcs. Additionally, it injects
`%defattr(-,root,root)` as a reasonable default for the `%files`
blocks so as to allow rpmbuild to be happy with what we're providing.
* Inject default file mode unconditionally
We should be safe to inject this unconditionally instead of special
casing on whether or not we have no actual files.
* Fixup wrong append
For the `symlink` attribute of `pkg_tar`, if the path begins
with ./, preserve the path rather than prefixing package_dir.
This allows graceful migration for some uses who rely on
the previous package_dir behavior.
* Update to new stardoc so we can work with bzlmod.
* Requires update to merge.py to account for stardoc changes.
* put integration test back on bazel at head.
Fixes #808
Force the integration test (stardoc and packaging) to run with bazel
7.0.0.
Bazel at head seems to have broken --noenable_bzlmod in some cases. See
#808.
This also bumps platforms to the latest, but that was a red herring from
the error message in the underlying problem
This change brings the sub RPM rules closer to parity with both
reality and the parent RPM rules by adding the `Conflicts` and
`Obsoletes` fields to them.
- Do not create the subrpms files when we don't have to
- Do not add `buildsubdir .` That seems to cause rpmbuild failures with
older rpmbuild.
- General cleanup: Remove tests against None which is needlessly brittle.
Add feature as described in #832.
RELNOTES: Automatic creation of parent directory specifications for
paths with depth can be prevented in `pkg_tar` archives by setting `create_parents=False`.
* rpm: Add support for sub packages to make_rpm.py script
Before we can enable support for sub RPM building as part of a single
`pkg_rpm()` rule we must add the underlying support to make_pkg.py
which is the underlying driver for `pkg_rpm()`.
This covers three pieces:
* specifying `buildsubdir` rpm variable
* capturing multiple RPM files as outputs
* injecting sub RPM definitions into specfile
* rpm: Factor out rpm_ctx helper
The various processing functions pass around a bunch of collections
everywhere which is a bit fragile. This collects them together into a
struct to make it a bit less messy.
* rpm: Factor out _process_dep() helper function
_process_dep() handles processing an individual dep and is currently
called from the processing loop. We'll need to re-use this logic for
processing individual sub RPMs as well so we want it in a helper.
* rpm: Capture generated output RPM files in rpm_ctx
Currently we only generate one RPM file, but once we generate sub RPM
files we'll need to be able to capture those outputs as well. This
prepares us for that step.
* rpm: Add args for make_rpm to rpm_ctx
We'll need to add additional arguments to make_rpm for sub RPM
building. It's easier to capture this in our context object than to
try to shuttle these bits around.
* rpm: Pass correct `--name` argument to make_rpm
If we don't pass the correct RPM name to `make_rpm.py` than we won't be
able to correctly determine the subrpm names. Currently, the name is
only used by `make_rpm` to generate some progress output, so this
shouldn't break anything internally.
* rpm: Implementation of `pkg_sub_rpm` rule
This introduces a `pkg_sub_rpm` rule that allows us to generate and
capture RPM subpackages and generate them as part of a single RPM
invocation in lieu of cobbling this together from multiple RPM rules.
This has a few benefits:
- faster execution due to single rpmbuild invocation
- sharing configuration between RPMs in the same fashion as vanilla
RPM building from a specfile
- will enable the proper construction of debuginfo RPMs in a later PR
The current implementation *only* works with non-specfile based rules
and currently allows for a subset of the general RPM configuration.
Internally, the process is for `pkg_sub_rpm` to generate a
PackageSubRPMInfo provider that will subsequently get consumed by the
`pkg_rpm` rule that generates the actual RPMs. We re-use the internal
dependency processing logic that's used by the top-level RPM to
generate the content related to the sub-RPM.
* rpm: Update entry points for RPM rules to include pkg_sub_rpm
This change updates `rpm.bzl` in two ways to account for sub RPMs:
- it adds an entrypoint for `pkg_sub_rpm`
- it adds a check in `pkg_rpm` to assert incompatibility with
`spec_file`
* examples: Add an example of how to use the subrpm rule
This provides a basic example using the `pkg_sub_rpm` rule to generate
multiple RPMs.
* Fix buildifier noise
* Fix make_rpm failures
* doc: Clean up sub-RPM docstring and add to doc_build
The initial docstring for pkg_sub_rpm is not great, so this change
fixes that while adding this to the list of rules to have
documentation generated for them.
* doc: Additional documentation for subrpms in pkg_rpm
This clarifies the `subrpms` attribute usage as well as indicating the
incompatibility with `spec_file` mode.
* Fix issue in subrpm passthrough
When adding the check to verify if we can use subrpms, this pass
through wasn't added.
* Add a basic test for pkg_sub_rpm
This introduces a basic test with a single sub RPM and main RPM each
containing a single source file.
* Tweaks to test
* Test fixes for CentOS 7
The `rpm` version on CentOS 7 doesn't work exactly the same way as
newer versions of rpm and requires a `-p` parameter to inspect
non-installed RPMs. CentOS 7 also inserts a `Relocations` field that
we didn't see on other platforms so we'll filter that out to make our
test a bit more portable.
* Move PackageSubRPMInfoProvider into rpm_pfg.bzl
This is private to the RPM rules so should probably live there.
Some glob patterns do not match any file, which makes building with --incompatible_disallow_empty_glob produce an error.
We should be able to just remove them.
* Move prebuilt_rpmbuild example into rpm subdirectory
This will help us be a bit more organized as we add a few more
RPM-based examples for rules_pkg.
* Fix WORKSPACE file to account for subdirectory move
One we moved this into a subdirectory it was broke because of how it
uses rules_pkg as a local repository.
* s/readme.md/README.md/
The latter seems to be more the norm, so renaming for consistency and
visibility.
* Add more detail to README.md for prebuilt_rpmbuild example
This adds a bit more detail the description of the `prebuilt_rpmbuild`
example so that someone can understand it a bit more.
* Add an example of using system rpmbuild for pkg_rpm()
This example is essentially the same as the prebuilt_rpmbuild example
except that it uses find_system_rpmbuild() to register the local
version of `rpmbuild` as the toolchain to use.
* Add an example with bzlmod using system rpmbuild
This is very similar to the non-bzlmod version except we've replaced
the WORKSPACE file with a MODULE.bazel file.
* Add an example that doesn't use a specfile for pkg_rpm()
This example uses pkg_rpm() to generate the full rpm in lieu of using
a standalone specfile.
elsk