* install: test uses pathlib.
Cleans up a lot of code!
* install: add --wipe_destdir option.
If specified, wipe destination directory before installing.
Fixes #893.
* install: Update doc for --wipe_destdir.
Clarify that this will delete the whole directory.
---------
Co-authored-by: HONG Yifan <elsk@google.com>
After this change, log levels are defined by the following:
* -q: only print errors (not even warnings)
* (default): print errors, warnings, infos
* -v: print above & also debug logs.
There are no errors and warnings logs at this moment.
The existing verbose logs about file operations are
categorized as debug logs, so they are only printed
when -v is set.
By default (if no flag is set), the destdir is printed
as an INFO level log.
Also improve the logging format to be similar to bazel's output, e.g.
INFO: Installing to /tmp/inst
Fixes #889
Co-authored-by: HONG Yifan <elsk@google.com>
* rpm: Refine prefix matching for sub RPMs to reduce fragility
The current prefix matching for sub RPMs can potentially fail in two
ways:
- if we have two overlapping prefixes we may match the shorter prefix
against the wrong RPM
- the shorter prefix will match repeatedly even after finding its
target
This change keeps track of which sub RPMs have already match so avoid
double matching a shorter prefix. Additionally, we sort the sub RPM
list by name length in reverse order. This ensures that we prefer to
match the longest prefixes first and avoid double matching with the
same prefix.
* Expand subrpm test for similarly named RPMs
This change expands the basic subrpm test to account for multiple
subrpms that have matching prefixes.
Implementation notes:
Relative paths are interpreted against BUILD_WORKSPACE_DIRECTORY, not
BUILD_WORKING_DIRECTORY. This is for the following reasons:
The TODO tag explicitly convey the intention of using
BUILD_WORKSPACE_DIRECTORY for relative paths.
If destdir is specified in the attribute of the pkg_install() target,
interpreting it against BUILD_WORKSPACE_DIRECTORY is much stabler.
That is, no matter where your current cwd is, the destdir attribute
always refers to a path relative to the workspace root. For example:
```
pkg_install(name = "my_pkg_install", destdir = "out/dest")
```
```
cd <workspace_root>/<some_subdir>
bazel run //:my_pkg_install
```
This clearly conveys that the default destdir is
<workspace_root>/out/dest regardless of where the user runs the command.
The cost is that the --destdir command line argument becomes trickier
to understand. For example, if one is not familiar with pkg_install,
and below the workspace root they run:
```
cd <workspace_root>/out
bazel run //:my_pkg_install -- --destdir dest
```
They may expect the destdir to be set to <workspace_root>/out/dest; but
it is in fact `<workspace_root>/dest`.
We could also interpret the target attribute & the command line argument
separately (e.g. pkg_install(destdir_against_workspace)), but honestly
I think that's even more confusing when they interpret relative paths
differently. Please let me know if this is preferred by the maintainers.
Co-authored-by: HONG Yifan <elsk@google.com>
* install: Delete unused params from internal functions.
They are so confusing; I would've thought that because they
took the parameters, they would set the user/group for me.
But in fact, _chown_chmod needs to be called afterwards.
* pkg_install: Support TreeArtifacts.
Implementation notes:
For tree artifacts, when creating directories, we mostly follow the
modes set for the whole TreeArtifact, but also +x to allow searching
the directory. This is similar to how pkg_tar etc. handles things.
Link: https://github.com/bazelbuild/rules_pkg/issues/308
* pkg_install: Also add test for modes in TreeArtifacts.
---------
Co-authored-by: HONG Yifan <elsk@google.com>
* Add support for repo mapping
Esp. when include_runfiles = True
Closes #769
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Update windows golden manifest
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Split repo_mapping manifest checking logic into its own function
Signed-off-by: Thomas Lam <thomaslam@canva.com>
* Comment
Signed-off-by: Thomas Lam <thomaslam@canva.com>
---------
Signed-off-by: Thomas Lam <thomaslam@canva.com>
AlmaLinux 9.3 appears to behave as a CentOS variant so adding it in
the same places. This change also cleans up the way we declare
debuginfo_type to make it a bit less verbose.
Duplicate path entries are made possible within tar archives as
discussed in feature request #849. This includes an interaction with
create parents, where the only logical scenario which would require
inference of a parent directory is when one does not already exist.
This is because allowance of duplicates is only useful when explicit
paths are declared.
RELNOTES: Duplicate path entries supported within tar archives
* Fix pkg_files_contents_test
The test wasn't asserting anything about the destination path because it
was missing the `env` positional parameter; the "assert_true" was tested
against the assertion message (which always evaluates to True as a
non-empty string).
* Test pkg_files runfiles destination paths
* Append workspace name in runfiles in pkg_files
Commit a811e7f44f recently fixed an issue
where the workspace name was missing from the path for runfiles added
via `pkg_tar` and some other rules.
This extends the fix to `pkg_files` as well.
* Exclude release from filename when using release_file
When we're using `release_file` in lieu of `release` we're just
pointing rpmbuild at the file containing the `Release` string and we
don't have it available to inject into the filename resulting in a
strange looking filename of the form `Foo-version-.arch.rpm`.
This change extracts the RPM name generation to a single helper,
`_make_rpm_filename` and tweaks it s.t. if we're missing the value for
`release` we'll just exclude it from the filename format instead.
* Fix test broken test
The test had was using the odd RPM name structure and this change
tweaks it so that the test passes.
* Enable creation and capture of debuginfo RPMs
This change enables the creation and capture of debuginfo RPMs on
Fedora40 and CentOS7.
See:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Debuginfo/
Fedora 40 expects the RPM contents to be located in a subdirectory
which is specified using the `buildsubdir` variable. In order to
account for this, we need to tweak some of the location details if
debuginfo is enabled.
CentOS expects `buildsubdir` to have a value like `.` instead.
In both cases, we disable debugsource packages by ensuring that we
undefine `_debugsource_packages`, otherwise we'll try to generate them
alongside the debuginfo packages and will fail.
We only want this method of producing debuginfo to apply when we're
using the system `rpmbuild` because the underlying behaviour is
controlled by a combination of the rpmbuild version, macro
definitions, find-debuginfo.sh, and debugedit. If we were to expand
this to use a hermetic debuginfo then a different approach might be
desirable.
* Add an RPM example that generates debuginfo
This provides a basic example that generates a debuginfo RPM
configured to run on CentOS7.
* Upgrade rules_python to 0.31.0
rules_python seems to fail us when we're generating debuginfo RPMs
unless we upgrade to a version more recent than 0.24.0.
* Only generate debuginfo RPM when pkg_rpm() asks for it
In lieu of enabling this behaviour by default on the supported
platforms, we add an additional argument to the pkg_rpm() rule that
will allow us to enable it for pkg_rpm() targets. This prevents us
from enabling it in cases where it's not desired.
* Add test for building debuginfo RPM
This test is modelled on the subrpm test. In lieu of using a simple
text file as an input it instead generates a binary that includes
debug symbosl from a C source file and includes that in the RPM.
The baseline comparison strips out the `.build-id` paths because the
hashes that are generated may not be stable.x
* Remove architecture and size from debuginfo test output
These values may vary depending on the platform that this is being run
on and we don't really care about them.
* Add period to docstring
* Enable debuginfo support for CentOS Stream 9
CentOS Stream 9 appears to work more or less the same for debuginfo
generation as CentOS 7. `os-release` describes it as os == `centos`
and version == `9`. This change creates an extra token for `centos9`
and sticks it in the places where we currently have controls for
`centos7`.
The runfiles directory is broken down by workspace name. This means that
files belonging to the main workspace are placed at
"${name}.runfiles/${ctx.workspace_name}/${short_path}". Files belonging
to externals have a short_path starting with "../${external_name}".
Because we currently don't append ctx.workspace_name to the computed
runfiles path, files belonging to the main workspace are placed at the
top level of the runfiles directory, which is incorrect. Even worse is
that files belonging to external repositories end up alongside the
runfiles directory, instead of being contained within.
* Enable pkg_rpm and pkg_subrpm to create empty RPMs
At present we fail in two ways if we try to create empty RPMs:
- we expect srcs to be both non-empty and will fail if it is
empty and we have no spec file
- we don't emit anything for the `%files` block in the RPM if
there are no actual files and rpmbuild doesn't like this
This change tweaks the former condition so that srcs has to be
non-None or we have to have a specfile, but will allow us to have an
empty (`[]`) value for srcs. Additionally, it injects
`%defattr(-,root,root)` as a reasonable default for the `%files`
blocks so as to allow rpmbuild to be happy with what we're providing.
* Inject default file mode unconditionally
We should be safe to inject this unconditionally instead of special
casing on whether or not we have no actual files.
* Fixup wrong append
For the `symlink` attribute of `pkg_tar`, if the path begins
with ./, preserve the path rather than prefixing package_dir.
This allows graceful migration for some uses who rely on
the previous package_dir behavior.
This change brings the sub RPM rules closer to parity with both
reality and the parent RPM rules by adding the `Conflicts` and
`Obsoletes` fields to them.
- Do not create the subrpms files when we don't have to
- Do not add `buildsubdir .` That seems to cause rpmbuild failures with
older rpmbuild.
- General cleanup: Remove tests against None which is needlessly brittle.
Add feature as described in #832.
RELNOTES: Automatic creation of parent directory specifications for
paths with depth can be prevented in `pkg_tar` archives by setting `create_parents=False`.
* rpm: Add support for sub packages to make_rpm.py script
Before we can enable support for sub RPM building as part of a single
`pkg_rpm()` rule we must add the underlying support to make_pkg.py
which is the underlying driver for `pkg_rpm()`.
This covers three pieces:
* specifying `buildsubdir` rpm variable
* capturing multiple RPM files as outputs
* injecting sub RPM definitions into specfile
* rpm: Factor out rpm_ctx helper
The various processing functions pass around a bunch of collections
everywhere which is a bit fragile. This collects them together into a
struct to make it a bit less messy.
* rpm: Factor out _process_dep() helper function
_process_dep() handles processing an individual dep and is currently
called from the processing loop. We'll need to re-use this logic for
processing individual sub RPMs as well so we want it in a helper.
* rpm: Capture generated output RPM files in rpm_ctx
Currently we only generate one RPM file, but once we generate sub RPM
files we'll need to be able to capture those outputs as well. This
prepares us for that step.
* rpm: Add args for make_rpm to rpm_ctx
We'll need to add additional arguments to make_rpm for sub RPM
building. It's easier to capture this in our context object than to
try to shuttle these bits around.
* rpm: Pass correct `--name` argument to make_rpm
If we don't pass the correct RPM name to `make_rpm.py` than we won't be
able to correctly determine the subrpm names. Currently, the name is
only used by `make_rpm` to generate some progress output, so this
shouldn't break anything internally.
* rpm: Implementation of `pkg_sub_rpm` rule
This introduces a `pkg_sub_rpm` rule that allows us to generate and
capture RPM subpackages and generate them as part of a single RPM
invocation in lieu of cobbling this together from multiple RPM rules.
This has a few benefits:
- faster execution due to single rpmbuild invocation
- sharing configuration between RPMs in the same fashion as vanilla
RPM building from a specfile
- will enable the proper construction of debuginfo RPMs in a later PR
The current implementation *only* works with non-specfile based rules
and currently allows for a subset of the general RPM configuration.
Internally, the process is for `pkg_sub_rpm` to generate a
PackageSubRPMInfo provider that will subsequently get consumed by the
`pkg_rpm` rule that generates the actual RPMs. We re-use the internal
dependency processing logic that's used by the top-level RPM to
generate the content related to the sub-RPM.
* rpm: Update entry points for RPM rules to include pkg_sub_rpm
This change updates `rpm.bzl` in two ways to account for sub RPMs:
- it adds an entrypoint for `pkg_sub_rpm`
- it adds a check in `pkg_rpm` to assert incompatibility with
`spec_file`
* examples: Add an example of how to use the subrpm rule
This provides a basic example using the `pkg_sub_rpm` rule to generate
multiple RPMs.
* Fix buildifier noise
* Fix make_rpm failures
* doc: Clean up sub-RPM docstring and add to doc_build
The initial docstring for pkg_sub_rpm is not great, so this change
fixes that while adding this to the list of rules to have
documentation generated for them.
* doc: Additional documentation for subrpms in pkg_rpm
This clarifies the `subrpms` attribute usage as well as indicating the
incompatibility with `spec_file` mode.
* Fix issue in subrpm passthrough
When adding the check to verify if we can use subrpms, this pass
through wasn't added.
* Add a basic test for pkg_sub_rpm
This introduces a basic test with a single sub RPM and main RPM each
containing a single source file.
* Tweaks to test
* Test fixes for CentOS 7
The `rpm` version on CentOS 7 doesn't work exactly the same way as
newer versions of rpm and requires a `-p` parameter to inspect
non-installed RPMs. CentOS 7 also inserts a `Relocations` field that
we didn't see on other platforms so we'll filter that out to make our
test a bit more portable.
* Move PackageSubRPMInfoProvider into rpm_pfg.bzl
This is private to the RPM rules so should probably live there.
Some glob patterns do not match any file, which makes building with --incompatible_disallow_empty_glob produce an error.
We should be able to just remove them.
* Explicitly set the FILE bit in zip external attributes.
This should not be needed, but it seems to be for some Azure users.
Fixes #802
* remove a comment that proved unneeded
* Allow substitution of user-defined variables in RPM preamble
It's desirable to be able to parameterize some variables in the
preamble such as architecture when RPM packages. This change enables
variable substitution in the preamble section so that the values may
be injected in this fashion in lieu of only using statically defined
values.
* Deal with mismatched variable definitions
Currently we don't handle things like $(foo or (bar) correctly.
Lacking regex matching, we can compensate for this somewhat by
attempting to find matching pairs of $( and ) and failing if we see
the start of a variable declaration but not its termination.
For some RPM packaging scenarios users may wish to define additional
macros. This change enables us to do this using the `--rpmbuild_arg`
argument to the `make_rpm.py` script by passing a dict of defines to
the `pkg_rpm()` rule.
rules_pkg has baked in the assumption that the value of `%{_builddir}`
is going to be `%{_topdir}/BUILD` which is where rpmbuild will `cd` to
when being run. When using the built in system rpmbuild in a
situation where the value of `%{_builddir}` has been overriden with a
custom local macro, this assumption may be broken which will result in
internal rpmbuild failures.
Because we make this assumption about the value of the macro, we can
instead be explicit about what we want the value of `%{_builddir}` to
be so as to avoid this problem altogether.
* Bring tar runfiles up to feature parity with pkg_files.runfiles.
The bulk of the change is to have pkg_tar use pkg_files%add_label_list to process `srcs`. That brings it in conformance with pkg_files and pkg_zip. This changes the behavior of runfiles to be useful, but it is a breaking
change.
Fixes: #153
Fixes: #579
Other bits:
- do not fail verify_archive_test without a min_size or max_size
- change `data` file in the sample executable from BUILD to foo.cc so
that it is easier to disambiguate from other files.
- fix bug in runfiles support where files were not added to file_deps list.
- fix bug in runfiles support where the runfiles base was not computed
correctly.
- some buildifier fixes to make the linters happy.
Currently if the consumer doesn't provide a value for the pre or post
scriplets we inject an empty scriptlet which has the effect of
constructing an RPM that appears to have these scriplets (although
they're empty). Instead, we likely want to not provide them at all in
these cases.
Currently there doesn't appear to be any support `Obsoletes` in our
RPM structure. This change introduces the support for the `Obsoletes`
tag in the RPM specfile. Unlikely `Requires`, there doesn't appear to
be a context-specific `Obsoletes`, so this is not added here.
* Refactor all the input processing code.
- Create a MappingContext to hold behaviors and defaults
- Change calls
- from `process_x(ctx, content_map, ..., default_x, default_y, ...)`
- to `process_x(mapping_context, ...)
1. This brings the code more in line with the new preferred rule writing style.
1. A major motivation for the change is adding include_runfiles to pkg_tar, where we want
to sometimes have add_label_list do the runfiles inclusion for us, and
other times we want to revert to legacy behavior. If we always passed it
in `ctx`, that would not be possible.
* Syle nits
* make buildifier happier
Get bzlmod CI working
* fix a lot of python which needs tochange
* make extension for rpmbuild
* restore macos CI to rolling
* better documentation on how to use rpmbuild.
* no docbuild with bzlmod -> stardoc problems
* disable a tar test that can not work with bzlmod
Next: Split rpmbuild to a separately distributable artifact
* Correct for case where tar has prefix_dir and files/symlinks, but the
user has already mapped the prefix_dir into those paths. Under the old
behavior, prefix_dir was (incorrectly) not added to symlinks, so some
users made the desired prefix part of the link. Protecting them
against the double inclusion of the prefix is probably the least
surprising behavior.
Add tests for this, which required improving verify_archive_test.
* make buildifier happy
* do not run the symlink test on windows
* huh? why are symlinks in a tree failing on CI?
* Align pkg_rpm returned files with other rules
pkg_rpm returns the package file named after `package_file_name` and a
symbolic link to the file called `target_name`, while other rules do
not return the symbolic link.
* Merge branch 'bazelbuild:main' into main
dpkg has officially supported zstandard compression for the data tarball
since version 1.21.18, and despite shipping with dpkg version 1.21.1,
most .deb packages for Ubuntu 22.04 are also in zst format.
While this PR does not do anything to make it easier to create
zstd-compressed tarballs, it does make it possible to use them when
creating .deb packages.
Closes #759
elsk