Commit Graph

510 Commits

Author SHA1 Message Date
elsk 5c6aec6462
install: Add --wipe_destdir option (#894)
* install: test uses pathlib.

Cleans up a lot of code!

* install: add --wipe_destdir option.

If specified, wipe destination directory before installing.

Fixes #893.

* install: Update doc for --wipe_destdir.

Clarify that this will delete the whole directory.

---------

Co-authored-by: HONG Yifan <elsk@google.com>
2024-09-27 21:27:52 -04:00
Ivo List 6a9eaf2ce9
Load py_binary from rules_python (#895) 2024-09-25 09:44:50 -04:00
peter woodman 25fd41cbe2
make_rpm: suppress unconditional debug prints (#892)
these look like they were left in accidentally, and they muck up build
output pretty significantly as-is.
2024-09-16 09:21:04 -04:00
elsk df9f345aa2
pkg_install: modify log levels & print destdir by default. (#891)
After this change, log levels are defined by the following:
* -q: only print errors (not even warnings)
* (default): print errors, warnings, infos
* -v: print above & also debug logs.

There are no errors and warnings logs at this moment.
The existing verbose logs about file operations are
categorized as debug logs, so they are only printed
when -v is set.

By default (if no flag is set), the destdir is printed
as an INFO level log.

Also improve the logging format to be similar to bazel's output, e.g.

    INFO: Installing to /tmp/inst

Fixes #889

Co-authored-by: HONG Yifan <elsk@google.com>
2024-09-16 09:18:15 -04:00
Dahan Gong 03cbc6c715
Make compression_level configurable for .tar.gz (#888) 2024-09-16 09:17:41 -04:00
Dahan Gong 5047485545
Fix an unexpected "index out of range" (#890) 2024-09-06 06:03:41 -06:00
Mike Kelly ec08e7fccf
rpm: Refine prefix matching for sub RPMs to reduce fragility (#879)
* rpm: Refine prefix matching for sub RPMs to reduce fragility

The current prefix matching for sub RPMs can potentially fail in two
ways:

- if we have two overlapping prefixes we may match the shorter prefix
  against the wrong RPM

- the shorter prefix will match repeatedly even after finding its
  target

This change keeps track of which sub RPMs have already match so avoid
double matching a shorter prefix.  Additionally, we sort the sub RPM
list by name length in reverse order.  This ensures that we prefer to
match the longest prefixes first and avoid double matching with the
same prefix.

* Expand subrpm test for similarly named RPMs

This change expands the basic subrpm test to account for multiple
subrpms that have matching prefixes.
2024-09-02 14:48:35 -04:00
elsk 4afd0b3ccd
pkg_install: Add destdir attr & read rel paths. (#886)
Implementation notes:

Relative paths are interpreted against BUILD_WORKSPACE_DIRECTORY, not
BUILD_WORKING_DIRECTORY. This is for the following reasons:

The TODO tag explicitly convey the intention of using
BUILD_WORKSPACE_DIRECTORY for relative paths.

If destdir is specified in the attribute of the pkg_install() target,
interpreting it against BUILD_WORKSPACE_DIRECTORY is much stabler.
That is, no matter where your current cwd is, the destdir attribute
always refers to a path relative to the workspace root. For example:
```
pkg_install(name = "my_pkg_install", destdir = "out/dest")
```
```
cd <workspace_root>/<some_subdir>
bazel run //:my_pkg_install
```
This clearly conveys that the default destdir is
<workspace_root>/out/dest regardless of where the user runs the command.

The cost is that the --destdir command line argument becomes trickier
to understand. For example, if one is not familiar with pkg_install,
and below the workspace root they run:

```
cd <workspace_root>/out
bazel run //:my_pkg_install -- --destdir dest
```

They may expect the destdir to be set to <workspace_root>/out/dest; but
it is in fact `<workspace_root>/dest`.

We could also interpret the target attribute & the command line argument
separately (e.g. pkg_install(destdir_against_workspace)), but honestly
I think that's even more confusing when they interpret relative paths
differently. Please let me know if this is preferred by the maintainers.

Co-authored-by: HONG Yifan <elsk@google.com>
2024-08-28 15:14:02 -04:00
elsk 412c8fe955
pkg_install: Support TreeArtifacts. (#885)
* install: Delete unused params from internal functions.

They are so confusing; I would've thought that because they
took the parameters, they would set the user/group for me.
But in fact, _chown_chmod needs to be called afterwards.

* pkg_install: Support TreeArtifacts.

Implementation notes:
For tree artifacts, when creating directories, we mostly follow the
modes set for the whole TreeArtifact, but also +x to allow searching
the directory. This is similar to how pkg_tar etc. handles things.

Link: https://github.com/bazelbuild/rules_pkg/issues/308

* pkg_install: Also add test for modes in TreeArtifacts.

---------

Co-authored-by: HONG Yifan <elsk@google.com>
2024-08-28 15:12:15 -04:00
Thomas Lam 447fb8e246
Add support for repo mapping (#878)
* Add support for repo mapping

Esp. when include_runfiles = True

Closes #769

Signed-off-by: Thomas Lam <thomaslam@canva.com>

* Update windows golden manifest

Signed-off-by: Thomas Lam <thomaslam@canva.com>

* Split repo_mapping manifest checking logic into its own function

Signed-off-by: Thomas Lam <thomaslam@canva.com>

* Comment

Signed-off-by: Thomas Lam <thomaslam@canva.com>

---------

Signed-off-by: Thomas Lam <thomaslam@canva.com>
2024-08-14 00:21:59 -04:00
aiuto 6a44f01087
Revert compatibility_level back to 1 so we can push to BCR. (#877)
* Revert compatibility_level back to 1 so we can push to BCR.

See https://github.com/bazelbuild/bazel-central-registry/issues/2232
for discussion.

* update version#
2024-07-09 14:22:24 -04:00
Mike Kelly fbc7cffa54
Add AlmaLinux 9.3 debuginfo support (#874)
AlmaLinux 9.3 appears to behave as a CentOS variant so adding it in
the same places.  This change also cleans up the way we declare
debuginfo_type to make it a bit less verbose.
2024-06-30 22:05:23 -04:00
aiuto b4ac7e3197
1.0 release (#869) 2024-06-03 13:12:43 -04:00
Erik Stringwell 2247f5d88d
Support duplicate paths in tar archives (#850)
Duplicate path entries are made possible within tar archives as
discussed in feature request #849. This includes an interaction with
create parents, where the only logical scenario which would require
inference of a parent directory is when one does not already exist.
This is because allowance of duplicates is only useful when explicit
paths are declared.

RELNOTES: Duplicate path entries supported within tar archives
2024-05-29 16:51:36 -04:00
Sitaktif bf4609b277
Append workspace name to the runfiles directory name for `pkg_files` (#864)
* Fix pkg_files_contents_test

The test wasn't asserting anything about the destination path because it
was missing the `env` positional parameter; the "assert_true" was tested
against the assertion message (which always evaluates to True as a
non-empty string).

* Test pkg_files runfiles destination paths

* Append workspace name in runfiles in pkg_files

Commit a811e7f44f recently fixed an issue
where the workspace name was missing from the path for runfiles added
via `pkg_tar` and some other rules.

This extends the fix to `pkg_files` as well.
2024-05-01 09:12:19 -04:00
Mike Kelly c8d6a0294d
Exclude release from filename when using release_file (#863)
* Exclude release from filename when using release_file

When we're using `release_file` in lieu of `release` we're just
pointing rpmbuild at the file containing the `Release` string and we
don't have it available to inject into the filename resulting in a
strange looking filename of the form `Foo-version-.arch.rpm`.

This change extracts the RPM name generation to a single helper,
`_make_rpm_filename` and tweaks it s.t. if we're missing the value for
`release` we'll just exclude it from the filename format instead.

* Fix test broken test

The test had was using the odd RPM name structure and this change
tweaks it so that the test passes.
2024-04-29 10:22:14 -04:00
Elvis M. Wianda 9616a33e43
fix: Not all runfiles are single files (#860)
* fix: Not all runfiles are single files

Handle cases where the defaultInfo file is a directory

* Add test case
2024-04-29 10:02:00 -04:00
Tom van der Woerdt 21e1cccbc3
Add support for "Epoch" attributes in RPMs (#858) 2024-04-24 16:14:25 -04:00
Mike Kelly 581a86a294
Add support for generating debuginfo RPMs (#842)
* Enable creation and capture of debuginfo RPMs

This change enables the creation and capture of debuginfo RPMs on
Fedora40 and CentOS7.

See:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Debuginfo/

Fedora 40 expects the RPM contents to be located in a subdirectory
which is specified using the `buildsubdir` variable.  In order to
account for this, we need to tweak some of the location details if
debuginfo is enabled.

CentOS expects `buildsubdir` to have a value like `.` instead.

In both cases, we disable debugsource packages by ensuring that we
undefine `_debugsource_packages`, otherwise we'll try to generate them
alongside the debuginfo packages and will fail.

We only want this method of producing debuginfo to apply when we're
using the system `rpmbuild` because the underlying behaviour is
controlled by a combination of the rpmbuild version, macro
definitions, find-debuginfo.sh, and debugedit.  If we were to expand
this to use a hermetic debuginfo then a different approach might be
desirable.

* Add an RPM example that generates debuginfo

This provides a basic example that generates a debuginfo RPM
configured to run on CentOS7.

* Upgrade rules_python to 0.31.0

rules_python seems to fail us when we're generating debuginfo RPMs
unless we upgrade to a version more recent than 0.24.0.

* Only generate debuginfo RPM when pkg_rpm() asks for it

In lieu of enabling this behaviour by default on the supported
platforms, we add an additional argument to the pkg_rpm() rule that
will allow us to enable it for pkg_rpm() targets.  This prevents us
from enabling it in cases where it's not desired.

* Add test for building debuginfo RPM

This test is modelled on the subrpm test.  In lieu of using a simple
text file as an input it instead generates a binary that includes
debug symbosl from a C source file and includes that in the RPM.

The baseline comparison strips out the `.build-id` paths because the
hashes that are generated may not be stable.x

* Remove architecture and size from debuginfo test output

These values may vary depending on the platform that this is being run
on and we don't really care about them.

* Add period to docstring

* Enable debuginfo support for CentOS Stream 9

CentOS Stream 9 appears to work more or less the same for debuginfo
generation as CentOS 7.  `os-release` describes it as os == `centos`
and version == `9`.  This change creates an extra token for `centos9`
and sticks it in the places where we currently have controls for
`centos7`.
2024-04-24 16:03:11 -04:00
Ed Schouten a811e7f44f
Append the workspace name to the runfiles directory name (#856)
The runfiles directory is broken down by workspace name. This means that
files belonging to the main workspace are placed at
"${name}.runfiles/${ctx.workspace_name}/${short_path}". Files belonging
to externals have a short_path starting with "../${external_name}".

Because we currently don't append ctx.workspace_name to the computed
runfiles path, files belonging to the main workspace are placed at the
top level of the runfiles directory, which is incorrect. Even worse is
that files belonging to external repositories end up alongside the
runfiles directory, instead of being contained within.
2024-04-23 23:36:35 -04:00
Mike Kelly 7849529f89
Enable pkg_rpm and pkg_subrpm to create empty RPMs (#859)
* Enable pkg_rpm and pkg_subrpm to create empty RPMs

At present we fail in two ways if we try to create empty RPMs:

- we expect srcs to be both non-empty and will fail if it is
  empty and we have no spec file

- we don't emit anything for the `%files` block in the RPM if
  there are no actual files and rpmbuild doesn't like this

This change tweaks the former condition so that srcs has to be
non-None or we have to have a specfile, but will allow us to have an
empty (`[]`) value for srcs.  Additionally, it injects
`%defattr(-,root,root)` as a reasonable default for the `%files`
blocks so as to allow rpmbuild to be happy with what we're providing.

* Inject default file mode unconditionally

We should be safe to inject this unconditionally instead of special
casing on whether or not we have no actual files.

* Fixup wrong append
2024-04-23 22:56:55 -04:00
aiuto e604010237
pkg_tar: if symlink starts with ./, keep it absolute (#854)
For the `symlink` attribute of `pkg_tar`, if the path begins
with ./, preserve the path rather than prefixing package_dir.
This allows graceful migration for some uses who rely on
the previous package_dir behavior.
2024-04-18 00:25:07 -04:00
aiuto dc21c28186
create empty MODULE.bazel files in examples to bazel does not complain (#855) 2024-04-17 18:44:38 -04:00
aiuto 36433a92a4
Update to new stardoc so we can work with bzlmod. (#853)
* Update to new stardoc so we can work with bzlmod.
* Requires update to merge.py to account for stardoc changes.
* put integration test back on bazel at head.

Fixes #808
2024-04-16 22:17:21 -04:00
aiuto 626de3ac59
add missing license(). remove licenses(notice) (#742) 2024-04-16 15:43:22 -04:00
Yun Peng bf30ed9e9b
Force stardoc integration test to run with bazel 7.0.0 (#852)
Force the integration test (stardoc and packaging) to run with bazel
7.0.0.
Bazel at head seems to have broken --noenable_bzlmod in some cases. See
#808.

This also bumps platforms to the latest, but that was a red herring from
the error message in the underlying problem
2024-04-16 10:38:15 +02:00
Tony Aiuto ea21d3860c use bazel 7.0.0 for stardoc test 2024-04-15 23:06:01 -04:00
Tony Aiuto 2b15badc46 bump platforms to latest. The hope is that this fixes what is going wrong in the integration test 2024-04-15 22:54:25 -04:00
Mike Kelly 0b5d943a50
rpm: Add conflicts and obsoletes to sub RPMs (#851)
This change brings the sub RPM rules closer to parity with both
reality and the parent RPM rules by adding the `Conflicts` and
`Obsoletes` fields to them.
2024-04-15 22:21:27 -04:00
lberki 59e682ddf1
Indirect .files_to_run and .default_runfiles through DefaultInfo. (#848)
The old ways are going away.
2024-04-10 15:35:02 +02:00
Lukacs T. Berki 79d1f2cd61 Indirect .files_to_run and .default_runfiles through DefaultInfo.
The old ways are going away.
2024-04-10 11:16:16 +00:00
aiuto a56b1d9d33
trying scorecard update one more time (#841)
* trying scorecard update one more time
2024-04-02 11:48:37 -04:00
aiuto e14245aa96
use sys.exit() instead of exit() (#844) 2024-03-28 10:34:38 -04:00
aiuto dcce3e6448
Improve handling of sub rpms. (#833)
- Do not create the subrpms files when we don't have to
- Do not add `buildsubdir .`  That seems to cause rpmbuild failures with
  older rpmbuild.
- General cleanup: Remove tests against None which is needlessly brittle.
2024-03-25 10:51:05 -04:00
aiuto 15d628ddfc
Update scorecard.yml (#840)
Update scorecard workflow as per https://github.com/ossf/scorecard-action
2024-03-22 10:10:05 -04:00
aiuto 00a9a05559
update scorecard workflow to newer nodejs setup (#839)
All the runs started failing with messages from github about node 16 not
being supported and needing node 20. I hop this helps
2024-03-22 09:29:11 -04:00
aiuto f6d5046fa7
Stop using datetime.utcfromtimestamp() (#838)
Fixes #826
2024-03-21 15:25:17 -04:00
Erik Stringwell a0eb69a1f3
Add support for no parent directory inference (#832) (#834)
Add feature as described in #832.

RELNOTES: Automatic creation of parent directory specifications for
paths with depth can be prevented in `pkg_tar` archives by setting `create_parents=False`.
2024-03-21 10:04:19 -04:00
aiuto 20100ce982
Remove expand_yml.py. (#837)
It is not used by anything and my security team is complaing about vulns.
This is the path of least resistance.
2024-03-20 23:44:11 -04:00
aiuto c53ff51070
fix some nits and buildifier warnings (#831) 2024-03-15 10:53:53 -04:00
Mike Kelly 2aa2b8e842
Add pkg_sub_rpm rule for RPM subpackages (#824)
* rpm: Add support for sub packages to make_rpm.py script

Before we can enable support for sub RPM building as part of a single
`pkg_rpm()` rule we must add the underlying support to make_pkg.py
which is the underlying driver for `pkg_rpm()`.

This covers three pieces:

 * specifying `buildsubdir` rpm variable
 * capturing multiple RPM files as outputs
 * injecting sub RPM definitions into specfile

* rpm: Factor out rpm_ctx helper

The various processing functions pass around a bunch of collections
everywhere which is a bit fragile.  This collects them together into a
struct to make it a bit less messy.

* rpm: Factor out _process_dep() helper function

_process_dep() handles processing an individual dep and is currently
called from the processing loop.  We'll need to re-use this logic for
processing individual sub RPMs as well so we want it in a helper.

* rpm: Capture generated output RPM files in rpm_ctx

Currently we only generate one RPM file, but once we generate sub RPM
files we'll need to be able to capture those outputs as well.  This
prepares us for that step.

* rpm: Add args for make_rpm to rpm_ctx

We'll need to add additional arguments to make_rpm for sub RPM
building.  It's easier to capture this in our context object than to
try to shuttle these bits around.

* rpm: Pass correct `--name` argument to make_rpm

If we don't pass the correct RPM name to `make_rpm.py` than we won't be
able to correctly determine the subrpm names.  Currently, the name is
only used by `make_rpm` to generate some progress output, so this
shouldn't break anything internally.

* rpm: Implementation of `pkg_sub_rpm` rule

This introduces a `pkg_sub_rpm` rule that allows us to generate and
capture RPM subpackages and generate them as part of a single RPM
invocation in lieu of cobbling this together from multiple RPM rules.
This has a few benefits:

- faster execution due to single rpmbuild invocation
- sharing configuration between RPMs in the same fashion as vanilla
  RPM building from a specfile
- will enable the proper construction of debuginfo RPMs in a later PR

The current implementation *only* works with non-specfile based rules
and currently allows for a subset of the general RPM configuration.

Internally, the process is for `pkg_sub_rpm` to generate a
PackageSubRPMInfo provider that will subsequently get consumed by the
`pkg_rpm` rule that generates the actual RPMs.  We re-use the internal
dependency processing logic that's used by the top-level RPM to
generate the content related to the sub-RPM.

* rpm: Update entry points for RPM rules to include pkg_sub_rpm

This change updates `rpm.bzl` in two ways to account for sub RPMs:

- it adds an entrypoint for `pkg_sub_rpm`

- it adds a check in `pkg_rpm` to assert incompatibility with
  `spec_file`

* examples: Add an example of how to use the subrpm rule

This provides a basic example using the `pkg_sub_rpm` rule to generate
multiple RPMs.

* Fix buildifier noise

* Fix make_rpm failures

* doc: Clean up sub-RPM docstring and add to doc_build

The initial docstring for pkg_sub_rpm is not great, so this change
fixes that while adding this to the list of rules to have
documentation generated for them.

* doc: Additional documentation for subrpms in pkg_rpm

This clarifies the `subrpms` attribute usage as well as indicating the
incompatibility with `spec_file` mode.

* Fix issue in subrpm passthrough

When adding the check to verify if we can use subrpms, this pass
through wasn't added.

* Add a basic test for pkg_sub_rpm

This introduces a basic test with a single sub RPM and main RPM each
containing a single source file.

* Tweaks to test

* Test fixes for CentOS 7

The `rpm` version on CentOS 7 doesn't work exactly the same way as
newer versions of rpm and requires a `-p` parameter to inspect
non-installed RPMs.  CentOS 7 also inserts a `Relocations` field that
we didn't see on other platforms so we'll filter that out to make our
test a bit more portable.

* Move PackageSubRPMInfoProvider into rpm_pfg.bzl

This is private to the RPM rules so should probably live there.
2024-03-14 22:36:15 -04:00
Diego Ortin 61132feb47
fix: remove unused glob patterns (#825)
Some glob patterns do not match any file, which makes building with --incompatible_disallow_empty_glob produce an error.
We should be able to just remove them.
2024-02-29 11:04:55 -04:00
Mike Kelly 56072594a8
Provide more examples for pkg_rpm() (#823)
* Move prebuilt_rpmbuild example into rpm subdirectory

This will help us be a bit more organized as we add a few more
RPM-based examples for rules_pkg.

* Fix WORKSPACE file to account for subdirectory move

One we moved this into a subdirectory it was broke because of how it
uses rules_pkg as a local repository.

* s/readme.md/README.md/

The latter seems to be more the norm, so renaming for consistency and
visibility.

* Add more detail to README.md for prebuilt_rpmbuild example

This adds a bit more detail the description of the `prebuilt_rpmbuild`
example so that someone can understand it a bit more.

* Add an example of using system rpmbuild for pkg_rpm()

This example is essentially the same as the prebuilt_rpmbuild example
except that it uses find_system_rpmbuild() to register the local
version of `rpmbuild` as the toolchain to use.

* Add an example with bzlmod using system rpmbuild

This is very similar to the non-bzlmod version except we've replaced
the WORKSPACE file with a MODULE.bazel file.

* Add an example that doesn't use a specfile for pkg_rpm()

This example uses pkg_rpm() to generate the full rpm in lieu of using
a standalone specfile.
2024-02-28 10:14:13 -04:00
tonyaiuto d7ff3e6e52
Update changelog and version numbers. (#819)
* update changelog

* freshdocs
2024-02-08 17:46:43 -05:00
aiuto 52996b06c3
Temporarily restore //mappings.bzl and //pkg.bzl (#817)
* Restore mappings.bzl and pkg.bzl as a stopgap until https://github.com/protocolbuffers/protobuf/issues/15779 is fixed.

* bump version number for upcoming patch release
2024-02-08 15:43:00 -05:00
Tomasz Wojno 37cccd4f77
Fix RPM package release and version files expansion (#816) 2024-02-08 10:05:58 -05:00
Diego Ortin 3a2508f9f9
Apply tar remap_paths to runfiles full paths (#812) 2024-02-01 07:06:29 -07:00
Chuck Grindel 1cbb70dfa2
Use raw string for docstring that contains a backslash (#815)
Python 3.12 started to complain about invalid escape sequences.
https://docs.python.org/dev/whatsnew/3.12.html#other-language-changes

One supposed to use r""" when dostrings contains backslashes """
https://stackoverflow.com/a/33734332
2024-02-01 07:04:28 -07:00
Alex Bozhenko 6cad5fb7a6
Update build_tar.py
Python 3.12 started to complain about invalid escape sequences.
https://docs.python.org/dev/whatsnew/3.12.html#other-language-changes

One supposed to use r""" when dostrings contains backslashes """
https://stackoverflow.com/a/33734332
2024-01-31 14:39:04 -08:00
aiuto 39ab08d680
doc and changelog update to go with 0.10.0 (#809)
* add docs for 10
2024-01-10 16:07:09 -05:00