Pop!_OS support was broken for me as I am currently running 22.04.
#62 seemed to mix Pop!_OS with Linux Mint and specific version checks, but AFAIK Pop!_OS uses the same version as Ubuntu. This change simply treats Pop!_OS as Ubuntu.
It is not easy to run LLVM 14 on the latest archlinux container which
come with a gcc distribution with compressed sections for its object
files, and LLVM 14 does not have zlib support in ld.lld. LLVM 13 does,
so let's run the tests for archlinux with LLVM 13.
We needed the symlink hack only for the tools explicitly defined by
bazel. Replacing the directory with individually symlinked files is a
better design now that we have not discovered any other uses of the hack
in some time.
It also helps avoid warnings like in
https://github.com/grailbio/bazel-toolchain/issues/125#issuecomment-1100226305
Some rules_rust tests needed more fixes in the patch (and the PR).
Also reorganized how we are listing the tests to run to be more clear
and be able to skip tests incompatible with the platform.
Also update rules_go and rules_rust dependencies. Tests for
compatibility with rules_rust have been simplified.
This change will also help avoid pull in unnecessary dependencies when
using bazel modules.
As of bazel 5.0, --all_incompatible_changes flag is a no-op. So we
should switch to using bazelisk for checking any migration issues.
https://github.com/bazelbuild/bazel/issues/13892
Because bazelisk does not allow specifying flag overrides on the command
line, we can not override the known failing incompatible flags, and so
our migration test will now fail. This is not ideal, but is the only
option going forward.
The original use case of #103 is better satisfied through
the users specifying `link_libs = {"": ["-lpthread", "-ldl"]}` in the
repository rule if they need it globally. Else they can specify it on a
per-target basis as `linkopt`.
We have started supporting the use of stdlib implementations that are
not libc++ and therefore do not require lpthread and ldl. I think the
historic availability of these libraries was by coincidence and we can
expect users to manually specify them if they need. This is also the
default with the LLVM toolchain outside of bazel.
Most users require some configurability of the toolchain through being
able to override the default:
1. stdlib
2. c++ standard
3. linker
The first two have already been solved through attributes to the
repository rule. Instead of continuing to add specialized repository
rules, we should pass through all flag groups in bazel's
unix_cc_toolchain_config.bzl as repository rule attributes. The existing
defaults continue to exist, but users now have the option to override
these defaults for each target os-arch pair.
These flag overrides can also contain a placeholder to the root
directory for the LLVM distribution.
An example solution to satisfy #132 (using mold linker) could be one of:
- Pick up mold linker and libstdc++ from the host.
```
llvm_toolchain(
name = "llvm_toolchain",
link_flags = {"": [
"-lm",
"-no-canonical-prefixes",
"-fuse-ld=mold",
"-Wl,--build-id=md5",
"-Wl,--hash-style=gnu",
"-Wl,-z,relro,-z,now",
"-l:libstdc++.a",
]},
stdlib = {"": "stdc++"},
llvm_version = "12.0.0",
)
```
- Pick up mold linker from the host but continue to link bundled libc++.
```
llvm_toolchain(
name = "llvm_toolchain",
link_flags = {"": [
"-lm",
"-no-canonical-prefixes",
"-fuse-ld=mold",
"-Wl,--build-id=md5",
"-Wl,--hash-style=gnu",
"-Wl,-z,relro,-z,now",
"-L{toolchain_path_prefix}lib",
"-l:libc++.a",
"-l:libc++abi.a",
"-l:libunwind.a",
"-rtlib=compiler-rt",
]},
llvm_version = "12.0.0",
)
```
Current strategy for detecting OS name and matching it to an LLVM
distribution archive has many corner cases. Moreover, people might have
their own URLs where they host these archives.
Having an explicit dict of URLs and sha256, keyed by OS name, version
and arch, will help people get over these corner cases, and be able to
try new LLVM releases without waiting for an update to this repository.
Note that the keys here are different than the `toolchain_roots`
attribute.
While this method does have an extra setup step for each new OS type
that the user's workspace needs to support, this approach is more
flexible.
If we notice that people are using this feature more than the
auto inferred URLs, or that the llvm_release_names.py script is out of
date, we may just retain this feature, and delete the other way of
getting archives.
Additionally, fixes #125. People with that use case can now use the
`urls` attribute, or use the new convenience aliases.
Changes:
- fallback on Ubuntu to latest supported version (for example use `linux-gnu-ubuntu-20.10` for 11.1.0 on Ubuntu 21.10);
- don't try to non-exists `linux-gnu-ubuntu-18.04` for clang 11.1.0 and 11.0.1.
Users wishing to use openmp through `-fopenmp` currently can not do so,
because libomp is available only as a shared library.
Shared libraries are not exposed by this toolchain because there is limited
utility in doing so (only the sandboxed compile step will succeed, and because
the lib is missing from runfiles, the executable won't run).
The proper way to depend on openmp will be the same as depending on other
shared libraries, i.e. either depend on the host/sysroot to provide it, or provide
it through `deps` using perhaps a `cc_import`.
While it was technically possible to create a `cc_import` target for libomp, it
was a little awkward because of the different extensions between darwin and
linux. So we now provide a convenience target `@llvm_toolchain//:omp` and
include a unit test as a demo.
Co-authored-by: Siddhartha Bagaria <sbagaria@grailbio.com>
Fixes #114.
* toolchain: update the tool names
* toolchain: add machinery to expose information about host tools to `cc_toolchain_config`
`cc_toolchain_config` is not a repository rule and cannot poke at the host tools itself
* toolchain: add logic to detect if a tool supports arg files
* toolchain: generalize the `cc_wrapper` plumbing to support "wrapper files"
* toolchain: move the `host_tools` utils to a struct
* toolchain: add a libtool wrapper that flattens arg files
* toolchain: use the libtool wrapper when needed
this commit also changes `cc_toolchain_config.bzl` to check that `strip` and `ld`
exist on the host machine when we use them
we use tags here instead of the `/usr/bin/` paths so that we can easily transition
to using `rctx.which` to find where the tools live instead if we want to do so in
the future
* misc: fmt
There is not enough customization to warrant keeping it in this repo
where it is untested, does not fit well with multiple target toolchains,
and does not have a clear use case.
* Fix including symlink extra files in dwp/objcopy
* Add strip filegroups to support strip inside sandbox
* tests: add a `.stripped` test
* tests: test that `.dwp` targets can be built
This has some things we'll eventually want to remove; see #109 for some context.
* tests: have `dwp_file` transition to `-c dbg` so .dwo files are generated with clang-12+
Issue #109 and [this comment](https://github.com/grailbio/bazel-toolchain/pull/108#issuecomment-928839768)
have some context; the short version is that `-gsplit-dwarf` no longer implies `-g2` which means under
`-c fastbuild` (implies `-g0`) no `.dwo` files are created.
There's a patch to `unix_cc_toolchain_config.bzl` but for now we'll just use `-c dbg` for this one
target using a transition.
* tests: migration fixes
Co-authored-by: Rahul Butani <rr.butani@gmail.com>
* Symlink '/usr/bin/ld' to llvm toolchain directory on Mac
Mac systems still use the local `ld` executable since `lld.ld` is still
experimental. Some applications, e.g. rustc when compiling rust, expect
to find `ld` within the same directory as the compiler and so when
compiling rust we run into an issue where the linker is not found.
The first immediate solution is to use the rust rule's
`extra_rustc_flags` to specify the linker explicitly (via
`-Clinker=/usr/bin/ld`. This works for some crates but fails for crates
using a build script, since the extra flags are not propagated to rustc
invocations within the exec configuration.
Another solution attempt was to link `ld` into the toolchain
configuration repository instead of directly in the llvm repository,
such that `ld` becomes a sibling file to `cc_wrapper.sh`. Unfortunately
rustc was still unable to find the linker with this configuration.
The final fully working solution is to link `ld` into the bin folder of
the llvm repository. This should be fine as LLVM's own linker exists
under `ld.lld` and symlinking the local linker will not clobber any
files.
* tests: add a `rules_rust` test
I picked `git2-rs` since it links against a C library and has tests but isn't _massive_.
There are a couple of hacks/patches here to workaround some bugs in `rules_rust`'s `crate-universe`
but on the whole it's not too messy.
* Add --incompatible_no_rule_outputs_param=false to run_external_tests.sh
The rules_rust repository is incompatible with this flag enabled. Have
disabled it for now.
* Add --incompatible_no_rule_outputs_param=false to run_tests.sh
Flag already added to 'run_external_tests.sh' but also needed here.
* Move 'git2-rs-cargo-toml.patch' to 'tests/rust'
* tests: move some shared test args for bazel into `bazel.sh`
Note that this also drops the check for `$TEST_MIGRATION` in run_external_tests.sh; we don't run that script in migration.yml
* tests: add a note about using `git2-rs` instead of the `@rules_rust` tests
Co-authored-by: Rahul Butani <rr.butani@gmail.com>
Ezekiel Warren