luxolus
/
open-vault
2
0
Fork 0
Code Issues 1 Pull Requests Releases Activity
open-vault/.github/workflows/build-vault-ce.yml

111 lines
3.9 KiB
YAML
Raw Normal View History

[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
---
name: build_vault
# This workflow is intended to be called by the build workflow for each Vault
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
# binary that needs to be built and packaged. The ci make targets that are
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
# utilized automatically determine build metadata and handle building and
# packing vault.
on:
workflow_call:
inputs:
cgo-enabled:
type: string
default: 0
create-packages:
type: boolean
default: true
goos:
required: true
type: string
goarch:
required: true
type: string
go-tags:
type: string
package-name:
type: string
default: vault
vault-version:
type: string
required: true
[QT-576] Optimize build workflow (#21486) (#21601) Improve our build workflow execution time by using custom runners, improved caching and conditional Web UI builds. Runners ------- We improve our build times[0] by using larger custom runners[1] when building the UI and Vault. Caching ------- We improve Vault caching by keeping a cache for each build job. This strategy has the following properties which should result in faster build times when `go.sum` hasn't been changed from prior builds, or when a pull request is retried or updated after a prior successful build: * Builds will restore cached Go modules and Go build cache according to the Go version, platform, architecture, go tags, and hash of `go.sum` that relates to each individual build workflow. This reduces the amount of time it will take to download the cache on hits and upload the cache on misses. * Parallel build workflows won't clobber each others build cache. This results in much faster compile times after cache hits because the Go compiler can reuse the platform, architecture, and tag specific build cache that it created on prior runs. * Older modules and build cache will not be uploaded when creating a new cache. This should result in lean cache sizes on an ongoing basis. * On cache misses we will have to upload our compressed module and build cache. This will slightly extend the build time for pull requests that modify `go.sum`. Web UI ------ We no longer build the web UI in every build workflow. Instead we separate the UI building into its own workflow and cache the resulting assets. The same UI assets are restored from cache during build worklows. This strategy has the following properties: * If the `ui` directory has not changed from prior builds we'll restore `http/web_ui` from cache and skip building the UI for no reason. * We continue to use the built-in `yarn` caching functionality in `action/setup-node`. The default mode saves the `yarn` global cache. to improve UI build times if the cache has not been modified. Changes ------- * Add per platform/archicture Go module and build caching * Move UI building into a separate job and cache the result * Restore UI cache during build * Pin workflows Notes ----- [0] https://hashicorp.atlassian.net/browse/QT-578 [1] https://github.com/hashicorp/vault/actions/runs/5415830307/jobs/9844829929 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-05 21:19:49 +00:00
web-ui-cache-key:
type: string
required: true
[QT-506] Use enos scenario samples for testing (#22641) (#22933) Replace our prior implementation of Enos test groups with the new Enos sampling feature. With this feature we're able to describe which scenarios and variant combinations are valid for a given artifact and allow enos to create a valid sample field (a matrix of all compatible scenarios) and take an observation (select some to run) for us. This ensures that every valid scenario and variant combination will now be a candidate for testing in the pipeline. See QT-504[0] for further details on the Enos sampling capabilities. Our prior implementation only tested the amd64 and arm64 zip artifacts, as well as the Docker container. We now include the following new artifacts in the test matrix: * CE Amd64 Debian package * CE Amd64 RPM package * CE Arm64 Debian package * CE Arm64 RPM package Each artifact includes a sample definition for both pre-merge/post-merge (build) and release testing. Changes: * Remove the hand crafted `enos-run-matrices` ci matrix targets and replace them with per-artifact samples. * Use enos sampling to generate different sample groups on all pull requests. * Update the enos scenario matrices to handle HSM and FIPS packages. * Simplify enos scenarios by using shared globals instead of cargo-culted locals. Note: This will require coordination with vault-enterprise to ensure a smooth migration to the new system. Integrating new scenarios or modifying existing scenarios/variants should be much smoother after this initial migration. [0] https://github.com/hashicorp/enos/pull/102 Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-08 19:31:09 +00:00
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
jobs:
build:
[QT-576] Optimize build workflow (#21486) (#21601) Improve our build workflow execution time by using custom runners, improved caching and conditional Web UI builds. Runners ------- We improve our build times[0] by using larger custom runners[1] when building the UI and Vault. Caching ------- We improve Vault caching by keeping a cache for each build job. This strategy has the following properties which should result in faster build times when `go.sum` hasn't been changed from prior builds, or when a pull request is retried or updated after a prior successful build: * Builds will restore cached Go modules and Go build cache according to the Go version, platform, architecture, go tags, and hash of `go.sum` that relates to each individual build workflow. This reduces the amount of time it will take to download the cache on hits and upload the cache on misses. * Parallel build workflows won't clobber each others build cache. This results in much faster compile times after cache hits because the Go compiler can reuse the platform, architecture, and tag specific build cache that it created on prior runs. * Older modules and build cache will not be uploaded when creating a new cache. This should result in lean cache sizes on an ongoing basis. * On cache misses we will have to upload our compressed module and build cache. This will slightly extend the build time for pull requests that modify `go.sum`. Web UI ------ We no longer build the web UI in every build workflow. Instead we separate the UI building into its own workflow and cache the resulting assets. The same UI assets are restored from cache during build worklows. This strategy has the following properties: * If the `ui` directory has not changed from prior builds we'll restore `http/web_ui` from cache and skip building the UI for no reason. * We continue to use the built-in `yarn` caching functionality in `action/setup-node`. The default mode saves the `yarn` global cache. to improve UI build times if the cache has not been modified. Changes ------- * Add per platform/archicture Go module and build caching * Move UI building into a separate job and cache the result * Restore UI cache during build * Pin workflows Notes ----- [0] https://hashicorp.atlassian.net/browse/QT-578 [1] https://github.com/hashicorp/vault/actions/runs/5415830307/jobs/9844829929 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-05 21:19:49 +00:00
runs-on: custom-linux-xl-vault-latest
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
name: Vault ${{ inputs.goos }} ${{ inputs.goarch }} v${{ inputs.vault-version }}
steps:
[QT-576] Optimize build workflow (#21486) (#21601) Improve our build workflow execution time by using custom runners, improved caching and conditional Web UI builds. Runners ------- We improve our build times[0] by using larger custom runners[1] when building the UI and Vault. Caching ------- We improve Vault caching by keeping a cache for each build job. This strategy has the following properties which should result in faster build times when `go.sum` hasn't been changed from prior builds, or when a pull request is retried or updated after a prior successful build: * Builds will restore cached Go modules and Go build cache according to the Go version, platform, architecture, go tags, and hash of `go.sum` that relates to each individual build workflow. This reduces the amount of time it will take to download the cache on hits and upload the cache on misses. * Parallel build workflows won't clobber each others build cache. This results in much faster compile times after cache hits because the Go compiler can reuse the platform, architecture, and tag specific build cache that it created on prior runs. * Older modules and build cache will not be uploaded when creating a new cache. This should result in lean cache sizes on an ongoing basis. * On cache misses we will have to upload our compressed module and build cache. This will slightly extend the build time for pull requests that modify `go.sum`. Web UI ------ We no longer build the web UI in every build workflow. Instead we separate the UI building into its own workflow and cache the resulting assets. The same UI assets are restored from cache during build worklows. This strategy has the following properties: * If the `ui` directory has not changed from prior builds we'll restore `http/web_ui` from cache and skip building the UI for no reason. * We continue to use the built-in `yarn` caching functionality in `action/setup-node`. The default mode saves the `yarn` global cache. to improve UI build times if the cache has not been modified. Changes ------- * Add per platform/archicture Go module and build caching * Move UI building into a separate job and cache the result * Restore UI cache during build * Pin workflows Notes ----- [0] https://hashicorp.atlassian.net/browse/QT-578 [1] https://github.com/hashicorp/vault/actions/runs/5415830307/jobs/9844829929 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-05 21:19:49 +00:00
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
[QT-589] Use the go module cache between CI and build (#21764) (#21790) In order to reliably store Go test times in the Github Actions cache we need to reduce our cache thrashing by not using more than 10gb over all of our caches. This change reduces our cache usage significantly by sharing Go module cache between our Go CI workflows and our build workflows. We lose our per-builder cache which will result in a bit of performance hit, but we'll enable better automatic rebalancing of our CI workflows. Overall we should see a per branch reduction in cache sizes from ~17gb to ~850mb. Some preliminary investigation into this new strategy: Prior build workflow strategy on a cache miss: Download modules: ~20s Build Vault: ~40s Upload cache: ~30s Total: ~1m30s Prior build workflow strategy on a cache hit: Download and decompress modules and build cache: ~12s Build Vault: ~15s Total: ~28s New build workflow strategy on a cache miss: Download modules: ~20 Build Vault: ~40s Upload cache: ~6s Total: ~1m6s New build workflow strategy on a cache hit: Download and decompress modules: ~3s Build Vault: ~40s Total: ~43s Expected time if we used no Go caching: Download modules: ~20 Build Vault: ~40s Total: ~1m Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-07-12 19:26:00 +00:00
- uses: ./.github/actions/set-up-go
[QT-576] Optimize build workflow (#21486) (#21601) Improve our build workflow execution time by using custom runners, improved caching and conditional Web UI builds. Runners ------- We improve our build times[0] by using larger custom runners[1] when building the UI and Vault. Caching ------- We improve Vault caching by keeping a cache for each build job. This strategy has the following properties which should result in faster build times when `go.sum` hasn't been changed from prior builds, or when a pull request is retried or updated after a prior successful build: * Builds will restore cached Go modules and Go build cache according to the Go version, platform, architecture, go tags, and hash of `go.sum` that relates to each individual build workflow. This reduces the amount of time it will take to download the cache on hits and upload the cache on misses. * Parallel build workflows won't clobber each others build cache. This results in much faster compile times after cache hits because the Go compiler can reuse the platform, architecture, and tag specific build cache that it created on prior runs. * Older modules and build cache will not be uploaded when creating a new cache. This should result in lean cache sizes on an ongoing basis. * On cache misses we will have to upload our compressed module and build cache. This will slightly extend the build time for pull requests that modify `go.sum`. Web UI ------ We no longer build the web UI in every build workflow. Instead we separate the UI building into its own workflow and cache the resulting assets. The same UI assets are restored from cache during build worklows. This strategy has the following properties: * If the `ui` directory has not changed from prior builds we'll restore `http/web_ui` from cache and skip building the UI for no reason. * We continue to use the built-in `yarn` caching functionality in `action/setup-node`. The default mode saves the `yarn` global cache. to improve UI build times if the cache has not been modified. Changes ------- * Add per platform/archicture Go module and build caching * Move UI building into a separate job and cache the result * Restore UI cache during build * Pin workflows Notes ----- [0] https://hashicorp.atlassian.net/browse/QT-578 [1] https://github.com/hashicorp/vault/actions/runs/5415830307/jobs/9844829929 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-05 21:19:49 +00:00
with:
[QT-589] Use the go module cache between CI and build (#21764) (#21790) In order to reliably store Go test times in the Github Actions cache we need to reduce our cache thrashing by not using more than 10gb over all of our caches. This change reduces our cache usage significantly by sharing Go module cache between our Go CI workflows and our build workflows. We lose our per-builder cache which will result in a bit of performance hit, but we'll enable better automatic rebalancing of our CI workflows. Overall we should see a per branch reduction in cache sizes from ~17gb to ~850mb. Some preliminary investigation into this new strategy: Prior build workflow strategy on a cache miss: Download modules: ~20s Build Vault: ~40s Upload cache: ~30s Total: ~1m30s Prior build workflow strategy on a cache hit: Download and decompress modules and build cache: ~12s Build Vault: ~15s Total: ~28s New build workflow strategy on a cache miss: Download modules: ~20 Build Vault: ~40s Upload cache: ~6s Total: ~1m6s New build workflow strategy on a cache hit: Download and decompress modules: ~3s Build Vault: ~40s Total: ~43s Expected time if we used no Go caching: Download modules: ~20 Build Vault: ~40s Total: ~1m Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-07-12 19:26:00 +00:00
github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }}
[QT-576] Optimize build workflow (#21486) (#21601) Improve our build workflow execution time by using custom runners, improved caching and conditional Web UI builds. Runners ------- We improve our build times[0] by using larger custom runners[1] when building the UI and Vault. Caching ------- We improve Vault caching by keeping a cache for each build job. This strategy has the following properties which should result in faster build times when `go.sum` hasn't been changed from prior builds, or when a pull request is retried or updated after a prior successful build: * Builds will restore cached Go modules and Go build cache according to the Go version, platform, architecture, go tags, and hash of `go.sum` that relates to each individual build workflow. This reduces the amount of time it will take to download the cache on hits and upload the cache on misses. * Parallel build workflows won't clobber each others build cache. This results in much faster compile times after cache hits because the Go compiler can reuse the platform, architecture, and tag specific build cache that it created on prior runs. * Older modules and build cache will not be uploaded when creating a new cache. This should result in lean cache sizes on an ongoing basis. * On cache misses we will have to upload our compressed module and build cache. This will slightly extend the build time for pull requests that modify `go.sum`. Web UI ------ We no longer build the web UI in every build workflow. Instead we separate the UI building into its own workflow and cache the resulting assets. The same UI assets are restored from cache during build worklows. This strategy has the following properties: * If the `ui` directory has not changed from prior builds we'll restore `http/web_ui` from cache and skip building the UI for no reason. * We continue to use the built-in `yarn` caching functionality in `action/setup-node`. The default mode saves the `yarn` global cache. to improve UI build times if the cache has not been modified. Changes ------- * Add per platform/archicture Go module and build caching * Move UI building into a separate job and cache the result * Restore UI cache during build * Pin workflows Notes ----- [0] https://hashicorp.atlassian.net/browse/QT-578 [1] https://github.com/hashicorp/vault/actions/runs/5415830307/jobs/9844829929 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-05 21:19:49 +00:00
- name: Restore UI from cache
uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
with:
[QT-589] Use the go module cache between CI and build (#21764) (#21790) In order to reliably store Go test times in the Github Actions cache we need to reduce our cache thrashing by not using more than 10gb over all of our caches. This change reduces our cache usage significantly by sharing Go module cache between our Go CI workflows and our build workflows. We lose our per-builder cache which will result in a bit of performance hit, but we'll enable better automatic rebalancing of our CI workflows. Overall we should see a per branch reduction in cache sizes from ~17gb to ~850mb. Some preliminary investigation into this new strategy: Prior build workflow strategy on a cache miss: Download modules: ~20s Build Vault: ~40s Upload cache: ~30s Total: ~1m30s Prior build workflow strategy on a cache hit: Download and decompress modules and build cache: ~12s Build Vault: ~15s Total: ~28s New build workflow strategy on a cache miss: Download modules: ~20 Build Vault: ~40s Upload cache: ~6s Total: ~1m6s New build workflow strategy on a cache hit: Download and decompress modules: ~3s Build Vault: ~40s Total: ~43s Expected time if we used no Go caching: Download modules: ~20 Build Vault: ~40s Total: ~1m Signed-off-by: Ryan Cragun <me@ryan.ec> Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-07-12 19:26:00 +00:00
# Restore the UI asset from the UI build workflow. Never use a partial restore key.
[QT-576] Optimize build workflow (#21486) (#21601) Improve our build workflow execution time by using custom runners, improved caching and conditional Web UI builds. Runners ------- We improve our build times[0] by using larger custom runners[1] when building the UI and Vault. Caching ------- We improve Vault caching by keeping a cache for each build job. This strategy has the following properties which should result in faster build times when `go.sum` hasn't been changed from prior builds, or when a pull request is retried or updated after a prior successful build: * Builds will restore cached Go modules and Go build cache according to the Go version, platform, architecture, go tags, and hash of `go.sum` that relates to each individual build workflow. This reduces the amount of time it will take to download the cache on hits and upload the cache on misses. * Parallel build workflows won't clobber each others build cache. This results in much faster compile times after cache hits because the Go compiler can reuse the platform, architecture, and tag specific build cache that it created on prior runs. * Older modules and build cache will not be uploaded when creating a new cache. This should result in lean cache sizes on an ongoing basis. * On cache misses we will have to upload our compressed module and build cache. This will slightly extend the build time for pull requests that modify `go.sum`. Web UI ------ We no longer build the web UI in every build workflow. Instead we separate the UI building into its own workflow and cache the resulting assets. The same UI assets are restored from cache during build worklows. This strategy has the following properties: * If the `ui` directory has not changed from prior builds we'll restore `http/web_ui` from cache and skip building the UI for no reason. * We continue to use the built-in `yarn` caching functionality in `action/setup-node`. The default mode saves the `yarn` global cache. to improve UI build times if the cache has not been modified. Changes ------- * Add per platform/archicture Go module and build caching * Move UI building into a separate job and cache the result * Restore UI cache during build * Pin workflows Notes ----- [0] https://hashicorp.atlassian.net/browse/QT-578 [1] https://github.com/hashicorp/vault/actions/runs/5415830307/jobs/9844829929 Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-07-05 21:19:49 +00:00
enableCrossOsArchive: true
fail-on-cache-miss: true
path: http/web_ui
key: ${{ inputs.web-ui-cache-key }}
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
- name: Build Vault
env:
cherrypick of https://github.com/hashicorp/vault/commit/a9a4b0b9ff4459deaeab4ba62927a2e2633be878 (#22813)
2023-09-06 17:24:39 +00:00
GO_TAGS: ${{ inputs.go-tags }}
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
CGO_ENABLED: ${{ inputs.cgo-enabled }}
GOARCH: ${{ inputs.goarch }}
GOOS: ${{ inputs.goos }}
cherrypick of https://github.com/hashicorp/vault/commit/a9a4b0b9ff4459deaeab4ba62927a2e2633be878 (#22813)
2023-09-06 17:24:39 +00:00
VERSION: ${{ inputs.vault-version }}
run:
make ci-build
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
- name: Determine artifact basename
env:
GOARCH: ${{ inputs.goarch }}
GOOS: ${{ inputs.goos }}
cherrypick of https://github.com/hashicorp/vault/commit/a9a4b0b9ff4459deaeab4ba62927a2e2633be878 (#22813)
2023-09-06 17:24:39 +00:00
VERSION: ${{ inputs.vault-version }}
Migrating CircleCI Jobs to GHA Workflow (#19662) * address lint reports * add diff-oss-ci and test-ui jobs to ci GHA workflow * Add actions linter workflow * Fix actions linter errors * pin 3rd party components with SHA hash and limit actionlint workflow to pull requests touching paths under .github directory * Fix actionlint runner * pin SHA hash of 3rd party components use .go-version file to provide go version to setup-go action remove unncessary ref parameter in checkout action --------- Co-authored-by: Brian Shore <bshore@hashicorp.com>
2023-03-22 19:02:06 +00:00
run: echo "ARTIFACT_BASENAME=$(make ci-get-artifact-basename)" >> "$GITHUB_ENV"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
- name: Bundle Vault
env:
BUNDLE_PATH: out/${{ env.ARTIFACT_BASENAME }}.zip
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
run: make ci-bundle
ci: request vpc quota increase (#20360) * Fix regions on two service quotas * Request an increase in VPCs per region * Pin github actions workflows Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-05-22 17:18:06 +00:00
- uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
with:
name: ${{ env.ARTIFACT_BASENAME }}.zip
path: out/${{ env.ARTIFACT_BASENAME }}.zip
if-no-files-found: error
- if: ${{ inputs.create-packages }}
uses: hashicorp/actions-packaging-linux@v1
with:
name: ${{ github.event.repository.name }}
description: Vault is a tool for secrets management, encryption as a service, and privileged access management.
arch: ${{ inputs.goarch }}
version: ${{ inputs.vault-version }}
maintainer: HashiCorp
homepage: https://github.com/hashicorp/vault
license: MPL-2.0
binary: dist/${{ inputs.package-name }}
deb_depends: openssl
rpm_depends: openssl
config_dir: .release/linux/package/
preinstall: .release/linux/preinst
postinstall: .release/linux/postinst
postremove: .release/linux/postrm
- if: ${{ inputs.create-packages }}
name: Determine package file names
run: |
Migrating CircleCI Jobs to GHA Workflow (#19662) * address lint reports * add diff-oss-ci and test-ui jobs to ci GHA workflow * Add actions linter workflow * Fix actions linter errors * pin 3rd party components with SHA hash and limit actionlint workflow to pull requests touching paths under .github directory * Fix actionlint runner * pin SHA hash of 3rd party components use .go-version file to provide go version to setup-go action remove unncessary ref parameter in checkout action --------- Co-authored-by: Brian Shore <bshore@hashicorp.com>
2023-03-22 19:02:06 +00:00
echo "RPM_PACKAGE=$(basename out/*.rpm)" >> "$GITHUB_ENV"
echo "DEB_PACKAGE=$(basename out/*.deb)" >> "$GITHUB_ENV"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
- if: ${{ inputs.create-packages }}
ci: request vpc quota increase (#20360) * Fix regions on two service quotas * Request an increase in VPCs per region * Pin github actions workflows Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-05-22 17:18:06 +00:00
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
with:
name: ${{ env.RPM_PACKAGE }}
path: out/${{ env.RPM_PACKAGE }}
if-no-files-found: error
- if: ${{ inputs.create-packages }}
ci: request vpc quota increase (#20360) * Fix regions on two service quotas * Request an increase in VPCs per region * Pin github actions workflows Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-05-22 17:18:06 +00:00
uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
with:
name: ${{ env.DEB_PACKAGE }}
path: out/${{ env.DEB_PACKAGE }}
if-no-files-found: error