luxolus
/
open-vault
2
0
Fork 0
Code Issues 1 Pull Requests Releases Activity
open-vault/scripts/ci-helper.sh

318 lines
7.4 KiB
Bash
Raw Normal View History

[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
#!/usr/bin/env bash
adding copyright header (#19555) * adding copyright header * fix fmt and a test
2023-03-15 16:00:52 +00:00
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
# The ci-helper is used to determine build metadata, build Vault binaries,
# package those binaries into artifacts, and execute tests with those artifacts.
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
set -euo pipefail
# We don't want to get stuck in some kind of interactive pager
export GIT_PAGER=cat
# Get the full version information
function version() {
local version
local prerelease
local metadata
version=$(version_base)
prerelease=$(version_pre)
metadata=$(version_metadata)
if [ -n "$metadata" ] && [ -n "$prerelease" ]; then
echo "$version-$prerelease+$metadata"
elif [ -n "$metadata" ]; then
echo "$version+$metadata"
elif [ -n "$prerelease" ]; then
echo "$version-$prerelease"
else
echo "$version"
fi
}
# Get the base version
function version_base() {
: "${VAULT_VERSION:=""}"
if [ -n "$VAULT_VERSION" ]; then
echo "$VAULT_VERSION"
return
fi
Move version out of SDK. (#14229) Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 18:29:51 +00:00
: "${VERSION_FILE:=$(repo_root)/version/version_base.go}"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
awk '$1 == "Version" && $2 == "=" { gsub(/"/, "", $3); print $3 }' < "$VERSION_FILE"
}
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
# Get the version major
function version_major() {
version_base | cut -d '.' -f 1
}
# Get the version minor
function version_minor() {
version_base | cut -d '.' -f 2
}
# Get the version patch
function version_patch() {
version_base | cut -d '.' -f 3
}
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
# Get the version pre-release
function version_pre() {
: "${VAULT_PRERELEASE:=""}"
if [ -n "$VAULT_PRERELEASE" ]; then
echo "$VAULT_PRERELEASE"
return
fi
Move version out of SDK. (#14229) Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 18:29:51 +00:00
: "${VERSION_FILE:=$(repo_root)/version/version_base.go}"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
awk '$1 == "VersionPrerelease" && $2 == "=" { gsub(/"/, "", $3); print $3 }' < "$VERSION_FILE"
}
# Get the version metadata, which is commonly the edition
function version_metadata() {
: "${VAULT_METADATA:=""}"
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
if [[ (-n "$VAULT_METADATA") && ("$VAULT_METADATA" != "oss") ]]; then
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
echo "$VAULT_METADATA"
return
fi
Move version out of SDK. (#14229) Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 18:29:51 +00:00
: "${VERSION_FILE:=$(repo_root)/version/version_base.go}"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
awk '$1 == "VersionMetadata" && $2 == "=" { gsub(/"/, "", $3); print $3 }' < "$VERSION_FILE"
}
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
# Get the version formatted for Debian and RHEL packages
function version_package() {
version | awk '{ gsub("-","~",$1); print $1 }'
}
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
# Get the build date from the latest commit since it can be used across all
# builds
function build_date() {
# It's tricky to do an RFC3339 format in a cross platform way, so we hardcode UTC
: "${DATE_FORMAT:="%Y-%m-%dT%H:%M:%SZ"}"
git show --no-show-signature -s --format=%cd --date=format:"$DATE_FORMAT" HEAD
}
# Get the revision, which is the latest commit SHA
function build_revision() {
git rev-parse HEAD
}
# Determine our repository by looking at our origin URL
function repo() {
basename -s .git "$(git config --get remote.origin.url)"
}
# Determine the root directory of the repository
function repo_root() {
git rev-parse --show-toplevel
}
# Determine the artifact basename based on metadata
function artifact_basename() {
: "${PKG_NAME:="vault"}"
: "${GOOS:=$(go env GOOS)}"
: "${GOARCH:=$(go env GOARCH)}"
echo "${PKG_NAME}_$(version)_${GOOS}_${GOARCH}"
}
# Build the UI
function build_ui() {
local repo_root
repo_root=$(repo_root)
pushd "$repo_root"
mkdir -p http/web_ui
popd
pushd "$repo_root/ui"
yarn install --ignore-optional
npm rebuild node-sass
yarn --verbose run build
popd
}
# Build Vault
function build() {
local version
local revision
local prerelease
local build_date
local ldflags
local msg
# Get or set our basic build metadata
version=$(version_base)
revision=$(build_revision)
metadata=$(version_metadata)
prerelease=$(version_pre)
build_date=$(build_date)
: "${GO_TAGS:=""}"
: "${KEEP_SYMBOLS:=""}"
# Build our ldflags
msg="--> Building Vault v$version, revision $revision, built $build_date"
# Strip the symbol and dwarf information by default
if [ -n "$KEEP_SYMBOLS" ]; then
ldflags=""
else
ldflags="-s -w "
fi
Move version out of SDK. (#14229) Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 18:29:51 +00:00
ldflags="${ldflags}-X github.com/hashicorp/vault/version.Version=$version -X github.com/hashicorp/vault/version.GitCommit=$revision -X github.com/hashicorp/vault/version.BuildDate=$build_date"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
if [ -n "$prerelease" ]; then
msg="${msg}, prerelease ${prerelease}"
Move version out of SDK. (#14229) Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 18:29:51 +00:00
ldflags="${ldflags} -X github.com/hashicorp/vault/version.VersionPrerelease=$prerelease"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
fi
if [ -n "$metadata" ]; then
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
msg="${msg}, metadata ${metadata}"
Move version out of SDK. (#14229) Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 18:29:51 +00:00
ldflags="${ldflags} -X github.com/hashicorp/vault/version.VersionMetadata=$metadata"
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
fi
# Build vault
echo "$msg"
pushd "$(repo_root)"
mkdir -p dist
mkdir -p out
set -x
go build -v -tags "$GO_TAGS" -ldflags "$ldflags" -o dist/
set +x
popd
}
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
# Bundle the dist directory into a zip
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
function bundle() {
: "${BUNDLE_PATH:=$(repo_root)/vault.zip}"
echo "--> Bundling dist/* to $BUNDLE_PATH"
zip -r -j "$BUNDLE_PATH" dist/
}
# Prepare legal requirements for packaging
function prepare_legal() {
: "${PKG_NAME:="vault"}"
pushd "$(repo_root)"
mkdir -p dist
curl -o dist/EULA.txt https://eula.hashicorp.com/EULA.txt
curl -o dist/TermsOfEvaluation.txt https://eula.hashicorp.com/TermsOfEvaluation.txt
mkdir -p ".release/linux/package/usr/share/doc/$PKG_NAME"
cp dist/EULA.txt ".release/linux/package/usr/share/doc/$PKG_NAME/EULA.txt"
cp dist/TermsOfEvaluation.txt ".release/linux/package/usr/share/doc/$PKG_NAME/TermsOfEvaluation.txt"
popd
}
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
# Determine the matrix group number that we'll select for execution. If the
# MATRIX_TEST_GROUP environment variable has set then it will always return
# that value. If has not been set, we will randomly select a number between 1
# and the value of MATRIX_MAX_TEST_GROUPS.
function matrix_group_id() {
: "${MATRIX_TEST_GROUP:=""}"
if [ -n "$MATRIX_TEST_GROUP" ]; then
echo "$MATRIX_TEST_GROUP"
return
fi
: "${MATRIX_MAX_TEST_GROUPS:=1}"
awk -v min=1 -v max=$MATRIX_MAX_TEST_GROUPS 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'
}
# Filter matrix file reads in the contents of MATRIX_FILE and filters out
# scenarios that are not in the current test group and/or those that have not
# met minimux or maximum version requirements.
function matrix_filter_file() {
: "${MATRIX_FILE:=""}"
if [ -z "$MATRIX_FILE" ]; then
echo "You must specify the MATRIX_FILE variable for this command" >&2
exit 1
fi
: "${MATRIX_TEST_GROUP:=$(matrix_group_id)}"
local path
local matrix
path=$(readlink -f $MATRIX_FILE)
matrix=$(cat "$path" | jq ".include |
map(. |
select(
((.min_minor_version == null) or (.min_minor_version <= $(version_minor))) and
((.max_minor_version == null) or (.max_minor_version >= $(version_minor))) and
((.test_group == null) or (.test_group == $MATRIX_TEST_GROUP))
)
)"
)
echo "{\"include\":$matrix}" | jq -c .
}
# Run the CI Helper
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
function main() {
case $1 in
artifact-basename)
artifact_basename
;;
build)
build
;;
build-ui)
build_ui
;;
bundle)
bundle
;;
date)
build_date
;;
prepare-legal)
prepare_legal
;;
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
matrix-filter-file)
matrix_filter_file
;;
matrix-group-id)
matrix_group_id
;;
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
revision)
build_revision
;;
version)
version
;;
version-base)
version_base
;;
version-pre)
version_pre
;;
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
version-major)
version_major
;;
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
version-meta)
version_metadata
;;
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
version-minor)
version_minor
;;
version-package)
version_package
;;
version-patch)
version_patch
;;
[QT-358] Unify CRT and local builder workflows (#17766) Here we make the following major changes: * Centralize CRT builder logic into a script utility so that we can share the logic for building artifacts in CI or locally. * Simplify the build workflow by calling a reusable workflow many times instead of repeating the contents. * Create a workflow that validates whether or not the build workflow and all child workflows have succeeded to allow for merge protection. Motivation * We need branch requirements for the build workflow and all subsequent integration tests (QT-353) * We need to ensure that the Enos local builder works (QT-558) * Debugging build failures can be difficult because one has to hand craft the steps to recreate the build * Merge conflicts between Vault OSS and Vault ENT build workflows are quite painful. As the build workflow must be the same file and name we'll reduce what is contained in each that is unique. Implementations of building will be unique per edition so we don't have to worry about conflict resolution. * Since we're going to be touching the build workflow to do the first two items we might as well try and improve those other issues at the same time to reduce the overhead of backports and conflicts. Considerations * Build logic for Vault OSS and Vault ENT differs * The Enos local builder was duplicating a lot of what we did in the CRT build workflow * Version and other artifact metadata has been an issue before. Debugging it has been tedious and error prone. * The build workflow is full of brittle copy and paste that is hard to understand, especially for all of the release editions in Vault Enterprise * Branch check requirements for workflows are incredibly painful to use for workflows that are dynamic or change often. The required workflows have to be configured in Github settings by administrators. They would also prevent us from having simple docs PRs since required integration workflows always have to run to satisfy branch requirements. * Doormat credentials requirements that are coming will require us to modify which event types trigger workflows. This changes those ahead of time since we're doing so much to build workflow. The only noticeable impact will be that the build workflow no longer runs on pushes to non-main or release branches. In order to test other branches it requires a workflow_dispatch from the Actions tab or a pull request. Solutions * Centralize the logic that determines build metadata and creates releasable Vault artifacts. Instead of cargo-culting logic multiple times in the build workflow and the Enos local modules, we now have a crt-builder script which determines build metadata and also handles building the UI, Vault, and the package bundle. There are make targets for all of the available sub-commands. Now what we use in the pipeline is the same thing as the local builder, and it can be executed locally by developers. The crt-builder script works in OSS and Enterprise so we will never have to deal with them being divergent or with special casing things in the build workflow. * Refactor the bulk of the Vault building into a reusable workflow that we can call multiple times. This allows us to define Vault builds in a much simpler manner and makes resolving merge conflicts much easier. * Rather than trying to maintain a list and manually configure the branch check requirements for build, we'll trigger a single workflow that uses the github event system to determine if the build workflow (all of the sub-workflows included) have passed. We'll then create branch restrictions on that single workflow down the line. Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 20:14:43 +00:00
*)
echo "unknown sub-command" >&2
exit 1
;;
esac
}
main "$@"