luxolus/open-vault
2
0
Fork 0
Code Issues 1 Pull requests Releases Activity
open-vault/.github/workflows/build.yml
hc-github-team-secure-vault-core 59cbdcda39
[QT-589] Use the go module cache between CI and build (#21764) (#21790) 
In order to reliably store Go test times in the Github Actions cache we
need to reduce our cache thrashing by not using more than 10gb over all
of our caches. This change reduces our cache usage significantly by
sharing Go module cache between our Go CI workflows and our build
workflows. We lose our per-builder cache which will result in a bit of
performance hit, but we'll enable better automatic rebalancing of our CI
workflows. Overall we should see a per branch reduction in cache sizes
from ~17gb to ~850mb.

Some preliminary investigation into this new strategy:

Prior build workflow strategy on a cache miss:
  Download modules: ~20s
  Build Vault: ~40s
  Upload cache: ~30s
  Total: ~1m30s

Prior build workflow strategy on a cache hit:
  Download and decompress modules and build cache: ~12s
  Build Vault: ~15s
  Total: ~28s

New build workflow strategy on a cache miss:
  Download modules: ~20
  Build Vault: ~40s
  Upload cache: ~6s
  Total: ~1m6s

New build workflow strategy on a cache hit:
  Download and decompress modules: ~3s
  Build Vault: ~40s
  Total: ~43s

Expected time if we used no Go caching:
  Download modules: ~20
  Build Vault: ~40s
  Total: ~1m

Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-07-12 19:26:00 +00:00

305 lines
12 KiB
YAML
Raw Blame History

---
name: build
on:
workflow_dispatch:
pull_request:
# The default types for pull_request are [ opened, synchronize, reopened ].
# This is insufficient for our needs, since we're skipping stuff on PRs in
# draft mode. By adding the ready_for_review type, when a draft pr is marked
# ready, we run everything, including the stuff we'd have skipped up until now.
types: [opened, synchronize, reopened, ready_for_review]
push:
branches:
- main
- release/**
concurrency:
group: ${{ github.head_ref || github.run_id }}-build
cancel-in-progress: true
jobs:
# verify-changes determines if the changes are only for docs (website)
verify-changes:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
outputs:
is_docs_change: ${{ steps.get-changeddir.outputs.is_docs_change }}
steps:
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0 # Use fetch depth 0 for comparing changes to base branch
- name: Get changed directories
id: get-changeddir
env:
TYPE: ${{ github.event_name }}
REF_NAME: ${{ github.ref_name }}
BASE: ${{ github.base_ref }}
run: ./.github/scripts/verify_changes.sh ${{ env.TYPE }} ${{ env.REF_NAME }} ${{ env.BASE }}
product-metadata:
# do not run build and test steps for docs changes
# Following https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/troubleshooting-required-status-checks#handling-skipped-but-required-checks
# we conditionally skip the build and tests for docs(website) changes
if: ${{ needs.verify-changes.outputs.is_docs_change == 'false' }}
runs-on: ubuntu-latest
needs: verify-changes
outputs:
build-date: ${{ steps.get-metadata.outputs.build-date }}
filepath: ${{ steps.generate-metadata-file.outputs.filepath }}
matrix-test-group: ${{ steps.get-metadata.outputs.matrix-test-group }}
package-name: ${{ steps.get-metadata.outputs.package-name }}
vault-revision: ${{ steps.get-metadata.outputs.vault-revision }}
vault-version: ${{ steps.get-metadata.outputs.vault-version }}
vault-base-version: ${{ steps.get-metadata.outputs.vault-base-version }}
steps:
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
- name: Ensure Go modules are cached
uses: ./.github/actions/set-up-go
id: set-up-go
with:
github-token: ${{ secrets.ELEVATED_GITHUB_TOKEN }}
no-restore: true # don't download them on a cache hit
- name: Get metadata
id: get-metadata
env:
# MATRIX_MAX_TEST_GROUPS is required to determine the randomly selected
# test group. It should be set to the highest test_group used in the
# enos-run-matrices.
MATRIX_MAX_TEST_GROUPS: 5
run: |
# shellcheck disable=SC2129
echo "build-date=$(make ci-get-date)" >> "$GITHUB_OUTPUT"
echo "matrix-test-group=$(make ci-get-matrix-group-id)" >> "$GITHUB_OUTPUT"
echo "package-name=vault" >> "$GITHUB_OUTPUT"
echo "vault-base-version=$(make ci-get-version-base)" >> "$GITHUB_OUTPUT"
echo "vault-revision=$(make ci-get-revision)" >> "$GITHUB_OUTPUT"
echo "vault-version=$(make ci-get-version)" >> "$GITHUB_OUTPUT"
- uses: hashicorp/actions-generate-metadata@v1
id: generate-metadata-file
with:
version: ${{ steps.get-metadata.outputs.vault-version }}
product: ${{ steps.get-metadata.outputs.package-name }}
- uses: actions/upload-artifact@0b7f8abb1508181956e8e162db84b466c27e18ce # v3.1.2
with:
name: metadata.json
path: ${{ steps.generate-metadata-file.outputs.filepath }}
if-no-files-found: error
build-ui:
name: UI
runs-on: custom-linux-xl-vault-latest
outputs:
cache-key: ui-${{ steps.ui-hash.outputs.ui-hash }}
steps:
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
- name: Get UI hash
id: ui-hash
run: echo "ui-hash=$(git ls-tree HEAD ui --object-only)" >> "$GITHUB_OUTPUT"
- name: Set up UI asset cache
id: cache-ui-assets
uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1
with:
enableCrossOsArchive: true
lookup-only: true
path: http/web_ui
# Only restore the UI asset cache if we haven't modified anything in the ui directory.
# Never do a partial restore of the web_ui if we don't get a cache hit.
key: ui-${{ steps.ui-hash.outputs.ui-hash }}
- if: steps.cache-ui-assets.outputs.cache-hit != 'true'
name: Set up node and yarn
uses: actions/setup-node@64ed1c7eab4cce3362f8c340dee64e5eaeef8f7c # v3.6.0
with:
node-version-file: ui/package.json
cache: yarn
cache-dependency-path: ui/yarn.lock
- if: steps.cache-ui-assets.outputs.cache-hit != 'true'
name: Build UI
run: make ci-build-ui
build-other:
name: Other
needs:
- product-metadata
- build-ui
strategy:
matrix:
goos: [freebsd, windows, netbsd, openbsd, solaris]
goarch: [386, amd64, arm]
exclude:
- goos: solaris
goarch: 386
- goos: solaris
goarch: arm
- goos: windows
goarch: arm
fail-fast: true
uses: ./.github/workflows/build-vault-oss.yml
with:
create-packages: false
goarch: ${{ matrix.goarch }}
goos: ${{ matrix.goos }}
go-tags: ui
package-name: ${{ needs.product-metadata.outputs.package-name }}
web-ui-cache-key: ${{ needs.build-ui.outputs.cache-key }}
vault-version: ${{ needs.product-metadata.outputs.vault-version }}
secrets: inherit
build-linux:
name: Linux
needs:
- product-metadata
- build-ui
strategy:
matrix:
goos: [linux]
goarch: [arm, arm64, 386, amd64]
fail-fast: true
uses: ./.github/workflows/build-vault-oss.yml
with:
goarch: ${{ matrix.goarch }}
goos: ${{ matrix.goos }}
go-tags: ui
package-name: ${{ needs.product-metadata.outputs.package-name }}
web-ui-cache-key: ${{ needs.build-ui.outputs.cache-key }}
vault-version: ${{ needs.product-metadata.outputs.vault-version }}
secrets: inherit
build-darwin:
name: Darwin
needs:
- product-metadata
- build-ui
strategy:
matrix:
goos: [darwin]
goarch: [amd64, arm64]
fail-fast: true
uses: ./.github/workflows/build-vault-oss.yml
with:
create-packages: false
goarch: ${{ matrix.goarch }}
goos: ${{ matrix.goos }}
go-tags: ui
package-name: ${{ needs.product-metadata.outputs.package-name }}
web-ui-cache-key: ${{ needs.build-ui.outputs.cache-key }}
vault-version: ${{ needs.product-metadata.outputs.vault-version }}
secrets: inherit
build-docker:
name: Docker image
needs:
- product-metadata
- build-linux
runs-on: ubuntu-latest
strategy:
matrix:
arch: [arm, arm64, 386, amd64]
steps:
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
- uses: hashicorp/actions-docker-build@v1
with:
version: ${{ needs.product-metadata.outputs.vault-version }}
target: default
arch: ${{ matrix.arch }}
zip_artifact_name: vault_${{ needs.product-metadata.outputs.vault-version }}_linux_${{ matrix.arch }}.zip
tags: |
docker.io/hashicorp/${{ github.event.repository.name }}:${{ needs.product-metadata.outputs.vault-version }}
public.ecr.aws/hashicorp/${{ github.event.repository.name }}:${{ needs.product-metadata.outputs.vault-version }}
build-ubi:
name: UBI image
needs:
- product-metadata
- build-linux
runs-on: ubuntu-latest
strategy:
matrix:
arch: [amd64]
steps:
- uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
- uses: hashicorp/actions-docker-build@v1
with:
version: ${{ needs.product-metadata.outputs.vault-version }}
target: ubi
arch: ${{ matrix.arch }}
zip_artifact_name: vault_${{ needs.product-metadata.outputs.vault-version }}_linux_${{ matrix.arch }}.zip
redhat_tag: quay.io/redhat-isv-containers/5f89bb5e0b94cf64cfeb500a:${{ needs.product-metadata.outputs.vault-version }}-ubi
test:
name: Test ${{ matrix.build-artifact-name }}
# Only run the Enos workflow against branches that are created from the
# hashicorp/vault repository. This has the effect of limiting execution of
# Enos scenarios to branches that originate from authors that have write
# access to hashicorp/vault repository. This is required as Github Actions
# will not populate the required secrets for branches created by outside
# contributors in order to protect the secrets integrity.
# This condition can be removed in future if enos workflow is updated to
# workflow_run event
if: "! github.event.pull_request.head.repo.fork"
needs:
- product-metadata
- build-linux
uses: ./.github/workflows/test-run-enos-scenario-matrix.yml
strategy:
fail-fast: false
matrix:
include:
- matrix-file-name: build-github-oss-linux-amd64-zip
build-artifact-name: vault_${{ needs.product-metadata.outputs.vault-version }}_linux_amd64.zip
- matrix-file-name: build-github-oss-linux-arm64-zip
build-artifact-name: vault_${{ needs.product-metadata.outputs.vault-version }}_linux_arm64.zip
with:
build-artifact-name: ${{ matrix.build-artifact-name }}
matrix-file-name: ${{ matrix.matrix-file-name }}
matrix-test-group: ${{ needs.product-metadata.outputs.matrix-test-group }}
vault-edition: oss
vault-revision: ${{ needs.product-metadata.outputs.vault-revision }}
ssh-key-name: ${{ github.event.repository.name }}-ci-ssh-key
secrets: inherit
test-docker-k8s:
name: Test Docker K8s
# Only run the Enos workflow against branches that are created from the
# hashicorp/vault repository. This has the effect of limiting execution of
# Enos scenarios to branches that originate from authors that have write
# access to hashicorp/vault repository. This is required as Github Actions
# will not populate the required secrets for branches created by outside
# contributors in order to protect the secrets integrity.
# GHA secrets are only ready on workflow_run for public repo
# This condition can be removed in future if enos workflow is updated to
# workflow_run event
if: "! github.event.pull_request.head.repo.fork"
needs:
- product-metadata
- build-docker
uses: ./.github/workflows/enos-run-k8s.yml
with:
artifact-build-date: ${{ needs.product-metadata.outputs.build-date }}
artifact-name: ${{ github.event.repository.name }}_default_linux_amd64_${{ needs.product-metadata.outputs.vault-version }}_${{ needs.product-metadata.outputs.vault-revision }}.docker.tar
artifact-revision: ${{ needs.product-metadata.outputs.vault-revision }}
artifact-version: ${{ needs.product-metadata.outputs.vault-version }}
secrets: inherit
completed-successfully:
# We force a failure if any of the dependent jobs fail,
# this is a workaround for the issue reported https://github.com/actions/runner/issues/2566
if: |
always() && (cancelled() ||
contains(needs.*.result, 'cancelled') ||
contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
needs:
- build-other
- build-linux
- build-darwin
- build-docker
- build-ubi
- test
- test-docker-k8s
steps:
- run: |
echo "Some of the required build and test workflows have failed!"
exit 1
Reference in a new issue View git blame Copy permalink