* Update tests-completed job in CI to appear as success when one of the required checks is skipped (but not cancelled)
* Fix typo in tests-completed
---------
Co-authored-by: Marc Boudreau <marc.boudreau@hashicorp.com>
* Add a new category of runners to the CI workflow; use new, dedicated runners on OSS; adjust runner sizes for jobs
Co-authored-by: Marc Boudreau <marc.boudreau@hashicorp.com>
* deprecation check
* adding script
* add execute permission to script
* revert changes
* adding the script back
* added working script for local and GHA
* give execute permissions
* updating revgrep
* adding changes to script, tools
* run go mod tidy
* removing default ref
* make bootstrap
* adding to makefile
* address lint reports
* add diff-oss-ci and test-ui jobs to ci GHA workflow
* Add actions linter workflow
* Fix actions linter errors
* pin 3rd party components with SHA hash and limit actionlint workflow to pull requests touching paths under .github directory
* Fix actionlint runner
* pin SHA hash of 3rd party components
use .go-version file to provide go version to setup-go action
remove unncessary ref parameter in checkout action
---------
Co-authored-by: Brian Shore <bshore@hashicorp.com>
* Add a GHA job running Go tests with race detection enabled to the CI workflow
* Incorporate logic from test-go-race into the test-go testing matrix
* Make test-go testing matrix job names more meaningful
* Fix some a bug in script's logic
* Experiment: bump wait time in the failing TestLoginMFASinglePhase test to see if that makes a difference
* Lower the wait time in TestLoginMFASinglePhase
* Change the wait time in TestLoginMFASinglePhase to 15
* Add more detail to test-go testing matrix job names
* Test whether we already have access to larger runners
* Run Go tests with enabled data race detection from a separate job than the standard suite of tests
* Tweak runner sizes for OSS
* Try rebalancing test buckets
* Change instance type for larger ENT runners
* Undo rebalancing of test buckets as it changed nothing
* Change instance type for larger OSS runners
* Change the way we generate names for matrix jobs
* Consolidate the Go build tags variables, update them to use comma as a separator and fix the if statement in test-go
* Fix a typo
* replace use of os.Unsetenv in test with t.Setenv and remove t.Parallel from test that rely on env being modified.
* experiment with using fromJSON function
* revert previous experiment
* including double quotes in the output value for the string ubuntu-latest
* use go run to launch gofumpt
* Migrate subset of CircleCI ci workflow to GitHub Actions
Runs test-go and test-go-remote-docker with a static splitting of test packages
* [skip actions] add comment to explain the purpose of test-generate-test-package-lists.sh and what to do if it fails
* change trigger to push
---------
Co-authored-by: Kuba Wieczorek <kuba.wieczorek@hashicorp.com>
* example for checking go doc tests
* add analyzer test and action
* get metadata step
* install revgrep
* fix for ci
* add revgrep to go.mod
* clarify how analysistest works
This uses aws-nuke and awslimitchecker to monitor the new vault CI account to clean up and prevent resource quota exhaustion. AWS-nuke will scan all regions of the accounts for lingering resources enos/terraform didn't clean up, and if they don't match exclusion criteria, delete them every night. By default, we exclude corp-sec created resources, our own CI resources, and when possible, anything created within the past 72 hours. Because this account is dedicated to CI, users should not expect resources to persist beyond this without additional configuration.
- enos-run workflow will checkout the `main` branch by default, which would pass incorrect metadata to the workflow
so we use the `revision` passed by the calling workflow to checkout the sha and get the relavant metadata`
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
Signed-off-by: Jaymala Sinha <jaymala@hashicorp.com>
Introducing a new approach to testing Vault artifacts before merge
and after merge/notorization/signing. Rather than run a few static
scenarios across the artifacts, we now have the ability to run a
pseudo random sample of scenarios across many different build artifacts.
We've added 20 possible scenarios for the AMD64 and ARM64 binary
bundles, which we've broken into five test groups. On any given push to
a pull request branch, we will now choose a random test group and
execute its corresponding scenarios against the resulting build
artifacts. This gives us greater test coverage but lets us split the
verification across many different pull requests.
The post-merge release testing pipeline behaves in a similar fashion,
however, the artifacts that we use for testing have been notarized and
signed prior to testing. We've also reduce the number of groups so that
we run more scenarios after merge to a release branch.
We intend to take what we've learned building this in Github Actions and
roll it into an easier to use feature that is native to Enos. Until then,
we'll have to manually add scenarios to each matrix file and manually
number the test group. It's important to note that Github requires every
matrix to include at least one vector, so every artifact that is being
tested must include a single scenario in order for all workflows to pass
and thus satisfy branch merge requirements.
* Add support for different artifact types to enos-run
* Add support for different runner type to enos-run
* Add arm64 scenarios to build matrix
* Expand build matrices to include different variants
* Update Consul versions in Enos scenarios and matrices
* Refactor enos-run environment
* Add minimum version filtering support to enos-run. This allows us to
automatically exclude scenarios that require a more recent version of
Vault
* Add maximum version filtering support to enos-run. This allows us to
automatically exclude scenarios that require an older version of
Vault
* Fix Node 12 deprecation warnings
* Rename enos-verify-stable to enos-release-testing-oss
* Convert artifactory matrix into enos-release-testing-oss matrices
* Add all Vault editions to Enos scenario matrices
* Fix verify version with complex Vault edition metadata
* Rename the crt-builder to ci-helper
* Add more version helpers to ci-helper and Makefile
* Update CODEOWNERS for quality team
* Add support for filtering matrices by group and version constraints
* Add support for pseudo random test scenario execution
Signed-off-by: Ryan Cragun <me@ryan.ec>
* Added a workflow dispatch trigger for bootstrap workflow, updated ssh key name
* Ensure the bootstrap workflow is only run for PRs that change the bootstrapping code
Kuba Wieczorek