luxolus/open-vault
2
0
Fork 0
Code Issues 1 Pull requests Releases Activity
open-vault/.github/enos-run-matrices/enos_release_testing_oss-artifactory-oss-linux-amd64-zip.json

55 lines
2.3 KiB
JSON
Raw Normal View History

[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
{
"include": [
{
"scenario": "smoke backend:raft consul_version:1.14.2 distro:ubuntu seal:shamir arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
[QT-525] enos: use spot instances for Vault targets (#20037) The previous strategy for provisioning infrastructure targets was to use the cheapest instances that could reliably perform as Vault cluster nodes. With this change we introduce a new model for target node infrastructure. We've replaced on-demand instances for a spot fleet. While the spot price fluctuates based on dynamic pricing, capacity, region, instance type, and platform, cost savings for our most common combinations range between 20-70%. This change only includes spot fleet targets for Vault clusters. We'll be updating our Consul backend bidding in another PR. * Create a new `vault_cluster` module that handles installation, configuration, initializing, and unsealing Vault clusters. * Create a `target_ec2_instances` module that can provision a group of instances on-demand. * Create a `target_ec2_spot_fleet` module that can bid on a fleet of spot instances. * Extend every Enos scenario to utilize the spot fleet target acquisition strategy and the `vault_cluster` module. * Update our Enos CI modules to handle both the `aws-nuke` permissions and also the privileges to provision spot fleets. * Only use us-east-1 and us-west-2 in our scenario matrices as costs are lower than us-west-1. Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-04-13 19:44:43 +00:00
"aws_region": "us-east-1",
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
"test_group": 2
},
{
"scenario": "smoke backend:raft consul_version:1.13.4 distro:rhel seal:awskms arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
"aws_region": "us-west-2",
"test_group": 1
},
{
"scenario": "smoke backend:consul consul_version:1.14.2 distro:ubuntu seal:shamir arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
[QT-525] enos: use spot instances for Vault targets (#20037) The previous strategy for provisioning infrastructure targets was to use the cheapest instances that could reliably perform as Vault cluster nodes. With this change we introduce a new model for target node infrastructure. We've replaced on-demand instances for a spot fleet. While the spot price fluctuates based on dynamic pricing, capacity, region, instance type, and platform, cost savings for our most common combinations range between 20-70%. This change only includes spot fleet targets for Vault clusters. We'll be updating our Consul backend bidding in another PR. * Create a new `vault_cluster` module that handles installation, configuration, initializing, and unsealing Vault clusters. * Create a `target_ec2_instances` module that can provision a group of instances on-demand. * Create a `target_ec2_spot_fleet` module that can bid on a fleet of spot instances. * Extend every Enos scenario to utilize the spot fleet target acquisition strategy and the `vault_cluster` module. * Update our Enos CI modules to handle both the `aws-nuke` permissions and also the privileges to provision spot fleets. * Only use us-east-1 and us-west-2 in our scenario matrices as costs are lower than us-west-1. Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-04-13 19:44:43 +00:00
"aws_region": "us-east-1",
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
"test_group": 2
},
{
"scenario": "smoke backend:consul consul_version:1.13.4 distro:rhel seal:awskms arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
"aws_region": "us-west-2",
"test_group": 1
},
{
"scenario": "smoke backend:consul consul_version:1.12.7 distro:ubuntu seal:shamir arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
[QT-525] enos: use spot instances for Vault targets (#20037) The previous strategy for provisioning infrastructure targets was to use the cheapest instances that could reliably perform as Vault cluster nodes. With this change we introduce a new model for target node infrastructure. We've replaced on-demand instances for a spot fleet. While the spot price fluctuates based on dynamic pricing, capacity, region, instance type, and platform, cost savings for our most common combinations range between 20-70%. This change only includes spot fleet targets for Vault clusters. We'll be updating our Consul backend bidding in another PR. * Create a new `vault_cluster` module that handles installation, configuration, initializing, and unsealing Vault clusters. * Create a `target_ec2_instances` module that can provision a group of instances on-demand. * Create a `target_ec2_spot_fleet` module that can bid on a fleet of spot instances. * Extend every Enos scenario to utilize the spot fleet target acquisition strategy and the `vault_cluster` module. * Update our Enos CI modules to handle both the `aws-nuke` permissions and also the privileges to provision spot fleets. * Only use us-east-1 and us-west-2 in our scenario matrices as costs are lower than us-west-1. Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-04-13 19:44:43 +00:00
"aws_region": "us-east-1",
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
"test_group": 2
},
{
"scenario": "upgrade backend:raft consul_version:1.14.2 distro:rhel seal:awskms arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
"aws_region": "us-west-2",
"test_group": 1
},
{
"scenario": "upgrade backend:raft consul_version:1.14.2 distro:ubuntu seal:shamir arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
[QT-525] enos: use spot instances for Vault targets (#20037) The previous strategy for provisioning infrastructure targets was to use the cheapest instances that could reliably perform as Vault cluster nodes. With this change we introduce a new model for target node infrastructure. We've replaced on-demand instances for a spot fleet. While the spot price fluctuates based on dynamic pricing, capacity, region, instance type, and platform, cost savings for our most common combinations range between 20-70%. This change only includes spot fleet targets for Vault clusters. We'll be updating our Consul backend bidding in another PR. * Create a new `vault_cluster` module that handles installation, configuration, initializing, and unsealing Vault clusters. * Create a `target_ec2_instances` module that can provision a group of instances on-demand. * Create a `target_ec2_spot_fleet` module that can bid on a fleet of spot instances. * Extend every Enos scenario to utilize the spot fleet target acquisition strategy and the `vault_cluster` module. * Update our Enos CI modules to handle both the `aws-nuke` permissions and also the privileges to provision spot fleets. * Only use us-east-1 and us-west-2 in our scenario matrices as costs are lower than us-west-1. Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-04-13 19:44:43 +00:00
"aws_region": "us-east-1",
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
"test_group": 2
},
{
"scenario": "upgrade backend:consul consul_version:1.14.2 distro:rhel seal:awskms arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
"aws_region": "us-west-2",
"test_group": 1
},
{
"scenario": "upgrade backend:consul consul_version:1.13.4 distro:ubuntu seal:shamir arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
[QT-525] enos: use spot instances for Vault targets (#20037) The previous strategy for provisioning infrastructure targets was to use the cheapest instances that could reliably perform as Vault cluster nodes. With this change we introduce a new model for target node infrastructure. We've replaced on-demand instances for a spot fleet. While the spot price fluctuates based on dynamic pricing, capacity, region, instance type, and platform, cost savings for our most common combinations range between 20-70%. This change only includes spot fleet targets for Vault clusters. We'll be updating our Consul backend bidding in another PR. * Create a new `vault_cluster` module that handles installation, configuration, initializing, and unsealing Vault clusters. * Create a `target_ec2_instances` module that can provision a group of instances on-demand. * Create a `target_ec2_spot_fleet` module that can bid on a fleet of spot instances. * Extend every Enos scenario to utilize the spot fleet target acquisition strategy and the `vault_cluster` module. * Update our Enos CI modules to handle both the `aws-nuke` permissions and also the privileges to provision spot fleets. * Only use us-east-1 and us-west-2 in our scenario matrices as costs are lower than us-west-1. Signed-off-by: Ryan Cragun <me@ryan.ec>
2023-04-13 19:44:43 +00:00
"aws_region": "us-east-1",
[QT-436] Pseudo random artifact test scenarios (#18056) Introducing a new approach to testing Vault artifacts before merge and after merge/notorization/signing. Rather than run a few static scenarios across the artifacts, we now have the ability to run a pseudo random sample of scenarios across many different build artifacts. We've added 20 possible scenarios for the AMD64 and ARM64 binary bundles, which we've broken into five test groups. On any given push to a pull request branch, we will now choose a random test group and execute its corresponding scenarios against the resulting build artifacts. This gives us greater test coverage but lets us split the verification across many different pull requests. The post-merge release testing pipeline behaves in a similar fashion, however, the artifacts that we use for testing have been notarized and signed prior to testing. We've also reduce the number of groups so that we run more scenarios after merge to a release branch. We intend to take what we've learned building this in Github Actions and roll it into an easier to use feature that is native to Enos. Until then, we'll have to manually add scenarios to each matrix file and manually number the test group. It's important to note that Github requires every matrix to include at least one vector, so every artifact that is being tested must include a single scenario in order for all workflows to pass and thus satisfy branch merge requirements. * Add support for different artifact types to enos-run * Add support for different runner type to enos-run * Add arm64 scenarios to build matrix * Expand build matrices to include different variants * Update Consul versions in Enos scenarios and matrices * Refactor enos-run environment * Add minimum version filtering support to enos-run. This allows us to automatically exclude scenarios that require a more recent version of Vault * Add maximum version filtering support to enos-run. This allows us to automatically exclude scenarios that require an older version of Vault * Fix Node 12 deprecation warnings * Rename enos-verify-stable to enos-release-testing-oss * Convert artifactory matrix into enos-release-testing-oss matrices * Add all Vault editions to Enos scenario matrices * Fix verify version with complex Vault edition metadata * Rename the crt-builder to ci-helper * Add more version helpers to ci-helper and Makefile * Update CODEOWNERS for quality team * Add support for filtering matrices by group and version constraints * Add support for pseudo random test scenario execution Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 20:46:04 +00:00
"test_group": 2
},
{
"scenario": "upgrade backend:consul consul_version:1.12.7 distro:rhel seal:awskms arch:amd64 artifact_source:artifactory edition:oss artifact_type:bundle",
"aws_region": "us-west-2",
"test_group": 1
}
]
}
Reference in a new issue Copy permalink