open-vault/enos

Enos

Enos is an quality testing framework that allows composing and executing quality requirement scenarios as code. For Vault, it is currently used to perform infrastructure integration testing using the artifacts that are created as part of the build workflow. While intended to be executed via Github Actions using the results of the build workflow, scenarios are also executable from a developer machine that has the requisite dependencies and configuration.

Refer to the Enos documentation for further information regarding installation, execution or composing Enos scenarios.

When to use Enos

Determining whether to use vault.NewTestCluster() or Enos for testing a feature or scenario is ultimately up to the author. Sometimes one, the other, or both might be appropriate depending on the requirements. Generally, vault.NewTestCluster() is going to give you faster feedback and execution time, whereas Enos is going to give you a real-world execution and validation of the requirement. Consider the following cases as examples of when one might opt for an Enos scenario:

• The feature require third-party integrations. Whether that be networked dependencies like a real Consul backend, a real KMS key to test awskms auto-unseal, auto-join discovery using AWS tags, or Cloud hardware KMS's.
• The feature might behave differently under multiple configuration variants and therefore should be tested with both combinations, e.g. auto-unseal and manual shamir unseal or replication in HA mode with integrated storage or Consul storage.
• The scenario requires coordination between multiple targets. For example, consider the complex lifecycle event of migrating the seal type or storage, or manually triggering a raft disaster scenario by partitioning the network between the leader and follower nodes. Or perhaps an auto-pilot upgrade between a stable version of Vault and our candidate version.
• The scenario has specific deployment strategy requirements. For example, if we want to add a regression test for an issue that only arises when the software is deployed in a certain manner.
• The scenario needs to use actual build artifacts that will be promoted through the pipeline.

Requirements

• AWS access. HashiCorp Vault developers should use Doormat.
• Terraform >= 1.2
• Enos >= v0.0.10. You can install it from a release channel.
• Access to the QTI org in Terraform Cloud. HashiCorp Vault developers can access a shared token in 1Password or request their own in #team-quality on Slack.
• An SSH keypair in the AWS region you wish to run the scenario. You can use Doormat to log in to the AWS console to create or upload an existing keypair.
• A Vault install bundle downloaded from releases.hashicorp.com or Artifactory when using the builder:crt variants. When using the builder:local variants Enos will build a Vault bundle from the current branch for you.

Scenario Variables

In CI, each scenario is executed via Github Actions and has been configured using environment variable inputs that follow the ENOS_VAR_varname pattern.

For local execution you can specify all the required variables using environment variables, or you can update enos.vars.hcl with values and uncomment the lines.

Variables that are required:

• aws_ssh_keypair_name
• aws_ssh_private_key_path
• tfc_api_token
• vault_bundle_path
• vault_license_path (only required for non-OSS editions)

See enos.vars.hcl or enos-variables.hcl for further descriptions of the variables.

Executing Scenarios

From the enos directory:

# List all available scenarios
enos scenario list
# Run the smoke or upgrade scenario with an artifact that is built locally. Make sure
# the local machine has been configured as detailed in the requirements
# section. This will execute the scenario and clean up any resources if successful.
enos scenario run smoke builder:local
enos scenario run upgrade builder:local
# To run the same scenario variants that are run in CI, refer to the scenarios listed
# in .github/workflows/enos-run.yml under `jobs.enos.strategy.matrix.include`,
# adding `builder:local` to run locally.
enos scenario run smoke backend:consul consul_version:1.12.3 distro:ubuntu seal:awskms builder:local arch:amd64 edition:oss
# Launch an individual scenario but leave infrastructure up after execution
enos scenario launch smoke builder:local
# Check an individual scenario for validity. This is useful during scenario
# authoring and debugging.
enos scenario validate smoke builder:local
# If you've run the tests and desire to see the outputs, such as the URL or
# credentials, you can run the output command to see them. Please note that
# after "run" or destroy there will be no "outputs" as the infrastructure
# will have been destroyed and state cleared.
enos scenario output smoke builder:local
# Explicitly destroy all existing infrastructure
enos scenario destroy smoke builder:local

Refer to the Enos documentation for further information regarding installation, execution or composing scenarios.

Scenarios

There are current two scenarios: smoke and upgrade. Both begin by building Vault as specified by the selected builder variant (see Variants section below for more information).

Smoke

The smoke scenario creates a Vault cluster using the version from the current branch (either in CI or locally), with the backend specified by the backend variant (raft or consul). Next, it unseals with the appropriate method (awskms or shamir) and performs different verifications depending on the backend and seal type.

Upgrade

The upgrade scenario creates a Vault cluster using the version specified in vault_upgrade_initial_release, with the backend specified by the backend variant (raft or consul). Next, it upgrades the Vault binary that is determined by the builder variant. After the upgrade, it verifies that cluster is at the desired version, along with additional verifications.

Autopilot

The autopilot scenario creates a Vault cluster using the version specified in vault_upgrade_initial_release. Next, it creates additional nodes with the candiate version of Vault as determined by the builder variant. The module uses AWS auto-join to handle discovery and unseals with auto-unseal or Shamir depending on the seal variant. After the new nodes have joined and been unsealed, it waits for Autopilot to upgrade the new nodes and demote the old nodes.

Variants

Both scenarios support a matrix of variants. In order to achieve broad coverage while keeping test run time reasonable, the variants executed by the enos-run Github Actions are tailored to maximize variant distribution per scenario.

builder:crt

This variant is designed for use in Github Actions. The enos-run.yml workflow downloads the artifact built by the build.yml workflow, unzips it, and sets the vault_bundle_path to the zip file and the vault_local_binary_path to the binary.

builder:local

This variant is for running the Enos scenario locally. It builds the Vault bundle from the current branch, placing the bundle at the vault_bundle_path and the unzipped Vault binary at the vault_local_binary_path.