open-nomad/e2e
Tim Gross 693a8a2613
e2e: fix platform path for installing for Linux from s3 (#8708)
2020-08-21 09:20:09 -04:00
..
affinities e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
bin e2e/bin/run: run & update only attempt to contact linux servers (#8517) 2020-07-24 10:52:12 -04:00
cli e2e/cli/command: Wait() after execution 2019-11-25 10:56:40 -08:00
clientstate e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
connect e2e: connect jobID code golf 2020-07-10 10:24:13 -05:00
consul test: deflake consul e2e tests 2020-08-19 14:29:22 -07:00
consulacls e2e: minimize Consul ACL policies used in e2e tests 2020-03-30 12:53:40 -06:00
consultemplate e2e: add a Windows client to test runner (#6735) 2019-11-25 13:31:00 -05:00
csi e2e: spread CSI controller plugins across multiple DCs (#8629) 2020-08-10 16:41:39 -04:00
deployment e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
e2eutil test: deflake consul e2e tests 2020-08-19 14:29:22 -07:00
example e2e/cli: fix formatting 2018-07-31 13:52:25 -04:00
execagent e2e: add NomadAgent and basic client state test 2019-03-21 07:14:34 -07:00
fabio e2e: add a Windows client to test runner (#6735) 2019-11-25 13:31:00 -05:00
framework E2E: move Nomad installation to script on remote hosts (#8706) 2020-08-20 16:10:00 -04:00
hostvolumes e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
metrics build: use hashicorp hclfmt 2020-05-24 18:31:57 -05:00
migrations chore: Format hcl configurations 2019-07-20 16:55:07 +02:00
nomad09upgrade e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
nomadexec e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
podman skip podman e2e 2020-08-14 09:02:56 -04:00
prometheus build: use hashicorp hclfmt 2020-05-24 18:31:57 -05:00
rescheduling chore: Format hcl configurations 2019-07-20 16:55:07 +02:00
spread e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
systemsched simplify job, better error 2020-02-04 13:59:39 -05:00
taskevents e2e: setup consul ACLs a little more correctly 2020-01-31 19:06:11 -06:00
terraform e2e: fix platform path for installing for Linux from s3 (#8708) 2020-08-21 09:20:09 -04:00
upgrades script e2e/upgrades: cluster upgrade scripts 2019-09-24 14:35:45 -04:00
vault test: explicitly pass vars vs enclosing them 2020-02-14 11:10:33 -08:00
.gitignore e2e: have TF write-out HCL for CSI volume registration (#7599) 2020-04-02 12:16:43 -04:00
README.md docs: clarify e2e tests 2020-08-19 20:32:31 -07:00
e2e_test.go base podman e2e test and provisioning updates (#8104) 2020-06-03 14:06:58 -04:00

README.md

End to End Tests

This package contains integration tests. Unlike tests alongside Nomad code, these tests expect there to already be a functional Nomad cluster accessible (either on localhost or via the NOMAD_ADDR env var).

The terraform folder has provisioning code to spin up a Nomad cluster on AWS. The tests work with the NOMAD_ADDR environment variable which can be set either to a local dev Nomad agent or a Nomad client on AWS.

The NOMAD_E2E=1 environment variable must be set for these tests to run.

Local Nomad Development

When developing tests locally, provisioning is not required when only the tests change. See framework/doc.go for how to write tests.

When making changes to the Nomad agent itself, use ./bin/update $(which nomad) /usr/local/bin/nomad and ./bin/run sudo systemctl restart nomad to destructively modify the provisioned cluster.

Provisioning Test Infrastructure on AWS

You'll need Terraform and AWS credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) to setup AWS instances on which e2e tests will run. See the README for details. The number of servers and clients is configurable, as is the configuration file for each client and server.

Provisioning e2e Framework Nomad Cluster

You can use the Terraform output from the previous step to generate a provisioning configuration file for the e2e framework.

# from the ./e2e/terraform directory
terraform output provisioning | jq . > ../provisioning.json

By default the provisioning.json will not include the Nomad version that will be deployed to each node. You can pass the following flags to go test to set the version for all nodes:

  • -nomad.local_file=string: provision this specific local binary of Nomad. This is a path to a Nomad binary on your own host. Ex. -nomad.local_file=/home/me/nomad
  • -nomad.sha=string: provision this specific sha from S3. This is a Nomad binary identified by its full commit SHA that's stored in a shared s3 bucket that Nomad team developers can access. That commit SHA can be from any branch that's pushed to remote. Ex. -nomad.sha=0b6b475e7da77fed25727ea9f01f155a58481b6c
  • -nomad.version=string: provision this version from releases.hashicorp.com. Ex. -nomad.version=0.10.2

Then deploy Nomad to the cluster by passing -provision.terraform without a Nomad version flag:

NOMAD_E2E=1 go test -v .                   \
  -timeout 20m                             \
  -nomad.local_file=$(which nomad)         \
  -provision.terraform=./provisioning.json \
  -skipTests
  • -skipTests: provisioning can take time, so it's best to skip tests

  • -timeout 20m: depending on your cluster size and upload bandwidth the default 10m timeout may not be long enough for provisioning to finish

Running

After completing the provisioning step above, you can set the client environment for NOMAD_ADDR and run the tests as shown below:

# from the ./e2e/terraform directory, set your client environment
# if you haven't already
$(terraform output environment)

cd ..
go test -v .

If you want to run a specific suite, you can specify the -suite flag as shown below. Only the suite with a matching Framework.TestSuite.Component will be run, and all others will be skipped.

go test -v -suite=Consul .

If you want to run a specific test, you'll need to regex-escape some of the test's name so that the test runner doesn't skip over framework struct method names in the full name of the tests:

go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup'
                              ^       ^             ^               ^
                              |       |             |               |
                          Component   |             |           Test func    
                                      |             |
                                  Go Package      Struct

I Want To...

...SSH Into One Of The Test Machines

You can use the Terraform output to find the IP address. The keys will in the ./terraform/keys/ directory.

ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR}

Run terraform output for IP addresses and details.

...Deploy a Cluster of Mixed Nomad Versions

The provisioning.json file output by Terraform has a blank field for nomad_sha for each node of the cluster (server and client). You can manually edit the file to replace this value with a nomad_sha, nomad_local_binary, or nomad_version for each node to create a cluster of mixed versions. The provisioning framework accepts any of the following options for those fields:

  • nomad_sha: This is a Nomad binary identified by its full commit SHA that's stored in a shared s3 bucket that Nomad team developers can access. That commit SHA can be from any branch that's pushed to remote. (Ex. "nomad_sha": "0b6b475e7da77fed25727ea9f01f155a58481b6c")
  • nomad_local_binary: This is a path to a Nomad binary on your own host. (Ex. "nomad_local_binary": "/home/me/nomad")
  • nomad_version: This is a version number of Nomad that's been released to HashiCorp. (Ex. "nomad_version": "0.10.2")

Then deploy Nomad to the cluster by passing -provision.terraform without a Nomad version flag:

go test -v . -provision.terraform ./provisioning.json -skipTests

...Deploy Custom Configuration Files

The provisioning.json file includes a bundles section for each node of the cluster (server and client). You can manually edit this file to add, remove, or replace

"bundles": [
  {
    "destination": "/ops/shared/nomad/base.hcl",
    "source": "/home/me/custom.hcl"
  }
]

...Deploy More Than 4 Linux Clients

Right now the framework doesn't support this out-of-the-box because of the way the provisioning script adds specific client configurations to each client node (for constraint testing). You'll need to add additional configuration files to ./e2e/terraform/shared/nomad/indexed.