e2e: document e2e provisioning process (#6976)
This commit is contained in:
Tim Gross
GitHub
parent
936b96af5e
commit
457e3ad5c6
|
|
@ -0,0 +1 @@
|
|||
provisioning.json
|
||||
|
|
@ -1,23 +1,90 @@
|
|||
End to End Tests
|
||||
================
|
||||
# End to End Tests
|
||||
|
||||
This package contains integration tests.
|
||||
This package contains integration tests.
|
||||
|
||||
The `terraform` folder has provisioning code to spin up a Nomad cluster on AWS. The tests work with the `NOMAD_ADDR` environment variable which can be set either to a local dev Nomad agent or a Nomad client on AWS.
|
||||
The `terraform` folder has provisioning code to spin up a Nomad cluster on AWS.
|
||||
The tests work with the `NOMAD_ADDR` environment variable which can be set
|
||||
either to a local dev Nomad agent or a Nomad client on AWS.
|
||||
|
||||
Local Development
|
||||
=================
|
||||
The workflow when developing end to end tests locally is to run the provisioning step described below once, and then run the tests as described below.
|
||||
When making local changes, use `./bin/update $(which nomad) /usr/local/bin/nomad` and `./bin/run sudo systemctl restart nomad` to destructively modify the provisioned cluster.
|
||||
## Local Development
|
||||
|
||||
Provisioning
|
||||
============
|
||||
You'll need AWS credentials (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`) to create the Nomad cluster. See the [README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md) for details. The number of servers and clients is configurable, as is the configuration file for each client and server.
|
||||
The workflow when developing end to end tests locally is to run the
|
||||
provisioning step described below once, and then run the tests as described
|
||||
below.
|
||||
|
||||
Running
|
||||
===========
|
||||
After completing the provisioning step above, you should see CLI output showing the IP addresses of Nomad client machines. To run the tests, set the NOMAD_ADDR variable to `http://[client IP]:4646/`
|
||||
When making local changes, use `./bin/update $(which nomad) /usr/local/bin/nomad`
|
||||
and `./bin/run sudo systemctl restart nomad` to destructively modify the
|
||||
provisioned cluster.
|
||||
|
||||
## Provisioning Test Infrastructure on AWS
|
||||
|
||||
You'll need Terraform and AWS credentials (`AWS_ACCESS_KEY_ID` and
|
||||
`AWS_SECRET_ACCESS_KEY`) to setup AWS instances on which e2e tests
|
||||
will run. See the [README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md)
|
||||
for details. The number of servers and clients is configurable, as is
|
||||
the configuration file for each client and server.
|
||||
|
||||
## Provisioning e2e Framework Nomad Cluster
|
||||
|
||||
You can use the Terraform output from the previous step to generate a
|
||||
provisioning configuration file for the e2e framework.
|
||||
|
||||
```sh
|
||||
# from the ./e2e/terraform directory
|
||||
terraform output provisioning | jq . > ../provisioning.json
|
||||
```
|
||||
$ NOMAD_ADDR=<> NOMAD_E2E=1 go test -v
|
||||
|
||||
By default the `provisioning.json` will include a `nomad_sha` field
|
||||
for each node. You can edit this file to change the version of Nomad
|
||||
you want to deploy. Because each node has its own value, you can
|
||||
create cluster of mixed versions. The provisioning framework accepts
|
||||
any of the following options:
|
||||
|
||||
- `nomad_sha`: This is a Nomad binary identified by its full commit SHA that's
|
||||
stored in a shared s3 bucket that Nomad team developers can access. That
|
||||
commit SHA can be from any branch that's pushed to remote. (Ex.
|
||||
`"nomad_sha": "0b6b475e7da77fed25727ea9f01f155a58481b6c"`)
|
||||
- `nomad_local_binary`: This is a path to a Nomad binary on your own host.
|
||||
(Ex. `"nomad_local_binary": "/home/me/nomad"`)
|
||||
- `nomad_version`: This is a version number of Nomad that's been released to
|
||||
HashiCorp. (Ex. `"nomad_version": "0.10.2"`)
|
||||
|
||||
You can pass the following flags to `go test` to override the values
|
||||
in `provisioning.json` for all nodes:
|
||||
|
||||
- `-nomad.local_file=string`: provision this specific local binary of Nomad
|
||||
- `-nomad.sha=string`: provision this specific sha from S3
|
||||
- `-nomad.version=string`: provision this version from
|
||||
[releases.hashicorp.com](https://releases.hashicorp.com/nomad)
|
||||
|
||||
Deploy Nomad to the cluster:
|
||||
|
||||
```sh
|
||||
# from the ./e2e/terraform directory, set your client environment
|
||||
$(terraform output environment)
|
||||
|
||||
cd ..
|
||||
go test -v . -provision.terraform ./provisioning.json -skipTests
|
||||
```
|
||||
|
||||
## Running
|
||||
|
||||
After completing the provisioning step above, you can set the client
|
||||
environment for `NOMAD_ADDR` and run the tests as shown below:
|
||||
|
||||
```sh
|
||||
# from the ./e2e/terraform directory, set your client environment
|
||||
# if you haven't already
|
||||
$(terraform output environment)
|
||||
|
||||
cd ..
|
||||
go test -v .
|
||||
```
|
||||
|
||||
If you want to run a specific test, you'll need to regex-escape some of the
|
||||
test's name so that the test runner doesn't skip over framework struct method
|
||||
names in the full name of the tests:
|
||||
|
||||
```sh
|
||||
go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup'
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,39 +1,84 @@
|
|||
Terraform provisioner for end to end tests
|
||||
==========================================
|
||||
# Terraform infrastructure
|
||||
|
||||
This folder contains terraform resources for provisioning a nomad cluster on AWS for end to end tests.
|
||||
It uses a Nomad binary identified by its commit SHA that's stored in a shared s3 bucket that Nomad team
|
||||
developers can access. The commit SHA can be from any branch that's pushed to remote.
|
||||
This folder contains terraform resources for provisioning EC2 instances on AWS
|
||||
to use as the target of end-to-end tests.
|
||||
|
||||
Use [envchain](https://github.com/sorah/envchain) to store your AWS credentials.
|
||||
Terraform provisions the AWS infrastructure only, whereas the Nomad
|
||||
cluster is deployed to that infrastructure by the e2e
|
||||
framework. Terraform's output will include a `provisioning` stanza
|
||||
that can be written to a JSON file used by the e2e framework's
|
||||
provisioning step.
|
||||
|
||||
You can use Terraform to output the provisioning parameter JSON file the e2e
|
||||
framework uses by setting the `nomad_sha` variable.
|
||||
|
||||
```
|
||||
$ cd e2e/terraform/
|
||||
$ TF_VAR_nomad_sha=<nomad_sha> envchain nomadaws terraform apply
|
||||
## Setup
|
||||
|
||||
You'll need Terraform 0.12+, as well as AWS credentials (`AWS_ACCESS_KEY_ID`
|
||||
and `AWS_SECRET_ACCESS_KEY`) to create the Nomad cluster. Use
|
||||
[envchain](https://github.com/sorah/envchain) to store your AWS credentials.
|
||||
|
||||
Optionally, edit the `terraform.tfvars` file to change the number of
|
||||
Linux clients, Windows clients, or the Nomad build. You'll usually
|
||||
want to have the `nomad_sha` variable set here (or via the
|
||||
`TF_VAR_nomad_sha` env var) so that `terraform output provisioning` is
|
||||
populated with the build you want.
|
||||
|
||||
```hcl
|
||||
region = "us-east-1"
|
||||
instance_type = "t2.medium"
|
||||
server_count = "3"
|
||||
client_count = "4"
|
||||
windows_client_count = "1"
|
||||
|
||||
# alternatively, set this via env var: TF_VAR_nomad_sha
|
||||
# nomad_sha = ""
|
||||
```
|
||||
|
||||
After this step, you should have a nomad client address to point the end to end tests in the `e2e` folder to.
|
||||
Run Terraform apply to deploy the infrastructure:
|
||||
|
||||
```sh
|
||||
cd e2e/terraform/
|
||||
TF_VAR_nomad_sha=<nomad_sha> envchain nomadaws terraform apply
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
After deploying the infrastructure, you can get connection information
|
||||
about the cluster:
|
||||
|
||||
- `$(terraform output environment)` will set your current shell's
|
||||
`NOMAD_ADDR` and `CONSUL_HTTP_ADDR` to point to one of the cluster's
|
||||
server nodes, and set the `NOMAD_E2E` variable.
|
||||
- `terraform output servers` will output the list of server node IPs.
|
||||
- `terraform output linux_clients` will output the list of Linux
|
||||
client node IPs.
|
||||
- `terraform output windows_clients` will output the list of Windows
|
||||
client node IPs.
|
||||
- `terraform output provisioning | jq .` will output the JSON used by
|
||||
the e2e framework for provisioning.
|
||||
|
||||
## SSH
|
||||
|
||||
Terraform will output node IPs that may be accessed via ssh:
|
||||
You can use Terraform outputs above to access nodes via ssh:
|
||||
|
||||
```
|
||||
```sh
|
||||
ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR}
|
||||
```
|
||||
|
||||
The Windows client runs OpenSSH for conveniences, but has a different user and will drop you into a Powershell shell instead of bash:
|
||||
The Windows client runs OpenSSH for convenience, but has a different
|
||||
user and will drop you into a Powershell shell instead of bash:
|
||||
|
||||
```
|
||||
```sh
|
||||
ssh -i keys/nomad-e2e-*.pem Administrator@${EC2_IP_ADDR}
|
||||
```
|
||||
|
||||
|
||||
## Teardown
|
||||
The terraform state file stores all the info, so the nomad_sha doesn't need to be valid during teardown.
|
||||
|
||||
```
|
||||
$ cd e2e/terraform/
|
||||
$ envchain nomadaws TF_VAR_nomad_sha=yyyzzz terraform destroy
|
||||
The terraform state file stores all the info, so the `nomad_sha`
|
||||
doesn't need to be valid during teardown.
|
||||
|
||||
```sh
|
||||
cd e2e/terraform/
|
||||
envchain nomadaws TF_VAR_nomad_sha=yyyzzz terraform destroy
|
||||
```
|
||||
|
|
|
|||
|
|
@ -3,3 +3,6 @@ instance_type = "t2.medium"
|
|||
server_count = "3"
|
||||
client_count = "4"
|
||||
windows_client_count = "1"
|
||||
|
||||
# alternatively, set this via env var: TF_VAR_nomad_sha
|
||||
# nomad_sha = ""
|
||||
|
|
|
|||
Loading…
Reference in New Issue