open-consul/test/integration/connect/envoy
Luke Kysow 12ac96c356
Remove terminal colouring from test output so it is (#14810)
more readable in CI.

```
Running primary verification step for case-ingress-gateway-multiple-services...
�[34;1mverify.bats
�[0m�[1G   ingress proxy admin is up on :20000�[K�[75G 1/12�[2G�[1G ✓ ingress proxy admin is up on :20000�[K
�[0m�[1G   s1 proxy admin is up on :19000�[K�[75G 2/12�[2G�[1G ✓ s1 proxy admin is up on :19000�[K
�[0m�[1G   s2 proxy admin is up on :19001�[K�[75G 3/12�[2G�[1G ✓ s2 proxy admin is up on :19001�[K
�[0m�[1G   s1 proxy listener should be up and have right cert�[K�[75G 4/12�[2G�[1G ✓ s1 proxy listener should be up and have right cert�[K
�[0m�[1G   s2 proxy listener should be up and have right cert�[K�[75G 5/12�[2G�[1G ✓ s2 proxy listener should be up and have right cert�[K
�[0m�[1G   ingress-gateway should have healthy endpoints for s1�[K�[75G 6/12�[2G�[31;1m�[1G ✗ ingress-gateway should have healthy endpoints for s1�[K
�[0m�[31;22m   (from function `assert_upstream_has_endpoints_in_status' in file /workdir/primary/bats/helpers.bash, line 385,
```

versus

```
Running primary verification step for case-ingress-gateway-multiple-services...
1..12
ok 1 ingress proxy admin is up on :20000
ok 2 s1 proxy admin is up on :19000
ok 3 s2 proxy admin is up on :19001
ok 4 s1 proxy listener should be up and have right cert
ok 5 s2 proxy listener should be up and have right cert
not ok 6 ingress-gateway should have healthy endpoints for s1
not ok 7 s1 proxy should have been configured with max_connections in services
ok 8 ingress-gateway should have healthy endpoints for s2
```
2022-10-04 08:35:19 -07:00
..
case-badauthz test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-basic syncing changes back from enterprise (#12701) 2022-04-05 15:46:56 -05:00
case-centralconf Add DC and NS support for Envoy metrics (#9207) 2020-11-16 16:37:19 -07:00
case-cfg-resolver-cluster-peering-failover Implement Cluster Peering Redirects (#14445) 2022-09-09 13:58:28 -04:00
case-cfg-resolver-dc-failover-gateways-none Refactor failover code to use Envoy's aggregate clusters (#14178) 2022-08-12 14:30:46 -04:00
case-cfg-resolver-dc-failover-gateways-remote Refactor failover code to use Envoy's aggregate clusters (#14178) 2022-08-12 14:30:46 -04:00
case-cfg-resolver-defaultsubset test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cfg-resolver-features test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cfg-resolver-subset-onlypassing test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cfg-resolver-subset-redirect test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cfg-resolver-svc-failover Refactor failover code to use Envoy's aggregate clusters (#14178) 2022-08-12 14:30:46 -04:00
case-cfg-resolver-svc-redirect-http test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cfg-resolver-svc-redirect-tcp test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cfg-router-features Integration tests for all new header manip features 2021-09-10 21:09:24 +01:00
case-cfg-splitter-features Integration tests for all new header manip features 2021-09-10 21:09:24 +01:00
case-consul-exec test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-cross-peers peering: default to false (#13963) 2022-08-01 15:22:36 -04:00
case-cross-peers-http peering: default to false (#13963) 2022-08-01 15:22:36 -04:00
case-cross-peers-http-router peering: default to false (#13963) 2022-08-01 15:22:36 -04:00
case-cross-peers-resolver-redirect-tcp peering: default to false (#13963) 2022-08-01 15:22:36 -04:00
case-dogstatsd-udp test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-expose-checks xds: only try to create an ipv6 expose checks listener if ipv6 is supported by the kernel (#9765) 2021-02-19 14:38:43 -06:00
case-gateway-without-services Support Incremental xDS mode (#9855) 2021-04-29 13:54:05 -05:00
case-gateways-local test: run Envoy integration tests against both servers and clients (#13610) 2022-06-28 13:15:45 +01:00
case-gateways-remote test: run Envoy integration tests against both servers and clients (#13610) 2022-06-28 13:15:45 +01:00
case-grpc connect: update supported envoy versions to 1.18.2, 1.17.2, 1.16.3, and 1.15.4 (#10101) 2021-04-29 15:22:03 -05:00
case-http syncing changes back from enterprise (#12701) 2022-04-05 15:46:56 -05:00
case-http-badauthz test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-ingress-gateway-grpc test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-ingress-gateway-http Add Envoy integration test to show Header manip can interpolate Envoy variables 2021-09-10 21:09:24 +01:00
case-ingress-gateway-multiple-services fix flaky integration test (#14843) 2022-10-03 16:55:05 -04:00
case-ingress-gateway-sds connect: Remove support for Envoy 1.16 (#11354) 2021-10-27 18:51:35 -07:00
case-ingress-gateway-simple fix flaky integration test (#14843) 2022-10-03 16:55:05 -04:00
case-ingress-gateway-tls test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-ingress-mesh-gateways-resolver test: run Envoy integration tests against both servers and clients (#13610) 2022-06-28 13:15:45 +01:00
case-l7-intentions test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-mesh-to-lambda Add Consul Lambda integration tests (#13770) 2022-07-21 09:54:56 -07:00
case-multidc-rsa-ca test: run Envoy integration tests against both servers and clients (#13610) 2022-06-28 13:15:45 +01:00
case-prometheus Add DC and NS support for Envoy metrics (#9207) 2020-11-16 16:37:19 -07:00
case-stats-proxy Add DC and NS support for Envoy metrics (#9207) 2020-11-16 16:37:19 -07:00
case-statsd-udp test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-terminating-gateway-hostnames Update assertion to not check for port 2021-04-06 17:10:38 +08:00
case-terminating-gateway-simple test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-terminating-gateway-subsets test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-terminating-gateway-without-services test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-upstream-config test: use direct service registration in envoy integration tests (#9138) 2020-11-09 13:59:46 -06:00
case-wanfed-gw Run integration tests locally using amd64 (#14365) 2022-08-29 16:13:49 -07:00
case-zipkin xds: default to speaking xDS v3, but allow for v2 to be spoken upon request (#9658) 2021-02-26 16:23:15 -06:00
consul-base-cfg Support Incremental xDS mode (#9855) 2021-04-29 13:54:05 -05:00
test-sds-server Minor improvements to SDS server from review 2021-09-23 10:13:41 +01:00
.gitignore activate most discovery chain features in xDS for envoy (#6024) 2019-07-01 22:10:51 -05:00
Dockerfile-bats chore(test): Update bats version 2022-05-24 11:56:08 -04:00
Dockerfile-consul-envoy Run integration tests locally using amd64 (#14365) 2022-08-29 16:13:49 -07:00
Dockerfile-tcpdump peering: mesh gateways are required for cross-peer service mesh communication (#13410) 2022-06-09 11:05:18 -05:00
Dockerfile-test-sds-server ci: upgrade bats and the circle machine executors to get integration tests to function again (#12918) 2022-05-03 11:21:32 -05:00
README.md Add more content to integration test docs (#14613) 2022-09-14 16:13:23 -07:00
defaults.sh peering: mesh gateways are required for cross-peer service mesh communication (#13410) 2022-06-09 11:05:18 -05:00
down.sh speed up envoy integration tests by removing docker-compose (#8982) 2020-10-22 13:20:31 -05:00
helpers.bash Remove terminal colouring from test output so it is (#14810) 2022-10-04 08:35:19 -07:00
main_test.go re-run gofmt on 1.17 (#11579) 2021-11-16 12:04:01 -06:00
run-tests.sh Remove terminal colouring from test output so it is (#14810) 2022-10-04 08:35:19 -07:00

README.md

Envoy Integration Tests

Overview

These tests validate that Consul is configuring Envoy correctly. They set up various scenarios using Docker containers and then run Bats (a Bash test framework) tests to validate the expected results.

Running Tests

To run the tests locally, cd into the root of the repo and run:

make test-envoy-integ

To run a specific test, run:

make test-envoy-integ GO_TEST_FLAGS="-run TestEnvoy/case-basic"

Where case-basic can be replaced by any directory name from this directory.

How Do These Tests Work

  1. The tests are all run through Go test via the main_test.go file. Each directory prefixed by case- is a subtest, for example, TestEnvoy/case-basic and TestEnvoy/case-wanfed-gw.
  2. The real framework for this test suite lives in run-tests.sh. Under the hood, main_test.go just runs run-tests.sh with various arguments.
  3. The tests use your local code by building a Docker image from your local directory just before executing. Note: this is implemented as the docker-envoy-integ Makefile target which is a prerequisite to the test-envoy-integ target, so if you are running the tests by invoking run-tests.sh or go test manually, be sure to rebuild the Docker image to ensure you are running your latest code.
  4. The tests run Docker containers connected by a shared Docker network. All tests have at least one Consul server running and then depending on the test case they will spin up additional services or gateways. Some tests run multiple Consul servers to test multi-DC setups. See the case-wanfed-gateway test for an example of this.
  5. At a high level, tests are set up by executing the setup.sh script in each directory. This script uses helper functions defined in helpers.bash. Once the test case is set up, the validations in verify.bats are run.
  6. If there exists a vars.sh file in the top-level of the case directory, the test runner will source it prior to invoking the run_tests, test_teardown and capture_logs phases of the test scenario.
  7. If there exists a capture.sh file in the top-level of the case directory, it will be executed after the test is done, but prior to the containers being removed. This is useful for capturing logs or Envoy snapshots for debugging test failures.
  8. Any files matching the *.hcl glob will be copied to the container $WORKDIR/$CLUSTER/consul directory prior to running the tests. This is useful for defining Consul configuration for each agent process to load on start up.
  9. In CI, the tests are executed against different Envoy versions and with both XDS_TARGET=client and XDS_TARGET=server. If set to client, a Consul server and client are run, and services are registered against the client. If set to server, only a Consul server is run, and services are registered against the server. By default, XDS_TARGET is set to server. See this comment for more information.

Investigating Test Failures

  • When tests fail in CI, logs and additional debugging data are available in the artifacts of the test run.
  • You can re-run the tests locally by running make test-envoy-integ GO_TEST_FLAGS="-run TestEnvoy/<case-directory>" where <case-directory> is replaced with the name of the directory, e.g. case-basic.
  • Locally, all the logs of the failed test will be available in workdir in this directory.
  • You can run with DEBUG=1 to print out all the commands being run, e.g. DEBUG=1 make test-envoy-integ GO_TEST_FLAGS="-run TestEnvoy/case-basic".
  • If you want to prevent the Docker containers from being spun down after test failure, add a sleep 9999 to the verify.bats test case that's failing.

Creating a New Test

Below is a rough outline for creating a new test. For the example, assume our test case will be called my-feature.

  1. Create a new directory named case-my-feature
  2. If the test involves multiple datacenters/clusters, create a separate subdirectory for each cluster (eg. case-my-feature/{dc1,dc2})
  3. Add any necessary configuration to *.hcl files in the respective cluster subdirectory (or the test case directory when using a single cluster).
  4. Create a setup.sh file in the case directory
  5. Create a capture.sh file in the case directory
  6. Create a verify.bats file in the case directory
  7. Populate the setup.sh, capture.sh and verify.bats files with the appropriate code for running your test, validating its state and capturing any logs or snapshots.