f4a66ebd28
Pretty sure Consul / Nomad clients are often not ready yet after
the ConsulACLs test disables ACLs, by the time the next test starts
running.
Running locally things tend to work, but in TeamCity this seems to
be a recurring problem. However, when running locally sometimes I do
see that the "show status" step after disabling ACLs, some nodes are
still initializing, suggesting we're right on the border of not waiting
long enough
nomad node status
ID DC Name Class Drain Eligibility Status
0e4dfce2 dc1 EC2AMAZ-JB3NF9P <none> false eligible ready
6b90aa06 dc2 ip-172-31-16-225 <none> false eligible ready
7068558a dc2 ip-172-31-20-143 <none> false eligible ready
e0ae3c5c dc1 ip-172-31-25-165 <none> false eligible ready
15b59ed6 dc1 ip-172-31-23-199 <none> false eligible initializing
Going to try waiting a full 2 minutes after disabling ACLs, hopefully that
will help things Just Work. In the future, we should probably be parsing the
output of the status checks and actually confirming all nodes are ready.
Even better, maybe that's something shipyard will have built-in.
|
||
|---|---|---|
| .. | ||
| acl-disable.hcl | ||
| acl-enable.hcl | ||
| acl-pre-enable.hcl | ||
| consul-acls-manage.sh | ||
| consul-client-default-token.hcl | ||
| consul-client-policy.hcl | ||
| consul-server-policy.hcl | ||
| manage.go | ||
| nomad-client-consul.hcl | ||
| nomad-client-policy.hcl | ||
| nomad-server-consul.hcl | ||
| nomad-server-policy.hcl | ||
| README.md | ||
Configure Consul ACLs
This directory contains a set of scripts for re-configuring Consul in the TF provisioned e2e environment to enable Consul ACLs.
Usage
The consul-acls-manage.sh script can be used to manipulate the Consul cluster
to activate or de-activate Consul ACLs. There are 3 targets into the script, only
2 of which should be used from e2e framework tests. The script should be run from
the e2e directory (i.e. the directory from wich the e2e framework also runs).
bootstrap
The command consul-acls-manage.sh bootstrap should NOT be used from e2e
framework tests. It's merely a convenience entry-point for doing development /
debugging on the script itself.
The bootstrap process will upload "reasonable" ACL policy files to Consul Servers, Consul Clients, Nomad Servers, and Nomad Clients.
The bootstrap process creates a file on local disk which contains the generated
Consul ACL master token. The file is named based on the current TF state file
serial number. /tmp/e2e-consul-bootstrap-<serial>.token
enable
The command consul-acls-manage.sh enable will enable Consul ACLs, going through
the bootstrap process only if necessary. Whether the bootstrap process is necessary
depends on the existence of a token file that matches the current TF state serial
number. If no associated token file exists for the current TF state, the bootstrap
process is required. Otherwise, the bootstrap process is skipped.
If the bootstrap process was not required (i.e. it already occurred and a
Consul master token already exists for the current TF state), the script will
activate ACLs in the Consul Server configurations and restart those agents. After
using enable, the disable command can be used to turn Consul ACLs back off,
without destroying any of the existing ACL configuration.
disable
The command consul-acls-manage.sh disable will disable Consul ACLs. This does
not "cleanup" the policy files for Consul / Nomad agents, it merely deactivates
ACLs in the Consul Server configurations and restarts those agents. After using
disable, the enable command can be used to turn Consul ACLs back on, using
the same ACL token(s) generated before.