--- layout: docs page_title: Manual Secure Configuration - Consul on AWS Elastic Container Service (ECS) description: >- Securely configure Consul Service Mesh on Amazon Web Services ECS to protect the communication of sensitive data in production workloads. Learn how to manually configure auth methods to create tokens for clients and services and then apply tokens by role or policy. --- # Manual Secure Configuration of Consul on AWS Elastic Container Service (ECS) This topic describes how to enable Consul security features for your production workloads. ## Prerequisites The following features must be configured for your Consul server cluster: - [TLS encryption](/docs/security/encryption#rpc-encryption-with-tls) for RPC communication between Consul clients and servers. - [Gossip encryption](/docs/security/encryption#gossip-encryption) for encrypting gossip traffic. - [Access control lists (ACLs)](/docs/security/acl) for authentication and authorization for Consul clients and services on the mesh. You should already have followed the [manual installation instructions](/docs/ecs/manual/install) to define the necessary components of the task definition for Consul on ECS. You should be familiar with [specifying sensitive data](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html) on ECS. You should be familiar with configuring Consul's secure features, including how to create ACL tokens and policies. Refer to the [ACL system documentation](/docs/security/acl) and [Day 1: Security tutorials](https://learn.hashicorp.com/collections/consul/security) for an introduction and additional information. ## Auth Method Tokens are artifacts within the ACL system that authenticate users, services, and Consul agents. Tokens are linked to policies that specify the resources the token bearer has access to when making requests in the network. Auth Methods are a Consul server component that performs authentication against a trusted external party to authorize the creation of ACL tokens. The [AWS IAM auth method](/docs/security/acl/auth-methods/aws-iam) is used to enable an ECS task to automatically obtain ACL tokens when the task starts up. There are two types of ACL tokens for Consul on ECS: * **Client tokens:** used by the `consul-client` containers to join the Consul cluster * **Service tokens:** used by sidecar containers for service registration and health syncing This section describes how to manually configure the AWS IAM auth method for Consul on ECS. Alternatively, you can install the ACL controller to ease the burden of creating these resources. The ACL controller can automatically configure ACL resources for Consul on ECS. For additional details, refer to [ACL Controller](/docs/ecs/manual/acl-controller) and [Architecture](/docs/ecs/architecture). ### ECS Task Role Configuration The ECS [task role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) is an IAM role associated with an ECS task. When an ECS task starts up, it runs a `consul login` command. The `consul login` command obtains credentials for the task role role from AWS. It uses those credentials to sign the login request to the AWS IAM auth method. This proves the ECS task's identity to the Consul servers. The task role must be configured with the following details to compatible with the AWS IAM auth method. * An `iam:GetRole` permission to fetch itself. See [IAM Policies](/docs/security/acl/auth-methods/aws-iam#iam-policies) for details. * A `consul.hashicorp.com.service-name` tag on the task role which contains the Consul service name for the application in this task. * A consul.hashicorp.com.namespace tag on the task role indicating the Consul Enterprise namespace where this service is registering. The following sections describe how to configure the auth method to enable task roles to successfully authenticate to the AWS IAM auth method and obtain tokens with the necessary permissions. ### Auth Method for Client Tokens The following steps configure an instance of the auth method that creates client tokens for tasks. 1. Create the auth method instance 2. Create the client policy and role 3. Create the binding rule #### Create Auth Method for Client Tokens The following Consul CLI command creates an instance of the auth method for client tokens. ```shell consul acl auth-method create \ -type aws-iam \ -name iam-ecs-client-token \ -description="AWS IAM auth method for ECS client tokens" \ -config '{ "BoundIAMPrincipalArns": ["arn:aws:iam:::role/consul-ecs/*"], "EnableIAMEntityDetails": true }' ``` The following flags are required: | Flag | Type | Description | | --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- | | `-type` | string | Must be `aws-iam`. | | `-name` | string | A name of your choice. Must be unique among all auth methods. | | `-description` | string | A description of your choice. | | `-config` | string | A JSON string containing the [configuration](/docs/security/acl/auth-methods/aws-iam#config-parameters) for the the auth method. | In the `-config` option, the following fields are required: | Field | Type | Description | | ------------------------- | ------- | --------------------------------------------------------------------------------------------------------- | | `BoundIAMPrincipalArns` | list | The list of trusted IAM roles. We recommend using a wildcard to trust IAM roles at a particular path. | | `EnableIAMEntityDetails` | boolean | Must be true so that the auth method can retrieve IAM role details, such as the role path and role tags. | #### Create Client Policy and Role Configure the following ACL policy for Consul client tokens: ```hcl node_prefix "" { policy = "write" } service_prefix "" { policy = "read" } ``` The policy allows `node:write` for any node name, which is necessary because the Consul node names on ECS are not known until runtime. You can add the policy in Consul using the [`consul acl policy create`](/commands/acl/policy/create) command or the [`[PUT] /v1/acl/policy`](/api-docs/acl/policies#create-a-policy) API endpoint. After the policy is created, create a Consul role associated with the policy by using the [`consul acl role create`](/commands/acl/role/create) command or the [`[PUT] /v1/acl/role`](/api-docs/acl/roles#create-a-role) API endpoint. The following example shows how to use the Consul CLI to create the client policy and role. ```shell consul acl policy create -name consul-ecs-client-policy -rules @client-token-policy.hcl consul acl role create -name consul-ecs-client-role -policy-name consul-ecs-client-policy ``` #### Create Binding Rule for Client Tokens The following creates a binding rule for the auth method for client tokens. The binding rule associates the client role with each token created by a successful login to this auth method instance. ```shell consul acl binding-rule create -method iam-ecs-client-token \ -description 'Bind a role for Consul clients on ECS' \ -bind-type role \ -bind-name consul-ecs-client-role ``` ### Auth Method for Service Tokens The following steps configure an instance of the auth method that creates service tokens for tasks. * Create the auth method instance * Create the binding rule #### Create Auth Method for Service Tokens The following uses the Consul CLI to create an instance of the auth method for service tokens. This configures the auth method to associate a service identity to each token created during login to this auth method instance. ```shell consul acl auth-method create \ -type aws-iam \ -name iam-ecs-service-token \ -description="AWS IAM auth method for ECS service tokens" \ -config '{ "BoundIAMPrincipalArns": ["arn:aws:iam:::role/consul-ecs/*"], "EnableIAMEntityDetails": true, "IAMEntityTags": [ "consul.hashicorp.com.service-name" ] }' ``` The following flags are required: | Flag | Type | Description | | --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- | | `-type` | string | Must be `aws-iam`. | | `-name` | string | A name of your choice. Must be unique among all auth methods. | | `-description` | string | A description of your choice. | | `-config` | string | A JSON string containing the [configuration](/docs/security/acl/auth-methods/aws-iam#config-parameters) for the the auth method. | In the `-config` option, the following fields are required: | Field | Type | Description | | --- | --- | --- | | `BoundIAMPrincipalArns` | list | The list of trusted IAM roles. We recommend using a wildcard to trust IAM roles at a particular path. | | `EnableIAMEntityDetails` | boolean | Must be true so that the auth method can retrieve IAM role details, such as the role path and role tags. | | `IAMEntityTags` | list | The list of IAM role tags to make available to binding rules. Must include the service name tag as shown. | The following binding rule is used to associate a service identity with each token created by successful login to this instance of the auth method. The service identity name is taken from the `consul.hashicorp.com.service-name` tag from the authenticating IAM role identity. #### Create Binding Rule ```shell consul acl binding-rule create \ -method iam-ecs-service-token \ -description 'Bind a service identity from IAM role tags for ECS service tokens' \ -bind-type service \ -bind-name '${entity_tags.consul.hashicorp.com.service-name}' ``` ### Configuration for Consul Enterprise When using Consul Enterprise namespaces and admin partitions, pass the `-partition ` option to the Consul CLI when creating Consul ACL roles, policies, auth methods, and binding rules, in order to create these resources in a particular partition. The following shows how to create the ACL policy for client tokens. This ensures permissions for the client token are scoped to a particular partition. ```hcl partition "" { node_prefix "" { policy = "write" } namespace_prefix "" { service_prefix "" { policy = "read" } } } ``` The following commands show how to use the Consul CLI to create the policy and role. ```shell consul acl policy create -partition \ -name consul-ecs-client-policy \ -rules @client-token-policy-ent.hcl consul acl role create \ -partition \ -name consul-ecs-client-role \ -policy-name consul-ecs-client-policy ``` The auth method for *service tokens* requires the following additional configuration to include a namespace binding rule. This ensures the service tokens are created in the right namespace during login. (The namespace binding rule must not be configured on the instance of the auth method instance for *client tokens*.) ```shell consul acl auth-method create \ -partition \ -type aws-iam \ -name iam-ecs-service-token \ -description="AWS IAM auth method for ECS service tokens" \ -namespace-rule-selector 'entity_tags["consul.hashicorp.com.namespace"] != ""' \ -namespace-rule-bind-namespace '${entity_tags.consul.hashicorp.com.namespace}' \ -config '{ "BoundIAMPrincipalArns": ["arn:aws:iam:::role/consul-ecs/*"], "EnableIAMEntityDetails": true, "IAMEntityTags": [ "consul.hashicorp.com.service-name", "consul.hashicorp.com.namespace" ] }' ``` | Field | Type | Description | | -------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-partition` | string | The Consul Enterprise admin partition in which the auth method is created. | | `-namespace-rule-selector` | string | When this expression evaluates to true during login, the `-namespace-rule-bind-namespace` is applied. As shown, it evaluates to true when the `consul.hashicorp.com.namespace` tag is non-empty on the task IAM role. | | `-namespace-rule-bind-namespace` | string | This expression is evaluated to determine the namespace where the token is created during login. As shown, it uses the namespace from the `consul.hashicorp.com.namespace` tag on the task IAM role. | | `IAMEntityTags` | list | Must include `consul.hashicorp.com.namespace` to enable use of this tag in binding rules. | ## Secret storage You should securely store the following secrets in order to make them available to ECS tasks. 1. Consul Server CA certificates. More than one may be required for different Consul protocols. 2. Consul gossip encryption key These secrets can be securely stored and passed to ECS tasks using either of the following AWS secret services: * [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-parameters.html) * [AWS Secrets Manager](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html) Once the secrets are stored they can be referenced using their ARN. The following shows example secret ARNs when using AWS Secrets Manager: | Secret | Sample Secret ARN | | ---------------------- | ---------------------------------------------------------------------------------- | | Consul Server CA Cert for RPC | `arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-ca-cert` | | Consul Server CA Cert for HTTPS | `arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-https-ca-cert` | | Gossip encryption key | `arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-gossip-key` | ## Configure `consul-client` The following secrets must be passed to the `consul-client` container: * Consul server CA certificates * Gossip encryption key The following example shows how to include these secrets in the task definition. The `secrets` list specifies environment variable `name`s that will be set to the secret values for this container. ECS automatically fetches the secret values specified in the `valueFrom` fields during task provisioning. ```json { "containerDefinitions": [ { "name": "consul-client" "image": "public.ecr.aws/hashicorp/consul:", "secrets": [ { "name": "CONSUL_CACERT_PEM", "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-ca-cert" }, { "name": "CONSUL_HTTPS_CACERT_PEM", "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-https-ca-cert" }, { "name": "CONSUL_GOSSIP_ENCRYPTION_KEY", "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-gossip-key" } ] }, ... ] } ``` Next, update the Consul client startup script with the `consul login` command and additional Consul configuration options for a secure configuration. The following is an example of the *additional* content to include in the `consul-client` startup script. Refer to the [install page](/docs/ecs/manual/install#consul-client-container) for the remainder of the startup script and how to pass this script to the container. ```shell ... # Obtain details from the task metadata ECS_TASK_META=$(curl -s $ECS_CONTAINER_METADATA_URI_V4/task) TASK_REGION=$(echo "$ECS_TASK_META" | jq -r .TaskARN | cut -d ':' -f 4) TASK_ID=$(echo "$ECS_TASK_META" | jq -r .TaskARN | cut -d '/' -f 3) CLUSTER_ARN=$(echo "$ECS_TASK_META" | jq -r .TaskARN | sed -E 's|:task/([^/]+).*|:cluster/\1|') # Write the CA certs to a files in the shared volume echo "$CONSUL_CACERT_PEM" > /consul/consul-ca-cert.pem echo "$CONSUL_HTTPS_CACERT_PEM" > /consul/consul-https-ca-cert.pem consul_login() { echo "Logging into auth method" consul login \ -http-addr "" \ -ca-file /consul/consul-https-ca-cert.pem \ -partition "" \ -type aws-iam \ -method iam-ecs-client-token \ -meta "consul.hashicorp.com/task-id=$TASK_ID" \ -meta "consul.hashicorp.com/cluster=$CLUSTER_ARN" \ -aws-region "$TASK_REGION" \ -aws-auto-bearer-token -aws-include-entity \ -token-sink-file /consul/client-token } read_token_stale() { consul acl token read -http-addr ${ consul_http_addr } \ -ca-file /consul/consul-https-ca-cert.pem \ -stale -self -token-file /consul/client-token \ > /dev/null } # Retry in order to login successfully. while ! consul_login; do sleep 2 done # Allow the health-sync container to read this token for consul logout. chmod 0644 /consul/client-token # Wait for raft replication to hopefully occur. Without this, an "ACL not found" may be cached for a while. COUNT=20 while [ "$COUNT" -gt 0 ]; do echo "Checking that the ACL token exists when reading it in the stale consistency mode ($COUNT attempts remaining)" if read_token_stale; then echo "Successfully read ACL token from the server" break fi sleep 0.1 COUNT=$((COUNT - 1)) done if [ "$COUNT" -eq 0 ]; then echo "Unable to read ACL token from a Consul server; please check that your server cluster is healthy" exit 1 fi # This is interpolated into the agent-defaults.hcl export AGENT_TOKEN=$(cat /consul/client-token) # Write the Consul agent configuration file. cat << EOF > /consul/agent-defaults.hcl ... # Configure gossip encryption key encrypt = "$CONSUL_GOSSIP_ENCRYPTION_KEY" # Configure TLS settings auto_encrypt = { tls = true ip_san = ["$ECS_IPV4"] } tls { defaults { ca_file = "/consul/consul-ca-cert.pem" verify_outgoing = true } } # Configure ACLs acl { enabled = true default_policy = "deny" down_policy = "async-cache" tokens { agent = "$AGENT_TOKEN" } } partition = "" EOF ``` The following describes the additional steps added to the startup script: * Fetch additional details from the task metadata: the AWS region, task id, and cluster arn. These details are necessary for the `consul login` command used to obtain a Consul client token. * Write CA certificates to files for Consul CLI and Consul client * Run the `consul login` command in a retry loop. * Wait for Raft replication to hopefully occur for this token. * Configure the Consul client config file with additional fields necessary for secure configuration. The following flags are passed to the `consul login` command: | Field name | Type | Description | | ------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------ | | [`-http-addr`](/commands/login#http-addr) | string | HTTP(S) address of the Consul server. | | [`-ca-file`](/commands/login#ca-file) | string | Path of the CA cert for Consul's HTTPS interface. Not required when using Consul servers on HCP. | | `-partition` | string | The Consul Enterprise admin partition the auth method belongs to. | | [`-type`](/commands/login#type) | string | The auth method type. Must be `aws-iam`. | | [`-method`](/commands/login#type) | string | The auth method name. Must be the name of the auth method for ECS client tokens. | | [`-meta`](/commands/login#meta) | string | Metadata to set in description of the created token. Should include the task id and cluster as shown. | | `-aws-region` | string | The AWS region where the task is running. | | `-aws-auto-bearer-token` | | Must be set to login to the AWS IAM auth method. | | `-aws-include-entity` | | Must be set to enable IAM role details. | The following table describes the additional fields that must be included in the Consul client configuration file. | Field name | Type | Description | | ------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------ | | [`encrypt`](/docs/agent/config/cli-flags#_encrypt) | string | Gossip encryption key | | [`tls.defaults.ca_file`](/docs/agent/config/config-files#tls_defaults_ca_file) | string | Consul server CA cert for TLS verification. | | [`acl.enabled`](/docs/agent/config/config-files#acl_enabled) | boolean | Enable ACLs for this agent. | | [`acl.tokens.agent`](/docs/agent/config/config-files#acl_tokens_agent) | string | Consul client token which authorizes this agent with Consul servers. | | [`partition`](/docs/agent/config/config-files#partition-1) | string | The Consul Enterprise admin partition this agent belongs to. | ### Configure Audit Logging [Audit logging](/docs/enterprise/audit-logging) is supported on clients running Consul Enterprise with ACLs enabled. To enable audit logging, update the startup script to add an `audit` stanza to the Consul client configuration file. The following example modifies the `consul-client` startup script to configure audit logs to be written to the `stdout` of the `consul-client` container. ```shell ... # Write the Consul agent configuration file. cat << EOF > /consul/agent-defaults.hcl ... partition = "" audit { enabled = true sink "stdout" { type = "file" format = "json" path = "/dev/stdout" delivery_guarantee = "best-effort" } } EOF ``` The following table describes the fields that must be included to configure audit logging. | Field name | Type | Description | | ------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------ | | [`audit.enabled`](/docs/agent/config/config-files#enabled) | boolean | Enable audit logging for this agent. | | [`audit.sink`](/docs/agent/config/config-files#sink) | object | The audit logging sink for this agent. | ## Configure `consul-ecs-mesh-init` and `consul-ecs-health-sync` The following *additional* options should be set in the [`CONSUL_ECS_CONFIG_JSON`](/docs/ecs/manual/install#consul_ecs_config_json) environment variable. When these options are specified, the `consul-ecs mesh-init` command will run the `consul login` command to obtain a service token from the Consul AWS IAM Auth method. The `consul-ecs health-sync` command is responsible for running a `consul logout` command for both the service and client tokens when the task shuts down. ```json { "consulHTTPAddr": "", "consulCACertFile": "/consul/consul-https-ca-cert.pem", "consulLogin": { "enabled": true, "method": "iam-ecs-service-token", "extraLoginFlags": ["-partition", ""] }, "bootstrapDir": "/consul", "healthSyncContainers": [], ... } ``` The following table explains the additional fields for the `CONSUL_ECS_CONFIG_JSON`: | Field name | Type | Description | | ----------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `consulHTTPAddr` | string | HTTP(S) address for the Consul server. | | `consulCACertFile` | string | Path of the CA cert file for Consul's HTTPS interface. Not required for Consul servers in HCP. | | `consulLogin.enabled` | boolean | Must be set to true to log in to the auth method. | | `consulLogin.method` | string | Must be set to the name of the auth method instance for service tokens. | | `consulLogin.extraLoginFlags` | list | Additional flags passed to the `consul login` command. This shows how to pass the Consul Enterprise admin partition to the `consul login` command. |