open-consul/website/content/docs/ecs/manual/secure-configuration.mdx

---
layout: docs
page_title: Manual Secure Configuration - Consul on AWS Elastic Container Service (ECS)
description: >-
  Securely configure Consul Service Mesh on Amazon Web Services ECS to protect the communication of sensitive data in production workloads. Learn how to manually configure auth methods to create tokens for clients and services and then apply tokens by role or policy.
---

# Manual Secure Configuration of Consul on AWS Elastic Container Service (ECS)

This topic describes how to enable Consul security features for your production workloads.

## Prerequisites

The following features must be configured for your Consul server cluster:

- [TLS encryption](/docs/security/encryption#rpc-encryption-with-tls) for RPC communication between Consul clients and servers.
- [Gossip encryption](/docs/security/encryption#gossip-encryption) for encrypting gossip traffic.
- [Access control lists (ACLs)](/docs/security/acl) for authentication and authorization for Consul clients and services on the mesh.

You should already have followed the [manual installation instructions](/docs/ecs/manual/install) to define the necessary components of the task definition for Consul on ECS.

You should be familiar with [specifying sensitive data](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html) on ECS.

You should be familiar with configuring Consul's secure features, including how to create ACL tokens and policies. Refer to the [ACL system documentation](/docs/security/acl) and  [Day 1: Security tutorials](https://learn.hashicorp.com/collections/consul/security) for an introduction and additional information.

## Auth Method

Tokens are artifacts within the ACL system that authenticate users, services, and Consul agents. Tokens are linked to policies that specify the resources the token bearer has access to when making requests in the network.

Auth Methods are a Consul server component that performs authentication against a trusted external party to authorize the creation of ACL tokens. The [AWS IAM auth method](/docs/security/acl/auth-methods/aws-iam) is used to enable an ECS task to automatically obtain ACL tokens when the task starts up.

There are two types of ACL tokens for Consul on ECS:

* **Client tokens:** used by the `consul-client` containers to join the Consul cluster
* **Service tokens:** used by sidecar containers for service registration and health syncing

This section describes how to manually configure the AWS IAM auth method for Consul on ECS. Alternatively, you can install the ACL controller to ease the burden of creating these resources. The ACL controller can automatically configure ACL resources for Consul on ECS. For additional details, refer to [ACL Controller](/docs/ecs/manual/acl-controller) and [Architecture](/docs/ecs/architecture).

### ECS Task Role Configuration

The ECS [task role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)
is an IAM role associated with an ECS task.

When an ECS task starts up, it runs a `consul login` command. The `consul login` command obtains
credentials for the task role role from AWS. It uses those credentials to sign the login request to
the AWS IAM auth method. This proves the ECS task's identity to the Consul servers.

The task role must be configured with the following details to compatible with the AWS IAM auth
method.

* An `iam:GetRole` permission to fetch itself. See
  [IAM Policies](/docs/security/acl/auth-methods/aws-iam#iam-policies) for details.
* A `consul.hashicorp.com.service-name` tag on the task role which contains the Consul service name
  for the application in this task.
* <EnterpriseAlert inline /> A <code>consul.hashicorp.com.namespace</code> tag on the task role
  indicating the Consul Enterprise namespace where this service is registering.

The following sections describe how to configure the auth method to enable task roles to
successfully authenticate to the AWS IAM auth method and obtain tokens with the necessary
permissions.

### Auth Method for Client Tokens

The following steps configure an instance of the auth method that creates client tokens for tasks.

1. Create the auth method instance
2. Create the client policy and role
3. Create the binding rule

#### Create Auth Method for Client Tokens

The following Consul CLI command creates an instance of the auth method for client tokens.

```shell
consul acl auth-method create \
    -type aws-iam \
    -name iam-ecs-client-token \
    -description="AWS IAM auth method for ECS client tokens" \
    -config '{
  "BoundIAMPrincipalArns": ["arn:aws:iam::<ACCOUNT>:role/consul-ecs/*"],
  "EnableIAMEntityDetails": true
}'
```

The following flags are required:

| Flag            | Type    | Description                                                                                                                       |
| --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `-type`         | string  | Must be `aws-iam`.                                                                                                                |
| `-name`         | string  | A name of your choice. Must be unique among all auth methods.                                                                     |
| `-description`  | string  | A description of your choice.                                                                                                     |
| `-config`       | string  | A JSON string containing the [configuration](/docs/security/acl/auth-methods/aws-iam#config-parameters) for the the auth method.  |

In the `-config` option, the following fields are required:

| Field                     | Type    | Description                                                                                               |
| ------------------------- | ------- | --------------------------------------------------------------------------------------------------------- |
| `BoundIAMPrincipalArns`   | list    | The list of trusted IAM roles. We recommend using a wildcard to trust IAM roles at a particular path.     |
| `EnableIAMEntityDetails`  | boolean | Must be true so that the auth method can retrieve IAM role details, such as the role path and role tags.  |

#### Create Client Policy and Role

Configure the following ACL policy for Consul client tokens:

<CodeBlockConfig filename="client-token-policy.hcl">

```hcl
node_prefix "" {
  policy = "write"
}
service_prefix "" {
  policy = "read"
}
```

</CodeBlockConfig>

The policy allows `node:write` for any node name, which is necessary because the Consul node names on ECS are not known until runtime.

You can add the policy in Consul using the [`consul acl policy create`](/commands/acl/policy/create) command or the [`[PUT] /v1/acl/policy`](/api-docs/acl/policies#create-a-policy) API endpoint.

After the policy is created, create a Consul role associated with the policy by using the [`consul acl role create`](/commands/acl/role/create) command or the [`[PUT] /v1/acl/role`](/api-docs/acl/roles#create-a-role) API endpoint.

The following example shows how to use the Consul CLI to create the client policy and role.

```shell
consul acl policy create -name consul-ecs-client-policy -rules @client-token-policy.hcl

consul acl role create -name consul-ecs-client-role -policy-name consul-ecs-client-policy
```

#### Create Binding Rule for Client Tokens

The following creates a binding rule for the auth method for client tokens. The binding rule
associates the client role with each token created by a successful login to this auth
method instance.

```shell
consul acl binding-rule create -method iam-ecs-client-token \
   -description 'Bind a role for Consul clients on ECS' \
   -bind-type role \
   -bind-name consul-ecs-client-role
```

### Auth Method for Service Tokens


The following steps configure an instance of the auth method that creates service tokens for tasks.

* Create the auth method instance
* Create the binding rule

#### Create Auth Method for Service Tokens

The following uses the Consul CLI to create an instance of the auth method
for service tokens. This configures the auth method to associate a service identity
to each token created during login to this auth method instance.

```shell
consul acl auth-method create \
    -type aws-iam \
    -name iam-ecs-service-token \
    -description="AWS IAM auth method for ECS service tokens" \
    -config '{
  "BoundIAMPrincipalArns": ["arn:aws:iam::<ACCOUNT>:role/consul-ecs/*"],
  "EnableIAMEntityDetails": true,
  "IAMEntityTags": [
    "consul.hashicorp.com.service-name"
  ]
}'
```

The following flags are required:

| Flag            | Type    | Description                                                                                                                       |
| --------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `-type`         | string  | Must be `aws-iam`.                                                                                                                |
| `-name`         | string  | A name of your choice. Must be unique among all auth methods.                                                                     |
| `-description`  | string  | A description of your choice.                                                                                                     |
| `-config`       | string  | A JSON string containing the [configuration](/docs/security/acl/auth-methods/aws-iam#config-parameters) for the the auth method.  |

In the `-config` option, the following fields are required:

| Field                    | Type    | Description                                                                                               |
| ---                      | ---     | ---                                                                                                       |
| `BoundIAMPrincipalArns`  | list    | The list of trusted IAM roles. We recommend using a wildcard to trust IAM roles at a particular path.     |
| `EnableIAMEntityDetails` | boolean | Must be true so that the auth method can retrieve IAM role details, such as the role path and role tags.  |
| `IAMEntityTags`          | list    | The list of IAM role tags to make available to binding rules. Must include the service name tag as shown. |

The following binding rule is used to associate a service identity with each token created by
successful login to this instance of the auth method. The service identity name is taken from the
`consul.hashicorp.com.service-name` tag from the authenticating IAM role identity.

#### Create Binding Rule

```shell
consul acl binding-rule create \
   -method iam-ecs-service-token \
   -description 'Bind a service identity from IAM role tags for ECS service tokens' \
   -bind-type service \
   -bind-name '${entity_tags.consul.hashicorp.com.service-name}'
```

### Configuration for Consul Enterprise<EnterpriseAlert inline />

When using Consul Enterprise namespaces and admin partitions, pass the `-partition <partition-name>`
option to the Consul CLI when creating Consul ACL roles, policies, auth methods, and binding rules,
in order to create these resources in a particular partition.

The following shows how to create the ACL policy for client tokens. This ensures permissions for the
client token are scoped to a particular partition.

<CodeBlockConfig filename="client-token-policy-ent.hcl">

```hcl
partition "<partition-name>" {
  node_prefix "" {
    policy = "write"
  }
  namespace_prefix "" {
    service_prefix "" {
      policy = "read"
    }
  }
}
```

</CodeBlockConfig>

The following commands show how to use the Consul CLI to create the policy and role.

```shell
consul acl policy create -partition <partition-name> \
    -name consul-ecs-client-policy \
    -rules @client-token-policy-ent.hcl

consul acl role create \
  -partition <partition-name> \
  -name consul-ecs-client-role \
  -policy-name consul-ecs-client-policy
```

The auth method for *service tokens* requires the following additional configuration to include a
namespace binding rule. This ensures the service tokens are created in the right namespace during
login. (The namespace binding rule must not be configured on the instance of the auth method
instance for *client tokens*.)

<CodeBlockConfig highlight="2-2,6-7,13-13">

```shell
consul acl auth-method create \
    -partition <partition-name> \
    -type aws-iam \
    -name iam-ecs-service-token \
    -description="AWS IAM auth method for ECS service tokens" \
    -namespace-rule-selector 'entity_tags["consul.hashicorp.com.namespace"] != ""' \
    -namespace-rule-bind-namespace '${entity_tags.consul.hashicorp.com.namespace}' \
    -config '{
  "BoundIAMPrincipalArns": ["arn:aws:iam::<ACCOUNT>:role/consul-ecs/*"],
  "EnableIAMEntityDetails": true,
  "IAMEntityTags": [
    "consul.hashicorp.com.service-name",
    "consul.hashicorp.com.namespace"
  ]
}'
```

</CodeBlockConfig>

| Field                            | Type   | Description                                                                                                                                                                                                           |
| -------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-partition`                     | string | The Consul Enterprise admin partition in which the auth method is created.                                                                                                                                            |
| `-namespace-rule-selector`       | string | When this expression evaluates to true during login, the `-namespace-rule-bind-namespace` is applied. As shown, it evaluates to true when the `consul.hashicorp.com.namespace` tag is non-empty on the task IAM role. |
| `-namespace-rule-bind-namespace` | string | This expression is evaluated to determine the namespace where the token is created during login. As shown, it uses the namespace from the `consul.hashicorp.com.namespace` tag on the task IAM role.                   |
| `IAMEntityTags`                  | list   | Must include `consul.hashicorp.com.namespace` to enable use of this tag in binding rules.                                                                                                                             |

## Secret storage

You should securely store the following secrets in order to make them available to ECS tasks.

1. Consul Server CA certificates. More than one may be required for different Consul protocols.
2. Consul gossip encryption key

These secrets can be securely stored and passed to ECS tasks using either of the following AWS secret services:

* [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-parameters.html)
* [AWS Secrets Manager](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html)

Once the secrets are stored they can be referenced using their ARN. The following shows
example secret ARNs when using AWS Secrets Manager:

| Secret                          | Sample Secret ARN                                                                  |
| ----------------------          | ---------------------------------------------------------------------------------- |
| Consul Server CA Cert for RPC   | `arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-ca-cert`           |
| Consul Server CA Cert for HTTPS | `arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-https-ca-cert`     |
| Gossip encryption key           | `arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-gossip-key`        |

## Configure `consul-client`

The following secrets must be passed to the `consul-client` container:

* Consul server CA certificates
* Gossip encryption key

The following example shows how to include these secrets in the task definition. The `secrets`
list specifies environment variable `name`s that will be set to the secret values for this container.
ECS automatically fetches the secret values specified in the `valueFrom` fields during task provisioning.

```json
{
  "containerDefinitions": [
    {
      "name": "consul-client"
      "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
      "secrets": [
        {
          "name": "CONSUL_CACERT_PEM",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-ca-cert"
        },
        {
          "name": "CONSUL_HTTPS_CACERT_PEM",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-https-ca-cert"
        },
        {
          "name": "CONSUL_GOSSIP_ENCRYPTION_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-west-2:000000000000:secret:my-consul-gossip-key"
        }
      ]
    },
    ...
  ]
}
```

Next, update the Consul client startup script with the `consul login` command and additional Consul configuration
options for a secure configuration.

The following is an example of the *additional* content to include in the `consul-client` startup script. Refer to the [install
page](/docs/ecs/manual/install#consul-client-container) for the remainder of the startup script and how to pass this
script to the container.

<CodeBlockConfig highlight="3-60,66-91">

```shell
...

# Obtain details from the task metadata
ECS_TASK_META=$(curl -s $ECS_CONTAINER_METADATA_URI_V4/task)
TASK_REGION=$(echo "$ECS_TASK_META" | jq -r .TaskARN | cut -d ':' -f 4)
TASK_ID=$(echo "$ECS_TASK_META" | jq -r .TaskARN | cut -d '/' -f 3)
CLUSTER_ARN=$(echo "$ECS_TASK_META" | jq -r .TaskARN | sed -E 's|:task/([^/]+).*|:cluster/\1|')

# Write the CA certs to a files in the shared volume
echo "$CONSUL_CACERT_PEM" > /consul/consul-ca-cert.pem
echo "$CONSUL_HTTPS_CACERT_PEM" > /consul/consul-https-ca-cert.pem

consul_login() {
    echo "Logging into auth method"
    consul login \
      -http-addr "<consul server address>" \
      -ca-file /consul/consul-https-ca-cert.pem \
      -partition "<partition>" \
      -type aws-iam \
      -method iam-ecs-client-token \
      -meta "consul.hashicorp.com/task-id=$TASK_ID" \
      -meta "consul.hashicorp.com/cluster=$CLUSTER_ARN" \
      -aws-region "$TASK_REGION" \
      -aws-auto-bearer-token -aws-include-entity \
      -token-sink-file /consul/client-token
}

read_token_stale() {
    consul acl token read -http-addr ${ consul_http_addr } \
      -ca-file /consul/consul-https-ca-cert.pem \
      -stale -self -token-file /consul/client-token \
      > /dev/null
}

# Retry in order to login successfully.
while ! consul_login; do
    sleep 2
done

# Allow the health-sync container to read this token for consul logout.
chmod 0644 /consul/client-token

# Wait for raft replication to hopefully occur. Without this, an "ACL not found" may be cached for a while.
COUNT=20
while [ "$COUNT" -gt 0 ]; do
    echo "Checking that the ACL token exists when reading it in the stale consistency mode ($COUNT attempts remaining)"
    if read_token_stale; then
        echo "Successfully read ACL token from the server"
        break
    fi
    sleep 0.1
    COUNT=$((COUNT - 1))
done
if [ "$COUNT" -eq 0 ]; then
   echo "Unable to read ACL token from a Consul server; please check that your server cluster is healthy"
   exit 1
fi

# This is interpolated into the agent-defaults.hcl
export AGENT_TOKEN=$(cat /consul/client-token)

# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
...

# Configure gossip encryption key
encrypt = "$CONSUL_GOSSIP_ENCRYPTION_KEY"

# Configure TLS settings
auto_encrypt = {
  tls = true
  ip_san = ["$ECS_IPV4"]
}
tls {
  defaults {
    ca_file = "/consul/consul-ca-cert.pem"
    verify_outgoing = true
  }
}

# Configure ACLs
acl {
  enabled = true
  default_policy = "deny"
  down_policy = "async-cache"
  tokens {
    agent = "$AGENT_TOKEN"
  }
}

partition = "<partition>"

EOF
```

</CodeBlockConfig>

The following describes the additional steps added to the startup script:

* Fetch additional details from the task metadata: the AWS region, task id, and cluster arn.
  These details are necessary for the `consul login` command used to obtain a Consul client token.
* Write CA certificates to files for Consul CLI and Consul client
* Run the `consul login` command in a retry loop.
* Wait for Raft replication to hopefully occur for this token.
* Configure the Consul client config file with additional fields necessary for secure configuration.

The following flags are passed to the `consul login` command:

| Field name                                                                      | Type    | Description                                                                                           |
| ------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------                  |
| [`-http-addr`](/commands/login#http-addr)                                       | string  | HTTP(S) address of the Consul server.                                                                 |
| [`-ca-file`](/commands/login#ca-file)                                           | string  | Path of the CA cert for Consul's HTTPS interface. Not required when using Consul servers on HCP.      |
| `-partition`                                                                    | string  | <EnterpriseAlert inline /> The Consul Enterprise admin partition the auth method belongs to.          |
| [`-type`](/commands/login#type)                                                 | string  | The auth method type. Must be `aws-iam`.                                                              |
| [`-method`](/commands/login#type)                                               | string  | The auth method name. Must be the name of the auth method for ECS client tokens.                      |
| [`-meta`](/commands/login#meta)                                                 | string  | Metadata to set in description of the created token. Should include the task id and cluster as shown. |
| `-aws-region`                                                                   | string  | The AWS region where the task is running.                                                             |
| `-aws-auto-bearer-token`                                                        |         | Must be set to login to the AWS IAM auth method.                                                      |
| `-aws-include-entity`                                                           |         | Must be set to enable IAM role details.                                                               |

The following table describes the additional fields that must be included in the Consul client configuration file.

| Field name                                                                      | Type    | Description                                                                             |
| ------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------    |
| [`encrypt`](/docs/agent/config/cli-flags#_encrypt)                              | string  | Gossip encryption key                                                                   |
| [`tls.defaults.ca_file`](/docs/agent/config/config-files#tls_defaults_ca_file)  | string  | Consul server CA cert for TLS verification.                                             |
| [`acl.enabled`](/docs/agent/config/config-files#acl_enabled)                    | boolean | Enable ACLs for this agent.                                                             |
| [`acl.tokens.agent`](/docs/agent/config/config-files#acl_tokens_agent)          | string  | Consul client token which authorizes this agent with Consul servers.                    |
| [`partition`](/docs/agent/config/config-files#partition-1)                      | string  | <EnterpriseAlert inline /> The Consul Enterprise admin partition this agent belongs to. |

### Configure Audit Logging <EnterpriseAlert inline />
[Audit logging](/docs/enterprise/audit-logging) is supported on clients running Consul Enterprise with ACLs enabled.
To enable audit logging, update the startup script to add an `audit` stanza to the Consul client configuration file.

The following example modifies the `consul-client` startup script to configure audit logs to be written to the `stdout` of the `consul-client` container.

<CodeBlockConfig highlight="9-17">

```shell
...

# Write the Consul agent configuration file.
cat << EOF > /consul/agent-defaults.hcl
...

partition = "<partition>"

audit {
  enabled = true
  sink "stdout" {
    type   = "file"
    format = "json"
    path   = "/dev/stdout"
    delivery_guarantee = "best-effort"
  }
}

EOF
```

</CodeBlockConfig>

The following table describes the fields that must be included to configure audit logging.

| Field name                                                                      | Type    | Description                                                                             |
| ------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------    |
| [`audit.enabled`](/docs/agent/config/config-files#enabled)                      | boolean | Enable audit logging for this agent.                                                    |
| [`audit.sink`](/docs/agent/config/config-files#sink)                            | object  | The audit logging sink for this agent.                                                  |

## Configure `consul-ecs-mesh-init` and `consul-ecs-health-sync`

The following *additional* options should be set in the [`CONSUL_ECS_CONFIG_JSON`](/docs/ecs/manual/install#consul_ecs_config_json) environment variable. When these options are specified, the `consul-ecs mesh-init` command will run the `consul login` command to obtain a service token from the Consul AWS IAM Auth method. The `consul-ecs health-sync` command is responsible for running a `consul logout` command for both the service and client tokens when the task shuts down.

<CodeBlockConfig highlight="2-8">

```json
{
  "consulHTTPAddr": "<Consul server address>",
  "consulCACertFile": "/consul/consul-https-ca-cert.pem",
  "consulLogin": {
    "enabled": true,
    "method": "iam-ecs-service-token",
    "extraLoginFlags": ["-partition", "<partition>"]
  },
  "bootstrapDir": "/consul",
  "healthSyncContainers": [],
  ...
}
```

</CodeBlockConfig>

The following table explains the additional fields for the `CONSUL_ECS_CONFIG_JSON`:

| Field name                    | Type    | Description                                                                                                                                                                   |
| ----------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `consulHTTPAddr`              | string  | HTTP(S) address for the Consul server.                                                                                                                                        |
| `consulCACertFile`            | string  | Path of the CA cert file for Consul's HTTPS interface. Not required for Consul servers in HCP.                                                                                |
| `consulLogin.enabled`         | boolean | Must be set to true to log in to the auth method.                                                                                                                             |
| `consulLogin.method`          | string  | Must be set to the name of the auth method instance for service tokens.                                                                                                       |
| `consulLogin.extraLoginFlags` | list    | Additional flags passed to the `consul login` command. <EnterpriseAlert inline /> This shows how to pass the Consul Enterprise admin partition to the `consul login` command. |