2018-05-29 21:07:40 +00:00
---
2020-04-07 18:55:19 +00:00
layout: docs
2022-09-13 20:30:45 +00:00
page_title: Service Mesh Certificate Authority: Overview
description: >-
Consul uses a certificate authority (CA) to generate, use, manage, sign, and store certificates for your service mesh. Learn about certificate management, including configuration, root cert rotation, cross-signing, and regenerating the CA.
2018-05-29 21:07:40 +00:00
---
2022-09-13 20:30:45 +00:00
# Service Mesh Certificate Authority Overview
2018-05-29 21:07:40 +00:00
2018-06-24 21:55:26 +00:00
Certificate management in Connect is done centrally through the Consul
2018-06-06 17:50:13 +00:00
servers using the configured CA (Certificate Authority) provider. A CA provider
2018-06-24 21:55:26 +00:00
manages root and intermediate certificates and performs certificate signing
operations. The Consul leader orchestrates CA provider operations as necessary,
such as when a service needs a new certificate or during CA rotation events.
The CA provider abstraction enables Consul to support multiple systems for
storing and signing certificates. Consul ships with a
2020-04-09 23:46:54 +00:00
[built-in CA](/docs/connect/ca/consul) which generates and stores the
2018-06-25 18:12:53 +00:00
root certificate and private key on the Consul servers. Consul also has
2021-08-19 18:18:55 +00:00
support for using
2020-04-09 23:46:54 +00:00
[Vault as a CA](/docs/connect/ca/vault). With Vault, the root certificate
2020-10-20 16:58:28 +00:00
and private key material remain with the Vault cluster.
2018-06-24 21:55:26 +00:00
2022-01-12 21:10:00 +00:00
### CA and Certificate relationship
This diagram shows the relationship between the CA certificates in a Consul primary datacenter and a
secondary Consul datacenter.
Leaf certificates are created for two purposes:
- the Leaf Cert Service is used by envoy proxies in the mesh to perform mTLS with other
services.
- the Leaf Cert Client Agent is created by auto-encrypt and auto-config. It is used by
client agents for HTTP API TLS, and for mTLS for RPC requests to servers.
Any secondary datacenters receive an intermediate certificate, signed by the Primary Root
CA, which is used as the CA certificate to sign leaf certificates in the secondary
datacenter.
2018-06-24 21:55:26 +00:00
## CA Bootstrapping
CA initialization happens automatically when a new Consul leader is elected
as long as
2021-08-19 18:18:55 +00:00
[Connect is enabled](/docs/connect/configuration#agent-configuration),
and the CA system has not already been initialized. This initialization process
2018-06-24 21:55:26 +00:00
will generate the initial root certificates and setup the internal Consul server
state.
For the initial bootstrap, the CA provider can be configured through the
2022-01-11 01:30:50 +00:00
[Agent configuration](/docs/agent/config/config-files#connect_ca_config). After
2018-06-24 21:55:26 +00:00
initialization, the CA can only be updated through the
2022-03-30 21:16:26 +00:00
[Update CA Configuration API endpoint](/api-docs/connect/ca#update-ca-configuration).
2018-06-24 21:55:26 +00:00
If a CA is already initialized, any changes to the CA configuration in the
agent configuration file (including removing the configuration completely)
will have no effect.
If no specific provider is configured when Connect is enabled, the built-in
Consul CA provider will be used and a private key and root certificate will
be generated automatically.
## Viewing Root Certificates
Root certificates can be queried with the
2022-03-30 21:16:26 +00:00
[list CA Roots endpoint](/api-docs/connect/ca#list-ca-root-certificates).
2018-06-24 21:55:26 +00:00
With this endpoint, you can see the list of currently trusted root certificates.
When a cluster first initializes, this will only list one trusted root. Multiple
roots may appear as part of
2020-10-14 15:23:05 +00:00
[rotation](#root-certificate-rotation).
2018-06-06 17:50:13 +00:00
2020-05-19 18:32:38 +00:00
```shell-session
2018-06-24 21:55:26 +00:00
$ curl http://localhost:8500/v1/connect/ca/roots
2018-06-06 17:50:13 +00:00
{
"ActiveRootID": "31:6c:06:fb:49:94:42:d5:e4:55:cc:2e:27:b3:b2:2e:96:67:3e:7e",
"TrustDomain": "36cb52cd-4058-f811-0432-6798a240c5d3.consul",
"Roots": [
{
"ID": "31:6c:06:fb:49:94:42:d5:e4:55:cc:2e:27:b3:b2:2e:96:67:3e:7e",
"Name": "Consul CA Root Cert",
"SerialNumber": 7,
2019-09-23 17:52:35 +00:00
"SigningKeyID": "19:45:8b:30:a1:45:84:ae:23:52:db:8d:1b:ff:a9:09:db:fc:2a:72:39:ae:da:11:53:f4:37:5c:de:d1:68:d8",
"ExternalTrustDomain": "a1499528-fbf6-df7b-05e5-ae81e1873fc4",
2018-06-06 17:50:13 +00:00
"NotBefore": "2018-06-06T17:35:25Z",
"NotAfter": "2028-06-03T17:35:25Z",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIICmDCCAj6gAwIBAgIBBzAKBggqhkjOPQQDAjAWMRQwEgYDVQQDEwtDb25zdWwg\nQ0EgNzAeFw0xODA2MDYxNzM1MjVaFw0yODA2MDMxNzM1MjVaMBYxFDASBgNVBAMT\nC0NvbnN1bCBDQSA3MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEgo09lpx63bHw\ncSXeeoSpHpHgyzX1Q8ewJ3RUg6Ie8Howbs/QBz1y/kGxsF35HXij3YrqhgQyPPx4\nbQ8FH2YR4aOCAXswggF3MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/\nMGgGA1UdDgRhBF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToyMzo1MjpkYjo4ZDox\nYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1MzpmNDozNzo1Yzpk\nZTpkMTo2ODpkODBqBgNVHSMEYzBhgF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToy\nMzo1MjpkYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1\nMzpmNDozNzo1YzpkZTpkMTo2ODpkODA/BgNVHREEODA2hjRzcGlmZmU6Ly8zNmNi\nNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMuY29uc3VsMD0GA1UdHgEB\n/wQzMDGgLzAtgiszNmNiNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMu\nY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIHl6UDdouw8Fzn/oDHputAxt3UFbVg/U\nvC6jWPuqqMwmAiEAkvMadtwjtNU7m/AQRJrj1LeG3eXw7dWO8SlI2fEs0yY=\n-----END CERTIFICATE-----\n",
"IntermediateCerts": null,
"Active": true,
2019-09-23 17:52:35 +00:00
"PrivateKeyType": "",
"PrivateKeyBits": 0,
2018-06-06 17:50:13 +00:00
"CreateIndex": 8,
"ModifyIndex": 8
}
]
}
```
2018-06-24 21:55:26 +00:00
## CA Configuration
2018-06-06 17:50:13 +00:00
2018-06-24 21:55:26 +00:00
After initialization, the CA provider configuration can be viewed with the
2022-03-30 21:16:26 +00:00
[Get CA Configuration API endpoint](/api-docs/connect/ca#get-ca-configuration).
2018-06-24 21:55:26 +00:00
Consul will filter sensitive values from this endpoint depending on the
provider in use, so the configuration may not be complete.
2018-06-06 17:50:13 +00:00
2020-05-19 18:32:38 +00:00
```shell-session
2018-06-24 21:55:26 +00:00
$ curl http://localhost:8500/v1/connect/ca/configuration
2018-06-06 17:50:13 +00:00
{
"Provider": "consul",
"Config": {
2018-07-16 09:46:23 +00:00
"LeafCertTTL": "72h",
2020-01-17 22:27:13 +00:00
"IntermediateCertTTL": "8760h"
2018-06-06 17:50:13 +00:00
},
"CreateIndex": 5,
"ModifyIndex": 5
}
```
2018-06-24 21:55:26 +00:00
The CA provider can be reconfigured using the
2022-03-30 21:16:26 +00:00
[Update CA Configuration API endpoint](/api-docs/connect/ca#update-ca-configuration).
2018-06-24 21:55:26 +00:00
Specific options for reconfiguration can be found in the specific
CA provider documentation in the sidebar to the left.
2018-06-06 17:50:13 +00:00
2018-06-24 21:55:26 +00:00
## Root Certificate Rotation
2018-06-06 17:50:13 +00:00
Whenever the CA's configuration is updated in a way that causes the root key to
2019-11-11 21:36:22 +00:00
change, a special rotation process will be triggered in order to smoothly
transition to the new certificate. This rotation is automatically orchestrated
by Consul.
~> If the current CA Provider doesn't support cross-signing, this process can't
be followed. See [Forced Rotation Without
Cross-Signing](#forced-rotation-without-cross-signing).
2018-06-24 21:55:26 +00:00
This also automatically occurs when a completely different CA provider is
configured (since this changes the root key). Therefore, this automatic rotation
process can also be used to cleanly transition between CA providers. For example,
updating Connect to use Vault instead of the built-in CA.
2018-06-06 17:50:13 +00:00
2019-11-11 21:36:22 +00:00
During rotation, an intermediate CA certificate is requested from the new root,
which is then cross-signed by the old root. This cross-signed certificate is
then distributed alongside any newly-generated leaf certificates used by the
proxies once the new root becomes active, and provides a chain of trust back to
the old root certificate in the event that a certificate signed by the new root
is presented to a proxy that has not yet updated its bundle of trusted root CA
certificates to include the new root.
2018-06-06 17:50:13 +00:00
After the cross-signed certificate has been successfully generated and the new root
certificate or CA provider has been set up, the new root becomes the active one
and is immediately used for signing any new incoming certificate requests.
2019-11-11 21:36:22 +00:00
If we check the [list CA roots
2022-03-30 21:16:26 +00:00
endpoint](/api-docs/connect/ca#list-ca-root-certificates) after updating the
2019-11-11 21:36:22 +00:00
configuration with a new root certificate, we can see both the old and new root
certificates are present, and the currently active root has an intermediate
certificate which has been generated and cross-signed automatically by the old
root during the rotation process:
2018-06-06 17:50:13 +00:00
2020-05-19 18:32:38 +00:00
```shell-session
2018-06-06 17:50:13 +00:00
$ curl localhost:8500/v1/connect/ca/roots
{
"ActiveRootID": "d2:2c:41:94:1e:50:04:ea:86:fc:08:d6:b0:45:a4:af:8a:eb:76:a0",
"TrustDomain": "36cb52cd-4058-f811-0432-6798a240c5d3.consul",
"Roots": [
{
"ID": "31:6c:06:fb:49:94:42:d5:e4:55:cc:2e:27:b3:b2:2e:96:67:3e:7e",
"Name": "Consul CA Root Cert",
"SerialNumber": 7,
2019-09-23 17:52:35 +00:00
"SigningKeyID": "19:45:8b:30:a1:45:84:ae:23:52:db:8d:1b:ff:a9:09:db:fc:2a:72:39:ae:da:11:53:f4:37:5c:de:d1:68:d8",
"ExternalTrustDomain": "a1499528-fbf6-df7b-05e5-ae81e1873fc4",
2018-06-06 17:50:13 +00:00
"NotBefore": "2018-06-06T17:35:25Z",
"NotAfter": "2028-06-03T17:35:25Z",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIICmDCCAj6gAwIBAgIBBzAKBggqhkjOPQQDAjAWMRQwEgYDVQQDEwtDb25zdWwg\nQ0EgNzAeFw0xODA2MDYxNzM1MjVaFw0yODA2MDMxNzM1MjVaMBYxFDASBgNVBAMT\nC0NvbnN1bCBDQSA3MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEgo09lpx63bHw\ncSXeeoSpHpHgyzX1Q8ewJ3RUg6Ie8Howbs/QBz1y/kGxsF35HXij3YrqhgQyPPx4\nbQ8FH2YR4aOCAXswggF3MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/\nMGgGA1UdDgRhBF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToyMzo1MjpkYjo4ZDox\nYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1MzpmNDozNzo1Yzpk\nZTpkMTo2ODpkODBqBgNVHSMEYzBhgF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToy\nMzo1MjpkYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1\nMzpmNDozNzo1YzpkZTpkMTo2ODpkODA/BgNVHREEODA2hjRzcGlmZmU6Ly8zNmNi\nNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMuY29uc3VsMD0GA1UdHgEB\n/wQzMDGgLzAtgiszNmNiNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMu\nY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIHl6UDdouw8Fzn/oDHputAxt3UFbVg/U\nvC6jWPuqqMwmAiEAkvMadtwjtNU7m/AQRJrj1LeG3eXw7dWO8SlI2fEs0yY=\n-----END CERTIFICATE-----\n",
"IntermediateCerts": null,
"Active": false,
2019-09-23 17:52:35 +00:00
"PrivateKeyType": "",
"PrivateKeyBits": 0,
2018-06-06 17:50:13 +00:00
"CreateIndex": 8,
"ModifyIndex": 24
},
{
"ID": "d2:2c:41:94:1e:50:04:ea:86:fc:08:d6:b0:45:a4:af:8a:eb:76:a0",
"Name": "Consul CA Root Cert",
"SerialNumber": 16238269036752183483,
"SigningKeyID": "",
2019-09-23 17:52:35 +00:00
"ExternalTrustDomain": "a1499528-fbf6-df7b-05e5-ae81e1873fc4",
2018-06-06 17:50:13 +00:00
"NotBefore": "2018-06-06T17:37:03Z",
"NotAfter": "2028-06-03T17:37:03Z",
"RootCert": "-----BEGIN CERTIFICATE-----\nMIIDijCCAnKgAwIBAgIJAOFZ66em1qC7MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV\nBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRIwEAYDVQQKDAlIYXNoaUNvcnAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0x\nODA2MDYxNzM3MDNaFw0yODA2MDMxNzM3MDNaMGIxCzAJBgNVBAYTAlVTMRMwEQYD\nVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRIwEAYDVQQK\nDAlIYXNoaUNvcnAxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAK6ostXN6W093EpI3RDNQDwC1Gq3lPNoodL5XRaVVIBU\n3X5iC+Ttk02p67cHUguh4ZrWr3o3Dzxm+gKK0lfZLW0nNYNPAIGZWQD9zVSx1Lqt\n8X0pd+fhMV5coQrh3YIG/vy17IBTSBuRUX0mXOKjOeJJlrw1HQZ8pfm7WX6LFul2\nXszvgn5K1XR+9nhPy6K2bv99qsY0sm7AqCS2BjYBW8QmNngJOdLPdhyFh7invyXe\nPqgujc/KoA3P6e3/G7bJZ9+qoQMK8uwD7PxtA2hdQ9t0JGPsyWgzhwfBxWdBWRzV\nRvVi6Yu2tvw3QrjdeKQ5Ouw9FUb46VnTU7jTO974HjkCAwEAAaNDMEEwPwYDVR0R\nBDgwNoY0c3BpZmZlOi8vMzZjYjUyY2QtNDA1OC1mODExLTA0MzItNjc5OGEyNDBj\nNWQzLmNvbnN1bDANBgkqhkiG9w0BAQsFAAOCAQEATHgCro9VXj7JbH/tlB6f/KWf\n7r98+rlUE684ZRW9XcA9uUA6y265VPnemsC/EykPsririoh8My1jVPuEfgMksR39\n9eMDJKfutvSpLD1uQqZE8hu/hcYyrmQTFKjW71CfGIl/FKiAg7wXEw2ljLN9bxNv\nGG118wrJyMZrRvFjC2QKY025QQSJ6joNLFMpftsZrJlELtRV+nx3gMabpiDRXhIw\nJM6ti26P1PyVgGRPCOG10v+OuUtwe0IZoOqWpPJN8jzSuqZWf99uolkG0xuqLNz6\nd8qvTp1YF9tTmysgvdeGALez/02HTF035RVTsQfH9tM/+4yG1UnmjLpz3p4Fow==\n-----END CERTIFICATE-----",
"IntermediateCerts": [
"-----BEGIN CERTIFICATE-----\nMIIDTzCCAvWgAwIBAgIBFzAKBggqhkjOPQQDAjAWMRQwEgYDVQQDEwtDb25zdWwg\nQ0EgNzAeFw0xODA2MDYxNzM3MDNaFw0yODA2MDMxNzM3MDNaMGIxCzAJBgNVBAYT\nAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2Nv\nMRIwEAYDVQQKDAlIYXNoaUNvcnAxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJ\nKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK6ostXN6W093EpI3RDNQDwC1Gq3lPNo\nodL5XRaVVIBU3X5iC+Ttk02p67cHUguh4ZrWr3o3Dzxm+gKK0lfZLW0nNYNPAIGZ\nWQD9zVSx1Lqt8X0pd+fhMV5coQrh3YIG/vy17IBTSBuRUX0mXOKjOeJJlrw1HQZ8\npfm7WX6LFul2Xszvgn5K1XR+9nhPy6K2bv99qsY0sm7AqCS2BjYBW8QmNngJOdLP\ndhyFh7invyXePqgujc/KoA3P6e3/G7bJZ9+qoQMK8uwD7PxtA2hdQ9t0JGPsyWgz\nhwfBxWdBWRzVRvVi6Yu2tvw3QrjdeKQ5Ouw9FUb46VnTU7jTO974HjkCAwEAAaOC\nARswggEXMGgGA1UdDgRhBF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToyMzo1Mjpk\nYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1MzpmNDoz\nNzo1YzpkZTpkMTo2ODpkODBqBgNVHSMEYzBhgF8xOTo0NTo4YjozMDphMTo0NTo4\nNDphZToyMzo1MjpkYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpk\nYToxMTo1MzpmNDozNzo1YzpkZTpkMTo2ODpkODA/BgNVHREEODA2hjRzcGlmZmU6\nLy8zNmNiNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMuY29uc3VsMAoG\nCCqGSM49BAMCA0gAMEUCIBp46tRDot7GFyDXu7egq7lXBvn+UUHD5MmlFvdWmtnm\nAiEAwKBzEMcLd5kCBgFHNGyksRAMh/AGdEW859aL6z0u4gM=\n-----END CERTIFICATE-----\n"
],
"Active": true,
2019-09-23 17:52:35 +00:00
"PrivateKeyType": "",
"PrivateKeyBits": 0,
2018-06-06 17:50:13 +00:00
"CreateIndex": 24,
"ModifyIndex": 24
}
]
}
```
The old root certificate will be automatically removed once enough time has elapsed
for any leaf certificates signed by it to expire.
2019-11-11 21:36:22 +00:00
### Forced Rotation Without Cross-Signing
If the CA provider that is currently in use does not support cross-signing, then
attempts to change the root key or CA provider will fail. This is to ensure
operators don't make the change without understanding that there is additional
risk involved.
It is possible to force the change to happen anyway by setting the
`ForceWithoutCrossSigning` field in the CA configuration to `true`.
The downside is that all new certificates will immediately start being signed
with the new root key, but it will take some time for agents throughout the
cluster to observe the root CA change and reconfigure applications and proxies
to accept certificates signed by this new root. This will mean connections made
with a new certificate may fail for a short period after the CA change.
Typically all connected agents will have observed the new roots within seconds
even in a large deployment so the impact should be contained. But it is possible
for a disconnected, overloaded or misconfigured agent to not see the new root
for an unbounded amount of time during which new connections to services on that
host will fail. The issue will resolve as soon as the agent can reconnect to
servers.
Currently both Consul and Vault CA providers _do_ support cross signing. As more
providers are added this documentation will list any that this section applies
2020-04-06 20:27:35 +00:00
to.
2021-02-11 21:38:12 +00:00
### Recovering From Expired Certificates
If the built-in CA provider is misconfigured or unavailable, Consul service mesh requests eventually
stop functioning due to expiration of intermediate and root certificates. To recover manually, use the
[CLI helper](/commands/tls/ca#consul-tls-ca-create) to generate CA certificates.
#### Example - Regenerating the built in CA
```shell-session
$ consul tls ca create -cluster-id test -common-name "Consul Agent CA" -days=365 -domain consul
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem
```
The example above generates a new CA with a validity of 365 days. The cluster-id argument is specific
to each cluster and can be looked up by examining the `TrustDomain` field in
the [List CA Roots](/api-docs/connect/ca#list-ca-root-certificates) endpoint.
The contents of the generated cert and private key files from the above step should then be used with
2022-03-30 21:16:26 +00:00
the [Update CA Configuration](/api-docs/connect/ca#update-ca-configuration) endpoint. Once the CA configuration is
2021-02-11 21:38:12 +00:00
updated on the primary datacenter, all secondary datacenters will pick up the changes and regenerate their intermediate
and leaf certificates, after which any new requests that require certificate verification will succeed.
