2019-06-14 05:52:50 +00:00
|
|
|
---
|
2020-04-07 18:55:19 +00:00
|
|
|
layout: docs
|
|
|
|
|
page_title: Connect - Architecture
|
2020-04-07 23:56:08 +00:00
|
|
|
sidebar_title: 'Architecture'
|
2020-04-07 18:55:19 +00:00
|
|
|
sidebar_current: docs-connect-internals
|
|
|
|
|
description: >-
|
|
|
|
|
This page details the internals of Consul Connect: mutual TLS, agent caching
|
|
|
|
|
and performance, intention and certificate authority replication.
|
2019-06-14 05:52:50 +00:00
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# How Connect Works
|
|
|
|
|
|
|
|
|
|
This page details the inner workings of some of Connect's core features.
|
|
|
|
|
Understanding how these features work isn't a prerequisite for using Connect,
|
|
|
|
|
but will help you build a mental model of what's going on under the hood, which
|
|
|
|
|
may help you reason about Connect's behavior in more complex deployment
|
|
|
|
|
scenarios.
|
|
|
|
|
|
2020-04-16 17:56:31 +00:00
|
|
|
To try Connect locally, complete the [Getting Started with Consul service
|
|
|
|
|
mesh](https://learn.hashicorp.com/consul/gs-consul-service-mesh/understand-consul-service-mesh?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS)
|
|
|
|
|
guide.
|
|
|
|
|
|
2019-06-14 05:52:50 +00:00
|
|
|
## Mutual Transport Layer Security (mTLS)
|
|
|
|
|
|
|
|
|
|
The core of Connect is based on [mutual TLS](https://en.wikipedia.org/wiki/Mutual_authentication).
|
|
|
|
|
|
|
|
|
|
Connect provides each service with an identity encoded as a TLS certificate.
|
|
|
|
|
This certificate is used to establish and accept connections to and from other
|
|
|
|
|
services. The identity is encoded in the TLS certificate in compliance with
|
|
|
|
|
the [SPIFFE X.509 Identity Document](https://github.com/spiffe/spiffe/blob/master/standards/X509-SVID.md).
|
|
|
|
|
This enables Connect services to establish and accept connections with
|
|
|
|
|
other SPIFFE-compliant systems.
|
|
|
|
|
|
|
|
|
|
The client service verifies the destination service certificate
|
2020-04-09 23:46:54 +00:00
|
|
|
against the [public CA bundle](/api/connect/ca#list-ca-root-certificates).
|
2019-06-14 05:52:50 +00:00
|
|
|
This is very similar to a typical HTTPS web browser connection. In addition
|
|
|
|
|
to this, the client provides its own client certificate to show its
|
|
|
|
|
identity to the destination service. If the connection handshake succeeds,
|
|
|
|
|
the connection is encrypted and authorized.
|
|
|
|
|
|
|
|
|
|
The destination service verifies the client certificate
|
2020-04-09 23:46:54 +00:00
|
|
|
against the [public CA bundle](/api/connect/ca#list-ca-root-certificates).
|
2019-06-14 05:52:50 +00:00
|
|
|
After verifying the certificate, it must also call the
|
2020-04-09 23:46:54 +00:00
|
|
|
[authorization API](/api/agent/connect#authorize) to authorize
|
|
|
|
|
the connection against the configured set of Consul [intentions](/docs/connect/intentions).
|
2019-06-14 05:52:50 +00:00
|
|
|
If the authorization API responds successfully, the connection is established.
|
|
|
|
|
Otherwise, the connection is rejected.
|
|
|
|
|
|
|
|
|
|
To generate and distribute certificates, Consul has a built-in CA that
|
|
|
|
|
requires no other dependencies, and
|
2020-04-09 23:46:54 +00:00
|
|
|
also ships with built-in support for [Vault](/docs/connect/ca/vault). The PKI system is designed to be pluggable
|
2019-06-14 05:52:50 +00:00
|
|
|
and can be extended to support any system by adding additional CA providers.
|
|
|
|
|
|
|
|
|
|
All APIs required for Connect typically respond in microseconds and impose
|
2020-01-14 22:59:27 +00:00
|
|
|
minimal overhead to existing services. To ensure this, Connect-related API calls
|
2019-06-14 05:52:50 +00:00
|
|
|
are all made to the local Consul agent over a loopback interface, and all [agent
|
2020-04-09 23:46:54 +00:00
|
|
|
Connect endpoints](/api/agent/connect) implement local caching, background
|
2019-06-14 05:52:50 +00:00
|
|
|
updating, and support blocking queries. Most API calls operate on purely local
|
|
|
|
|
in-memory data.
|
|
|
|
|
|
|
|
|
|
## Agent Caching and Performance
|
|
|
|
|
|
|
|
|
|
To enable fast responses on [agent Connect API
|
2020-04-09 23:46:54 +00:00
|
|
|
endpoints](/api/agent/connect), the Consul agent locally caches most
|
2019-06-14 05:52:50 +00:00
|
|
|
Connect-related data and sets up background [blocking
|
2020-04-09 23:46:54 +00:00
|
|
|
queries](/api/features/blocking) against the server to update the cache in
|
2019-06-14 05:52:50 +00:00
|
|
|
the background. This allows most API calls such as retrieving certificates or
|
|
|
|
|
authorizing connections to use in-memory data and respond very quickly.
|
|
|
|
|
|
|
|
|
|
All data cached locally by the agent is populated on demand. Therefore, if
|
|
|
|
|
Connect is not used at all, the cache does not store any data. On first request,
|
|
|
|
|
the data is loaded from the server and cached. The set of data cached is: public
|
|
|
|
|
CA root certificates, leaf certificates, intentions, and service discovery
|
|
|
|
|
results for upstreams. For leaf certificates and intentions, only data related
|
|
|
|
|
to the service requested is cached, not the full set of data.
|
|
|
|
|
|
|
|
|
|
Further, the cache is partitioned by ACL token and datacenters. This is done
|
|
|
|
|
to minimize the complexity of the cache and prevent bugs where an ACL token
|
|
|
|
|
may see data it shouldn't from the cache. This results in higher memory usage
|
|
|
|
|
for cached data since it is duplicated per ACL token, but with the benefit
|
|
|
|
|
of simplicity and security.
|
|
|
|
|
|
|
|
|
|
With Connect enabled, you'll likely see increased memory usage by the
|
|
|
|
|
local Consul agent. The total memory is dependent on the number of intentions
|
|
|
|
|
related to the services registered with the agent accepting Connect-based
|
|
|
|
|
connections. The other data (leaf certificates and public CA certificates)
|
|
|
|
|
is a relatively fixed size per service. In most cases, the overhead per
|
|
|
|
|
service should be relatively small: single digit kilobytes at most.
|
|
|
|
|
|
|
|
|
|
The cache does not evict entries due to memory pressure. If memory capacity
|
|
|
|
|
is reached, the process will attempt to swap. If swap is disabled, the Consul
|
|
|
|
|
agent may begin failing and eventually crash. Cache entries do have TTLs
|
|
|
|
|
associated with them and will evict their entries if they're not used. Given
|
|
|
|
|
a long period of inactivity (3 days by default), the cache will empty itself.
|
|
|
|
|
|
2019-07-08 14:12:42 +00:00
|
|
|
## Connections Across Datacenters
|
2019-06-14 05:52:50 +00:00
|
|
|
|
2020-04-09 23:46:54 +00:00
|
|
|
Sidecar proxy's [upstream configuration](/docs/connect/registration/service-registration#upstream-configuration-reference)
|
2019-07-08 14:12:42 +00:00
|
|
|
may specify an alternative datacenter or a prepared query that can address services
|
|
|
|
|
in multiple datacenters (such as the [geo failover](https://learn.hashicorp.com/consul/developer-discovery/geo-failover) pattern).
|
2019-06-14 05:52:50 +00:00
|
|
|
|
2020-04-09 23:46:54 +00:00
|
|
|
[Intentions](/docs/connect/intentions) verify connections between services by
|
2019-07-08 14:12:42 +00:00
|
|
|
source and destination name seamlessly across datacenters.
|
2019-06-14 05:52:50 +00:00
|
|
|
|
2019-11-06 04:34:46 +00:00
|
|
|
Connections can be made via gateways to enable when communicating across
|
2019-07-08 14:12:42 +00:00
|
|
|
network topologies allowing connections between services in each datacenter
|
|
|
|
|
without externally routable IPs at the service level.
|
|
|
|
|
|
|
|
|
|
## Intention Replication
|
|
|
|
|
|
|
|
|
|
Intention replication happens automatically but requires the
|
2020-04-09 23:46:54 +00:00
|
|
|
[`primary_datacenter`](/docs/agent/options#primary_datacenter)
|
2019-07-08 14:12:42 +00:00
|
|
|
configuration to be set to specify a datacenter that is authoritative
|
|
|
|
|
for intentions. In production setups with ACLs enabled, the
|
2020-04-09 23:46:54 +00:00
|
|
|
[replication token](/docs/agent/options#acl_tokens_replication) must also
|
2019-07-08 14:12:42 +00:00
|
|
|
be set in the secondary datacenter server's configuration.
|
|
|
|
|
|
|
|
|
|
## Certificate Authority Federation
|
|
|
|
|
|
|
|
|
|
The primary datacenter also acts as the root Certificate Authority (CA) for Connect.
|
|
|
|
|
The primary datacenter generates a trust-domain UUID and obtains a root certificate
|
|
|
|
|
from the configured CA provider which defaults to the built-in one.
|
|
|
|
|
|
|
|
|
|
Secondary datacenters fetch the root CA public key and trust-domain ID from the
|
|
|
|
|
primary and generate their own key and Certificate Signing Request (CSR) for an
|
|
|
|
|
intermediate CA certificate. This CSR is signed by the root in the primary
|
|
|
|
|
datacenter and the certificate is returned. The secondary datacenter can now use
|
|
|
|
|
this intermediate to sign new Connect certificates in the secondary datacenter
|
|
|
|
|
without WAN communication. CA keys are never replicated between datacenters.
|
|
|
|
|
|
|
|
|
|
The secondary maintains watches on the root CA certificate in the primary. If the
|
|
|
|
|
CA root changes for any reason such as rotation or migration to a new CA, the
|
|
|
|
|
secondary automatically generates new keys and has them signed by the primary
|
|
|
|
|
datacenter's new root before initiating an automatic rotation of all issued
|
|
|
|
|
certificates in use throughout the secondary datacenter. This makes CA root key
|
|
|
|
|
rotation fully automatic and with zero downtime across multiple datacenters.
|
