438 lines
14 KiB
Plaintext
438 lines
14 KiB
Plaintext
---
|
||
layout: docs
|
||
page_title: Connecting Services Across Datacenters
|
||
description: |-
|
||
Connect services and secure inter-service communication across datacenters
|
||
using Consul Connect and mesh gateways.
|
||
---
|
||
|
||
## Introduction
|
||
|
||
Consul Connect is Consul’s service mesh offering, which allows users to observe
|
||
and secure service-to-service communication. Because Connect implements mutual
|
||
TLS between services, it also enabled us to build mesh gateways, which provide
|
||
users with a way to help services in different datacenters communicate with each
|
||
other. Mesh gateways take advantage of Server Name Indication (SNI), which is an
|
||
extension to TLS that allows them to see the destination of inter-datacenter
|
||
traffic without decrypting the message payload.
|
||
|
||
Using mesh gateways for inter-datacenter communication can prevent each Connect
|
||
proxy from needing an accessible IP address, and frees operators from worrying
|
||
about IP address overlap between datacenters.
|
||
|
||
In this guide, you will configure Consul Connect across multiple Consul
|
||
datacenters and use mesh gateways to enable inter-service traffic between them.
|
||
|
||
Specifically, you will:
|
||
|
||
1. Enable Connect in both datacenters
|
||
1. Deploy the two mesh gateways
|
||
1. Register services and Connect sidecar proxies
|
||
1. Configure intentions
|
||
1. Test that your services can communicate with each other
|
||
|
||
For the remainder of this guide we will refer to mesh gateways as "gateways".
|
||
Anywhere in this guide where you see the word gateway, assume it is specifically
|
||
a mesh gateway (as opposed to an API or other type of gateway).
|
||
|
||
## Prerequisites
|
||
|
||
To complete this guide you will need two wide area network (WAN) joined Consul
|
||
datacenters with access control list (ACL) replication enabled. If you are
|
||
starting from scratch, follow these guides to set up your datacenters, or use
|
||
them to check that you have the proper configuration:
|
||
|
||
- [Deployment Guide](/consul/datacenter-deploy/deployment-guide)
|
||
- [Securing Consul with ACLs](/consul/security-networking/production-acls)
|
||
- [Basic Federation with WAN Gossip](/consul/security-networking/datacenters)
|
||
|
||
You will also need to enable ACL replication, which you can do by following the
|
||
[ACL Replication for Multiple
|
||
Datacenters](/consul/day-2-operations/acl-replication) guide with the following
|
||
modification.
|
||
|
||
When creating the [replication token for ACL
|
||
management](/consul/day-2-operations/acl-replication#create-the-replication-token-for-acl-management),
|
||
it will need the following policy:
|
||
|
||
```json
|
||
{
|
||
"acl": "write",
|
||
"operator": "write",
|
||
"service_prefix": {
|
||
"": {
|
||
"policy": "read"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
The replication token needs different permissions depending on what you want to
|
||
accomplish. The above policy allows for ACL policy, role, and token replication
|
||
with `acl:write`, CA replication with `operator:write` and intention and
|
||
configuration entry replication with `service:*:read`.
|
||
|
||
You will also need to install [Envoy](https://www.envoyproxy.io/) alongside your
|
||
Consul clients. Both the gateway and sidecar proxies will need to get
|
||
configuration and updates from a local Consul client.
|
||
|
||
Lastly you should set [`enable_central_service_config = true`](/docs/agent/options#enable_central_service_config)
|
||
on your Consul clients, which will allow them to centrally configrure the
|
||
sidecar and mesh gateway proxies.
|
||
|
||
## Enable Connect in Both Datacenters
|
||
|
||
Once you have your datacenters set up and ACL replication configured, it’s time
|
||
to enable Connect in each of them sequentially. Connect’s certificate authority
|
||
(which is distinct from the Consul certificate authority that you manage using
|
||
the CLI) will automatically bootstrap as soon as a server with Connect enabled
|
||
becomes the server cluster’s leader. You can also use [Vault as a Connect
|
||
CA](/docs/connect/ca/vault).
|
||
|
||
!> **Warning:** If you are using this guide as a production playbook, we
|
||
strongly recommend that you enable Connect in each of your datacenters by
|
||
following the [Connect in Production
|
||
guide](/consul/developer-segmentation/connect-production),
|
||
which includes production security recommendations.
|
||
|
||
### Enable Connect in the primary datacenter
|
||
|
||
Enable Connect in the primary data center and bootstrap the Connect CA by adding
|
||
the following snippet to the server configuration for each of your servers.
|
||
|
||
```json
|
||
connect {
|
||
"enabled": true
|
||
}
|
||
```
|
||
|
||
Load the new configuration by restarting each server one at a time, making sure
|
||
to maintain quorum. This will be a similar process to performing a [rolling
|
||
restart during
|
||
upgrades](/docs/upgrading#standard-upgrades).
|
||
|
||
Stop the first server by running the following [leave
|
||
command](/docs/commands/leave).
|
||
|
||
```shell-session
|
||
$ consul leave
|
||
```
|
||
|
||
Once the server shuts down restart it and make sure that it is healthy and
|
||
rejoins the other servers. Repeat this process until you've restarted all the
|
||
servers with Connect enabled.
|
||
|
||
### Enable Connect in the secondary datacenter
|
||
|
||
Once Connect is enabled in the primary datacenter, follow the same process to
|
||
enable Connect in the secondary datacenter. Add the following configuration to
|
||
the configuration for your servers, and restart them one at a time, making sure
|
||
to maintain quorum.
|
||
|
||
```json
|
||
connect {
|
||
"enabled": true
|
||
}
|
||
```
|
||
|
||
The `primary_datacenter` setting that was required in order to enable ACL
|
||
replication between datacenters also specifies which datacenter will write
|
||
intentions and act as the [root CA for Connect](/docs/connect/connect-internals#connections-across-datacenters).
|
||
Intentions, which allow or deny inter-service communication, are automatically
|
||
replicated to the secondary datacenter.
|
||
|
||
## Deploy Gateways
|
||
|
||
Connect mesh gateways proxy requests from services in one datacenter to services
|
||
in another, so you will need to deploy your gateways on nodes that can reach
|
||
each other over the network. As we mentioned in the prerequisites,
|
||
you will need to make sure that both Envoy and Consul are installed on the
|
||
gateway nodes. You won’t want to run any services on these nodes other than
|
||
Consul and Envoy because they necessarily will have access to the WAN.
|
||
|
||
### Generate Tokens for the Gateways
|
||
|
||
You’ll need to [generate a
|
||
token](/consul/security-networking/production-acls#apply-individual-tokens-to-the-services)
|
||
for each gateway that gives it read access to the entire catalog.
|
||
|
||
Create a file named `mesh-gateway-policy.json` containing the following content.
|
||
|
||
```json
|
||
{
|
||
"node_prefix": {
|
||
"": {
|
||
"policy": "read"
|
||
}
|
||
}
|
||
}
|
||
{
|
||
"service_prefix": {
|
||
"": {
|
||
"policy": "read"
|
||
}
|
||
}
|
||
}
|
||
{
|
||
"service": {
|
||
"mesh-gateway": {
|
||
"policy": "write"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Next, create and name a new ACL policy using the file you just made.
|
||
|
||
```shell-session
|
||
$ consul acl policy create \
|
||
-name mesh-gateway \
|
||
-rules @mesh-gateway-policy.json
|
||
```
|
||
|
||
Generate a token for each gateway from the new policy.
|
||
|
||
```shell-session
|
||
$ consul acl token create -description "mesh-gateway primary datacenter token" \
|
||
-policy-name mesh-gateway
|
||
```
|
||
|
||
```shell-session
|
||
$ consul acl token create \
|
||
-description "mesh-gateway secondary datacenter token" \
|
||
-policy-name mesh-gateway
|
||
```
|
||
|
||
You’ll apply those tokens when you deploy the gateways.
|
||
|
||
### Deploy the Gateway for your primary datacenter
|
||
|
||
Register and start the gateway in your primary datacenter with the following
|
||
command.
|
||
|
||
```shell-session
|
||
$ consul connect envoy -mesh-gateway -register \
|
||
-service-name "gateway-primary"
|
||
-address "<your private address>" \
|
||
-wan-address "<your externally accessible address>"\
|
||
-token=<token for the primary dc gateway>
|
||
```
|
||
|
||
### Deploy the Gateway for your Secondary Datacenter
|
||
|
||
Register and start the gateway in your secondary datacenter with the following
|
||
command.
|
||
|
||
```shell-session
|
||
$ consul connect envoy -mesh-gateway -register \
|
||
-service-name "gateway-secondary"
|
||
-address "<your private address>" \
|
||
-wan-address "<your externally accessible address>"\
|
||
-token=<token for the secondary dc gateway>
|
||
```
|
||
|
||
### Configure Sidecar Proxies to use Gateways
|
||
|
||
Next, create a [centralized
|
||
configuration](/docs/agent/config_entries/proxy-defaults)
|
||
file for all the sidecar proxies in both datacenters called
|
||
`proxy-defaults.json`. This file will instruct the sidecar proxies to send all
|
||
their inter-datacenter traffic through the gateways. It should contain the
|
||
following:
|
||
|
||
```json
|
||
{
|
||
"Kind": "proxy-defaults",
|
||
"Name": "global",
|
||
"MeshGateway": "local"
|
||
}
|
||
```
|
||
|
||
Write the centralized configuration you just created with the following command.
|
||
|
||
```shell-session
|
||
$ consul config write proxy-defaults.json
|
||
```
|
||
|
||
Once this step is complete, you will have set up Consul Connect with gateways
|
||
across multiple datacenters. Now you are ready to register the services that
|
||
will use Connect.
|
||
|
||
## Register a Service in Each Datacenter to Use Connect
|
||
|
||
You can register a service to use a sidecar proxy by including a sidecar proxy
|
||
stanza in its registration file. For this guide, you can use socat to act as a
|
||
backend service and register a dummy service called web to represent the client
|
||
service. Those names are used in our examples. If you have services that you
|
||
would like to connect, feel free to use those instead.
|
||
|
||
~> **Caution:** Connect takes its default intention policy from Consul’s default
|
||
ACL policy. If you have set your default ACL policy to deny (as is recommended
|
||
for secure operation) and are adding Connect to already registered services,
|
||
those services may lose connection to each other until you set an intention
|
||
between them to allow communication.
|
||
|
||
### Register a back end service in one datacenter
|
||
|
||
In one datacenter register a backend service and add an Envoy sidecar proxy
|
||
registration. To do this you will either create a new registration file or edit
|
||
an existing one to include a sidecar proxy stanza. If you are using socat as
|
||
your backend service, you will create a new file called `socat.json` that will
|
||
contain the below snippet. Since you have ACLs enabled, you will have to [create
|
||
a token for the
|
||
service](/consul/security-networking/production-acls#apply-individual-tokens-to-the-services).
|
||
|
||
```json
|
||
{
|
||
"service": {
|
||
"name": "socat",
|
||
"port": 8181,
|
||
"token": "<token here>",
|
||
"connect": { "sidecar_service": {} }
|
||
}
|
||
}
|
||
```
|
||
|
||
Note the Connect stanza of the registration with the `sidecar_service` and
|
||
`token` options. This is what you would add to an existing service registration
|
||
if you are not using socat as an example.
|
||
|
||
Reload the client with the new or modified registration.
|
||
|
||
```shell-session
|
||
$ consul reload
|
||
```
|
||
|
||
Then start Envoy specifying which service it will proxy.
|
||
|
||
```shell-session
|
||
$ consul connect envoy -sidecar-for socat
|
||
```
|
||
|
||
If you are using socat as your example, start it now on the port you specified
|
||
in your registration by running the following command.
|
||
|
||
```shell-session
|
||
$ socat -v tcp-l:8181,fork exec:"/bin/cat"
|
||
```
|
||
|
||
Check that the socat service is running by accessing it using netcat on the same
|
||
node. It will echo back anything you type.
|
||
|
||
```shell-session
|
||
$ nc 127.0.0.1 8181
|
||
hello
|
||
hello
|
||
echo
|
||
echo
|
||
```
|
||
|
||
Stop the running netcat service by typing `ctrl + c`.
|
||
|
||
### Register a front end service in the other datacenter
|
||
|
||
Now in your other datacenter, you will register a service (with a sidecar proxy)
|
||
that calls your backend service. Your registration will need to list the backend
|
||
service as your upstream. Like the backend service, you can use an example
|
||
service, which will be called web, or append the connect stanza to an existing
|
||
registration with some customization.
|
||
|
||
To use web as your front end service, create a registration file called
|
||
`web.json` that contains the following snippet.
|
||
|
||
```json
|
||
{
|
||
"service": {
|
||
"name": "web",
|
||
"port": 8080,
|
||
"token": "<token here>",
|
||
"connect": {
|
||
"sidecar_service": {
|
||
"proxy": {
|
||
"upstreams": [
|
||
{
|
||
"destination_name": "socat",
|
||
"datacenter": "primary",
|
||
"local_bind_port": 8181
|
||
}
|
||
]
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Note the Connect part of the registration, which specifies socat as an
|
||
upstream. If you are using another service as a back end, replace `socat` with
|
||
its name and the `8181` with its port.
|
||
|
||
Reload the client with the new or modified registration.
|
||
|
||
```shell-session
|
||
$ consul reload
|
||
```
|
||
|
||
Then start Envoy and specify which service it will proxy.
|
||
|
||
```shell-session
|
||
$ consul connect envoy -sidecar-for web
|
||
```
|
||
|
||
## Configure Intentions to Allow Communication Between Services
|
||
|
||
Now that your services both use Connect, you will need to configure intentions
|
||
in order for them to communicate with each other. Add an intention to allow the
|
||
front end service to access the back end service. For web and socat the command
|
||
would look like this.
|
||
|
||
```shell-session
|
||
$ consul intention create web socat
|
||
```
|
||
|
||
Consul will automatically forward intentions initiated in the in the secondary
|
||
datacenter to the primary datacenter, where the servers will write them. The
|
||
servers in the primary datacenter will then automatically replicate the written
|
||
intentions back to the secondary datacenter.
|
||
|
||
## Test the connection
|
||
|
||
Now that you have services using Connect, verify that they can contact each
|
||
other. If you have been using the example web and socat services, from the node
|
||
and datacenter where you registered the web service, start netcat and type
|
||
something for it to echo.
|
||
|
||
```shell-session
|
||
$ nc 127.0.0.1 8181
|
||
hello
|
||
hello
|
||
echo
|
||
echo
|
||
```
|
||
|
||
## Summary
|
||
|
||
In this guide you configured two WAN-joined datacenters to use Consul Connect,
|
||
deployed gateways in each datacenter, and connected two services to each other
|
||
across datacenters.
|
||
|
||
Gateways know where to route traffic because of Server Name Indication (SNI)
|
||
where the client service sends the destination as part of the TLS handshake.
|
||
Because gateways rely on TLS to discover the traffic’s destination, they require
|
||
Consul Connect to route traffic.
|
||
|
||
### Next Steps
|
||
|
||
Now that you’ve seen how to deploy gateways to proxy inter-datacenter traffic,
|
||
you can deploy multiple gateways for redundancy or availability. The gateways
|
||
and proxies will automatically round-robin load balance traffic between the
|
||
gateways.
|
||
|
||
If you are using Kubernetes you can configure Connect and deploy gateways for
|
||
your Kubernetes cluster using the Helm chart. Learn more in the [Consul’s
|
||
Kubernetes documentation](/docs/platform/k8s/helm)
|
||
|
||
Visit the Consul documentation for a full list of configurations for [Consul
|
||
Connect](/docs/connect), including [mesh gateway
|
||
configuration options](/docs/connect/mesh-gateway).
|