318 lines
11 KiB
Plaintext
318 lines
11 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Single Consul Datacenter in Multiple Kubernetes Clusters - Kubernetes
|
|
description: Single Consul Datacenter deployed in multiple Kubernetes clusters
|
|
---
|
|
|
|
# Single Consul Datacenter in Multiple Kubernetes Clusters
|
|
|
|
-> Requires consul-helm v0.32.1 or higher.
|
|
|
|
This page describes how to deploy a single Consul datacenter in multiple Kubernetes clusters,
|
|
with both servers and clients running in one cluster, and only clients running in the rest of the clusters.
|
|
In this example, we will use two Kubernetes clusters, but this approach could be extended to using more than two.
|
|
|
|
~> **Note:** This deployment topology requires that your Kubernetes clusters have a flat network
|
|
for both pods and nodes, so that pods or nodes from one cluster can connect
|
|
to pods or nodes in another.
|
|
|
|
## Deploying Consul servers and clients in the first cluster
|
|
|
|
First, we will deploy the Consul servers with Consul clients in the first cluster.
|
|
For that, we will use the following Helm configuration:
|
|
|
|
<CodeBlockConfig filename="cluster1-config.yaml">
|
|
|
|
```yaml
|
|
global:
|
|
datacenter: dc1
|
|
tls:
|
|
enabled: true
|
|
enableAutoEncrypt: true
|
|
acls:
|
|
manageSystemACLs: true
|
|
gossipEncryption:
|
|
secretName: consul-gossip-encryption-key
|
|
secretKey: key
|
|
connectInject:
|
|
enabled: true
|
|
controller:
|
|
enabled: true
|
|
ui:
|
|
service:
|
|
type: NodePort
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
Note that we are deploying in a secure configuration, with gossip encryption,
|
|
TLS for all components, and ACLs. We are enabling the Consul Service Mesh and the controller for CRDs
|
|
so that we can use them to later verify that our services can connect with each other across clusters.
|
|
|
|
We're also setting UI's service type to be `NodePort`.
|
|
This is needed so that we can connect to servers from another cluster without using the pod IPs of the servers,
|
|
which are likely going to change.
|
|
|
|
To deploy, first we need to generate the Gossip encryption key and save it as a Kubernetes secret.
|
|
|
|
```shell
|
|
$ kubectl create secret generic consul-gossip-encryption-key --from-literal=key=$(consul keygen)
|
|
```
|
|
|
|
Now we can install our Consul cluster with Helm:
|
|
```shell
|
|
$ helm install cluster1 -f cluster1-config.yaml hashicorp/consul
|
|
```
|
|
|
|
~> **Note:** The Helm release name must be unique for each Kubernetes cluster.
|
|
That is because the Helm chart will use the Helm release name as a prefix for the
|
|
ACL resources that it creates, such as tokens and auth methods. If the names of the Helm releases
|
|
are the same, the Helm installation in subsequent clusters will clobber existing ACL resources.
|
|
|
|
Once the installation finishes and all components are running and ready,
|
|
we need to extract the gossip encryption key we've created, the CA certificate
|
|
and the ACL bootstrap token generated during installation,
|
|
so that we can apply them to our second Kubernetes cluster.
|
|
|
|
```shell
|
|
kubectl get secret consul-gossip-encryption-key cluster1-consul-ca-cert cluster1-consul-bootstrap-acl-token -o yaml > cluster1-credentials.yaml
|
|
```
|
|
|
|
## Deploying Consul clients in the second cluster
|
|
~> **Note:** If multiple Kubernetes clusters will be joined to the Consul Datacenter, then the following instructions will need to be repeated for each additional Kubernetes cluster.
|
|
|
|
Now we can switch to the second Kubernetes cluster where we will deploy only the Consul clients
|
|
that will join the first Consul cluster.
|
|
|
|
First, we need to apply credentials we've extracted from the first cluster to the second cluster:
|
|
|
|
```shell
|
|
$ kubectl apply -f cluster1-credentials.yaml
|
|
```
|
|
To deploy in the second cluster, we will use the following Helm configuration:
|
|
|
|
<CodeBlockConfig filename="cluster2-config.yaml" highlight="6-11,15-17">
|
|
|
|
```yaml
|
|
global:
|
|
enabled: false
|
|
datacenter: dc1
|
|
acls:
|
|
manageSystemACLs: true
|
|
bootstrapToken:
|
|
secretName: cluster1-consul-bootstrap-acl-token
|
|
secretKey: token
|
|
gossipEncryption:
|
|
secretName: consul-gossip-encryption-key
|
|
secretKey: key
|
|
tls:
|
|
enabled: true
|
|
enableAutoEncrypt: true
|
|
caCert:
|
|
secretName: cluster1-consul-ca-cert
|
|
secretKey: tls.crt
|
|
externalServers:
|
|
enabled: true
|
|
# This should be any node IP of the first k8s cluster
|
|
hosts: ["10.0.0.4"]
|
|
# The node port of the UI's NodePort service
|
|
httpsPort: 31557
|
|
tlsServerName: server.dc1.consul
|
|
# The address of the kube API server of this Kubernetes cluster
|
|
k8sAuthMethodHost: https://kubernetes.example.com:443
|
|
client:
|
|
enabled: true
|
|
join: ["provider=k8s kubeconfig=/consul/userconfig/cluster1-kubeconfig/kubeconfig label_selector=\"app=consul,component=server\""]
|
|
extraVolumes:
|
|
- type: secret
|
|
name: cluster1-kubeconfig
|
|
load: false
|
|
connectInject:
|
|
enabled: true
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
Note that we're referencing secrets from the first cluster in ACL, gossip, and TLS configuration.
|
|
|
|
Next, we need to set up the `externalServers` configuration.
|
|
|
|
The `externalServers.hosts` and `externalServers.httpsPort`
|
|
refer to the IP and port of the UI's NodePort service deployed in the first cluster.
|
|
Set the `externalServers.hosts` to any Node IP of the first cluster,
|
|
which you can see by running `kubectl get nodes -o wide`.
|
|
Set `externalServers.httpsPort` to the `nodePort` of the `cluster1-consul-ui` service.
|
|
In our example, the port is `31557`.
|
|
|
|
```shell
|
|
$ kubectl get service cluster1-consul-ui --context cluster1
|
|
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
|
cluster1-consul-ui NodePort 10.0.240.80 <none> 443:31557/TCP 40h
|
|
```
|
|
|
|
We set the `externalServer.tlsServerName` to `server.dc1.consul`. This the DNS SAN
|
|
(Subject Alternative Name) that is present in the Consul server's certificate.
|
|
We need to set it because we're connecting to the Consul servers over the node IP,
|
|
but that IP isn't present in the server's certificate.
|
|
To make sure that the hostname verification succeeds during the TLS handshake, we need to set the TLS
|
|
server name to a DNS name that *is* present in the certificate.
|
|
|
|
Next, we need to set `externalServers.k8sAuthMethodHost` to the address of the second Kubernetes API server.
|
|
This should be the address that is reachable from the first cluster, and so it cannot be the internal DNS
|
|
available in each Kubernetes cluster. Consul needs it so that `consul login` with the Kubernetes auth method will work
|
|
from the second cluster.
|
|
More specifically, the Consul server will need to perform the verification of the Kubernetes service account
|
|
whenever `consul login` is called, and to verify service accounts from the second cluster it needs to
|
|
reach the Kubernetes API in that cluster.
|
|
The easiest way to get it is to set it from your `kubeconfig` by running `kubectl config view` and grabbing
|
|
the value of `cluster.server` for the second cluster.
|
|
|
|
Lastly, we need to set up the clients so that they can discover the servers in the first cluster.
|
|
For this, we will use Consul's cloud auto-join feature
|
|
for the [Kubernetes provider](https://www.consul.io/docs/install/cloud-auto-join#kubernetes-k8s).
|
|
To use it we need to provide a way for the Consul clients to reach the first Kubernetes cluster.
|
|
To do that, we need to save the `kubeconfig` for the first cluster as a Kubernetes secret in the second cluster
|
|
and reference it in the `clients.join` value. Note that we're making that secret available to the client pods
|
|
by setting it in `client.extraVolumes`.
|
|
|
|
~> **Note:** The kubeconfig you're providing to the client should have minimal permissions.
|
|
The cloud auto-join provider will only need permission to read pods.
|
|
Please see [Kubernetes Cloud auto-join](https://www.consul.io/docs/install/cloud-auto-join#kubernetes-k8s)
|
|
for more details.
|
|
|
|
Now we're ready to install!
|
|
|
|
```shell
|
|
helm install cluster2 -f cluster2-config.yaml hashicorp/consul
|
|
```
|
|
|
|
## Verifying the Consul Service Mesh works
|
|
|
|
~> When Transparent proxy is enabled, services in one Kubernetes cluster that need to communicate with a service in another Kubernetes cluster must have a explicit upstream configured through the ["consul.hashicorp.com/connect-service-upstreams"](https://www.consul.io/docs/k8s/connect#consul-hashicorp-com-connect-service-upstreams) annotation.
|
|
|
|
Now that we have our Consul cluster in multiple k8s clusters up and running, we will
|
|
deploy two services and verify that they can connect to each other.
|
|
|
|
First, we'll deploy `static-server` service in the first cluster:
|
|
|
|
<CodeBlockConfig filename="static-server.yaml">
|
|
|
|
```yaml
|
|
---
|
|
apiVersion: consul.hashicorp.com/v1alpha1
|
|
kind: ServiceIntentions
|
|
metadata:
|
|
name: static-server
|
|
spec:
|
|
destination:
|
|
name: static-server
|
|
sources:
|
|
- name: static-client
|
|
action: allow
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: static-server
|
|
spec:
|
|
type: ClusterIP
|
|
selector:
|
|
app: static-server
|
|
ports:
|
|
- protocol: TCP
|
|
port: 80
|
|
targetPort: 8080
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: static-server
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: static-server
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: static-server
|
|
template:
|
|
metadata:
|
|
name: static-server
|
|
labels:
|
|
app: static-server
|
|
annotations:
|
|
"consul.hashicorp.com/connect-inject": "true"
|
|
spec:
|
|
containers:
|
|
- name: static-server
|
|
image: hashicorp/http-echo:latest
|
|
args:
|
|
- -text="hello world"
|
|
- -listen=:8080
|
|
ports:
|
|
- containerPort: 8080
|
|
name: http
|
|
serviceAccountName: static-server
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
Note that we're defining a Service intention so that our services are allowed to talk to each other.
|
|
|
|
Then we'll deploy `static-client` in the second cluster with the following configuration:
|
|
|
|
<CodeBlockConfig filename="static-client.yaml">
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: static-client
|
|
spec:
|
|
selector:
|
|
app: static-client
|
|
ports:
|
|
- port: 80
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: static-client
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: static-client
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: static-client
|
|
template:
|
|
metadata:
|
|
name: static-client
|
|
labels:
|
|
app: static-client
|
|
annotations:
|
|
"consul.hashicorp.com/connect-inject": "true"
|
|
"consul.hashicorp.com/connect-service-upstreams": "static-server:1234"
|
|
spec:
|
|
containers:
|
|
- name: static-client
|
|
image: curlimages/curl:latest
|
|
command: [ "/bin/sh", "-c", "--" ]
|
|
args: [ "while true; do sleep 30; done;" ]
|
|
serviceAccountName: static-client
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
Once both services are up and running, we can connect to the `static-server` from `static-client`:
|
|
|
|
```shell
|
|
$ kubectl exec deploy/static-client -- curl -s localhost:1234
|
|
"hello world"
|
|
```
|