562 lines
20 KiB
Plaintext
562 lines
20 KiB
Plaintext
---
|
||
layout: docs
|
||
page_title: Consul Service Mesh on Kubernetes
|
||
description: >-
|
||
Consul Service Mesh is a feature built into to Consul that enables automatic
|
||
service-to-service authorization and connection encryption across your Consul
|
||
services. Consul Service Mesh can be used with Kubernetes to secure pod communication with
|
||
other services.
|
||
---
|
||
|
||
# Consul Service Mesh on Kubernetes
|
||
|
||
[Consul Service Mesh](/docs/connect) is a feature built into to Consul that enables
|
||
automatic service-to-service authorization and connection encryption across
|
||
your Consul services. Consul Service Mesh can be used with Kubernetes to secure pod
|
||
communication with other pods and external Kubernetes services. Consul Connect is used interchangeably with the name
|
||
Consul Service Mesh and is what will be used to refer to for Service Mesh functionality within Consul.
|
||
|
||
The Connect sidecar running Envoy can be automatically injected into pods in
|
||
your cluster, making configuration for Kubernetes automatic.
|
||
This functionality is provided by the
|
||
[consul-k8s project](https://github.com/hashicorp/consul-k8s) and can be
|
||
automatically installed and configured using the
|
||
[Consul Helm chart](/docs/k8s/installation/install#helm-chart-installation).
|
||
|
||
## Usage
|
||
|
||
When the
|
||
[Connect injector is installed](/docs/k8s/connect#installation-and-configuration),
|
||
the Connect sidecar can be automatically added to all pods. This sidecar can both
|
||
accept and establish connections using Connect, enabling the pod to communicate
|
||
to clients and dependencies exclusively over authorized and encrypted
|
||
connections.
|
||
|
||
-> **Note:** The examples in this section are valid and use
|
||
publicly available images. If you've installed the Connect injector, feel free
|
||
to run the examples in this section to try Connect with Kubernetes.
|
||
Please note the documentation below this section on how to properly install
|
||
and configure the Connect injector.
|
||
|
||
### Accepting Inbound Connections
|
||
|
||
An example Deployment is shown below with Connect enabled to accept inbound
|
||
connections. Notice that the Deployment would still be fully functional without
|
||
Connect. Minimal to zero modifications are required to enable Connect in Kubernetes.
|
||
Notice also that even though we're using a Deployment here, the same configuration
|
||
would work on a Pod, a StatefulSet, or a DaemonSet.
|
||
|
||
This Deployment specification starts a server that responds to any
|
||
HTTP request with the static text "hello world".
|
||
|
||
-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes
|
||
service is **required** to run services on the Consul Service Mesh.
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
# This name will be the service name in Consul.
|
||
name: static-server
|
||
spec:
|
||
selector:
|
||
app: static-server
|
||
ports:
|
||
- protocol: TCP
|
||
port: 80
|
||
targetPort: 8080
|
||
---
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: static-server
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: static-server
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: static-server
|
||
template:
|
||
metadata:
|
||
name: static-server
|
||
labels:
|
||
app: static-server
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
spec:
|
||
containers:
|
||
- name: static-server
|
||
image: hashicorp/http-echo:latest
|
||
args:
|
||
- -text="hello world"
|
||
- -listen=:8080
|
||
ports:
|
||
- containerPort: 8080
|
||
name: http
|
||
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
|
||
serviceAccountName: static-server
|
||
```
|
||
|
||
The only change for Connect is the addition of the
|
||
`consul.hashicorp.com/connect-inject` annotation. This enables injection
|
||
for the Pod in this Deployment. The injector can also be
|
||
[configured](/docs/k8s/connect#installation-and-configuration)
|
||
to automatically inject unless explicitly disabled, but the default
|
||
installation requires opt-in using the annotation shown above.
|
||
|
||
~> **A common mistake** is to set the annotation on the Deployment or
|
||
other resource. Ensure that the injector annotations are specified on
|
||
the _pod specification template_ as shown above.
|
||
|
||
This will start a sidecar proxy that listens on port `20000` registered
|
||
with Consul and proxies valid inbound connections to port 8080 in the pod.
|
||
To establish a connection to the pod using Connect, a client must use another Connect
|
||
proxy. The client Connect proxy will use Consul service discovery to find
|
||
all available upstream proxies and their public ports.
|
||
|
||
In the example above, the server is listening on `:8080`.
|
||
By default, the Consul Service Mesh runs in [transparent proxy](/docs/connect/transparent-proxy) mode.
|
||
This means that even though the server binds to all interfaces,
|
||
the inbound and outbound connections will automatically go through to the sidecar proxy.
|
||
It also allows you to use Kubernetes DNS like you normally would without the
|
||
Consul Service Mesh.
|
||
|
||
-> **Note:** As of consul `v1.10.0`, consul-k8s `v0.26.0` and Consul Helm `v0.32.0`,
|
||
all Consul Service Mesh services will run with transparent proxy enabled by default. Running with transparent
|
||
proxy will enforce all inbound and outbound traffic to go through the Envoy proxy.
|
||
|
||
The service name registered in Consul will be set to the name of the Kubernetes service
|
||
associated with the Pod. This can be customized with the `consul.hashicorp.com/connect-service`
|
||
annotation. If using ACLs, this name must be the same as the Pod's `ServiceAccount` name.
|
||
|
||
### Connecting to Connect-Enabled Services
|
||
|
||
The example Deployment specification below configures a Deployment that is capable
|
||
of establishing connections to our previous example "static-server" service. The
|
||
connection to this static text service happens over an authorized and encrypted
|
||
connection via Connect.
|
||
|
||
-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes
|
||
Service is **required** to run services on the Consul Service Mesh.
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
# This name will be the service name in Consul.
|
||
name: static-client
|
||
spec:
|
||
selector:
|
||
app: static-client
|
||
ports:
|
||
- port: 80
|
||
---
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: static-client
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: static-client
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: static-client
|
||
template:
|
||
metadata:
|
||
name: static-client
|
||
labels:
|
||
app: static-client
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
spec:
|
||
containers:
|
||
- name: static-client
|
||
image: curlimages/curl:latest
|
||
# Just spin & wait forever, we'll use `kubectl exec` to demo
|
||
command: ['/bin/sh', '-c', '--']
|
||
args: ['while true; do sleep 30; done;']
|
||
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
|
||
serviceAccountName: static-client
|
||
```
|
||
|
||
By default when ACLs are enabled or when ACLs default policy is `allow`,
|
||
Consul will automatically configure proxies with all upstreams from the same datacenter.
|
||
When ACLs are enabled with default `deny` policy,
|
||
you must supply an [intention](/docs/connect/intentions) to tell Consul which upstream you need to talk to.
|
||
|
||
When upstreams are specified explicitly with the
|
||
[`consul.hashicorp.com/connect-service-upstreams` annotation](/docs/k8s/annotations-and-labels#consul-hashicorp-com-connect-service-upstreams),
|
||
the injector will also set environment variables `<NAME>_CONNECT_SERVICE_HOST`
|
||
and `<NAME>_CONNECT_SERVICE_PORT` in every container in the Pod for every defined
|
||
upstream. This is analogous to the standard Kubernetes service environment variables, but
|
||
point instead to the correct local proxy port to establish connections via
|
||
Connect.
|
||
|
||
We can verify access to the static text server using `kubectl exec`.
|
||
Because transparent proxy is enabled by default,
|
||
we use Kubernetes DNS to connect to our desired upstream.
|
||
|
||
```shell-session
|
||
$ kubectl exec deploy/static-client -- curl --silent http://static-server/
|
||
"hello world"
|
||
```
|
||
|
||
We can control access to the server using [intentions](/docs/connect/intentions).
|
||
If you use the Consul UI or [CLI](/commands/intention/create) to
|
||
create a deny [intention](/docs/connect/intentions) between
|
||
"static-client" and "static-server", connections are immediately rejected
|
||
without updating either of the running pods. You can then remove this
|
||
intention to allow connections again.
|
||
|
||
```shell-session
|
||
$ kubectl exec deploy/static-client -- curl --silent http://static-server/
|
||
command terminated with exit code 52
|
||
```
|
||
|
||
### Kubernetes Pods with Multiple ports
|
||
To configure a pod with multiple ports to be a part of the service mesh and receive and send service mesh traffic, you
|
||
will need to add configuration so that a Consul service can be registered per port. This is because services in Consul
|
||
currently support a single port per service instance.
|
||
|
||
In the following example, suppose we have a pod which exposes 2 ports, `8080` and `9090`, both of which will need to
|
||
receive service mesh traffic.
|
||
|
||
First, decide on the names for the two Consul services that will correspond to those ports. In this example, the user
|
||
chooses the names `web` for `8080` and `web-admin` for `9090`.
|
||
|
||
Create two service accounts for `web` and `web-admin`:
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: web
|
||
---
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: web-admin
|
||
```
|
||
Create two Service objects for `web` and `web-admin`:
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: web
|
||
spec:
|
||
selector:
|
||
app: web
|
||
ports:
|
||
- protocol: TCP
|
||
port: 80
|
||
targetPort: 8080
|
||
---
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: web-admin
|
||
spec:
|
||
selector:
|
||
app: web
|
||
ports:
|
||
- protocol: TCP
|
||
port: 80
|
||
targetPort: 9090
|
||
```
|
||
`web` will target `containerPort` `8080` and select pods labeled `app: web`. `web-admin` will target `containerPort`
|
||
`9090` and will also select the same pods.
|
||
|
||
Create a Deployment with any chosen name, and use the following annotations:
|
||
```yaml
|
||
consul.hashicorp.com/connect-inject: true
|
||
consul.hashicorp.com/transparent-proxy: false
|
||
consul.hashicorp.com/connect-service: web,web-admin
|
||
consul.hashicorp.com/connect-service-port: 8080,9090
|
||
```
|
||
Note that the order the ports are listed in the same order as the service names, i.e. the first service name `web`
|
||
corresponds to the first port, `8080`, and the second service name `web-admin` corresponds to the second port, `9090`.
|
||
|
||
The service account on the pod spec for the deployment should be set to the first service name `web`:
|
||
```yaml
|
||
serviceAccountName: web
|
||
```
|
||
|
||
For reference, the full deployment example could look something like the following:
|
||
```yaml
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: web
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: web
|
||
template:
|
||
metadata:
|
||
name: web
|
||
labels:
|
||
app: web
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
'consul.hashicorp.com/transparent-proxy': 'false'
|
||
'consul.hashicorp.com/connect-service': 'web,web-admin'
|
||
'consul.hashicorp.com/connect-service-port': '8080,9090'
|
||
spec:
|
||
containers:
|
||
- name: web
|
||
image: hashicorp/http-echo:latest
|
||
args:
|
||
- -text="hello world"
|
||
- -listen=:8080
|
||
ports:
|
||
- containerPort: 8080
|
||
name: http
|
||
- name: web-admin
|
||
image: hashicorp/http-echo:latest
|
||
args:
|
||
- -text="hello world from 9090"
|
||
- -listen=:9090
|
||
ports:
|
||
- containerPort: 9090
|
||
name: http
|
||
serviceAccountName: web
|
||
```
|
||
|
||
After deploying the `web` application, you can test service mesh connections by deploying the `static-client`
|
||
application with the configuration in the [previous section](#connecting-to-connect-enabled-services) and add the
|
||
following annotation to the pod template on `static-client`:
|
||
```yaml
|
||
consul.hashicorp.com/connect-service-upstreams: "web:1234,web-admin:2234"
|
||
```
|
||
|
||
If you exec on to a static-client pod, using a command like:
|
||
```shell-session
|
||
$ kubectl exec -it static-client-5bd667fbd6-kk6xs -- /bin/sh
|
||
```
|
||
you can then run:
|
||
```shell-session
|
||
$ curl localhost:1234
|
||
```
|
||
to see the output `hello world` and run:
|
||
```shell-session
|
||
$ curl localhost:2234
|
||
```
|
||
to see the output `hello world from 9090`.
|
||
|
||
The way this works is that a Consul service instance is being registered per port on the Pod, so there are 2 Consul
|
||
services in this case. An additional Envoy sidecar proxy and `connect-init` init container are also deployed per port in
|
||
the Pod. So the upstream configuration can use the individual service names to reach each port as seen in the example.
|
||
|
||
#### Caveats for Multi-port Pods
|
||
* Transparent proxy is not supported for multi-port Pods.
|
||
* Metrics and metrics merging is not supported for multi-port Pods.
|
||
* Upstreams will only be set on the first service's Envoy sidecar proxy for the pod.
|
||
* This means that ServiceIntentions from a multi-port pod to elsewhere, will need to use the first service's name,
|
||
`web` in the example above to accept connections from either `web` or `web-admin`. ServiceIntentions from elsewhere
|
||
to a multi-port pod can use the individual service names within the multi-port Pod.
|
||
* Health checking is done on a per-Pod basis, so if any Kubernetes health checks (like readiness, liveness, etc) are
|
||
failing for any container on the Pod, the entire Pod is marked unhealthy, and any Consul service referencing that Pod
|
||
will also be marked as unhealthy. So, if `web` has a failing health check, `web-admin` would also be marked as
|
||
unhealthy for service mesh traffic.
|
||
|
||
## Installation and Configuration
|
||
|
||
The Connect sidecar proxy is injected via a
|
||
[mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks)
|
||
provided by the
|
||
[consul-k8s project](https://github.com/hashicorp/consul-k8s).
|
||
This enables the automatic pod mutation shown in the usage section above.
|
||
Installation of the mutating admission webhook is automated using the
|
||
[Helm chart](/docs/k8s/installation/install).
|
||
|
||
To install the Connect injector, enable the Connect injection feature using
|
||
[Helm values](/docs/k8s/helm#configuration-values) and
|
||
upgrade the installation using `helm upgrade` for existing installs or
|
||
`helm install` for a fresh install.
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
|
||
controller:
|
||
enabled: true
|
||
```
|
||
|
||
This will configure the injector to inject when the
|
||
[injection annotation](#consul-hashicorp-com-connect-inject)
|
||
is set to `true`. Other values in the Helm chart can be used to limit the namespaces
|
||
the injector runs in, enable injection by default, and more.
|
||
|
||
### Verifying the Installation
|
||
|
||
To verify the installation, run the
|
||
["Accepting Inbound Connections"](/docs/k8s/connect#accepting-inbound-connections)
|
||
example from the "Usage" section above. After running this example, run
|
||
`kubectl get pod static-server --output yaml`. In the raw YAML output, you should
|
||
see injected Connect containers and an annotation
|
||
`consul.hashicorp.com/connect-inject-status` set to `injected`. This
|
||
confirms that injection is working properly.
|
||
|
||
If you do not see this, then use `kubectl logs` against the injector pod
|
||
and note any errors.
|
||
|
||
### Controlling Injection Via Annotation
|
||
|
||
By default, the injector will inject only when the
|
||
[injection annotation](#consul-hashicorp-com-connect-inject)
|
||
on the pod (not the deployment) is set to `true`:
|
||
|
||
```yaml
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
```
|
||
|
||
### Injection Defaults
|
||
|
||
If you wish for the injector to always inject, you can set the default to `true`
|
||
in the Helm chart:
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
default: true
|
||
```
|
||
|
||
You can then exclude specific pods via annotation:
|
||
|
||
```yaml
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'false'
|
||
```
|
||
|
||
### Controlling Injection Via Namespace
|
||
|
||
You can control which Kubernetes namespaces are allowed to be injected via
|
||
the `k8sAllowNamespaces` and `k8sDenyNamespaces` keys:
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
k8sAllowNamespaces: ['*']
|
||
k8sDenyNamespaces: []
|
||
```
|
||
|
||
In the default configuration (shown above), services from all namespaces are allowed
|
||
to be injected. Whether or not they're injected depends on the value of `connectInject.default`
|
||
and the `consul.hashicorp.com/connect-inject` annotation.
|
||
|
||
If you wish to only enable injection in specific namespaces, you can list only those
|
||
namespaces in the `k8sAllowNamespaces` key. In the configuration below
|
||
only the `my-ns-1` and `my-ns-2` namespaces will be enabled for injection.
|
||
All other namespaces will be ignored, even if the connect inject [annotation](#consul-hashicorp-com-connect-inject)
|
||
is set.
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
|
||
k8sDenyNamespaces: []
|
||
```
|
||
|
||
If you wish to enable injection in every namespace _except_ specific namespaces, you can
|
||
use `*` in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list:
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
k8sAllowNamespaces: ['*']
|
||
k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']
|
||
```
|
||
|
||
-> **NOTE:** The deny list takes precedence over the allow list. If a namespace
|
||
is listed in both lists, it will **not** be synced.
|
||
|
||
~> **NOTE:** The `kube-system` and `kube-public` namespaces will never be injected.
|
||
|
||
### Consul Enterprise Namespaces
|
||
|
||
Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered
|
||
into Consul, you can control which Consul namespace they are registered into.
|
||
|
||
There are three options available:
|
||
|
||
1. **Single Destination Namespace** – Register all Kubernetes pods, regardless of namespace,
|
||
into the same Consul namespace.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
connectInject:
|
||
enabled: true
|
||
consulNamespaces:
|
||
consulDestinationNamespace: 'my-consul-ns'
|
||
```
|
||
|
||
-> **NOTE:** If the destination namespace does not exist we will create it.
|
||
|
||
1. **Mirror Namespaces** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace.
|
||
For example, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `ns-1`.
|
||
If a mirrored namespace does not exist in Consul, it will be created.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
connectInject:
|
||
enabled: true
|
||
consulNamespaces:
|
||
mirroringK8S: true
|
||
```
|
||
|
||
1. **Mirror Namespaces With Prefix** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes
|
||
namespace **with a prefix**.
|
||
For example, given a prefix `k8s-`, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `k8s-ns-1`.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
connectInject:
|
||
enabled: true
|
||
consulNamespaces:
|
||
mirroringK8S: true
|
||
mirroringK8SPrefix: 'k8s-'
|
||
```
|
||
|
||
### Consul Enterprise Namespace Upstreams
|
||
|
||
When [transparent proxy](/docs/connect/transparent-proxy) is enabled and ACLs are disabled,
|
||
the upstreams will be configured automatically across Consul namespaces.
|
||
When ACLs are enabled, you must configure it by specifying an [intention](/docs/connect/intentions),
|
||
allowing services across Consul namespaces to talk to each other.
|
||
|
||
If you wish to specify an upstream explicitly via the `consul.hashicorp.com/connect-service-upstreams` annotation,
|
||
use the format `[service-name].[namespace]:[port]:[optional datacenter]`:
|
||
|
||
```yaml
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'
|
||
```
|
||
|
||
See [consul.hashicorp.com/connect-service-upstreams](#consul-hashicorp-com-connect-service-upstreams) for more details.
|
||
|
||
-> **Note:** When you specify upstreams via an upstreams annotation, you will need to use
|
||
`localhost:<port>` with the port from the upstreams annotation instead of KubeDNS to connect to your upstream
|
||
application.
|