luxolus
/
open-consul
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity
open-consul/website/content/docs/k8s/connect/index.mdx

561 lines
22 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
layout: docs
page_title: Service Mesh - Kubernetes
description: >-
Connect is a feature built into to Consul that enables automatic
service-to-service authorization and connection encryption across your Consul
services. Connect can be used with Kubernetes to secure pod communication with
other services.
---
# Connect Service Mesh on Kubernetes
[Connect](/docs/connect) is a feature built into to Consul that enables
automatic service-to-service authorization and connection encryption across
your Consul services. Connect can be used with Kubernetes to secure pod
communication with other pods and external Kubernetes services.
The Connect sidecar running Envoy can be automatically injected into pods in
your cluster, making configuration for Kubernetes automatic.
This functionality is provided by the
[consul-k8s project](https://github.com/hashicorp/consul-k8s) and can be
automatically installed and configured using the
[Consul Helm chart](/docs/k8s/installation/install).
## Usage
When the
[Connect injector is installed](/docs/k8s/connect#installation-and-configuration),
the Connect sidecar can be automatically added to all pods. This sidecar can both
accept and establish connections using Connect, enabling the pod to communicate
to clients and dependencies exclusively over authorized and encrypted
connections.
-> **Note:** The examples in this section are valid and use
publicly available images. If you've installed the Connect injector, feel free
to run the examples in this section to try Connect with Kubernetes.
Please note the documentation below this section on how to properly install
and configure the Connect injector.
### Accepting Inbound Connections
An example Deployment is shown below with Connect enabled to accept inbound
connections. Notice that the Deployment would still be fully functional without
Connect. Minimal to zero modifications are required to enable Connect in Kubernetes.
Notice also that even though we're using a Deployment here, the same configuration
would work on a Pod, a StatefulSet, or a DaemonSet.
This Deployment specification starts a server that responds to any
HTTP request with the static text "hello world".
-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes
service is **required** to run services on the Consul Service Mesh.
```yaml
apiVersion: v1
kind: Service
metadata:
# This name will be the service name in Consul.
name: static-server
spec:
selector:
app: static-server
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: static-server
spec:
replicas: 1
selector:
matchLabels:
app: static-server
template:
metadata:
name: static-server
labels:
app: static-server
annotations:
'consul.hashicorp.com/connect-inject': 'true'
spec:
containers:
- name: static-server
image: hashicorp/http-echo:latest
args:
- -text="hello world"
- -listen=:8080
ports:
- containerPort: 8080
name: http
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
serviceAccountName: static-server
```
The only change for Connect is the addition of the
`consul.hashicorp.com/connect-inject` annotation. This enables injection
for the Pod in this Deployment. The injector can also be
[configured](/docs/k8s/connect#installation-and-configuration)
to automatically inject unless explicitly disabled, but the default
installation requires opt-in using the annotation shown above.
~> **A common mistake** is to set the annotation on the Deployment or
other resource. Ensure that the injector annotations are specified on
the _pod specification template_ as shown above.
This will start a sidecar proxy that listens on port `20000` registered
with Consul and proxies valid inbound connections to port 8080 in the pod.
To establish a connection to the pod using Connect, a client must use another Connect
proxy. The client Connect proxy will use Consul service discovery to find
all available upstream proxies and their public ports.
In the example above, the server is listening on `:8080`.
By default, the Consul Service Mesh runs in [transparent proxy](/docs/connect/transparent-proxy) mode.
This means that even though the server binds to all interfaces,
the inbound and outbound connections will automatically go through to the sidecar proxy.
It also allows you to use Kubernetes DNS like you normally would without the
Consul Service Mesh.
-> **Note:** As of consul `v1.10.0`, consul-k8s `v0.26.0` and Consul Helm `v0.32.0`,
all Consul Service Mesh services will run with transparent proxy enabled by default. Running with transparent
proxy will enforce all inbound and outbound traffic to go through the Envoy proxy.
The service name registered in Consul will be set to the name of the Kubernetes service
associated with the Pod. This can be customized with the `consul.hashicorp.com/connect-service`
annotation. If using ACLs, this name must be the same as the Pod's `ServiceAccount` name.
### Connecting to Connect-Enabled Services
The example Deployment specification below configures a Deployment that is capable
of establishing connections to our previous example "static-server" service. The
connection to this static text service happens over an authorized and encrypted
connection via Connect.
-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes
Service is **required** to run services on the Consul Service Mesh.
```yaml
apiVersion: v1
kind: Service
metadata:
# This name will be the service name in Consul.
name: static-client
spec:
selector:
app: static-client
ports:
- port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: static-client
spec:
replicas: 1
selector:
matchLabels:
app: static-client
template:
metadata:
name: static-client
labels:
app: static-client
annotations:
'consul.hashicorp.com/connect-inject': 'true'
spec:
containers:
- name: static-client
image: tutum/curl:latest
# Just spin & wait forever, we'll use `kubectl exec` to demo
command: ['/bin/sh', '-c', '--']
args: ['while true; do sleep 30; done;']
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
serviceAccountName: static-client
```
By default when ACLs are enabled or when ACLs default policy is `allow`,
Consul will automatically configure proxies with all upstreams from the same datacenter.
When ACLs are enabled with default `deny` policy,
you must supply an [intention](/docs/connect/intentions) to tell Consul which upstream you need to talk to.
When upstreams are specified explicitly with the
[`consul.hashicorp.com/connect-service-upstreams` annotation](/docs/k8s/connect#consul-hashicorp-com-connect-service-upstreams),
the injector will also set environment variables `<NAME>_CONNECT_SERVICE_HOST`
and `<NAME>_CONNECT_SERVICE_PORT` in every container in the Pod for every defined
upstream. This is analogous to the standard Kubernetes service environment variables, but
point instead to the correct local proxy port to establish connections via
Connect.
We can verify access to the static text server using `kubectl exec`.
Because transparent proxy is enabled by default,
we use Kubernetes DNS to connect to our desired upstream.
```shell-session
$ kubectl exec deploy/static-client -- curl -s http://static-server/
"hello world"
```
We can control access to the server using [intentions](/docs/connect/intentions).
If you use the Consul UI or [CLI](/commands/intention/create) to
create a deny [intention](/docs/connect/intentions) between
"static-client" and "static-server", connections are immediately rejected
without updating either of the running pods. You can then remove this
intention to allow connections again.
```shell-session
$ kubectl exec deploy/static-client -- curl -s http://static-server/
command terminated with exit code 52
```
### Available Annotations
Pod annotations can be used to configure the injection behavior.
- `consul.hashicorp.com/connect-inject` - If this is "true" then injection
is enabled. If this is "false" then injection is explicitly disabled.
The default injector behavior requires pods to opt-in to injection by
specifying this value as "true". This default can be changed in the
injector's configuration if desired.
- `consul.hashicorp.com/transparent-proxy` - If this is "true", this Pod
will run with transparent proxy enabled. This means you can use Kubernetes
DNS to access upstream services and all inbound and outbound traffic within
the pod is redirected to go through the proxy.
- `consul.hashicorp.com/transparent-proxy-overwrite-probes` - If this is "true"
and transparent proxy is enabled, the Connect injector will overwrite Kubernetes
HTTP probes to point to the Envoy proxy.
- `consul.hashicorp.com/transparent-proxy-exclude-inbound-ports` - A comma-separated
list of inbound ports to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/transparent-proxy-exclude-inbound-cidrs` - A comma-separated
list of inbound CIDRs to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/transparent-proxy-exclude-outbound-ports` - A comma-separated
list of outbound ports to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/transparent-proxy-exclude-uids` - A comma-separated
list of additional user IDs to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/connect-service` - For pods that accept inbound
connections, this specifies the name of the service that is being
served. This defaults to the name of the Kubernetes service associated with the pod.
If using ACLs, this must be the same name as the Pod's `ServiceAccount`.
- `consul.hashicorp.com/connect-service-port` - For pods that accept inbound
connections, this specifies the port to route inbound connections to. This
is the port that the service is listening on. The service port defaults to
the first exposed port on any container in the pod. If specified, the value
can be the _name_ of a configured port, such as "http" or it can be a direct
port value such as "8080". This is the port of the _service_, the proxy
public listener will listen on a dynamic port.
- `consul.hashicorp.com/connect-service-upstreams` - The list of upstream
services that this pod needs to connect to via Connect along with a static
local port to listen for those connections. When transparent proxy is enabled,
this annotation is optional.
- Services
The name of the service is the name of the service registered with Consul. You can optionally specify datacenters with this annotation.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter]"
```
- Consul Enterprise Namespaces
If running Consul Enterprise 1.7+, your upstream services may be running in different
namespaces. The upstream namespace can be specified after the service name
as `[service-name].[namespace]`. See [Consul Enterprise Namespaces](#consul-enterprise-namespaces)
below for more details on configuring the injector.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].[service-namespace]:[port]:[optional datacenter]"
```
-> **NOTE:** If the namespace is not specified it will default to the namespace
of the source service.
~> **WARNING:** Setting a namespace when not using Consul Enterprise or using a version < 1.7
is not supported. It will be treated as part of the service name.
- [Prepared Query](/docs/connect/proxies#dynamic-upstreams-require-native-integration)
```yaml
annotations:
'consul.hashicorp.com/connect-service-upstreams': 'prepared_query:[query name]:[port]'
```
- Multiple Upstreams
If you would like to specify multiple services or upstreams, delimit them with commas
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],[service-name]:[port]:[optional datacenter]"
```
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],prepared_query:[query name]:[port]"
```
- `consul.hashicorp.com/envoy-extra-args` - A space-separated list of [arguments](https://www.envoyproxy.io/docs/envoy/latest/operations/cli)
to be passed to the injected envoy binary.
```yaml
annotations:
consul.hashicorp.com/envoy-extra-args: '--log-level debug --disable-hot-restart'
```
- `consul.hashicorp.com/service-tags` - A comma separated list of tags that will
be applied to the Consul service and its sidecar.
```yaml
annotations:
consul.hashicorp.com/service-tags: foo,bar,baz
```
- `consul.hashicorp.com/service-meta-<YOUR_KEY>` - Set Consul meta key/value
pairs that will be applied to the Consul service and its sidecar.
The key will be what comes after `consul.hashicorp.com/service-meta-`, e.g.
`consul.hashicorp.com/service-meta-foo: bar` will result in `foo: bar`.
```yaml
annotations:
consul.hashicorp.com/service-meta-foo: baz
consul.hashicorp.com/service-meta-bar: baz
```
- `consul.hashicorp.com/sidecar-proxy-` - Override default resource settings for
the sidecar proxy container.
The defaults are set in Helm config via the [`connectInject.sidecarProxy.resources`](/docs/k8s/helm#v-connectinject-sidecarproxy-resources) key.
- `consul.hashicorp.com/sidecar-proxy-cpu-limit` - Override the default CPU limit.
- `consul.hashicorp.com/sidecar-proxy-cpu-request` - Override the default CPU request.
- `consul.hashicorp.com/sidecar-proxy-memory-limit` - Override the default memory limit.
- `consul.hashicorp.com/sidecar-proxy-memory-request` - Override the default memory request.
- `consul.hashicorp.com/enable-metrics` - Override the default Helm value [`connectInject.metrics.defaultEnabled`](/docs/k8s/helm#v-connectinject-metrics-defaultenabled).
- `consul.hashicorp.com/enable-metrics-merging` - Override the default Helm value [`connectInject.metrics.defaultEnableMerging`](/docs/k8s/helm#v-connectinject-metrics-defaultenablemerging).
- `consul.hashicorp.com/merged-metrics-port` - Override the default Helm value [`connectInject.metrics.defaultMergedMetricsPort`](/docs/k8s/helm#v-connectinject-metrics-defaultmergedmetricsport).
- `consul.hashicorp.com/prometheus-scrape-port` - Override the default Helm value [`connectInject.metrics.defaultPrometheusScrapePort`](/docs/k8s/helm#v-connectinject-metrics-defaultprometheusscrapeport).
- `consul.hashicorp.com/prometheus-scrape-path` - Override the default Helm value [`connectInject.metrics.defaultPrometheusScrapePath`](/docs/k8s/helm#v-connectinject-metrics-defaultprometheusscrapepath).
- `consul.hashicorp.com/service-metrics-port` - Set the port where the Connect service exposes metrics.
- `consul.hashicorp.com/service-metrics-path` - Set the path where the Connect service exposes metrics.
## Installation and Configuration
The Connect sidecar proxy is injected via a
[mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks)
provided by the
[consul-k8s project](https://github.com/hashicorp/consul-k8s).
This enables the automatic pod mutation shown in the usage section above.
Installation of the mutating admission webhook is automated using the
[Helm chart](/docs/k8s/installation/install).
To install the Connect injector, enable the Connect injection feature using
[Helm values](/docs/k8s/helm#configuration-values) and
upgrade the installation using `helm upgrade` for existing installs or
`helm install` for a fresh install.
```yaml
connectInject:
enabled: true
controller:
enabled: true
```
This will configure the injector to inject when the
[injection annotation](#consul-hashicorp-com-connect-inject)
is set to `true`. Other values in the Helm chart can be used to limit the namespaces
the injector runs in, enable injection by default, and more.
### Controlling Injection Via Annotation
By default, the injector will inject only when the
[injection annotation](#consul-hashicorp-com-connect-inject)
on the pod (not the deployment) is set to `true`:
```yaml
annotations:
'consul.hashicorp.com/connect-inject': 'true'
```
### Injection Defaults
If you wish for the injector to always inject, you can set the default to `true`
in the Helm chart:
```yaml
connectInject:
enabled: true
default: true
```
You can then exclude specific pods via annotation:
```yaml
annotations:
'consul.hashicorp.com/connect-inject': 'false'
```
### Controlling Injection Via Namespace
You can control which Kubernetes namespaces are allowed to be injected via
the `k8sAllowNamespaces` and `k8sDenyNamespaces` keys:
```yaml
connectInject:
enabled: true
k8sAllowNamespaces: ['*']
k8sDenyNamespaces: []
```
In the default configuration (shown above), services from all namespaces are allowed
to be injected. Whether or not they're injected depends on the value of `connectInject.default`
and the `consul.hashicorp.com/connect-inject` annotation.
If you wish to only enable injection in specific namespaces, you can list only those
namespaces in the `k8sAllowNamespaces` key. In the configuration below
only the `my-ns-1` and `my-ns-2` namespaces will be enabled for injection.
All other namespaces will be ignored, even if the connect inject [annotation](#consul-hashicorp-com-connect-inject)
is set.
```yaml
connectInject:
enabled: true
k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
k8sDenyNamespaces: []
```
If you wish to enable injection in every namespace _except_ specific namespaces, you can
use `*` in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list:
```yaml
connectInject:
enabled: true
k8sAllowNamespaces: ['*']
k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']
```
-> **NOTE:** The deny list takes precedence over the allow list. If a namespace
is listed in both lists, it will **not** be synced.
~> **NOTE:** The `kube-system` and `kube-public` namespaces will never be injected.
### Consul Enterprise Namespaces
Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered
into Consul, you can control which Consul namespace they are registered into.
There are three options available:
1. **Single Destination Namespace** – Register all Kubernetes pods, regardless of namespace,
into the same Consul namespace.
This can be configured with:
```yaml
global:
enableConsulNamespaces: true
connectInject:
enabled: true
consulNamespaces:
consulDestinationNamespace: 'my-consul-ns'
```
-> **NOTE:** If the destination namespace does not exist we will create it.
1. **Mirror Namespaces** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace.
For example, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `ns-1`.
If a mirrored namespace does not exist in Consul, it will be created.
This can be configured with:
```yaml
global:
enableConsulNamespaces: true
connectInject:
enabled: true
consulNamespaces:
mirroringK8S: true
```
1. **Mirror Namespaces With Prefix** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes
namespace **with a prefix**.
For example, given a prefix `k8s-`, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `k8s-ns-1`.
This can be configured with:
```yaml
global:
enableConsulNamespaces: true
connectInject:
enabled: true
consulNamespaces:
mirroringK8S: true
mirroringK8SPrefix: 'k8s-'
```
### Consul Enterprise Namespace Upstreams
When [transparent proxy](/docs/connect/transparent-proxy) is enabled and ACLs are disabled,
the upstreams will be configured automatically across Consul namespaces.
When ACLs are enabled, you must configure it by specifying an [intention](/docs/connect/intentions),
allowing services across Consul namespaces to talk to each other.
If you wish to specify an upstream explicitly via the `consul.hashicorp.com/connect-service-upstreams` annotation,
use the format `[service-name].[namespace]:[port]:[optional datacenter]`:
```yaml
annotations:
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'
```
See [consul.hashicorp.com/connect-service-upstreams](#consul-hashicorp-com-connect-service-upstreams) for more details.
-> **Note:** When you specify upstreams via an upstreams annotation, you will need to use
`localhost:<port>` with the port from the upstreams annotation instead of KubeDNS to connect to your upstream
application.
### Verifying the Installation
To verify the installation, run the
["Accepting Inbound Connections"](/docs/k8s/connect#accepting-inbound-connections)
example from the "Usage" section above. After running this example, run
`kubectl get pod static-server -o yaml`. In the raw YAML output, you should
see injected Connect containers and an annotation
`consul.hashicorp.com/connect-inject-status` set to `injected`. This
confirms that injection is working properly.
If you do not see this, then use `kubectl logs` against the injector pod
and note any errors.
Reference in New Issue View Git Blame Copy Permalink