luxolus
/
open-consul
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity
open-consul/website/source/docs/platform/k8s/run.html.md

421 lines
17 KiB
Markdown
Raw Blame History

---
layout: "docs"
page_title: "Installing Consul on Kubernetes - Kubernetes"
sidebar_current: "docs-platform-k8s-run"
description: |-
Consul can run directly on Kubernetes, both in server or client mode. For pure-Kubernetes workloads, this enables Consul to also exist purely within Kubernetes. For heterogeneous workloads, Consul agents can join a server running inside or outside of Kubernetes.
---
# Installing Consul on Kubernetes
Consul can run directly on Kubernetes, both in server or client mode.
For pure-Kubernetes workloads, this enables Consul to also exist purely
within Kubernetes. For heterogeneous workloads, Consul agents can join
a server running inside or outside of Kubernetes.
This page starts with a large how-to section for various specific tasks.
To learn more about the general architecture of Consul on Kubernetes, scroll
down to the [architecture](/docs/platform/k8s/run.html#architecture) section. If you would like to get hands-on experience testing Consul on Kubernetes, try the step-by-step beginner tutorial with an accompanying video in the [Minikube with Consul guide](https://learn.hashicorp.com/consul/getting-started-k8s/minikube?utm_source=consul.io&utm_medium=docs)
## Helm Chart Installation
The recommended way to run Consul on Kubernetes is via the
[Helm chart](/docs/platform/k8s/helm.html). This will install and configure
all the necessary components to run Consul. The configuration enables you
to run just a server cluster, just a client cluster, or both. Using the Helm
chart, you can have a full Consul deployment up and running in minutes.
A step-by-step beginner tutorial and accompanying video can be found at the
[Minikube with Consul guide](https://learn.hashicorp.com/consul/getting-started-k8s/minikube?utm_source=consul.io&utm_medium=docs).
While the Helm chart exposes dozens of useful configurations and automatically
sets up complex resources, it **does not automatically operate Consul.**
You are still responsible for learning how to monitor, backup,
upgrade, etc. the Consul cluster.
The Helm chart has no required configuration and will install a Consul
cluster with sane defaults out of the box. Prior to going to production,
it is highly recommended that you
[learn about the configuration options](/docs/platform/k8s/helm.html#configuration-values-).
~> **Security Warning:** By default, the chart will install an insecure configuration
of Consul. This provides a less complicated out-of-box experience for new users,
but is not appropriate for a production setup. It is highly recommended to use
a properly secured Kubernetes cluster or make sure that you understand and enable
the [recommended security features](/docs/internals/security.html). Currently,
some of these features are not supported in the Helm chart and require additional
manual configuration.
### Prerequisites
The Consul Helm chart works with Helm 2 and Helm 3. If using Helm 2, you will
need to install Tiller by following the
[Helm 2 Installation Guide](https://v2.helm.sh/docs/using_helm/#quickstart-guide).
### Installing Consul
Determine the latest version of the Consul Helm chart
by visiting [https://github.com/hashicorp/consul-helm/releases](https://github.com/hashicorp/consul-helm/releases).
Clone the chart at that version. For example, if the latest version is
`v0.8.1`, you would run:
```bash
$ git clone --single-branch --branch v0.8.1 https://github.com/hashicorp/consul-helm.git
Cloning into 'consul-helm'...
...
You are in 'detached HEAD' state...
```
Ensure you've checked out the correct version with `helm inspect chart`:
```bash
$ helm inspect chart ./consul-helm
apiVersion: v1
description: Install and configure Consul on Kubernetes.
home: https://www.consul.io
name: consul
sources:
- https://github.com/hashicorp/consul
- https://github.com/hashicorp/consul-helm
- https://github.com/hashicorp/consul-k8s
version: 0.8.1
```
Now you're ready to install Consul! To install Consul with the default
configuration using Helm 3 run:
```sh
$ helm install hashicorp ./consul-helm
NAME: hashicorp
...
```
-> If using Helm 2, run: `helm install --name hashicorp ./consul-helm`
_That's it._ The Helm chart does everything to setup a recommended
Consul-on-Kubernetes deployment.
In a couple minutes, a Consul cluster will be formed and a leader
elected and every node will have a running Consul agent.
### Customizing Your Installation
If you want to customize your installation,
create a `config.yaml` file to override the default settings.
You can learn what settings are available by running `helm inspect values ./consul-helm`
or by reading the [Helm Chart Reference](/docs/platform/k8s/helm.html).
Once you've created your `config.yaml` file, run `helm install` with the `-f` flag:
```bash
$ helm install hashicorp ./consul-helm -f config.yaml
```
If you've already installed Consul and want to make changes, you'll need to run
`helm upgrade`. See the [Upgrading Consul on Kubernetes](/docs/platform/k8s/run.html#upgrading-consul-on-kubernetes)
section for more details.
## Uninstalling Consul
Consul can be uninstalled via the `helm delete` command:
```bash
$ helm delete hashicorp
release "hashicorp" uninstalled
```
-> If using Helm 2, run `helm delete --purge hashicorp`
After deleting the Helm release, you need to delete the `PersistentVolumeClaim`'s
for the persistent volumes that store Consul's data. These are not deleted by Helm due to a [bug](https://github.com/helm/helm/issues/5156).
To delete, run:
```bash
$ kubectl get pvc -l chart=consul-helm
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-default-hashicorp-consul-server-0 Bound pvc-32cb296b-1213-11ea-b6f0-42010a8001db 10Gi RWO standard 17m
data-default-hashicorp-consul-server-1 Bound pvc-32d79919-1213-11ea-b6f0-42010a8001db 10Gi RWO standard 17m
data-default-hashicorp-consul-server-2 Bound pvc-331581ea-1213-11ea-b6f0-42010a8001db 10Gi RWO standard 17m
$ kubectl delete pvc -l chart=consul-helm
persistentvolumeclaim "data-default-hashicorp-consul-server-0" deleted
persistentvolumeclaim "data-default-hashicorp-consul-server-1" deleted
persistentvolumeclaim "data-default-hashicorp-consul-server-2" deleted
```
## Viewing the Consul UI
The Consul UI is enabled by default when using the Helm chart.
For security reasons, it isn't exposed via a `LoadBalancer` Service by default so you must
use `kubectl port-forward` to visit the UI:
```
$ kubectl port-forward service/hashicorp-consul-server 8500:8500
...
```
Once the port is forwarded navigate to [http://localhost:8500](http://localhost:8500).
If you want to expose the UI via a Kubernetes Service, configure
the [`ui.service` chart values](/docs/platform/k8s/helm.html#v-ui-service).
This service will allow requests to the Consul servers so it should
not be open to the world.
## Joining an Existing Consul Cluster
If you have a Consul cluster already running, you can configure your
Kubernetes nodes to join this existing cluster.
```yaml
# config.yaml
global:
enabled: false
client:
enabled: true
join:
- "provider=my-cloud config=val ..."
```
The `config.yaml` file to configure the Helm chart sets the proper
configuration to join an existing cluster.
The `global.enabled` value first disables all chart components by default
so that each component is opt-in. This allows us to _only_ setup the client
agents. We then opt-in to the client agents by setting `client.enabled` to
`true`.
Next, `client.join` is set to an array of valid
[`-retry-join` values](/docs/agent/options.html#retry-join). In the
example above, a fake [cloud auto-join](/docs/agent/cloud-auto-join.html)
value is specified. This should be set to resolve to the proper addresses of
your existing Consul cluster.
-> **Networking:** Note that for the Kubernetes nodes to join an existing
cluster, the nodes (and specifically the agent pods) must be able to connect
to all other server and client agents inside and _outside_ of Kubernetes.
If this isn't possible, consider running the Kubernetes agents as a separate
DC or adopting Enterprise for
[network segments](/docs/enterprise/network-segments/index.html).
## Accessing the Consul HTTP API
The Consul HTTP API should be accessed by communicating to the local agent
running on the same node. While technically any listening agent (client or
server) can respond to the HTTP API, communicating with the local agent
has important caching behavior, and allows you to use the simpler
[`/agent` endpoints for services and checks](/api/agent.html).
For Consul installed via the Helm chart, a client agent is installed on
each Kubernetes node. This is explained in the [architecture](/docs/platform/k8s/run.html#client-agents)
section. To access the agent, you may use the
[downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/).
An example pod specification is shown below. In addition to pods, anything
with a pod template can also access the downward API and can therefore also
access Consul: StatefulSets, Deployments, Jobs, etc.
```yaml
apiVersion: v1
kind: Pod
metadata:
name: consul-example
spec:
containers:
- name: example
image: "consul:latest"
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
command:
- "/bin/sh"
- "-ec"
- |
export CONSUL_HTTP_ADDR="${HOST_IP}:8500"
consul kv put hello world
restartPolicy: Never
```
An example `Deployment` is also shown below to show how the host IP can
be accessed from nested pod specifications:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: consul-example-deployment
spec:
replicas: 1
selector:
matchLabels:
app: consul-example
template:
metadata:
labels:
app: consul-example
spec:
containers:
- name: example
image: "consul:latest"
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
command:
- "/bin/sh"
- "-ec"
- |
export CONSUL_HTTP_ADDR="${HOST_IP}:8500"
consul kv put hello world
```
## Upgrading Consul on Kubernetes
To upgrade Consul on Kubernetes, we follow the same pattern as
[generally upgrading Consul](/docs/upgrading.html), except we can use
the Helm chart to step through a rolling deploy. It is important to understand
how to [generally upgrade Consul](/docs/upgrading.html) before reading this
section.
Upgrading Consul on Kubernetes will follow the same pattern: each server
will be updated one-by-one. After that is successful, the clients will
be updated in batches.
### Upgrading Consul Servers
To initiate the upgrade, change the `server.image` value to the
desired Consul version. For illustrative purposes, the example below will
use `consul:123.456`. Also set the `server.updatePartition` value
_equal to the number of server replicas_:
```yaml
server:
image: "consul:123.456"
replicas: 3
updatePartition: 3
```
The `updatePartition` value controls how many instances of the server
cluster are updated. Only instances with an index _greater than_ the
`updatePartition` value are updated (zero-indexed). Therefore, by setting
it equal to replicas, none should update yet.
Next, run the upgrade. You should run this with `--dry-run` first to verify
the changes that will be sent to the Kubernetes cluster.
```
$ helm upgrade consul ./
...
```
This should cause no changes (although the resource will be updated). If
everything is stable, begin by decreasing the `updatePartition` value by one,
and running `helm upgrade` again. This should cause the first Consul server
to be stopped and restarted with the new image.
Wait until the Consul server cluster is healthy again (30s to a few minutes)
then decrease `updatePartition` and upgrade again. Continue until
`updatePartition` is `0`. At this point, you may remove the
`updatePartition` configuration. Your server upgrade is complete.
### Upgrading Consul Clients
With the servers upgraded, it is time to upgrade the clients. To upgrade
the clients, set the `client.image` value to the desired Consul version.
Then, run `helm upgrade`. This will upgrade the clients in batches, waiting
until the clients come up healthy before continuing.
### Using Existing Persistent Volume Claims (PVCs)
The only way to use a pre-created PVC is to name them in the format Kubernetes expects:
```
data-<kubernetes namespace>-<helm release name>-consul-server-<ordinal>
```
The Kubernetes namespace you are installing into, helm release name, and ordinal
must match between your Consul servers and your pre-created PVCs. You only
need as many PVCs as you have Consul servers. For example, given a Kubernetes
namespace of "vault" and a release name of "consul" and 5 servers, you would need
to create PVCs with the following names:
```
data-vault-consul-consul-server-0
data-vault-consul-consul-server-1
data-vault-consul-consul-server-2
data-vault-consul-consul-server-3
data-vault-consul-consul-server-4
```
If you are using your own storage, you'll need to configure a storage class. See the
documentation for configuring storage classes [here](https://kubernetes.io/docs/concepts/storage/storage-classes/).
## Architecture
We recommend running Consul on Kubernetes with the same
[general architecture](/docs/internals/architecture.html)
as running it anywhere else. There are some benefits Kubernetes can provide
that eases operating a Consul cluster and we document those below. The standard
[production deployment guide](https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide) is still an
important read even if running Consul within Kubernetes.
Each section below will outline the different components of running Consul
on Kubernetes and an overview of the resources that are used within the
Kubernetes cluster.
### Server Agents
The server agents are run as a **StatefulSet**, using persistent volume
claims to store the server state. This also ensures that the
[node ID](/docs/agent/options.html#_node_id) is persisted so that servers
can be rescheduled onto new IP addresses without causing issues. The server agents
are configured with
[anti-affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity)
rules so that they are placed on different nodes. A readiness probe is
configured that marks the pod as ready only when it has established a leader.
A **Service** is registered to represent the servers and expose the various
ports. The DNS address of this service is used to join the servers to each
other without requiring any other access to the Kubernetes cluster. The
service is configured to publish non-ready endpoints so that it can be used
for joining during bootstrap and upgrades.
Additionally, a **PodDisruptionBudget** is configured so the Consul server
cluster maintains quorum during voluntary operational events. The maximum
unavailable is `(n/2)-1` where `n` is the number of server agents.
-> **Note:** Kubernetes and Helm do not delete Persistent Volumes or Persistent
Volume Claims when a
[StatefulSet is deleted](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-storage),
so this must done manually when removing servers.
### Client Agents
The client agents are run as a **DaemonSet**. This places one agent
(within its own pod) on each Kubernetes node.
The clients expose the Consul HTTP API via a static port (default 8500)
bound to the host port. This enables all other pods on the node to connect
to the node-local agent using the host IP that can be retrieved via the
Kubernetes downward API. See
[accessing the Consul HTTP API](/docs/platform/k8s/run.html#accessing-the-consul-http-api)
for an example.
There is a major limitation to this: there is no way to bind to a local-only
host port. Therefore, any other node can connect to the agent. This should be
considered for security. For a properly production-secured agent with TLS
and ACLs, this is safe.
Some people prefer to run **Consul agent per pod** architectures, since this
makes it easy to register the pod as a service. However, this turns
a pod into a "node" in Consul and also causes an explosion of resource usage
since every pod needs a Consul agent. We recommend instead running an
agent (in a dedicated pod) per node, via the DaemonSet. This maintains the
node equivalence in Consul. Service registration should be handled via the
catalog syncing feature with Services rather than pods.
-> **Note:** Due to a limitation of anti-affinity rules with DaemonSets,
a client-mode agent runs alongside server-mode agents in Kubernetes. This
duplication wastes some resources, but otherwise functions perfectly fine.
Reference in New Issue View Git Blame Copy Permalink