From 2163f79170572b0f609d743551da60b676c44354 Mon Sep 17 00:00:00 2001 From: Iryna Shustava Date: Tue, 21 Jan 2020 19:29:55 -0800 Subject: [PATCH] Add docs about rolling out TLS on k8s (#7096) * Add docs about gradually rolling out TLS on k8s Co-authored-by: Luke Kysow <1034429+lkysow@users.noreply.github.com> --- .../docs/platform/k8s/operations.html.md | 1 + .../k8s/tls-on-existing-cluster.html.md | 109 ++++++++++++++++++ .../docs/platform/k8s/upgrading.html.md | 9 +- website/source/layouts/docs.erb | 3 + 4 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 website/source/docs/platform/k8s/tls-on-existing-cluster.html.md diff --git a/website/source/docs/platform/k8s/operations.html.md b/website/source/docs/platform/k8s/operations.html.md index 21e733b25..1cb622a1f 100644 --- a/website/source/docs/platform/k8s/operations.html.md +++ b/website/source/docs/platform/k8s/operations.html.md @@ -11,4 +11,5 @@ description: |- This section holds documentation on various operational tasks you may need to perform. * [Upgrading](/docs/platform/k8s/upgrading.html) - Upgrading your Consul servers or clients and the Helm chart. +* [Configuring TLS on an Existing Cluster](/docs/platform/k8s/tls-on-existing-cluster.html) - Configuring TLS on an existing Consul cluster running in Kubernetes. * [Uninstalling](/docs/platform/k8s/uninstalling.html) - Uninstaling the Helm chart. diff --git a/website/source/docs/platform/k8s/tls-on-existing-cluster.html.md b/website/source/docs/platform/k8s/tls-on-existing-cluster.html.md new file mode 100644 index 000000000..16794adb9 --- /dev/null +++ b/website/source/docs/platform/k8s/tls-on-existing-cluster.html.md @@ -0,0 +1,109 @@ +--- +layout: "docs" +page_title: "Configuring TLS on an Existing Cluster" +sidebar_current: "docs-platform-k8s-ops-tls-on-existing-cluster" +description: |- + Configuring TLS on an existing Consul cluster running in Kubernetes +--- + +# Configuring TLS on an Existing Cluster + +As of Consul Helm version `0.16.0`, the chart supports TLS for communication +within the cluster. If you already have a Consul cluster deployed on Kubernetes, +you may want to configure TLS in a way that minimizes downtime to your applications. +Consul already supports rolling out TLS on an existing cluster without downtime. +However, depending on your Kubernetes use case, your upgrade procedure may be different. + +## Gradual TLS Rollout without Consul Connect + +If you're **not using Consul Connect**, follow this process. + + 1. Run a Helm upgrade with the following config: + + ```yaml + global: + tls: + enabled: true + # This configuration sets `verify_outgoing`, `verify_server_hostname`, + # and `verify_incoming` to `false` on servers and clients, + # which allows TLS-disabled nodes to join the cluster. + verify: false + server: + updatePartition: + ``` + + This upgrade will trigger a rolling update of the clients, as well as any + other `consul-k8s` components, such as sync catalog or client snapshot deployments. + + 1. Perform a rolling upgrade of the servers, as described in + [Upgrade Consul Servers](/docs/platform/k8s/upgrading.html#upgrading-consul-servers). + + 1. Repeat steps 1 and 2, turning on TLS verification by setting `global.tls.verify` + to `true`. + +## Gradual TLS Rollout with Consul Connect + +Because the sidecar Envoy proxies need to talk to the Consul client agent regularly +for service discovery, we can't enable TLS on the clients without also re-injecting a +TLS-enabled proxy into the application pods. To perform TLS rollout with minimal +downtime, we recommend instead to add a new Kubernetes node pool and migrate your +applications to it. + + 1. Add a new identical node pool. + + 1. Cordon all nodes in the **old** pool by running `kubectl cordon` + to ensure Kubernetes doesn't schedule any new workloads on those nodes + and instead schedules onto the new nodes, which shortly will be TLS-enabled. + + 1. Create the following Helm config file for the upgrade: + + ```yaml + global: + tls: + enabled: true + # This configuration sets `verify_outgoing`, `verify_server_hostname`, + # and `verify_incoming` to `false` on servers and clients, + # which allows TLS-disabled nodes to join the cluster. + verify: false + server: + updatePartition: + client: + updateStrategy: | + type: OnDelete + ``` + + In this configuration, we're setting `server.updatePartition` to the number of + server replicas as described in [Upgrade Consul Servers](/docs/platform/k8s/upgrading.html#upgrading-consul-servers) + and `client.updateStrategy` to `OnDelete` to manually trigger an upgrade of the clients. + + 1. Run `helm upgrade` with the above config file. The upgrade will trigger an update of all + components except clients and servers, such as the Consul Connect webhook deployment + or the sync catalog deployment. Note that the sync catalog and the client + snapshot deployments will not be in the `ready` state until the clients on their + nodes are upgraded. It is OK to proceed to the next step without them being ready + because Kubernetes will keep the old deployment pod around, and so there will be no + downtime. + + 1. Gradually perform an upgrade of the clients by deleting client pods on the **new** node + pool. + + 1. At this point, all components (e.g., Consul Connect webhook and sync catalog) should be running + on the new node pool. + + 1. Redeploy all your Connect-enabled applications. + One way to trigger a redeploy is to run `kubectl drain` on the nodes in the old pool. + Now that the Connect webhook is TLS-aware, it will add TLS configuration to + the sidecar proxy. Also, Kubernetes should schedule these applications on the new node pool. + + 1. Perform a rolling upgrade of the servers described in + [Upgrade Consul Servers](/docs/platform/k8s/upgrading.html#upgrading-consul-servers). + + 1. If everything is healthy, delete the old node pool. + + 1. Finally, set `global.tls.verify` to `true` in your Helm config file, remove the + `client.updateStrategy` property, and perform a rolling upgrade of the servers. + +-> **Note:** It is possible to do this upgrade without fully duplicating the node pool. +You could drain a subset of the Kubernetes nodes within your existing node pool and treat it +as your "new node pool." Then follow the above instructions. Repeat this process for the rest +of the nodes in the node pool. \ No newline at end of file diff --git a/website/source/docs/platform/k8s/upgrading.html.md b/website/source/docs/platform/k8s/upgrading.html.md index 7a825ce0f..d0c199678 100644 --- a/website/source/docs/platform/k8s/upgrading.html.md +++ b/website/source/docs/platform/k8s/upgrading.html.md @@ -22,7 +22,7 @@ be updated in batches. To initiate the upgrade, change the `server.image` value to the desired Consul version. For illustrative purposes, the example below will -use `consul:123.456`. Also set the `server.updatePartition` value +use `consul:123.456`. Also, set the `server.updatePartition` value _equal to the number of server replicas_: ```yaml @@ -61,3 +61,10 @@ With the servers upgraded, it is time to upgrade the clients. To upgrade the clients, set the `client.image` value to the desired Consul version. Then, run `helm upgrade`. This will upgrade the clients in batches, waiting until the clients come up healthy before continuing. + +## Configuring TLS on an Existing Cluster + +If you already have a Consul cluster deployed on Kubernetes and +would like to turn on TLS for internal Consul communication, +please see +[Configuring TLS on an Existing Cluster](/docs/platform/k8s/tls-on-existing-cluster.html). diff --git a/website/source/layouts/docs.erb b/website/source/layouts/docs.erb index 54582b21a..af8ca60bb 100644 --- a/website/source/layouts/docs.erb +++ b/website/source/layouts/docs.erb @@ -624,6 +624,9 @@ > Upgrading + > + Configuring TLS on an Existing Cluster + > Uninstalling