diff --git a/.changelog/16284.txt b/.changelog/16284.txt new file mode 100644 index 000000000..23dd2aa6f --- /dev/null +++ b/.changelog/16284.txt @@ -0,0 +1,3 @@ +```release-note:feature +cli: adds new CLI commands `consul troubleshoot upstreams` and `consul troubleshoot proxy` to troubleshoot Consul's service mesh configuration and network issues. +``` \ No newline at end of file diff --git a/website/content/commands/index.mdx b/website/content/commands/index.mdx index 2946d794b..415fc551d 100644 --- a/website/content/commands/index.mdx +++ b/website/content/commands/index.mdx @@ -56,6 +56,7 @@ Available commands are: services Interact with services snapshot Saves, restores and inspects snapshots of Consul server state tls Builtin helpers for creating CAs and certificates + troubleshoot Provides tools to troubleshoot Consul's service mesh configuration validate Validate config files/directories version Prints the Consul version watch Watch for changes in Consul diff --git a/website/content/commands/troubleshoot/index.mdx b/website/content/commands/troubleshoot/index.mdx new file mode 100644 index 000000000..521981a77 --- /dev/null +++ b/website/content/commands/troubleshoot/index.mdx @@ -0,0 +1,31 @@ +--- +layout: commands +page_title: 'Commands: Troubleshoot' +description: >- + The `consul troubleshoot` command provides tools to troubleshoot Consul's service mesh configuration. +--- + +# Consul Troubleshooting + +Command: `consul troubleshoot` + +Use the `troubleshoot` command to diagnose Consul service mesh configuration or network issues. + +## Usage + +```text +Usage: consul troubleshoot [options] + + # ... + +Subcommands: + + proxy Troubleshoots service mesh issues from the current Envoy instance + upstreams Gets upstream Envoy identifiers and IPs configured for the proxy +``` + +For more information, examples, and usage about a subcommand, click on the name +of the subcommand in the sidebar or one of the links below: + +- [proxy](/consul/commands/troubleshoot/proxy) +- [upstreams](/consul/commands/troubleshoot/upstreams) diff --git a/website/content/commands/troubleshoot/proxy.mdx b/website/content/commands/troubleshoot/proxy.mdx new file mode 100644 index 000000000..7a11983ae --- /dev/null +++ b/website/content/commands/troubleshoot/proxy.mdx @@ -0,0 +1,68 @@ +--- +layout: commands +page_title: 'Commands: Troubleshoot Proxy' +description: >- + The `consul troubleshoot proxy` diagnoses Consul service mesh configuration and network issues to an upstream. +--- + +# Consul Troubleshoot Proxy + +Command: `consul troubleshoot proxy` + +The `troubleshoot proxy` command diagnoses Consul service mesh configuration and network issues to an upstream. + +## Usage + +Usage: `consul troubleshoot proxy (-upstream-ip |-upstream-envoy-id ) [options]` +This command requires `-envoy-admin-endpoint` or `-upstream-ip` to specify the upstream. + +#### Command Options + +- `-envoy-admin-endpoint=` - Envoy admin endpoint address for the local Envoy instance. +Defaults to `127.0.0.1:19000`. + +- `-upstream-ip=` - The IP address of the upstream service; Use when the upstream is reached via a transparent proxy DNS address (KubeDNS or Consul DNS) + +- `-upstream-envoy-id=` - The Envoy identifier of the upstream service; Use when the upstream is explicitly configured on the downstream service. + +## Examples + +The following example illustrates how to troubleshoot Consul service mesh configuration and network issues to an upstream from a source proxy. + + ```shell-session + $ consul troubleshoot proxy -upstream-ip 10.4.6.160 + + ==> Validation + ✓ Certificates are valid + ✓ Envoy has 0 rejected configurations + ✓ Envoy has detected 0 connection failure(s) + ✓ Listener for upstream "backend" found + ✓ Route for upstream "backend" found + ✓ Cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found + ✓ Healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found + ✓ Cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found + ! No healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found + -> Check that your upstream service is healthy and running + -> Check that your upstream service is registered with Consul + -> Check that the upstream proxy is healthy and running + -> If you are explicitly configuring upstreams, ensure the name of the upstream is correct + ``` + +The following example troubleshoots Consul service mesh configuration and network issues from the local Envoy instance to the given upstream. + + ```shell-session + $ consul-k8s troubleshoot proxy -upstream-envoy-id db + + ==> Validation + ✓ Certificates are valid + ✓ Envoy has 0 rejected configurations + ✓ Envoy has detected 0 connection failure(s) + ✓ Listener for upstream "db" found + ✓ Route for upstream "db" found + ✓ Cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0.consul" for upstream "db" found + ✓ Healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0.consul" for upstream "db" found + + If you are still experiencing issues, you can: + -> Check intentions to ensure the upstream allows traffic from this source + -> If using transparent proxy, ensure DNS resolution is to the same IP you have verified here + ``` diff --git a/website/content/commands/troubleshoot/upstreams.mdx b/website/content/commands/troubleshoot/upstreams.mdx new file mode 100644 index 000000000..5cbeb232f --- /dev/null +++ b/website/content/commands/troubleshoot/upstreams.mdx @@ -0,0 +1,37 @@ +--- +layout: commands +page_title: 'Commands: Troubleshoot Upstreams' +description: >- + The `consul troubleshoot upstreams` lists the available upstreams in the Consul service mesh from the current service. +--- + +# Consul Troubleshoot Upstreams + +Command: `consul troubleshoot upstreams` + +The `troubleshoot upstreams` lists the available upstreams in the Consul service mesh from the current service. + +## Usage + +Usage: `consul troubleshoot upstreams [options]` + +#### Command Options + +- `-envoy-admin-endpoint=` - Envoy admin endpoint address for the local Envoy instance. +Defaults to `127.0.0.1:19000`. + +## Examples + +Display all transparent proxy upstreams in Consul service mesh from the current Envoy instance. + + ```shell-session + $ consul troubleshoot upstreams + ==> Upstreams (explicit upstreams only) (0) + + ==> Upstreams IPs (transparent proxy only) (1) + [10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul] + + If you cannot find the upstream address or cluster for a transparent proxy upstream: + - Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream. + - To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured. + ``` diff --git a/website/content/docs/k8s/k8s-cli.mdx b/website/content/docs/k8s/k8s-cli.mdx index 402d2327b..b3e6f76bc 100644 --- a/website/content/docs/k8s/k8s-cli.mdx +++ b/website/content/docs/k8s/k8s-cli.mdx @@ -33,6 +33,7 @@ You can use the following commands with `consul-k8s`. - [`proxy list`](#proxy-list): List all Pods running proxies managed by Consul. - [`proxy read`](#proxy-read): Inspect the Envoy configuration for a given Pod. - [`status`](#status): Check the status of a Consul installation on Kubernetes. + - [`troubleshoot`](#troubleshoot): Troubleshoot Consul service mesh and networking issues from a given pod. - [`uninstall`](#uninstall): Uninstall Consul deployment. - [`upgrade`](#upgrade): Upgrade Consul on Kubernetes from an existing installation. - [`version`](#version): Print the version of the Consul on Kubernetes CLI. @@ -490,6 +491,106 @@ $ consul-k8s status ✓ Consul clients healthy (3/3) ``` +### `troubleshoot` + +The `troubleshoot` command exposes two subcommands for troubleshooting Consul +service mesh and network issues from a given pod. + +- [`troubleshoot upstreams`](#troubleshoot-upstreams): List all Envoy upstreams in Consul service mesh from the given pod. +- [`troubleshoot proxy`](#troubleshoot-proxy): Troubleshoot Consul service mesh configuration and network issues between the given pod and the given upstream. + +### `troubleshoot upstreams` + +```shell-session +$ consul-k8s troubleshoot upstreams -pod +``` + +| Flag | Description | Default | +| ------------------------------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `-namespace`, `-n` | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. | +| `-envoy-admin-endpoint` | `String` Envoy sidecar address and port | `127.0.0.1:19000` | + +#### Example Commands + +The following example displays all transparent proxy upstreams in Consul service mesh from the given pod. + + ```shell-session + $ consul-k8s troubleshoot upstreams -pod frontend-767ccfc8f9-6f6gx + + ==> Upstreams (explicit upstreams only) (0) + + ==> Upstreams IPs (transparent proxy only) (1) + [10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul] + + If you cannot find the upstream address or cluster for a transparent proxy upstream: + - Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream. + - To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured. + ``` + +The following example displays all explicit upstreams from the given pod in the Consul service mesh. + + ```shell-session + $ consul-k8s troubleshoot upstreams -pod client-767ccfc8f9-6f6gx + + ==> Upstreams (explicit upstreams only) (1) + server + counting + + ==> Upstreams IPs (transparent proxy only) (0) + + If you cannot find the upstream address or cluster for a transparent proxy upstream: + - Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream. + - To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured. + ``` + +### `troubleshoot proxy` + +```shell-session +$ consul-k8s troubleshoot proxy -pod -upstream-ip +$ consul-k8s troubleshoot proxy -pod -upstream-envoy-id +``` + +| Flag | Description | Default | +| ------------------------------------ | ----------------------------------------------------------| ---------------------------------------------------------------------------------------------------------------------- | +| `-namespace`, `-n` | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. | +| `-envoy-admin-endpoint` | `String` Envoy sidecar address and port | `127.0.0.1:19000` | +| `-upstream-ip` | `String` The IP address of the upstream transparent proxy | | +| `-upstream-envoy-id` | `String` The Envoy identifier of the upstream | | + +#### Example Commands + +The following example troubleshoots the Consul service mesh configuration and network issues between the given pod and the given upstream IP. + + ```shell-session + $ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-ip 10.4.6.160 + + ==> Validation + ✓ certificates are valid + ✓ Envoy has 0 rejected configurations + ✓ Envoy has detected 0 connection failure(s) + ✓ listener for upstream "backend" found + ✓ route for upstream "backend" found + ✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found + ✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found + ✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found + ! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found + ``` + +The following example troubleshoots the Consul service mesh configuration and network issues between the given pod and the given upstream. + + ```shell-session + $ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-envoy-id db + + ==> Validation + ✓ certificates are valid + ✓ Envoy has 0 rejected configurations + ✓ Envoy has detected 0 connection failure(s) + ! no listener for upstream "db" found + ! no route for upstream "backend" found + ! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found + ! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found + ``` + ### `uninstall` The `uninstall` command removes Consul from Kubernetes. diff --git a/website/data/commands-nav-data.json b/website/data/commands-nav-data.json index 62ed01a40..aac55d795 100644 --- a/website/data/commands-nav-data.json +++ b/website/data/commands-nav-data.json @@ -528,6 +528,23 @@ } ] }, + { + "title": "troubleshoot", + "routes": [ + { + "title": "Overview", + "path": "troubleshoot" + }, + { + "title": "upstreams", + "path": "troubleshoot/upstreams" + }, + { + "title": "proxy", + "path": "troubleshoot/proxy" + } + ] + }, { "title": "validate", "path": "validate"