147 lines
8.1 KiB
Plaintext
147 lines
8.1 KiB
Plaintext
---
|
||
layout: docs
|
||
page_title: Configure metrics for Consul on Kubernetes
|
||
description: >-
|
||
Use the `connectInject.metrics` Helm values to enable Prometheus and Grafana integrations and capture metrics. Consul can collect metrics from the service mesh, sidecar proxies, agents, and gateways in a k8s cluster and then display service traffic metrics in Consul’s UI for additional observability.
|
||
---
|
||
|
||
# Configure Metrics for Consul on Kubernetes
|
||
|
||
Consul on Kubernetes integrates with Prometheus and Grafana to provide metrics for Consul Service Mesh. The metrics
|
||
available are:
|
||
|
||
* Connect Service metrics
|
||
* Sidecar proxy metrics
|
||
* Consul agent metrics
|
||
* Ingress, Terminating and Mesh Gateway metrics
|
||
|
||
Specific sidecar proxy metrics can also be seen in the Consul UI Topology Visualization view. This section documents how to enable each of these.
|
||
|
||
-> **Note:** Metrics will be supported in Consul-helm >= `0.31.0` and consul-k8s >= `0.25.0`. However, enabling the [metrics merging feature](#connect-service-and-sidecar-metrics-with-metrics-merging) with Helm value (`defaultEnableMerging`) or
|
||
annotation (`consul.hashicorp.com/enable-metrics-merging`) can only be used with Consul `1.10.0` and above. The
|
||
other metrics configuration can still be used before Consul `1.10.0`.
|
||
|
||
## Connect Service and Sidecar Metrics with Metrics Merging
|
||
|
||
Prometheus annotations are used to instruct Prometheus to scrape metrics from Pods. Prometheus annotations only support
|
||
scraping from one endpoint on a Pod, so Consul on Kubernetes supports metrics merging whereby service metrics and
|
||
sidecar proxy metrics are merged into one endpoint. If there are no service metrics, it also supports just scraping the
|
||
sidecar proxy metrics.
|
||
|
||
The diagram below illustrates how the metrics integration works when merging is enabled:
|
||
|
||
[](/img/metrics-arch.png)
|
||
|
||
Connect service metrics can be configured with the Helm values nested under [`connectInject.metrics`](/docs/k8s/helm#v-connectinject-metrics).
|
||
|
||
Metrics and metrics merging can be enabled by default for all connect-injected Pods with the following Helm values:
|
||
```yaml
|
||
connectInject:
|
||
metrics:
|
||
defaultEnabled: true # by default, this inherits from the value global.metrics.enabled
|
||
defaultEnableMerging: true
|
||
```
|
||
They can also be overridden on a per-Pod basis using the annotations `consul.hashicorp.com/enable-metrics` and
|
||
`consul.hashicorp.com/enable-metrics-merging`.
|
||
|
||
~> In most cases, the default settings will be sufficient. If you are encountering issues with colliding ports or service
|
||
metrics not being merged, you may need to change the defaults.
|
||
|
||
The Prometheus annotations configure the endpoint to scrape the metrics from. As shown in the diagram, the annotations point to a listener on `0.0.0.0:20200` on the Envoy sidecar. This listener and the corresponding Prometheus annotations can be configured with the following Helm values (or overridden on a per-Pod basis with Consul annotations `consul.hashicorp.com/prometheus-scrape-port` and `consul.hashicorp.com/prometheus-scrape-path`):
|
||
```yaml
|
||
connectInject:
|
||
metrics:
|
||
defaultPrometheusScrapePort: 20200
|
||
defaultPrometheusScrapePath: "/metrics"
|
||
```
|
||
Those Helm values will result in the following Prometheus annotations being automatically added to the Pod for scraping:
|
||
```yaml
|
||
metadata:
|
||
annotations:
|
||
prometheus.io/scrape: "true"
|
||
prometheus.io/path: "/metrics"
|
||
prometheus.io/port: "20200"
|
||
```
|
||
|
||
When metrics alone are enabled, the listener in the diagram on `0.0.0.0:20200` would point directly at the sidecar
|
||
metrics endpoint, rather than the merged metrics endpoint. The Prometheus scraping annotations would stay the same.
|
||
|
||
When metrics and metrics merging are *both* enabled, metrics are combined from the service and the sidecar proxy, and
|
||
exposed via a local server on the Consul sidecar for scraping. This endpoint is called the merged metrics endpoint and
|
||
defaults to `127.0.0.1:20100/stats/prometheus`. The listener will target the merged metrics endpoint in the above case.
|
||
It can be configured with the following Helm values (or overridden on a per-Pod basis with
|
||
`consul.hashicorp.com/merged-metrics-port`):
|
||
```yaml
|
||
connectInject:
|
||
metrics:
|
||
defaultMergedMetricsPort: 20100
|
||
```
|
||
|
||
The endpoint to scrape service metrics from can be configured only on a per-Pod basis via the Pod annotations `consul.hashicorp.com/service-metrics-port` and `consul.hashicorp.com/service-metrics-path`. If these are not configured, the service metrics port will default to the port used to register the service with Consul (`consul.hashicorp.com/connect-service-port`), which in turn defaults to the first port on the first container of the Pod. The service metrics path will default to `/metrics`.
|
||
|
||
## Consul Agent Metrics
|
||
|
||
Metrics from the Consul server and client Pods can be scraped via Prometheus by setting the field `global.metrics.enableAgentMetrics` to `true`. Additionally, one can configure the metrics retention time on the agents by configuring
|
||
the field `global.metrics.agentMetricsRetentionTime` which expects a duration and defaults to `"1m"`. This value must be greater than `"0m"` for the Consul servers and clients to emit metrics at all. As the Prometheus deployment currently does not support scraping TLS endpoints, agent metrics are currently *unsupported* when TLS is enabled.
|
||
|
||
```yaml
|
||
global:
|
||
metrics:
|
||
enabled: true
|
||
enableAgentMetrics: true
|
||
agentMetricsRetentionTime: "1m"
|
||
```
|
||
|
||
## Gateway Metrics
|
||
|
||
Metrics from the Consul gateways, namely the Ingress Gateways, Terminating Gateways and the Mesh Gateways can be scraped
|
||
via Prometheus by setting the field `global.metrics.enableGatewayMetrics` to `true`. The gateways emit standard Envoy proxy
|
||
metrics. To ensure that the metrics are not exposed to the public internet (as Mesh and Ingress gateways can have public
|
||
IPs), their metrics endpoints are exposed on the Pod IP of the respective gateway instance, rather than on all
|
||
interfaces on `0.0.0.0`.
|
||
|
||
```yaml
|
||
global:
|
||
metrics:
|
||
enabled: true
|
||
enableGatewayMetrics: true
|
||
```
|
||
|
||
## Metrics in the UI Topology Visualization
|
||
|
||
Consul's built-in UI has a topology visualization for services part of the Consul Service Mesh. The topology visualization has the ability to fetch basic metrics from a metrics provider for each service and display those metrics as part of the [topology visualization](/docs/connect/observability/ui-visualization).
|
||
|
||
The diagram below illustrates how the UI displays service metrics for a sample application:
|
||
|
||
|
||
|
||
The topology view is configured under `ui.metrics`. This will enable the Consul UI to query the provider specified by
|
||
`ui.metrics.provider` at the URL of the Prometheus server `ui.metrics.baseURL` to display sidecar proxy metrics for the
|
||
service. The UI will display some specific sidecar proxy Prometheus metrics when `ui.metrics.enabled` is `true` and
|
||
`ui.enabled` is true. The value of `ui.metrics.enabled` defaults to `"-"` which means it will inherit from the value of
|
||
`global.metrics.enabled.`
|
||
|
||
```yaml
|
||
ui:
|
||
enabled: true
|
||
metrics:
|
||
enabled: true # by default, this inherits from the value global.metrics.enabled
|
||
provider: "prometheus"
|
||
baseURL: http://prometheus-server
|
||
```
|
||
|
||
## Deploying Prometheus (_for demo and non-production use-cases only_)
|
||
|
||
The Helm chart contains demo manifests for deploying Prometheus. It can be installed with Helm via `prometheus.enabled`. This manifest is based on the community manifest for Prometheus.
|
||
The Prometheus deployment is designed to allow quick bootstrapping for trial and demo use cases, and is not recommended for production use-cases.
|
||
|
||
Prometheus will be installed in the same namespace as Consul, and will be installed
|
||
and uninstalled along with the Consul installation.
|
||
|
||
Grafana can optionally be utilized with Prometheus to display metrics. The installation and configuration of Grafana must be managed separately from the Consul Helm chart. The [Layer 7 Observability with Prometheus, Grafana, and Kubernetes](https://learn.hashicorp.com/tutorials/consul/kubernetes-layer7-observability?in=consul/kubernetes?in=consul/kubernetes)) tutorial provides an installation walkthrough using Helm.
|
||
|
||
```yaml
|
||
prometheus:
|
||
enabled: true
|
||
```
|