146 lines
7.8 KiB
Plaintext
146 lines
7.8 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Metrics
|
|
description: Metrics for Consul on Kubernetes
|
|
---
|
|
|
|
# Metrics
|
|
|
|
Consul on Kubernetes integrates with Prometheus and Grafana to provide metrics for Consul Service Mesh. The metrics
|
|
available are:
|
|
|
|
* Connect Service metrics
|
|
* Sidecar proxy metrics
|
|
* Consul agent metrics
|
|
* Ingress, Terminating and Mesh Gateway metrics
|
|
|
|
Specific sidecar proxy metrics can also be seen in the Consul UI Topology Visualization view. This section documents how to enable each of these.
|
|
|
|
-> **Note:** Metrics will be supported in Consul-helm >= `0.31.0` and consul-k8s >= `0.25.0`. However, enabling the [metrics merging feature](#connect-service-and-sidecar-metrics-with-metrics-merging) with Helm value (`defaultEnableMerging`) or
|
|
annotation (`consul.hashicorp.com/enable-metrics-merging`) can only be used with Consul `1.10.0` and above. The
|
|
other metrics configuration can still be used before Consul `1.10.0`.
|
|
|
|
## Connect Service and Sidecar Metrics with Metrics Merging
|
|
|
|
Prometheus annotations are used to instruct Prometheus to scrape metrics from Pods. Prometheus annotations only support
|
|
scraping from one endpoint on a Pod, so Consul on Kubernetes supports metrics merging whereby service metrics and
|
|
sidecar proxy metrics are merged into one endpoint. If there are no service metrics, it also supports just scraping the
|
|
sidecar proxy metrics.
|
|
|
|
The diagram below illustrates how the metrics integration works when merging is enabled:
|
|
|
|
[](/img/metrics-arch.png)
|
|
|
|
Connect service metrics can be configured with the Helm values nested under [`connectInject.metrics`](/docs/k8s/helm#v-connectinject-metrics).
|
|
|
|
Metrics and metrics merging can be enabled by default for all connect-injected Pods with the following Helm values:
|
|
```yaml
|
|
connectInject:
|
|
metrics:
|
|
defaultEnabled: true # by default, this inherits from the value global.metrics.enabled
|
|
defaultEnableMerging: true
|
|
```
|
|
They can also be overridden on a per-Pod basis using the annotations `consul.hashicorp.com/enable-metrics` and
|
|
`consul.hashicorp.com/enable-metrics-merging`.
|
|
|
|
~> In most cases, the default settings will be sufficient. If you are encountering issues with colliding ports or service
|
|
metrics not being merged, you may need to change the defaults.
|
|
|
|
The Prometheus annotations configure the endpoint to scrape the metrics from. As shown in the diagram, the annotations point to a listener on `0.0.0.0:20200` on the Envoy sidecar. This listener and the corresponding Prometheus annotations can be configured with the following Helm values (or overridden on a per-Pod basis with Consul annotations `consul.hashicorp.com/prometheus-scrape-port` and `consul.hashicorp.com/prometheus-scrape-path`):
|
|
```yaml
|
|
connectInject:
|
|
metrics:
|
|
defaultPrometheusScrapePort: 20200
|
|
defaultPrometheusScrapePath: "/metrics"
|
|
```
|
|
Those Helm values will result in the following Prometheus annotations being automatically added to the Pod for scraping:
|
|
```yaml
|
|
metadata:
|
|
annotations:
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/path: "/metrics"
|
|
prometheus.io/port: "20200"
|
|
```
|
|
|
|
When metrics alone are enabled, the listener in the diagram on `0.0.0.0:20200` would point directly at the sidecar
|
|
metrics endpoint, rather than the merged metrics endpoint. The Prometheus scraping annotations would stay the same.
|
|
|
|
When metrics and metrics merging are *both* enabled, metrics are combined from the service and the sidecar proxy, and
|
|
exposed via a local server on the Consul sidecar for scraping. This endpoint is called the merged metrics endpoint and
|
|
defaults to `127.0.0.1:20100/stats/prometheus`. The listener will target the merged metrics endpoint in the above case.
|
|
It can be configured with the following Helm values (or overridden on a per-Pod basis with
|
|
`consul.hashicorp.com/merged-metrics-port`):
|
|
```yaml
|
|
connectInject:
|
|
metrics:
|
|
defaultMergedMetricsPort: 20100
|
|
```
|
|
|
|
The endpoint to scrape service metrics from can be configured only on a per-Pod basis via the Pod annotations `consul.hashicorp.com/service-metrics-port` and `consul.hashicorp.com/service-metrics-path`. If these are not configured, the service metrics port will default to the port used to register the service with Consul (`consul.hashicorp.com/connect-service-port`), which in turn defaults to the first port on the first container of the Pod. The service metrics path will default to `/metrics`.
|
|
|
|
## Consul Agent Metrics
|
|
|
|
Metrics from the Consul server and client Pods can be scraped via Prometheus by setting the field `global.metrics.enableAgentMetrics` to `true`. Additionally, one can configure the metrics retention time on the agents by configuring
|
|
the field `global.metrics.agentMetricsRetentionTime` which expects a duration and defaults to `"1m"`. This value must be greater than `"0m"` for the Consul servers and clients to emit metrics at all. As the Prometheus deployment currently does not support scraping TLS endpoints, agent metrics are currently *unsupported* when TLS is enabled.
|
|
|
|
```yaml
|
|
global:
|
|
metrics:
|
|
enabled: true
|
|
enableAgentMetrics: true
|
|
agentMetricsRetentionTime: "1m"
|
|
```
|
|
|
|
## Gateway Metrics
|
|
|
|
Metrics from the Consul gateways, namely the Ingress Gateways, Terminating Gateways and the Mesh Gateways can be scraped
|
|
via Prometheus by setting the field `global.metrics.enableGatewayMetrics` to `true`. The gateways emit standard Envoy proxy
|
|
metrics. To ensure that the metrics are not exposed to the public internet (as Mesh and Ingress gateways can have public
|
|
IPs), their metrics endpoints are exposed on the Pod IP of the respective gateway instance, rather than on all
|
|
interfaces on `0.0.0.0`.
|
|
|
|
```yaml
|
|
global:
|
|
metrics:
|
|
enabled: true
|
|
enableGatewayMetrics: true
|
|
```
|
|
|
|
## Metrics in the UI Topology Visualization
|
|
|
|
Consul's built-in UI has a topology visualization for services part of the Consul Service Mesh. The topology visualization has the ability to fetch basic metrics from a metrics provider for each service and display those metrics as part of the [topology visualization](/docs/connect/observability/ui-visualization).
|
|
|
|
The diagram below illustrates how the UI displays service metrics for a sample application:
|
|
|
|
|
|
|
|
The topology view is configured under `ui.metrics`. This will enable the Consul UI to query the provider specified by
|
|
`ui.metrics.provider` at the URL of the Prometheus server `ui.metrics.baseURL` to display sidecar proxy metrics for the
|
|
service. The UI will display some specific sidecar proxy Prometheus metrics when `ui.metrics.enabled` is `true` and
|
|
`ui.enabled` is true. The value of `ui.metrics.enabled` defaults to `"-"` which means it will inherit from the value of
|
|
`global.metrics.enabled.`
|
|
|
|
```yaml
|
|
ui:
|
|
enabled: true
|
|
metrics:
|
|
enabled: true # by default, this inherits from the value global.metrics.enabled
|
|
provider: "prometheus"
|
|
baseURL: http://prometheus-server
|
|
```
|
|
|
|
## Deploying Prometheus (_for demo and non-production use-cases only_)
|
|
|
|
The Helm chart contains demo manifests for deploying Prometheus. It can be installed with Helm via `prometheus.enabled`. This manifest is based on the community manifest for Prometheus.
|
|
The Prometheus deployment is designed to allow quick bootstrapping for trial and demo use cases, and is not recommended for production use-cases.
|
|
|
|
Prometheus will be installed in the same namespace as Consul, and will be installed
|
|
and uninstalled along with the Consul installation.
|
|
|
|
Grafana can optionally be utilized with Prometheus to display metrics. The installation and configuration of Grafana must be managed separately from the Consul Helm chart. The [Layer 7 Observability with Prometheus, Grafana, and Kubernetes](https://learn.hashicorp.com/tutorials/consul/kubernetes-layer7-observability?in=consul/kubernetes?in=consul/kubernetes)) tutorial provides an installation walkthrough using Helm.
|
|
|
|
```yaml
|
|
prometheus:
|
|
enabled: true
|
|
```
|