The Consul agent collects various runtime metrics about the performance of different libraries and subsystems. These metrics are aggregated on a ten second interval and are retained for one minute.
These are some metrics emitted that can help you understand the health of your cluster at a glance. For a full list of metrics emitted by Consul, see [Metrics Reference](#metrics-reference)
### Transaction timing
| Metric Name | Description |
| :----------------------- | :---------- |
| `consul.kvs.apply` | This measures the time it takes to complete an update to the KV store. |
| `consul.txn.apply` | This measures the time spent applying a transaction operation. |
| `consul.raft.apply` | This counts the number of Raft transactions occurring over the interval. |
| `consul.raft.commitTime` | This measures the time it takes to commit a new entry to the Raft log on the leader. |
**Why they're important:** Taken together, these metrics indicate how long it takes to complete write operations in various parts of the Consul cluster. Generally these should all be fairly consistent and no more than a few milliseconds. Sudden changes in any of the timing values could be due to unexpected load on the Consul servers, or due to problems on the servers themselves.
**What to look for:** Deviations (in any of these metrics) of more than 50% from baseline over the previous hour.
### Leadership changes
| Metric Name | Description |
| :---------- | :---------- |
| `consul.raft.leader.lastContact` | Measures the time since the leader was last able to contact the follower nodes when checking its leader lease. |
| `consul.raft.state.candidate` | This increments whenever a Consul server starts an election. |
| `consul.raft.state.leader` | This increments whenever a Consul server becomes a leader. |
**Why they're important:** Normally, your Consul cluster should have a stable leader. If there are frequent elections or leadership changes, it would likely indicate network issues between the Consul servers, or that the Consul servers themselves are unable to keep up with the load.
**What to look for:** For a healthy cluster, you're looking for a `lastContact` lower than 200ms, `leader` > 0 and `candidate` == 0. Deviations from this might indicate flapping leadership.
### Autopilot
| Metric Name | Description |
| :---------- | :---------- |
| `consul.autopilot.healthy` | This tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0. |
**Why it's important:** Obviously, you want your cluster to be healthy.
**What to look for:** Alert if `healthy` is 0.
### Memory usage
| Metric Name | Description |
| :---------- | :---------- |
| `consul.runtime.alloc_bytes` | This measures the number of bytes allocated by the Consul process. |
| `consul.runtime.sys_bytes` | This is the total number of bytes of memory obtained from the OS. |
**Why they're important:** Consul keeps all of its data in memory. If Consul consumes all available memory, it will crash.
**What to look for:** If `consul.runtime.sys_bytes` exceeds 90% of total avaliable system memory.
### Garbage collection
| Metric Name | Description |
| :---------- | :---------- |
| `consul.runtime.total_gc_pause_ns` | Number of nanoseconds consumed by stop-the-world garbage collection (GC) pauses since Consul started. |
**Why it's important:** GC pause is a "stop-the-world" event, meaning that all runtime threads are blocked until GC completes. Normally these pauses last only a few nanoseconds. But if memory usage is high, the Go runtime may GC so frequently that it starts to slow down Consul.
**What to look for:** Warning if `total_gc_pause_ns` exceeds 2 seconds/minute, critical if it exceeds 5 seconds/minute.
**NOTE:** `total_gc_pause_ns` is a cumulative counter, so in order to calculate rates (such as GC/minute),
you will need to apply a function such as InfluxDB's [`non_negative_difference()`](https://docs.influxdata.com/influxdb/v1.5/query_language/functions/#non-negative-difference).
### Network activity - RPC Count
| Metric Name | Description |
| :---------- | :---------- |
| `consul.client.rpc` | Increments whenever a Consul agent in client mode makes an RPC request to a Consul server |
| `consul.client.rpc.exceeded` | Increments whenever a Consul agent in client mode makes an RPC request to a Consul server gets rate limited by that agent's [`limits`](/docs/agent/options.html#limits) configuration. |
| `consul.client.rpc.failed` | Increments whenever a Consul agent in client mode makes an RPC request to a Consul server and fails. |
**Why they're important:** These measurements indicate the current load created from a Consul agent, including when the load becomes high enough to be rate limited. A high RPC count, especially from `consul.client.rpcexceeded` meaning that the requests are being rate-limited, could imply a misconfigured Consul agent.
**What to look for:**
Sudden large changes to the `consul.client.rpc` metrics (greater than 50% deviation from baseline).
`consul.client.rpc.exceeded` or `consul.client.rpc.failed` count > 0, as it implies that an agent is being rate-limited or fails to make an RPC request to a Consul server
<td>This increments whenever a Consul agent in client mode makes an RPC request to a Consul server. This gives a measure of how much a given agent is loading the Consul servers. Currently, this is only generated by agents in client mode, not Consul servers.</td>
<td>requests</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.client.rpc.exceeded`</td>
<td>This increments whenever a Consul agent in client mode makes an RPC request to a Consul server gets rate limited by that agent's [`limits`](/docs/agent/options.html#limits) configuration. This gives an indication that there's an abusive application making too many requests on the agent, or that the rate limit needs to be increased. Currently, this only applies to agents in client mode, not Consul servers.</td>
<td>This tracks the number of running goroutines and is a general load pressure indicator. This may burst from time to time but should return to a steady state value.</td>
<td>number of goroutines</td>
<td>gauge</td>
</tr>
<tr>
<td>`consul.runtime.alloc_bytes`</td>
<td>This measures the number of bytes allocated by the Consul process. This may burst from time to time but should return to a steady state value.</td>
<td>bytes</td>
<td>gauge</td>
</tr>
<tr>
<td>`consul.runtime.heap_objects`</td>
<td>This measures the number of objects allocated on the heap and is a general memory pressure indicator. This may burst from time to time but should return to a steady state value.</td>
<td>This tracks how long it takes to service the given HTTP request for the given verb and path. Paths do not include details like service or key names, for these an underscore will be present as a placeholder (eg. `consul.http.GET.v1.kv._`)</td>
<td>This metric measures the time taken by the Consul agent to compact a log. This operation occurs only when the snapshot becomes large enough to justify the compaction .</td>
<td>This increments whenever a Consul server becomes a leader. If there are frequent leadership changes this may be indication that the servers are overloaded and aren't meeting the soft real-time requirements for Raft, or that there are networking problems between the servers.</td>
<td>leadership transitions / interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.raft.state.candidate`</td>
<td>This increments whenever a Consul server starts an election. If this increments without a leadership change occurring it could indicate that a single server is overloaded or is experiencing network connectivity issues.</td>
<td>election attempts / interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.raft.apply`</td>
<td>This counts the number of Raft transactions occurring over the interval, which is a general indicator of the write load on the Consul servers.</td>
<td>This metric counts the number of times the agent has started the barrier i.e the number of times it has
issued a blocking call, to ensure that the agent has all the pending operations that were queued, to be applied to the agent's FSM.</td>
<td>blocks / interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.raft.verify_leader`</td>
<td>This metric counts the number of times an agent checks whether it is still the leader or not</td>
<td>checks / interval</td>
<td>Counter</td>
</tr>
<tr>
<td>`consul.raft.restore`</td>
<td>This metric counts the number of times the restore operation has been performed by the agent. Here, restore refers to the action of raft consuming an external snapshot to restore its state.</td>
<td>This measures the time it takes to commit a new entry to the Raft log on the leader.</td>
<td>ms</td>
<td>timer</td>
</tr>
<tr>
<td>`consul.raft.leader.dispatchLog`</td>
<td>This measures the time it takes for the leader to write log entries to disk.</td>
<td>ms</td>
<td>timer</td>
</tr>
<tr>
<td>`consul.raft.replication.appendEntries`</td>
<td>This measures the time it takes to replicate log entries to followers. This is a general indicator of the load pressure on the Consul servers, as well as the performance of the communication between the servers.</td>
<td>This metric counts the number of times an agent has entered the follower mode. This happens when a new agent joins the cluster or after the end of a leader election.</td>
<td>This metric gives the number of times an agent has transistioned to the Candidate state, after receive no heartbeat messages from the last known leader.</td>
<td>timeouts / interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.raft.restoreUserSnapshot`</td>
<td>This metric measures the time taken by the agent to restore the FSM state from a user's snapshot</td>
<td>ms</td>
<td>timer</td>
</tr>
<tr>
<td>`consul.raft.rpc.processHeartBeat`</td>
<td>This metric measures the time taken to process a heartbeat request.</td>
<td>ms</td>
<td>timer</td>
</tr>
<tr>
<td>`consul.raft.rpc.appendEntries`</td>
<td>This metric measures the time taken to process an append entries RPC call from an agent.</td>
<td>This metric measures the time taken to process the outstanding log entries of an agent.</td>
<td>ms</td>
<td>timer</td>
</tr>
<tr>
<tr>
<td>`consul.raft.rpc.requestVote`</td>
<td>This metric measures the time taken to process the request vote RPC call.</td>
<td>ms</td>
<td>timer</td>
</tr>
<tr>
<td>`consul.raft.rpc.installSnapshot`</td>
<td>This metric measures the time taken to process the installSnapshot RPC call. This metric should only be seen on agents which are currently in the follower state.</td>
<td>This will only be emitted by the Raft leader and measures the time since the leader was last able to contact the follower nodes when checking its leader lease. It can be used as a measure for how stable the Raft timing is and how close the leader is to timing out its lease.<br><br>The lease timeout is 500 ms times the [`raft_multiplier` configuration](/docs/agent/options.html#raft_multiplier), so this telemetry value should not be getting close to that configured value, otherwise the Raft timing is marginal and might need to be tuned, or more powerful servers might be needed. See the [Server Performance](/docs/guides/performance.html) guide for more details.</td>
<td>This metric counts the number of times the agent has performed failure detection on an other agent at a slower probe rate. The agent uses its own health metric as an indicator to perform this action. (If its health score is low, means that the node is healthy, and vice versa.)</td>
<td>probes / interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.degraded.timeout`</td>
<td>This metric counts the number of times an agent was marked as a dead node, whilst not getting enough confirmations from a randomly selected list of agent nodes in an agent's membership.</td>
<td>occurrence / interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.msg.dead`</td>
<td>This metric counts the number of times an agent has marked another agent to be a dead node.</td>
<td>This metric describes a node's perception of its own health based on how well it is meeting the soft real-time requirements of the protocol. This metric ranges from 0 to 8, where 0 indicates "totally healthy". This health score is used to scale the time between outgoing probes, and higher scores translate into longer probing intervals. For more details see section IV of the Lifeguard paper: https://arxiv.org/pdf/1707.00788.pdf</td>
<td>This increments when an agent suspects another as failed when executing random probes as part of the gossip protocol. These can be an indicator of overloaded agents, network problems, or configuration errors where agents can not connect to each other on the [required ports](/docs/agent/options.html#ports).</td>
<td>This metric gives the number of gossips (messages) broadcasted to a set of randomly selected nodes.</td>
<td>messages / Interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.msg_alive`</td>
<td>This metric counts the number of alive agents, that the agent has mapped out so far, based on the message information given by the network layer.</td>
<td>nodes / Interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.msg_dead`</td>
<td>This metric gives the number of dead agents, that the agent has mapped out so far, based on the message information given by the network layer.</td>
<td>nodes / Interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.msg_suspect`</td>
<td>This metric gives the number of suspect nodes, that the agent has mapped out so far, based on the message information given by the network layer.</td>
<td>nodes / Interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.probeNode`</td>
<td>This metric measures the time taken to perform a single round of failure detection on a select agent.</td>
<td>nodes / Interval</td>
<td>counter</td>
</tr>
<tr>
<td>`consul.memberlist.pushPullNode`</td>
<td>This metric measures the number of agents that have exchanged state with this agent.</td>
<td>Available in Consul 0.7 and later, this increments when an agent is marked dead and then recovers within a short time period. This can be an indicator of overloaded agents, network problems, or configuration errors where agents can not connect to each other on the [required ports](/docs/agent/options.html#ports).</td>
<td>This increments when an agent processes an [event](/docs/commands/event.html). Consul uses events internally so there may be additional events showing in telemetry. There are also a per-event counters emitted as `consul.serf.events.<event name>`.</td>
<td>This tracks the number of voting servers that the cluster can lose while continuing to function.</td>
<td>servers</td>
<td>gauge</td>
</tr>
<tr>
<td>`consul.autopilot.healthy`</td>
<td>This tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0.</td>
