--- layout: "docs" page_title: "Telemetry" sidebar_current: "docs-internals-telemetry" description: |- Learn about the telemetry data available in Vault. --- # Telemetry The Vault agent collects various runtime metrics about the performance of different libraries and subsystems. These metrics are aggregated on a ten second interval and are retained for one minute. To view this data, you must send a signal to the Vault process: on Unix, this is `USR1` while on Windows it is `BREAK`. Once Vault receives the signal, it will dump the current telemetry information to the agent's `stderr`. This telemetry information can be used for debugging or otherwise getting a better view of what Vault is doing. Telemetry information can be streamed to both [statsite](https://github.com/armon/statsite) as well as statsd based on providing the appropriate configuration options. Below is sample output of a telemetry dump: ```text [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.num_goroutines': 12.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.free_count': 11882.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.total_gc_runs': 9.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.expire.num_leases': 1.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.alloc_bytes': 502992.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.sys_bytes': 3999992.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.malloc_count': 17315.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.heap_objects': 5433.000 [2015-04-20 12:24:30 -0700 PDT][G] 'vault.runtime.total_gc_pause_ns': 3794124.000 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.audit.log_response': Count: 2 Min: 0.001 Mean: 0.001 Max: 0.001 Stddev: 0.000 Sum: 0.002 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.route.read.secret-': Count: 1 Sum: 0.036 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.barrier.get': Count: 3 Min: 0.004 Mean: 0.021 Max: 0.050 Stddev: 0.025 Sum: 0.064 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.token.lookup': Count: 2 Min: 0.040 Mean: 0.074 Max: 0.108 Stddev: 0.048 Sum: 0.148 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.policy.get_policy': Count: 2 Min: 0.003 Mean: 0.004 Max: 0.005 Stddev: 0.001 Sum: 0.009 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.core.check_token': Count: 2 Min: 0.053 Mean: 0.087 Max: 0.121 Stddev: 0.048 Sum: 0.174 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.audit.log_request': Count: 2 Min: 0.001 Mean: 0.001 Max: 0.001 Stddev: 0.000 Sum: 0.002 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.barrier.put': Count: 3 Min: 0.004 Mean: 0.010 Max: 0.019 Stddev: 0.008 Sum: 0.029 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.route.write.secret-': Count: 1 Sum: 0.035 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.core.handle_request': Count: 2 Min: 0.097 Mean: 0.228 Max: 0.359 Stddev: 0.186 Sum: 0.457 [2015-04-20 12:24:30 -0700 PDT][S] 'vault.expire.register': Count: 1 Sum: 0.18 ``` You'll note that log entries are prefixed with the metric type as follows: - `[C]` is a counter - `[G]` is a gauge - `[S]` is a summary ## Key Metrics The following tables described the different Vault metrics. The metrics interval can be assumed to be 10 seconds when retrieving metrics using the above described signals. ### Internal Metrics These metrics represent operational aspects of the running Vault instance. | Metric | Description | Unit | Type | | ---------------- | ----------------------------------| ---- | ---- | |`vault.audit.log_request`| This measures the number of audit log requests | Number of requests | Summary | |`vault.audit.log_response`| This measures the number of audit log responses | Number of responses | Summary | |`vault.barrier.delete`| This measures the number of delete operations at the barrier | Number of operations | Summary | |`vault.barrier.get`| This measures the number of get operations at the barrier | Number of operations | Summary | |`vault.barrier.put`| This measures the number of put operations at the barrier | Number of operations | Summary | |`vault.barrier.list`| This measures the number of list operations at the barrier | Number of operations | Counter | |`vault.core.check_token`| This measures the number of token checks | Number of checks | Summary | |`vault.core.fetch_acl_and_token`| This measures the number of ACL and corresponding token entry fetches | Number of fetches | Summary | |`vault.core.handle_request`| This measures the number of requests | Number of requests | Summary | |`vault.core.handle_login_request`| This measures the number of login requests | Number of requests | Summary | |`vault.core.leadership_setup_failed`| This measures the number of cluster leadership setup failures | Number of failures | Summary | |`vault.core.leadership_lost`| This measures the number of cluster leadership losses | Number of losses | Summary | |`vault.core.post_unseal` | This measures the number of post-unseal operations | Number of operations | Gauge | |`vault.core.pre_seal`| This measures the number of pre-seal operations | Number of operations | Gauge | |`vault.core.seal-with-request`| This measures the number of requested seal operations | Number of operations | Gauge | |`vault.core.seal`| This measures the number of seal operations | Number of operations | Gauge | |`vault.core.seal-internal`| This measures the number of internal seal operations | Number of operations | Gauge | |`vault.core.step_down`| This measures the number of cluster leadership step downs | Number of stepdowns | Summary | |`vault.core.unseal`| This measures the number of unseal operations | Number of operations | Summary | |`vault.runtime.alloc_bytes` | This measures the number of bytes allocated by the Vault process. This may burst from time to time but should return to a steady state value.| Number of bytes | Gauge | |`vault.runtime.free_count`| This measures the number of `free` operations | Number of operations | Gauge | |`vault.runtime.heap_objects`| This measures the number of objects on the heap and is a good general memory pressure indicator | Number of heap objects | Gauge | |`vault.runtime.malloc_count`| This measures the number of `malloc` operations | Number of operations | Gauge | |`vault.runtime.num_goroutines`| This measures the number of goroutines and serves as a general load indicator | Number of goroutines| Gauge | |`vault.runtime.sys_bytes`| This measures the number of bytes allocated to Vault and includes what is being used by the heap and what has been reclaimed but not given back| Number of bytes | Gauge | |`vault.runtime.total_gc_pause_ns` | This measures the total garbage collector pause time since the Vault instance was last started | Nanosecond | Summary | | `vault.runtime.total_gc_runs` | Total number of garbage collection runs since the Vault instance was last started | Number of operations | Gauge | ### Policy and Token Metrics These metrics relate to policies and tokens. | Metric | Description | Unit | Type | | ---------------- | ----------------------------------| ---- | ---- | `vault.expire.fetch-lease-times`| This measures the number of lease time fetch operations | Number of operations | Gauge | `vault.expire.fetch-lease-times-by-token`| This measures the number of operations which compute lease times by token | Number of operations | Gauge | `vault.expire.num_leases`| This measures the number of expired leases | Number of expired leases | Gauge | `vault.expire.revoke`| This measures the number of revoke operations | Number of operations | Counter | `vault.expire.revoke-force`| This measures the number of forced revoke operations | Number of operations | Counter | `vault.expire.revoke-prefix`| This measures the number of operations used to revoke all secrets with a given prefix | Number of operations | Counter | `vault.expire.revoke-by-token`| This measures the number of operations used to revoke all secrets issued with a given token | Number of operations | Counter | `vault.expire.renew`| This measures the number of renew operations | Number of operations | Counter | `vault.expire.renew-token`| This measures the number of renew token operations to renew a token which does not need to invoke a logical backend | Number of operations | Gauge | `vault.expire.register`| This measures the number of register operations which take a request and response with an associated lease and register a lease entry with lease ID | Number of operations | Gauge | `vault.expire.register-auth`| This measures the number of register auth operations which create lease entries without lease ID | Number of operations | Gauge | `vault.policy.get_policy`| This measures the number of policy get operations | Number of operations | Counter | `vault.policy.list_policies`| This measures the number of policy list operations | Number of operations | Counter | `vault.policy.delete_policy`| This measures the number of policy delete operations | Number of operations | Counter | `vault.policy.set_policy`| This measures the number of policy set operations | Number of operations | Gauge | `vault.token.create`| This measures the number of token create operations | Number of operations | Gauge | `vault.token.createAccessor`| This measures the number of Token ID identifier operations | Number of operations | Gauge | `vault.token.lookup`| This measures the number of token lookups | Number of lookups | Counter | `vault.token.revoke`| This measures the number of token revocation operations | Number of operations | Gauge | `vault.token.revoke-tree`| This measures the number of revoke tree operations | Number of operations | Gauge | `vault.token.store`| This measures the number of operations to store an updated token entry without writing to the secondary index | Number of operations | Gauge | ### Auth Method Metrics These metrics relate to supported auth methods. | Metric | Description | Unit | Type | | ---------------- | ----------------------------------| ---- | ---- | | `vault.rollback.attempt.auth-token-` | This measures the number of rollback operations attempted for authentication tokens backend | Number of operations | Summary | | `vault.rollback.attempt.auth-ldap-` | This measures the number of rollback operations attempted for the LDAP auth method | Number of operations | Summary | | `vault.rollback.attempt.cubbyhole-` | This measures the number of rollback operations attempted for the cubbyhole auth method | Number of operations | Summary | | `vault.rollback.attempt.secret-` | This measures the number of rollback operations attempted for the kv secret backend | Number of operations | Summary | | `vault.rollback.attempt.sys-` | This measures the number of rollback operations attempted for the sys backend | Number of operations | Summary | | `vault.route.rollback.auth-ldap-` | This measures the number of rollback operations for the LDAP auth method | Number of operations | Summary | | `vault.route.rollback.auth-token-` | This measures the number of rollback operations for the authentication tokens backend | Number of operations | Summary | | `vault.route.rollback.cubbyhole-` | This measures the number of rollback operations for the cubbyhole auth method | Number of operations | Summary | | `vault.route.rollback.secret-` | This measures the number of rollback operations for the kv secret backend | Number of operations | Summary | | `vault.route.rollback.sys-` | This measures the number of rollback operations for the sys backend | Number of operations | Summary | ### Storage Backend Metrics These metrics relate to supported storage backends. | Metric | Description | Unit | Type | | ---------------- | ----------------------------------| ---- | ---- | |`vault.azure.put` | This measures the number of put operations against the Azure storage backend | Number of operations | Gauge | |`vault.azure.get` | This measures the number of get operations against the Azure storage backend | Number of operations | Gauge | |`vault.azure.delete` | This measures the number of delete operations against the Azure storage backend | Number of operations | Gauge | |`vault.azure.list` | This measures the number of list operations against the Azure storage backend | Number of operations | Gauge | |`vault.consul.put` | This measures the number of put operations against the Consul storage backend | Number of operations | Gauge | |`vault.consul.get` | This measures the number of get operations against the Consul storage backend | Number of operations | Gauge | |`vault.consul.delete` | This measures the number of delete operations against the Consul storage backend | Number of operations | Gauge | |`vault.consul.list` | This measures the number of list operations against the Consul storage backend | Number of operations | Gauge | |`vault.dynamodb.put` | This measures the number of put operations against the DynamoDB storage backend | Number of operations | Gauge | |`vault.dynamodb.get` | This measures the number of get operations against the DynamoDB storage backend | Number of operations | Gauge | |`vault.dynamodb.delete` | This measures the number of delete operations against the DynamoDB storage backend | Number of operations | Gauge | |`vault.dynamodb.list` | This measures the number of list operations against the DynamoDB storage backend | Number of operations | Gauge | |`vault.etcd.put` | This measures the number of put operations against the etcd storage backend | Number of operations | Gauge | |`vault.etcd.get` | This measures the number of get operations against the etcd storage backend | Number of operations | Gauge | |`vault.etcd.delete` | This measures the number of delete operations against the etcd storage backend | Number of operations | Gauge | |`vault.etcd.list` | This measures the number of list operations against the etcd storage backend | Number of operations | Gauge | |`vault.gcs.put` | This measures the number of put operations against the Google Cloud Storage backend | Number of operations | Gauge | |`vault.gcs.get` | This measures the number of get operations against the Google Cloud Storage backend | Number of operations | Gauge | |`vault.gcs.delete` | This measures the number of delete operations against the Google Cloud Storage backend | Number of operations | Gauge | |`vault.gcs.list` | This measures the number of list operations against the Google Cloud Storage backend | Number of operations | Gauge | |`vault.mysql.put` | This measures the number of put operations against the MySQL backend | Number of operations | Gauge | |`vault.mysql.get` | This measures the number of get operations against the MySQL backend | Number of operations | Gauge | |`vault.mysql.delete` | This measures the number of delete operations against the MySQL backend | Number of operations | Gauge | |`vault.mysql.list` | This measures the number of list operations against the MySQL backend | Number of operations | Gauge | |`vault.postgres.put` | This measures the number of put operations against the PostgreSQL backend | Number of operations | Gauge | |`vault.postgres.get` | This measures the number of get operations against the PostgreSQL backend | Number of operations | Gauge | |`vault.postgres.delete` | This measures the number of delete operations against the PostgreSQL backend | Number of operations | Gauge | |`vault.postgres.list` | This measures the number of list operations against the PostgreSQL backend | Number of operations | Gauge | |`vault.s3.put` | This measures the number of put operations against the Amazon S3 backend | Number of operations | Gauge | |`vault.s3.get` | This measures the number of get operations against the Amazon S3 backend | Number of operations | Gauge | |`vault.s3.delete` | This measures the number of delete operations against the Amazon S3 backend | Number of operations | Gauge | |`vault.s3.list` | This measures the number of list operations against the Amazon S3 backend | Number of operations | Gauge | |`vault.swift.put` | This measures the number of put operations against the OpenStack Swift backend | Number of operations | Gauge | |`vault.swift.get` | This measures the number of get operations against the OpenStack Swift backend | Number of operations | Gauge | |`vault.swift.delete` | This measures the number of delete operations against the OpenStack Swift backend | Number of operations | Gauge | |`vault.swift.list` | This measures the number of list operations against the OpenStack Swift backend | Number of operations | Gauge | |`vault.zookeeper.put` | This measures the number of put operations against the ZooKeeper backend | Number of operations | Gauge | |`vault.zookeeper.get` | This measures the number of get operations against the ZooKeeper backend | Number of operations | Gauge | |`vault.zookeeper.delete` | This measures the number of delete operations against the ZooKeeper backend | Number of operations | Gauge | |`vault.zookeeper.list` | This measures the number of list operations against the ZooKeeper backend | Number of operations | Gauge |