2020-08-11 19:39:44 +00:00
|
|
|
---
|
|
|
|
layout: docs
|
|
|
|
page_title: 'Commands: operator debug'
|
|
|
|
description: |
|
|
|
|
Build an archive of debug data.
|
|
|
|
---
|
|
|
|
|
|
|
|
# Command: operator debug
|
|
|
|
|
|
|
|
The `operator debug` command builds an archive containing Nomad cluster
|
|
|
|
configuration and state information, Nomad server and client node
|
|
|
|
logs, and pprof profiles from the selected servers and client nodes.
|
|
|
|
|
|
|
|
If no selection option is specified, the debug archive contains only
|
|
|
|
cluster meta information.
|
|
|
|
|
|
|
|
## Usage
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
nomad operator debug [options]
|
|
|
|
```
|
|
|
|
|
|
|
|
This command accepts comma separated `server-id` and `node-id` IDs for
|
|
|
|
monitoring and pprof profiling. If IDs are provided, the command will
|
|
|
|
monitor logs for the `duration`, saving a snapshot of Nomad state
|
|
|
|
every `interval`. Captured logs and configurations are subjected to
|
|
|
|
redaction, but may still contain sensitive information and the archive
|
|
|
|
contents should be reviewed before sharing.
|
|
|
|
|
|
|
|
If an `output` path is provided, `debug` will create a timestamped
|
|
|
|
directory in that path instead of an archive. By default, the command
|
|
|
|
creates a compressed tar archive in the current directory.
|
|
|
|
|
|
|
|
Consul and Vault status and version information are included if
|
|
|
|
configured.
|
|
|
|
|
2020-11-19 21:38:08 +00:00
|
|
|
If ACLs are enabled, this command will require a token with the 'node:read'
|
|
|
|
capability to run. In order to collect information, the token will also
|
|
|
|
require the 'agent:read' and 'operator:read' capabilities, as well as the
|
2020-12-08 04:55:11 +00:00
|
|
|
'list-jobs' capability for all namespaces. To collect pprof profiles the
|
2021-03-31 13:43:17 +00:00
|
|
|
token will also require 'agent:write', or enable_debug configuration set to
|
2020-12-08 04:55:11 +00:00
|
|
|
true.
|
2020-11-19 21:38:08 +00:00
|
|
|
|
2020-08-11 19:39:44 +00:00
|
|
|
## General Options
|
|
|
|
|
2020-11-19 18:02:48 +00:00
|
|
|
@include 'general_options_no_namespace.mdx'
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
## Debug Options
|
|
|
|
|
|
|
|
- `-duration=2m`: Set the duration of the log monitor command.
|
|
|
|
Defaults to `"2m"`. Logs will be captured from specified servers and
|
|
|
|
nodes at `log-level`.
|
|
|
|
|
|
|
|
- `-interval=2m`: The interval between snapshots of the Nomad state.
|
2020-09-30 13:48:40 +00:00
|
|
|
If unspecified, only one snapshot is captured.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-log-level=DEBUG`: The log level to monitor. Defaults to `DEBUG`.
|
|
|
|
|
2021-03-31 13:43:17 +00:00
|
|
|
- `-max-nodes=<count>`: Cap the maximum number of client nodes included
|
|
|
|
in the capture. Defaults to 10, set to 0 for unlimited.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
2020-12-08 04:55:11 +00:00
|
|
|
- `-node-class=<node-class>`: Filter client nodes based on node class.
|
|
|
|
|
|
|
|
- `-node-id=<node1>,<node2>`: Comma separated list of Nomad client node ids,
|
2021-03-31 13:43:17 +00:00
|
|
|
to monitor for logs and include pprof profiles. Accepts id prefixes, and
|
2020-12-08 04:55:11 +00:00
|
|
|
"all" to select all nodes (up to count = max-nodes).
|
|
|
|
|
2021-03-17 19:13:04 +00:00
|
|
|
- `-pprof-duration=<duration>`: Duration for pprof collection. Defaults to 1s.
|
2020-12-08 04:55:11 +00:00
|
|
|
|
|
|
|
- `-server-id=s1,s2`: Comma separated list of Nomad server names, "leader", or
|
|
|
|
"all" to monitor for logs and include pprof profiles.
|
|
|
|
|
2021-03-17 19:13:04 +00:00
|
|
|
- `-stale=<true|false>`: If "false", the default, get membership data from the
|
2020-12-08 04:55:11 +00:00
|
|
|
cluster leader. If the cluster is in an outage unable to establish
|
|
|
|
leadership, it may be necessary to get the configuration from a non-leader
|
|
|
|
server.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-output=path`: Path to the parent directory of the output
|
2020-09-30 13:48:40 +00:00
|
|
|
directory. Defaults to the current directory. If specified, no
|
|
|
|
archive is built.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-consul-http-addr=<addr>`: The address and port of the Consul HTTP
|
|
|
|
agent. Overrides the `CONSUL_HTTP_ADDR` environment variable.
|
|
|
|
|
|
|
|
- `-consul-token=<token>`: Token used to query Consul. Overrides the
|
2020-09-30 13:48:40 +00:00
|
|
|
`CONSUL_HTTP_TOKEN` environment variable and the Consul token
|
|
|
|
file.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-consul-token-file=<path>`: Path to the Consul token file. Overrides the `CONSUL_HTTP_TOKEN_FILE`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-consul-client-cert=<path>`: Path to the Consul client cert file. Overrides the `CONSUL_CLIENT_CERT`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-consul-client-key=<path>`: Path to the Consul client key file. Overrides the `CONSUL_CLIENT_KEY`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-consul-ca-cert=<path>`: Path to a CA file to use with Consul. Overrides the `CONSUL_CACERT`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable and the Consul CA path.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-consul-ca-path=<path>`: Path to a directory of PEM encoded CA cert files to verify the Consul
|
2020-09-30 13:48:40 +00:00
|
|
|
certificate. Overrides the `CONSUL_CAPATH` environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-vault-address=<addr>`: The address and port of the Vault HTTP agent. Overrides the `VAULT_ADDR`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-vault-token=<token>`: Token used to query Vault. Overrides the `VAULT_TOKEN` environment
|
2020-09-30 13:48:40 +00:00
|
|
|
variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-vault-client-cert=<path>`: Path to the Vault client cert file. Overrides the `VAULT_CLIENT_CERT`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-vault-client-key=<path>`: Path to the Vault client key file. Overrides the `VAULT_CLIENT_KEY`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-vault-ca-cert=<path>`: Path to a CA file to use with Vault. Overrides the `VAULT_CACERT`
|
2020-09-30 13:48:40 +00:00
|
|
|
environment variable and the Vault CA path.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
- `-vault-ca-path=<path>`: Path to a directory of PEM encoded CA cert files to verify the Vault
|
2020-09-30 13:48:40 +00:00
|
|
|
certificate. Overrides the `VAULT_CAPATH` environment variable.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
## Output
|
|
|
|
|
2021-03-31 13:43:17 +00:00
|
|
|
This command prints a summary of the capture and the name of the timestamped
|
2020-12-08 04:55:11 +00:00
|
|
|
archive file produced.
|
2020-08-11 19:39:44 +00:00
|
|
|
|
|
|
|
## Examples
|
|
|
|
|
|
|
|
```shell-session
|
2020-12-08 04:55:11 +00:00
|
|
|
$ nomad operator debug -duration 5s -interval 5s -server-id all -node-id b5,20
|
|
|
|
Starting debugger...
|
|
|
|
|
|
|
|
Servers: (3/3) [server1.global server2.global server3.global]
|
|
|
|
Clients: (2/3) [b547cd3a-085f-68c2-55f4-e99beebb0433 20c0964b-72cc-4083-87fe-ec6905b6230a]
|
|
|
|
Interval: 5s
|
|
|
|
Duration: 5s
|
|
|
|
|
|
|
|
Capturing cluster data...
|
|
|
|
Capture interval 0000
|
|
|
|
Capture interval 0001
|
|
|
|
Capture interval 0002
|
|
|
|
Capture interval 0003
|
|
|
|
Created debug archive: nomad-debug-2020-12-08-034455Z.tar.gz
|
|
|
|
```
|
|
|
|
|
|
|
|
```shell-session
|
|
|
|
$ nomad operator debug -duration 5s -interval 5s -server-id all -node-id all -max-nodes=1
|
|
|
|
Starting debugger...
|
|
|
|
|
|
|
|
Servers: (3/3) [server1.global server2.global server3.global]
|
|
|
|
Clients: (1/3) [b547cd3a-085f-68c2-55f4-e99beebb0433]
|
|
|
|
Max node count reached (1)
|
|
|
|
Interval: 5s
|
|
|
|
Duration: 5s
|
|
|
|
|
|
|
|
Capturing cluster data...
|
2020-08-11 19:39:44 +00:00
|
|
|
Capture interval 0000
|
|
|
|
Capture interval 0001
|
|
|
|
Capture interval 0002
|
|
|
|
Capture interval 0003
|
2020-12-08 04:55:11 +00:00
|
|
|
Created debug archive: nomad-debug-2020-12-08-034113Z.tar.gz
|
2020-08-11 19:39:44 +00:00
|
|
|
```
|