229 lines
9.9 KiB
Plaintext
229 lines
9.9 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: operator diagnose - Command
|
|
description: |-
|
|
"vault operator diagnose" is a new operator-centric command, focused on providing a clear description
|
|
of what is working in Vault, and what is not working. The command focuses on why Vault cannot serve requests,
|
|
but will also warn on configurations or statuses that it deems to be unsafe in some way.
|
|
|
|
---
|
|
|
|
# operator diagnose
|
|
|
|
The operator diagnose command should be used primarily when vault is down or
|
|
partially inoperational. The command can be used safely regardless of the state
|
|
vault is in, but may return meaningless results for some of the test cases if the
|
|
vault server is already running.
|
|
|
|
Note: if you run the diagnose command proactively, either before a server
|
|
starts or while a server is operational, please consult the documentation
|
|
on the individual checks below to see which checks are returning false error
|
|
messages or warnings.
|
|
|
|
## Usage
|
|
|
|
The following flags are available in addition to the [standard set of
|
|
flags](/docs/commands) included on all commands.
|
|
|
|
### Output Options
|
|
|
|
- `-format` `(string: "table")` - Print the output in the given format. Valid
|
|
formats are "table", "json", or "yaml". This can also be specified via the
|
|
`VAULT_FORMAT` environment variable.
|
|
|
|
#### Output Layout
|
|
|
|
The operator diagnose command will output a set of lines in the CLI.
|
|
Each line will begin with a prefix in parenthesis. These are:.
|
|
|
|
- `[ success ]` - Denotes that the check was successful.
|
|
- `[ warning ]` - Denotes that the check has passed, but that there may be potential
|
|
issues to look into that may relate to the issues vault is experiencing. Diagnose warns
|
|
frequently. These warnings are meant to serve as starting points in the debugging process.
|
|
- `[ failure ]` - Denotes that the check has failed. Failures are critical issues in the eyes
|
|
of the diagnose command.
|
|
|
|
In addition to these prefixed lines, there may be output lines that are not prefixed, but are
|
|
color-coded purple. These are advice lines from Diagnose, and are meant to offer general guidance
|
|
on how to go about fixing potential warnings or failures that may arise.
|
|
|
|
Warn or fail prefixes in nested checks will bubble up to the parent if the prefix superceeds the
|
|
parent prefix. Fail superceeds warn, and warn superceeds ok. For example, if the TLS checks under
|
|
the Storage check fails, the `[ failure ]` prefix will bubble up to the Storage check.
|
|
|
|
### Command Options
|
|
|
|
- `-config` `(string; "")` - The path to the vault configuration file used by
|
|
the vault server on startup.
|
|
|
|
### Diagnose Checks
|
|
|
|
The following section details the various checks that Diagnose runs. Check names in documentation
|
|
will be separated by slashes to denote that they are nested, when applicable. For example, a check
|
|
documented as `A / B` will show up as `B` in the `operator diagnose` output, and will be nested
|
|
(indented) under `A`.
|
|
|
|
#### Vault Diagnose
|
|
|
|
`Vault Diagnose` is the top level check that contains the rest of the checks. It will report the status
|
|
of the check
|
|
|
|
#### Check Operating System / Check Open File Limit
|
|
|
|
`Check Open File Limit` verifies that the open file limit value is set high enough for vault
|
|
to run effectively. We recommend setting these limits to at least 1024768.
|
|
|
|
This check will be skipped on openbsd, arm, and windows.
|
|
|
|
#### Check Operating System / Check Disk Usage
|
|
|
|
`Check Disk Usage` will report disk usage for each partition. For each partition on a prod host,
|
|
we recommend having at least 5% of the partition free to use, and at least 1 GB of space.
|
|
|
|
This check will be skipped on openbsd and arm.
|
|
|
|
#### Parse Configuration
|
|
|
|
`Parse Configuration` will check the vault server config file for syntax errors. It will check
|
|
for extra values in the configuration file, repeated stanzas, and stanzas that do not belong
|
|
in the configuration file (for example a "tcpp" listener as opposed to a tcp listener).
|
|
|
|
Currently, the `storage` stanza is not checked.
|
|
|
|
#### Check Storage / Create Storage Backend
|
|
|
|
`Create Storage Backend` ensures that the storage stanza configured in the vault server config
|
|
has enough information to create a storage object internally. Common errors will have to do
|
|
with misconfigured fields in the storage stanza.
|
|
|
|
#### Check Storage / Check Consul TLS
|
|
|
|
`Check Consul TLS` verifies TLS information included in the storage stanza if the storage type
|
|
is consul. If a certificate chain is provided, Diagnose parses the root, intermediate, and leaf
|
|
certificates, and checks each one for correctness.
|
|
|
|
#### Check Storage / Check Consul Direct Storage Access
|
|
|
|
`Check Consul Direct Storage Access` is a consul-specific check that ensures Vault is not accessing
|
|
the consul server directly, but rather through a local agent.
|
|
|
|
#### Check Storage / Check Raft Folder Permissions
|
|
|
|
`Check Raft Folder Permissions` computes the permissions on the raft folder, checks that a boltDB file
|
|
has been initialized within the folder previously, and ensures that the folder is not too permissive, but
|
|
at the same time has enough permissions to be used. The raft folder should not have `other` permissions, but
|
|
should have `group rw` or `owner rw`, depending on different setups. This check also warns if it detects a
|
|
symlink being used.
|
|
|
|
Note that this check will warn that a raft file has not been created if diagnose is run without any
|
|
pre-existing server runs.
|
|
|
|
This check will be skipped on windows.
|
|
|
|
#### Check Storage / Check Raft Folder Ownership
|
|
|
|
`Check Raft Folder Ownership` ensures that vault does not need to run as root to access the boltDB folder.
|
|
|
|
Note that this check will warn that a raft file has not been created if diagnose is run without any
|
|
pre-existing server runs.
|
|
|
|
This check will be skipped on windows.
|
|
|
|
#### Check Storage / Check For Raft Quorum
|
|
|
|
`Check For Raft Quorum` uses the FSM to ensure that there were an odd number of voters in the raft quorum when
|
|
vault was last running.
|
|
|
|
Note that this check will warn that there are 0 voters if diagnose is run without any pre-existing server runs.
|
|
|
|
#### Check Storage / Check Storage Access
|
|
|
|
`Check Storage Access` will try to write a dud value, named `diagnose/latency/<uuid>`, to storage.
|
|
Ensure that there is no important data at this location before running diagnose, as this check
|
|
will overwrite that data. This check will then try to list and read the value it wrote to ensure
|
|
the name and value is as expected.
|
|
|
|
`Check Storage Access` will warn if any operation takes longer than 100ms, and error out if the
|
|
entire check takes longer than 30s.
|
|
|
|
#### Check Service Discovery / Check Consul Service Discovery TLS
|
|
|
|
`Check Consul Service Discovery TLS` verifies TLS information included in the service discovery
|
|
stanza if the storage type is consul. If a certificate chain is provided, Diagnose parses
|
|
the root, intermediate, and leaf certificates, and checks each one for correctness.
|
|
|
|
#### Check Service Discovery / Check Consul Direct Service Discovery
|
|
|
|
`Check Consul Direct Service Discovery` is a consul-specific check that ensures Vault
|
|
is not accessing the consul server directly, but rather through a local agent.
|
|
|
|
#### Create Vault Server Configuration Seals
|
|
|
|
`Create Vault Server Configuration Seals` creates seals from the vault configuration
|
|
stanza and verifies they can be initialized and finalized.
|
|
|
|
#### Check Transit Seal TLS
|
|
|
|
`Check Transit Seal TLS` checks the TLS client certificate, key, and CA certificate
|
|
provided in a transit seal stanza (if one exists) for correctness.
|
|
|
|
#### Create Core Configuration / Initialize Randomness for Core
|
|
|
|
`Initialize Randomness for Core` ensures that vault has access to the randReader that
|
|
the vault core uses.
|
|
|
|
#### HA Storage
|
|
|
|
This check and any nested checks will be the same as the `Check Storage` checks.
|
|
The only difference is that the checks here will be run on whatever is specified in the
|
|
`ha_storage` section of the vault configuration, as opposed to the `storage` section.
|
|
|
|
#### Determine Redirect Address
|
|
|
|
Ensures that one of the `VAULT_API_ADDR`, `VAULT_REDIRECT_ADDR`, or `VAULT_ADVERTISE_ADDR`
|
|
environment variables are set, or that the redirect address is specified in the vault
|
|
configuration.
|
|
|
|
#### Check Cluster Address
|
|
|
|
Parses the cluster address from the `VAULT_CLUSTER_ADDR` environment variable, or from the
|
|
redirect address or cluster address specified in the vault configuration, and checks that
|
|
the address is of the form `host:port`.
|
|
|
|
#### Check Core Creation
|
|
|
|
`Check Core Creation` verifies the logical configuration checks that vault does when it
|
|
creates a core object. These are runtime checks, meaning any errors thrown by this diagnose
|
|
test will also be thrown by the vault server itself when it is run.
|
|
|
|
#### Check For Autoloaded License
|
|
|
|
`Check For Autoloaded License` is an enterprise diagnose check, which verifies that vault
|
|
has access to a valid autoloaded license that will not expire in the next 30 days.
|
|
|
|
#### Start Listeners / Check Listener TLS
|
|
|
|
`Check Listener TLS` verifies the server certificate file and key are valid and matching.
|
|
It also checks the client CA file, if one is provided, for a valid certificate, and performs
|
|
the standard runtime listener checks on the listener configuration stanza, such as verifying
|
|
that the minimum and maximum TLS versions are within the bounds of what vault supports.
|
|
|
|
Like all the other Diagnose TLS checks, it will warn if any of the certificates provided are
|
|
set to expire within the next month.
|
|
|
|
#### Start Listeners / Create Listeners
|
|
|
|
`Create Listeners` uses the listener configuration to initialize the listeners, erroring with
|
|
a server error if anything goes wrong.
|
|
|
|
#### Check Autounseal Encryption
|
|
|
|
`Check Autounseal Encryption` will initialize the barrier using the seal stanza, if the seal
|
|
type is not a shamir seal, and use it to encrypt and decrypt a dud value.
|
|
|
|
#### Check Server Before Runtime
|
|
|
|
`Check Server Before Runtime` achieves parity with the server run command, running through
|
|
the runtime code checks before the server is initialized to ensure that nothing fails.
|
|
This check will never fail without another diagnose check failing.
|