f3df55ad58
* Adding check-legacy-links-format workflow * Adding test-link-rewrites workflow * Updating docs-content-check-legacy-links-format hash * Migrating links to new format Co-authored-by: Kendall Strautman <kendallstrautman@gmail.com>
78 lines
3.4 KiB
Plaintext
78 lines
3.4 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Recovery Mode
|
|
description: Recovery mode allows for doing surgery on a Vault that won't start.
|
|
---
|
|
|
|
# Recovery Mode
|
|
|
|
Vault can be started using the `-recovery` flag to bring it up in Recovery Mode.
|
|
The main purpose of recovery mode is to allow direct access to storage in case
|
|
Vault isn't starting up due to some newly discovered bug. This probably won't
|
|
be helpful without a Vault expert on hand to advise.
|
|
|
|
Differences between recovery mode and regular Vault operation:
|
|
- none of the usual subsystems run, e.g. expiration, clustering, RPCs from other nodes
|
|
- instead of a regular unseal request, unseal a node by generating a recovery token
|
|
- all requests are to `sys/raw` and are authenticated using the recovery token
|
|
|
|
## Recovery process
|
|
|
|
The usual way recovery mode is used is:
|
|
- seal or stop all nodes in the cluster
|
|
- if using Integrated Storage, run `vault status` on each node to find the highest-index ones
|
|
(this will require they be running and sealed, as if unsealed a new leader might be
|
|
elected and writes could happen, confusing the issue)
|
|
- restart the target node in recovery mode
|
|
- generate a recovery token on that node
|
|
- use the recovery token to perform sys/raw requests to repair the node
|
|
- if using Integrated Storage, reform the raft cluster as described below
|
|
|
|
## Integrated storage for HA only (ha_storage)
|
|
|
|
If Integrated Storage is used in hybrid mode (i.e. for `ha_storage`),
|
|
recovery mode will not allow for changes to the Raft data but instead allow for
|
|
modification of the underlying physical data that is associated with Vault's
|
|
storage backend. This means that the notes regarding Integrated Storage in
|
|
this doc do not apply.
|
|
|
|
## Integrated storage
|
|
|
|
With Integrated Storage, not all nodes are equal. It's possible that some
|
|
nodes are further behind - i.e. haven't applied as many Raft logs. It is
|
|
important when choosing a node to use for recovery that it has the highest
|
|
AppliedIndex found in the cluster.
|
|
|
|
Each node's AppliedIndex value can be obtained by running `vault status` against
|
|
the node sealed nodes of the cluster after bringing it down.
|
|
|
|
## Recovery tokens
|
|
|
|
Recovery tokens are issued in much the same way as root tokens are generated,
|
|
only using a different endpoint, and the Vault node must be sealed first.
|
|
Unlike root tokens, the recovery token is not persisted, so if Vault
|
|
is restarted into recovery mode a new one must be generated.
|
|
|
|
Only a single recovery token can be generated. If lost, restart Vault and
|
|
generate a new one.
|
|
|
|
## Raw requests
|
|
|
|
Requests can be issued to `sys/raw` in just the same way as in regular Vault
|
|
server mode. The only difference is that in recovery mode, `X-Vault-Token`
|
|
must contain a recovery token instead of a service or batch token.
|
|
|
|
## Reform the raft cluster
|
|
|
|
Recovery mode Vault automatically resizes the cluster to size 1. This is
|
|
necessary because the Raft protocol won't allow changes to be made without a
|
|
quorum, and in recovery mode we wish to make changes using a single node.
|
|
|
|
This means that after having used recovery mode, part of the procedure for
|
|
returning to active service must include re-forming the raft cluster. There
|
|
are two ways to do so: either delete the vault data directory on the other nodes
|
|
and re-join them to the recovered node, or use the
|
|
[Manual Recovery Using peers.json](/vault/docs/concepts/integrated-storage#manual-recovery-using-peers-json)
|
|
approach to get all nodes to agree on what nodes are part of the cluster.
|
|
|