251 lines
9.8 KiB
Plaintext
251 lines
9.8 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Integrated Storage
|
|
sidebar_title: Integrated Storage
|
|
description: Learn about the integrated raft storage in Vault.
|
|
---
|
|
|
|
# Integrated Storage
|
|
|
|
Vault supports a number of storage options for the durable storage of Vault's
|
|
information. As of Vault 1.4 an integrated storage option is offered. This
|
|
storage backend does not rely on any third party systems, implements high
|
|
availability semantics, supports Enterprise Replication features, and provides
|
|
backup/restore workflows.
|
|
|
|
The integrated storage option stores Vault's data on the server's filesystem and
|
|
uses a consensus protocol to replicate data to each server in the cluster. More
|
|
information on the internals of integrated storage can be found in the
|
|
[integrated storage internals
|
|
documentation](/docs/internals/integrated-storage/). Additionally, the
|
|
[Configuration](/docs/configuration/storage/raft/) docs can help in configuring
|
|
Vault to use integrated storage.
|
|
|
|
The sections below go into various details on how to operate Vault with
|
|
integrated storage.
|
|
|
|
## Cluster Membership
|
|
|
|
This section will outline how to bootstrap and manage a cluster of Vault nodes
|
|
running integrated storage.
|
|
|
|
Integrated storage is bootstrapped during the [initialization
|
|
process](https://learn.hashicorp.com/vault/getting-started/deploy#initializing-the-vault),
|
|
and results in a cluster of size 1. Depending on the [desired deployment
|
|
size](/docs/internals/integrated-storage/#deployment-table), nodes can be joined
|
|
to the active Vault node.
|
|
|
|
### Joining Nodes
|
|
|
|
Joining is the process of taking an uninitialized Vault node and making it a
|
|
member of an existing cluster. In order to authenticate the new node to the
|
|
cluster it must use the same seal mechanism. If using a Auto Unseal the node
|
|
must be configured to use the same KMS provider and Key as the cluster it's
|
|
attempting to join. If using a Shamir seal the unseal keys must be provided to
|
|
the new node before the join process can complete. Once a node has successfully
|
|
joined, data from the active node can begin to replicate to it. Once a node has
|
|
been joined it cannot be re-joined to a different cluster.
|
|
|
|
You can either join the node automatically via the config file or manually through the
|
|
API (both methods described below). When joining a node, the API address of the leader node must be used. We
|
|
recommend setting the [api_addr](/docs/concepts/ha/#direct-access) configuration
|
|
option on all nodes to make joining simpler.
|
|
|
|
#### retry_join Configuration
|
|
|
|
This method enables setting one, or more, target leader nodes in the config file. When an
|
|
uninitialized Vault server starts up it will attempt to join each potential
|
|
leader that is defined, retrying until successful. When one of the specified
|
|
leaders become active this node will successfully join. When using Shamir seal,
|
|
the joined nodes will still need to be unsealed manually. When using Auto Unseal
|
|
the node will be able to join and unseal automatically.
|
|
|
|
An example [retry_join](/docs/configuration/storage/raft/#retry_join-stanza)
|
|
config can be seen below:
|
|
|
|
```hcl
|
|
storage "raft" {
|
|
path = "/var/raft/"
|
|
node_id = "node3"
|
|
retry_join {
|
|
leader_api_addr = "https://node1.vault.local:8200"
|
|
}
|
|
retry_join {
|
|
leader_api_addr = "https://node2.vault.local:8200"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Join from the CLI
|
|
|
|
Alternatively you can use the [join CLI
|
|
command](/docs/commands/operator/raft/#join) or the API to join a node. The
|
|
active node's API address will need to be specified:
|
|
|
|
```shell
|
|
$ vault operator raft join https://node1.vault.local:8200
|
|
```
|
|
|
|
#### Non-Voting Nodes (Enterprise Only)
|
|
|
|
Nodes that are joined to a cluster can be specified as non-voters. A non-voting
|
|
node has all of Vault's data replicated to it, but does not contribute to the
|
|
quorum count. This can be used in conjunction with [Performance
|
|
Standby](/docs/enterprise/performance-standby/) nodes to add read scalability to
|
|
a cluster in cases where a high volume of reads to servers are needed.
|
|
|
|
```shell
|
|
$ vault operator raft join -non-voter https://node1.vault.local:8200
|
|
```
|
|
|
|
### Removing Peers
|
|
|
|
Removing a peer node is a necessary step when you no longer want the node in the
|
|
cluster. This could happen if the node is rotated for a new one, the hostname
|
|
permanently changes and can no longer be accessed, you're attempting to shrink
|
|
the size of the cluster, or for many other reasons. Removing the peer will
|
|
ensure the cluster stays at the desired size, and that quorum is maintained.
|
|
|
|
To remove the peer you can issue a
|
|
[remove-peer](/docs/commands/operator/raft/#remove-peer) command and provide the
|
|
node ID you wish to remove:
|
|
|
|
```shell
|
|
$ vault operator raft remove-peer node1
|
|
Peer removed successfully!
|
|
```
|
|
|
|
### Listing Peers
|
|
|
|
To see the current peer set for the cluster you can issue a
|
|
[list-peers](/docs/commands/operator/raft/#list-peers) command. All the voting
|
|
nodes that are listed here contribute to the quorum and a majority must be alive
|
|
for integrated storage to continue to operate.
|
|
|
|
```shell
|
|
$ vault operator raft list-peers
|
|
Node Address State Voter
|
|
---- ------- ----- -----
|
|
node1 node1.vault.local:8201 follower true
|
|
node2 node2.vault.local:8201 follower true
|
|
node3 node3.vault.local:8201 leader true
|
|
```
|
|
|
|
## Server-to-Server Communication
|
|
|
|
Once nodes are joined to one another they begin to communicate using mTLS over
|
|
Vault's cluster port. The cluster port defaults to `8201`. The TLS information
|
|
is exchanged at join time and is rotated on a cadence.
|
|
|
|
A requirement for integrated storage is that the
|
|
[cluster_addr](/docs/concepts/ha/#per-node-cluster-address) configuration option
|
|
is set. This allows Vault to assign an address to the node ID at join time.
|
|
|
|
## Outage Recovery
|
|
|
|
### Quorum Maintained
|
|
|
|
This section outlines the steps to take when a single server or multiple servers
|
|
are in a failed state but quorum is still maintained. This means the remaining
|
|
alive servers are still operational, can elect a leader, and are able to process
|
|
write requests.
|
|
|
|
If the failed server is recoverable, the best option is to bring it back online
|
|
and have it reconnect to the cluster with the same host address. This will return
|
|
the cluster to a fully healthy state.
|
|
|
|
If this is impractical, you need to remove the failed server. Usually, you can
|
|
issue a remove-peer command to remove the failed server if it's still a member
|
|
of the cluster.
|
|
|
|
If the remove-peer command isn't possible or you'd rather manually re-write the
|
|
cluster membership a `raft/peers.json` file can be written to the configured
|
|
data directory.
|
|
|
|
### Quorum Lost
|
|
|
|
In the event that multiple servers are lost, causing a loss of quorum and a
|
|
complete outage, partial recovery is still possible.
|
|
|
|
If the failed servers are recoverable, the best option is to bring them back
|
|
online and have them reconnect to the cluster using the same host addresses.
|
|
This will return the cluster to a fully healthy state.
|
|
|
|
If the failed servers are not recoverable, partial recovery is possible using
|
|
data on the remaining servers in the cluster. There may be data loss in this
|
|
situation because multiple servers were lost, so information about what's
|
|
committed could be incomplete. The recovery process implicitly commits all
|
|
outstanding Raft log entries, so it's also possible to commit data that was
|
|
uncommitted before the failure.
|
|
|
|
See the section below on manual recovery using peers.json for details of the
|
|
recovery procedure. You include only the remaining servers in the
|
|
raft/peers.json recovery file. The cluster should be able to elect a leader once
|
|
the remaining servers are all restarted with an identical raft/peers.json
|
|
configuration.
|
|
|
|
Any servers you introduce later can be fresh with totally clean data
|
|
directories and joined using Vault's join command.
|
|
|
|
In extreme cases, it should be possible to recover with just a single remaining
|
|
server by starting that single server with itself as the only peer in the
|
|
raft/peers.json recovery file.
|
|
|
|
### Manual Recovery Using peers.json
|
|
|
|
Using raft/peers.json for recovery can cause uncommitted Raft log entries to be
|
|
implicitly committed, so this should only be used after an outage where no other
|
|
option is available to recover a lost server. Make sure you don't have any
|
|
automated processes that will put the peers file in place on a periodic basis.
|
|
|
|
To begin, stop all remaining servers.
|
|
|
|
The next step is to go to the [configured data
|
|
path](/docs/configuration/storage/raft/#path) of each Vault server. Inside that
|
|
directory, there will be a raft/ sub-directory. We need to create a
|
|
raft/peers.json file. The file should be formatted as a JSON array containing
|
|
the node ID, address:port, and suffrage information of each Vault server you
|
|
wish to be in the cluster.
|
|
|
|
```json
|
|
[
|
|
{
|
|
"id": "node1",
|
|
"address": "node1.vault.local:8201",
|
|
"non_voter": false
|
|
},
|
|
{
|
|
"id": "node2",
|
|
"address": "node2.vault.local:8201",
|
|
"non_voter": false
|
|
},
|
|
{
|
|
"id": "node3",
|
|
"address": "node3.vault.local:8201",
|
|
"non_voter": false
|
|
}
|
|
]
|
|
```
|
|
|
|
* `id` `(string: <required>)` - Specifies the node ID of the server. This can be
|
|
found in the config file, or inside the node-id file in the server's data
|
|
directory if it was auto-generated.
|
|
* `address` `(string: <required>)` - Specifies the host and port of the server. The
|
|
port is the server's cluster port.
|
|
* `non_voter` `(bool: <false>)` - This controls whether the server is a non-voter.
|
|
If omitted, it will default to false, which is typical for most clusters. This
|
|
is an enterprise only feature.
|
|
|
|
Create entries for all servers. You must confirm that servers you do not
|
|
include here have indeed failed and will not later rejoin the cluster. Ensure
|
|
that this file is the same across all remaining server nodes.
|
|
|
|
At this point, you can restart all the remaining servers. The cluster should be
|
|
in an operable state again. One of the nodes should claim leadership and become
|
|
active.
|
|
|
|
### Other Recovery Methods
|
|
|
|
For other, non-quorum related recovery [Vault's
|
|
recovery](/docs/concepts/recovery-mode/) mode can be used.
|