open-vault/website/content/docs/commands/operator/migrate.mdx

135 lines
5 KiB
Plaintext
Raw Normal View History

---
layout: docs
page_title: operator migrate - Command
description: >-
The "operator migrate" command copies data between storage backends to
facilitate
migrating Vault between configurations. It operates directly at the storage
level, with no decryption involved.
---
# operator migrate
The `operator migrate` command copies data between storage backends to facilitate
migrating Vault between configurations. It operates directly at the storage
level, with no decryption involved. Keys in the destination storage backend will
2018-12-04 16:49:42 +00:00
be overwritten, and the destination should _not_ be initialized prior to the
migrate operation. The source data is not modified, with the exception of a small lock
key added during migration.
This is intended to be an offline operation to ensure data consistency, and Vault
will not allow starting the server if a migration is in progress.
## Examples
Migrate all keys:
```shell-session
$ vault operator migrate -config migrate.hcl
2018-09-20T14:23:23.656-0700 [INFO ] copied key: data/core/seal-config
2018-09-20T14:23:23.657-0700 [INFO ] copied key: data/core/wrapping/jwtkey
2018-09-20T14:23:23.658-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/archive/metadata
2018-09-20T14:23:23.660-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/metadata/5kKFZ4YnzgNfy9UcWOzxxzOMpqlp61rYuq6laqpLQDnB3RawKpqi7yBTrawj1P
...
```
Migration is done in a consistent, sorted order. If the migration is halted or
exits before completion (e.g. due to a connection error with a storage backend),
it may be resumed from an arbitrary key prefix:
```shell-session
$ vault operator migrate -config migrate.hcl -start "data/logical/fd"
```
## Configuration
The `operator migrate` command uses a dedicated configuration file to specify the source
and destination storage backends. The format of the storage stanzas is identical
to that used to [configure Vault](/docs/configuration/storage),
with the only difference being that two stanzas are required: `storage_source` and `storage_destination`.
```hcl
storage_source "mysql" {
username = "user1234"
password = "secret123!"
database = "vault"
}
storage_destination "consul" {
address = "127.0.0.1:8500"
path = "vault/"
}
```
## Migrating to integrated raft storage
### Example Configuration
The below configuration will migrate away from Consul storage to integrated
raft storage. The raft data will be stored on the local filesystem in the
defined `path`. `node_id` can optionally be set to identify this node.
[cluster_addr](/docs/configuration#cluster_addr) must be set to the
cluster hostname of this node. For more configuration options see the [raft
storage configuration documentation](/docs/configuration/storage/raft).
raft: add support for using backend for ha_storage (#9193) * raft: initial work on raft ha storage support * add note on join * add todo note * raft: add support for bootstrapping and joining existing nodes * raft: gate bootstrap join by reading leader api address from storage * raft: properly check for raft-only for certain conditionals * raft: add bootstrap to api and cli * raft: fix bootstrap cli command * raft: add test for setting up new cluster with raft HA * raft: extend TestRaft_HA_NewCluster to include inmem and consul backends * raft: add test for updating an existing cluster to use raft HA * raft: remove debug log lines, clean up verifyRaftPeers * raft: minor cleanup * raft: minor cleanup * Update physical/raft/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/ha.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/ha.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/logical_system_raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * address feedback comments * address feedback comments * raft: refactor tls keyring logic * address feedback comments * Update vault/raft.go Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * address feedback comments * testing: fix import ordering * raft: rename var, cleanup comment line * docs: remove ha_storage restriction note on raft * docs: more raft HA interaction updates with migration and recovery mode * docs: update the raft join command * raft: update comments * raft: add missing isRaftHAOnly check for clearing out state set earlier * raft: update a few ha_storage config checks * Update command/operator_raft_bootstrap.go Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com> * raft: address feedback comments * raft: fix panic when checking for config.HAStorage.Type * Update vault/raft.go Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * Update website/pages/docs/commands/operator/raft.mdx Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * raft: remove bootstrap cli command * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * raft: address review feedback * raft: revert vendored sdk * raft: don't send applied index and node ID info if we're HA-only Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-06-23 19:04:13 +00:00
If the original configuration uses "raft" for `ha_storage` a different
`path` needs to be declared for the path in `storage_destination` and the new
configuration for the node post-migration.
```hcl
storage_source "consul" {
address = "127.0.0.1:8500"
path = "vault"
}
storage_destination "raft" {
path = "/path/to/raft/data"
node_id = "raft_node_1"
}
cluster_addr = "http://127.0.0.1:8201"
```
### Run the migration
Vault will need to be offline during the migration process. First, stop Vault.
Then, run the migration on the server you wish to become a the new Vault node.
```shell-session
$ vault operator migrate -config migrate.hcl
2018-09-20T14:23:23.656-0700 [INFO ] copied key: data/core/seal-config
2018-09-20T14:23:23.657-0700 [INFO ] copied key: data/core/wrapping/jwtkey
2018-09-20T14:23:23.658-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/archive/metadata
2018-09-20T14:23:23.660-0700 [INFO ] copied key: data/logical/fd1bed89-ffc4-d631-00dd-0696c9f930c6/31c8e6d9-2a17-d98f-bdf1-aa868afa1291/metadata/5kKFZ4YnzgNfy9UcWOzxxzOMpqlp61rYuq6laqpLQDnB3RawKpqi7yBTrawj1P
...
```
After migration has completed, the data is stored on the local file system. To
use the new storage backend with Vault, update Vault's configuration file as
described in the [raft storage configuration
documentation](/docs/configuration/storage/raft). Then start and unseal the
vault server.
### Join additional nodes
After migration the raft cluster will only have a single node. Additional peers
should be joined to this node.
raft: add support for using backend for ha_storage (#9193) * raft: initial work on raft ha storage support * add note on join * add todo note * raft: add support for bootstrapping and joining existing nodes * raft: gate bootstrap join by reading leader api address from storage * raft: properly check for raft-only for certain conditionals * raft: add bootstrap to api and cli * raft: fix bootstrap cli command * raft: add test for setting up new cluster with raft HA * raft: extend TestRaft_HA_NewCluster to include inmem and consul backends * raft: add test for updating an existing cluster to use raft HA * raft: remove debug log lines, clean up verifyRaftPeers * raft: minor cleanup * raft: minor cleanup * Update physical/raft/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/ha.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/ha.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/logical_system_raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * address feedback comments * address feedback comments * raft: refactor tls keyring logic * address feedback comments * Update vault/raft.go Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * address feedback comments * testing: fix import ordering * raft: rename var, cleanup comment line * docs: remove ha_storage restriction note on raft * docs: more raft HA interaction updates with migration and recovery mode * docs: update the raft join command * raft: update comments * raft: add missing isRaftHAOnly check for clearing out state set earlier * raft: update a few ha_storage config checks * Update command/operator_raft_bootstrap.go Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com> * raft: address feedback comments * raft: fix panic when checking for config.HAStorage.Type * Update vault/raft.go Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * Update website/pages/docs/commands/operator/raft.mdx Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> * raft: remove bootstrap cli command * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * Update vault/raft.go Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> * raft: address review feedback * raft: revert vendored sdk * raft: don't send applied index and node ID info if we're HA-only Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com> Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com> Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-06-23 19:04:13 +00:00
If the cluster was previously HA-enabled using "raft" as the `ha_storage`, the
nodes will have to re-join to the migrated node before unsealing.
## Usage
The following flags are available for the `operator migrate` command.
- `-config` `(string: <required>)` - Path to the migration configuration file.
- `-start` `(string: "")` - Migration starting key prefix. Only keys at or after this value will be copied.
- `-reset` - Reset the migration lock. A lock file is added during migration to prevent
starting the Vault server or another migration. The `-reset` option can be used to
remove a stale lock file if present.