* physical/spanner: use separate client for updating locks
We believe this mitigates an issue where a large influx of requests
cause the leader to be unable to update the lock table (since it cannot
grab a client from the pool or the client has no more open connections),
which causes cascading failure.
* raft: initial work on raft ha storage support
* add note on join
* add todo note
* raft: add support for bootstrapping and joining existing nodes
* raft: gate bootstrap join by reading leader api address from storage
* raft: properly check for raft-only for certain conditionals
* raft: add bootstrap to api and cli
* raft: fix bootstrap cli command
* raft: add test for setting up new cluster with raft HA
* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends
* raft: add test for updating an existing cluster to use raft HA
* raft: remove debug log lines, clean up verifyRaftPeers
* raft: minor cleanup
* raft: minor cleanup
* Update physical/raft/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/logical_system_raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* address feedback comments
* address feedback comments
* raft: refactor tls keyring logic
* address feedback comments
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* address feedback comments
* testing: fix import ordering
* raft: rename var, cleanup comment line
* docs: remove ha_storage restriction note on raft
* docs: more raft HA interaction updates with migration and recovery mode
* docs: update the raft join command
* raft: update comments
* raft: add missing isRaftHAOnly check for clearing out state set earlier
* raft: update a few ha_storage config checks
* Update command/operator_raft_bootstrap.go
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* raft: address feedback comments
* raft: fix panic when checking for config.HAStorage.Type
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update website/pages/docs/commands/operator/raft.mdx
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* raft: remove bootstrap cli command
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft: address review feedback
* raft: revert vendored sdk
* raft: don't send applied index and node ID info if we're HA-only
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* initial work on improving snapshot performance
* Work on snapshots
* rename a few functions
* Cleanup the snapshot file
* vendor the safeio library
* Add a test
* Add more tests
* Some review comments
* Fix comment
* Update physical/raft/snapshot.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update physical/raft/snapshot.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Review feedback
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
* Adds a safety switch to configuration files.
This requires a user to either use TLS, or acknowledge that they are sending
credentials over plaintext.
* Warn if plaintext credentials will be passed
* Add true/false support to the plaintext transmission ack
* Updated website docs and ensured ToLower is used for true comparison
* storage/raft: Advertise the configured cluster address
* Don't allow raft to start with unspecified IP
* Fix concurrent map write panic
* Add test file
* changelog++
* changelog++
* changelog++
* Update tcp_layer.go
* Update tcp_layer.go
* Only set the adverise addr if set
* storage/raft: Add committed and applied indexes to the status output
* Update api vendor
* changelog++
* Update http/sys_leader.go
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* stub out reusable storage test
* implement reusable inmem test
* work on reusable raft test
* stub out simple raft test
* switch to reusable raft storage
* cleanup tests
* cleanup tests
* refactor tests
* verify raft configuration
* cleanup tests
* stub out reuseStorage
* use common base address across clusters
* attempt to reuse raft cluster
* tinker with test
* fix typo
* start debugging
* debug raft configuration
* add BaseClusterListenPort to TestCluster options
* use BaseClusterListenPort in test
* raft join works now
* misc cleanup of raft tests
* use configurable base port for raft test
* clean up raft tests
* add parallelized tests for all backends
* clean up reusable storage tests
* remove debugging code from startClusterListener()
* improve comments in testhelpers
* improve comments in teststorage
* improve comments and test logging
* fix typo in vault/testing
* fix typo in comments
* remove debugging code
* make number of cores parameterizable in test
* raft: use file paths for TLS info in the retry_join stanza
* raft: maintain backward compat for existing tls params
* docs: update raft docs with new file-based TLS params
* Update godoc comment, fix docs
* raft: check for nil on concrete type in SetupCluster
* raft: move check to its own func
* raft: func cleanup
* raft: disallow disable_clustering = true when raft storage is used
* docs: update disable_clustering to mention new behavior
* Disallow non-voter setting from peers.json
* Fix bug that would make the actual fix a no-op
* Change order of evaluation
* Error out instead of resetting the value
* Update physical/raft/raft.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* Print node ID
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* adding support for TLS 1.3 for TCP listeners
* removed test as CI uses go 1.12
* removed Cassandra support, added deprecation notice
* re-added TestTCPListener_tls13
* Guard against using Raft as a seperate HA Storage
* Document that Raft cannot be used as a seperate ha_storage backend at this time
* remove duplicate imports from updating with master
* use observer pattern for service discovery
* update perf standby method
* fix test
* revert usersTags to being called serviceTags
* use previous consul code
* vault isnt a performance standby before starting
* log err
* changes from feedback
* add Run method to interface
* changes from feedback
* fix core test
* update example
* Raft retry join
* update
* Make retry join work with shamir seal
* Return upon context completion
* Update vault/raft.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Address some review comments
* send leader information slice as a parameter
* Make retry join work properly with Shamir case. This commit has a blocking issue
* Fix join goroutine exiting before the job is done
* Polishing changes
* Don't return after a successful join during unseal
* Added config parsing test
* Add test and fix bugs
* minor changes
* Address review comments
* Fix build error
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* move ServiceDiscovery into methods
* add ServiceDiscoveryFactory
* add serviceDiscovery field to vault.Core
* refactor ConsulServiceDiscovery into separate struct
* cleanup
* revert accidental change to go.mod
* cleanup
* get rid of un-needed struct tags in vault.CoreConfig
* add service_discovery parser
* add ServiceDiscovery to config
* cleanup
* cleanup
* add test for ConfigServiceDiscovery to Core
* unit testing for config service_discovery stanza
* cleanup
* get rid of un-needed redirect_addr stuff in service_discovery stanza
* improve test suite
* cleanup
* clean up test a bit
* create docs for service_discovery
* check if service_discovery is configured, but storage does not support HA
* tinker with test
* tinker with test
* tweak docs
* move ServiceDiscovery into its own package
* tweak a variable name
* fix comment
* rename service_discovery to service_registration
* tweak service_registration config
* Revert "tweak service_registration config"
This reverts commit 5509920a8ab4c5a216468f262fc07c98121dce35.
* simplify naming
* refactor into ./serviceregistration/consul
* physical/posgresql: add ability to use CONNECTION_URL environment variable instead of requiring it to be configured in the Vault config file.
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* storage/postgresql: update configuration documentation for postgresql storage backend to include connection_url configuration via the PG_CONNECTION_URL environment variable
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* physical/postgresql: add a configuration file and tests for getting the connection_url from the config file or environment
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* physical/postgresql: update postgresql backend to pull the required connection_url from the PG_CONNECTION_URL environment variable if it exists, otherwise, fallback to using the config file
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* physical/postgresql: remove configure*.go files and prefer the postgresql*.go files
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* physical/postgresql: move and simplify connectionURL function
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* physical/postgresql: update connectionURL test to use an unordered map instead of slice to avoid test flakiness
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* physical/postgresql: update config env to be prefixed with VAULT_ - VAULT_PG_CONNECTION_URL
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* docs/web: update postgresql backend docs to use updated, VAULT_ prefixed config env
Signed-off-by: Colton McCurdy <mccurdyc22@gmail.com>
* refactor test code to avoid panic if tests ran multiple times
* cleanup: don't actually send just close
* move comment to a better location
* move error check to a more obvious spot
* Revert "move error check to a more obvious spot"
Reverting because methods like this should only be called on the main
goroutine running the test:
- https://golang.org/pkg/testing/#T
This reverts commit db7641948317785bff15b3d9dbe6fb18a2d19c2c.
* Fix unordered imports
* Allow Raft node ID to be set via the environment variable `VAULT_RAFT_NODE_ID`
* Allow Raft path to be set via the environment variable `VAULT_RAFT_PATH`
* Prioritize the environment when fetching the Raft configuration values
Values in environment variables should override the config as per the
documentation as well as common sense.
* Initial work
* rework
* s/dr/recovery
* Add sys/raw support to recovery mode (#7577)
* Factor the raw paths out so they can be run with a SystemBackend.
# Conflicts:
# vault/logical_system.go
* Add handleLogicalRecovery which is like handleLogical but is only
sufficient for use with the sys-raw endpoint in recovery mode. No
authentication is done yet.
* Integrate with recovery-mode. We now handle unauthenticated sys/raw
requests, albeit on path v1/raw instead v1/sys/raw.
* Use sys/raw instead raw during recovery.
* Don't bother persisting the recovery token. Authenticate sys/raw
requests with it.
* RecoveryMode: Support generate-root for autounseals (#7591)
* Recovery: Abstract config creation and log settings
* Recovery mode integration test. (#7600)
* Recovery: Touch up (#7607)
* Recovery: Touch up
* revert the raw backend creation changes
* Added recovery operation token prefix
* Move RawBackend to its own file
* Update API path and hit it using CLI flag on generate-root
* Fix a panic triggered when handling a request that yields a nil response. (#7618)
* Improve integ test to actually make changes while in recovery mode and
verify they're still there after coming back in regular mode.
* Refuse to allow a second recovery token to be generated.
* Resize raft cluster to size 1 and start as leader (#7626)
* RecoveryMode: Setup raft cluster post unseal (#7635)
* Setup raft cluster post unseal in recovery mode
* Remove marking as unsealed as its not needed
* Address review comments
* Accept only one seal config in recovery mode as there is no scope for migration
Calvin Leung Huang