* k8s doc: update for 0.9.1 and 0.8.0 releases (#10825)
* k8s doc: update for 0.9.1 and 0.8.0 releases
* Update website/content/docs/platform/k8s/helm/configuration.mdx
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
* Autopilot initial commit
* Move autopilot related backend implementations to its own file
* Abstract promoter creation
* Add nil check for health
* Add server state oss no-ops
* Config ext stub for oss
* Make way for non-voters
* s/health/state
* s/ReadReplica/NonVoter
* Add synopsis and description
* Remove struct tags from AutopilotConfig
* Use var for config storage path
* Handle nin-config when reading
* Enable testing autopilot by using inmem cluster
* First passing test
* Only report the server as known if it is present in raft config
* Autopilot defaults to on for all existing and new clusters
* Add locking to some functions
* Persist initial config
* Clarify the command usage doc
* Add health metric for each node
* Fix audit logging issue
* Don't set DisablePerformanceStandby to true in test
* Use node id label for health metric
* Log updates to autopilot config
* Less aggressively consume config loading failures
* Return a mutable config
* Return early from known servers if raft config is unable to be pulled
* Update metrics name
* Reduce log level for potentially noisy log
* Add knob to disable autopilot
* Don't persist if default config is in use
* Autopilot: Dead server cleanup (#10857)
* Dead server cleanup
* Initialize channel in any case
* Fix a bunch of tests
* Fix panic
* Add follower locking in heartbeat tracker
* Add LastContactFailureThreshold to config
* Add log when marking node as dead
* Update follower state locking in heartbeat tracker
* Avoid follower states being nil
* Pull test to its own file
* Add execution status to state response
* Optionally enable autopilot in some tests
* Updates
* Added API function to fetch autopilot configuration
* Add test for default autopilot configuration
* Configuration tests
* Add State API test
* Update test
* Added TestClusterOptions.PhysicalFactoryConfig
* Update locking
* Adjust locking in heartbeat tracker
* s/last_contact_failure_threshold/left_server_last_contact_threshold
* Add disabling autopilot as a core config option
* Disable autopilot in some tests
* s/left_server_last_contact_threshold/dead_server_last_contact_threshold
* Set the lastheartbeat of followers to now when setting up active node
* Don't use config defaults from CLI command
* Remove config file support
* Remove HCL test as well
* Persist only supplied config; merge supplied config with default to operate
* Use pointer to structs for storing follower information
* Test update
* Retrieve non voter status from configbucket and set it up when a node comes up
* Manage desired suffrage
* Consider bucket being created already
* Move desired suffrage to its own entry
* s/DesiredSuffrageKey/LocalNodeConfigKey
* s/witnessSuffrage/recordSuffrage
* Fix test compilation
* Handle local node config post a snapshot install
* Commit to storage first; then record suffrage in fsm
* No need of local node config being nili case, post snapshot restore
* Reconcile autopilot config when a new leader takes over duty
* Grab fsm lock when recording suffrage
* s/Suffrage/DesiredSuffrage in FollowerState
* Instantiate autopilot only in leader
* Default to old ways in more scenarios
* Make API gracefully handle 404
* Address some feedback
* Make IsDead an atomic.Value
* Simplify follower hearbeat tracking
* Use uber.atomic
* Don't have multiple causes for having autopilot disabled
* Don't remove node from follower states if we fail to remove the dead server
* Autopilot server removals map (#11019)
* Don't remove node from follower states if we fail to remove the dead server
* Use map to track dead server removals
* Use lock and map
* Use delegate lock
* Adjust when to remove entry from map
* Only hold the lock while accessing map
* Fix race
* Don't set default min_quorum
* Fix test
* Ensure follower states is not nil before starting autopilot
* Fix race
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
* upgrade vault dependency set
* etcd and grpc issues:
* better for tests
* testing
* all upgrades for hashicorp deps
* kubernetes plugin upgrade seems to work
* kubernetes plugin upgrade seems to work
* etcd and a bunch of other stuff
* all vulnerable packages upgraded
* k8s is broken in linux env but not locally
* test fixes
* fix testing
* fix etcd and grpc
* fix etcd and grpc
* use master branch of go-testing-interface
* roll back etcd upgrade
* have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd
* rolling back in the replace directives
* a few more testing dependencies to clean up
* fix go mod vendor
* basic pool and start testing
* refactor a bit for testing
* workFunc, start/stop safety, testing
* cleanup function for worker quit, more tests
* redo public/private members
* improve tests, export types, switch uuid package
* fix loop capture bug, cleanup
* cleanup tests
* update worker pool file name, other improvements
* add job manager prototype
* remove remnants
* add functions to wait for job manager and worker pool to stop, other fixes
* test job manager functionality, fix bugs
* encapsulate how jobs are distributed to workers
* make worker job channel read only
* add job interface, more testing, fixes
* set name for dispatcher
* fix test races
* dispatcher and job manager constructors don't return errors
* logger now dependency injected
* make some members private, test fcn to get worker pool size
* make GetNumWorkers public
* Update helper/fairshare/jobmanager_test.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* make workerpool private
* remove custom worker names
* concurrency improvements
* remove worker pool cleanup function
* remove cleanup func from job manager, remove non blocking stop from fairshare
* stop fairshare when started in tests
* stop leaking job manager goroutine
* prototype channel for waking up to assign work
* fix typo/bug and add tests
* improve job manager wake up, fix test typo
* put channel drain back
* better start/pause test for job manager
* go mod vendor
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Adding snowflake as a bundled database secrets plugin
* Add snowflake-database-plugin to expected bundled plugins
* Add snowflake plugin name to the mockBuiltinRegistry
* fix racy activity log tests and move testing utilities elsewhere
* remove TODO
* move SetEnable out of activity log
* clarify not waiting on waitgroup
* remove todo
* merge activity log invalidation work from vault-enterprise PR 1546
* skip failing test due to enabled config on oss
Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
* first commit
* update
* removed some ent features from backport
* final refactor
* backport patch
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MBP.hitronhub.home>
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
* added retry to mssql testing
* setting num retry to 3
* removed a comment and moved svc into loop
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MBP.hitronhub.home>
Fix some places where raft wasn't hooking into the core logger as it should.
Revisited the code that was setting the log level to Error during cleanup: it's normal for there to be a bunch of errors then, which makes it harder to see what went wrong up to the point where the test was deemed to have failed. So now, instead of setting log level to Error, we actually stop logging altogether. This only applies if the test didn't pass in its own logger during cluster creation, but we should be moving away from that anyway.
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* TOB-018 remediation
* Make key derivation an optional config flag, off by default, for backwards compatibility
* Fix unit tests
* Address some feedback
* Set config on unit test
* Fix another test failure
* One more conf fail
* Switch one of the test cases to not use a derive dkey
* wip
* comments
* raft: initial work on raft ha storage support
* add note on join
* add todo note
* raft: add support for bootstrapping and joining existing nodes
* raft: gate bootstrap join by reading leader api address from storage
* raft: properly check for raft-only for certain conditionals
* raft: add bootstrap to api and cli
* raft: fix bootstrap cli command
* raft: add test for setting up new cluster with raft HA
* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends
* raft: add test for updating an existing cluster to use raft HA
* raft: remove debug log lines, clean up verifyRaftPeers
* raft: minor cleanup
* raft: minor cleanup
* Update physical/raft/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/logical_system_raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* address feedback comments
* address feedback comments
* raft: refactor tls keyring logic
* address feedback comments
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* address feedback comments
* testing: fix import ordering
* raft: rename var, cleanup comment line
* docs: remove ha_storage restriction note on raft
* docs: more raft HA interaction updates with migration and recovery mode
* docs: update the raft join command
* raft: update comments
* raft: add missing isRaftHAOnly check for clearing out state set earlier
* raft: update a few ha_storage config checks
* Update command/operator_raft_bootstrap.go
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* raft: address feedback comments
* raft: fix panic when checking for config.HAStorage.Type
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update website/pages/docs/commands/operator/raft.mdx
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* raft: remove bootstrap cli command
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft: address review feedback
* raft: revert vendored sdk
* raft: don't send applied index and node ID info if we're HA-only
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* Resolve merge conflicts and updates from running a test
* move testing/_test.go over to legacy
* updates
* Add core of plugin test framework Stepwise (#9166)
* adding stepwise testing, but there are protocol buff error :/
* move file and update sdk/go.mo
* update/sync modules
* update from other branch
* update sdk/go.mod
* some cleanups after feedback
* remove enviornments from this PR
* update vendor
* change from running go mod tidy
* change from go mod tidy
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* change panic to error
* Update sdk/testing/stepwise/helpers.go
return `nil` and not `err` at the end
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Defer close() on successful Open of a file
* document the re-creation of steps
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* remove unused BarrierKeys()
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* updates from feedback
* fix return with bad arguments
* Rename things:
- StepOperation -> Operation
- StepwiseEnvironment -> Environment
- StepCheckFunc -> AssertionFunc
- step.Check -> step.Assert
* document the environment interface methods
* rename EnvironmentOptions to MountOptions
* rename Name to RegistryName
* remove ExpectError because it's redundant
* minor doc update
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* add checkShouldRun function
* remove redundant return
* remove vestigial PreCheck function
* add tt.Helper() to makeRequest
* minor code formatting and document 1-based index for log output of Steps
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* minor updates
* update sdk
* use local reference for api, vault dep
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* cleanup some defer functions
* call fatal if environment setup fails, and don't call teardown
* defer re-setting client token in makeRequest
* Move legacy logicaltest back to testhelpers
* update mods and test files with go mod tidy
* go mod vendor
* remove relative replace directives
* restore old logical test location
* move declaration to main stepwise file
* remove index var and use i+1
* add testing for write, delete paths of makeRequest
* update stepwise core testing to do request counting
* remove unused methods
* Update sdk/testing/stepwise/stepwise.go
remove dead line
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update sdk/testing/stepwise/stepwise.go
fix capitalization in code comment
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* update code comments for SkipTeardown to clarify its use
* update stepwise
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Replaced ClusterMetricSink's cluster name with an atomic.Value.
This should permit go-race tests to pass which seal and unseal
the core.
* Replace metric sink before unseal to avoid data races.
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
Vishal Nayak