* upgrade vault dependency set
* etcd and grpc issues:
* better for tests
* testing
* all upgrades for hashicorp deps
* kubernetes plugin upgrade seems to work
* kubernetes plugin upgrade seems to work
* etcd and a bunch of other stuff
* all vulnerable packages upgraded
* k8s is broken in linux env but not locally
* test fixes
* fix testing
* fix etcd and grpc
* fix etcd and grpc
* use master branch of go-testing-interface
* roll back etcd upgrade
* have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd
* rolling back in the replace directives
* a few more testing dependencies to clean up
* fix go mod vendor
* basic pool and start testing
* refactor a bit for testing
* workFunc, start/stop safety, testing
* cleanup function for worker quit, more tests
* redo public/private members
* improve tests, export types, switch uuid package
* fix loop capture bug, cleanup
* cleanup tests
* update worker pool file name, other improvements
* add job manager prototype
* remove remnants
* add functions to wait for job manager and worker pool to stop, other fixes
* test job manager functionality, fix bugs
* encapsulate how jobs are distributed to workers
* make worker job channel read only
* add job interface, more testing, fixes
* set name for dispatcher
* fix test races
* dispatcher and job manager constructors don't return errors
* logger now dependency injected
* make some members private, test fcn to get worker pool size
* make GetNumWorkers public
* Update helper/fairshare/jobmanager_test.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* make workerpool private
* remove custom worker names
* concurrency improvements
* remove worker pool cleanup function
* remove cleanup func from job manager, remove non blocking stop from fairshare
* stop fairshare when started in tests
* stop leaking job manager goroutine
* prototype channel for waking up to assign work
* fix typo/bug and add tests
* improve job manager wake up, fix test typo
* put channel drain back
* better start/pause test for job manager
* go mod vendor
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Adding snowflake as a bundled database secrets plugin
* Add snowflake-database-plugin to expected bundled plugins
* Add snowflake plugin name to the mockBuiltinRegistry
* fix racy activity log tests and move testing utilities elsewhere
* remove TODO
* move SetEnable out of activity log
* clarify not waiting on waitgroup
* remove todo
* merge activity log invalidation work from vault-enterprise PR 1546
* skip failing test due to enabled config on oss
Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
* first commit
* update
* removed some ent features from backport
* final refactor
* backport patch
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MBP.hitronhub.home>
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
* added retry to mssql testing
* setting num retry to 3
* removed a comment and moved svc into loop
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MBP.hitronhub.home>
Fix some places where raft wasn't hooking into the core logger as it should.
Revisited the code that was setting the log level to Error during cleanup: it's normal for there to be a bunch of errors then, which makes it harder to see what went wrong up to the point where the test was deemed to have failed. So now, instead of setting log level to Error, we actually stop logging altogether. This only applies if the test didn't pass in its own logger during cluster creation, but we should be moving away from that anyway.
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* TOB-018 remediation
* Make key derivation an optional config flag, off by default, for backwards compatibility
* Fix unit tests
* Address some feedback
* Set config on unit test
* Fix another test failure
* One more conf fail
* Switch one of the test cases to not use a derive dkey
* wip
* comments
* raft: initial work on raft ha storage support
* add note on join
* add todo note
* raft: add support for bootstrapping and joining existing nodes
* raft: gate bootstrap join by reading leader api address from storage
* raft: properly check for raft-only for certain conditionals
* raft: add bootstrap to api and cli
* raft: fix bootstrap cli command
* raft: add test for setting up new cluster with raft HA
* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends
* raft: add test for updating an existing cluster to use raft HA
* raft: remove debug log lines, clean up verifyRaftPeers
* raft: minor cleanup
* raft: minor cleanup
* Update physical/raft/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/logical_system_raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* address feedback comments
* address feedback comments
* raft: refactor tls keyring logic
* address feedback comments
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* address feedback comments
* testing: fix import ordering
* raft: rename var, cleanup comment line
* docs: remove ha_storage restriction note on raft
* docs: more raft HA interaction updates with migration and recovery mode
* docs: update the raft join command
* raft: update comments
* raft: add missing isRaftHAOnly check for clearing out state set earlier
* raft: update a few ha_storage config checks
* Update command/operator_raft_bootstrap.go
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* raft: address feedback comments
* raft: fix panic when checking for config.HAStorage.Type
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update website/pages/docs/commands/operator/raft.mdx
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* raft: remove bootstrap cli command
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft: address review feedback
* raft: revert vendored sdk
* raft: don't send applied index and node ID info if we're HA-only
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* Resolve merge conflicts and updates from running a test
* move testing/_test.go over to legacy
* updates
* Add core of plugin test framework Stepwise (#9166)
* adding stepwise testing, but there are protocol buff error :/
* move file and update sdk/go.mo
* update/sync modules
* update from other branch
* update sdk/go.mod
* some cleanups after feedback
* remove enviornments from this PR
* update vendor
* change from running go mod tidy
* change from go mod tidy
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* change panic to error
* Update sdk/testing/stepwise/helpers.go
return `nil` and not `err` at the end
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Defer close() on successful Open of a file
* document the re-creation of steps
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* remove unused BarrierKeys()
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* updates from feedback
* fix return with bad arguments
* Rename things:
- StepOperation -> Operation
- StepwiseEnvironment -> Environment
- StepCheckFunc -> AssertionFunc
- step.Check -> step.Assert
* document the environment interface methods
* rename EnvironmentOptions to MountOptions
* rename Name to RegistryName
* remove ExpectError because it's redundant
* minor doc update
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* add checkShouldRun function
* remove redundant return
* remove vestigial PreCheck function
* add tt.Helper() to makeRequest
* minor code formatting and document 1-based index for log output of Steps
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* minor updates
* update sdk
* use local reference for api, vault dep
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* cleanup some defer functions
* call fatal if environment setup fails, and don't call teardown
* defer re-setting client token in makeRequest
* Move legacy logicaltest back to testhelpers
* update mods and test files with go mod tidy
* go mod vendor
* remove relative replace directives
* restore old logical test location
* move declaration to main stepwise file
* remove index var and use i+1
* add testing for write, delete paths of makeRequest
* update stepwise core testing to do request counting
* remove unused methods
* Update sdk/testing/stepwise/stepwise.go
remove dead line
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update sdk/testing/stepwise/stepwise.go
fix capitalization in code comment
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* update code comments for SkipTeardown to clarify its use
* update stepwise
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Replaced ClusterMetricSink's cluster name with an atomic.Value.
This should permit go-race tests to pass which seal and unseal
the core.
* Replace metric sink before unseal to avoid data races.
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
* Extract certificate helpers for use in non-mongodb packages
* Created mTLS/X509 test for MySQL secrets engine.
* Ensure mysql username and passwords aren't url encoded
* Skip mTLS test for circleCI
* move adjustForSealMigration to vault package
* fix adjustForSealMigration
* begin working on new seal migration test
* create shamir seal migration test
* refactor testhelpers
* add VerifyRaftConfiguration to testhelpers
* stub out TestTransit
* Revert "refactor testhelpers"
This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.
* get shamir test working again
* stub out transit join
* work on transit join
* remove debug code
* initTransit now works with raft join
* runTransit works with inmem
* work on runTransit with raft
* runTransit works with raft
* cleanup tests
* TestSealMigration_TransitToShamir_Pre14
* TestSealMigration_ShamirToTransit_Pre14
* split for pre-1.4 testing
* add simple tests for transit and shamir
* fix typo in test suite
* debug wrapper type
* test debug
* test-debug
* refactor core migration
* Revert "refactor core migration"
This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.
* begin refactor of adjustForSealMigration
* fix bug in adjustForSealMigration
* clean up tests
* clean up core refactoring
* fix bug in shamir->transit migration
* stub out test that brings individual nodes up and down
* refactor NewTestCluster
* pass listeners into newCore()
* simplify cluster address setup
* simplify extra test core setup
* refactor TestCluster for readability
* refactor TestCluster for readability
* refactor TestCluster for readability
* add shutdown func to TestCore
* add cleanup func to TestCore
* create RestartCore
* stub out TestSealMigration_ShamirToTransit_Post14
* refactor address handling in NewTestCluster
* fix listener setup in newCore()
* remove unnecessary lock from setSealsForMigration()
* rename sealmigration test package
* use ephemeral ports below 30000
* work on post-1.4 migration testing
* clean up pre-1.4 test
* TestSealMigration_ShamirToTransit_Post14 works for non-raft
* work on raft TestSealMigration_ShamirToTransit_Post14
* clean up test code
* refactor TestClusterCore
* clean up TestClusterCore
* stub out some temporary tests
* use HardcodedServerAddressProvider in seal migration tests
* work on raft for TestSealMigration_ShamirToTransit_Post14
* always use hardcoded raft address provider in seal migration tests
* debug TestSealMigration_ShamirToTransit_Post14
* fix bug in RestartCore
* remove debug code
* TestSealMigration_ShamirToTransit_Post14 works now
* clean up debug code
* clean up tests
* cleanup tests
* refactor test code
* stub out TestSealMigration_TransitToShamir_Post14
* set seals properly for transit->shamir migration
* migrateFromTransitToShamir_Post14 works for inmem
* migrateFromTransitToShamir_Post14 works for raft
* use base ports per-test
* fix seal verification test code
* simplify seal migration test suite
* simplify test suite
* cleanup test suite
* use explicit ports below 30000
* simplify use of numTestCores
* Update vault/external_tests/sealmigration/seal_migration_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update vault/external_tests/sealmigration/seal_migration_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* clean up imports
* rename to StartCore()
* Update vault/testing.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* simplify test suite
* clean up tests
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* enable seal wrap in all seal migration tests
* move adjustForSealMigration to vault package
* fix adjustForSealMigration
* begin working on new seal migration test
* create shamir seal migration test
* refactor testhelpers
* add VerifyRaftConfiguration to testhelpers
* stub out TestTransit
* Revert "refactor testhelpers"
This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.
* get shamir test working again
* stub out transit join
* work on transit join
* Revert "move resuable storage test to avoid creating import cycle"
This reverts commit b3ff2317381a5af12a53117f87d1c6fbb093af6b.
* remove debug code
* initTransit now works with raft join
* runTransit works with inmem
* work on runTransit with raft
* runTransit works with raft
* get rid of dis-used test
* cleanup tests
* TestSealMigration_TransitToShamir_Pre14
* TestSealMigration_ShamirToTransit_Pre14
* split for pre-1.4 testing
* add simple tests for transit and shamir
* fix typo in test suite
* debug wrapper type
* test debug
* test-debug
* refactor core migration
* Revert "refactor core migration"
This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.
* begin refactor of adjustForSealMigration
* fix bug in adjustForSealMigration
* clean up tests
* clean up core refactoring
* fix bug in shamir->transit migration
* remove unnecessary lock from setSealsForMigration()
* rename sealmigration test package
* use ephemeral ports below 30000
* simplify use of numTestCores
* Add token creation counters.
* Created a utility to change TTL to bucket name.
* Add counter covering token creation for response wrapping.
* Fix namespace label, with a new utility function.
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
Clint