Commit graph

670 commits

Author SHA1 Message Date
Tero Saarni 944332d12d
Update Go client libraries for etcd (#11980)
* Update Go client libraries for etcd

* Added etcd server container to run etcd3 tests automatically.

* Removed etcd2 test case: it fails the backend tests but the failure is
  unrelated to the uplift.  The etcd2 backend implementation does not
  remove empty nested nodes when removing leaf (see comments in #11980).
2021-09-29 14:28:13 -04:00
akshya96 c643dc1d53
Add Custom metadata field to alias (#12502)
* adding changes

* removing q.Q

* removing empty lines

* testing

* checking tests

* fixing tests

* adding changes

* added requested changes

* added requested changes

* added policy templating changes and fixed tests

* adding proto changes

* making changes

* adding unit tests

* using suggested function
2021-09-17 11:03:47 -07:00
Pratyoy Mukhopadhyay fa29e780f0
[NO-TICKET] Upgrade protoc-gen-go to 1.26, upgrade protoc to 3.17.3 (#12171)
* [NO-TICKET] Set protoc-gen-go to 1.23, upgrade protoc to 3.17.3

* [NO-TICKET] Upgrade version of protoc-gen-go to 1.26
2021-07-28 14:51:36 -07:00
Jeff Mitchell f7147025dd
Migrate to sdk/internalshared libs in go-secure-stdlib (#12090)
* Swap sdk/helper libs to go-secure-stdlib

* Migrate to go-secure-stdlib reloadutil

* Migrate to go-secure-stdlib kv-builder

* Migrate to go-secure-stdlib gatedwriter
2021-07-15 20:17:31 -04:00
Hridoy Roy f3895f7cce
oss part of license diagnose (#11939) 2021-06-25 14:18:34 -07:00
swayne275 597c3d1296
Vault-1983: Use fairsharing to distribute workers between queues (#11789)
* prelim fairshare prototype, untested and prototype status

* add tests for new fairshare infra - this likely fails tests for being racy

* probably fix races for code and test

* one more lock to fix for races

* fairsharing queue work distribution, tests, fixes, etc

* comment, shorten wait time

* typos and comments

* fix inverted worker count logic

* Update helper/fairshare/jobmanager.go

typo

* Update helper/fairshare/jobmanager.go

clarify comment

* move back to round robin between queues

* improvements from self review

* add job manager stress test
2021-06-25 14:06:49 -06:00
Michael Golowka 7f6a1739a3
Cassandra: Refactor PEM parsing logic (#11861)
* Refactor TLS parsing

The ParsePEMBundle and ParsePKIJSON functions in the certutil package assumes
both a client certificate and a custom CA are specified. Cassandra needs to
allow for either a client certificate, a custom CA, or both. This revamps the
parsing of pem_json and pem_bundle to accomodate for any of these configurations
2021-06-21 11:38:08 -06:00
Lars Lehtonen 1480f26e3d
helper: deprecate errwrap.Wrapf() (#11556) 2021-05-07 08:53:20 -04:00
swayne275 335e4c3711
Introduce Logical Unrecoverable Error, Use it in Expiration Manager (#11477)
* build out zombie lease system

* add typo for CI

* undo test CI commit

* time equality test isn't working on CI, so let's see what this does...

* add unrecoverable proto error, make proto, go mod vendor

* zombify leases if unrecoverable error, tests

* test fix: somehow pointer in pointer rx is null after pointer rx called

* tweaks based on roy feedback

* improve zombie errors

* update which errors are unrecoverable

* combine zombie logic

* keep subset of zombie lease in memory
2021-05-03 17:56:06 -06:00
Josh Black ec105f288f
Switch to shared raft-boltdb library and add metrics (#11269) 2021-04-26 16:01:26 -07:00
Nick Cabatoff e3a88c59e9
Add metrics for perf standby and replication node type. (#11472) 2021-04-26 13:54:19 -04:00
Nick Cabatoff a21e7139ff
Update tests to reflect new HC public pgp keys. (#11446) 2021-04-23 12:32:46 -04:00
Nick Cabatoff 7ee6acf780
Fix a couple of 64bit-isms. (#11429) 2021-04-21 09:53:25 -04:00
Michael Golowka 4279bc8b34
Validate hostnames when using TLS in Cassandra (#11365) 2021-04-16 15:52:35 -06:00
Nick Cabatoff 242d258e94
Fix goroutine leak caused by updating rate quotas (#11371)
Make sure that when we modify a rate quota, we stop the existing goroutine before starting the new one.
2021-04-16 14:00:01 -04:00
Brian Kassouf 303c2aee7c
Run a more strict formatter over the code (#11312)
* Update tooling

* Run gofumpt

* go mod vendor
2021-04-08 09:43:39 -07:00
Nick Cabatoff 5a2f0cbe51
OSS changes from ent fix for openbsd memory lookups. (#11088) 2021-03-11 10:25:15 -05:00
Vishal Nayak 586a5a73f0
Fix diff (#11057) 2021-03-08 13:20:01 -05:00
Vishal Nayak 3e55e79a3f
Autopilot: Server Stabilization, State and Dead Server Cleanup (#10856)
* k8s doc: update for 0.9.1 and 0.8.0 releases (#10825)

* k8s doc: update for 0.9.1 and 0.8.0 releases

* Update website/content/docs/platform/k8s/helm/configuration.mdx

Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

* Autopilot initial commit

* Move autopilot related backend implementations to its own file

* Abstract promoter creation

* Add nil check for health

* Add server state oss no-ops

* Config ext stub for oss

* Make way for non-voters

* s/health/state

* s/ReadReplica/NonVoter

* Add synopsis and description

* Remove struct tags from AutopilotConfig

* Use var for config storage path

* Handle nin-config when reading

* Enable testing autopilot by using inmem cluster

* First passing test

* Only report the server as known if it is present in raft config

* Autopilot defaults to on for all existing and new clusters

* Add locking to some functions

* Persist initial config

* Clarify the command usage doc

* Add health metric for each node

* Fix audit logging issue

* Don't set DisablePerformanceStandby to true in test

* Use node id label for health metric

* Log updates to autopilot config

* Less aggressively consume config loading failures

* Return a mutable config

* Return early from known servers if raft config is unable to be pulled

* Update metrics name

* Reduce log level for potentially noisy log

* Add knob to disable autopilot

* Don't persist if default config is in use

* Autopilot: Dead server cleanup (#10857)

* Dead server cleanup

* Initialize channel in any case

* Fix a bunch of tests

* Fix panic

* Add follower locking in heartbeat tracker

* Add LastContactFailureThreshold to config

* Add log when marking node as dead

* Update follower state locking in heartbeat tracker

* Avoid follower states being nil

* Pull test to its own file

* Add execution status to state response

* Optionally enable autopilot in some tests

* Updates

* Added API function to fetch autopilot configuration

* Add test for default autopilot configuration

* Configuration tests

* Add State API test

* Update test

* Added TestClusterOptions.PhysicalFactoryConfig

* Update locking

* Adjust locking in heartbeat tracker

* s/last_contact_failure_threshold/left_server_last_contact_threshold

* Add disabling autopilot as a core config option

* Disable autopilot in some tests

* s/left_server_last_contact_threshold/dead_server_last_contact_threshold

* Set the lastheartbeat of followers to now when setting up active node

* Don't use config defaults from CLI command

* Remove config file support

* Remove HCL test as well

* Persist only supplied config; merge supplied config with default to operate

* Use pointer to structs for storing follower information

* Test update

* Retrieve non voter status from configbucket and set it up when a node comes up

* Manage desired suffrage

* Consider bucket being created already

* Move desired suffrage to its own entry

* s/DesiredSuffrageKey/LocalNodeConfigKey

* s/witnessSuffrage/recordSuffrage

* Fix test compilation

* Handle local node config post a snapshot install

* Commit to storage first; then record suffrage in fsm

* No need of local node config being nili case, post snapshot restore

* Reconcile autopilot config when a new leader takes over duty

* Grab fsm lock when recording suffrage

* s/Suffrage/DesiredSuffrage in FollowerState

* Instantiate autopilot only in leader

* Default to old ways in more scenarios

* Make API gracefully handle 404

* Address some feedback

* Make IsDead an atomic.Value

* Simplify follower hearbeat tracking

* Use uber.atomic

* Don't have multiple causes for having autopilot disabled

* Don't remove node from follower states if we fail to remove the dead server

* Autopilot server removals map (#11019)

* Don't remove node from follower states if we fail to remove the dead server

* Use map to track dead server removals

* Use lock and map

* Use delegate lock

* Adjust when to remove entry from map

* Only hold the lock while accessing map

* Fix race

* Don't set default min_quorum

* Fix test

* Ensure follower states is not nil before starting autopilot

* Fix race

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
2021-03-03 13:59:50 -05:00
Brian Kassouf 1bc410783d OSS/ENT Drift 2021-03-01 10:51:04 -08:00
Brian Kassouf a112161f60
expiration: Add a few metrics to measure revoke queue lengths (#10955)
* expiration: Add a few metrics to measure revoke queue lengths

* Update the metric names

* Add appropriate cluster labels

* Add metrics to docs

* Update jobmanager.go
2021-02-26 16:00:39 -08:00
swayne275 fe9e6e3d89
Make fairsharing stop function async (#11012)
* stop fairsharing layer async

* update function doc
2021-02-25 14:33:02 -07:00
Nick Cabatoff c1ddfbb538
OSS parts of the new client controlled consistency feature (#10974) 2021-02-24 06:58:10 -05:00
swayne275 38a647c6e5
remove noisy log, simplify job interface (#10975) 2021-02-22 15:00:24 -07:00
Clint 2aff402279
Bundle new Vault plugin: Terraform secrets (#10931)
* Bundle Terraform secrets engine

* update go.mod/sum

* vendor update

* add changelog entry

* add secrets terraform
2021-02-19 16:38:56 -06:00
Brian Kassouf 6f3d179635
helper/metricsutil: Prevent potential Ticker leak (#10913) 2021-02-18 20:31:53 -08:00
Hridoy Roy 4a96126d5a
Revert "Vault Dependency Upgrades [VAULT-871] (#10903)" (#10939)
This reverts commit eb74ca61fc4dcb7038f39defb127d5d639ba0ca1.
2021-02-18 15:40:18 -05:00
Hridoy Roy a26d1300e8
Vault Dependency Upgrades [VAULT-871] (#10903)
* upgrade vault dependency set

* etcd and grpc issues:

* better for tests

* testing

* all upgrades for hashicorp deps

* kubernetes plugin upgrade seems to work

* kubernetes plugin upgrade seems to work

* etcd and a bunch of other stuff

* all vulnerable packages upgraded

* k8s is broken in linux env but not locally

* test fixes

* fix testing

* fix etcd and grpc

* fix etcd and grpc

* use master branch of go-testing-interface

* roll back etcd upgrade

* have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd

* rolling back in the replace directives

* a few more testing dependencies to clean up

* fix go mod vendor
2021-02-18 12:31:57 -08:00
Michael Golowka e494e8a141
Cassandra - Add username customization (#10906) 2021-02-16 12:39:24 -07:00
swayne275 3994579603
VAULT-1401 and 1402 - preliminary fair sharing (#1701) (#10917)
* basic pool and start testing

* refactor a bit for testing

* workFunc, start/stop safety, testing

* cleanup function for worker quit, more tests

* redo public/private members

* improve tests, export types, switch uuid package

* fix loop capture bug, cleanup

* cleanup tests

* update worker pool file name, other improvements

* add job manager prototype

* remove remnants

* add functions to wait for job manager and worker pool to stop, other fixes

* test job manager functionality, fix bugs

* encapsulate how jobs are distributed to workers

* make worker job channel read only

* add job interface, more testing, fixes

* set name for dispatcher

* fix test races

* dispatcher and job manager constructors don't return errors

* logger now dependency injected

* make some members private, test fcn to get worker pool size

* make GetNumWorkers public

* Update helper/fairshare/jobmanager_test.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* make workerpool private

* remove custom worker names

* concurrency improvements

* remove worker pool cleanup function

* remove cleanup func from job manager, remove non blocking stop from fairshare

* stop fairshare when started in tests

* stop leaking job manager goroutine

* prototype channel for waking up to assign work

* fix typo/bug and add tests

* improve job manager wake up, fix test typo

* put channel drain back

* better start/pause test for job manager

* go mod vendor

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-02-12 14:51:52 -07:00
Michael Golowka 108d4c6a68
MySQL - Add username customization (#10834) 2021-02-11 14:08:32 -07:00
Lauren Voswinkel 7189a67a33
Adding snowflake as a bundled database secrets plugin (#10603)
* Adding snowflake as a bundled database secrets plugin

* Add snowflake-database-plugin to expected bundled plugins

* Add snowflake plugin name to the mockBuiltinRegistry
2021-01-07 09:30:24 -08:00
swayne275 88eaf5f4c3
Fix Racy Activity Log Tests (#10484)
* fix racy activity log tests and move testing utilities elsewhere

* remove TODO

* move SetEnable out of activity log

* clarify not waiting on waitgroup

* remove todo
2020-12-02 13:48:13 -07:00
Michael Golowka cc7efd393d
MySQL - Fix username generation length bug (#10433) 2020-12-01 15:24:51 -07:00
Nick Cabatoff 6a377a3df3
Add helper used in ent. (#10445) 2020-11-25 08:27:45 -05:00
Hridoy Roy 6261afb343
Port: Telemetry For Lease Expiration Times (#10375)
* port lease metrics

* go mod vendor

* caught a bug
2020-11-13 10:26:58 -08:00
swayne275 dffd85e09a
Backport invalidation changes (#10292)
* merge activity log invalidation work from vault-enterprise PR 1546

* skip failing test due to enabled config on oss

Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
2020-10-30 18:11:12 -06:00
Brian Kassouf 81a86f48e8
Backport some OSS changes (#10267)
* Backport some OSS changes

* go mod vendor
2020-10-29 16:47:34 -07:00
Hridoy Roy 0259be04e0
Port: Add metrics to report mount table sizes for auth and logical [Vault 671] (#10201)
* first commit

* update

* removed some ent features from backport

* final refactor

* backport patch

Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MBP.hitronhub.home>
2020-10-27 08:24:43 -07:00
Nick Cabatoff 0d6a929a4c
Same seal migration oss (#10224)
* Refactoring and test improvements.

* Support migrating from a given type of autoseal to that same type but with different parameters.
2020-10-23 14:16:04 -04:00
Tom Proctor 53875004df
DBPW - Migrate Redshift database plugin to v5 interface (#10195) 2020-10-23 14:10:57 +01:00
Michael Golowka bd79fbafb3
Add couchbase, elasticsearch, and mongodbatlas back (#10222)
Updated the `Serve` function so these can be added back into Vault
2020-10-22 17:20:17 -06:00
Michael Golowka e6c8ee24ea
DBPW - Enables AutoMTLS for DB plugins (#10220)
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
2020-10-22 15:43:19 -06:00
Nick Cabatoff 66274607b7
OSS changes for enterprise automated snapshots (#10160) 2020-10-16 14:57:11 -04:00
Brian Kassouf 84dbca38a1
Revert "Migrate internalshared out (#9727)" (#10141)
This reverts commit ee6391b691ac12ab6ca13c3912404f1d3a842bd6.
2020-10-13 16:38:21 -07:00
Jeff Mitchell e6881c8147
Migrate internalshared out (#9727)
* Migrate internalshared out

* fix merge issue

* fix merge issue

* go mod vendor

Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
2020-10-12 11:56:24 -07:00
Lauren Voswinkel 55761c1d43
Migrate MySQL db to v5 database engine (#10110)
* Migrate MySQL db to v5 database engine
2020-10-09 10:32:38 -07:00
Hridoy Roy d7a673321d
Retry Logic to Mssql Tests [VAULT-637] (#10039)
* added retry to mssql testing

* setting num retry to 3

* removed a comment and moved svc into loop

Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MBP.hitronhub.home>
2020-09-28 09:03:23 -07:00
ncabatoff 3fb20ed529
Logging tweaks, mostly for tests (#10028)
Fix some places where raft wasn't hooking into the core logger as it should.
Revisited the code that was setting the log level to Error during cleanup: it's normal for there to be a bunch of errors then, which makes it harder to see what went wrong up to the point where the test was deemed to have failed.  So now, instead of setting log level to Error, we actually stop logging altogether.  This only applies if the test didn't pass in its own logger during cluster creation, but we should be moving away from that anyway.
2020-09-23 13:40:00 -04:00
Brian Kassouf 3f30fc5f4e
Port changes from enterprise lease fix (#10020) 2020-09-22 14:47:13 -07:00
ncabatoff 2872702995
Use a longer timeout for DeriveActiveCore in the hopes that giving more time will allow for raft leader election failure to recover. (#9988) 2020-09-18 11:07:35 -04:00
ncabatoff 0f77d0e282
Move the code that creates Consul containers out of teststorage. This allows importers of teststorage that don't need consul to run as a non-docker test. (#9975) 2020-09-17 15:44:29 -04:00
ncabatoff 64301d3b22
Allow for testing Consul with a different repo that requires auth. (#9983) 2020-09-17 13:05:24 -04:00
Lauren Voswinkel 5740e1ff9e
5844 AWS Root Credential Rotation (#9921)
* strip redundant field type declarations

* root credential rotation for aws creds plugin

* Change location of mocks awsutil and update methods that no longer exist

* Update website/pages/docs/auth/aws.mdx

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* Update sdk version to get the awsutil mock file

* Re-vendor modules to pass CI

* Use write lock for the entirety of AWS root cred rotation

* Update docs for AWS root cred rotation for clarity

Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-09-15 15:26:56 -07:00
ncabatoff b615da43d7
Run CI tests in docker instead of a machine. (#8948) 2020-09-15 10:01:26 -04:00
Scott Miller 5b003b06f8
Trail of bits 018 (#9674)
* TOB-018 remediation

* Make key derivation an optional config flag, off by default, for backwards compatibility

* Fix unit tests

* Address some feedback

* Set config on unit test

* Fix another test failure

* One more conf fail

* Switch one of the test cases to not use a derive dkey

* wip

* comments
2020-08-17 11:36:16 -05:00
ncabatoff 4134ef2e98
Ensure that perf standbys can perform seal migrations. (#9690) 2020-08-10 08:35:57 -04:00
Tom Proctor 4ca978598f
Bundle couchbase database plugin with vault (#9664) 2020-08-07 11:01:04 +01:00
ncabatoff 73b8696b46
Don't assume that time.Duration can be represented by an int. (#9554) 2020-07-22 12:04:52 -04:00
Freddy 59ffc7af5c
Allow consul repo override via env (#9500) 2020-07-16 10:48:12 -04:00
Josh Black 38fc012817
Backport the pieces of the replication API changes (#9425) 2020-07-09 15:11:37 -07:00
Brian Kassouf f8df68b673
seal: Fix issue migrating from Auto->Shamir and improve tests (#9430)
* Fix issue migrating from Auto->Shamir and improve tests

* Undo newline

* fix panic in test

* Fix test panic
2020-07-09 12:28:17 -07:00
Mark Gritter 97d415d024
Token gauge metrics implementation. (#9239)
* Token gauge metrics implementation.
* Enable gauges only when interval is nonzero.
* Added count by TTL
* Yandle "in restore mode" error specifically.
* Refactored initialization code for gauge collection processes.
* Fixed for multiple namespaces.
* Ability to disable individual gauges with environment variable.
* changelog++
2020-06-23 18:36:24 -05:00
Calvin Leung Huang 101b0d3054
mfa: fix import path on test file (#9303) 2020-06-23 16:00:56 -07:00
Calvin Leung Huang c45bdca0b3
raft: add support for using backend for ha_storage (#9193)
* raft: initial work on raft ha storage support

* add note on join

* add todo note

* raft: add support for bootstrapping and joining existing nodes

* raft: gate bootstrap join by reading leader api address from storage

* raft: properly check for raft-only for certain conditionals

* raft: add bootstrap to api and cli

* raft: fix bootstrap cli command

* raft: add test for setting up new cluster with raft HA

* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends

* raft: add test for updating an existing cluster to use raft HA

* raft: remove debug log lines, clean up verifyRaftPeers

* raft: minor cleanup

* raft: minor cleanup

* Update physical/raft/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/ha.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/ha.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/logical_system_raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* address feedback comments

* address feedback comments

* raft: refactor tls keyring logic

* address feedback comments

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* address feedback comments

* testing: fix import ordering

* raft: rename var, cleanup comment line

* docs: remove ha_storage restriction note on raft

* docs: more raft HA interaction updates with migration and recovery mode

* docs: update the raft join command

* raft: update comments

* raft: add missing isRaftHAOnly check for clearing out state set earlier

* raft: update a few ha_storage config checks

* Update command/operator_raft_bootstrap.go

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* raft: address feedback comments

* raft: fix panic when checking for config.HAStorage.Type

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update website/pages/docs/commands/operator/raft.mdx

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* raft: remove bootstrap cli command

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* raft: address review feedback

* raft: revert vendored sdk

* raft: don't send applied index and node ID info if we're HA-only

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-06-23 12:04:13 -07:00
Clint 6b4bdb1882
VLT091 plugin testing framework stepwise (#9270)
* Resolve merge conflicts and updates from running a test

* move testing/_test.go over to legacy

* updates

* Add core of plugin test framework Stepwise  (#9166)

* adding stepwise testing, but there are protocol buff error :/

* move file and update sdk/go.mo

* update/sync modules

* update from other branch

* update sdk/go.mod

* some cleanups after feedback

* remove enviornments from this PR

* update vendor

* change from running go mod tidy

* change from go mod tidy

* Update sdk/testing/stepwise/helpers.go

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* Update sdk/testing/stepwise/helpers.go

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* change panic to error

* Update sdk/testing/stepwise/helpers.go

return `nil` and not `err` at the end

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* Defer close() on successful Open of a file

* document the re-creation of steps

* Update sdk/testing/stepwise/stepwise.go

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* remove unused BarrierKeys()

* Update sdk/testing/stepwise/stepwise.go

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* updates from feedback

* fix return with bad arguments

* Rename things:

- StepOperation -> Operation
- StepwiseEnvironment -> Environment
- StepCheckFunc -> AssertionFunc
- step.Check -> step.Assert

* document the environment interface methods

* rename EnvironmentOptions to MountOptions

* rename Name to RegistryName

* remove ExpectError because it's redundant

* minor doc update

* Update sdk/testing/stepwise/stepwise.go

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* add checkShouldRun function

* remove redundant return

* remove vestigial PreCheck function

* add tt.Helper() to makeRequest

* minor code formatting and document 1-based index for log output of Steps

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>

* minor updates

* update sdk

* use local reference for api, vault dep

* Update sdk/testing/stepwise/stepwise.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* Update sdk/testing/stepwise/stepwise.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* cleanup some defer functions

* call fatal if environment setup fails, and don't call teardown

* defer re-setting client token in makeRequest

* Move legacy logicaltest back to testhelpers

* update mods and test files with go mod tidy

* go mod vendor

* remove relative replace directives

* restore old logical test location

* move declaration to main stepwise file

* remove index var and use i+1

* add testing for write, delete paths of makeRequest

* update stepwise core testing to do request counting

* remove unused methods

* Update sdk/testing/stepwise/stepwise.go

remove dead line

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update sdk/testing/stepwise/stepwise.go

fix capitalization in code comment

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* update code comments for SkipTeardown to clarify its use

* update stepwise

Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
2020-06-23 06:01:39 -05:00
Mark Gritter cc6a64d4d8
Replaced ClusterMetricSink's cluster name with an atomic.Value. (#9252)
* Replaced ClusterMetricSink's cluster name with an atomic.Value.
This should permit go-race tests to pass which seal and unseal
the core.

* Replace metric sink before unseal to avoid data races.
2020-06-18 12:55:50 -05:00
Michael Golowka f77bcc53c4
Move sdk/helper/random -> helper/random (#9226)
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
2020-06-17 14:24:38 -06:00
Lauren Voswinkel a5ae18d285
Allow mTLS for mysql secrets engine (#9181)
* Extract certificate helpers for use in non-mongodb packages
* Created mTLS/X509 test for MySQL secrets engine.
* Ensure mysql username and passwords aren't url encoded
* Skip mTLS test for circleCI
2020-06-17 11:46:01 -07:00
Mike Jarmy e608503139
Test Shamir-to-Transit and Transit-to-Shamir Seal Migration for post-1.4 Vault. (#9214)
* move adjustForSealMigration to vault package

* fix adjustForSealMigration

* begin working on new seal migration test

* create shamir seal migration test

* refactor testhelpers

* add VerifyRaftConfiguration to testhelpers

* stub out TestTransit

* Revert "refactor testhelpers"

This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.

* get shamir test working again

* stub out transit join

* work on transit join

* remove debug code

* initTransit now works with raft join

* runTransit works with inmem

* work on runTransit with raft

* runTransit works with raft

* cleanup tests

* TestSealMigration_TransitToShamir_Pre14

* TestSealMigration_ShamirToTransit_Pre14

* split for pre-1.4 testing

* add simple tests for transit and shamir

* fix typo in test suite

* debug wrapper type

* test debug

* test-debug

* refactor core migration

* Revert "refactor core migration"

This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.

* begin refactor of adjustForSealMigration

* fix bug in adjustForSealMigration

* clean up tests

* clean up core refactoring

* fix bug in shamir->transit migration

* stub out test that brings individual nodes up and down

* refactor NewTestCluster

* pass listeners into newCore()

* simplify cluster address setup

* simplify extra test core setup

* refactor TestCluster for readability

* refactor TestCluster for readability

* refactor TestCluster for readability

* add shutdown func to TestCore

* add cleanup func to TestCore

* create RestartCore

* stub out TestSealMigration_ShamirToTransit_Post14

* refactor address handling in NewTestCluster

* fix listener setup in newCore()

* remove unnecessary lock from setSealsForMigration()

* rename sealmigration test package

* use ephemeral ports below 30000

* work on post-1.4 migration testing

* clean up pre-1.4 test

* TestSealMigration_ShamirToTransit_Post14 works for non-raft

* work on raft TestSealMigration_ShamirToTransit_Post14

* clean up test code

* refactor TestClusterCore

* clean up TestClusterCore

* stub out some temporary tests

* use HardcodedServerAddressProvider in seal migration tests

* work on raft for TestSealMigration_ShamirToTransit_Post14

* always use hardcoded raft address provider in seal migration tests

* debug TestSealMigration_ShamirToTransit_Post14

* fix bug in RestartCore

* remove debug code

* TestSealMigration_ShamirToTransit_Post14 works now

* clean up debug code

* clean up tests

* cleanup tests

* refactor test code

* stub out TestSealMigration_TransitToShamir_Post14

* set seals properly for transit->shamir migration

* migrateFromTransitToShamir_Post14 works for inmem

* migrateFromTransitToShamir_Post14 works for raft

* use base ports per-test

* fix seal verification test code

* simplify seal migration test suite

* simplify test suite

* cleanup test suite

* use explicit ports below 30000

* simplify use of numTestCores

* Update vault/external_tests/sealmigration/seal_migration_test.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/external_tests/sealmigration/seal_migration_test.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* clean up imports

* rename to StartCore()

* Update vault/testing.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* simplify test suite

* clean up tests

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-06-16 14:12:22 -04:00
Mark Gritter ebac0443d5
Configure metrics wrapper with the "global" object, not just the fanout. (#9099) 2020-06-16 10:50:24 -05:00
Mike Jarmy 4303790aae
Test pre-1.4 seal migration (#9085)
* enable seal wrap in all seal migration tests

* move adjustForSealMigration to vault package

* fix adjustForSealMigration

* begin working on new seal migration test

* create shamir seal migration test

* refactor testhelpers

* add VerifyRaftConfiguration to testhelpers

* stub out TestTransit

* Revert "refactor testhelpers"

This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.

* get shamir test working again

* stub out transit join

* work on transit join

* Revert "move resuable storage test to avoid creating import cycle"

This reverts commit b3ff2317381a5af12a53117f87d1c6fbb093af6b.

* remove debug code

* initTransit now works with raft join

* runTransit works with inmem

* work on runTransit with raft

* runTransit works with raft

* get rid of dis-used test

* cleanup tests

* TestSealMigration_TransitToShamir_Pre14

* TestSealMigration_ShamirToTransit_Pre14

* split for pre-1.4 testing

* add simple tests for transit and shamir

* fix typo in test suite

* debug wrapper type

* test debug

* test-debug

* refactor core migration

* Revert "refactor core migration"

This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.

* begin refactor of adjustForSealMigration

* fix bug in adjustForSealMigration

* clean up tests

* clean up core refactoring

* fix bug in shamir->transit migration

* remove unnecessary lock from setSealsForMigration()

* rename sealmigration test package

* use ephemeral ports below 30000

* simplify use of numTestCores
2020-06-11 15:07:59 -04:00
Mark Gritter 475fe0eede
Token creation counters (#9052)
* Add token creation counters.
* Created a utility to change TTL to bucket name.
* Add counter covering token creation for response wrapping.
* Fix namespace label, with a new utility function.
2020-06-02 13:40:54 -05:00
ncabatoff 8870b2e51c
Add mongodbatlas static roles support (#8987)
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
2020-05-29 14:21:23 -04:00
Lauren Voswinkel 4d98430964
Use parameters when executing prepared statements rather than fmt.Sprintf (#9013)
* Don't use string formatting to prepare queries.

We should, when possible, use the built-in params and ? format when
preparing and executing a query. This is done to prevent SQL Injection
attacks.

* Revert some changes due to failing tests, update mssql go driver

* Add docker container startup for some MSSQL tests

* Remove acceptance test flagging, add more SQL injection protection

* Refactor MSSQL prepareTestContainer to a test helper

Also, remove all ? references and convert them to @p*
2020-05-21 16:07:18 -07:00
Josh Black 6e92c8cbd2
Add a new "vault monitor" command (#8477)
Add a new "vault monitor" command

Co-authored-by: ncabatoff <ncabatoff@hashicorp.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-21 13:07:50 -07:00
Mark Gritter a82862a118
Utility for collecting new gauge metrics (#9017) 2020-05-19 17:54:43 -05:00
Jeff Mitchell b4f5d38916
Update to latest go-kms-wrapping and fix protos/etcd (#8996) 2020-05-14 18:45:10 -04:00
Mike Jarmy 8f0183a981
move resuable storage test to avoid creating import cycle (#8999) 2020-05-14 13:53:56 -04:00
Jeff Mitchell 1d3d89e2aa
Create configutil and move some common config and setup functions there (#8362) 2020-05-14 09:19:27 -04:00
Mike Jarmy 724af764bb
Test reusable storage (#8983)
* stub out reusable storage test

* implement reusable inmem test

* work on reusable raft test

* stub out simple raft test

* switch to reusable raft storage

* cleanup tests

* cleanup tests

* refactor tests

* verify raft configuration

* cleanup tests

* stub out reuseStorage

* use common base address across clusters

* attempt to reuse raft cluster

* tinker with test

* fix typo

* start debugging

* debug raft configuration

* add BaseClusterListenPort to TestCluster options

* use BaseClusterListenPort in test

* raft join works now

* misc cleanup of raft tests

* use configurable base port for raft test

* clean up raft tests

* add parallelized tests for all backends

* clean up reusable storage tests

* remove debugging code from startClusterListener()

* improve comments in testhelpers

* improve comments in teststorage

* improve comments and test logging

* fix typo in vault/testing

* fix typo in comments

* remove debugging code

* make number of cores parameterizable in test
2020-05-14 08:31:02 -04:00
Mark Gritter bd766d7bae
Metrics wrapper that adds the cluster name as a label. (#8961) 2020-05-12 21:00:59 -05:00
ncabatoff d0754790ab
Define Consul version to test against in one place, and let it be overriden by environment. Exception: some tests that verify interoperability with older Consul versions still specify a version explicitly. (#8901) 2020-04-30 13:06:24 -04:00
Brian Kassouf 21cdba6fb5
identity: Add batch entity deletion endpoint (#8785)
* identity: Add batch entity deletion endpoint

* Update the parameter description

* Update error message

* Update helper/storagepacker/storagepacker.go

Co-Authored-By: Vishal Nayak <vishalnayak@users.noreply.github.com>

* Review feedback

* Update vault/identity_store_entities.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-04-23 15:25:13 -07:00
Brian Kassouf 33b3e6857e
identity: group refresh shouldn't lock unless an update is needed (#8795) 2020-04-23 11:31:22 -07:00
Mike Jarmy ca711c2ac4
Expand seal migration tests (#8658)
* rename some seal migration tests

* add comments to seal migration tests

* add comments to seal migration tests

* rename sealhelper

* creat TestSealMigration_ShamirToTestSeal

* fix typo in test suite

* create TestSealMigration_TransitToTestSeal
2020-04-01 13:21:32 -04:00
ncabatoff c6518cc3f0
Make sure if a user gets removed from all groups in the external system, Vault updates itself accordingly. This is CVE-2020-10660. (#8606) 2020-03-23 18:00:26 -04:00
ncabatoff 20b4ca6474
Add helper for testing with a real autounseal via transit. (#8392) 2020-02-19 17:46:29 -05:00
Jeff Mitchell da5aef4878 Move kv-builder to internalshared 2020-02-19 11:13:13 -05:00
Jason O'Donnell dd9f25a118
Add OpenLDAP Secret Plugin (#8360)
* Add openldap secret plugin

* go mod vendor

* Revert to go-ldap 3.1.3

* go mod vendor
2020-02-15 13:21:07 -05:00
Brian Kassouf 3bbd9dc34c
Port some changes back to OSS (#8359) 2020-02-14 16:39:13 -08:00
Michel Vocks 985acc4ce5
Fix ldap client upndomain (#8333) 2020-02-14 10:26:30 -08:00
Michael Golowka 635b957e76
Add x509 Client Auth to MongoDB Database Plugin (#8329)
* Mark deprecated plugins as deprecated

* Add redaction capability to database plugins

* Add x509 client auth

* Update vendored files

* Add integration test for x509 client auth

* Remove redaction logic pending further discussion

* Update vendored files

* Minor updates from code review

* Updated docs with x509 client auth

* Roles are required

* Disable x509 test because it doesn't work in CircleCI

* Add timeouts for container lifetime
2020-02-13 15:54:00 -07:00
Vishal Nayak c87d34d1a4
Seal migration with Raft (#8103)
* Seal migration after unsealing

* Refactor migration fields migrationInformation in core

* Perform seal migration as part of postUnseal

* Remove the sleep logic

* Use proper seal in the unseal function

* Fix migration from Auto to Shamir

* Fix the recovery config missing issue

* Address the non-ha migration case

* Fix the multi cluster case

* Avoid re-running seal migration

* Run the post migration code in new leaders

* Fix the issue of wrong recovery being set

* Address review feedback

* Add more complete testing coverage for seal migrations.   (#8247)

* Add more complete testing coverage for seal migrations.  Also remove VAULT_ACC gate from some tests that just depend on docker, cleanup dangling recovery config in storage after migration, and fix a call in adjustCoreForSealMigration that seems broken.

* Fix the issue of wrong recovery key being set

* Adapt tests to work with multiple cores.

* Add missing line to disable raft join.

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* Fix all known issues

* Remove warning

* Review feedback.

* Revert my previous change that broke raft tests.  We'll need to come back and at least comment
this once we better understand why it's needed.

* Don't allow migration between same types for now

* Disable auto to auto tests for now since it uses migration between same types which is not allowed

* Update vault/core.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Add migration logs

* Address review comments

* Add the recovery config check back

* Skip a few steps if migration is already done

* Return from waitForLeadership if migration fails

Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-02-13 16:27:31 -05:00
Calvin Leung Huang dac3382e15
external_tests: ensure derived cores are stable before proceeding on tests (#8342)
* external_tests: ensure derived cores are stable before proceeding on tests

* testhelpers: add min duration tolerance when checking stability on derived core
2020-02-13 13:18:53 -08:00
Jeff Malnick 27ad920399
Add redshift database plugin (#8299)
* feat: add redshift database plugin

* build: update vendored libraries

* docs: add reference doc for redshift variant of the database secrets engine

* feat: set middlewear type name for better metrics naming (#8346)

Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
2020-02-13 09:42:30 -08:00
Jeff Mitchell fedb451a12 Move reload and listenerutil into sdk 2020-02-13 10:40:02 -05:00
Jim Kalafut 2ee7b76469
Bundle MongoDB Atlas (#8309) 2020-02-07 14:09:39 -08:00
Jeff Mitchell d3adf1286d Migrate gated-writer to sdk 2020-01-23 14:00:15 -05:00
Jeff Mitchell 7826f8a284
Update gated-writer logic a bit (#8227)
This is to smooth some other changes coming once
https://github.com/hashicorp/go-hclog/pull/56 lands
2020-01-23 13:57:18 -05:00