Commit graph

2308 commits

Author SHA1 Message Date
Mark Gritter fd55aa8378
Implement sys/seal-status and sys/leader in system backend (#10725)
* Implement sys/seal-status and sys/leader as normal API calls
(so that they can be used in namespaces.)
* Added changelog.
2021-01-20 14:04:24 -06:00
Nick Cabatoff 8cbc63d572
Add configuration to specify a TLS ServerName to use in the TLS handshake when performing a raft join. (#10698) 2021-01-19 17:54:28 -05:00
Nick Cabatoff c2bdeb9e7d
Minimal change to ensure that the bulky leaseEntry isn't kept in memory. (#10726) 2021-01-19 17:51:41 -05:00
Hridoy Roy 0becd555cf
Protect part of emitMetrics from panic behavior during post-seal (#10708)
* vault/core_metrics.go

* changelog

* comments
2021-01-19 14:06:50 -08:00
Scott Miller 77d27cb968
Add NIST guidance on rotating keys used for AES-GCM encryption (#10612)
* Add NIST guidance on rotating keys used for AES-GCM encryption

* Capture more places barrier encryption is used

* spacing issue

* Probabilistically track an estimated encryption count by key term

* Un-reorder imports

* wip

* get rid of sampling
2021-01-07 15:37:37 -06:00
Scott Miller c3e0d06216
Make the error response to the sys/internal/ui/mounts with no client token consistent (#10650)
* Make the error response to the sys/internal/ui/mounts with no client token consistent

* changelog

* Don't test against an empty mount path

* One other spot

* Instead, do all token checks first and early out before even looking for the mount
2021-01-07 11:46:08 -06:00
Lauren Voswinkel 7189a67a33
Adding snowflake as a bundled database secrets plugin (#10603)
* Adding snowflake as a bundled database secrets plugin

* Add snowflake-database-plugin to expected bundled plugins

* Add snowflake plugin name to the mockBuiltinRegistry
2021-01-07 09:30:24 -08:00
Mark Gritter d076d95d37
Feature flags API (#10613)
* Added sys/internal/ui/feature-flags endpoint.
* Added documentation for new API endpoint.
* Added integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
2021-01-06 16:05:00 -06:00
Nick Cabatoff e856174d15
Fix test for expiring root tokens creating non-expiring root tokens (#10632)
Test was failing (once we specified the expected error to check) because when we create a token via the TokenStore, without registering the lease in the expiration manager, lookupInternal will see that there is an expiring token with no lease and delete it immediately, yielding the "no parent found" error.
2021-01-04 09:48:22 -05:00
swayne275 a961bdc318
Fix setting Activity Log enable flag through the API (#10594)
* fix setting enable, update tests

* improve wording

* fix typo - left the testing enabled set in originally

* improve warning handling

* move from nested if to switch - TIL
2020-12-18 11:20:32 -07:00
Mark Gritter 8c67bed7ae
Send a test message before committing a new audit device. (#10520)
* Send a test message before committing a new audit device.
Also, lower timeout on connection attempts in socket device.
* added changelog
* go mod vendor (picked up some unrelated changes.)
* Skip audit device check in integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
2020-12-16 16:00:32 -06:00
Aleksandr Bezobchuk ae6267cc9b
core: add warning when disabling activity (#10485) 2020-12-15 14:11:28 -05:00
Michel Vocks 191aa65bc3
Fix UI custom header values (#10511)
* Fix UI custom header values

* Fix changelog entry

* Introduce param for multi values

* Fix multivalue

* multivalue should be bool

* Sort imports

* Fix conflict

* Remove changelog entry

* Revert entry delete
2020-12-15 15:58:03 +01:00
swayne275 cdf933adf1
say how many leases there are when threshold exceeded (#10567) 2020-12-14 16:00:19 -07:00
Aleksandr Bezobchuk 3bce568535
rate limit: fix initialize defaults (#10536) 2020-12-14 14:55:52 -05:00
Brian Kassouf 275ca323e8
core: Record the time a node became active (#10489)
* core: Record the time a node became active

* Update vault/core.go

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Add omitempty field

* Update vendor

* Added CL entry and fixed test

* Fix test

* Fix command package tests

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2020-12-11 16:50:19 -08:00
Calvin Leung Huang f137c945d7
ha: update godoc on grabLockOrStop (#10547) 2020-12-11 16:04:00 -08:00
Seth Bunce a6a437a1ce
fix deadlock on core state lock (#10456)
* fix race that can cause deadlock on core state lock

The bug is in the grabLockOrStop function. For specific concurrent
executions the grabLockOrStop function can return stopped=true when
the lock is still held. A comment in grabLockOrStop indicates that the
function is only used when the stateLock is held, but grabLockOrStop is
being used to acquire the stateLock. If there are concurrent goroutines
using grabLockOrStop then some concurrent executions result in
stopped=true being returned when the lock is acquired.

The fix is to add a lock and some state around which the parent and
child goroutine in the grabLockOrStop function can coordinate so that
the different concurrent executions can be handled.

This change includes a non-deterministic unit test which reliably
reproduces the problem before the fix.

* use rand instead of time for random test stopCh close

Using time.Now().UnixNano()%2 ends up being system dependent because
different operating systems and hardware have different clock
resolution. A lower resolution will return the same unix time for a
longer period of time.

It is better to avoid this issue by using a random number generator.
This change uses the rand package default random number generator. It's
generally good to avoid using the default random number generator,
because it creates extra lock contention. For a test it should be fine.
2020-12-10 06:50:11 -05:00
Nick Cabatoff b425be1a93
Fix race with test that mutates KeyRotateGracePeriod: make the global be a Core field instead. (#10512) 2020-12-08 13:57:44 -05:00
Nick Cabatoff 84d566db9e
Be consistent with how we report init status. (#10498)
Also make half-joined raft peers consider storage to be initialized, whether or not they're sealed.
2020-12-08 13:55:34 -05:00
Hridoy Roy 0ada870a52
Only use entropy augmentation for root token creation [VAULT-670] (#10487)
* Only use entropy augmentation for root token creation

* changelog

* change wording of changelog entry
2020-12-04 09:44:04 -08:00
swayne275 88eaf5f4c3
Fix Racy Activity Log Tests (#10484)
* fix racy activity log tests and move testing utilities elsewhere

* remove TODO

* move SetEnable out of activity log

* clarify not waiting on waitgroup

* remove todo
2020-12-02 13:48:13 -07:00
Michael Golowka cc7efd393d
MySQL - Fix username generation length bug (#10433) 2020-12-01 15:24:51 -07:00
Scott Miller 32cb144d0d
Update HCL dependency to fix ParseACLPolicy error on invalid syntax (#10156) 2020-11-30 09:17:33 -06:00
Nick Cabatoff 818f8aeff2
Fix test failures of the form "bad start timestamp. expected: 1606313752 got: 1606313753". Also re-enable a test that probably shouldn't be skipped, and delete a test that's meant for ent (see also https://github.com/hashicorp/vault-enterprise/pull/1613) (#10452) 2020-11-25 13:49:47 -05:00
Nick Cabatoff d8e7d2e2b8
Use a lock to address race. (#10447) 2020-11-25 12:24:28 -05:00
Nick Cabatoff 6faef07fd5
Factor out the consul-using sealmigration tests to their own package, so that the remaining tests can run in the CI job that doesn't need docker. (#10342)
Remove the file-storage-backed tests: they don't add anything, and they don't represent a viable cluster storage solution that can be used in prod.
2020-11-20 07:53:31 -05:00
Mark Gritter 0bc1226084
Disable test that fails on OSS. (#10401) 2020-11-16 12:23:21 -06:00
Hridoy Roy 6261afb343
Port: Telemetry For Lease Expiration Times (#10375)
* port lease metrics

* go mod vendor

* caught a bug
2020-11-13 10:26:58 -08:00
Hridoy Roy c5aa1c715f
reverting the tests to not use metrics when unnecessary (#10350) 2020-11-11 15:35:09 -08:00
Mark Gritter 52ddad2bc5
Wait for asynchronous deletion to finish before restarting. (#10345) 2020-11-06 18:21:04 -06:00
Scott Miller d53e26fb7a
Backport last quota fix changes to OSS (#10335)
* Backport last quota fix changes to OSS

* Get all unit tests

* dupe test
2020-11-06 09:46:31 -06:00
Vishal Nayak 36a5bd946a
Fix quota conflict error (#10285)
Co-authored-by: Scott Miller <smiller@hashicorp.com>
2020-11-05 10:18:07 -06:00
Mark Gritter 91ca298a14
Move "counters" path to the logical system's local path list. (#10314) 2020-11-02 21:59:55 -06:00
swayne275 dffd85e09a
Backport invalidation changes (#10292)
* merge activity log invalidation work from vault-enterprise PR 1546

* skip failing test due to enabled config on oss

Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
2020-10-30 18:11:12 -06:00
Mark Gritter 7f01a58aee
Reintroduce a feature flag to disable the activity log entirely. (#10288)
* Reintroduce a feature flag to disable the activity log entirely.
* Add log message when disabled.
2020-10-30 18:27:35 -05:00
Brian Kassouf 8af08c3221
Add an env var to enable a permit pool that limits lease expirations (#10268)
* Add a flag to enable a permit pool to gate lease expiration

* Use the env var to get the size

* Add logs and metris to help debug this

Co-authored-by: Hridoy Roy <roy@hashicorp.com>
2020-10-30 14:45:44 -07:00
Brian Kassouf 81a86f48e8
Backport some OSS changes (#10267)
* Backport some OSS changes

* go mod vendor
2020-10-29 16:47:34 -07:00
Vishal Nayak 90a9528610
added test for concurrency call of remount handler and proposed fix for logic to avoid duplication of mount names (#10264)
Co-authored-by: bruj0 <ramakandra@gmail.com>
2020-10-29 14:39:41 -04:00
Vishal Nayak 30fe58a458
Fix remount tests (#10265) 2020-10-29 14:31:58 -04:00
Matt Greenfield 2f369730e0
Validate to/from parameters when remounting a backend (#9890)
Vault uses http.ServeMux which issues an HTTP 301 redirect if the
request path contains a double slash (`//`). Additionally, vault
handles all paths to ensure that the path only contains printable
characters. Therefore use the same validation on the to/from parameters
for remounting.

Not doing this can result in a Vault mount that was originally mounted
at `pki/foo` to being remounted at `pki/foo//bar` resulting in mounts
that cannot be accessed.

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-10-29 14:06:07 -04:00
Hridoy Roy fc94e16805
port external test fix (#10263) 2020-10-29 10:55:26 -07:00
Hridoy Roy f8a248ce48
Port: change leader status metric name to active (#10245)
* change active node metric name

* comment to see if commit is fine

Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MacBook-Pro.local>
2020-10-29 10:30:45 -07:00
Hridoy Roy 0259be04e0
Port: Add metrics to report mount table sizes for auth and logical [Vault 671] (#10201)
* first commit

* update

* removed some ent features from backport

* final refactor

* backport patch

Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MBP.hitronhub.home>
2020-10-27 08:24:43 -07:00
Jeff Mitchell 3b93a18ef2
Consolidate locking for sys/health (#9876)
* Consolidate locking for sys/health

This avoids a second state lock read-lock on every sys/health hit

* Address review feedback

Co-authored-by: Vishal Nayak <vishalnayakv@gmail.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-10-26 16:47:54 -04:00
Calvin Leung Huang ca8435bf4e
auth: store period value on tokens created via login (#7885)
* auth: store period value on tokens created via login

* test: reduce potentially flaskiness due to ttl check

* test: govet on package declaration

* changelog++

* Temporarily remove CL entry

* Add back the CL entry

Co-authored-by: Vishal Nayak <vishalnayakv@gmail.com>
2020-10-26 16:25:56 -04:00
Clint 95810d1360
Return logical.StatusBadRequest on requests with missing token (#8457)
* Add test for 400 status on missing token

* Return logical.StatusBadRequest on missing token

* remove commented out code

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-10-26 16:17:25 -04:00
Jeff Mitchell a07b6ba1d2
Add omitempty's to MountEntry and MountConfig (#7154)
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-10-26 15:34:55 -04:00
Aleksandr Bezobchuk 95bbd8d920
Merge PR #10192: Auto-Join: Configurable Scheme & Port (and add k8s provider) 2020-10-23 16:13:09 -04:00
Nick Cabatoff 0d6a929a4c
Same seal migration oss (#10224)
* Refactoring and test improvements.

* Support migrating from a given type of autoseal to that same type but with different parameters.
2020-10-23 14:16:04 -04:00
Michael Golowka bd79fbafb3
Add couchbase, elasticsearch, and mongodbatlas back (#10222)
Updated the `Serve` function so these can be added back into Vault
2020-10-22 17:20:17 -06:00
Michael Golowka e6c8ee24ea
DBPW - Enables AutoMTLS for DB plugins (#10220)
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
2020-10-22 15:43:19 -06:00
Aleksandr Bezobchuk 0d6a0ec589
Merge PR #10010: Rate Limit Quotas: Allow Exempt Paths to be Configurable 2020-10-16 14:58:19 -04:00
Nick Cabatoff 66274607b7
OSS changes for enterprise automated snapshots (#10160) 2020-10-16 14:57:11 -04:00
Hridoy Roy bd2dc7734c
Backport leader status telemetry [VAULT-672] (#10147)
* backport VAULT-672

* backport VAULT-672

* go mod tidy

* go mod tidy

* add back indirect import

* replace go mod and go sum with master version

* go mod vendor

* more go mod vendor

Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MBP.hitronhub.home>
Co-authored-by: Hridoy Roy <hridoyroy@Hridoys-MacBook-Pro.local>
2020-10-15 14:15:58 -07:00
Michael Golowka 1888323243
DBPW - Copy newdbplugin package to dbplugin/v5 (#10151)
This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.
2020-10-15 13:20:12 -06:00
Brian Kassouf 84dbca38a1
Revert "Migrate internalshared out (#9727)" (#10141)
This reverts commit ee6391b691ac12ab6ca13c3912404f1d3a842bd6.
2020-10-13 16:38:21 -07:00
Aleksandr Bezobchuk d37be9af6e
Merge PR #10095: Integrated Storage Cloud Auto-Join 2020-10-13 16:26:39 -04:00
Connor 8b1a3301f0
Add telemetry for LRU cache (#10079)
Vault creates an LRU cache that is used when interacting with the
physical backend. Add telemetry when the cache is hit, missed, written
to and deleted from. Use the MetricSink from ClusterMetrics
2020-10-13 10:11:54 -05:00
Jeff Mitchell e6881c8147
Migrate internalshared out (#9727)
* Migrate internalshared out

* fix merge issue

* fix merge issue

* go mod vendor

Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
2020-10-12 11:56:24 -07:00
Lauren Voswinkel 55761c1d43
Migrate MySQL db to v5 database engine (#10110)
* Migrate MySQL db to v5 database engine
2020-10-09 10:32:38 -07:00
Brian Kassouf cb37fda0a7
Expiration: Fix lease counting logic (#10106) 2020-10-07 17:27:45 -07:00
Ivan Lemeshev c84be1c874
Fix panic in RekeyVerifyRestart (#9930) (#10099) 2020-10-07 11:06:17 -07:00
Ivan Lemeshev 8a2842b467
Fix panic in handleAuditNonLogical if vault is sealed (#9310) (#10103) 2020-10-07 08:30:36 -04:00
Aleksandr Bezobchuk a3cfa7c447
Merge PR #10059: Port OSS changes from #1497 2020-10-01 15:15:20 -04:00
Brian Kassouf b0d3d9bf49
Update lease timer logic (#10030) 2020-09-23 11:46:22 -07:00
ncabatoff 3fb20ed529
Logging tweaks, mostly for tests (#10028)
Fix some places where raft wasn't hooking into the core logger as it should.
Revisited the code that was setting the log level to Error during cleanup: it's normal for there to be a bunch of errors then, which makes it harder to see what went wrong up to the point where the test was deemed to have failed.  So now, instead of setting log level to Error, we actually stop logging altogether.  This only applies if the test didn't pass in its own logger during cluster creation, but we should be moving away from that anyway.
2020-09-23 13:40:00 -04:00
Brian Kassouf 3f30fc5f4e
Port changes from enterprise lease fix (#10020) 2020-09-22 14:47:13 -07:00
Mark Gritter 38ae7efca3
Switch to PerfStandby() where possible. (#9993) 2020-09-22 14:48:30 -05:00
Michael Golowka 60e0cbbc37
[DBPW 4/X] Update DB engine to support v4 and v5 interfaces with password policies (#9878) 2020-09-18 15:10:54 -06:00
ncabatoff 0f77d0e282
Move the code that creates Consul containers out of teststorage. This allows importers of teststorage that don't need consul to run as a non-docker test. (#9975) 2020-09-17 15:44:29 -04:00
ncabatoff 2cb1e03c1b
Fix flaky TestCluster_ForwardRequest tests. (#9973)
We now wait to see that clients are aware of a new leader before we hit them, and we wait to see that the leader has setup the forwarding handler.
2020-09-16 15:31:06 -04:00
Mark Gritter 587ed7d499
Disable usage metrics on performance standby nodes. (#9966) 2020-09-15 17:12:28 -05:00
ncabatoff b615da43d7
Run CI tests in docker instead of a machine. (#8948) 2020-09-15 10:01:26 -04:00
Mark Gritter 4633f5a8fc
Disable flaky test case. (#9926) 2020-09-10 17:54:31 -05:00
swayne275 22f7a0ebef
VAULT-417: check expired context in entity API (#1445) (#9925) 2020-09-10 16:31:32 -06:00
Mark Gritter 1b2c20e07c
Merge activity log work to date on enterprise back into oss. (#9900)
* Added stub class for activity logging. (#1435)
* Define activity fragments and starter methods for manipulating them. (#1441)
2020-09-08 14:22:09 -05:00
Mark Gritter 3690774f7b
Fix crash when KV store has a zero-length key. (#9881)
* Fix crash when KV store has a zero-length key.
* Add PR to changelog.
2020-09-02 17:43:44 -05:00
ncabatoff 30eba1eed1
Update retryablehttp to fix a data race (#9551) 2020-08-31 11:10:52 -04:00
Alexander Bezobchuk f873863263
Merge PR #9667: Rate Limit Backoff 2020-08-16 22:09:18 -04:00
ncabatoff 4134ef2e98
Ensure that perf standbys can perform seal migrations. (#9690) 2020-08-10 08:35:57 -04:00
Rodrigo D. L d0df8bfa21
adding new config flag disable_sentinel_trace (#9696) 2020-08-10 06:23:44 -04:00
Tom Proctor 4ca978598f
Bundle couchbase database plugin with vault (#9664) 2020-08-07 11:01:04 +01:00
ncabatoff b6fd378ee8
Make manualStepDownCh a 1-buffered channel to ensure StepDown actually steps down in tests. (#9622) 2020-07-31 10:01:51 -04:00
ncabatoff 1154b36b56
Log sanitized config at startup and when it changes. (#9637)
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
2020-07-30 13:15:00 -04:00
Alexander Bezobchuk 1e262e5648
Merge PR #9581: Rate Limit Quota Headers 2020-07-29 15:15:05 -04:00
Scott Miller 16ca436dd8
Increase expiration timeouts on leases to avoid races in NoopBackend (#9600)
* Increase expiration timeouts on leases to avoid races in NoopBackend

* Set timeouts depending on whether they are relevant to the test: 1s for irrelevant, back to 20ms if they are

* revert one more
2020-07-29 10:13:47 -05:00
ncabatoff 003bccd16e
Eliminate global that caused race tests to fail in ent with an internal config setting. (#9604) 2020-07-27 16:10:26 -04:00
ncabatoff ee6e2344dd
Fix a race caused by assignment to core.metricSink (#9560) 2020-07-22 13:52:10 -04:00
ncabatoff 7484fd7c72
Handle "invalid request" failures stemming from a kvv2 upgrade. (#9550) 2020-07-22 10:56:00 -04:00
ncabatoff 3fbc0f35c2
Make runTransit tolerate a non-core-0 leader. (#9548) 2020-07-21 15:50:01 -04:00
Brian Kassouf fd72d92434
raft: Fix some snapshot restore issues (#9533)
* raft: Remove double read lock

* Reload TLS keyring after reloading the barrier keys
2020-07-21 10:59:07 -07:00
ncabatoff d2436a9c56
Make standbyStopCh atomic to avoid data races (#9539) 2020-07-21 08:34:07 -04:00
ncabatoff d777708fde
Improve logging, and add polling to the post-stepdown leader check. (#9530) 2020-07-20 12:44:23 -04:00
ncabatoff a31fd27069
Use the accessor method so state lock is used to check perf standby status. (#9496) 2020-07-20 10:34:16 -04:00
ncabatoff 3ddc837ce3
Make sure cluster is stopped before wiping storage. (#9526) 2020-07-20 09:32:38 -04:00
Vishal Nayak 14779e9f9d
Remove prefix for exception paths; add sys/internal/ui/mounts (#9512) 2020-07-17 10:55:38 -04:00
Mike Jarmy 93ff4c098c
Add a lock to seal migration (#9485)
* add a lock to seal migration

* switch to CompareAndSwapInt32

* switch to uber go-atomic
2020-07-16 15:14:29 -04:00
Alexander Bezobchuk ea13485c7d
Merge PR #9502: Resource Quotas: Remove 'burst' Param from Rate Limiter 2020-07-16 14:34:43 -04:00
Vishal Nayak 4e4cb5289a
Fetch quota by name before updating it (#9466)
* Fix quotas update

* Update doc
2020-07-15 13:25:00 -04:00
Mark Gritter c4dbbccef3
Remove namespace from mount_point label. (#9436)
* Remove namespace from mount_point label.
* Fix the other two places where vault.token.creation is emitted.
2020-07-14 14:28:11 -05:00
Calvin Leung Huang cf354f6438
quotas: fix data race that could occur if ApplyQuota was called durin… (#9458)
* quotas: fix data race that could occur if ApplyQuota was called during a db reset

* Abstract out the locking caller

* Remove unneeded lock

* Update

Co-authored-by: Vishal Nayak <vishalnayakv@gmail.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-07-13 11:42:39 -07:00
ncabatoff c822a4bffb
Modifying base can break tests that share the same coreConfig passed to NewTestCluster. (#9451) 2020-07-10 13:11:18 -04:00
Josh Black 38fc012817
Backport the pieces of the replication API changes (#9425) 2020-07-09 15:11:37 -07:00
Alexander Bezobchuk f7c1907683
Merge PR #9437: tests: Port #1315 from enterprise 2020-07-09 17:16:31 -04:00
Brian Kassouf f8df68b673
seal: Fix issue migrating from Auto->Shamir and improve tests (#9430)
* Fix issue migrating from Auto->Shamir and improve tests

* Undo newline

* fix panic in test

* Fix test panic
2020-07-09 12:28:17 -07:00
Alexander Bezobchuk dfb28a8fcc
Merge PR #9390: http: revert resource quota changes 2020-07-07 00:05:28 -04:00
Calvin Leung Huang 67444d85b8
test/migration: ensure that leader client is used for storage read check (#9403) 2020-07-06 16:22:07 -07:00
Scott Miller dfbc60cc49
Add mount type to request earlier, so it is populated on audit entries even if the request fails authorization (#9389) 2020-07-06 11:17:41 -05:00
Alexander Bezobchuk f1534a0ed0
Add nil check for quota manager (#9379)
* Add nil check for quota manager

* Add missing nil checks
2020-07-01 18:14:33 -07:00
Alexander Bezobchuk fb9cd9db2a
Merge PR #9372: Fix Unauthenticated list 2020-07-01 16:02:14 -04:00
Mark Gritter 707fdea702
Don't return quota error on revoke. (#9374)
Changed log messages to be clearer about quota operations.
This should fix enterprise unit test failures.
2020-07-01 14:41:42 -05:00
Calvin Leung Huang 37c0e51142
logical/system: re-introduce ns-awareness in pathInternalUIMountsRead (#9373) 2020-07-01 12:37:11 -07:00
Scott Miller a6f62359a9
Don't setup plugin reload on perf standbys (#9352) 2020-06-30 17:32:06 -05:00
Mike Jarmy 4b2cdfee72
re-enable seal migration (#9351)
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-06-30 18:21:18 -04:00
Scott Miller ad292bec73
Fix wrong err return value in plugin reload status command (#9348)
* Fix wrong return value (discovered when merging to ENT)

* go.mod

* go mod vendor

* Add setup plugin reload hook

* All reloads return something now
2020-06-30 13:33:30 -05:00
Scott Miller 001ee861bd
Global Plugin Reload: OSS Changes Take II (#9347)
* Carefully move changes from the plugin-cluster-reload branch into this clean branch off master.

* Don't test this at this level, adequately covered in the api level tests

* Change PR link

* go.mod

* Vendoring

* Vendor api/sys_plugins.go
2020-06-30 10:26:52 -05:00
Scott Miller e92f8f5a81
Revert global plugin reload commits (#9344)
* Revert "Some of the OSS changes were clobbered when merging with quotas out of, master (#9343)"

This reverts commit 8719a9b7c4d6ca7afb2e0a85e7c570cc17081f41.

* Revert "OSS side of Global Plugin Reload (#9340)"

This reverts commit f98afb998ae50346849050e882b6be50807983ad.
2020-06-29 17:36:22 -05:00
Scott Miller cc51427584
Some of the OSS changes were clobbered when merging with quotas out of, master (#9343)
* OSS side of Global Plugin Reload
2020-06-29 16:58:51 -05:00
Scott Miller a83fe0fc6d
OSS side of Global Plugin Reload (#9340)
* OSS side of Global Plugin Reload

* changelog++
2020-06-29 16:23:28 -05:00
ncabatoff d42ee4f7ef
Ensure "initialized" service registration tag is also present whenever Vault is unsealed, on both Consul and K8s (#8990)
* Add the initialized tag to Consul registration for parity with k8s (and for easy automated testing).  Ensure that whenever we flag Vault as unsealed, we also flag it as initialized.

* Update API docs.

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
2020-06-29 16:02:49 -04:00
Calvin Leung Huang babaa93a0f
monitor: watch for seal state during monitor request (#9341)
* monitor: watch for seal state during monitor request

* monitor: return error regardless of how upstream handles it
2020-06-29 12:58:41 -07:00
Mark Gritter 873acbefbd
Unit test that fails to demonstrate identity store problem. (#9339)
Fix test compilation error.
2020-06-29 13:33:25 -05:00
Vishal Nayak 6bd5674345
Reset quota manager during shutdown (#9331) 2020-06-29 13:23:10 -04:00
Vishal Nayak c6876fe00f
Resource Quotas: Rate Limiting (#9330) 2020-06-26 17:13:16 -04:00
Javier Ramos 16070564cb
Calculate percentage when displaying progress in tidy operation (#9233)
* Calculate percentage when displaying progress in tidy operation

* Update vault/token_store.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/token_store.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-06-26 12:44:45 -07:00
Mark Gritter a5d2edece2
Enable identity entity collection. (#9324) 2020-06-25 18:54:38 -05:00
Scott Miller 57c6ae4233
Test for overflow of the capacity value (#9317) 2020-06-25 11:22:13 -05:00
Mark Gritter a54a3b6a66
Entity and alias counts (#9262)
* Added gauge collectors for entity counts.
* Entity and alias gauges.
* Locking around accessor to core.identityStore and core.mount.
2020-06-23 19:45:59 -05:00
Mark Gritter 97d415d024
Token gauge metrics implementation. (#9239)
* Token gauge metrics implementation.
* Enable gauges only when interval is nonzero.
* Added count by TTL
* Yandle "in restore mode" error specifically.
* Refactored initialization code for gauge collection processes.
* Fixed for multiple namespaces.
* Ability to disable individual gauges with environment variable.
* changelog++
2020-06-23 18:36:24 -05:00
Calvin Leung Huang c45bdca0b3
raft: add support for using backend for ha_storage (#9193)
* raft: initial work on raft ha storage support

* add note on join

* add todo note

* raft: add support for bootstrapping and joining existing nodes

* raft: gate bootstrap join by reading leader api address from storage

* raft: properly check for raft-only for certain conditionals

* raft: add bootstrap to api and cli

* raft: fix bootstrap cli command

* raft: add test for setting up new cluster with raft HA

* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends

* raft: add test for updating an existing cluster to use raft HA

* raft: remove debug log lines, clean up verifyRaftPeers

* raft: minor cleanup

* raft: minor cleanup

* Update physical/raft/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/ha.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/ha.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/logical_system_raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* address feedback comments

* address feedback comments

* raft: refactor tls keyring logic

* address feedback comments

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* address feedback comments

* testing: fix import ordering

* raft: rename var, cleanup comment line

* docs: remove ha_storage restriction note on raft

* docs: more raft HA interaction updates with migration and recovery mode

* docs: update the raft join command

* raft: update comments

* raft: add missing isRaftHAOnly check for clearing out state set earlier

* raft: update a few ha_storage config checks

* Update command/operator_raft_bootstrap.go

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* raft: address feedback comments

* raft: fix panic when checking for config.HAStorage.Type

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update website/pages/docs/commands/operator/raft.mdx

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* raft: remove bootstrap cli command

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* raft: address review feedback

* raft: revert vendored sdk

* raft: don't send applied index and node ID info if we're HA-only

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-06-23 12:04:13 -07:00
Mark Gritter ce88903638
Add counter for creation of new entity. (#9263) 2020-06-19 18:24:05 -05:00
Mark Gritter 3e220f300a
Fix to failing test compilation. (#9272) 2020-06-19 15:19:21 -05:00
Mark Gritter b3c3635f49
Added gauges to count KV secrets. (#9250)
* Added gauges to count KV secrets.
* Use real KV implementation in test.
2020-06-19 14:01:35 -05:00
Mark Gritter 239b2375aa
Counter that increments on every secret engine lease creation. (#9244) 2020-06-18 15:36:21 -05:00
Mark Gritter cc6a64d4d8
Replaced ClusterMetricSink's cluster name with an atomic.Value. (#9252)
* Replaced ClusterMetricSink's cluster name with an atomic.Value.
This should permit go-race tests to pass which seal and unseal
the core.

* Replace metric sink before unseal to avoid data races.
2020-06-18 12:55:50 -05:00
Michael Golowka f77bcc53c4
Move sdk/helper/random -> helper/random (#9226)
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
2020-06-17 14:24:38 -06:00
Scott Miller 0b9a40a64e
Add a simple sealed gauge, updated when seal status changes (#9177)
* Add a simple unsealed gauge, updated when seal status changes
2020-06-17 09:50:28 -05:00
Mike Jarmy e608503139
Test Shamir-to-Transit and Transit-to-Shamir Seal Migration for post-1.4 Vault. (#9214)
* move adjustForSealMigration to vault package

* fix adjustForSealMigration

* begin working on new seal migration test

* create shamir seal migration test

* refactor testhelpers

* add VerifyRaftConfiguration to testhelpers

* stub out TestTransit

* Revert "refactor testhelpers"

This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.

* get shamir test working again

* stub out transit join

* work on transit join

* remove debug code

* initTransit now works with raft join

* runTransit works with inmem

* work on runTransit with raft

* runTransit works with raft

* cleanup tests

* TestSealMigration_TransitToShamir_Pre14

* TestSealMigration_ShamirToTransit_Pre14

* split for pre-1.4 testing

* add simple tests for transit and shamir

* fix typo in test suite

* debug wrapper type

* test debug

* test-debug

* refactor core migration

* Revert "refactor core migration"

This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.

* begin refactor of adjustForSealMigration

* fix bug in adjustForSealMigration

* clean up tests

* clean up core refactoring

* fix bug in shamir->transit migration

* stub out test that brings individual nodes up and down

* refactor NewTestCluster

* pass listeners into newCore()

* simplify cluster address setup

* simplify extra test core setup

* refactor TestCluster for readability

* refactor TestCluster for readability

* refactor TestCluster for readability

* add shutdown func to TestCore

* add cleanup func to TestCore

* create RestartCore

* stub out TestSealMigration_ShamirToTransit_Post14

* refactor address handling in NewTestCluster

* fix listener setup in newCore()

* remove unnecessary lock from setSealsForMigration()

* rename sealmigration test package

* use ephemeral ports below 30000

* work on post-1.4 migration testing

* clean up pre-1.4 test

* TestSealMigration_ShamirToTransit_Post14 works for non-raft

* work on raft TestSealMigration_ShamirToTransit_Post14

* clean up test code

* refactor TestClusterCore

* clean up TestClusterCore

* stub out some temporary tests

* use HardcodedServerAddressProvider in seal migration tests

* work on raft for TestSealMigration_ShamirToTransit_Post14

* always use hardcoded raft address provider in seal migration tests

* debug TestSealMigration_ShamirToTransit_Post14

* fix bug in RestartCore

* remove debug code

* TestSealMigration_ShamirToTransit_Post14 works now

* clean up debug code

* clean up tests

* cleanup tests

* refactor test code

* stub out TestSealMigration_TransitToShamir_Post14

* set seals properly for transit->shamir migration

* migrateFromTransitToShamir_Post14 works for inmem

* migrateFromTransitToShamir_Post14 works for raft

* use base ports per-test

* fix seal verification test code

* simplify seal migration test suite

* simplify test suite

* cleanup test suite

* use explicit ports below 30000

* simplify use of numTestCores

* Update vault/external_tests/sealmigration/seal_migration_test.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/external_tests/sealmigration/seal_migration_test.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* clean up imports

* rename to StartCore()

* Update vault/testing.go

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>

* simplify test suite

* clean up tests

Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-06-16 14:12:22 -04:00
Mark Gritter 50b388a93c
Changes to expiration manager to walk tokens (#9182)
* Changes to expiration manager to walk tokens (including non-expiring ones.)
* Count by namespace in token manager.
* Keep a dictionary of policy lists and deduplicate based on it.
2020-06-15 18:54:36 -05:00
Austin Gebauer a826bd1d23
fix: configutil redeclared as imported package name (#9211) 2020-06-11 18:08:46 -07:00
Michael Golowka a89f09802d
Integrate password policies into RabbitMQ secret engine (#9143)
* Add password policies to RabbitMQ & update docs
* Also updates some parts of the password policies to aid/fix testing
2020-06-11 16:08:20 -06:00
Mike Jarmy 4303790aae
Test pre-1.4 seal migration (#9085)
* enable seal wrap in all seal migration tests

* move adjustForSealMigration to vault package

* fix adjustForSealMigration

* begin working on new seal migration test

* create shamir seal migration test

* refactor testhelpers

* add VerifyRaftConfiguration to testhelpers

* stub out TestTransit

* Revert "refactor testhelpers"

This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.

* get shamir test working again

* stub out transit join

* work on transit join

* Revert "move resuable storage test to avoid creating import cycle"

This reverts commit b3ff2317381a5af12a53117f87d1c6fbb093af6b.

* remove debug code

* initTransit now works with raft join

* runTransit works with inmem

* work on runTransit with raft

* runTransit works with raft

* get rid of dis-used test

* cleanup tests

* TestSealMigration_TransitToShamir_Pre14

* TestSealMigration_ShamirToTransit_Pre14

* split for pre-1.4 testing

* add simple tests for transit and shamir

* fix typo in test suite

* debug wrapper type

* test debug

* test-debug

* refactor core migration

* Revert "refactor core migration"

This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.

* begin refactor of adjustForSealMigration

* fix bug in adjustForSealMigration

* clean up tests

* clean up core refactoring

* fix bug in shamir->transit migration

* remove unnecessary lock from setSealsForMigration()

* rename sealmigration test package

* use ephemeral ports below 30000

* simplify use of numTestCores
2020-06-11 15:07:59 -04:00
ncabatoff 2ec9049ef7
Add ssh signing algorithm as a role option. (#9096) 2020-06-11 08:10:13 -04:00
ncabatoff fdba917b66
Fix feature flag persistence: we shouldn't have excluded dr primaries, they too must write feature flags. DR secondaries might not need depend on feature flags being there, but a DR primary could also be (or become) a perf primary. (#9148) 2020-06-04 13:00:33 -04:00
Mark Gritter 475fe0eede
Token creation counters (#9052)
* Add token creation counters.
* Created a utility to change TTL to bucket name.
* Add counter covering token creation for response wrapping.
* Fix namespace label, with a new utility function.
2020-06-02 13:40:54 -05:00
ncabatoff 8870b2e51c
Add mongodbatlas static roles support (#8987)
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
2020-05-29 14:21:23 -04:00
Scott Miller 12d704d97f
Provide token ttl and issue time in the audit log. (#9091)
* Populate a token_ttl and token_issue_time field on the Auth struct of audit log entries, and in the Auth portion of a response for login methods

* Revert go fmt, better zero checking

* Update unit tests

* changelog++
2020-05-29 12:30:47 -05:00
Michael Golowka b52950f884
Add user configurable password policies available to secret engines (#8637)
* Add random string generator with rules engine

This adds a random string generation library that validates random
strings against a set of rules. The library is designed for use as generating
passwords, but can be used to generate any random strings.
2020-05-27 12:28:00 -06:00
Josh Black 6e92c8cbd2
Add a new "vault monitor" command (#8477)
Add a new "vault monitor" command

Co-authored-by: ncabatoff <ncabatoff@hashicorp.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-21 13:07:50 -07:00
Mark Gritter 71b3de0450
Switch expiration manager's pending map to a sync.Map. (#8589) 2020-05-21 12:41:03 -05:00
ncabatoff 5376e87360
Don't audit replication status requests or responses. (#8877) 2020-05-21 09:20:05 -04:00
Brian Kassouf c8dde052f2
storage/raft: Advertise the configured cluster address (#9008)
* storage/raft: Advertise the configured cluster address

* Don't allow raft to start with unspecified IP

* Fix concurrent map write panic

* Add test file

* changelog++

* changelog++

* changelog++

* Update tcp_layer.go

* Update tcp_layer.go

* Only set the adverise addr if set
2020-05-18 18:22:25 -07:00
Brian Kassouf 1bb0bd489d
storage/raft: Add committed and applied indexes to the status output (#9011)
* storage/raft: Add committed and applied indexes to the status output

* Update api vendor

* changelog++

* Update http/sys_leader.go

Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>

Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
2020-05-18 16:07:27 -07:00
Calvin Leung Huang 8cefbca1c9
Refactor service registration (#8976)
* serivceregistration: refactor service registration logic to run later

* move state check to the internal func

* sr/kubernetes: update setInitialStateInternal godoc

* sr/kubernetes: remove return in setInitialState

* core/test: fix mockServiceRegistration

* address review feedback
2020-05-15 11:06:58 -07:00
Jeff Mitchell b4f5d38916
Update to latest go-kms-wrapping and fix protos/etcd (#8996) 2020-05-14 18:45:10 -04:00
Jeff Mitchell 1d3d89e2aa
Create configutil and move some common config and setup functions there (#8362) 2020-05-14 09:19:27 -04:00
Mike Jarmy 724af764bb
Test reusable storage (#8983)
* stub out reusable storage test

* implement reusable inmem test

* work on reusable raft test

* stub out simple raft test

* switch to reusable raft storage

* cleanup tests

* cleanup tests

* refactor tests

* verify raft configuration

* cleanup tests

* stub out reuseStorage

* use common base address across clusters

* attempt to reuse raft cluster

* tinker with test

* fix typo

* start debugging

* debug raft configuration

* add BaseClusterListenPort to TestCluster options

* use BaseClusterListenPort in test

* raft join works now

* misc cleanup of raft tests

* use configurable base port for raft test

* clean up raft tests

* add parallelized tests for all backends

* clean up reusable storage tests

* remove debugging code from startClusterListener()

* improve comments in testhelpers

* improve comments in teststorage

* improve comments and test logging

* fix typo in vault/testing

* fix typo in comments

* remove debugging code

* make number of cores parameterizable in test
2020-05-14 08:31:02 -04:00
Mark Gritter bd766d7bae
Metrics wrapper that adds the cluster name as a label. (#8961) 2020-05-12 21:00:59 -05:00
Scott Miller c3b6db2f26
Reject requests read and write requests to cubbyhole with an empty path (#8971)
* Reject requests read and write requests to cubbyhole with an empty path
2020-05-11 14:15:36 -05:00
Calvin Leung Huang cef02f4803
sys: use full mount path for checkAccess on internal/ui/mounts/:path (#8962) 2020-05-08 16:02:40 -07:00
ncabatoff 862beef59d
Factor out test logger creation so that it can be used when the caller wants to specify a logger explicitly, e.g. because they also want to use it when creating storage. (#8916) 2020-05-05 12:11:36 -04:00
ncabatoff a6809fdbaf
Write NewTestCluster logs to files, which get deleted if the test passes (#8911) 2020-05-01 12:26:26 -04:00
Calvin Leung Huang ec8448ab56
token: disallow periods on custom token IDs (#8646)
* token: disallow periods on custom token IDs

* docs: update token API docs
2020-04-27 09:39:33 -07:00
Dustin Decker 08571a0ac3
Add identity num_entities gauge metric (#8816)
Signed-off-by: Dustin Decker <dustindecker@protonmail.com>
2020-04-23 19:29:42 -05:00
Brian Kassouf 21cdba6fb5
identity: Add batch entity deletion endpoint (#8785)
* identity: Add batch entity deletion endpoint

* Update the parameter description

* Update error message

* Update helper/storagepacker/storagepacker.go

Co-Authored-By: Vishal Nayak <vishalnayak@users.noreply.github.com>

* Review feedback

* Update vault/identity_store_entities.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
2020-04-23 15:25:13 -07:00
Brian Kassouf 33b3e6857e
identity: group refresh shouldn't lock unless an update is needed (#8795) 2020-04-23 11:31:22 -07:00
Calvin Leung Huang 2659c34910
raft: check for nil on concrete type in SetupCluster (#8784)
* raft: check for nil on concrete type in SetupCluster

* raft: move check to its own func

* raft: func cleanup

* raft: disallow disable_clustering = true when raft storage is used

* docs: update disable_clustering to mention new behavior
2020-04-21 13:45:07 -07:00
Calvin Leung Huang df23b481a6
core: change rawConfig to be atomic.Value (#8755)
This avoids SetConfig from having to grab a write lock which is called on a SIGHUP, and may block, along with a long-running requests that has a read lock held, any other operation that requires a state lock.
2020-04-16 16:34:46 -07:00
Vishal Nayak b93bd39b97
Support unwrapping tokens that does not contain data (#8714)
* Support unwrapping tokens that does not contain data

* s/token/secret
2020-04-09 14:09:23 -04:00
Calvin Leung Huang c3caac76f1
token/renewal: return full set of token and identity policies in the … (#8535)
* token/renewal: return full set of token and identity policies in the policies field

* extend tests to cover additional token and identity policies on a token

* verify identity_policies returned on login and renewals
2020-03-31 13:36:45 -07:00
ncabatoff 236eb7e19f
Fix a test failure I observed on ent re cluster listener (#8647)
Panics when the cluster listener changes while we're setting up request forwarding.
2020-03-31 13:47:39 -04:00
Brian Kassouf 6b8b02f77a
identity: Fix potential deadlock on error to load groups (#8613) 2020-03-25 09:42:23 -07:00
ncabatoff c6518cc3f0
Make sure if a user gets removed from all groups in the external system, Vault updates itself accordingly. This is CVE-2020-10660. (#8606) 2020-03-23 18:00:26 -04:00
Brian Kassouf 05eea911bd
storage/raft: Refresh TLS keyring on snapshot restore (#8546) 2020-03-13 13:39:14 -07:00
ncabatoff 5fe1ab766b
Add option to detect deadlocks in Core.stateLock using build tag deadlock (#8524) 2020-03-10 16:01:20 -04:00
Jim Kalafut 90e8d9267d
Fix identity token caching (#8412)
The namespace-partitioned cache flushing was not being used correctly,
which could leave standby nodes with stale information.

Fixes #8284
2020-02-26 15:56:19 -05:00
ncabatoff e5721310ac
Add persistent feature flags to be used on enterprise non-primaries. (#8391) 2020-02-19 18:06:53 -05:00
Jeff Mitchell 844b2c3a5d Bump API/SDK and adapt to move from SDK stuff 2020-02-15 14:58:05 -05:00
Vitaly Velikodny 3ae6568512
Add missed description field for GET /sys/auth/:path/tune endpoint (#8193)
* fix #7623: add missed description field for GET /sys/auth/:path/tune endpoint

* fix #7623: allow empty description

* fix #7623: update tests with description field
2020-02-15 10:32:47 -08:00
ncabatoff 9dd18d8487
Unless we've been asked to stop, most failures should not result in (#7732)
waitForLeadership returning.
2020-02-14 18:28:37 -08:00
Joe Dollard 8f74b4d2b6
provide vault server flag to exit on core shutdown (#7561)
* provide vault server flag to exit on core shutdown

* Update command/server.go

Co-Authored-By: Jeff Mitchell <jeffrey.mitchell@gmail.com>

Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-02-14 18:07:31 -08:00
Brian Kassouf 3bbd9dc34c
Port some changes back to OSS (#8359) 2020-02-14 16:39:13 -08:00
Security Sauce d74463bbcc
Identity: Allow specifying a custom OIDC client_id field (#8165) 2020-02-13 23:15:35 -08:00
Vishal Nayak c87d34d1a4
Seal migration with Raft (#8103)
* Seal migration after unsealing

* Refactor migration fields migrationInformation in core

* Perform seal migration as part of postUnseal

* Remove the sleep logic

* Use proper seal in the unseal function

* Fix migration from Auto to Shamir

* Fix the recovery config missing issue

* Address the non-ha migration case

* Fix the multi cluster case

* Avoid re-running seal migration

* Run the post migration code in new leaders

* Fix the issue of wrong recovery being set

* Address review feedback

* Add more complete testing coverage for seal migrations.   (#8247)

* Add more complete testing coverage for seal migrations.  Also remove VAULT_ACC gate from some tests that just depend on docker, cleanup dangling recovery config in storage after migration, and fix a call in adjustCoreForSealMigration that seems broken.

* Fix the issue of wrong recovery key being set

* Adapt tests to work with multiple cores.

* Add missing line to disable raft join.

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* Fix all known issues

* Remove warning

* Review feedback.

* Revert my previous change that broke raft tests.  We'll need to come back and at least comment
this once we better understand why it's needed.

* Don't allow migration between same types for now

* Disable auto to auto tests for now since it uses migration between same types which is not allowed

* Update vault/core.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Add migration logs

* Address review comments

* Add the recovery config check back

* Skip a few steps if migration is already done

* Return from waitForLeadership if migration fails

Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-02-13 16:27:31 -05:00
Calvin Leung Huang dac3382e15
external_tests: ensure derived cores are stable before proceeding on tests (#8342)
* external_tests: ensure derived cores are stable before proceeding on tests

* testhelpers: add min duration tolerance when checking stability on derived core
2020-02-13 13:18:53 -08:00
Jeff Malnick 27ad920399
Add redshift database plugin (#8299)
* feat: add redshift database plugin

* build: update vendored libraries

* docs: add reference doc for redshift variant of the database secrets engine

* feat: set middlewear type name for better metrics naming (#8346)

Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
2020-02-13 09:42:30 -08:00
Jeff Mitchell 86327b8010 Bump api/sdk and fix imports 2020-02-13 10:41:16 -05:00
Jim Kalafut 2ee7b76469
Bundle MongoDB Atlas (#8309) 2020-02-07 14:09:39 -08:00
ncabatoff a0ac4bdd2a
Audit generate root requests and responses. (#8301) 2020-02-06 11:56:37 -05:00
ncabatoff 2c8b012d14
Changes needed so that benchmark-vault can run with Prometheus monitoring (#8295) 2020-02-05 13:45:16 -05:00
Sebastien Williams-Wynn 9f99ff4912
Fix minor typo in doc string (#8277) 2020-02-02 20:12:59 +01:00
ncabatoff 30d262f149
Fix flaky test of api renewer by moving away from legacy api. (#8265) 2020-01-30 15:12:21 -05:00
Jim Kalafut f17fc4e5c1
Run goimports (#8251) 2020-01-27 21:11:00 -08:00
Becca Petrin 759f9b38f7
Observer pattern for service registration interface (#8123)
* use observer pattern for service discovery

* update perf standby method

* fix test

* revert usersTags to being called serviceTags

* use previous consul code

* vault isnt a performance standby before starting

* log err

* changes from feedback

* add Run method to interface

* changes from feedback

* fix core test

* update example
2020-01-24 09:42:03 -08:00
Vishal Nayak fb93b0bf9c Use Shamir as KeK when migrating from auto-seal to shamir (#8172)
* Use Shamir as KeK when migrating from auto-seal to shamir

* Use the correct number of shares/threshold for the migrated seal.

* Fix log message

* Add WaitForActiveNode to test

* Make test fail

* Minor updates

* Test with more shares and a threshold

* Add seal/unseal step to the test

* Update the logic that prepares seal migration (#8187)

* Update the logic that preps seal migration

* Add test and update recovery logic

Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-01-21 09:24:33 -08:00
Brian Kassouf 1e37eadfa0
Fix panic when listener fails to startup (#8174) 2020-01-17 09:07:53 -08:00
Brian Kassouf f32a86ee7a
Create network layer abstraction to allow in-memory cluster traffic (#8173) 2020-01-16 23:03:02 -08:00
Calvin Leung Huang 67c0773df9
ldap, okta: fix renewal when login policies are empty (#8072)
* ldap, okta: fix renewal when login policies are empty

* test/policy: add test for login renewal without configured policy

* test/policy: remove external dependency on tests, refactor lease duration check
2020-01-16 09:42:35 -08:00
Vishal Nayak 8891f2ba88 Raft retry join (#7856)
* Raft retry join

* update

* Make retry join work with shamir seal

* Return upon context completion

* Update vault/raft.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Address some review comments

* send leader information slice as a parameter

* Make retry join work properly with Shamir case. This commit has a blocking issue

* Fix join goroutine exiting before the job is done

* Polishing changes

* Don't return after a successful join during unseal

* Added config parsing test

* Add test and fix bugs

* minor changes

* Address review comments

* Fix build error

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-01-13 17:02:16 -08:00
Jeff Mitchell a0694943cc
Migrate built in auto seal to go-kms-wrapping (#8118) 2020-01-10 20:39:52 -05:00
Brian Kassouf 549faf47f2
Add identity templating helper to sdk/framework (#8088)
* Add identity templating helper to sdk/framework

* Cleanup a bit

* Fix length issue when groups/aliases are filtered due to ns

* review feedback
2020-01-06 10:16:52 -08:00
Jim Kalafut 5821fe48c7 Fix identity token panic during invalidation (#8015)
* Fix identity token crash during invalidation

* Check for nil namespace

* Fix test

* Add nil check test

* Check OIDC cache errors
2019-12-17 10:43:38 -08:00
ncabatoff d3ff2684bb
Fix panic when creating batch tokens for role that doesn't exist. (#8021) 2019-12-16 09:31:32 -05:00
Calvin Leung Huang 7727c8b913
plugin: fix panic on router.MatchingSystemView if backend is nil (#7991)
* plugin: fix panic on router.MatchingSystemView if backend is nil

* correctly determine the plugin binary file in the directory

* docs: simplify plugin file removal
2019-12-10 10:48:30 -08:00
Mike Jarmy e42bc0ffc0
Introduce optional service_registration stanza (#7887)
* move ServiceDiscovery into methods

* add ServiceDiscoveryFactory

* add serviceDiscovery field to vault.Core

* refactor ConsulServiceDiscovery into separate struct

* cleanup

* revert accidental change to go.mod

* cleanup

* get rid of un-needed struct tags in vault.CoreConfig

* add service_discovery parser

* add ServiceDiscovery to config

* cleanup

* cleanup

* add test for ConfigServiceDiscovery to Core

* unit testing for config service_discovery stanza

* cleanup

* get rid of un-needed redirect_addr stuff in service_discovery stanza

* improve test suite

* cleanup

* clean up test a bit

* create docs for service_discovery

* check if service_discovery is configured, but storage does not support HA

* tinker with test

* tinker with test

* tweak docs

* move ServiceDiscovery into its own package

* tweak a variable name

* fix comment

* rename service_discovery to service_registration

* tweak service_registration config

* Revert "tweak service_registration config"

This reverts commit 5509920a8ab4c5a216468f262fc07c98121dce35.

* simplify naming

* refactor into ./serviceregistration/consul
2019-12-06 09:46:39 -05:00
Steve Gore f991b92dc3 Fix "does not exists" grammar (#7950)
* Fix "does not exists" grammar

* Fix "does not exists" grammar

* Revert vendor and go.mod
2019-12-03 16:25:09 -06:00
Lars Lehtonen 019fbe4b9c vault: fix dropped error in TestTokenStore_HandleRequest_CreateToken_NotAllowedEntityAlias(). (#7934) 2019-12-02 10:03:24 -08:00
Brian Kassouf 5cb6600e83
Remove unnecessary locking during heartbeating (#7877) 2019-11-20 17:24:49 -08:00
Calvin Leung Huang afa9510835 tests: eliminate flakiness on TestRecovery (#7897) 2019-11-20 20:21:34 -05:00
Jeff Mitchell ec17f373e4 Fix test for Go 1.13 2019-11-18 23:10:37 -05:00
Jeff Mitchell 9b5392bc8f
Fix cluster cipher test (#7900)
Go 1.13 flipped TLS 1.3 to opt-out instead of opt-in, and its TLS 1.3
support does not allow configuring cipher suites. Simply remove the
affected test; it's not relevant going forward and there's ample
evidence it works properly prior to Go 1.13.
2019-11-18 23:04:49 -05:00
Brian Kassouf 5e8901b5db Fix token counters test (#7867) 2019-11-12 13:33:28 -05:00
Michel Vocks 83d38480fc Backport OSS changes from enterprise (#7864) 2019-11-12 09:17:37 -08:00
Brian Kassouf a003abd6e2
Update recovery_test.go 2019-11-11 16:29:04 -08:00
Calvin Leung Huang bbaa7f8ea9
core: revoke the proper token on partial failures from token-related requests (#7835)
* core: revoke the proper token on partial failures from token-related requests

* move test to vault package, move test trigger to expiration manager

* update logging messages for clarity

* docstring fix
2019-11-08 13:14:03 -08:00
Vishal Nayak 9d3d53b4f4
Ensure only primary stores the case sensitivity state (#7820)
* Ensure only primary stores the case sensitivity state

* Update the check
2019-11-08 12:13:53 -05:00
Jeff Mitchell f2f984557e
Add ability to renew by accessor (#7817)
* Add renewing by accessor

* Add accessor renewing test and fix bug

* Update website docs

* Remove extra newline

* Add command-level test
2019-11-08 11:32:01 -05:00
Jeff Mitchell a8aa5f0dd3 go-ldap update; bump dep, api, secrets-ad plugin, and vendor 2019-11-08 11:24:46 -05:00
Calvin Leung Huang dac03d44e6
port namespace lease revocation fix (#7836) 2019-11-07 14:10:47 -08:00
Jim Kalafut 59e526614d
Run go fmt (#7823) 2019-11-07 08:54:34 -08:00
Brian Kassouf be883161be
Port OSS changes from perf standby fix (#7818)
* Port OSS changes from perf standby fix

* Fix build
2019-11-06 14:36:47 -08:00
Jeff Mitchell 44e899afd1
Don't allow registering a non-root zero TTL token lease (#7524)
* Don't allow registering a non-root zero TTL token lease

This is defense-in-depth in that such a token was not allowed to be
used; however it's also a bug fix in that this would then cause no lease
to be generated but the token entry to be written, meaning the token
entry would stick around until it was attempted to be used or tidied (in
both cases the internal lookup would see that this was invalid and do a
revoke on the spot).

* Fix tests

* tidy
2019-11-05 16:11:13 -05:00
Lars Lehtonen bda6a6da78 core/policy & core/token: Remove Dead Test Code (#7774)
* vault: remove dead test helper function testMakeBatchTokenViaCore()

* vault: remove dead test helper function testMakeBatchTokenViaBackend()

* vault: remove dead test helper function mockPolicyStoreNoCache()

* vault: remove dead test helper function mockPolicyStore()

* vault: remove unused test imports
2019-11-04 10:36:07 +01:00
Calvin Leung Huang 8c31e45860
hostutil: query stats with context, update gopsutil, refactor tests (#7769)
* hostutil: query stats with context, update gopsutil, refactor tests

* go mod vendor

* minor comment wording
2019-11-01 10:12:22 -07:00
Mike Jarmy d3bb5d60b2
Fix the token counter test so it doesn't miscount the number of tokens. (#7760) 2019-10-30 08:03:03 -04:00
Michel Vocks e5a921d277
Harden mount/auth filter evaluation error handling (#7754) 2019-10-29 17:42:13 +01:00
Jeff Mitchell 4e1470f483
Handpick cluster cipher suites when they're not user-set (#7487)
* Handpick cluster cipher suites when they're not user-set

There is an undocumented way for users to choose cluster cipher suites
but for the most part this is to paper over the fact that there are
undesirable suites in TLS 1.2.

If not explicitly set, have the set of cipher suites for the cluster
port come from a hand-picked list; either the allowed TLS 1.3 set (for
forwards compatibility) or the three identical ones for TLS 1.2.

The 1.2 suites have been supported in Go until at least as far back as
Go 1.9 from two years ago. As a result in cases where no specific suites
have been chosen this _ought_ to have no compatibility issues.

Also includes a useful test script.
2019-10-28 12:51:45 -04:00
Brian Kassouf d05b401cd8
Update token_store.go 2019-10-28 09:31:58 -07:00
Jack Kleeman 65c67dd6f3 Add a counter for root token creation (#7172)
It would be useful to be able to page on root token creation. This PR
adds a counter which increments on this event.
2019-10-28 09:30:11 -07:00
Jeff Mitchell 0c88218dd4 Port some changes that got out of sync 2019-10-28 11:38:14 -04:00
Brian Kassouf a20e73c2da
Port filtered paths changes back to OSS (#7741)
* Port filtered paths changes back to OSS

* Fix build
2019-10-27 13:30:38 -07:00
Matt Morrison 1e7acd0800 path-help missing or incorrect for raft paths (#7326) 2019-10-25 12:37:48 -04:00
Mike Jarmy 56725e694f
fix token counter test so the token won't time out (#7737) 2019-10-25 10:55:38 -04:00
ncabatoff 7c6cc95a24
Fix a regression introduced in #7698 that breaks root token generation. (#7727) 2019-10-24 10:23:31 -04:00
Vishal Nayak 23b0fb62de Abstract generate-root authentication into the strategy interface (#7698)
* Abstract generate-root authentication into the strategy interface

* Generate root strategy ncabatoff (#7700)

* Adapt to new shamir-as-kek reality.

* Don't try to verify the master key when we might still be sealed (in
recovery mode).  Instead, verify it in the authenticate methods.
2019-10-23 09:52:28 -07:00
ncabatoff 13c00dfa38
Use docker instead of an external LDAP server that sometimes goes down (#7522) 2019-10-22 13:37:41 -04:00
ncabatoff 68656a86cb
Fix a nil map pointer in mergeEntity. (#7711) 2019-10-22 09:57:24 -04:00
ncabatoff 8543da27bd
TestSysRekey_Verification would fail sometimes when recovery=true (#7710)
because when unsealing it wouldn't wait for core 0 to come up and become
the active node. Much of our testing code assumes that core0 is the
active node.
2019-10-22 09:35:48 -04:00
ncabatoff 1c98152fa0
Shamir seals now come in two varieties: legacy and new-style. (#7694)
Shamir seals now come in two varieties: legacy and new-style. Legacy
Shamir is automatically converted to new-style when a rekey operation
is performed. All new Vault initializations using Shamir are new-style.

New-style Shamir writes an encrypted master key to storage, just like
AutoUnseal. The stored master key is encrypted using the shared key that
is split via Shamir's algorithm. Thus when unsealing, we take the key
fragments given, combine them into a Key-Encryption-Key, and use that
to decrypt the master key on disk. Then the master key is used to read
the keyring that decrypts the barrier.
2019-10-18 14:46:00 -04:00
Lexman c86fe212c0
oss changes for entropy augmentation feature (#7670)
* oss changes for entropy augmentation feature

* fix oss command/server/config tests

* update go.sum

* fix logical_system and http/ tests

* adds vendored files

* removes unused variable
2019-10-17 10:33:00 -07:00
Michael Gaffney 4f7bd872ed
core: Log if an error is returned from postSealMigration (#7675) 2019-10-16 14:00:00 -04:00
Michael Gaffney 24f663403e
core: add postSealMigration method (#7579)
* core: add postSealMigration method

The postSealMigration method is called at the end of the postUnseal
method if a seal migration has occurred. This starts a seal rewrap
process in the enterprise version of. It is a no-op in the OSS version.
2019-10-16 12:52:37 -04:00
Michael Gaffney 6ae92ede11
core: add hook for initializing seals for migration (#7666)
* core: add hook for initializing seals for migration

Needed in enterprise version.
2019-10-15 15:48:23 -04:00
Vishal Nayak 0d077d7945
Recovery Mode (#7559)
* Initial work

* rework

* s/dr/recovery

* Add sys/raw support to recovery mode (#7577)

* Factor the raw paths out so they can be run with a SystemBackend.

# Conflicts:
#	vault/logical_system.go

* Add handleLogicalRecovery which is like handleLogical but is only
sufficient for use with the sys-raw endpoint in recovery mode.  No
authentication is done yet.

* Integrate with recovery-mode.  We now handle unauthenticated sys/raw
requests, albeit on path v1/raw instead v1/sys/raw.

* Use sys/raw instead raw during recovery.

* Don't bother persisting the recovery token.  Authenticate sys/raw
requests with it.

* RecoveryMode: Support generate-root for autounseals (#7591)

* Recovery: Abstract config creation and log settings

* Recovery mode integration test. (#7600)

* Recovery: Touch up (#7607)

* Recovery: Touch up

* revert the raw backend creation changes

* Added recovery operation token prefix

* Move RawBackend to its own file

* Update API path and hit it using CLI flag on generate-root

* Fix a panic triggered when handling a request that yields a nil response. (#7618)

* Improve integ test to actually make changes while in recovery mode and
verify they're still there after coming back in regular mode.

* Refuse to allow a second recovery token to be generated.

* Resize raft cluster to size 1 and start as leader (#7626)

* RecoveryMode: Setup raft cluster post unseal (#7635)

* Setup raft cluster post unseal in recovery mode

* Remove marking as unsealed as its not needed

* Address review comments

* Accept only one seal config in recovery mode as there is no scope for migration
2019-10-15 00:55:31 -04:00
Matthew Irish 87d4e6e068
UI - raft config and snapshotting (#7410)
* add storage route

* template out the routes and new raft storage overview

* fetch raft config and add new server model

* pngcrush the favicon

* add view components and binary-file component

* add form-save-buttons component

* adjust rawRequest so that it can send a request body and returns the response on errors

* hook up restore

* rename binary-file to file-to-array-buffer

* add ember-service-worker

* use forked version of ember-service-worker for now

* scope the service worker to a single endpoint

* show both download buttons for now

* add service worker download with a fallback to JS in-mem download

* add remove peer functionality

* lint go file

* add storage-type to the cluster and node models

* update edit for to take a cancel action

* separate out a css table styles to be used by http-requests-table and on the raft-overview component

* add raft-join adapter, model, component and use on the init page

* fix styling and gate the menu item on the cluster using raft storage

* style tweaks to the raft-join component

* fix linting

* add form-save-buttons component to storybook

* add cancel functionality for backup uploads, and add a success message for successful uploads

* add component tests

* add filesize.js

* add filesize and modified date to file-to-array-buffer

* fix linting

* fix server section showing in the cluster nav

* don't use babel transforms in service worker lib because we don't want 2 copies of babel polyfill

* add file-to-array-buffer to storybook

* add comments and use removeObjectURL to raft-storage-overview

* update alert-banner markdown

* messaging change for upload alert banner

* Update ui/app/templates/components/raft-storage-restore.hbs

Co-Authored-By: Joshua Ogle <joshua@joshuaogle.com>

* more comments

* actually render the label if passed and update stories with knobs
2019-10-14 13:23:29 -05:00
Brian Kassouf 1167fad704
Improve raft write performance by utilizing FSM Batching (#7527)
* Start benchmark work

* Add batching FSM function

* dedupe some code

* Update dependency on chunking FSM

* fix raft external tests

* fix go.mod

* Add batching test

* uncomment test

* update raft deps

* update vendor

* Update physical/raft/fsm.go

Co-Authored-By: Michel Vocks <michelvocks@gmail.com>

* Update physical/raft/fsm.go
2019-10-14 09:25:07 -06:00
Brian Kassouf 024c29c36a
OSS portions of raft non-voters (#7634)
* OSS portions of raft non-voters

* add file

* Update vault/raft.go

Co-Authored-By: Vishal Nayak <vishalnayak@users.noreply.github.com>
2019-10-11 11:56:59 -07:00
isbric e6e20e9eb3 Correct spelling of error message (#7630) 2019-10-11 11:14:41 -04:00
ncabatoff c16e3bbceb
Cache whether we've been initialized to reduce load on storage (#7549) 2019-10-08 17:52:38 -04:00