Commit graph

2249 commits

Author SHA1 Message Date
swayne275 6664846a65
Vault-2713: Retry Irrevocable Lease Revocation Periodically (#11977)
* prototype function to retry irrevocable lease revocation, clean up comment

* setup irrevocable lease revoke retry on expiration manager creation

* reduce expiration lag to 1 hour, use core context and timeout on retry attempts

* add lease namespace to revocation call

* start irrevocable revocation attempts in setupExpiration

* grab reference to channel to avoid race test failures
2021-07-06 15:05:04 -06:00
Pratyoy Mukhopadhyay c7e6a07939
Augment lease revocation metrics (#11954)
* [VAULT-1982] Add metric for irrevocable leases

* [VAULT-1982] Add metric for tracking worker count per queue

* [VAULT-1982] Update external docs

* [VAULT-1982] Add nil check on metrics sink

* [VAULT-1982] Update metric name, fix access patterns

* [VAULT-1982] Add locking when marking leases as irrevocable

* [VAULT-1982] Remove workers per mount metric

* [vault-1982] Misc review fixes

* [VAULT-1982] minor refactor

* [VAULT-1982] Add/update some comments
2021-06-30 14:37:36 -07:00
swayne275 fd517e6114
remove trailing / on leases API endpoint (#11959) 2021-06-30 13:02:26 -06:00
swayne275 f882c31856
update oss files with changes from ent PR #1928 (#11965) 2021-06-29 15:02:30 -06:00
Lars Lehtonen 27679e4fda
fix dropped errors in external_tests (#11947)
* vault/external_tests/token: fix dropped test error

* vault/external_tests/identity: fix dropped test errors
2021-06-29 09:04:26 -04:00
swayne275 67bc4fdabd
Require special privileges to list irrevocable leases (#11888)
* add leases path to sudo required set

* update TestSystemBackend_RootPaths with new special privilege paths

* note that list-leases requires sudo

* minor typo fixes
2021-06-28 16:51:47 -06:00
Hridoy Roy f3895f7cce
oss part of license diagnose (#11939) 2021-06-25 14:18:34 -07:00
hghaf099 6d7dbc85b5
diagnose: Add seal transit tls check (#11922)
* diagnose: Add seal transit tls check

* Fixing the path to the config file and the path to the cert files

* Addressing comment

* Addressing seal transit tls check comments
2021-06-24 18:30:42 -04:00
Hridoy Roy a8cc5e4557
TLS Verification Bugfixes (#11910)
* tls verification bugfix

* tls verification bugfix

* allow diagnose fail to report status when there are also warnings

* allow diagnose fail to report status when there are also warnings

* Update vault/diagnose/helpers_test.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* comments

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-24 10:43:49 -07:00
hghaf099 658a4ea276
Checking Validity of all Certs in the chain [VAULT-2114] (#11883)
* Checking Validity of all Certs in the chain

* Addressing Comments for TLS cert validation

* Fixing tls_verification tests

* Fixing minor issue in tls_verification tests

* Addressing Comments, Rebasing with main

* Adding comment on top of a test
2021-06-22 16:23:55 -04:00
Hridoy Roy bbfd450319
remove prints (#11918) 2021-06-22 09:40:53 -07:00
hghaf099 38b753046b
Tls check client certs [VAULT-2117] (#11860)
* add expiration warning to certificate checking for diagnose

* Adding TLS Client CA checks to diagnose
Adding checks for tls_disable_client_certs and tls_require_and_verify_client_cert flags

* Check validity of TLSClientCAFile
Adding related tests

* Addressing comments

* Fixing some tls tests

* Addressing comments

Co-authored-by: HridoyRoy <roy@hashicorp.com>
2021-06-18 19:35:38 -04:00
Scott Miller ee0d6603f3
Wire configuration checks into diagnose and fix resulting bugs. (#11854)
* Actually call config.Validate in diagnose

* Wire configuration checks into diagnose and fix resulting bugs.

* go mod vendor

* Merge to vendorless version

* Remove sentinel section to allow diagnose_ok to pass

* Fix unit tests
2021-06-17 13:09:37 -05:00
Hridoy Roy e38f991054
Diagnose checks for raft quorum status and file backend permissions (#11771)
* raft file and quorum checks

* raft checks

* backup

* raft file checks test

* address comments and add more raft and file and process checks

* syntax issues

* modularize functions to compile differently on different os

* compile raft checks everywhere

* more build tag issues

* raft-diagnose

* correct file permission checks

* upgrade tests and add a getConfigOffline test that currently does not work

* comment

* update file checks method signature on windows

* Update physical/raft/raft_test.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* raft tests

* add todo comment for windows root ownership

* voter count message

* raft checks test fixes

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-06-17 10:04:21 -07:00
Luke Jolly 0779d260ec
Fix error log typo for failing to decrypt keys (#11873) 2021-06-17 10:12:13 -04:00
Lars Lehtonen f390cad6e4
vault: fix dropped identity store test errors (#11867) 2021-06-16 11:32:20 -04:00
Hridoy Roy f46764abbf
Diagnose TLS Cert Expiry Warning Check: change 1 week expiry to 1 month expiry (#11871)
* change 1 week expiry to 1 month expiry

* report time left to expiry

* report time left to expiry
2021-06-15 12:31:28 -07:00
Hridoy Roy e58c39033b
Add expiration warning to certificate checking for diagnose [VAULT-1018] (#11850)
* add expiration warning to certificate checking for diagnose

* Update serviceregistration/consul/consul_service_registration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* review comments

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-15 09:53:29 -07:00
Nick Cabatoff 2ce2617a8a
Reorganize request handling code so that we don't touch storage until we have the stateLock. (#11835) 2021-06-11 13:18:16 -04:00
Lars Lehtonen 5c8b3db0e5
vault: token store dropped test errors (#11799) 2021-06-11 11:41:31 -04:00
Brian Kassouf b42529dd17
Omit wrapping tokens and control groups from client counts (#11826)
* Omit wrapping tokens and control groups from client counts

* add changelog note
2021-06-10 15:57:51 -07:00
Hridoy Roy 65e3489c45
Diagnose resource creation checks (#11627)
* initial refactoring of unseal step in run

* remove waitgroup

* remove waitgroup

* backup work

* backup

* backup

* completely modularize run and move into diagnose

* add diagnose errors for incorrect number of unseal keys

* comment tests back in

* backup

* first subspan

* finished subspanning but running into error with timeouts

* remove runtime checks

* merge main branch

* meeting updates

* remove telemetry block

* roy comment

* subspans for seal finalization and wrapping diagnose latency checks

* backup while I fix something else

* fix storage latency test errors

* runtime checks

* diagnose with timeout on seal
2021-06-10 12:29:32 -07:00
swayne275 7c5c9c830e
fix irrevocable leases API deadlock on m.coreStateLock (#11815)
* fix m.coreStateLock deadlock for API handler code

* improve lock comment
2021-06-10 11:01:14 -06:00
Mark Gritter 3e8a4bad1a
Acquire a per-lock lease to make renew and revoke atomic wrt each other. (#11122)
* Acquire a per-lock lease to make renew and revoke atomic wrt each other.
This means we don't have to hold pendingLock during I/O.

* Attempted fix for deadlock in token revocation.

* Comment fix.

* Fix error checking in loadEntry.

* Add benchmark

* Add a few additional locking locations

* Improve benchmark slightly

* Update vault/expiration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Update vault/expiration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Add a lease lock into tidy

Co-authored-by: Scott Miller <smiller@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-09 17:28:41 -07:00
Nick Cabatoff c07711b488
Add some paths that use nonstandard auth. (#11804) 2021-06-09 17:04:06 -04:00
Nick Cabatoff abdf54c77e
Use the perfStandby field instead of the method because we don't want to try to grab the readlock when we're servicing a request (#11793) 2021-06-08 11:55:38 -04:00
Scott Miller b795643146
Third time trying to get the os_openbsd_arm.go right. (#11786)
* Fix the third dumb thing in this file

* ...
2021-06-07 17:19:47 -05:00
Scott Miller 69d0242db9
Add Advice and Advise functions for adding an advice section to any span. (#11760)
* wip

* wip

* Finish implementing advice handling and word wrapping

* Properly word wrap messages and warnings

* Remove debugging

* Remove debugging

* Remove unnecessary test

* unit test bug

* go vendor
2021-06-07 11:29:36 -05:00
Scott Miller c896c3a6aa
Fix SpotSkipped on openbsd/arm (#11781) 2021-06-07 11:29:15 -05:00
Hridoy Roy 1782b4e880
oss part of control groups upgrade (#11772)
* oss part of control groups upgrade

* changelog and docs

* formatting

* formatting
2021-06-07 09:15:35 -07:00
swayne275 62a92a77d7
bug fixes and improvements from ent namespace work (#11774) 2021-06-04 13:25:41 -06:00
Josh Black c8cfcd9514
OSS parts of sighup license reload (#11767) 2021-06-04 10:24:35 -07:00
Scott Miller fe8f0476fd
Fix build boolean logic causing os_common to be skipped wrongly (#11766) 2021-06-03 17:29:34 -05:00
Josh Black c6c0424a8e
OSS parts of sys/config/reload/license (#11695) 2021-06-03 10:30:30 -07:00
Scott Miller 9db384e0f7
Edge cases in the skipping gathering and output logic (#11752)
* Edge cases in the skipping gathering and output logic

* Fix unit test
2021-06-03 11:01:14 -05:00
Scott Miller d151b0b55b
Segment out disk checks to disable on openbsd/arm (#11749)
* Segment out disk checks to disable on openbsd/arm

Also add a spot skipped helper.

* Expected results may be fewer than actual because of variable length tests like disk usage

* Move to os_common and build on windows
2021-06-02 12:17:52 -05:00
swayne275 35aad1df4a
Vault 2256: fix lease count quotas causing panics on dr secondaries (#11742)
* lift relevant changes from ent to oss

* fix silent error bug in quotas
2021-06-02 10:12:05 -06:00
swayne275 9724f59180
Vault 1979: Query API for Irrevocable Leases (#11607)
* build out lease count (not fully working), start lease list

* build out irrevocable lease list

* bookkeeping

* test irrevocable lease counts for API/CLI

* fix listIrrevocableLeases, test listIrrevocableLeases, cleanup

* test expiration API limit

* namespace tweaks, test force flag on lease list

* integration test leases/count API, plenty of fixes and improvements

* test lease list API, fixes and improvements

* test force flag for irrevocable lease list API

* i guess this wasn't saved on the last refactor...

* fixes and improvements found during my review

* better test error msg

* Update vault/logical_system_paths.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* return warning with data if more than default leases to list without force flag

* make api doc more generalized

* list leases in general, not by mount point

* change force flag to include_large_results

* sort leases by LeaseID for consistent API response

* switch from bool flag for API limit to string value

* sort first by leaseID, then stable sort by expiration

* move some utils to be in oss and ent

* improve sort efficiency for API response

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-06-02 10:11:30 -06:00
Scott Miller 59bea5f6fd
Add ulimit check, and tidy unit test cases (#11678)
* Add ulimit check, and tidy unit test cases to avoid needing to have all results and perfect ordering

* Make order independent check recursive

* Fix unit tests

* Try a 5s request timeout
2021-06-01 12:43:51 -05:00
Cory Bolar 6c726bde67
Only fail auth plugin backend mounting for custom plugins (#11690)
Checking if a plugin is a builtin by comparing it's type to those in the builtin
registry allows for a custom plugin loaded with the same name to be considered a
builtin during error handling of the mounting process.  This can cause the vault
installation to brick itself because it cannot be unsealed to register a new
sha256 or file path for a previously loaded custom plugin.  Improve this logic
by checking the plugin catalog rather than the builtin registry.

Fixes #11687
2021-05-27 13:37:45 -05:00
Scott Miller 46da8ebae6
Diagnose Disk usage checks (#11673)
* Disk usage checks

* Move disk free earlier

* Move logic to helpers

* Bring over test logic from the ulimit PR

* imports

* Report error

* Get unit tests working
2021-05-27 10:17:52 -07:00
Hridoy Roy b6e1d62d33
Modularize Run Command (#11573)
* initial refactoring of unseal step in run

* remove waitgroup

* remove waitgroup

* backup work

* backup

* backup

* completely modularize run and move into diagnose

* add diagnose errors for incorrect number of unseal keys

* comment tests back in

* backup

* first subspan

* finished subspanning but running into error with timeouts

* remove runtime checks

* meeting updates

* remove telemetry block

* roy comment

* subspans for seal finalization and wrapping diagnose latency checks

* fix storage latency test errors

* review comments

* use random uuid for latency checks instead of static id
2021-05-25 15:23:20 -07:00
Scott Miller e305f78e9c
Add a Success helper to set successful spans' messages (#11621) 2021-05-21 19:21:23 -07:00
Scott Miller 941d01eee3
Initial Diagnose CLI output (#11583)
* Create helpers which integrate with OpenTelemetry for diagnose collection

* Go mod vendor

* Comments

* Update vault/diagnose/helpers.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Add unit test/example

* tweak output

* More comments

* add spot check concept

* Get unit tests working on Result structs

* wip

* Fix unit test

* Get unit tests working, and make diagnose sessions local rather than global

* Comments

* Last comments

* No need for init

* :|

* Fix helpers_test

* wip

* wip

* wip

* Revendor otel

* Fix merge related problems

* imports

* Fix unit tests

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-05-21 19:21:11 -07:00
Nick Cabatoff 2adef1f878
OSS parts of #1891 (sys/health license addition) (#11676) 2021-05-20 13:32:15 -04:00
Nick Cabatoff 01f96f18ce
VAULT-2439: OSS parts of #1889 (raft licensing init) (#11665) 2021-05-19 16:07:58 -04:00
Brian Kassouf eabfda0c79
Make sure we are checking the final peerset (#11664) 2021-05-19 12:01:52 -07:00
Brian Kassouf f498d0d389
Reload raft TLS keys on active startup (#11660) 2021-05-19 10:03:32 -07:00
Scott Miller 6b8d7fe2e6
Patch expiration fix over from ENT (#11650)
* Patch expiration fix over from ENT

* Rename changelog
2021-05-18 14:55:38 -07:00
Nick Cabatoff e212ec5d8e
OSS parts of ent PR #1857: license autoloading init changes. (#11623) 2021-05-17 14:10:26 -04:00