Commit graph

2538 commits

Author SHA1 Message Date
Nick Cabatoff 4230749d9e
Revert #13679 since TestCluster_ListenForRequest now fails intermittently in alarming ways. (#13714) 2022-01-20 08:34:40 -05:00
Sung Hon Wu 194c9e32d3
Enhance sys/raw to read and write values that cannot be encoded in json (#13537) 2022-01-20 07:52:53 -05:00
Scott Miller f7e3ab72a8
OSS side of the managed key interfaces (#13699)
* OSS side of the managed key interfaces

* Not needed in OSS
2022-01-19 11:15:33 -06:00
Nick Cabatoff 21be98ee7a
Support go-sockaddr templates in top-level cluster_addr config (#13678)
In doing some testing I found that the listener clusteraddr isn't really used, or at least isn't as important as the top-level clusteraddr setting.  As such, go-sockaddr templating needs to be implemented for the top-level `cluster_addr` setting or it's unusable for HA.

Also fix a nil pointer panic I discovered at the same time.
2022-01-19 10:56:04 -05:00
Nick Cabatoff d96298461f
Don't say we've shut down cluster listener before having done so (#13679) 2022-01-19 10:51:40 -05:00
Rémi Lapeyre 0d6c2acbd9
Make auth/token/revoke-accessor idempotent (#13661)
The auth/token/revoke will not error out if the token does not exists, it
always tries to revoke the token and return success to the client whether
or not the token exists. This makes the behavior of
auth/token/revoke-accessor coherent with this and remove the need to
check whether the token still exists.
2022-01-18 06:56:38 -05:00
Nick Cabatoff 400996ef0d
Parallel retry join (#13606) 2022-01-17 10:33:03 -05:00
akshya96 319a76d8d1
Vault-3991 Code Scanning Alerts Changes (#13667)
* code scanning alerts changes

* adding changelog
2022-01-14 15:35:27 -08:00
Chris Capurso d52d69e4bb
Add HTTP PATCH support for KV key metadata (#13215)
* go get vault-plugin-secrets-kv@vault-4290-patch-metadata

* add kv metadata patch command

* add changelog entry

* success tests for kv metadata patch flags

* add more kv metadata patch flags tests

* add kv metadata patch cas warning test

* add kv-v2 key metadata patch API docs

* add kv metadata patch to docs

* prevent unintentional field overwriting in kv metadata put cmd

* like create/update ops, prevent patch to paths ending in /

* fix kv metadata patch cmd in docs

* fix flag defaults for kv metadata put

* go get vault-plugin-secrets-kv@vault-4290-patch-metadata

* fix TestKvMetadataPatchCommand_Flags test

* doc fixes

* go get vault-plugin-secrets-kv@master; go mod tidy
2022-01-12 12:05:27 -05:00
Chris Capurso c925324057
add retry logic to TestKV_Patch_RootToken (#13586) 2022-01-07 14:46:26 -05:00
Scott Miller 89f617a97c
Convert to Go 1.17 go:build directive (#13579) 2022-01-05 12:02:03 -06:00
Scott Miller 4a4c9932b1
Add the most basic interfaces to OSS (#13561) 2022-01-04 14:07:18 -06:00
Steven Clark b9e5aeb459
Attempt to address a data race issue within identity store - take 2 (#13476)
* Attempt to address a data race issue within identity store
* Testcase TestIdentityStore_LocalAliasInvalidations identified a data race issue.
* This reverts the previous attempt to address the issue from #13093
2021-12-22 09:51:13 -05:00
Anthony (Ryo) Wright e0ac921b8f
Fixed null token panic from 'v1/auth/token/' endpoints and returned p… (#13233)
* Fixed null token panic from 'v1/auth/token/' endpoints and returned proper error response

* added changelog entry for PR #13233

* changed error message from 'bad token' to 'null token'

* rebased off of main

* Revert "changed error message from 'bad token' to 'null token'"

This reverts commit 381ed9b32c5ddd5e47adb1643ef7e46fb768bc76.

* changed 'bad token' error message to 'invalid token' after revert

* remove unnecessary vault-data folder
2021-12-21 09:46:56 -08:00
Scott Miller 82f333002a
One small diff between ENT and OSS managed key config (#13482)
* Changes to bring ENT in line with OSS for managed key changes

* Revert this
2021-12-20 15:20:18 -06:00
Victor Rodriguez 1136381637
Add no-op method setupManagedKeyRegistry(). (#13433) 2021-12-14 16:00:17 -05:00
Steven Clark 0bb0996072
Fix logging statement using formatting args (#13407) 2021-12-13 15:28:35 -05:00
Ben Ash 6ec3367648
Support clearing an identity alias' custom_metadata (#13395)
* Support clearing an identity alias' custom_metadata

Previously, an update to an entity alias supported updating the
custom_metadata as long as the update was not empty, which makes it
impossible to clear the metadata values completely.

Fixes:
- empty custom_metadata parameters are honoured on entity alias update
- update related tests
- drop dependency on mapstructure
- reformat with gofumpt
2021-12-10 18:07:47 -05:00
Nick Cabatoff fed74de3fe
Remove another use gopsutil/host. (#13390) 2021-12-10 09:59:52 -05:00
Nick Cabatoff f729dfc18f
Use os.Hostname instead of a dependency that doesn't work on OpenBSD. (#13389) 2021-12-10 08:57:15 -05:00
hghaf099 65845c7531
VAULT-1564 report in-flight requests (#13024)
* VAULT-1564 report in-flight requests

* adding a changelog

* Changing some variable names and fixing comments

* minor style change

* adding unauthenticated support for in-flight-req

* adding documentation for the listener.profiling stanza

* adding an atomic counter for the inflight requests
addressing comments

* addressing comments

* logging completed requests

* fixing a test

* providing log_requests_info as a config option to determine at which level requests should be logged

* removing a member and a method from the StatusHeaderResponseWriter struct

* adding api docks

* revert changes in NewHTTPResponseWriter

* Fix logging invalid log_requests_info value

* Addressing comments

* Fixing a test

* use an tomic value for logRequestsInfo, and moving the CreateClientID function to Core

* fixing go.sum

* minor refactoring

* protecting InFlightRequests from data race

* another try on fixing a data race

* another try to fix a data race

* addressing comments

* fixing couple of tests

* changing log_requests_info to log_requests_level

* minor style change

* fixing a test

* removing the lock in InFlightRequests

* use single-argument form for interface assertion

* adding doc for the new configuration paramter

* adding the new doc to the nav data file

* minor fix
2021-12-08 17:34:42 -05:00
Matt Schultz 85f5cfc356
Adds support for SHA-3 to transit (#13367)
* Adding support for SHA3 in the transit backend.

* Adds SHA-3 tests for transit sign/verify path. Adds SHA-3 tests for logical system tools path hash functionality. Updates documentation to include SHA-3 algorithms in system tools path hashing.

* Adds changelog entry.

Co-authored-by: robison jacka <robison@packetized.io>
2021-12-08 12:29:33 -06:00
Jim Kalafut 22c4ae5933
Rename master key to root key (#13324)
* See what it looks like to replace "master key" with "root key".  There are two places that would require more challenging code changes: the storage path `core/master`, and its contents (the JSON-serialized EncodedKeyringtructure.)

* Restore accidentally deleted line

* Add changelog

* Update root->recovery

* Fix test

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2021-12-06 17:12:20 -08:00
Anthony (Ryo) Wright bcd29f2b68
Fix null token type bug (#13236)
* Fixed null token panic from 'v1/auth/token/' endpoints and returned proper error response

* Fixed panic resulting from null token_type in /auth/token/roles/{role_name} to returne proper error response

* added changelog entry for PR #13236

* edit changelog entry for PR #13236
2021-12-06 09:38:53 -08:00
Eugene R c2a92cd351
Fix possible nil pointer dereference (#13318) 2021-12-02 08:23:41 -05:00
Pavlos Tzianos 0abc8f43fa
Add helper for encoding/decoding root tokens and OTP generation in SDK module (#10504) (#10505) 2021-12-01 08:05:49 -05:00
Nick Cabatoff a47a2c9fc4
Add "operator members" command to list nodes in the cluster. (#13292) 2021-11-30 14:49:58 -05:00
Hridoy Roy 1ff2e8d9d0
some small version test changes (#13310) 2021-11-30 09:52:33 -08:00
John-Michael Faircloth 32152e10fd
Identity: check NextSigningKey existence during key rotation (#13298)
* oidc: fix key rotation panic

* refactor and update unit tests

* add changelog
2021-11-29 15:10:58 -06:00
hghaf099 9640d35136
Unify HTTPResponseWriter and StatusHeaderResponseWriter (#13200)
* Unify NewHTTPResponseWriter ant NewStatusHeaderResponseWriter to fix ResponseWriter issues

* adding changelog

* removing unnecessary function from the WrappingResponseWriter interface

* changing logical requests responseWriter type

* reverting change to HTTPResponseWriter
2021-11-23 14:30:25 -05:00
akshya96 f77223bfe5
Authenticate to "login" endpoint for non-existent mount path bug (#13162)
* changing response from missing client token to permission denied

* removing todo comment

* fix tests

* adding changelog

* fixing changelog
2021-11-22 17:06:59 -08:00
Austin Gebauer d5f4fbecc1
identity/oidc: optional nonce parameter for authorize request (#13231) 2021-11-22 09:42:22 -08:00
divyapola5 5236fe93aa
Add a new parameter "allowed_managed_keys" to mount config (#13202)
* Add a new parameter "allowed_managed_keys" to mount config

* Adjust formatting in mount.go

* Add changelog entry
2021-11-21 19:08:38 -06:00
swayne275 3458c22df0
Vault-2257: don't log token error on DR Secondary (#13137)
* don't log token error on DR Secondary

* stop gauge collector expiration errors on dr secondary

* don't check dr secondary for token create

* see if CI hits panic

* Revert "don't check dr secondary for token create"

This reverts commit c036a1a544d3a20d29d046f1ee239ab1563ce4d9.

* don't check dr secondary for token create

* Revert "see if CI hits panic"

This reverts commit 1e15aa535cac6e4d1684aaf47c8746c094068eb8.

* remove condition on log
2021-11-17 09:21:54 -07:00
Nick Cabatoff 1ec904976a
Note that versionTimestamps are only loaded on the active node. 2021-11-16 15:05:59 -05:00
Nick Cabatoff c2d9215d1d
Fix startup failures when aliases from a pre-1.9 vault version exist (#13169)
* Add AllowMissing to local_bucket_key schema, preventing startup failures in post-unseal when aliases from an older version exist.
2021-11-16 14:56:34 -05:00
Scott Miller a5e55f6b05
Fix a data race in the new autoseal health check (#13136)
* Move the ctx capture outside the goroutine to avoid a race

* refactor the toggleable wrapper to avoid races

* Move the capture back outside the goroutine

* defer
2021-11-12 15:58:46 -06:00
Scott Miller 10270b6985
Add a periodic test of the autoseal to detect loss of connectivity. (#13078)
* Add a periodic test of the autoseal to detect loss of connectivity

* Keep the logic adjacent to autoseal

* imports

* typo, plus unnecessary constant time compare

* changelog

* pr feedback

* More feedback

* Add locking and a unit test

* unnecessary

* Add timeouts to encrypt/decrypt operations, capture activeContext before starting loop

* Add a block scope for the timeout

* copy/paste ftl

* Refactor to use two timeouts, and cleanup the repetitive failure code

* Readd 0ing gauge

* use millis

* Invert the unit test logic
2021-11-10 14:46:07 -06:00
John-Michael Faircloth e6ffaaf835
OIDC: return full issuer uri on read provider (#13058)
* return full issuer uri on read provider

* remove err check

* simplify full issuer logic
2021-11-10 12:35:31 -06:00
swayne275 e137045050
Vault 936: use core.activeContext in ActivityLog (#13083)
* update activity log to use core's activeContext for cleaner worker termination

* update tests to use core activeContext instead of generic context

* pass context around instead

* revert context change

* undo test context changes

* change worker context

* accidentally undid context for fcn signature changes
2021-11-09 11:47:39 -07:00
Steven Clark 8c50afc07c
Address a data race issue within identity_store_util::processLocalAlias (#13093)
- When loading an existing alias within processLocalAlias we aren't
   cloning the object from the memory store. There seems to be a data
   race within the function when calling entity.UpsertAlias and
   a concurrent invalidation routine.

 ==================
WARNING: DATA RACE
Read at 0x00c00bd03d08 by goroutine 94:
  google.golang.org/protobuf/internal/impl.pointer.Elem()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3
  google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265
  google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a
  google.golang.org/protobuf/internal/impl.(*MessageInfo).size()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95
  google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c
  google.golang.org/protobuf/proto.MarshalOptions.marshal()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3
  google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5
  github.com/golang/protobuf/proto.marshalAppend()
      /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4
  github.com/golang/protobuf/proto.Marshal()
      /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64
  github.com/hashicorp/vault/helper/identity.(*Entity).Clone()
      /go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150
  github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn()
      /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306
  github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate()
      /go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c
  github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm()
      /go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d
  github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey()
      /go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a
  github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey()
      <autogenerated>:1 +0x7d
  github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey()
      /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390
  github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler()
      /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b
  github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453()
      /go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71

Previous write at 0x00c00bd03d08 by goroutine 52:
  github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias()
      /go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271
  github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias()
      /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672
  github.com/hashicorp/vault/vault.possiblyForwardEntityCreation()
      /go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286
  github.com/hashicorp/vault/vault.(*Core).handleLoginRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a
  github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11
  github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5
  github.com/hashicorp/vault/vault.(*Core).HandleRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2
  github.com/hashicorp/vault/http.request()
      /go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1
  github.com/hashicorp/vault/http.handleLogicalInternal.func1()
      /go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.handleRequestForwarding.func1()
      /go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/go/src/net/http/server.go:2424 +0xc5
  github.com/hashicorp/vault/http.wrapHelpHandler.func1()
      /go/src/github.com/hashicorp/vault/http/help.go:23 +0x281
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.wrapCORSHandler.func1()
      /go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1()
      /go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1()
      /go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.wrapGenericHandler.func1()
      /go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1()
      /go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2878 +0x89a
  net/http.initALPNRequest.ServeHTTP()
      /usr/local/go/src/net/http/server.go:3479 +0x34d
  net/http.(*initALPNRequest).ServeHTTP()
      <autogenerated>:1 +0x8f
  net/http.Handler.ServeHTTP-fm()
      /usr/local/go/src/net/http/server.go:87 +0x75
  net/http.(*http2serverConn).runHandler()
      /usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd
  net/http.(*http2serverConn).processHeaders·dwrap·31()
      /usr/local/go/src/net/http/h2_bundle.go:5562 +0x64
2021-11-09 10:00:26 -05:00
Hridoy Roy e7e881c559
Port: Allow Routing to Partial Monthly Client Count From Namespaces (#13086)
* add function for routing activity log client counts to ent namespaces

* changelog
2021-11-08 15:38:35 -08:00
Rémi Lapeyre ae0b5e41e0
Add read support to sys/mounts/:path (#12792)
* Add read support to sys/mounts/:path

Closes https://github.com/hashicorp/vault/issues/12349

* Add changelog entry

* Empty commit to trigger CI

* Empty commit to trigger CI
2021-11-08 10:32:01 -08:00
Nick Cabatoff 40640ef43f
Fix errors logged on standbys when we try to write versions to storage (#13042) 2021-11-08 10:04:17 -05:00
Meggie 627c43e496
1.10.0-dev version bump (#12987)
* 1.10.0-dev version bump

* Remove hard-coded versions from tests. (#13026)

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2021-11-03 12:50:12 -04:00
Brian Kassouf 0e1b2c1b7b
Fix some more error shadowing issues (#12990) 2021-11-01 11:43:00 -07:00
John-Michael Faircloth 792400a24c
ensure errors are checked (#12989) 2021-11-01 13:34:51 -05:00
Austin Gebauer 83dcbc0cd5
Deny access to UserInfo endpoint if client no longer allowed by provider (#12949) 2021-10-28 09:46:07 -07:00
Austin Gebauer e5b45b5416
Adds check to ensure authorization code can only be exchanged by the issuing provider (#12948) 2021-10-28 09:45:41 -07:00
Brian Kassouf 350ecd6bc5
Fix leak of restore worker go routines after an error (#12933) 2021-10-27 09:08:10 -07:00
Austin Gebauer b3fab954fe
Use mutex in OIDC configuration handlers (#12932) 2021-10-27 08:23:05 -07:00
swayne275 92e6972b86
oss LastDRWAL (#12931) 2021-10-26 15:17:20 -06:00
Hridoy Roy 1d29b93eb5
fix adding clientID to request in audit log (#12919)
* fix adding clientID to request in audit log

* fix boolean statement

* use standard encoding for client ID instead of urlEncoding

* change encoding in tests

* add in client counts to request handling

* remove redundant client ID generation in request handling

* directly add clientID to req after handling token usage
2021-10-26 09:02:13 -07:00
Austin Gebauer a25cfe822e
Adds missing unlock of RWMutex in OIDC delete key (#12916) 2021-10-25 09:59:26 -07:00
swayne275 95e5cdd000
VAULT 2844: remove legacy lease revocation strategy (#12888)
* remove legacy lease revocation strategy

* add deprecation change log note

* remove VAULT_16_REVOKE_PERMITPOOL

* update changelog
2021-10-22 17:37:01 -06:00
hghaf099 442b5920e2
Entities may have duplicate policies (#12812)
* Entities may have duplicate plicies

* Adding changelog

* removing duplicates on reading entity policies

* fix changelog
2021-10-22 19:28:31 -04:00
Nick Cabatoff ff74f49047
Move to go 1.17 (#12868)
Also ensure that the go 1.17 breaking changes to net.ParseCIDR don't make us choke on stored CIDRs that were acceptable to older Go versions.
2021-10-21 09:32:03 -04:00
Vishal Nayak 6eead9f09b
Fix entity alias deletion (#12834)
* Fix entity alias deletion

* Fix tests

* Add CL
2021-10-19 15:05:06 -04:00
Brian Kassouf c705adc79c
Fix some linting errors (#12860) 2021-10-18 17:29:47 -07:00
Austin Gebauer 4e5b865c4f
Rename scopes to scopes_supported for OIDC providers (#12851) 2021-10-15 19:33:32 -07:00
Hridoy Roy 19822781cc
use IsRoot helper before clientID generation in activity log (#12846)
* use IsRoot helper in activity log

* use IsRoot helper in activity log
2021-10-15 14:04:55 -07:00
Vishal Nayak 476fb08e0d
Local aliases OSS patch (#12848)
* Local aliases OSS patch

* build fix
2021-10-15 15:20:00 -04:00
saltperfect 66369469d9
Removed unpublished:true for sys/internal/* endpoints (#12713)
* removed unpublished:true for sys/internal/* endpoints

* added changelog file

* updated change log and added placeholder summary as these endpoints are not mentioned in docs.

* added documentation for internal/ui/namspaces and resultant-acl

* updated log configs
2021-10-15 14:50:14 -04:00
Pratyoy Mukhopadhyay 148109b8ed
[VAULT-3252] Disallow alias creation if entity/accessor combination exists (#12747)
* Disallow alias creation if entity/accessor combination exists

* Add changelog

* Address review comments

* Add handling to aliasUpdate, some field renaming

* Update tests to work under new entity-alias constraint

* Add check to entity merge, other review fixes

* Log duplicated accessors only once

* Fix flaky test

* Add note about new constraint to docs

* Update entity merge warn log
2021-10-14 09:52:07 -07:00
Hridoy Roy 1c427d3286
Port: add client ID to TWEs in activity log [vault-3136] (#12820)
* port for tracking twes as clients

* comment clean up

* changelog

* change changelog entry phrasing
2021-10-14 09:10:59 -07:00
Austin Gebauer 0551f91068
Adds OIDC Token and UserInfo endpoints (#12711) 2021-10-13 18:59:36 -07:00
swayne275 76556c867b
port relevant ent changes (#12823) 2021-10-13 18:15:54 -06:00
Chris Capurso bbb4ab4a41
Add HTTP PATCH support to KV (#12687)
* handle HTTP PATCH requests as logical.PatchOperation

* update go.mod, go.sum

* a nil response for logical.PatchOperation should result in 404

* respond with 415 for incorrect MIME type in PATCH Content-Type header

* add abstraction to handle PatchOperation requests

* add ACLs for patch

* Adding JSON Merge support to the API client

* add HTTP PATCH tests to check high level response logic

* add permission-based 'kv patch' tests in prep to add HTTP PATCH

* adding more 'kv patch' CLI command tests

* fix TestHandler_Patch_NotFound

* Fix TestKvPatchCommand_StdinValue

* add audit log test for HTTP PATCH

* patch CLI changes

* add patch CLI tests

* change JSONMergePatch func to accept a ctx

* fix TestKVPatchCommand_RWMethodNotExists and TestKVPatchCommand_RWMethodSucceeds to specify -method flag

* go fmt

* add a test to verify patching works by default with the root token

* add changelog entry

* get vault-plugin-secrets-kv@add-patch-support

* PR feedback

* reorder some imports; go fmt

* add doc comment for HandlePatchOperation

* add json-patch@v5.5.0 to go.mod

* remove unnecessary cancelFunc for WriteBytes

* remove default for -method

* use stable version of json-patch; go mod tidy

* more PR feedback

* temp go get vault-plugin-secrets-kv@master until official release

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
2021-10-13 15:24:31 -04:00
John-Michael Faircloth 2abf916ddb
Add support to parameterize unauthenticated paths (#12668)
* store unauthenticated path wildcards in map

* working unauthenticated paths with basic unit tests

* refactor wildcard logic

* add parseUnauthenticatedPaths unit tests

* use parseUnauthenticatedPaths when reloading backend

* add more wildcard test cases

* update special paths doc; add changelog

* remove buggy prefix check; add test cases

* prevent false positives for prefix matches

If we ever encounter a mismatched segment, break and set a flag to
prevent false positives for prefix matches.

If it is a match we need to do a prefix check. But we should not return
unless HasPrefix also evaluates to true. Otherwise we should let the for
loop continue to check other possibilities and only return false once
all wildcard paths have been evaluated.

* refactor switch and add more test cases

* remove comment leftover from debug session

* add more wildcard path validation and test cases

* update changelong; feature -> improvement

* simplify wildcard segment matching logic

* refactor wildcard matching into func

* fix glob matching, add more wildcard validation, refactor

* refactor common wildcard errors to func

* move doc comment to logical.Paths

* optimize wildcard paths storage with pre-split slices

* fix comment typo

* fix test case after changing wildcard paths storage type

* move prefix check to parseUnauthenticatedPaths

* tweak regex, remove unneeded array copy, refactor

* add test case around wildcard and glob matching
2021-10-13 11:51:20 -05:00
hghaf099 ad2ef412cc
Customizing HTTP headers in the config file (#12485)
* Customizing HTTP headers in the config file

* Add changelog, fix bad imports

* fixing some bugs

* fixing interaction of custom headers and /ui

* Defining a member in core to set custom response headers

* missing additional file

* Some refactoring

* Adding automated tests for the feature

* Changing some error messages based on some recommendations

* Incorporating custom response headers struct into the request context

* removing some unused references

* fixing a test

* changing some error messages, removing a default header value from /ui

* fixing a test

* wrapping ResponseWriter to set the custom headers

* adding a new test

* some cleanup

* removing some extra lines

* Addressing comments

* fixing some agent tests

* skipping custom headers from agent listener config,
removing two of the default headers as they cause issues with Vault in UI mode
Adding X-Content-Type-Options to the ui default headers
Let Content-Type be set as before

* Removing default custom headers, and renaming some function varibles

* some refacotring

* Refactoring and addressing comments

* removing a function and fixing comments
2021-10-13 11:06:33 -04:00
Victor Rodriguez ae40d1c2c9
Wait for expiration manager to be out of restore mode while testing. (#12779) 2021-10-13 08:58:02 -04:00
Matt Greenfield 0b3eea4441
Dedup from_entity_ids when merging two entities (#10101)
Fixes #10100
2021-10-12 15:35:19 -04:00
Pratyoy Mukhopadhyay c52ff27e9d
Update naptime for retrytoken test (#12799) 2021-10-12 11:12:06 -07:00
John-Michael Faircloth f30c3ac621
Filter identity token keys (#12780)
* filter identity token keys

* Update test cases to associate keys with roles

* use getOIDCRole helper

* add func comment and test assertion

* add changelog

* remove unnecessary code

* build list of keys to return by starting with a list of roles

* move comment

* update changelog
2021-10-12 11:14:03 -05:00
swayne275 2edac287ae
update function signature and call (#12806) 2021-10-11 18:21:38 -06:00
John-Michael Faircloth e72c15426b
OIDC: Unique prefix to client_secret (#12745)
* Unique prefix to client_id and client_secret

* only prefix client secret and use constants

* split client secret checks
2021-10-08 16:02:20 -05:00
Scott Miller 1097f356af
Add a TRACE log with TLS connection details on replication connections (#12754)
* remove cruft
use helper
Add a helper for getting public key sizes
wip

* error names

* Fix ecdsa

* only if trace is on

* Log listener side as well

* rename

* Add remote address

* Make the log level configurable via the env var, and a member of the Listener and thus modifiable by tests

* Fix certutil_test
2021-10-07 14:17:31 -05:00
Austin Gebauer e09657e1f3
Adds ability to define an inline policy and internal metadata on tokens (#12682)
* Adds ability to define an inline policy and internal metadata to tokens

* Update comment on fetchEntityAndDerivedPolicies

* Simplify handling of inline policy

* Update comment on InternalMeta

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Improve argument name

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Use explicit SkipIdentityInheritance token field instead of implicit InlinePolicy behavior

* Add SkipIdentityInheritance to pb struct in token store create method

* Rename SkipIdentityInheritance to NoIdentityPolicies

* Merge latest from main and make proto

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-10-07 10:36:22 -07:00
swayne275 b9fde1dd6f
oss port (#12755) 2021-10-07 11:25:16 -06:00
vinay-gopalan 458927c2ed
[VAULT-3157] Move mergeStates utils from Agent to api module (#12731)
* move merge and compare states to vault core

* move MergeState, CompareStates and ParseRequiredStates to api package

* fix merge state reference in API Proxy

* move mergeStates test to api package

* add changelog

* ghost commit to trigger CI

* rename CompareStates to CompareReplicationStates

* rename MergeStates and make compareStates and parseStates private methods

* improved error messaging in parseReplicationState

* export ParseReplicationState for enterprise files
2021-10-06 10:57:06 -07:00
hghaf099 a3796997d9
Fix a Deadlock on HA leadership transfer (#12691)
* Fix a Deadlock on HA leadership transfer when standby
was actively forwarding a request
fixes GH #12601

* adding the changelog
2021-10-04 13:55:15 -04:00
vinay-gopalan 680e8515fa
[VAULT-3472] Cap Client id_token_ttl field to associated Key's verification_ttl (#12677) 2021-10-01 10:47:40 -07:00
Matt Greenfield 8577602395
Fix entity group associations (#10085)
- When two entities are merged, remove the from entity ID in any
  associated groups.
- When two entities are merged, also merge their associated group
  memberships.

Fixes #10084
2021-10-01 10:22:52 -04:00
Brian Kassouf 39a9727c8b
Update protobuf & grpc libraries and protoc plugins (#12679) 2021-09-29 18:25:15 -07:00
Tero Saarni 944332d12d
Update Go client libraries for etcd (#11980)
* Update Go client libraries for etcd

* Added etcd server container to run etcd3 tests automatically.

* Removed etcd2 test case: it fails the backend tests but the failure is
  unrelated to the uplift.  The etcd2 backend implementation does not
  remove empty nested nodes when removing leaf (see comments in #11980).
2021-09-29 14:28:13 -04:00
swayne275 af3136d393
move from static sleep to timed loop waiting for lease revocation (#12627) 2021-09-27 14:23:17 -06:00
Pratyoy Mukhopadhyay 73ae1fcfff
Update flaky expiration test (#12628)
* Update flaky expiration test

* Remove extraneous comment

Co-authored-by: swayne275 <swayne275@gmail.com>

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-09-27 11:51:27 -07:00
Austin Gebauer b58913ad9f
Adds OIDC Authorization Endpoint to OIDC providers (#12538) 2021-09-27 10:55:29 -07:00
Austin Gebauer da394f34b1
Adds additional OIDC discovery metadata (#12623) 2021-09-27 10:05:55 -07:00
Austin Gebauer 93f8d248d3
Use durations for ID and access token TTLs (#12632) 2021-09-27 10:05:28 -07:00
Calvin Leung Huang 7ad62f5be4
core: set namespace within GeneratePasswordFromPolicy (#12635)
* core: set namespace from the sysview's mount entry on GeneratePasswordFromPolicy

* test: update TestDynamicSystemView to be ns-aware, update tests

* add changelog entry
2021-09-27 09:08:07 -07:00
Pratyoy Mukhopadhyay 8e6698fb4a
[VAULT-3519] Return no_default_policy on token role read (#12565)
* [VAULT-3519] Return no_default_policy on token role read if set

* [VAULT-3519] Add changelog

* [VAULT-3519] Always return token_no_default_policy on role read

* Fix broken test

* Update role read response in docs
2021-09-21 09:53:08 -07:00
Tiernan a538936367
Allow globbing dis/allowed_policies_glob in token roles (#7277)
* Add allowed_policies_glob and disallowed_policies_glob that are the same as allowed_policies and disallowed_policies but allow glob matching.

* Update changelog, docs, tests, and comments for (dis)allowed_token_glob token role feature.

* Improve docs and unit tests for auth/token role policy globbing.
2021-09-21 08:25:06 -07:00
Nick Cabatoff 2bd95232cf
Fail alias rename if the resulting (name,accessor) exists already (#12473) 2021-09-21 08:19:44 -04:00
akshya96 c643dc1d53
Add Custom metadata field to alias (#12502)
* adding changes

* removing q.Q

* removing empty lines

* testing

* checking tests

* fixing tests

* adding changes

* added requested changes

* added requested changes

* added policy templating changes and fixed tests

* adding proto changes

* making changes

* adding unit tests

* using suggested function
2021-09-17 11:03:47 -07:00
John-Michael Faircloth bee9f25277
OIDC provider: show success message in CLI (#12574) 2021-09-17 11:41:08 -05:00
Michael Ward c15ac1053f
Expose secret_id_accessor as WrappedAccessor when wrapping secret-id creation. (#12425)
* Expose secret_id_accessor as WrappedAccessor when wrapping secret-id creation.

* Add changelog.

* Minor updates as suggested.

* Adding external test for wrapped accessor.

* Add check that mounttype is approle.

* Update changelog text to use improvement
2021-09-16 10:47:49 -07:00
John-Michael Faircloth 4ed0eb4493
identity: fix bug and increase logging for jwks cache control max age test (#12561)
* identity: increase logging for jwks cache control max age test

* clarify comment

* add more logging for jwks cache control max age in test
2021-09-15 14:38:29 -05:00
John-Michael Faircloth c42c9993a0
feature: OIDC keys endpoint (#12525)
* add keys path and initial handler

* read provider public keys

* add test cases

* remove some debug logs

* update tests after merging main

* refactor list all clients

* refactor logic to collect Key IDs
2021-09-14 15:37:53 -05:00
hghaf099 2576be23d0
Fixing a flaky test TestQuotas_RateLimitQuota_ExemptPaths (#12532)
* Fixing a flaky test TestQuotas_RateLimitQuota_ExemptPaths

* fixing a string formatting by removing an extra parameter
2021-09-10 17:41:14 -04:00
John-Michael Faircloth c42bbb369c
Identity: prepublish jwt signing keys (#12414)
* pre-publish new signing keys for `rotation_period` of time before using

* Work In Progress: Prepublish JWKS and even cache control

* remove comments

* use math/rand instead of math/big

* update tests

* remove debug comment

* refactor cache control logic into func

* don't set expiry when create/update key

* update cachecontrol name in oidccache for test

* fix bug in periodicfunc test case

* add changelog

* remove confusing comment

* add logging and comments

* update change log from bug to improvement

Co-authored-by: Ian Ferguson <ian.ferguson@datadoghq.com>
2021-09-09 13:47:42 -05:00
John-Michael Faircloth 0d04a9892a
identity: enforce key param and key existence on role creation (#12208)
* identity: handle creation of role without a key parameter

* update docs to not require key parameter for creation of a role

* add changelog

* require key param when creating a role

* lock create/update role; remove now redundant key check

* update changelog and UTs

* update change log to refelct actual implementation

* remove deprecated test case
2021-09-08 10:46:58 -05:00
Daniel Kimsey b4b61efc75
Auto-join support for IPv6 discovery (#12366)
* Auto-join support for IPv6 discovery

The go-discover library returns IP addresses and not URLs. It just so
happens net.URL parses "127.0.0.1", which isn't a valid URL.

Instead, we construct the URL ourselves. Being careful to check if it's
an ipv6 address and making sure it's in explicit form if so.

Fixes #12323

* feedback: addrs & ipv6 test

Rename addrs to clusterIPs to improve clarity and intent

Tighten up our IPv6 address detection to be more correct and to ensure
it's actually in implicit form
2021-09-07 11:55:07 -07:00
John-Michael Faircloth 01011973a3
feature: OIDC discovery endpoint (#12481)
* OIDC Provider: implement discovery endpoint

* handle case when provider does not exist

* refactor providerDiscover struct and add scopes_supported

* fix authz endpoint
2021-09-07 13:35:23 -05:00
akshya96 f4bd14ed3f
Vault 2823 cc namespace (#12393)
* vault-2823 adding changes

* VAULT-2823 adding alias

* Vault-2823 addressing comments

* Vault-2823 removing comments

* Vault-2823 removing comments

* vault-2823 removing q debug

* adding changelog

* Vault-2823 updating external test

* adding approved changes

* fixing returns

* fixing returns
2021-09-07 09:16:12 -07:00
Nick Cabatoff 45a83d8e0f
Add code to api.RaftSnapshot to detect incomplete snapshots (#12388) 2021-09-07 11:16:37 -04:00
Scott Miller 8b20cae44f
Make necessary OSS changes for contexts in quotas (#12480)
* Make necessary OSS changes for contexts in quotas

* missed one
2021-09-02 09:54:21 -05:00
Scott Miller 0f6543fb41
Upgrade go-limiter to fix building on 1.17 (#12358)
* Upgrade go-limiter

* Modify quota system to pass contexts to upgraded go-limiter

* One more spot

* Add context vars to unit tests

* missed one
2021-09-01 16:28:47 -05:00
vinay-gopalan c99cf35b6a
[VAULT-3347] Ensure Deduplication in Provider and Client APIs in OIDC Provider (#12460)
* add deduplication for Provider

* add deduplication to provider client API

* add changelog

* delete changelog

* update comments

* update test names
2021-08-30 13:57:28 -07:00
Nick Cabatoff 0762f9003d
Refactor usages of Core in IdentityStore so they can be decoupled. (#12461) 2021-08-30 15:31:11 -04:00
vinay-gopalan 6338f3dc62
feature: OIDC Provider API (#12279) 2021-08-26 10:13:51 -07:00
John-Michael Faircloth 924cab3b06
OIDC Client API: add more test coverage (#12392)
* initial commit

* add read and delete operations

* fix bug in delete and add list unit test

* func doc typo fix

* add existence check for assignment

* remove locking on the assignment resource

It is not needed at this time.

* convert Callbacks to Operations

- convert Callbacks to Operations
- add test case for update operations

* add CRUD operations and test cases

* add client api and tests

* remove use of oidcCache

* remove use of oidcCache

* add template validation and update tests

* remove usage of oidcCache

* refactor struct and var names

* harmonize test name conventions

* refactor struct and var names

* add changelog and refactor

- add changelog
- be more explicit in the case where we do not recieve a path field

* refactor

be more explicit in the case where a field is not provided

* remove extra period from changelog

* update scope path to be OIDC provider specific

* refactor naming conventions

* update assignment path

* update scope path

* enforce key existence on client creation

* removed unused name field

* removed unused name field

* removed unused name field

* prevent assignment deletion when ref'ed by a client

* enfoce assignment existence on client create/update

* update scope template description

* error when attempting to created scope with openid reserved name

* fix UT failures after requiring assignment existence

* disallow key deletion when ref'ed by existing client

* generate client_id and client_secret on CreateOp

* do not allow key modification on client update

* return client_id and client_secret on read ops

* small refactor

* fix bug in delete assignment op

* remove client secret get call

* OIDC Client API: add more test coverage

* change name convention in tests
2021-08-23 14:08:03 -05:00
John-Michael Faircloth fec8e8b21a
feature: OIDC provider client API (#12272)
* initial commit

* add read and delete operations

* fix bug in delete and add list unit test

* func doc typo fix

* add existence check for assignment

* remove locking on the assignment resource

It is not needed at this time.

* convert Callbacks to Operations

- convert Callbacks to Operations
- add test case for update operations

* add CRUD operations and test cases

* add client api and tests

* remove use of oidcCache

* remove use of oidcCache

* add template validation and update tests

* remove usage of oidcCache

* refactor struct and var names

* harmonize test name conventions

* refactor struct and var names

* add changelog and refactor

- add changelog
- be more explicit in the case where we do not recieve a path field

* refactor

be more explicit in the case where a field is not provided

* remove extra period from changelog

* update scope path to be OIDC provider specific

* refactor naming conventions

* update assignment path

* update scope path

* enforce key existence on client creation

* removed unused name field

* removed unused name field

* removed unused name field

* prevent assignment deletion when ref'ed by a client

* enfoce assignment existence on client create/update

* update scope template description

* error when attempting to created scope with openid reserved name

* fix UT failures after requiring assignment existence

* disallow key deletion when ref'ed by existing client

* generate client_id and client_secret on CreateOp

* do not allow key modification on client update

* return client_id and client_secret on read ops

* small refactor

* fix bug in delete assignment op

* remove client secret get call
2021-08-23 08:42:31 -05:00
John-Michael Faircloth 2b3da872b3
feature: OIDC provider scope API (#12266)
* initial commit

* add read and delete operations

* fix bug in delete and add list unit test

* func doc typo fix

* add existence check for assignment

* remove locking on the assignment resource

It is not needed at this time.

* convert Callbacks to Operations

- convert Callbacks to Operations
- add test case for update operations

* add CRUD operations and test cases

* remove use of oidcCache

* remove use of oidcCache

* add template validation and update tests

* refactor struct and var names

* harmonize test name conventions

* refactor struct and var names

* add changelog and refactor

- add changelog
- be more explicit in the case where we do not recieve a path field

* refactor

be more explicit in the case where a field is not provided

* remove extra period from changelog

* update scope path to be OIDC provider specific

* update assignment path

* update scope path

* removed unused name field

* removed unused name field

* update scope template description

* error when attempting to created scope with openid reserved name
2021-08-18 13:20:27 -05:00
Hridoy Roy 2554563268
(OSS Port) Restrict Quota Deletion to Primary Cluster [vault-2399] (#12339)
* oss part of vault 2399

* Update vault/quotas/quotas.go

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* use OSS PR number as changelog entry as indicated by the changelog guide

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2021-08-17 15:34:43 -07:00
John-Michael Faircloth 40fd60342a
feature: OIDC provider assignment API (#12198)
* initial commit

* add read and delete operations

* fix bug in delete and add list unit test

* func doc typo fix

* add existence check for assignment

* remove locking on the assignment resource

It is not needed at this time.

* convert Callbacks to Operations

- convert Callbacks to Operations
- add test case for update operations

* remove use of oidcCache

* refactor struct and var names

* harmonize test name conventions

* add changelog and refactor

- add changelog
- be more explicit in the case where we do not recieve a path field

* remove extra period from changelog

* update assignment path

* removed unused name field
2021-08-17 15:55:06 -05:00
Lars Lehtonen 3bb1c68996
vault: fix dropped test errors (#12242)
* vault: fix dropped test error

* vault/activity: fix dropped test error
2021-08-16 09:52:08 -04:00
Pratyoy Mukhopadhyay 691a43030d
[VAULT-2822] Sort client count response (#12308) 2021-08-12 09:35:15 -07:00
hghaf099 f885d97774
VAULT-2285 adding capability to accept comma separated entries for au… (#12126)
* VAULT-2285 adding capability to accept comma separated entries for auth enable/tune

* Adding changelog

* Adding logic to detect invalid input parameter for auth enable config

* Updating tune.mdx

* Updating secret enable/tune for comma separated parameters

* Adding further parameter checks for auth/secret tests
Fixing changelog
using builtin type for a switch statement
Fixing a possible panic scenario

* Changing a function name, using deep.Equal instead of what reflect package provides

* Fixing auth/secret enable/tune mdx files

* One more mdx file fix

* Only when users provide a single comma separated string in a curl command, split the entries by commas

* Fixing API docs for auth/mount enable/tune for comma separated entries

* updating docs, removing an unnecessary switch case
2021-08-09 15:37:03 -04:00
John-Michael Faircloth 0d94a6530f
identity: allow creating a role with a non-existent key (#12251)
* identity: allow creating a role with a non-existent key

* remove whitespace

* add changelog
2021-08-04 11:01:13 -07:00
Hridoy Roy eaa88155d1
change advice to be used with spot warning (#12237) 2021-08-02 14:41:12 -07:00
Hridoy Roy a3fefdca35
oss part of license diagnose test fix (#12234)
* oss part of license diagnose test fix

* cl
2021-08-02 10:50:49 -07:00
Hridoy Roy 28f33d2384
Fix Diagnose Formatting In Disk Usage Checks (#12229)
* save

* fix diagnose formatting errors

* fix diagnose formatting errors

* change powers

* change powers

* use humanize instead of doing the conversion to mb manually

* cl
2021-08-02 10:06:04 -07:00
Pratyoy Mukhopadhyay 113b6885c3
[VAULT-2852] deprecate req counters in oss (#12197) 2021-07-29 10:21:40 -07:00
John-Michael Faircloth 39c744ca4e
identity: do not allow a role's token_ttl to be longer than verification_ttl (#12151)
* do not allow token_ttl to be longer than verification_ttl

* add verification when updating an existing key

When updating a key, ensure any roles referencing the key do not already
have a token_ttl greater than the key's verification_ttl

* add changelog

* remove unneeded UT check and comment

* refactor based on PR comments

- remove make slice in favor of var delcaration
- remove unneeded if check
- validate expiry value during token generation
- update changelog as bug

* refactor get roles referencing target key names logic

* add note about thread safety to helper func

* update func comment

* sort array and refactor func names

* add warning to return response

* remove unnecessary code from unit test

* Update vault/identity_store_oidc.go

Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>

Co-authored-by: Austin Gebauer <34121980+austingebauer@users.noreply.github.com>
2021-07-28 20:34:52 -05:00
Pratyoy Mukhopadhyay fa29e780f0
[NO-TICKET] Upgrade protoc-gen-go to 1.26, upgrade protoc to 3.17.3 (#12171)
* [NO-TICKET] Set protoc-gen-go to 1.23, upgrade protoc to 3.17.3

* [NO-TICKET] Upgrade version of protoc-gen-go to 1.26
2021-07-28 14:51:36 -07:00
Pratyoy Mukhopadhyay 11a3c3d72d
[VAULT-2807] Count entity tokens on use (#12153) 2021-07-22 14:01:49 -07:00
Romain Aviolat c76a4e8bd1
fix: typo in function documentation (#11852) 2021-07-21 13:21:45 -07:00
Jeff Mitchell f7147025dd
Migrate to sdk/internalshared libs in go-secure-stdlib (#12090)
* Swap sdk/helper libs to go-secure-stdlib

* Migrate to go-secure-stdlib reloadutil

* Migrate to go-secure-stdlib kv-builder

* Migrate to go-secure-stdlib gatedwriter
2021-07-15 20:17:31 -04:00
Scott Miller 555eb2ae0b
Add advice as a trace option to spot checks (#12105)
* Add advice as a trace option to spot checks

* typo

* Collect advice when forming the tree
2021-07-15 18:01:51 -05:00
Nick Cabatoff f027a1b1ff
Revert #12061 due to failures in TestLogical_RequestSizeLimit (#12093) 2021-07-15 12:55:09 -04:00
hghaf099 f7635ec1b8
Add namespace in error (#12061)
* hghaf099-VAULT-1303-Adding namespace in error when it is set

* casting ResponseWriter in handleMonitor to logical.NamespaceResponseWriter

* Casting ResponseWriter conditionally for http.Flusher
Adding changelog

* Improving changlog message
2021-07-14 15:55:55 -04:00
Scott Miller 1aa98e81f7
Make any section skippable via string contains skip checking. (#12055)
* Make any section skippable via string contains skip checking

* Fix unit test
2021-07-13 18:25:04 -05:00
Hridoy Roy d3034f2851
Diagnose Language Pass (#11909)
* save

* save

* save

* first round of the diagnose language pass

* capitalization

* first round of feedback

* fix bug in advise

* a few more nouns to verbs
2021-07-11 15:44:19 -07:00
Chris Capurso 349378454a
[VAULT-1836] Support kv-v1 generic mounts for vault.kv.secret.count metric (#12020)
* [VAULT-1836] Support kv-v1 generic mounts for vault.kv.secret.count metric

* [VAULT-1836] Add changelog entry
2021-07-09 11:05:05 -04:00
Vishal Nayak eecb39a57f
OSS parts of Autopilot in DR secondaries (#12014) 2021-07-08 12:30:01 -04:00
Pratyoy Mukhopadhyay adbaad17db
[VAULT-708] Zero out request counter on preSeal (#11970)
* [VAULT-708] Zero out request counter on preSeal

* [VAULT-708] Added changelog entry

* [VAULT-708] Add comment clarifying request counter zeroing
2021-07-07 14:03:39 -05:00
Nick Cabatoff e49f125fe5
Return a more meaningful error when trying to use a partially-replicated token. (#11993) 2021-07-07 08:43:58 -04:00
Pratyoy Mukhopadhyay a8b62bd93e
[VAULT-1981] Add OSS changes (#11999) 2021-07-06 17:12:24 -05:00
swayne275 6664846a65
Vault-2713: Retry Irrevocable Lease Revocation Periodically (#11977)
* prototype function to retry irrevocable lease revocation, clean up comment

* setup irrevocable lease revoke retry on expiration manager creation

* reduce expiration lag to 1 hour, use core context and timeout on retry attempts

* add lease namespace to revocation call

* start irrevocable revocation attempts in setupExpiration

* grab reference to channel to avoid race test failures
2021-07-06 15:05:04 -06:00
Pratyoy Mukhopadhyay c7e6a07939
Augment lease revocation metrics (#11954)
* [VAULT-1982] Add metric for irrevocable leases

* [VAULT-1982] Add metric for tracking worker count per queue

* [VAULT-1982] Update external docs

* [VAULT-1982] Add nil check on metrics sink

* [VAULT-1982] Update metric name, fix access patterns

* [VAULT-1982] Add locking when marking leases as irrevocable

* [VAULT-1982] Remove workers per mount metric

* [vault-1982] Misc review fixes

* [VAULT-1982] minor refactor

* [VAULT-1982] Add/update some comments
2021-06-30 14:37:36 -07:00
swayne275 fd517e6114
remove trailing / on leases API endpoint (#11959) 2021-06-30 13:02:26 -06:00
swayne275 f882c31856
update oss files with changes from ent PR #1928 (#11965) 2021-06-29 15:02:30 -06:00
Lars Lehtonen 27679e4fda
fix dropped errors in external_tests (#11947)
* vault/external_tests/token: fix dropped test error

* vault/external_tests/identity: fix dropped test errors
2021-06-29 09:04:26 -04:00
swayne275 67bc4fdabd
Require special privileges to list irrevocable leases (#11888)
* add leases path to sudo required set

* update TestSystemBackend_RootPaths with new special privilege paths

* note that list-leases requires sudo

* minor typo fixes
2021-06-28 16:51:47 -06:00
Hridoy Roy f3895f7cce
oss part of license diagnose (#11939) 2021-06-25 14:18:34 -07:00
hghaf099 6d7dbc85b5
diagnose: Add seal transit tls check (#11922)
* diagnose: Add seal transit tls check

* Fixing the path to the config file and the path to the cert files

* Addressing comment

* Addressing seal transit tls check comments
2021-06-24 18:30:42 -04:00
Hridoy Roy a8cc5e4557
TLS Verification Bugfixes (#11910)
* tls verification bugfix

* tls verification bugfix

* allow diagnose fail to report status when there are also warnings

* allow diagnose fail to report status when there are also warnings

* Update vault/diagnose/helpers_test.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* comments

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-24 10:43:49 -07:00
hghaf099 658a4ea276
Checking Validity of all Certs in the chain [VAULT-2114] (#11883)
* Checking Validity of all Certs in the chain

* Addressing Comments for TLS cert validation

* Fixing tls_verification tests

* Fixing minor issue in tls_verification tests

* Addressing Comments, Rebasing with main

* Adding comment on top of a test
2021-06-22 16:23:55 -04:00
Hridoy Roy bbfd450319
remove prints (#11918) 2021-06-22 09:40:53 -07:00
hghaf099 38b753046b
Tls check client certs [VAULT-2117] (#11860)
* add expiration warning to certificate checking for diagnose

* Adding TLS Client CA checks to diagnose
Adding checks for tls_disable_client_certs and tls_require_and_verify_client_cert flags

* Check validity of TLSClientCAFile
Adding related tests

* Addressing comments

* Fixing some tls tests

* Addressing comments

Co-authored-by: HridoyRoy <roy@hashicorp.com>
2021-06-18 19:35:38 -04:00
Scott Miller ee0d6603f3
Wire configuration checks into diagnose and fix resulting bugs. (#11854)
* Actually call config.Validate in diagnose

* Wire configuration checks into diagnose and fix resulting bugs.

* go mod vendor

* Merge to vendorless version

* Remove sentinel section to allow diagnose_ok to pass

* Fix unit tests
2021-06-17 13:09:37 -05:00
Hridoy Roy e38f991054
Diagnose checks for raft quorum status and file backend permissions (#11771)
* raft file and quorum checks

* raft checks

* backup

* raft file checks test

* address comments and add more raft and file and process checks

* syntax issues

* modularize functions to compile differently on different os

* compile raft checks everywhere

* more build tag issues

* raft-diagnose

* correct file permission checks

* upgrade tests and add a getConfigOffline test that currently does not work

* comment

* update file checks method signature on windows

* Update physical/raft/raft_test.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* raft tests

* add todo comment for windows root ownership

* voter count message

* raft checks test fixes

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-06-17 10:04:21 -07:00
Luke Jolly 0779d260ec
Fix error log typo for failing to decrypt keys (#11873) 2021-06-17 10:12:13 -04:00
Lars Lehtonen f390cad6e4
vault: fix dropped identity store test errors (#11867) 2021-06-16 11:32:20 -04:00
Hridoy Roy f46764abbf
Diagnose TLS Cert Expiry Warning Check: change 1 week expiry to 1 month expiry (#11871)
* change 1 week expiry to 1 month expiry

* report time left to expiry

* report time left to expiry
2021-06-15 12:31:28 -07:00
Hridoy Roy e58c39033b
Add expiration warning to certificate checking for diagnose [VAULT-1018] (#11850)
* add expiration warning to certificate checking for diagnose

* Update serviceregistration/consul/consul_service_registration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* review comments

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-15 09:53:29 -07:00
Nick Cabatoff 2ce2617a8a
Reorganize request handling code so that we don't touch storage until we have the stateLock. (#11835) 2021-06-11 13:18:16 -04:00
Lars Lehtonen 5c8b3db0e5
vault: token store dropped test errors (#11799) 2021-06-11 11:41:31 -04:00
Brian Kassouf b42529dd17
Omit wrapping tokens and control groups from client counts (#11826)
* Omit wrapping tokens and control groups from client counts

* add changelog note
2021-06-10 15:57:51 -07:00
Hridoy Roy 65e3489c45
Diagnose resource creation checks (#11627)
* initial refactoring of unseal step in run

* remove waitgroup

* remove waitgroup

* backup work

* backup

* backup

* completely modularize run and move into diagnose

* add diagnose errors for incorrect number of unseal keys

* comment tests back in

* backup

* first subspan

* finished subspanning but running into error with timeouts

* remove runtime checks

* merge main branch

* meeting updates

* remove telemetry block

* roy comment

* subspans for seal finalization and wrapping diagnose latency checks

* backup while I fix something else

* fix storage latency test errors

* runtime checks

* diagnose with timeout on seal
2021-06-10 12:29:32 -07:00
swayne275 7c5c9c830e
fix irrevocable leases API deadlock on m.coreStateLock (#11815)
* fix m.coreStateLock deadlock for API handler code

* improve lock comment
2021-06-10 11:01:14 -06:00
Mark Gritter 3e8a4bad1a
Acquire a per-lock lease to make renew and revoke atomic wrt each other. (#11122)
* Acquire a per-lock lease to make renew and revoke atomic wrt each other.
This means we don't have to hold pendingLock during I/O.

* Attempted fix for deadlock in token revocation.

* Comment fix.

* Fix error checking in loadEntry.

* Add benchmark

* Add a few additional locking locations

* Improve benchmark slightly

* Update vault/expiration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Update vault/expiration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Add a lease lock into tidy

Co-authored-by: Scott Miller <smiller@hashicorp.com>
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-09 17:28:41 -07:00
Nick Cabatoff c07711b488
Add some paths that use nonstandard auth. (#11804) 2021-06-09 17:04:06 -04:00
Nick Cabatoff abdf54c77e
Use the perfStandby field instead of the method because we don't want to try to grab the readlock when we're servicing a request (#11793) 2021-06-08 11:55:38 -04:00
Scott Miller b795643146
Third time trying to get the os_openbsd_arm.go right. (#11786)
* Fix the third dumb thing in this file

* ...
2021-06-07 17:19:47 -05:00
Scott Miller 69d0242db9
Add Advice and Advise functions for adding an advice section to any span. (#11760)
* wip

* wip

* Finish implementing advice handling and word wrapping

* Properly word wrap messages and warnings

* Remove debugging

* Remove debugging

* Remove unnecessary test

* unit test bug

* go vendor
2021-06-07 11:29:36 -05:00
Scott Miller c896c3a6aa
Fix SpotSkipped on openbsd/arm (#11781) 2021-06-07 11:29:15 -05:00
Hridoy Roy 1782b4e880
oss part of control groups upgrade (#11772)
* oss part of control groups upgrade

* changelog and docs

* formatting

* formatting
2021-06-07 09:15:35 -07:00
swayne275 62a92a77d7
bug fixes and improvements from ent namespace work (#11774) 2021-06-04 13:25:41 -06:00
Josh Black c8cfcd9514
OSS parts of sighup license reload (#11767) 2021-06-04 10:24:35 -07:00
Scott Miller fe8f0476fd
Fix build boolean logic causing os_common to be skipped wrongly (#11766) 2021-06-03 17:29:34 -05:00
Josh Black c6c0424a8e
OSS parts of sys/config/reload/license (#11695) 2021-06-03 10:30:30 -07:00
Scott Miller 9db384e0f7
Edge cases in the skipping gathering and output logic (#11752)
* Edge cases in the skipping gathering and output logic

* Fix unit test
2021-06-03 11:01:14 -05:00
Scott Miller d151b0b55b
Segment out disk checks to disable on openbsd/arm (#11749)
* Segment out disk checks to disable on openbsd/arm

Also add a spot skipped helper.

* Expected results may be fewer than actual because of variable length tests like disk usage

* Move to os_common and build on windows
2021-06-02 12:17:52 -05:00
swayne275 35aad1df4a
Vault 2256: fix lease count quotas causing panics on dr secondaries (#11742)
* lift relevant changes from ent to oss

* fix silent error bug in quotas
2021-06-02 10:12:05 -06:00
swayne275 9724f59180
Vault 1979: Query API for Irrevocable Leases (#11607)
* build out lease count (not fully working), start lease list

* build out irrevocable lease list

* bookkeeping

* test irrevocable lease counts for API/CLI

* fix listIrrevocableLeases, test listIrrevocableLeases, cleanup

* test expiration API limit

* namespace tweaks, test force flag on lease list

* integration test leases/count API, plenty of fixes and improvements

* test lease list API, fixes and improvements

* test force flag for irrevocable lease list API

* i guess this wasn't saved on the last refactor...

* fixes and improvements found during my review

* better test error msg

* Update vault/logical_system_paths.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* return warning with data if more than default leases to list without force flag

* make api doc more generalized

* list leases in general, not by mount point

* change force flag to include_large_results

* sort leases by LeaseID for consistent API response

* switch from bool flag for API limit to string value

* sort first by leaseID, then stable sort by expiration

* move some utils to be in oss and ent

* improve sort efficiency for API response

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-06-02 10:11:30 -06:00
Scott Miller 59bea5f6fd
Add ulimit check, and tidy unit test cases (#11678)
* Add ulimit check, and tidy unit test cases to avoid needing to have all results and perfect ordering

* Make order independent check recursive

* Fix unit tests

* Try a 5s request timeout
2021-06-01 12:43:51 -05:00
Cory Bolar 6c726bde67
Only fail auth plugin backend mounting for custom plugins (#11690)
Checking if a plugin is a builtin by comparing it's type to those in the builtin
registry allows for a custom plugin loaded with the same name to be considered a
builtin during error handling of the mounting process.  This can cause the vault
installation to brick itself because it cannot be unsealed to register a new
sha256 or file path for a previously loaded custom plugin.  Improve this logic
by checking the plugin catalog rather than the builtin registry.

Fixes #11687
2021-05-27 13:37:45 -05:00
Scott Miller 46da8ebae6
Diagnose Disk usage checks (#11673)
* Disk usage checks

* Move disk free earlier

* Move logic to helpers

* Bring over test logic from the ulimit PR

* imports

* Report error

* Get unit tests working
2021-05-27 10:17:52 -07:00
Hridoy Roy b6e1d62d33
Modularize Run Command (#11573)
* initial refactoring of unseal step in run

* remove waitgroup

* remove waitgroup

* backup work

* backup

* backup

* completely modularize run and move into diagnose

* add diagnose errors for incorrect number of unseal keys

* comment tests back in

* backup

* first subspan

* finished subspanning but running into error with timeouts

* remove runtime checks

* meeting updates

* remove telemetry block

* roy comment

* subspans for seal finalization and wrapping diagnose latency checks

* fix storage latency test errors

* review comments

* use random uuid for latency checks instead of static id
2021-05-25 15:23:20 -07:00
Scott Miller e305f78e9c
Add a Success helper to set successful spans' messages (#11621) 2021-05-21 19:21:23 -07:00
Scott Miller 941d01eee3
Initial Diagnose CLI output (#11583)
* Create helpers which integrate with OpenTelemetry for diagnose collection

* Go mod vendor

* Comments

* Update vault/diagnose/helpers.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Add unit test/example

* tweak output

* More comments

* add spot check concept

* Get unit tests working on Result structs

* wip

* Fix unit test

* Get unit tests working, and make diagnose sessions local rather than global

* Comments

* Last comments

* No need for init

* :|

* Fix helpers_test

* wip

* wip

* wip

* Revendor otel

* Fix merge related problems

* imports

* Fix unit tests

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-05-21 19:21:11 -07:00
Nick Cabatoff 2adef1f878
OSS parts of #1891 (sys/health license addition) (#11676) 2021-05-20 13:32:15 -04:00
Nick Cabatoff 01f96f18ce
VAULT-2439: OSS parts of #1889 (raft licensing init) (#11665) 2021-05-19 16:07:58 -04:00
Brian Kassouf eabfda0c79
Make sure we are checking the final peerset (#11664) 2021-05-19 12:01:52 -07:00
Brian Kassouf f498d0d389
Reload raft TLS keys on active startup (#11660) 2021-05-19 10:03:32 -07:00
Scott Miller 6b8d7fe2e6
Patch expiration fix over from ENT (#11650)
* Patch expiration fix over from ENT

* Rename changelog
2021-05-18 14:55:38 -07:00
Nick Cabatoff e212ec5d8e
OSS parts of ent PR #1857: license autoloading init changes. (#11623) 2021-05-17 14:10:26 -04:00
Scott Miller 7b05fcf9bf
Don't backoff if a listener error was a timeout (#11594) 2021-05-13 17:02:25 -05:00
Jason O'Donnell 502cf3b212
core: updates to password policy generator (#11596)
* core: fix bug in password policies not using namespaces

* Add changelog
2021-05-13 09:55:46 -04:00
Josh Black 641a81b74b
Provide a new API endpoint for retrieving signed licenses (#11543) 2021-05-12 12:19:25 -07:00
Scott Miller 9dbf1a7dba
Add infrastructure and helpers for skipping diagnose checks (#11593)
* Add infrastructure for skipping tests

* Add infrastructure for skipping tests

* Set it

* Update vault/diagnose/helpers.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Implement type alias for test functions

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-05-12 12:54:40 -05:00
Scott Miller f0c3192f20
Add an exponential backoff to TCP listeners to avoid fast loops in error scenarios (#11588)
* Add an exponential backoff to TCP listeners to avoid fast loops in error scenarios

* reset loop delay

* changelog
2021-05-12 10:47:38 -05:00
swayne275 0229787a26
Vault 2303: Count irrevocable leases in quotas/metrics and other improvements (#11542)
* shrink generic error message

* move zombie loading to updatePendingInternal from loadEntryInternal

* probably the right metric/lease behavior for irrevocable leases...

* comment improvements

* test total lease count with valid and irrevocable leases
2021-05-11 14:04:06 -06:00
Lars Lehtonen 53dd619d2f
vault: deprecate errwrap.Wrapf() (#11577) 2021-05-11 13:12:54 -04:00
swayne275 f467d86805
don't panic if no data, improve log (#11581) 2021-05-11 10:41:57 -06:00
Scott Miller 40e0806f29
Add timeout helper (#11554)
* Add timeout helper

* cleanup-imports

* didnt need mainsection

Co-authored-by: HridoyRoy <roy@hashicorp.com>
2021-05-10 17:55:43 -07:00
swayne275 ee53fc5afb
Fix activity log fragment race test (#11565)
* grab reference to done channel while locked

* add clarifying comment

* lock and grab channel reference again for test races
2021-05-07 17:02:11 -06:00
Nick Cabatoff 53c7d1de7d
config for autoloading license (oss parts) 2021-05-07 08:55:41 -04:00