Commit Graph

2354 Commits

Author SHA1 Message Date
Nick Cabatoff 1ec904976a
Note that versionTimestamps are only loaded on the active node. 2021-11-16 15:05:59 -05:00
Nick Cabatoff c2d9215d1d
Fix startup failures when aliases from a pre-1.9 vault version exist (#13169)
* Add AllowMissing to local_bucket_key schema, preventing startup failures in post-unseal when aliases from an older version exist.
2021-11-16 14:56:34 -05:00
Scott Miller a5e55f6b05
Fix a data race in the new autoseal health check (#13136)
* Move the ctx capture outside the goroutine to avoid a race

* refactor the toggleable wrapper to avoid races

* Move the capture back outside the goroutine

* defer
2021-11-12 15:58:46 -06:00
Scott Miller 10270b6985
Add a periodic test of the autoseal to detect loss of connectivity. (#13078)
* Add a periodic test of the autoseal to detect loss of connectivity

* Keep the logic adjacent to autoseal

* imports

* typo, plus unnecessary constant time compare

* changelog

* pr feedback

* More feedback

* Add locking and a unit test

* unnecessary

* Add timeouts to encrypt/decrypt operations, capture activeContext before starting loop

* Add a block scope for the timeout

* copy/paste ftl

* Refactor to use two timeouts, and cleanup the repetitive failure code

* Readd 0ing gauge

* use millis

* Invert the unit test logic
2021-11-10 14:46:07 -06:00
John-Michael Faircloth e6ffaaf835
OIDC: return full issuer uri on read provider (#13058)
* return full issuer uri on read provider

* remove err check

* simplify full issuer logic
2021-11-10 12:35:31 -06:00
swayne275 e137045050
Vault 936: use core.activeContext in ActivityLog (#13083)
* update activity log to use core's activeContext for cleaner worker termination

* update tests to use core activeContext instead of generic context

* pass context around instead

* revert context change

* undo test context changes

* change worker context

* accidentally undid context for fcn signature changes
2021-11-09 11:47:39 -07:00
Steven Clark 8c50afc07c
Address a data race issue within identity_store_util::processLocalAlias (#13093)
- When loading an existing alias within processLocalAlias we aren't
   cloning the object from the memory store. There seems to be a data
   race within the function when calling entity.UpsertAlias and
   a concurrent invalidation routine.

 ==================
WARNING: DATA RACE
Read at 0x00c00bd03d08 by goroutine 94:
  google.golang.org/protobuf/internal/impl.pointer.Elem()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/pointer_unsafe.go:118 +0x2b3
  google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:76 +0x265
  google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:56 +0x12a
  google.golang.org/protobuf/internal/impl.(*MessageInfo).size()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:40 +0x95
  google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/encode.go:33 +0x6c
  google.golang.org/protobuf/proto.MarshalOptions.marshal()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:153 +0x1f3
  google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend()
      /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/encode.go:122 +0xa5
  github.com/golang/protobuf/proto.marshalAppend()
      /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xe4
  github.com/golang/protobuf/proto.Marshal()
      /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23 +0x64
  github.com/hashicorp/vault/helper/identity.(*Entity).Clone()
      /go/src/github.com/hashicorp/vault/helper/identity/identity.go:34 +0x150
  github.com/hashicorp/vault/vault.(*IdentityStore).MemDBEntitiesByBucketKeyInTxn()
      /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:1214 +0x306
  github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate()
      /go/src/github.com/hashicorp/vault/vault/identity_store.go:216 +0xd6c
  github.com/hashicorp/vault/vault.(*IdentityStore).Invalidate-fm()
      /go/src/github.com/hashicorp/vault/vault/identity_store.go:160 +0x6d
  github.com/hashicorp/vault/sdk/framework.(*Backend).InvalidateKey()
      /go/src/github.com/hashicorp/vault/sdk/framework/backend.go:347 +0x8a
  github.com/hashicorp/vault/vault.(*IdentityStore).InvalidateKey()
      <autogenerated>:1 +0x7d
  github.com/hashicorp/vault/vault.(*Core).asyncInvalidateKey()
      /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:58 +0x390
  github.com/hashicorp/vault/vault.(*Core).asyncInvalidateHandler()
      /go/src/github.com/hashicorp/vault/vault/replication_invalidation_ent.go:71 +0x9b
  github.com/hashicorp/vault/vault.startReplicationEnt·dwrap·453()
      /go/src/github.com/hashicorp/vault/vault/replication_util_ent.go:331 +0x71

Previous write at 0x00c00bd03d08 by goroutine 52:
  github.com/hashicorp/vault/helper/identity.(*Entity).UpsertAlias()
      /go/src/github.com/hashicorp/vault/helper/identity/identity.go:55 +0x271
  github.com/hashicorp/vault/vault.(*IdentityStore).processLocalAlias()
      /go/src/github.com/hashicorp/vault/vault/identity_store_util.go:720 +0x672
  github.com/hashicorp/vault/vault.possiblyForwardEntityCreation()
      /go/src/github.com/hashicorp/vault/vault/request_handling_util_ent.go:230 +0x286
  github.com/hashicorp/vault/vault.(*Core).handleLoginRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:1345 +0x234a
  github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:607 +0x1a11
  github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:442 +0x5b5
  github.com/hashicorp/vault/vault.(*Core).HandleRequest()
      /go/src/github.com/hashicorp/vault/vault/request_handling.go:408 +0xf2
  github.com/hashicorp/vault/http.request()
      /go/src/github.com/hashicorp/vault/http/handler.go:953 +0xb1
  github.com/hashicorp/vault/http.handleLogicalInternal.func1()
      /go/src/github.com/hashicorp/vault/http/logical.go:341 +0xca
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.handleRequestForwarding.func1()
      /go/src/github.com/hashicorp/vault/http/handler.go:887 +0x4eb
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  net/http.(*ServeMux).ServeHTTP()
      /usr/local/go/src/net/http/server.go:2424 +0xc5
  github.com/hashicorp/vault/http.wrapHelpHandler.func1()
      /go/src/github.com/hashicorp/vault/http/help.go:23 +0x281
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.wrapCORSHandler.func1()
      /go/src/github.com/hashicorp/vault/http/cors.go:29 +0xb0e
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1()
      /go/src/github.com/hashicorp/vault/http/util.go:97 +0xf28
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1()
      /go/src/github.com/hashicorp/vault/http/util_ent.go:81 +0x7e3
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/vault/http.wrapGenericHandler.func1()
      /go/src/github.com/hashicorp/vault/http/handler.go:465 +0x1843
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1()
      /go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0xc1
  net/http.HandlerFunc.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2046 +0x4d
  net/http.serverHandler.ServeHTTP()
      /usr/local/go/src/net/http/server.go:2878 +0x89a
  net/http.initALPNRequest.ServeHTTP()
      /usr/local/go/src/net/http/server.go:3479 +0x34d
  net/http.(*initALPNRequest).ServeHTTP()
      <autogenerated>:1 +0x8f
  net/http.Handler.ServeHTTP-fm()
      /usr/local/go/src/net/http/server.go:87 +0x75
  net/http.(*http2serverConn).runHandler()
      /usr/local/go/src/net/http/h2_bundle.go:5832 +0xdd
  net/http.(*http2serverConn).processHeaders·dwrap·31()
      /usr/local/go/src/net/http/h2_bundle.go:5562 +0x64
2021-11-09 10:00:26 -05:00
Hridoy Roy e7e881c559
Port: Allow Routing to Partial Monthly Client Count From Namespaces (#13086)
* add function for routing activity log client counts to ent namespaces

* changelog
2021-11-08 15:38:35 -08:00
Rémi Lapeyre ae0b5e41e0
Add read support to sys/mounts/:path (#12792)
* Add read support to sys/mounts/:path

Closes https://github.com/hashicorp/vault/issues/12349

* Add changelog entry

* Empty commit to trigger CI

* Empty commit to trigger CI
2021-11-08 10:32:01 -08:00
Nick Cabatoff 40640ef43f
Fix errors logged on standbys when we try to write versions to storage (#13042) 2021-11-08 10:04:17 -05:00
Meggie 627c43e496
1.10.0-dev version bump (#12987)
* 1.10.0-dev version bump

* Remove hard-coded versions from tests. (#13026)

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2021-11-03 12:50:12 -04:00
Brian Kassouf 0e1b2c1b7b
Fix some more error shadowing issues (#12990) 2021-11-01 11:43:00 -07:00
John-Michael Faircloth 792400a24c
ensure errors are checked (#12989) 2021-11-01 13:34:51 -05:00
Austin Gebauer 83dcbc0cd5
Deny access to UserInfo endpoint if client no longer allowed by provider (#12949) 2021-10-28 09:46:07 -07:00
Austin Gebauer e5b45b5416
Adds check to ensure authorization code can only be exchanged by the issuing provider (#12948) 2021-10-28 09:45:41 -07:00
Brian Kassouf 350ecd6bc5
Fix leak of restore worker go routines after an error (#12933) 2021-10-27 09:08:10 -07:00
Austin Gebauer b3fab954fe
Use mutex in OIDC configuration handlers (#12932) 2021-10-27 08:23:05 -07:00
swayne275 92e6972b86
oss LastDRWAL (#12931) 2021-10-26 15:17:20 -06:00
Hridoy Roy 1d29b93eb5
fix adding clientID to request in audit log (#12919)
* fix adding clientID to request in audit log

* fix boolean statement

* use standard encoding for client ID instead of urlEncoding

* change encoding in tests

* add in client counts to request handling

* remove redundant client ID generation in request handling

* directly add clientID to req after handling token usage
2021-10-26 09:02:13 -07:00
Austin Gebauer a25cfe822e
Adds missing unlock of RWMutex in OIDC delete key (#12916) 2021-10-25 09:59:26 -07:00
swayne275 95e5cdd000
VAULT 2844: remove legacy lease revocation strategy (#12888)
* remove legacy lease revocation strategy

* add deprecation change log note

* remove VAULT_16_REVOKE_PERMITPOOL

* update changelog
2021-10-22 17:37:01 -06:00
hghaf099 442b5920e2
Entities may have duplicate policies (#12812)
* Entities may have duplicate plicies

* Adding changelog

* removing duplicates on reading entity policies

* fix changelog
2021-10-22 19:28:31 -04:00
Nick Cabatoff ff74f49047
Move to go 1.17 (#12868)
Also ensure that the go 1.17 breaking changes to net.ParseCIDR don't make us choke on stored CIDRs that were acceptable to older Go versions.
2021-10-21 09:32:03 -04:00
Vishal Nayak 6eead9f09b
Fix entity alias deletion (#12834)
* Fix entity alias deletion

* Fix tests

* Add CL
2021-10-19 15:05:06 -04:00
Brian Kassouf c705adc79c
Fix some linting errors (#12860) 2021-10-18 17:29:47 -07:00
Austin Gebauer 4e5b865c4f
Rename scopes to scopes_supported for OIDC providers (#12851) 2021-10-15 19:33:32 -07:00
Hridoy Roy 19822781cc
use IsRoot helper before clientID generation in activity log (#12846)
* use IsRoot helper in activity log

* use IsRoot helper in activity log
2021-10-15 14:04:55 -07:00
Vishal Nayak 476fb08e0d
Local aliases OSS patch (#12848)
* Local aliases OSS patch

* build fix
2021-10-15 15:20:00 -04:00
saltperfect 66369469d9
Removed unpublished:true for sys/internal/* endpoints (#12713)
* removed unpublished:true for sys/internal/* endpoints

* added changelog file

* updated change log and added placeholder summary as these endpoints are not mentioned in docs.

* added documentation for internal/ui/namspaces and resultant-acl

* updated log configs
2021-10-15 14:50:14 -04:00
Pratyoy Mukhopadhyay 148109b8ed
[VAULT-3252] Disallow alias creation if entity/accessor combination exists (#12747)
* Disallow alias creation if entity/accessor combination exists

* Add changelog

* Address review comments

* Add handling to aliasUpdate, some field renaming

* Update tests to work under new entity-alias constraint

* Add check to entity merge, other review fixes

* Log duplicated accessors only once

* Fix flaky test

* Add note about new constraint to docs

* Update entity merge warn log
2021-10-14 09:52:07 -07:00
Hridoy Roy 1c427d3286
Port: add client ID to TWEs in activity log [vault-3136] (#12820)
* port for tracking twes as clients

* comment clean up

* changelog

* change changelog entry phrasing
2021-10-14 09:10:59 -07:00
Austin Gebauer 0551f91068
Adds OIDC Token and UserInfo endpoints (#12711) 2021-10-13 18:59:36 -07:00
swayne275 76556c867b
port relevant ent changes (#12823) 2021-10-13 18:15:54 -06:00
Chris Capurso bbb4ab4a41
Add HTTP PATCH support to KV (#12687)
* handle HTTP PATCH requests as logical.PatchOperation

* update go.mod, go.sum

* a nil response for logical.PatchOperation should result in 404

* respond with 415 for incorrect MIME type in PATCH Content-Type header

* add abstraction to handle PatchOperation requests

* add ACLs for patch

* Adding JSON Merge support to the API client

* add HTTP PATCH tests to check high level response logic

* add permission-based 'kv patch' tests in prep to add HTTP PATCH

* adding more 'kv patch' CLI command tests

* fix TestHandler_Patch_NotFound

* Fix TestKvPatchCommand_StdinValue

* add audit log test for HTTP PATCH

* patch CLI changes

* add patch CLI tests

* change JSONMergePatch func to accept a ctx

* fix TestKVPatchCommand_RWMethodNotExists and TestKVPatchCommand_RWMethodSucceeds to specify -method flag

* go fmt

* add a test to verify patching works by default with the root token

* add changelog entry

* get vault-plugin-secrets-kv@add-patch-support

* PR feedback

* reorder some imports; go fmt

* add doc comment for HandlePatchOperation

* add json-patch@v5.5.0 to go.mod

* remove unnecessary cancelFunc for WriteBytes

* remove default for -method

* use stable version of json-patch; go mod tidy

* more PR feedback

* temp go get vault-plugin-secrets-kv@master until official release

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
2021-10-13 15:24:31 -04:00
John-Michael Faircloth 2abf916ddb
Add support to parameterize unauthenticated paths (#12668)
* store unauthenticated path wildcards in map

* working unauthenticated paths with basic unit tests

* refactor wildcard logic

* add parseUnauthenticatedPaths unit tests

* use parseUnauthenticatedPaths when reloading backend

* add more wildcard test cases

* update special paths doc; add changelog

* remove buggy prefix check; add test cases

* prevent false positives for prefix matches

If we ever encounter a mismatched segment, break and set a flag to
prevent false positives for prefix matches.

If it is a match we need to do a prefix check. But we should not return
unless HasPrefix also evaluates to true. Otherwise we should let the for
loop continue to check other possibilities and only return false once
all wildcard paths have been evaluated.

* refactor switch and add more test cases

* remove comment leftover from debug session

* add more wildcard path validation and test cases

* update changelong; feature -> improvement

* simplify wildcard segment matching logic

* refactor wildcard matching into func

* fix glob matching, add more wildcard validation, refactor

* refactor common wildcard errors to func

* move doc comment to logical.Paths

* optimize wildcard paths storage with pre-split slices

* fix comment typo

* fix test case after changing wildcard paths storage type

* move prefix check to parseUnauthenticatedPaths

* tweak regex, remove unneeded array copy, refactor

* add test case around wildcard and glob matching
2021-10-13 11:51:20 -05:00
hghaf099 ad2ef412cc
Customizing HTTP headers in the config file (#12485)
* Customizing HTTP headers in the config file

* Add changelog, fix bad imports

* fixing some bugs

* fixing interaction of custom headers and /ui

* Defining a member in core to set custom response headers

* missing additional file

* Some refactoring

* Adding automated tests for the feature

* Changing some error messages based on some recommendations

* Incorporating custom response headers struct into the request context

* removing some unused references

* fixing a test

* changing some error messages, removing a default header value from /ui

* fixing a test

* wrapping ResponseWriter to set the custom headers

* adding a new test

* some cleanup

* removing some extra lines

* Addressing comments

* fixing some agent tests

* skipping custom headers from agent listener config,
removing two of the default headers as they cause issues with Vault in UI mode
Adding X-Content-Type-Options to the ui default headers
Let Content-Type be set as before

* Removing default custom headers, and renaming some function varibles

* some refacotring

* Refactoring and addressing comments

* removing a function and fixing comments
2021-10-13 11:06:33 -04:00
Victor Rodriguez ae40d1c2c9
Wait for expiration manager to be out of restore mode while testing. (#12779) 2021-10-13 08:58:02 -04:00
Matt Greenfield 0b3eea4441
Dedup from_entity_ids when merging two entities (#10101)
Fixes #10100
2021-10-12 15:35:19 -04:00
Pratyoy Mukhopadhyay c52ff27e9d
Update naptime for retrytoken test (#12799) 2021-10-12 11:12:06 -07:00
John-Michael Faircloth f30c3ac621
Filter identity token keys (#12780)
* filter identity token keys

* Update test cases to associate keys with roles

* use getOIDCRole helper

* add func comment and test assertion

* add changelog

* remove unnecessary code

* build list of keys to return by starting with a list of roles

* move comment

* update changelog
2021-10-12 11:14:03 -05:00
swayne275 2edac287ae
update function signature and call (#12806) 2021-10-11 18:21:38 -06:00
John-Michael Faircloth e72c15426b
OIDC: Unique prefix to client_secret (#12745)
* Unique prefix to client_id and client_secret

* only prefix client secret and use constants

* split client secret checks
2021-10-08 16:02:20 -05:00
Scott Miller 1097f356af
Add a TRACE log with TLS connection details on replication connections (#12754)
* remove cruft
use helper
Add a helper for getting public key sizes
wip

* error names

* Fix ecdsa

* only if trace is on

* Log listener side as well

* rename

* Add remote address

* Make the log level configurable via the env var, and a member of the Listener and thus modifiable by tests

* Fix certutil_test
2021-10-07 14:17:31 -05:00
Austin Gebauer e09657e1f3
Adds ability to define an inline policy and internal metadata on tokens (#12682)
* Adds ability to define an inline policy and internal metadata to tokens

* Update comment on fetchEntityAndDerivedPolicies

* Simplify handling of inline policy

* Update comment on InternalMeta

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Improve argument name

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Use explicit SkipIdentityInheritance token field instead of implicit InlinePolicy behavior

* Add SkipIdentityInheritance to pb struct in token store create method

* Rename SkipIdentityInheritance to NoIdentityPolicies

* Merge latest from main and make proto

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-10-07 10:36:22 -07:00
swayne275 b9fde1dd6f
oss port (#12755) 2021-10-07 11:25:16 -06:00
vinay-gopalan 458927c2ed
[VAULT-3157] Move `mergeStates` utils from Agent to api module (#12731)
* move merge and compare states to vault core

* move MergeState, CompareStates and ParseRequiredStates to api package

* fix merge state reference in API Proxy

* move mergeStates test to api package

* add changelog

* ghost commit to trigger CI

* rename CompareStates to CompareReplicationStates

* rename MergeStates and make compareStates and parseStates private methods

* improved error messaging in parseReplicationState

* export ParseReplicationState for enterprise files
2021-10-06 10:57:06 -07:00
hghaf099 a3796997d9
Fix a Deadlock on HA leadership transfer (#12691)
* Fix a Deadlock on HA leadership transfer when standby
was actively forwarding a request
fixes GH #12601

* adding the changelog
2021-10-04 13:55:15 -04:00
vinay-gopalan 680e8515fa
[VAULT-3472] Cap Client `id_token_ttl` field to associated Key's `verification_ttl` (#12677) 2021-10-01 10:47:40 -07:00
Matt Greenfield 8577602395
Fix entity group associations (#10085)
- When two entities are merged, remove the from entity ID in any
  associated groups.
- When two entities are merged, also merge their associated group
  memberships.

Fixes #10084
2021-10-01 10:22:52 -04:00
Brian Kassouf 39a9727c8b
Update protobuf & grpc libraries and protoc plugins (#12679) 2021-09-29 18:25:15 -07:00