* Move seal barrier type field from Access to autoSeal struct.
Remove method Access.SetType(), which was only being used by a single test, and
which can use the name option of NewTestSeal() to specify the type.
* Change method signatures of Access to match those of Wrapper.
* Turn seal.Access struct into an interface.
* Tweak Access implementation.
Change `access` struct to have a field of type wrapping.Wrapper, rather than
extending it.
* Add method Seal.GetShamirWrapper().
Add method Seal.GetShamirWrapper() for use by code that need to perform
Shamir-specific operations.
* add RequestResponseCallback to core/options
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* pass in router and apply function on requests
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* add callback
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* cleanup
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Update vault/core.go
* bad typo...
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* use pvt interface, can't downcast to child struct
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* finer grained errors
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* trim path for backend
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* remove entire mount point instead of just the first part of url
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* Update vault/testing.go
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* add doc string
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* update docstring
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* reformat
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* added changelog
---------
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
Also updates the event receieved to include a timestamp.
Websockets support both JSON and protobuf binary formats.
This can be used by either `wscat` or the new
`vault events subscribe`:
e.g.,
```sh
$ wscat -H "X-Vault-Token: $(vault print token)" --connect ws://127.0.0.1:8200/v1/sys/events/subscribe/abc?json=true
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
and
```sh
$ vault events subscribe abc
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Revert "Don't execute the seal recovery tests on ENT. (#18841)"
This reverts commit 990d3bacc203c229d0f6729929d7562e678a1ac2.
* Revert "Add the ability to unseal using recovery keys via an explicit seal option. (#18683)"
This reverts commit 2ffe49aab0fc1a527c5182637c8fa3ac39b08d45.
* wip
* wip
* Got it 'working', but not happy about cleanliness yet
* Switch to a dedicated defaultSeal with recovery keys
This is simpler than trying to hijack SealAccess as before. Instead, if the operator
has requested recovery unseal mode (via a flag in the seal stanza), we new up a shamir
seal with the recovery unseal key path instead of the auto seal. Then everything proceeds
as if you had a shamir seal to begin with.
* Handle recovery rekeying
* changelog
* Revert go.mod redirect
* revert multi-blob info
* Dumb nil unmarshal target
* More comments
* Update vault/seal.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* pr feedback
* Fix recovery rekey, which needs to fetch root keys and restore them under the new recovery split
* Better comment on recovery seal during adjustSealMigration
* Make it possible to migrate from an auto-seal in recovery mode to shamir
* Fix sealMigrated to account for a recovery seal
* comments
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Address PR feedback
* Refactor duplicated migration code into helpers, using UnsealRecoveryKey/RecoveryKey where appropriate
* Don't shortcut the reast of seal migration
* get rid of redundant transit server cleanup
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* return error for meta auth and mount listing if sealed
* some logging changes
* some more logging changes
* add panic recovery
* use ErrInternalError
* Add WriteForwardedStorage to sdk's plugin, logical in OSS
This should allow backends to specify paths to forward write
(storage.Put(...) and storage.Delete(...)) operations for.
Notably, these semantics are subject to change and shouldn't yet be
relied on.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Collect paths for write forwarding in OSS
This adds a path manager to Core, allowing tracking across all Vault
versions of paths which could use write forwarding if available. In
particular, even on OSS offerings, we'll need to template {{clusterId}}
into the paths, in the event of later upgrading to Enterprise. If we
didn't, we'd end up writing paths which will no longer be accessible
post-migration, due to write forwarding now replacing the sentinel with
the actual cluster identifier.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add forwarded writer implementation to OSS
Here, for paths given to us, we determine if we need to do cluster
translation and perform local writing. This is the OSS variant.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Wire up mount-specific request forwarding in OSS
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify that state lock needs to be held to call HAState in OSS
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move cluster sentinel constant to sdk/logical
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Expose ClusterID to Plugins via SystemView
This will let plugins learn what the Cluster's ID is, without having to
resort to hacks like writing a random string to its cluster-prefixed
namespace and then reading it once it has replicated.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add GRPC ClusterID implementation
For any external plugins which wish to use it.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
For example, using:
```sh
vault server -dev -experiment events.beta1
```
Tested by checking that the events were enabled and disabled
when the `-experiment events.beta1` flag was present and absent.
Also added a small fix to pass the `hclog.Logger` in now so that
the logging hierarchy and levels are respected.
Creates a new `eventbus` package under `vault` with
an implementation of the `go-eventlogger` broker.
Also creates a stub of a common broker that will be accessible
in the core, and creates a simple event sending interface.
* add core state lockd eadlock detection config option v2
* add changelog
* split out NewTestCluster function to maintain build flag
* replace long func with constant
* remove line
* rename file, and move where detect deadlock flag is set
* expand NodeStatusReporter with new fields
* only call IsRaftVoter if using raft storage
* add changelog entry
* fix listeners
* return LogLevel as enum
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* bump github.com/hashicorp/vault/vault/hcp_link/proto
* go mod tidy
* go get link proto @vault-11829-meta-get-cluster-status
* add HA status
* add HAEnabled method
* add raft config
* allocate HA nodes based on actual count
* add raft autopilot status
* add raft quorum warnings
* add ClusterID method
* add StorageType
* add ClusterID
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* fix raft config panic
* remove "Warning" quorum message prefix
* add error wrapping
* add Core.HAStateWithLock method
* reduce quorum warnings to single string
* fix HCP_API_HOST test env var check
* Revert "fix HCP_API_HOST test env var check"
This reverts commit 97c73c4798b77b84aea84f341f2c63c4d657914d.
* Allow mounting external plugins with same name/type as deprecated builtins
* Add some go tests for deprecation status handling
* Move timestamp storage to post-unseal
* Add upgrade-aware deprecation shutdown and tests
When issuing a core.Shutdown(), it is common to background the shutdown
request. This allows Vault to continue cleaning up, mainly to release
the stateLock. This allows the shutdown to complete, but is inherently
racy, so the core.shutdownDoneCh needs to be made atomic.
* Initial worker pool
* Run postUnsealFuncs in parallel
* Use the old logic for P=1
* changelog
* Use a CPU count relative worker pool
* Update vault/core.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Done must be called once per postUnsealFunc
* Defer is overkill
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
* adding oss file changes
* check disabled and read values from config
* isUserLocked, getUserLockout Configurations, check user lock before login and return error
* remove stale entry from storage during read
* added failed login process workflow
* success workflow updated
* user lockouts external tests
* changing update to support delete
* provide access to alias look ahead function
* adding path alias lookahead
* adding tests
* added changelog
* added comments
* adding changes from ent branch
* adding lock to UpdateUserFailedLoginInfo
* fix return default bug
* Skip plugin startup for missing plugins
* Skip secrets startup for missing plugins
* Add changelog for bugfix
* Make plugin handling on unseal version-aware
* Update plugin lazy-load logic/comments for readability
* Add register/mount/deregister/seal/unseal go test
* Consolidate lazy mount logic to prevent inconsistencies
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Create global quotas of each type in every NewTestCluster. Also switch some key locks to use DeadlockMutex to make it easier to discover deadlocks in testing.
NewTestCluster also now starts the cluster, and the Start method becomes a no-op. Unless SkipInit is provided, we also wait for a node to become active, eliminating the need for WaitForActiveNode. This was needed because otherwise we can't safely make the quota api call. We can't do it in Start because Start doesn't return an error, and I didn't want to begin storing the testing object T instead TestCluster just so we could call t.Fatal inside Start.
The last change here was to address the problem of how to skip setting up quotas when creating a cluster with a nonstandard handler that might not even implement the quotas endpoint. The challenge is that because we were taking a func pointer to generate the real handler func, we didn't have any way to compare that func pointer to the standard handler-generating func http.Handler without creating a circular dependency between packages vault and http. The solution was to pass a method instead of an anonymous func pointer so that we can do reflection on it.
* Fix typos
* Return http 400 when wrong unseal key is supplied
* Add changelog
* Add test cases and change one more return case to http 400
The new case is triggered when key length is within valid range
[16, 32], but it has uneven bytes, causing crypto/aes to return
invalid key size.
* remove expected in unit tests
* include error in the new error reason
* add multikey and autoseal test cases
* return invalid key for few more code paths
Add some metrics helpful for monitoring raft cluster state.
Furthermore, we weren't emitting bolt metrics on regular (non-perf) standbys, and there were other metrics
in metricsLoop that would make sense to include in OSS but weren't. We now have an active-node-only func,
emitMetricsActiveNode. This runs metricsLoop on the active node. Standbys and perf-standbys run metricsLoop
from a goroutine managed by the runStandby rungroup.
* HCP link integration
* update configure-git.yml
* more OSS stuff
* removing internal repos
* adding a nil check
* removing config test to be included in ENT only
* updating hcp-sdk-go to v0.22.0
* remove Hostname and AuthURL link config params
Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
* OSS portion of wrapper-v2
* Prefetch barrier type to avoid encountering an error in the simple BarrierType() getter
* Rename the OveriddenType to WrapperType and use it for the barrier type prefetch
* Fix unit test
* storage/raft: Make raftInfo atomic
This fixes some racy behavior discovered in parallel testing. Change the
core struct member to an atomic and update references throughout.
* VAULT-6613 add DetermineRoleFromLoginRequest function to Core
* Fix body handling
* Role resolution for rate limit quotas
* VAULT-6613 update precedence test
* Add changelog
* VAULT-6614 start of changes for roles in LCQs
* Expiration changes for leases
* Add role information to RequestAuth
* VAULT-6614 Test updates
* VAULT-6614 Add expiration test with roles
* VAULT-6614 fix comment
* VAULT-6614 Protobuf on OSS
* VAULT-6614 Add rlock to determine role code
* VAULT-6614 Try lock instead of rlock
* VAULT-6614 back to rlock while I think about this more
* VAULT-6614 Additional safety for nil dereference
* VAULT-6614 Use %q over %s
* VAULT-6614 Add overloading to plugin backends
* VAULT-6614 RLocks instead
* VAULT-6614 Fix return for backend factory
* add func to set level for specific logger
* add endpoints to modify log level
* initialize base logger with IndependentLevels
* test to ensure other loggers remain unchanged
* add DELETE loggers endpoints to revert back to config
* add API docs page
* add changelog entry
* remove extraneous line
* add log level field to Core struct
* add godoc for getLogLevel
* add some loggers to c.allLoggers
miagilepner