* Add a stronger warning about the usage of recovery keys
* Update website/content/docs/concepts/seal.mdx
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Keep the mitigation text in the warning box
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* test/plugin: refactor compilePlugin for reuse
- move compilePlugin to helper package
- make NewTestCluster use compilePlugin
* do not overwrite plugin directory in CoreConfig if set
* fix getting plugin directory path for go build
This isn't perfect for sure, but it's solidifying and becoming a useful
base to work off.
This routes events sent from auth and secrets plugins to the main
`EventBus` in the Vault Core. Events sent from plugins are automatically
tagged with the namespace and plugin information associated with them.
* Revert "Don't execute the seal recovery tests on ENT. (#18841)"
This reverts commit 990d3bacc203c229d0f6729929d7562e678a1ac2.
* Revert "Add the ability to unseal using recovery keys via an explicit seal option. (#18683)"
This reverts commit 2ffe49aab0fc1a527c5182637c8fa3ac39b08d45.
* Move some test helper stuff from the vault package to a new helper/testhelpers/corehelpers package. Consolidate on a single "noop audit" implementation.
* Regexp metacharacter `.` should be escaped when used literally
The paths including `/.well-known/` in the Vault API could currently
technically be invoked with any random character in place of the dot.
* Replace implementation of OpenAPI path translator with regexp AST-based one
* Add changelog
* Typo fix from PR review - thanks!
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Add comment based on review feedback
* Change style of error handling as suggested in code review
* Make a further tweak to the handling of the error case
* Add more tests, testing cases which fail with the previous implementation
* Resolve issue with a test, and improve comment
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* The verify-sign command in it's cleanest existing form.
* Working state
* Updates to proper verification syntax
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
* make fmt
* Git CI caught some stuff.
* Base functionality.
* make fmt; changelog
* pki issue command.
* Make fmt. Changelog.
* Error Handling Is Almost A Tutorial
* What I thought empty issuers response fix would be.
* Some tests
* PR-review updates.
* make fmt.
* Fix null response data for listing empty issuers causing a crash.
* Update command/pki_list_children_command.go
Fix double specifier
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Add test for pki_list_children.
* Fix tests.
* Update descriptions for correctness based on PR reviews.
* make fmt.
* Updates based on PR feedback.
* Allow multiple arguements (space separated)
* Remove bad merge-thing.
* White-space hell fix change.
* Tests, and return information for issue ca
* Fix make fmt error introduced here: https://github.com/hashicorp/vault/pull/18876
* Update command/pki_issue_intermediate.go
Puncutation.
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove smart quotes for standard quotes.
* More information as part of the help text.
* Better help text.
* Add missing "/" into error message.
---------
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Add timeout functionality to inmem
* Update vault/cluster/inmem_layer.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Add comment about forceTimeout
* Add comment about time
---------
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* wip
* wip
* Got it 'working', but not happy about cleanliness yet
* Switch to a dedicated defaultSeal with recovery keys
This is simpler than trying to hijack SealAccess as before. Instead, if the operator
has requested recovery unseal mode (via a flag in the seal stanza), we new up a shamir
seal with the recovery unseal key path instead of the auto seal. Then everything proceeds
as if you had a shamir seal to begin with.
* Handle recovery rekeying
* changelog
* Revert go.mod redirect
* revert multi-blob info
* Dumb nil unmarshal target
* More comments
* Update vault/seal.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* pr feedback
* Fix recovery rekey, which needs to fetch root keys and restore them under the new recovery split
* Better comment on recovery seal during adjustSealMigration
* Make it possible to migrate from an auto-seal in recovery mode to shamir
* Fix sealMigrated to account for a recovery seal
* comments
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Address PR feedback
* Refactor duplicated migration code into helpers, using UnsealRecoveryKey/RecoveryKey where appropriate
* Don't shortcut the reast of seal migration
* get rid of redundant transit server cleanup
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* named MFA method configurations
* fix a test
* CL
* fix an issue with same config name different ID and add a test
* feedback
* feedback on test
* consistent use of passcode for all MFA methods (#18611)
* make use of passcode factor consistent for all MFA types
* improved type for MFA factors
* add method name to login CLI
* minor refactoring
* only accept MFA method name with its namespace path in the login request MFA header
* fix a bug
* fixing an ErrorOrNil return value
* more informative error message
* Apply suggestions from code review
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* feedback
* test refactor a bit
* adding godoc for a test
* feedback
* remove sanitize method name
* guard a possbile nil ref
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* return error for meta auth and mount listing if sealed
* some logging changes
* some more logging changes
* add panic recovery
* use ErrInternalError
* Add global, cross-cluster revocation queue to PKI
This adds a global, cross-cluster replicated revocation queue, allowing
operators to revoke certificates by serial number across any cluster. We
don't support revoking with private key (PoP) in the initial
implementation.
In particular, building on the PBPWF work, we add a special storage
location for handling non-local revocations which gets replicated up to
the active, primary cluster node and back down to all secondary PR
clusters. These then check the pending revocation entry and revoke the
serial locally if it exists, writing a cross-cluster confirmation entry.
Listing capabilities are present under pki/certs/revocation-queue,
allowing operators to see which certs are present. However, a future
improvement to the tidy subsystem will allow automatic cleanup of stale
entries.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow tidying revocation queue entries
No manual operator control of revocation queue entries are allowed.
However, entries are stored with their request time, allowing tidy to,
after a suitable safety buffer, remove these unconfirmed and presumably
invalid requests.
Notably, when a cluster goes offline, it will be unable to process
cross-cluster revocations for certificates it holds. If tidy runs,
potentially valid revocations may be removed. However, it is up to the
administrator to ensure the tidy window is sufficiently long that any
required maintenance is done (or, prior to maintenance when an issue is
first noticed, tidy is temporarily disabled).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Only allow enabling global revocation queue on Vault Enterprise
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use a locking queue to handle revocation requests
This queue attempts to guarantee that PKI's invalidateFunc won't have
to wait long to execute: by locking only around access to the queue
proper, and internally using a list, we minimize the time spent locked,
waiting for queue accesses.
Previously, we held a lock during tidy and processing that would've
prevented us from processing invalidateFunc calls.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* use_global_queue->cross_cluster_revocation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Grab revocation storage lock when processing queue
We need to grab the storage lock as we'll actively be revoking new
certificates in the revocation queue. This ensures nobody else is
competing for storage access, across periodic funcs, new revocations,
and tidy operations.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix expected tidy status test
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow probing RollbackManager directly in tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address review feedback on revocationQueue
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add more cancel checks, fix starting manual tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add WriteForwardedStorage to sdk's plugin, logical in OSS
This should allow backends to specify paths to forward write
(storage.Put(...) and storage.Delete(...)) operations for.
Notably, these semantics are subject to change and shouldn't yet be
relied on.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Collect paths for write forwarding in OSS
This adds a path manager to Core, allowing tracking across all Vault
versions of paths which could use write forwarding if available. In
particular, even on OSS offerings, we'll need to template {{clusterId}}
into the paths, in the event of later upgrading to Enterprise. If we
didn't, we'd end up writing paths which will no longer be accessible
post-migration, due to write forwarding now replacing the sentinel with
the actual cluster identifier.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add forwarded writer implementation to OSS
Here, for paths given to us, we determine if we need to do cluster
translation and perform local writing. This is the OSS variant.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Wire up mount-specific request forwarding in OSS
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify that state lock needs to be held to call HAState in OSS
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move cluster sentinel constant to sdk/logical
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Expose ClusterID to Plugins via SystemView
This will let plugins learn what the Cluster's ID is, without having to
resort to hacks like writing a random string to its cluster-prefixed
namespace and then reading it once it has replicated.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add GRPC ClusterID implementation
For any external plugins which wish to use it.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* added audit-hash operations
* more audit paths
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* added audit fields
* add changelog file
* dynamic fields should be nil
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* start to add test helper
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* add tests for /sys/audit openapi paths
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <anton.averchenkov@hashicorp.com>
For example, using:
```sh
vault server -dev -experiment events.beta1
```
Tested by checking that the events were enabled and disabled
when the `-experiment events.beta1` flag was present and absent.
Also added a small fix to pass the `hclog.Logger` in now so that
the logging hierarchy and levels are respected.
* OpenAPI `generic_mount_paths` follow-up
An incremental improvement within larger context discussed in #18560.
* Following the revert in #18617, re-introduce the change from
`{mountPath}` to `{<path-of-mount>_mount_path}`; this is needed, as
otherwise paths from multiple plugins would clash - e.g. almost every
auth method would provide a conflicting definition for
`auth/{mountPath}/login`, and the last one written into the map would
win.
* Move the half of the functionality that was in `sdk/framework/` to
`vault/logical_system.go` with the rest; this is needed, as
`sdk/framework/` gets compiled in to externally built plugins, and
therefore there may be version skew between it and the Vault main
code. Implementing the `generic_mount_paths` feature entirely on one
side of this boundary frees us from problems caused by this.
* Update the special exception that recognizes `system` and `identity`
as singleton mounts to also include the other two singleton mounts,
`cubbyhole` and `auth/token`.
* Include a comment that documents to restricted circumstances in which
the `generic_mount_paths` option makes sense to use:
// Note that for this to actually be useful, you have to be using it with
// a Vault instance in which you have mounted one of each secrets engine
// and auth method of types you are interested in, at paths which identify
// their type, and for the KV secrets engine you will probably want to
// mount separate kv-v1 and kv-v2 mounts to include the documentation for
// each of those APIs.
* Fix tests
Also remove comment "// TODO update after kv repo update" which was
added 4 years ago in #5687 - the implied update has not happened.
* Add changelog
* Update 18663.txt
Creates a new `eventbus` package under `vault` with
an implementation of the `go-eventlogger` broker.
Also creates a stub of a common broker that will be accessible
in the core, and creates a simple event sending interface.
* add core state lockd eadlock detection config option v2
* add changelog
* split out NewTestCluster function to maintain build flag
* replace long func with constant
* remove line
* rename file, and move where detect deadlock flag is set
* Revert "Add mount path into the default generated openapi.json spec (UI) (#17926)"
This reverts commit db8efac708e5385ec871be9558507eeaf54ac972.
* Revert "Remove `generic_mount_paths` field (#18558)"
This reverts commit 79c8f626c59ca11bb8e7f460d40b09f5e0cec76d.
* expand NodeStatusReporter with new fields
* only call IsRaftVoter if using raft storage
* add changelog entry
* fix listeners
* return LogLevel as enum
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* bump github.com/hashicorp/vault/vault/hcp_link/proto
* go mod tidy
* go get link proto @vault-11829-meta-get-cluster-status
* add HA status
* add HAEnabled method
* add raft config
* allocate HA nodes based on actual count
* add raft autopilot status
* add raft quorum warnings
* add ClusterID method
* add StorageType
* add ClusterID
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* fix raft config panic
* remove "Warning" quorum message prefix
* add error wrapping
* add Core.HAStateWithLock method
* reduce quorum warnings to single string
* fix HCP_API_HOST test env var check
* Revert "fix HCP_API_HOST test env var check"
This reverts commit 97c73c4798b77b84aea84f341f2c63c4d657914d.
Nick Cabatoff