Commit graph

2817 commits

Author SHA1 Message Date
miagilepner c31a10b90a
VAULT-13763 normalize activity log mount paths (#19343)
* add slashes to mount paths in activity log

* cleanup test

* fix test
2023-02-24 16:57:41 +01:00
Christopher Swenson 6b36cc7587
When copying test binary, delete first (#19331)
For plugin tests, we copy the test binary. On macOS, if the
destination binary already exists, then copying over it will result
in an invalid signature.

The easiest workaround is to delete the file before copying.
2023-02-23 15:10:13 -08:00
Jason O'Donnell f69297e0b3
Fix inmem layer unlock bug (#19323) 2023-02-23 20:16:49 +00:00
John-Michael Faircloth 0a7656ae5c
test: Fix bug in TestAddTestPlugin test helper (#19313)
* fix external plugin test failing locally

* Ensure file is closed and written in TestAddTestPlugin
2023-02-23 17:07:48 +00:00
Leland Ursu 432fad12b1
added in the missing test cases to validate response structures (#19277)
* added in the missing test cases to validate response structures

* added changelog file

* remove unneeded changelog file

* removed comment to update when indentity/entity is implemented

---------

Co-authored-by: lursu <leland.ursu@hashicorp.com>
2023-02-22 12:46:46 -05:00
Chris Capurso 3a361e1e83
add error consistency in link node status resp (#19279) 2023-02-22 11:53:29 -05:00
John-Michael Faircloth 2cc6117f28
test/plugin: add more test scenarios for external plugins (#19257) 2023-02-21 09:44:54 -06:00
Christopher Swenson 404d7a57bb
events: WS protobuf messages should be binary (#19232)
The [WebSockets spec](https://www.rfc-editor.org/rfc/rfc6455) states
that text messages must be valid UTF-8 encoded strings, which protobuf
messages virtually never are. This now correctly sends the protobuf events
as binary messages.

We change the format to correspond to CloudEvents, as originally intended,
and remove a redundant timestamp and newline.

We also bump the eventlogger to fix a race condition that this code triggers.
2023-02-17 11:38:03 -08:00
John-Michael Faircloth eca810d06e
test/plugin: test external database plugin workflows (#19191)
* test/plugin: test external db plugin

* use test helper to get cluster and plugins

* create test helper to create a vault admin user

* add step to revoke lease

* make tests parallel and add reload test

* use more descriptive name for test group; check response
2023-02-16 15:52:24 -06:00
Daniel Huckins 448f5dd33e
VAULT-12112: add openapi response structures for /sys/config and /sys/generate-root endpoints (#18472)
* some config responses

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* added response structs

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* added changelog

* add test for config/cors

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* add (failing) tests

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* copy-pasta err

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* update tests for /sys/config/ui/headers/{header}

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

---------

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
2023-02-16 15:06:26 -05:00
Daniel Huckins 60488687ad
VAULT-12112: add openapi response structures for /sys/capabilities* endpoints (#18468)
* add capabilities

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* added change log

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* add test

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* use nil for dynamic fields

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

---------

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
2023-02-16 15:04:37 -05:00
Daniel Huckins a9d15f1252
VAULT-12112: add openapi response structures for /sys/auth/* endpoints (#18465)
* added responses to /sys/auth/.../tune

* add response structure for auth/...

* added changelog

* Update vault/logical_system_paths.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* its TypeString

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* use nil for dynamic fields

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* test auth endpoint schema

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* kicking off ci

---------

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
2023-02-16 15:03:19 -05:00
Tom Proctor 2472029a0a
Support event subscriptions with glob wildcards (#19205) 2023-02-16 17:22:56 +00:00
Leland Ursu 0704127020
added OpenAPI response objects for sys endpoints (#18633)
* added response objects for sys 3 section

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_raw.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_quotas.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_quotas.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_quotas.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* add tests and update based on reviews

* added changelog file

* finally got make fmt to work...

* fixed copy pasta test case

* updated based on review

* Update vault/logical_system_quotas.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* Update vault/logical_system_test.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_test.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

---------

Co-authored-by: lursu <leland.ursu@hashicorp.com>
Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
2023-02-15 15:02:21 -05:00
Leland Ursu 6425130605
Added OpenAPI response structures for sys endpoints (#18515)
* added response objects to all of the endpoints laid out by the ticket linked

* added changelog file and updated based on review

* added the required bool to the correct fields

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* updated based on review

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* updated based on review and added test cases for validating response structures

* fix copy pasta issues breaking tests

* Update vault/logical_system_paths.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* fix test failures

* fixed issue with refrencing the wrong req var name

* fixed another test case and double checked the rest

* updated based on review

* updated in all locations

* Update vault/logical_system_paths.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* Update vault/logical_system_paths.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* fixed my brain fart

* Update vault/logical_system_paths.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* address fmt error

---------

Co-authored-by: lursu <leland.ursu@hashicorp.com>
Co-authored-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
2023-02-15 15:00:06 -05:00
Daniel Huckins 7fde5ecb83
Validate response schema for integration tests (#19043)
* add RequestResponseCallback to core/options

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* pass in router and apply function on requests

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* add callback

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* cleanup

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* Update vault/core.go

* bad typo...

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* use pvt interface, can't downcast to child struct

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* finer grained errors

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* trim path for backend

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* remove entire mount point instead of just the first part of url

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* Update vault/testing.go

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* add doc string

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* update docstring

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* reformat

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* added changelog

---------

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
2023-02-15 14:57:57 -05:00
Christopher Swenson 020a3903ff
events: Remove subscriptions on timeout and cancel (#19185)
When subscriptions are too slow to accept messages on their channels,
then we should remove them from the fanout so that we don't have
dead subscriptions using up resources.

In addition, when a subscription is explicitly canceled, we should
also clean up after it remove the corresponding pipeline.

We also add a new metrics, `events.subscriptions`, to keep track
of the number of active subscriptions. This also helps us test.

Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
2023-02-15 11:13:15 -08:00
Christopher Swenson 7a977fd6ea
events: Check token and ACLs on request (#19138)
This checks the request against the `read` permission for
`sys/events/subscribe/{eventType}` on the initial subscribe.

Future work includes moving this to its own verb (`subscribe`)
and periodically rechecking the request.

Tested locally by minting a token with the wrong permissions
and verifying that they are rejected as expected, and that
they work if the policy is adjusted to `sys/event/subscribe/*`
(or the specific topic name) with `read` permissions.

I had to change the `core.checkToken()` to be publicly accessible,
as it seems like the easiest way to check the token on the
`logical.Request` against all relevant policies, but without
going into all of the complex logic further in `handleLogical()`.

Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
2023-02-10 20:56:00 +00:00
Christopher Swenson 7d3d404ee2
events: Add websockets and command (#19057)
Also updates the event receieved to include a timestamp.
Websockets support both JSON and protobuf binary formats.

This can be used by either `wscat` or the new
`vault events subscribe`:

e.g.,
```sh
$ wscat -H "X-Vault-Token: $(vault print token)" --connect ws://127.0.0.1:8200/v1/sys/events/subscribe/abc?json=true
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```

and

```sh
$ vault events subscribe abc
{"event":{"id":"5c5c8c83-bf43-7da5-fe88-fc3cac814b2e", "note":"testing"}, "eventType":"abc", "timestamp":"2023-02-07T18:40:50.598408Z"}
...
```

Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
2023-02-09 13:18:58 -08:00
Tom Proctor 78d83c9136
Make experiments API authenticated (#18966) 2023-02-09 20:18:14 +00:00
John-Michael Faircloth 34fd57ac08
test/plugin: test external plugin workflows (#19090)
* test/plugin: test external plugin workflows

* update secrets engine test
2023-02-09 10:16:16 -06:00
Nick Cabatoff ec2af04ec6
Remove the last vestiges of sdk/version. (#19068) 2023-02-08 12:30:27 -05:00
Scott Miller acee981753
Remove accidental addition of a hackweek file (#19016) 2023-02-06 16:45:55 -06:00
Scott Miller b43e4fbd9c
Add a stronger warning about the usage of recovery keys (#19011)
* Add a stronger warning about the usage of recovery keys

* Update website/content/docs/concepts/seal.mdx

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Keep the mitigation text in the warning box

---------

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2023-02-06 16:23:05 -06:00
miagilepner e873932bce
VAULT-8436 remove <-time.After statements in for loops (#18818)
* replace time.After with ticker in loops

* add semgrep rule

* update to use timers

* remove stop
2023-02-06 17:49:01 +01:00
miagilepner 9d09dba7ac
VAULT-13061: Fix mount path discrepancy in activity log (#18916)
* use single function to convert mount accessor to mount path

* add changelog

* more context and comments for the tests
2023-02-06 10:26:32 +01:00
John-Michael Faircloth 14e4d67026
test/plugin: refactor compilePlugin for reuse (#18952)
* test/plugin: refactor compilePlugin for reuse

- move compilePlugin to helper package
- make NewTestCluster use compilePlugin

* do not overwrite plugin directory in CoreConfig if set

* fix getting plugin directory path for go build
2023-02-03 16:27:11 -06:00
Christopher Swenson 5864075c30
Add events sending routed from plugins (#18834)
This isn't perfect for sure, but it's solidifying and becoming a useful
base to work off.

This routes events sent from auth and secrets plugins to the main
`EventBus` in the Vault Core. Events sent from plugins are automatically
tagged with the namespace and plugin information associated with them.
2023-02-03 13:24:16 -08:00
Chris Capurso c74c057bdb
fix sys/leases panic when lease_id is nil (#18951)
* fix sys/leases panic when lease_id is nil

* add changelog entry
2023-02-03 09:51:10 -05:00
Anton Averchenkov 619f5b2edf
openapi: Ensure mount_path parameters are marked as required (#18955) 2023-02-02 14:39:09 -05:00
Chris Capurso 53390eaddf
Add ClusteName to GetClusterStatus response (#18950)
* bump github.com/hashicorp/vault/vault/hcp_link/proto

* add ClusterName to GetClusterStatus response
2023-02-02 09:27:55 -05:00
Chris Capurso c8660ca2ea
add ClusterName to meta GetClusterStatusResponse (#18944)
* add ClusterName to meta GetClusterStatusResponse

* make proto
2023-02-01 15:15:04 -05:00
Scott Miller 20551261bd
Revert #18683 (#18942)
* Revert "Don't execute the seal recovery tests on ENT. (#18841)"

This reverts commit 990d3bacc203c229d0f6729929d7562e678a1ac2.

* Revert "Add the ability to unseal using recovery keys via an explicit seal option. (#18683)"

This reverts commit 2ffe49aab0fc1a527c5182637c8fa3ac39b08d45.
2023-02-01 13:34:53 -06:00
Hamid Ghaf 6672d3753f
check for nil references (#18871)
* check for nil references

* feedback

* feedback

* address vault-issue-18849

* Revert "address vault-issue-18849"

This reverts commit 081610c02398cf7ecdfb726cd3f622f3e69eafc4.
2023-02-01 09:51:52 -05:00
Nick Cabatoff c2b222a11a
Vault test cluster helper refactorings, mostly audit related (#18928)
* Move some test helper stuff from the vault package to a new helper/testhelpers/corehelpers package.  Consolidate on a single "noop audit" implementation.
2023-02-01 08:33:16 -05:00
Anton Averchenkov 7b356f0366
openapi: Add default values to the thing_mount_path parameters (#18935) 2023-01-31 19:37:16 -05:00
Max Bowsher 9d863a92ce
Fix multiple OpenAPI generation issues with new AST-based generator (#18554)
* Regexp metacharacter `.` should be escaped when used literally

The paths including `/.well-known/` in the Vault API could currently
technically be invoked with any random character in place of the dot.

* Replace implementation of OpenAPI path translator with regexp AST-based one

* Add changelog

* Typo fix from PR review - thanks!

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>

* Add comment based on review feedback

* Change style of error handling as suggested in code review

* Make a further tweak to the handling of the error case

* Add more tests, testing cases which fail with the previous implementation

* Resolve issue with a test, and improve comment

---------

Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
2023-01-31 16:27:39 -05:00
Hamid Ghaf 9c8fcaf5a5
prevent panic on mfa enforcement delete after a namespace is deleted (#18923)
* prevent panic on mfa enforcement delete after a namespace is deleted

* CL
2023-01-31 21:06:16 +00:00
Chris Capurso 6cb6157d37
return 403 for wrapping requests when no token provided (#18859)
* return 403 for wrapping requests when no token provided

* add changelog entry

* fix changelog

* use errors.As

* simplify error response string
2023-01-31 13:57:50 -05:00
Nick Cabatoff db41c5319d
TestClusterCore's TLSConfig becomes a method and does a Clone. (#18914) 2023-01-31 11:05:16 -05:00
akshya96 16ce923ddb
Brute forcing unlock user bug (#18890)
* brute forcing unlock user bug

* add changelog

* fix changelog
2023-01-30 13:06:10 -08:00
Kit Haines 5ece71109a
Vault 11798 vault cli issue intermediate (#18467)
* The verify-sign command in it's cleanest existing form.

* Working state

* Updates to proper verification syntax

Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>

* make fmt

* Git CI caught some stuff.

* Base functionality.

* make fmt; changelog

* pki issue command.

* Make fmt. Changelog.

* Error Handling Is Almost A Tutorial

* What I thought empty issuers response fix would be.

* Some tests

* PR-review updates.

* make fmt.

* Fix null response data for listing empty issuers causing a crash.

* Update command/pki_list_children_command.go

Fix double specifier

Co-authored-by: Steven Clark <steven.clark@hashicorp.com>

* Add test for pki_list_children.

* Fix tests.

* Update descriptions for correctness based on PR reviews.

* make fmt.

* Updates based on PR feedback.

* Allow multiple arguements (space separated)

* Remove bad merge-thing.

* White-space hell fix change.

* Tests, and return information for issue ca

* Fix make fmt error introduced here: https://github.com/hashicorp/vault/pull/18876

* Update command/pki_issue_intermediate.go

Puncutation.

Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove smart quotes for standard quotes.

* More information as part of the help text.

* Better help text.

* Add missing "/" into error message.

---------

Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
2023-01-27 16:41:16 -05:00
Jason O'Donnell e14f0216a0
Add timeout functionality to inmem (#18876)
* Add timeout functionality to inmem

* Update vault/cluster/inmem_layer.go

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Add comment about forceTimeout

* Add comment about time

---------

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2023-01-27 16:46:12 +00:00
Scott Miller af7df08921
Don't execute the seal recovery tests on ENT. (#18841)
* Don't execute the seal recovery tests on OSS.

* correct go:build syntax
2023-01-25 15:35:15 -06:00
Hamid Ghaf 2b5e5121e1
Provide IP Address in Duo Request (#18811)
* Provide IP Address in Duo Request

* CL
2023-01-24 17:28:59 -05:00
Scott Miller 25960fd034
Add the ability to unseal using recovery keys via an explicit seal option. (#18683)
* wip

* wip

* Got it 'working', but not happy about cleanliness yet

* Switch to a dedicated defaultSeal with recovery keys

This is simpler than trying to hijack SealAccess as before.  Instead, if the operator
has requested recovery unseal mode (via a flag in the seal stanza), we new up a shamir
seal with the recovery unseal key path instead of the auto seal.  Then everything proceeds
as if you had a shamir seal to begin with.

* Handle recovery rekeying

* changelog

* Revert go.mod redirect

* revert multi-blob info

* Dumb nil unmarshal target

* More comments

* Update vault/seal.go

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Update changelog/18683.txt

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* pr feedback

* Fix recovery rekey, which needs to fetch root keys and restore them under the new recovery split

* Better comment on recovery seal during adjustSealMigration

* Make it possible to migrate from an auto-seal in recovery mode to shamir

* Fix sealMigrated to account for a recovery seal

* comments

* Update changelog/18683.txt

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Address PR feedback

* Refactor duplicated migration code into helpers, using UnsealRecoveryKey/RecoveryKey where appropriate

* Don't shortcut the reast of seal migration

* get rid of redundant transit server cleanup

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2023-01-24 14:57:56 -06:00
Hamid Ghaf edbf093290
Allow Token Create Requests To Be Replicated (#18689)
* Allow Token Create Requests To Be Replicated

* adding a test

* revert a test
2023-01-24 14:00:27 -05:00
Hamid Ghaf 65a41d4f08
named Login MFA methods (#18610)
* named MFA method configurations

* fix a test

* CL

* fix an issue with same config name different ID and add a test

* feedback

* feedback on test

* consistent use of passcode for all MFA methods (#18611)

* make use of passcode factor consistent for all MFA types

* improved type for MFA factors

* add method name to login CLI

* minor refactoring

* only accept MFA method name with its namespace path in the login request MFA header

* fix a bug

* fixing an ErrorOrNil return value

* more informative error message

* Apply suggestions from code review

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* feedback

* test refactor a bit

* adding godoc for a test

* feedback

* remove sanitize method name

* guard a possbile nil ref

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2023-01-23 15:51:22 -05:00
Tom Proctor fc378c0908
Event system alpha experiment (#18795) 2023-01-23 19:26:49 +00:00
Chris Capurso 0eedcd979b
Fix link meta panics (#18774)
* return error for meta auth and mount listing if sealed

* some logging changes

* some more logging changes

* add panic recovery

* use ErrInternalError
2023-01-23 09:59:15 -05:00
Alexander Scheel b3dc380c82
Add cross-cluster revocation queues for PKI (#18784)
* Add global, cross-cluster revocation queue to PKI

This adds a global, cross-cluster replicated revocation queue, allowing
operators to revoke certificates by serial number across any cluster. We
don't support revoking with private key (PoP) in the initial
implementation.

In particular, building on the PBPWF work, we add a special storage
location for handling non-local revocations which gets replicated up to
the active, primary cluster node and back down to all secondary PR
clusters. These then check the pending revocation entry and revoke the
serial locally if it exists, writing a cross-cluster confirmation entry.

Listing capabilities are present under pki/certs/revocation-queue,
allowing operators to see which certs are present. However, a future
improvement to the tidy subsystem will allow automatic cleanup of stale
entries.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow tidying revocation queue entries

No manual operator control of revocation queue entries are allowed.
However, entries are stored with their request time, allowing tidy to,
after a suitable safety buffer, remove these unconfirmed and presumably
invalid requests.

Notably, when a cluster goes offline, it will be unable to process
cross-cluster revocations for certificates it holds. If tidy runs,
potentially valid revocations may be removed. However, it is up to the
administrator to ensure the tidy window is sufficiently long that any
required maintenance is done (or, prior to maintenance when an issue is
first noticed, tidy is temporarily disabled).

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Only allow enabling global revocation queue on Vault Enterprise

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use a locking queue to handle revocation requests

This queue attempts to guarantee that PKI's invalidateFunc won't have
to wait long to execute: by locking only around access to the queue
proper, and internally using a list, we minimize the time spent locked,
waiting for queue accesses.

Previously, we held a lock during tidy and processing that would've
prevented us from processing invalidateFunc calls.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* use_global_queue->cross_cluster_revocation

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Grab revocation storage lock when processing queue

We need to grab the storage lock as we'll actively be revoking new
certificates in the revocation queue. This ensures nobody else is
competing for storage access, across periodic funcs, new revocations,
and tidy operations.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Fix expected tidy status test

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow probing RollbackManager directly in tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Address review feedback on revocationQueue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add more cancel checks, fix starting manual tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 09:29:27 -05:00
Alexander Scheel 6930568076
Add path based primary write forwarding (PBPWF) - OSS (#18735)
* Add WriteForwardedStorage to sdk's plugin, logical in OSS

This should allow backends to specify paths to forward write
(storage.Put(...) and storage.Delete(...)) operations for.

Notably, these semantics are subject to change and shouldn't yet be
relied on.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Collect paths for write forwarding in OSS

This adds a path manager to Core, allowing tracking across all Vault
versions of paths which could use write forwarding if available. In
particular, even on OSS offerings, we'll need to template {{clusterId}}
into the paths, in the event of later upgrading to Enterprise. If we
didn't, we'd end up writing paths which will no longer be accessible
post-migration, due to write forwarding now replacing the sentinel with
the actual cluster identifier.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add forwarded writer implementation to OSS

Here, for paths given to us, we determine if we need to do cluster
translation and perform local writing. This is the OSS variant.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Wire up mount-specific request forwarding in OSS

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Clarify that state lock needs to be held to call HAState in OSS

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Move cluster sentinel constant to sdk/logical

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Expose ClusterID to Plugins via SystemView

This will let plugins learn what the Cluster's ID is, without having to
resort to hacks like writing a random string to its cluster-prefixed
namespace and then reading it once it has replicated.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add GRPC ClusterID implementation

For any external plugins which wish to use it.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-20 16:36:18 -05:00
Christopher Swenson 4a93097895
Use schema for events in event broker (#18693)
For the new events schema. Based on the CloudEvents schema.
2023-01-20 10:18:23 -08:00
Daniel Huckins fc6d13e29d
VAULT-12112: openapi response definitions: sys/audit (#18456)
* added audit-hash operations

* more audit paths

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* added audit fields

* add changelog file

* dynamic fields should be nil

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* start to add test helper

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>

* add tests for /sys/audit openapi paths

Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <anton.averchenkov@hashicorp.com>
2023-01-20 11:09:33 -05:00
Josh Black fa1447cb3c
Add new clients into the monthly breakdown (#18766)
* Add new clients into the monthly breakdown

* add changelog
2023-01-19 09:12:17 -08:00
Christopher Swenson fcbce0effd
Start events when core starts if enabled (#18742)
For example, using:

```sh
vault server -dev -experiment events.beta1
```

Tested by checking that the events were enabled and disabled
when the `-experiment events.beta1` flag was present and absent.

Also added a small fix to pass the `hclog.Logger` in now so that
the logging hierarchy and levels are respected.
2023-01-18 10:46:01 -08:00
Max Bowsher 4c5f583f39
OpenAPI generic_mount_paths follow-up (#18663)
* OpenAPI `generic_mount_paths` follow-up

An incremental improvement within larger context discussed in #18560.

* Following the revert in #18617, re-introduce the change from
  `{mountPath}` to `{<path-of-mount>_mount_path}`; this is needed, as
  otherwise paths from multiple plugins would clash - e.g. almost every
  auth method would provide a conflicting definition for
  `auth/{mountPath}/login`, and the last one written into the map would
  win.

* Move the half of the functionality that was in `sdk/framework/` to
  `vault/logical_system.go` with the rest; this is needed, as
  `sdk/framework/` gets compiled in to externally built plugins, and
  therefore there may be version skew between it and the Vault main
  code. Implementing the `generic_mount_paths` feature entirely on one
  side of this boundary frees us from problems caused by this.

* Update the special exception that recognizes `system` and `identity`
  as singleton mounts to also include the other two singleton mounts,
  `cubbyhole` and `auth/token`.

* Include a comment that documents to restricted circumstances in which
  the `generic_mount_paths` option makes sense to use:

	    // Note that for this to actually be useful, you have to be using it with
	    // a Vault instance in which you have mounted one of each secrets engine
	    // and auth method of types you are interested in, at paths which identify
	    // their type, and for the KV secrets engine you will probably want to
	    // mount separate kv-v1 and kv-v2 mounts to include the documentation for
	    // each of those APIs.

* Fix tests

Also remove comment "// TODO update after kv repo update" which was
added 4 years ago in #5687 - the implied update has not happened.

* Add changelog

* Update 18663.txt
2023-01-17 23:07:11 -05:00
akshya96 6e04e4ede1
Prevent brute forcing : telemetry oss changes (#18718)
* Prevent brute forcing : telemetry oss changes

* adding changelog
2023-01-17 15:10:50 -08:00
akshya96 b2276a369a
Prevent Brute Forcing: Create an api endpoint to list locked users OSS changes (#18675)
* api to list lockedusers oss changes

* add changelog
2023-01-17 14:25:56 -08:00
Josh Black c9763996d4
Enable undo logs by default (#18692)
* Enable undo logs by default

* add consul test

* update go.mod/sum

* add a better non-existent key
2023-01-17 13:38:18 -08:00
Christopher Swenson b95beeb675
Add basic event bus broker stub (#18640)
Creates a new `eventbus` package under `vault` with
an implementation of the `go-eventlogger` broker.

Also creates a stub of a common broker that will be accessible
in the core, and creates a simple event sending interface.
2023-01-17 13:34:37 -08:00
Tom Proctor d5c35f39c3
Add experiment system + events experiment (#18682) 2023-01-16 16:07:18 +00:00
Peter Wilson 59450ecb82
Revert "Add new clients into the monthly breakdown (#18629)" (#18726)
This reverts commit d641bbc28e5e8cc12b81d409e5d5fc1f2cb7f66c.
2023-01-16 15:51:19 +00:00
akshya96 78546af8fc
Vault 8308 Background thread to update locked user entries (#18673)
* background thread changes

* adding changelog

* fix changelog typo
2023-01-12 14:09:33 -08:00
Chris Capurso 03b23a7970
VAULT-12548: Fix data races in Link status reporting (#18674)
* pin hcp-scada-provider

* fix link reportStatus data race

* go get hcp-scada-provider@v0.2.1
2023-01-12 15:10:34 -05:00
Ellie 6f7757e949
add core state lock deadlock detection config option v2 (#18604)
* add core state lockd eadlock detection config option v2

* add changelog

* split out NewTestCluster function to maintain build flag

* replace long func with constant

* remove line

* rename file, and move where detect deadlock flag is set
2023-01-11 13:32:05 -06:00
Max Bowsher 6d6a726f9d
Fix HelpOperation on sudo-protected paths (#18568)
* Fix HelpOperation on sudo-protected paths

Fixes #18566

* Add changelog
2023-01-10 12:17:16 -06:00
Anton Averchenkov c7f40361eb
Revert "Add mount path into the default generated openapi.json spec (#17926)" (#18617)
* Revert "Add mount path into the default generated openapi.json spec (UI) (#17926)"

This reverts commit db8efac708e5385ec871be9558507eeaf54ac972.

* Revert "Remove `generic_mount_paths` field (#18558)"

This reverts commit 79c8f626c59ca11bb8e7f460d40b09f5e0cec76d.
2023-01-10 11:16:59 -05:00
Josh Black d3f822a938
Add new clients into the monthly breakdown (#18629)
* Add new clients into the monthly breakdown

* add changelog
2023-01-09 15:26:11 -08:00
Chris Capurso 25d0afae23
VAULT-11830: Expand NodeStatusReporter with new fields (#18302)
* expand NodeStatusReporter with new fields

* only call IsRaftVoter if using raft storage

* add changelog entry

* fix listeners

* return LogLevel as enum

* update github.com/hashicorp/vault/vault/hcp_link/proto

* add changelog entry

* bump github.com/hashicorp/vault/vault/hcp_link/proto

* go mod tidy
2023-01-06 20:53:09 -05:00
Chris Capurso bb0c92afe7
VAULT-11829: Add cluster status handler (#18351)
* go get link proto @vault-11829-meta-get-cluster-status

* add HA status

* add HAEnabled method

* add raft config

* allocate HA nodes based on actual count

* add raft autopilot status

* add raft quorum warnings

* add ClusterID method

* add StorageType

* add ClusterID

* update github.com/hashicorp/vault/vault/hcp_link/proto

* add changelog entry

* fix raft config panic

* remove "Warning" quorum message prefix

* add error wrapping

* add Core.HAStateWithLock method

* reduce quorum warnings to single string

* fix HCP_API_HOST test env var check

* Revert "fix HCP_API_HOST test env var check"

This reverts commit 97c73c4798b77b84aea84f341f2c63c4d657914d.
2023-01-06 17:06:54 -05:00
Chris Capurso 82eaecd745
change quorum warning to a singular string (#18619) 2023-01-06 15:31:27 -05:00
Chris Capurso de59e29bbc
VAULT-11829: Add GetClusterStatus rpc to meta capability (#18316)
* add GetClusterStatus rpc to meta capability

* consolidate HA-related info

* add storage type
2023-01-05 13:33:08 -05:00
Violet Hynes ae653a05a7
VAULT-12489 OSS Changes (#18607) 2023-01-05 13:00:55 -05:00
Josh Black c8a8c21cee
Account for mount counts when de-duplicating current and historical month data (#18598)
* Account for mount counts when de-duplicating current and historical month data

* add changelog
2023-01-05 09:34:05 -08:00
Chris Capurso e7787a94e0
VAULT-11827: Add new Link node level fields (#18203)
* add addition Link node-level status fields

* pin protoc to 3.21.9

* make proto

* change LogLevel to be a string

* add RaftStatus; IsActive -> Activetime

* use an enum for LogLevel
2023-01-05 11:16:48 -05:00
Max Bowsher 7d87548f4f
Fix duplicate definition of path sys/internal/specs/openapi (#18553)
This was accidentally duplicated in #5687.

Remove the second definition, which was shadowed by the first, and move
the documentation that was part of the second to the surviving version.
2023-01-04 22:48:40 -05:00
Ikko Eltociear Ashimine 6927478ab9
Fix typo in mount.go (#18575)
heirarchy -> hierarchy
2023-01-04 21:52:42 -05:00
vinay-gopalan bbd8ac9bbf
Upgrade go.opentelemetry.io/otel from v0.20.0 to v1.11.2 (#18589) 2023-01-04 11:31:30 -08:00
Chris Capurso 0635d304de
only update SCADA metadata if status changes (#18585)
* only update SCADA metadata if status changes

* add changelog entry
2023-01-04 11:09:51 -05:00
Max Bowsher c4e8e22963
Remove generic_mount_paths field (#18558)
PR #17926 already deleted the implementation of the
`generic_mount_paths` field so it needs to be removed from the declared
fields of the path too, so help and OpenAPI isn't misleading.
2023-01-03 19:14:29 -05:00
Max Bowsher 4052c785ea
Remove unreachable code (#18576)
I happened to spot that the `TemplateError` type is never instantiated.
Therefore delete it, and code referencing it.
2023-01-03 09:02:01 -05:00
akshya96 4126060d88
Prevent Brute Forcing: Create api endpoint to unlock users (#18279)
* code changes for unlock

* add test

* adding sys help

* adding sys help

* updating unlock user function

* edit test

* add changelog

* syshelp

* adding open api response definition

* removing response fields

* change path name
2022-12-19 14:24:42 -08:00
Josh Black cd7d6d5761
De-duplicate namespaces when historical and current month data are mixed (#18452)
* De-duplicate namespaces when historical and current month data are mixed

* add changelog
2022-12-16 16:02:42 -08:00
Scott Miller 53e73ceba2
Use a cleaner worker pattern (#18422) 2022-12-16 11:35:24 -06:00
divyaac f8ad8bc5a5
OSS PR for Config Changes PR (#18418)
* OSS PR for Config Changes PR

* Edited tests

* typo

* Added changelog

* Remove changelog
2022-12-15 12:19:19 -08:00
Nick Cabatoff 429916c135
Prevent panics in expiration invalidation, and make some changes for testing (#18401) 2022-12-15 18:09:36 +00:00
Mike Palmiotto cb3406b1eb
plugins: Handle mount/enable for shadowed builtins (#17879)
* Allow mounting external plugins with same name/type as deprecated builtins
* Add some go tests for deprecation status handling
* Move timestamp storage to post-unseal
* Add upgrade-aware deprecation shutdown and tests
2022-12-14 13:06:33 -05:00
Mike Palmiotto 809a04c8b4
core: Make shutdownDoneCh atomic (#18358)
When issuing a core.Shutdown(), it is common to background the shutdown
request. This allows Vault to continue cleaning up, mainly to release
the stateLock. This allows the shutdown to complete, but is inherently
racy, so the core.shutdownDoneCh needs to be made atomic.
2022-12-14 15:59:11 +00:00
Scott Miller 25bff579ea
Use a small pool of workers to run postUnsealFuncs in parallel (#18244)
* Initial worker pool

* Run postUnsealFuncs in parallel

* Use the old logic for P=1

* changelog

* Use a CPU count relative worker pool

* Update vault/core.go

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Done must be called once per postUnsealFunc

* Defer is overkill

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2022-12-12 17:07:53 -06:00
Chris Capurso 42f36605c7
use meta and link_control from link proto (#18290) 2022-12-09 12:35:06 -05:00
Chris Capurso c81485d5cd
copy over link_control and meta to link proto module (#18285) 2022-12-09 11:57:35 -05:00
Violet Hynes 176c149a38
VAULT-8336 Fix default rate limit paths (#18273)
* VAULT-8336 Fix default rate limit paths

* VAULT-8336 changelog
2022-12-09 08:49:17 -05:00
Chris Capurso 4dc5155c5f
Link OSS (#18228)
* add Link config, init, and capabilities

* add node status proto

* bump protoc version to 3.21.9

* make proto

* adding link tests

* remove wrapped link

* add changelog entry

* update changelog entry
2022-12-08 15:02:18 -05:00
Anton Averchenkov 493040d147
Add mount path into the default generated openapi.json spec (UI) (#17926) 2022-12-08 12:15:54 -05:00
Nick Cabatoff 1b745aef58
Prevent autopilot from demoting voters when they join a 2nd time (#18263) 2022-12-07 14:17:45 -05:00
Nick Cabatoff 342b61984a
Move version out of SDK. (#14229)
Move version out of SDK.  For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString.  That will have to wait until we've removed uses of useragent.String from all builtins.
2022-12-07 13:29:51 -05:00
akshya96 1801f09c6a
Vault 8307 user lockout workflow oss (#17951)
* adding oss file changes

* check disabled and read values from config

* isUserLocked, getUserLockout Configurations, check user lock before login and return error

* remove stale entry from storage during read

* added failed login process workflow

* success workflow updated

* user lockouts external tests

* changing update to support delete

* provide access to alias look ahead function

* adding path alias lookahead

* adding tests

* added changelog

* added comments

* adding changes from ent branch

* adding lock to UpdateUserFailedLoginInfo

* fix return default bug
2022-12-06 17:22:46 -08:00
Mike Palmiotto ea41e62e83
plugins: Mount missing plugin entries and skip loading (#18189)
* Skip plugin startup for missing plugins
* Skip secrets startup for missing plugins
* Add changelog for bugfix
* Make plugin handling on unseal version-aware
* Update plugin lazy-load logic/comments for readability
* Add register/mount/deregister/seal/unseal go test
* Consolidate lazy mount logic to prevent inconsistencies

Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
2022-12-02 13:16:31 -05:00
Tom Proctor 05aeab2752
Fix plugin list API when audit logging enabled (#18173)
* Add test that fails due to audit log panic
* Rebuild VersionedPlugin as map of primitive types before adding to response
* Changelog
* Fix casting in external plugin tests
2022-12-01 10:44:44 +00:00