Commit graph

2392 commits

Author SHA1 Message Date
Ellie 08ef61cc00
add error message when trying to rotate mssql root without password in configuration (#19103)
* add error message when trying to rotate mssql root without password in configuration

* add changelog
2023-02-13 07:31:13 -05:00
Kit Haines 14adb3b825
Telemetry Metrics Configuration. (#18186)
* Telemetry Metrics Configuration.

* Err Shadowing Fix (woah, semgrep is cool).

* Fix TestBackend_RevokePlusTidy_Intermediate

* Add Changelog.

* Fix memory leak.  Code cleanup as suggested by Steve.

* Turn off metrics by default, breaking-change.

* Show on tidy-status before start-up.

* Fix tests

* make fmt

* Add emit metrics to periodicFunc

* Test not delivering unavailable metrics + fix.

* Better error message.

* Fixing the false-error bug.

* make fmt.

* Try to fix race issue, remove confusing comments.

* Switch metric counter variables to an atomic.Uint32

 - Switch the metric counter variables to an atomic variable type
   so that we are forced to properly load/store values to it

* Fix race-issue better by trying until the metric is sunk.

* make fmt.

* empty commit to retrigger non-race tests that all pass locally

---------

Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
2023-02-10 21:31:56 +00:00
Michael Dempsey 1582b743aa
Add default to allowed values for algorithm_signer (#17894)
* Add default to allowed values for algorithm_signer

* Add possible values for algorithm signer in ui
2023-02-09 13:03:53 -05:00
Steven Clark 720ab09feb
Add a comment around why we are grabbing a lock to update an atomic boolean (#19087) 2023-02-09 09:12:37 -05:00
Steven Clark a4aee2fc95
Revert URL encoding OCSP GET fix #18938 (#19037)
- This fix was incorrect as now the tests and program are double
   URL encoding the OCSP GET requests, so the base64 + characters
   when using Vault proper are becoming space characters.
2023-02-08 11:24:26 -05:00
miagilepner e873932bce
VAULT-8436 remove <-time.After statements in for loops (#18818)
* replace time.After with ticker in loops

* add semgrep rule

* update to use timers

* remove stop
2023-02-06 17:49:01 +01:00
Nick Cabatoff 53afd2627b
Make API not depend on SDK (#18962) 2023-02-06 09:41:56 -05:00
Steven Clark 9e9d5d5645
Use the unified CRL on local CRL paths if UnifiedCRLOnExistingPaths is set (#18989)
* Use the unified CRL on legacy CRL paths if UnifiedCRLOnExistingPaths is set

 - If the crl configuration option unified_crl_on_existing_paths is set
   to true along with the unified_crl feature, provide the unified crl
   on the existing CRL paths.
 - Added some test helpers to help debugging, they are being used by
   the ENT test that validates this feature.

* Rename method to shouldLocalPathsUseUnified
2023-02-03 14:38:36 -05:00
Alexander Scheel b69055175a
Use UTC for leaf exceeding CA's notAfter (#18984)
* Use UTC for leaf exceeding CA's notAfter

When generating a leaf which exceeds the CA's validity period, Vault's
error message was confusing as the leaf would use the server's time
zone, but the CA's notAfter date would use UTC. This could cause
user confusion as the leaf's expiry might look before the latter, due
to using different time zones. E.g.:

> cannot satisfy request, as TTL would result in notAfter
> 2023-03-06T16:41:09.757694-08:00 that is beyond the expiration of
> the CA certificate at 2023-03-07T00:29:52Z

Consistently use UTC for this instead.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-02-03 17:00:42 +00:00
Steven Clark 449a0a68f5
Fix race accessing b.crls within cert auth (#18945)
* Fix race accessing b.crls within cert auth

 - Discovered by CircleCI the pathLogin, pathLoginRenew paths access
   and reloads the b.crls member variable without a lock.
 - Also discovered that pathLoginResolveRole never populated an empty
   b.crls before usage within b.verifyCredentials

* Add cl

* Misc cleanup

 - Introduce a login path wrapper instead of repeating in all the
   various login methods the crl reloading
 - Cleanup updatedConfig, never returned an error and nothing looked at
   the error returned
 - Make the test within TestCRLFetch a little less timing sensitive as
   I was able to trigger a failure due to my machine taking more than
   150ms to load the new CRL
2023-02-01 16:23:06 -05:00
Steven Clark baf66ff56e
Apply URL encoding/unencoding to OCSP Get requests (#18938)
* Apply URL encoding/unencoding to OCSP Get requests

 - Missed this during development and sadly the unit tests were written
   at a level that did not expose this issue originally, there are
   certain combinations of issuer cert + serial that lead to base64
   data containing a '/' which will lead to the OCSP handler not getting
   the full parameter.
 - Do as the spec says, this should be treated as url-encoded data.

* Add cl

* Add higher level PKI OCSP GET/POST tests

* Rename PKI ocsp files to path_ocsp to follow naming conventions

* make fmt
2023-02-01 11:03:43 -05:00
Alexander Scheel 5d17f9b142
Allow cleanup ssh dynamic keys host keys (#18939)
* Add ability to clean up host keys for dynamic keys

This adds a new endpoint, tidy/dynamic-keys that removes any stale host
keys still present on the mount. This does not clean up any pending
dynamic key leases and will not remove these keys from systems with
authorized hosts entries created by Vault.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-02-01 15:09:16 +00:00
Steven Clark 81689ae63d
Refactor an ENT test helper back to OSS to wait for updated CRLs within PKI tests (#18933)
- We had two separate helpers that were very close, one only in ENT, so unify them here in OSS.
2023-02-01 08:47:26 -05:00
Steven Clark 8d47ad792f
PKI Unified CRL/OCSP apis should be ent only (#18913)
* PKI Unified CRL/OCSP apis should be ent only

 - Do not enable any of the unified crl/ocsp related apis on OSS.

* Rollback refactoring of pathFetchCRLViaCertPath

 - As pointed out in the PR, this method isn't actually being used at
   the moment with the <serial> handler, pathFetchValid, matching
   everything under the cert/XXXX path.

* Fix schema for ent/oss diff

 - Define the OSS vs ENT urls we want to see within the schema
   definition even if they aren't really going to be used in the end.
2023-02-01 08:35:14 -05:00
Nick Cabatoff c2b222a11a
Vault test cluster helper refactorings, mostly audit related (#18928)
* Move some test helper stuff from the vault package to a new helper/testhelpers/corehelpers package.  Consolidate on a single "noop audit" implementation.
2023-02-01 08:33:16 -05:00
Alexander Scheel 881ae5a303
Remove dynamic keys from SSH Secrets Engine (#18874)
* Remove dynamic keys from SSH Secrets Engine

This removes the functionality of Vault creating keys and adding them to
the authorized keys file on hosts.

This functionality has been deprecated since Vault version 0.7.2.

The preferred alternative is to use the SSH CA method, which also allows
key generation but places limits on TTL and doesn't require Vault reach
out to provision each key on the specified host, making it much more
secure.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove dynamic ssh references from documentation

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove dynamic key secret type entirely

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Clarify changelog language

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add removal notice to the website

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-31 16:02:22 -05:00
Nick Cabatoff db41c5319d
TestClusterCore's TLSConfig becomes a method and does a Clone. (#18914) 2023-01-31 11:05:16 -05:00
Steven Clark b737777f15
Address pki::TestAutoRebuild flakiness (#18903)
* Address pki::TestAutoRebuild flakiness

 - Wait for a CRL change before progressing to the next step after
   we change configuration. Prior to this we would be racing against
   the CRL reloading from the configuration change.
2023-01-30 21:38:38 +00:00
akshya96 16ce923ddb
Brute forcing unlock user bug (#18890)
* brute forcing unlock user bug

* add changelog

* fix changelog
2023-01-30 13:06:10 -08:00
Matt Schultz 20c1645329
Add some managed key build tags to support incoming enterprise feature work. (#18902) 2023-01-30 14:57:01 -06:00
Alexander Scheel 2b9a8c6c49
Fix race in tidy status with cert counting (#18899)
* Read total cert counts with atomic.LoadUint32(...)

When generating the tidy status, we read the values of two backend
atomics, b.certCount and b.revokedCertCount, without using the atomic
load operation. This resulted in a data race when the status was read
at the same time as an on-going tidy operation:

    WARNING: DATA RACE
    Write at 0x00c00c77680c by goroutine 90522:
      sync/atomic.AddInt32()
          /usr/local/go/src/runtime/race_amd64.s:281 +0xb
      sync/atomic.AddUint32()
          <autogenerated>:1 +0x1a
      github.com/hashicorp/vault/builtin/logical/pki.(*backend).tidyStatusIncRevokedCertCount()
          /home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:1236 +0x107
      github.com/hashicorp/vault/builtin/logical/pki.(*backend).doTidyRevocationStore()
          /home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:525 +0x1404
      github.com/hashicorp/vault/builtin/logical/pki.(*backend).startTidyOperation.func1.1()
          /home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:290 +0x1a4
      github.com/hashicorp/vault/builtin/logical/pki.(*backend).startTidyOperation.func1()
          /home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:342 +0x278

    Previous read at 0x00c00c77680c by goroutine 90528:
      reflect.Value.Uint()
          /usr/local/go/src/reflect/value.go:2584 +0x195
      encoding/json.uintEncoder()
          /usr/local/go/src/encoding/json/encode.go:562 +0x45
      encoding/json.ptrEncoder.encode()
          /usr/local/go/src/encoding/json/encode.go:944 +0x3c2
      encoding/json.ptrEncoder.encode-fm()
          <autogenerated>:1 +0x90
      encoding/json.(*encodeState).reflectValue()
          /usr/local/go/src/encoding/json/encode.go:359 +0x88
      encoding/json.interfaceEncoder()
          /usr/local/go/src/encoding/json/encode.go:715 +0x17b
      encoding/json.mapEncoder.encode()
          /usr/local/go/src/encoding/json/encode.go:813 +0x854
      ... more stack trace pointing into JSON encoding and http
      handler...

In particular, because the tidy status was directly reading the uint
value without resorting to the atomic side, the JSON serialization could
race with a later atomic update.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Also use atomic load in tests

Because no tidy operation is running here, it should be safe to read the
pointed value directly, but use the safer atomic.Load for consistency.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-30 14:13:40 -05:00
Rachel Culpepper c151d5fd58
Vault-11713: Add the ability for transit to use managed keys (#18844)
* add the ability for transit to use managed keys

* add fields for rotate

* reorganize managed keys for better error messaging

* add aad for managed keys

* change encrypting key interface

* fix import error

* fix fmt errors

* move managed key rotation to a separate function

* fix fmt issue

* fix import error

* fix go.mod issue

* fix go.mod

* remove extra rotate param

* fix locking issue in rotation

* fix errors in policy tests

* move managed key params to fit existing policy functions

* fix sign and verify

* fix policy tests

* fix test error
2023-01-27 13:39:58 -06:00
Nick Cabatoff 9ca78845b7
Proposed more streamlined approach to validating schema responses. (#18865) 2023-01-27 14:34:55 -05:00
Alexander Scheel 419a92a632
Move cert auth backend setup into initialize (#18885)
* Move cert auth backend setup into initialize

In further review with new understanding after #18244, loading
configuration and CRLs within the backend's initialize function is the
ideal approach: Factory construction is strictly serial, resulting in
backend initialization blocking until config and CRLs are loaded.
By using an InitializeFunc(...), we delay loading until after all
backends are constructed (either right on startup in 1.12+, else during
the initial PeriodicFunc(...) invocation on 1.11 and earlier).

We also invoke initialize automatically on test Factory construction.

Resolves: #17847

Co-authored-by: valli_0x <personallune@mail.ru>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: valli_0x <personallune@mail.ru>
2023-01-27 17:42:13 +00:00
Steven Clark 235746b98d
Add t.Helper() to various PKI test helper methods (#18881)
- This has been done to help diagnose errors in the future so that
   we get the callers in the trace's when we fail and not just the
   helper's trace output.
2023-01-27 17:29:11 +00:00
Alexander Scheel a2c84ef236
Allow unification of revocations on other clusters (#18873)
* Allow unification of revocations on other clusters

If a BYOC revocation occurred on cluster A, while the cert was initially
issued and stored on cluster B, we need to use the invalidation on the
unified entry to detect this: the revocation queues only work for
non-PoP, non-BYOC serial only revocations and thus this BYOC would be
immediately accepted on cluster A. By checking all other incoming
revocations for duplicates on a given cluster, we can ensure that
unified revocation is consistent across clusters.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use time-of-use locking for global revocation processing

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-27 16:34:04 +00:00
Steven Clark d12534c2bd
Unified revocation migration code (#18866)
* Unified revocation migration code

 - Add a periodic function that will list the local revocations
   and if any are missing from the unified revocation area will
   force a write to the unified revocation folder/remote instance.

* PR Feedback

 - Do not transfer expired certificates to unified space from local
 - Move new periodic code into a periodic.go file
 - Add a flag so we only run this stuff once if all is good, with
   a force flag if we encounter errors or if unified_crl is toggled
   on

* PR feedback take 2
2023-01-27 15:49:20 +00:00
Steven Clark af5bd7a2d7
Return a detailed list response for unified-revoked API endpoint (#18862)
- Return a detailed reponse within the list api that an end-user can
   use to determine what clusters revoked the certificate on from the
   pki/certs/unified-revoked LIST api.

 - Return colon delimited serial numbers from the certs/revocation-queue
   LIST api
2023-01-26 19:12:35 +00:00
Alexander Scheel ea539070c6
Add tidy of cross-cluster revoked storage (#18860)
* Add new tidy operation for cross revoked certs

This operation allows tidying of the cross-cluster revocation storage.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Fix missing cancels, status values

Previous additions to tidy didn't have enough cancel operations and left
out some new values from the status operation.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-26 13:30:57 -05:00
Alexander Scheel cc57a0f73e
Clarify key bits for ssh (#18854)
* Clarify error on due to unsupported EC key bits

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove documentation about unsupported EC/224

Resolves: #18843

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-26 10:14:05 -05:00
Steven Clark 57ab764d68
Store the unified crl in a path that is not cluster local (#18848)
- I missed this in the original review, that we were storing the
   unified-crl in a cluster-local storage area so none of the other
   hosts would receive it.
 - Discovered while writing unit tests, the main cluster had the unified
   crl but the other clusters would return an empty response
2023-01-26 09:03:02 -05:00
Kit Haines 27be887bfd
Vault 9406 enablement certs need userid handling in role (#18397)
* The fields.

* UserID set, add to certificate

* Changelog.

* Fix test (set default).

* Add UserID constant to certutil, revert extension changes

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add user_ids as field for leaf signing

Presumably, this isn't necessary for CAs, given that CAs probably don't
have a user ID corresponding to them.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Support setting multiple user_ids in Subject

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow any User ID with sign-verbatim

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests for User IDs in PKI

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add docs about user_ids, allowed_user_ids

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-25 13:13:54 -05:00
Anton Averchenkov 5a6092f8ab
Add approle's remaining response schema definitions (#18772) 2023-01-24 13:12:41 -05:00
Alexander Scheel 7850f0e05e
Ensure unified_crl requires auto_rebuild (#18819)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-24 15:30:50 +00:00
Steven Clark 43f679bf05
Deny setting cross-cluster revocation features on local mounts (#18801)
- It does not make sense to allow operators to enable the cross-cluster
   revocation features on local mounts as they will never have a
   corresponding mount on the other cluster.
2023-01-23 20:21:39 +00:00
Alexander Scheel 15ae00d147
Add unified crl building (#18792)
* Add unified CRL config storage helpers

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add support to build unified CRLs

This allows us to build unified versions of both the complete and delta
CRLs. This mostly involved creating a new variant of the
unified-specific CRL builder, fetching certs from each cluster's storage
space.

Unlike OCSP, here we do not unify the node's local storage with the
cross-cluster storage: this node is the active of the performance
primary, so writes to unified storage happen exactly the same as
writes to cluster-local storage, meaning the two are always in
sync. Other performance secondaries do not rebuild the CRL, and hence
the out-of-sync avoidance that we'd like to solve with the OCSP
responder is not necessary to solve here.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add ability to fetch unified CRLs

This adds to the path-fetch APIs the ability to return the unified CRLs.
We update the If-Modified-Since infrastructure to support querying the
unified CRL specific data and fetchCertBySerial to support all unified
variants. This works for both the default/global fetch APIs and the
issuer-specific fetch APIs.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rebuild CRLs on unified status changes

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Handle rebuilding CRLs due to either changing

This allows detecting if the Delta CRL needs to be rebuilt because
either the local or the unified CRL needs to be rebuilt. We never
trigger rebuilding the unified delta on a non-primary cluster.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Ensure serials aren't added to unified CRL twice

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 19:17:34 +00:00
Alexander Scheel 1c85d611e2
Write delta WAL entries for unified CRLs (#18785)
* Write delta WAL entries for unified CRLs

When we'd ordinarily write delta WALs for local CRLs, we also need to
populate the cross-cluster delta WAL. This could cause revocation to
appear to fail if the two clusters are disconnected, but notably regular
cross-cluster revocation would also fail.

Notably, this commit also changes us to not write Delta WALs when Delta
CRLs is disabled (versus previously doing it when auto rebuild is
enabled in case Delta CRLs were later asked for), and instead,
triggering rebuilding a complete CRL so we don't need up-to-date Delta
WAL info.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Update IMS test for forced CRL rebuilds

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 16:56:08 +00:00
Alexander Scheel ec7502aa44
More cross cluster queue tweaks (#18789)
* Move comment about perf-primary only invalidation

Also remove noisy debug log.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove more noisy log statements during queue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Skip revocation entries from our current cluster

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add locking and comment about tidying revoke queue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Switch to time.Since for tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor tidyStatuses into path_tidy.go

Leaving these in backend.go often causes us to miss adding useful values
to tidyStatus when we add a new config parameter.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Track the number of deleted revocation request

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow tidy to remove confirmed revocation requests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add missing field to tidy test

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 16:52:38 +00:00
Steven Clark d0453ed40b
Add unified storage support to OCSP handler (#18788) 2023-01-23 15:49:07 +00:00
Steven Clark f3ce351e01
Add support for revoke by serial number to update the unified CRL (#18786) 2023-01-23 10:22:10 -05:00
Alexander Scheel b3dc380c82
Add cross-cluster revocation queues for PKI (#18784)
* Add global, cross-cluster revocation queue to PKI

This adds a global, cross-cluster replicated revocation queue, allowing
operators to revoke certificates by serial number across any cluster. We
don't support revoking with private key (PoP) in the initial
implementation.

In particular, building on the PBPWF work, we add a special storage
location for handling non-local revocations which gets replicated up to
the active, primary cluster node and back down to all secondary PR
clusters. These then check the pending revocation entry and revoke the
serial locally if it exists, writing a cross-cluster confirmation entry.

Listing capabilities are present under pki/certs/revocation-queue,
allowing operators to see which certs are present. However, a future
improvement to the tidy subsystem will allow automatic cleanup of stale
entries.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow tidying revocation queue entries

No manual operator control of revocation queue entries are allowed.
However, entries are stored with their request time, allowing tidy to,
after a suitable safety buffer, remove these unconfirmed and presumably
invalid requests.

Notably, when a cluster goes offline, it will be unable to process
cross-cluster revocations for certificates it holds. If tidy runs,
potentially valid revocations may be removed. However, it is up to the
administrator to ensure the tidy window is sufficiently long that any
required maintenance is done (or, prior to maintenance when an issue is
first noticed, tidy is temporarily disabled).

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Only allow enabling global revocation queue on Vault Enterprise

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use a locking queue to handle revocation requests

This queue attempts to guarantee that PKI's invalidateFunc won't have
to wait long to execute: by locking only around access to the queue
proper, and internally using a list, we minimize the time spent locked,
waiting for queue accesses.

Previously, we held a lock during tidy and processing that would've
prevented us from processing invalidateFunc calls.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* use_global_queue->cross_cluster_revocation

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Grab revocation storage lock when processing queue

We need to grab the storage lock as we'll actively be revoking new
certificates in the revocation queue. This ensures nobody else is
competing for storage access, across periodic funcs, new revocations,
and tidy operations.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Fix expected tidy status test

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow probing RollbackManager directly in tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Address review feedback on revocationQueue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add more cancel checks, fix starting manual tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 09:29:27 -05:00
Alexander Scheel 3adfed1af8
Add missing space in PKI error (#18778)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-20 11:02:17 -05:00
Alexander Scheel 6b4f770de9
Refactor CRL Building for unified CRLs (#18754)
* Refactor CRL building into separate functions

This will allow us to add the ability to add and build a unified CRL
across all clusters, reusing logic that is common to both, but letting
each have their own certificate lists.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rename localCRLConfigEntry->internalCRLConfigEntry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rename Delta WALs to Local Delta WALs

This adds clarity that we'll have a separate local and remote Delta CRL
and WALs for each.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-18 15:05:14 -05:00
Anton Averchenkov 6ae09f3074
Add AppRole response schema validation tests (#18636)
This PR modifies every test in `builtin/credentials/approle/path_role_test.go` with new validation checks to ensure that approle/path_role  successful responses align with the declared response schema.

It also introduces a test helper in `sdk/helper/testhelpers`:

```go
func FindResponseSchema(t *testing.T, ...)
```

This test helper will be useful for all plugins that require similar response schema validation in tests.

### Background

This PR is part of the ongoing work to add structured responses in Vault OpenAPI (VLT-234)
2023-01-13 15:23:36 -05:00
Steven Clark e0e957731b
Refactor the PKI revocation handler to prep for unified revocation (#18685)
* Rename revokeCert variable to identify serial number formatting

* Refactor out lease specific behavior out of revokeCert

 - Isolate the specific behavior regarding revoking lease specific
   certificates outside of the revokeCert function and into the only
   caller that leveraged used it.
 - This allows us to simplify revokeCert a little bit and keeps the
   function purely about revoking a certificate

* Within revokeCert short circuit the already revoked use-case

 - Make the function a little easier to process by exiting early
   if the certificate has already been revoked.

* Do not load certificates from storage multiple times during revocation

 - Isolate the loading of a certificate and parsing of a certificate
   into a single attempt, either when provided the certificate for BYOC
   revocation or strictly from storage for the other revocation types.

* With BYOC write certificate entry using dashes not the legacy colon char
2023-01-13 10:31:03 -05:00
Max Bowsher d1f2b101b5
Add option 'elide_list_responses' to audit backends (#18128)
This PR relates to a feature request logged through HashiCorp commercial
support.

Vault lacks pagination in its APIs. As a result, certain list operations
can return **very** large responses.  The user's chosen audit sinks may
experience difficulty consuming audit records that swell to tens of
megabytes of JSON.

In our case, one of the systems consuming audit log data could not cope,
and failed.

The responses of list operations are typically not very interesting, as
they are mostly lists of keys, or, even when they include a "key_info"
field, are not returning confidential information. They become even less
interesting once HMAC-ed by the audit system.

Some example Vault "list" operations that are prone to becoming very
large in an active Vault installation are:

    auth/token/accessors/
    identity/entity/id/
    identity/entity-alias/id/
    pki/certs/

In response, I've coded a new option that can be applied to audit
backends, `elide_list_responses`. When enabled, response data is elided
from audit logs, only when the operation type is "list".

For added safety, the elision only applies to the "keys" and "key_info"
fields within the response data - these are conventionally the only
fields present in a list response - see logical.ListResponse, and
logical.ListResponseWithInfo. However, other fields are technically
possible if a plugin author writes unusual code, and these will be
preserved in the audit log even with this option enabled.

The elision replaces the values of the "keys" and "key_info" fields with
an integer count of the number of entries. This allows even the elided
audit logs to still be useful for answering questions like "Was any data
returned?" or "How many records were listed?".
2023-01-11 16:15:52 -05:00
Anton Averchenkov 5f7e95fcb9
Fix AppRole / path_role response schema (#18637) 2023-01-11 12:15:29 -05:00
Alexander Scheel 44c3b736bf
Allow tidy to backup legacy CA bundles (#18645)
* Allow tidy to backup legacy CA bundles

With the new tidy_move_legacy_ca_bundle option, we'll use tidy to move
the legacy CA bundle from /config/ca_bundle to /config/ca_bundle.bak.
This does two things:

 1. Removes ca_bundle from the hot-path of initialization after initial
    migration has completed. Because this entry is seal wrapped, this
    may result in performance improvements.
 2. Allows recovery of this value in the event of some other failure
    with migration.

Notably, this cannot occur during migration in the unlikely (and largely
unsupported) case that the operator immediately downgrades to Vault
<1.11.x. Thus, we reuse issuer_safety_buffer; while potentially long,
tidy can always be run manually with a shorter buffer (and only this
flag) to manually move the bundle if necessary.

In the event of needing to recover or undo this operation, it is
sufficient to use sys/raw to read the backed up value and subsequently
write it to its old path (/config/ca_bundle).

The new entry remains seal wrapped, but otherwise isn't used within the
code and so has better performance characteristics.

Performing a fat deletion (DELETE /root) will again remove the backup
like the old legacy bundle, preserving its wipe characteristics.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation about new tidy parameter

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests for migration scenarios

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Clean up time comparisons

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-11 12:12:53 -05:00
Alexander Scheel a18187c643
Correctly distinguish empty issuer names in PKI (#18466)
* Correctly distinguish empty issuer names

When using client.Logical().JSONMergePatch(...) with an empty issuer
name, patch incorrectly reports:

> issuer name contained invalid characters

In this case, both the error in getIssuerName(...) is incorrect and
patch should allow setting an empty issuer name explicitly.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-10 10:04:30 -05:00
Alexander Scheel 38de21468e
Add cluster_aia_path templating variable (#18493)
* Add cluster_aia_path templating variable

Per discussion with maxb, allow using a non-Vault distribution point
which may use an insecure transport for RFC 5280 compliance.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Address feedback from Max

Co-authored-by: Max Bowsher <maxbowsher@gmail.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Max Bowsher <maxbowsher@gmail.com>
2023-01-10 09:51:37 -05:00