- This fix was incorrect as now the tests and program are double
URL encoding the OCSP GET requests, so the base64 + characters
when using Vault proper are becoming space characters.
* Use the unified CRL on legacy CRL paths if UnifiedCRLOnExistingPaths is set
- If the crl configuration option unified_crl_on_existing_paths is set
to true along with the unified_crl feature, provide the unified crl
on the existing CRL paths.
- Added some test helpers to help debugging, they are being used by
the ENT test that validates this feature.
* Rename method to shouldLocalPathsUseUnified
* Use UTC for leaf exceeding CA's notAfter
When generating a leaf which exceeds the CA's validity period, Vault's
error message was confusing as the leaf would use the server's time
zone, but the CA's notAfter date would use UTC. This could cause
user confusion as the leaf's expiry might look before the latter, due
to using different time zones. E.g.:
> cannot satisfy request, as TTL would result in notAfter
> 2023-03-06T16:41:09.757694-08:00 that is beyond the expiration of
> the CA certificate at 2023-03-07T00:29:52Z
Consistently use UTC for this instead.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix race accessing b.crls within cert auth
- Discovered by CircleCI the pathLogin, pathLoginRenew paths access
and reloads the b.crls member variable without a lock.
- Also discovered that pathLoginResolveRole never populated an empty
b.crls before usage within b.verifyCredentials
* Add cl
* Misc cleanup
- Introduce a login path wrapper instead of repeating in all the
various login methods the crl reloading
- Cleanup updatedConfig, never returned an error and nothing looked at
the error returned
- Make the test within TestCRLFetch a little less timing sensitive as
I was able to trigger a failure due to my machine taking more than
150ms to load the new CRL
* Apply URL encoding/unencoding to OCSP Get requests
- Missed this during development and sadly the unit tests were written
at a level that did not expose this issue originally, there are
certain combinations of issuer cert + serial that lead to base64
data containing a '/' which will lead to the OCSP handler not getting
the full parameter.
- Do as the spec says, this should be treated as url-encoded data.
* Add cl
* Add higher level PKI OCSP GET/POST tests
* Rename PKI ocsp files to path_ocsp to follow naming conventions
* make fmt
* Add ability to clean up host keys for dynamic keys
This adds a new endpoint, tidy/dynamic-keys that removes any stale host
keys still present on the mount. This does not clean up any pending
dynamic key leases and will not remove these keys from systems with
authorized hosts entries created by Vault.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* PKI Unified CRL/OCSP apis should be ent only
- Do not enable any of the unified crl/ocsp related apis on OSS.
* Rollback refactoring of pathFetchCRLViaCertPath
- As pointed out in the PR, this method isn't actually being used at
the moment with the <serial> handler, pathFetchValid, matching
everything under the cert/XXXX path.
* Fix schema for ent/oss diff
- Define the OSS vs ENT urls we want to see within the schema
definition even if they aren't really going to be used in the end.
* Move some test helper stuff from the vault package to a new helper/testhelpers/corehelpers package. Consolidate on a single "noop audit" implementation.
* Remove dynamic keys from SSH Secrets Engine
This removes the functionality of Vault creating keys and adding them to
the authorized keys file on hosts.
This functionality has been deprecated since Vault version 0.7.2.
The preferred alternative is to use the SSH CA method, which also allows
key generation but places limits on TTL and doesn't require Vault reach
out to provision each key on the specified host, making it much more
secure.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove dynamic ssh references from documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove dynamic key secret type entirely
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify changelog language
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add removal notice to the website
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address pki::TestAutoRebuild flakiness
- Wait for a CRL change before progressing to the next step after
we change configuration. Prior to this we would be racing against
the CRL reloading from the configuration change.
* Read total cert counts with atomic.LoadUint32(...)
When generating the tidy status, we read the values of two backend
atomics, b.certCount and b.revokedCertCount, without using the atomic
load operation. This resulted in a data race when the status was read
at the same time as an on-going tidy operation:
WARNING: DATA RACE
Write at 0x00c00c77680c by goroutine 90522:
sync/atomic.AddInt32()
/usr/local/go/src/runtime/race_amd64.s:281 +0xb
sync/atomic.AddUint32()
<autogenerated>:1 +0x1a
github.com/hashicorp/vault/builtin/logical/pki.(*backend).tidyStatusIncRevokedCertCount()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:1236 +0x107
github.com/hashicorp/vault/builtin/logical/pki.(*backend).doTidyRevocationStore()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:525 +0x1404
github.com/hashicorp/vault/builtin/logical/pki.(*backend).startTidyOperation.func1.1()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:290 +0x1a4
github.com/hashicorp/vault/builtin/logical/pki.(*backend).startTidyOperation.func1()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:342 +0x278
Previous read at 0x00c00c77680c by goroutine 90528:
reflect.Value.Uint()
/usr/local/go/src/reflect/value.go:2584 +0x195
encoding/json.uintEncoder()
/usr/local/go/src/encoding/json/encode.go:562 +0x45
encoding/json.ptrEncoder.encode()
/usr/local/go/src/encoding/json/encode.go:944 +0x3c2
encoding/json.ptrEncoder.encode-fm()
<autogenerated>:1 +0x90
encoding/json.(*encodeState).reflectValue()
/usr/local/go/src/encoding/json/encode.go:359 +0x88
encoding/json.interfaceEncoder()
/usr/local/go/src/encoding/json/encode.go:715 +0x17b
encoding/json.mapEncoder.encode()
/usr/local/go/src/encoding/json/encode.go:813 +0x854
... more stack trace pointing into JSON encoding and http
handler...
In particular, because the tidy status was directly reading the uint
value without resorting to the atomic side, the JSON serialization could
race with a later atomic update.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Also use atomic load in tests
Because no tidy operation is running here, it should be safe to read the
pointed value directly, but use the safer atomic.Load for consistency.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move cert auth backend setup into initialize
In further review with new understanding after #18244, loading
configuration and CRLs within the backend's initialize function is the
ideal approach: Factory construction is strictly serial, resulting in
backend initialization blocking until config and CRLs are loaded.
By using an InitializeFunc(...), we delay loading until after all
backends are constructed (either right on startup in 1.12+, else during
the initial PeriodicFunc(...) invocation on 1.11 and earlier).
We also invoke initialize automatically on test Factory construction.
Resolves: #17847
Co-authored-by: valli_0x <personallune@mail.ru>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: valli_0x <personallune@mail.ru>
- This has been done to help diagnose errors in the future so that
we get the callers in the trace's when we fail and not just the
helper's trace output.
* Allow unification of revocations on other clusters
If a BYOC revocation occurred on cluster A, while the cert was initially
issued and stored on cluster B, we need to use the invalidation on the
unified entry to detect this: the revocation queues only work for
non-PoP, non-BYOC serial only revocations and thus this BYOC would be
immediately accepted on cluster A. By checking all other incoming
revocations for duplicates on a given cluster, we can ensure that
unified revocation is consistent across clusters.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use time-of-use locking for global revocation processing
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Unified revocation migration code
- Add a periodic function that will list the local revocations
and if any are missing from the unified revocation area will
force a write to the unified revocation folder/remote instance.
* PR Feedback
- Do not transfer expired certificates to unified space from local
- Move new periodic code into a periodic.go file
- Add a flag so we only run this stuff once if all is good, with
a force flag if we encounter errors or if unified_crl is toggled
on
* PR feedback take 2
- Return a detailed reponse within the list api that an end-user can
use to determine what clusters revoked the certificate on from the
pki/certs/unified-revoked LIST api.
- Return colon delimited serial numbers from the certs/revocation-queue
LIST api
* Add new tidy operation for cross revoked certs
This operation allows tidying of the cross-cluster revocation storage.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix missing cancels, status values
Previous additions to tidy didn't have enough cancel operations and left
out some new values from the status operation.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify error on due to unsupported EC key bits
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove documentation about unsupported EC/224
Resolves: #18843
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- I missed this in the original review, that we were storing the
unified-crl in a cluster-local storage area so none of the other
hosts would receive it.
- Discovered while writing unit tests, the main cluster had the unified
crl but the other clusters would return an empty response
* The fields.
* UserID set, add to certificate
* Changelog.
* Fix test (set default).
* Add UserID constant to certutil, revert extension changes
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add user_ids as field for leaf signing
Presumably, this isn't necessary for CAs, given that CAs probably don't
have a user ID corresponding to them.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Support setting multiple user_ids in Subject
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow any User ID with sign-verbatim
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for User IDs in PKI
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add docs about user_ids, allowed_user_ids
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
- It does not make sense to allow operators to enable the cross-cluster
revocation features on local mounts as they will never have a
corresponding mount on the other cluster.
* Add unified CRL config storage helpers
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support to build unified CRLs
This allows us to build unified versions of both the complete and delta
CRLs. This mostly involved creating a new variant of the
unified-specific CRL builder, fetching certs from each cluster's storage
space.
Unlike OCSP, here we do not unify the node's local storage with the
cross-cluster storage: this node is the active of the performance
primary, so writes to unified storage happen exactly the same as
writes to cluster-local storage, meaning the two are always in
sync. Other performance secondaries do not rebuild the CRL, and hence
the out-of-sync avoidance that we'd like to solve with the OCSP
responder is not necessary to solve here.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add ability to fetch unified CRLs
This adds to the path-fetch APIs the ability to return the unified CRLs.
We update the If-Modified-Since infrastructure to support querying the
unified CRL specific data and fetchCertBySerial to support all unified
variants. This works for both the default/global fetch APIs and the
issuer-specific fetch APIs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rebuild CRLs on unified status changes
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Handle rebuilding CRLs due to either changing
This allows detecting if the Delta CRL needs to be rebuilt because
either the local or the unified CRL needs to be rebuilt. We never
trigger rebuilding the unified delta on a non-primary cluster.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Ensure serials aren't added to unified CRL twice
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Write delta WAL entries for unified CRLs
When we'd ordinarily write delta WALs for local CRLs, we also need to
populate the cross-cluster delta WAL. This could cause revocation to
appear to fail if the two clusters are disconnected, but notably regular
cross-cluster revocation would also fail.
Notably, this commit also changes us to not write Delta WALs when Delta
CRLs is disabled (versus previously doing it when auto rebuild is
enabled in case Delta CRLs were later asked for), and instead,
triggering rebuilding a complete CRL so we don't need up-to-date Delta
WAL info.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update IMS test for forced CRL rebuilds
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move comment about perf-primary only invalidation
Also remove noisy debug log.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove more noisy log statements during queue
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Skip revocation entries from our current cluster
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add locking and comment about tidying revoke queue
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to time.Since for tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor tidyStatuses into path_tidy.go
Leaving these in backend.go often causes us to miss adding useful values
to tidyStatus when we add a new config parameter.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Track the number of deleted revocation request
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow tidy to remove confirmed revocation requests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing field to tidy test
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add global, cross-cluster revocation queue to PKI
This adds a global, cross-cluster replicated revocation queue, allowing
operators to revoke certificates by serial number across any cluster. We
don't support revoking with private key (PoP) in the initial
implementation.
In particular, building on the PBPWF work, we add a special storage
location for handling non-local revocations which gets replicated up to
the active, primary cluster node and back down to all secondary PR
clusters. These then check the pending revocation entry and revoke the
serial locally if it exists, writing a cross-cluster confirmation entry.
Listing capabilities are present under pki/certs/revocation-queue,
allowing operators to see which certs are present. However, a future
improvement to the tidy subsystem will allow automatic cleanup of stale
entries.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow tidying revocation queue entries
No manual operator control of revocation queue entries are allowed.
However, entries are stored with their request time, allowing tidy to,
after a suitable safety buffer, remove these unconfirmed and presumably
invalid requests.
Notably, when a cluster goes offline, it will be unable to process
cross-cluster revocations for certificates it holds. If tidy runs,
potentially valid revocations may be removed. However, it is up to the
administrator to ensure the tidy window is sufficiently long that any
required maintenance is done (or, prior to maintenance when an issue is
first noticed, tidy is temporarily disabled).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Only allow enabling global revocation queue on Vault Enterprise
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use a locking queue to handle revocation requests
This queue attempts to guarantee that PKI's invalidateFunc won't have
to wait long to execute: by locking only around access to the queue
proper, and internally using a list, we minimize the time spent locked,
waiting for queue accesses.
Previously, we held a lock during tidy and processing that would've
prevented us from processing invalidateFunc calls.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* use_global_queue->cross_cluster_revocation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Grab revocation storage lock when processing queue
We need to grab the storage lock as we'll actively be revoking new
certificates in the revocation queue. This ensures nobody else is
competing for storage access, across periodic funcs, new revocations,
and tidy operations.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix expected tidy status test
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow probing RollbackManager directly in tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address review feedback on revocationQueue
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add more cancel checks, fix starting manual tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor CRL building into separate functions
This will allow us to add the ability to add and build a unified CRL
across all clusters, reusing logic that is common to both, but letting
each have their own certificate lists.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rename localCRLConfigEntry->internalCRLConfigEntry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rename Delta WALs to Local Delta WALs
This adds clarity that we'll have a separate local and remote Delta CRL
and WALs for each.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
This PR modifies every test in `builtin/credentials/approle/path_role_test.go` with new validation checks to ensure that approle/path_role successful responses align with the declared response schema.
It also introduces a test helper in `sdk/helper/testhelpers`:
```go
func FindResponseSchema(t *testing.T, ...)
```
This test helper will be useful for all plugins that require similar response schema validation in tests.
### Background
This PR is part of the ongoing work to add structured responses in Vault OpenAPI (VLT-234)
* Rename revokeCert variable to identify serial number formatting
* Refactor out lease specific behavior out of revokeCert
- Isolate the specific behavior regarding revoking lease specific
certificates outside of the revokeCert function and into the only
caller that leveraged used it.
- This allows us to simplify revokeCert a little bit and keeps the
function purely about revoking a certificate
* Within revokeCert short circuit the already revoked use-case
- Make the function a little easier to process by exiting early
if the certificate has already been revoked.
* Do not load certificates from storage multiple times during revocation
- Isolate the loading of a certificate and parsing of a certificate
into a single attempt, either when provided the certificate for BYOC
revocation or strictly from storage for the other revocation types.
* With BYOC write certificate entry using dashes not the legacy colon char
This PR relates to a feature request logged through HashiCorp commercial
support.
Vault lacks pagination in its APIs. As a result, certain list operations
can return **very** large responses. The user's chosen audit sinks may
experience difficulty consuming audit records that swell to tens of
megabytes of JSON.
In our case, one of the systems consuming audit log data could not cope,
and failed.
The responses of list operations are typically not very interesting, as
they are mostly lists of keys, or, even when they include a "key_info"
field, are not returning confidential information. They become even less
interesting once HMAC-ed by the audit system.
Some example Vault "list" operations that are prone to becoming very
large in an active Vault installation are:
auth/token/accessors/
identity/entity/id/
identity/entity-alias/id/
pki/certs/
In response, I've coded a new option that can be applied to audit
backends, `elide_list_responses`. When enabled, response data is elided
from audit logs, only when the operation type is "list".
For added safety, the elision only applies to the "keys" and "key_info"
fields within the response data - these are conventionally the only
fields present in a list response - see logical.ListResponse, and
logical.ListResponseWithInfo. However, other fields are technically
possible if a plugin author writes unusual code, and these will be
preserved in the audit log even with this option enabled.
The elision replaces the values of the "keys" and "key_info" fields with
an integer count of the number of entries. This allows even the elided
audit logs to still be useful for answering questions like "Was any data
returned?" or "How many records were listed?".
* Allow tidy to backup legacy CA bundles
With the new tidy_move_legacy_ca_bundle option, we'll use tidy to move
the legacy CA bundle from /config/ca_bundle to /config/ca_bundle.bak.
This does two things:
1. Removes ca_bundle from the hot-path of initialization after initial
migration has completed. Because this entry is seal wrapped, this
may result in performance improvements.
2. Allows recovery of this value in the event of some other failure
with migration.
Notably, this cannot occur during migration in the unlikely (and largely
unsupported) case that the operator immediately downgrades to Vault
<1.11.x. Thus, we reuse issuer_safety_buffer; while potentially long,
tidy can always be run manually with a shorter buffer (and only this
flag) to manually move the bundle if necessary.
In the event of needing to recover or undo this operation, it is
sufficient to use sys/raw to read the backed up value and subsequently
write it to its old path (/config/ca_bundle).
The new entry remains seal wrapped, but otherwise isn't used within the
code and so has better performance characteristics.
Performing a fat deletion (DELETE /root) will again remove the backup
like the old legacy bundle, preserving its wipe characteristics.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation about new tidy parameter
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for migration scenarios
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clean up time comparisons
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly distinguish empty issuer names
When using client.Logical().JSONMergePatch(...) with an empty issuer
name, patch incorrectly reports:
> issuer name contained invalid characters
In this case, both the error in getIssuerName(...) is incorrect and
patch should allow setting an empty issuer name explicitly.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add cluster_aia_path templating variable
Per discussion with maxb, allow using a non-Vault distribution point
which may use an insecure transport for RFC 5280 compliance.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address feedback from Max
Co-authored-by: Max Bowsher <maxbowsher@gmail.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Max Bowsher <maxbowsher@gmail.com>
This was not copied over when the this code was
copied in https://github.com/hashicorp/vault/pull/12340.
Also adds a stub for the `.copywrite.hcl` file (for when
Vault is onboarded to Copywrite) and adds the `pkcs7` and
`ui/node_modules` to the ignore pattern.
* Add issuer reference info on JSON endpoint
This endpoint is unauthenticated and shouldn't contain sensitive
information. However, listing the issuers (LIST /issuers) already
returns both the issuer ID and the issuer name (if any) so this
information is safe to return here.
When fetching /pki/issuer/default/json, it would be nice to know exactly
which issuer ID and name it corresponds to, without having to fetch the
authenticated endpoint as well.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Bad news: the hot patch we were using breaks in Go 1.19.4: 6109c07ec4
Good news: we can now patch with an environment variable at runtime.
Co-authored-by: Christopher Swenson <christopher.swenson@hashicorp.com>
* Return the partial success code override for all batch error types
* changelog
* docs
* Lost the actual override logic. :)
* And don't hardcode 400
* gate on success
* Rename path_config -> path_keys_config
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add config/keys to disable upserting
Transit would allow anyone with Create permissions on the encryption
endpoint to automatically create new encryption keys. This becomes hard
to reason about for operators, especially if typos are subtly
introduced (e.g., my-key vs my_key) -- there is no way to merge these
two keys afterwards.
Add the ability to globally disable upserting, so that if the
applications using Transit do not need the capability, it can be
globally disallowed even under permissive policies.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation on disabling upsert
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update website/content/api-docs/secret/transit.mdx
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
* Update website/content/api-docs/secret/transit.mdx
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
A lot of places took a (context, backend, request) tuple, ignoring the
request proper and only using it for its storage. This (modified) tuple
is exactly the set of elements in the shared storage context, so we
should be using that instead of manually passing all three elements
around.
This simplifies a few places where we'd generate a storage context at
the request level and then split it apart only to recreate it again
later (e.g., CRL building).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- Nick brought this to our attention, one of the PKI test suites
is overwriting the production code's value leading to a data race
issue.
- Remove the setting of the variable with the same value from the test
suite.
This PR modifies the path schema of `approle/path_role.go`, switching the old `Callbacks` to the equivalent `Operations` objects with a list of response fields for the 200 responses. This will allow us to generate a response structures in openapi.json. This PR is split out from #18055 along with #18192.
### Example
For `GET "/auth/approle/role/{role_name}/bind-secret-id"` path, it will update the response as follows:
```diff
"responses": {
"200": {
"description": "OK",
++ "content": {
++ "application/json": {
++ "schema": {
++ "$ref": "#/components/schemas/ApproleRoleBindSecretIdResponse"
++ }
++ }
}
}
}
```
And will add the actual response structure:
```diff
++ "ApproleRoleBindSecretIdResponse": {
++ "type": "object",
++ "properties": {
++ "bind_secret_id": {
++ "type": "boolean",
++ "description": "Impose secret_id to be presented when logging in using this role. Defaults to 'true'."
++ }
++ }
++ },
```
* Respond with data to all writes in PKI engine
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow templating of cluster-local AIA URIs
This adds a new configuration path, /config/cluster, which retains
cluster-local configuration. By extending /config/urls and its issuer
counterpart to include an enable_templating parameter, we can allow
operators to correctly identify the particular cluster a cert was
issued on, and tie its AIA information to this (cluster, issuer) pair
dynamically.
Notably, this does not solve all usage issues around AIA URIs: the CRL
and OCSP responder remain local, meaning that some merge capability is
required prior to passing it to other systems if they use CRL files and
must validate requests with certs from any arbitrary PR cluster.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation about templated AIAs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* AIA URIs -> AIA URLs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* issuer.AIAURIs might be nil
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow non-nil response to config/urls
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Always validate URLs on config update
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Ensure URLs lack templating parameters
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Review feedback
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address a nil panic when writing an empty POST request to the ocsp handler
- Seems when no JSON body is sent with a POST request Vault will not
populate the HTTPRequest member variable which caused the nil panic
- vault write -force pki/ocsp
- Add a check for it and the Body member variable to be nil before use.
* Add cl
* Add tests using client certificates
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor Go TLS client tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for CRLs
Note that Delta CRL support isn't present in nginx or apache, so we lack
a server-side test presently. Wget2 does appear to support it however,
if we wanted to add a client-side OpenSSL test.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add checks for delta CRL with wget2
This ensures the delta CRL is properly formatted and accepted by
OpenSSL.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Re-add missing test helpers
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rename clientFullChain->clientWireChain
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rename integation_test.go->integration_test.go
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add ability to fetch container's network addresses
This lets us return the on-network container address, allowing us to
spawn client containers which contact server containers.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add integration tests with nginx, curl, wget, Go
We build new integration tests, spawning a test instance on nginx and
ensuring we can connect with a variety of clients against a variety of
CA and leaf certificate types. This will ultimately let us detect issues
with compatibility as we expand the matrix of supported servers and
clients.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Make runner reference unique
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Attempt to fix CI with longer wait
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Finish moving nginx tests to pkiext package
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* make fmt
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add more debugging, work on CircleCI
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Removes _builtin_ versions from mount storage where it already exists
* Stops new builtin versions being put into storage on mount creation/tuning
* Stops the plugin catalog from returning a builtin plugin that has been overridden, so it more accurately reflects the plugins that are available to actually run
* New PKI API to generate and sign a CRL based on input data
- Add a new PKI API that allows an end-user to feed in all the
information required to generate and sign a CRL by a given issuer.
- This is pretty powerful API allowing an escape hatch for 3rd parties
to craft customized CRLs with extensions based on their individual
needs
* Add api-docs and error if reserved extension is provided as input
* Fix copy/paste error in Object Identifier constants
* Return nil on errors instead of partially filled slices
* Add cl
* wip
* Add cached OCSP client support to Cert Auth
* ->pointer
* Code cleanup
* Fix unit tests
* Use an LRU cache, and only persist up to 1000 of the most recently used values to stay under the storage entry limit
* Fix caching, add fail open mode parameter to cert auth roles
* reduce logging
* Add the retry client and GET then POST logic
* Drop persisted cache, make cache size configurable, allow for parallel testing of multiple servers
* dead code
* Update builtin/credential/cert/path_certs.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Hook invalidate to reinit the ocsp cache size
* locking
* Conditionally init the ocsp client
* Remove cache size config from cert configs, it's a backend global
* Add field
* Remove strangely complex validity logic
* Address more feedback
* Rework error returning logic
* More edge cases
* MORE edge cases
* Add a test matrix with a builtin responder
* changelog
* Use an atomic for configUpdated
* Actually use ocsp_enabled, and bind to a random port for testing
* Update builtin/credential/cert/path_login.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor unit tests
* Add status to cache
* Make some functions private
* Rename for testing, and attribute
* Up to date gofumpt
* remove hash from key, and disable the vault dependent unit test
* Comment out TestMultiOCSP
* imports
* more imports
* Address semgrep results
* Attempt to pass some sort of logging to test_responder
* fix overzealous search&replace
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add crl list capabilities to cert auth
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add docs on cert auth CRL listing
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for cert auth listing
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new PKI api to combine and sign different CRLs from the same issuer
- Add a new PKI api /issuer/<issuer ref>/resign-crls that will allow
combining and signing different CRLs that were signed by the same
issuer.
- This allows external actors to combine CRLs into a single CRL across
different Vault clusters that share the CA certificate and key material
such as performance replica clusters and the primary cluster
* Update API docs
* PR Feedback - Delta CRL rename
* Update to latest version of main
* PR Feedback - Get rid of the new caEntry struct
* Address PR feedback in api-docs and PEM encoded response
* Export CreateBackendWithStorage for pkiext
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move zlint_test.go to pkiext
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix mount all test to ignore pkiext
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Credit to Steve for finding this one.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Expose ssh algorithm_signer in web interface (#10114)
* Adds allowed values for algorithm_signer to ssh plugin API
* Adds algorithm_signer as field in UI
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add automatic tidy of expired issuers
To aid PKI users like Consul, which periodically rotate intermediates,
and provided a little more consistency with older versions of Vault
which would silently (and dangerously!) replace the configured CA on
root/intermediate generation, we introduce an automatic tidy of expired
issuers.
This includes a longer safety buffer (1 year) and logging of the
relevant issuer information prior to deletion (certificate contents, key
ID, and issuer ID/name) to allow admins to recover this value if
desired, or perform further cleanup of keys.
From my PoV, removal of the issuer is thus a relatively safe operation
compared to keys (which I do not feel comfortable removing) as they can
always be re-imported if desired. Additionally, this is an opt-in tidy
operation, not enabled by default. Lastly, most major performance
penalties comes with lots of issuers within the mount, not as much
large numbers of keys (as only new issuer creation/import operations are
affected, unlike LIST /issuers which is a public, unauthenticated
endpoint).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add docs on tidy of issuers
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Restructure logging
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing fields to expected tidy output
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Also remove one duplicate error masked by return.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly preserve other issuer config params
When setting a new default issuer, our helper function would overwrite
other parameters in the issuer configuration entry. However, up until
now, there were none.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new parameter to allow default to follow new
This parameter will allow operators to have the default issuer
automatically update when a new root is generated or a single issuer
with a key (potentially with others lacking key) is imported.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Storage migration tests fail on new members
These internal members shouldn't be tested by the storage migration
code, and so should be elided from the test results.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Follow new issuer on root generation, import
This updates the two places where issuers can be created (outside of
legacy CA bundle migration which already sets the default) to follow
newly created issuers when the config is set.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for new default-following behavior
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new API to PKI to list revoked certificates
- A new API that will return the list of serial numbers of
revoked certificates on the local cluster.
* Add cl
* PR feedback
* Ensure correct write ordering in rebuildIssuersChains
When troubleshooting a recent migration failure from 1.10->1.11, it was
noted that some PKI mounts had bad chain construction despite having
valid, chaining issuers. Due to the cluster's leadership trashing
between nodes, the migration logic was re-executed several times,
partially succeeding each time. While the legacy CA bundle migration
logic was written with this in mind, one shortcoming in the chain
building code lead us to truncate the ca_chain: by sorting the list of
issuers after including non-written issuers (with random IDs), these
issuers would occasionally be persisted prior to storage _prior_ to
existing CAs with modified chains.
The migration code carefully imported the active issuer prior to its
parents. However, due to this bug, there was a chance that, if write to
the pending parent succeeded but updating the active issuer didn't, the
active issuer's ca_chain field would only contain the self-reference and
not the parent's reference as well. Ultimately, a workaround of setting
and subsequently unsetting a manual chain would force a chain
regeneration.
In this patch, we simply fix the write ordering: because we need to
ensure a stable chain sorting, we leave the sort location in the same
place, but delay writing the provided referenceCert to the last
position. This is because the reference is meant to be the user-facing
action: without transactional write capabilities, other chains may
succeed, but if the last user-facing action fails, the user will
hopefully retry the action. This will also correct migration, by
ensuring the subsequent issuer import will be attempted again,
triggering another chain build and only persisting this issuer when
all other issuers have also been updated.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remigrate ca_chains to fix any missing issuers
In the previous commit, we identified an issue that would occur on
legacy issuer migration to the new storage format. This is easy enough
to detect for any given mount (by an operator), but automating scanning
and remediating all PKI mounts in large deployments might be difficult.
Write a new storage migration version to regenerate all chains on
upgrade, once.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add issue to PKI considerations documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correct %v -> %w in chain building errs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
When running the test suite in CI (where requests are centralized from
relatively few IPs), we'd occasionally hit Dockerhub's rate limits.
Luckily Hashicorp runs a (limited) public mirror of the containers we
need, so we can switch to them here in the tests.
For consistency between developer and CI, we've opted to have the tests
always pull from the Hashicorp mirror, rather than updating the CI
runner to prefer the mirror.
We exclude nomad and influxdb as we don't presently mirror these repos.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Return revocation info within existing certs/<serial> api
- The api already returned both the certificate and a revocation_time
field populated. Update the api to return revocation_time_rfc3339
as we do elsewhere and also the issuer id if it was revoked.
- This will allow callers to associate a revoked cert with an issuer
* Add cl
* PR feedback (docs update)
* Don't use a duplicate sync object for stepwise tests precheck
* Change STS test check to no longer look for a secret, add SetSourceIdentity policy to role
* Bump validity period check to satisfy CircleCI
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update builtin/logical/pki/backend_test.go
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add regression test for default CRL expiry
Also fixes a bug w.r.t. upgrading older entries and missing the Delta
Rebuild Interval field, setting it to the default.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog for earlier PR
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support for PKCSv1_5_NoOID signatures
This assumes a pre-hashed input has been provided to Vault, but we do
not write the hash's OID into the signature stream. This allows us to
generate the alternative PKCSv1_5_NoOID signature type rather than the
existing PKCSv1_5_DERnull signature type we presently use.
These are specified in RFC 3447 Section 9.2.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Exclude new none type from PSS based tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for PKCS#1v1.5 signatures
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow passing AssociatedData factories in keysutil
This allows the high-level, algorithm-agnostic Encrypt/Decrypt with
Factory to pass in AssociatedData, and potentially take multiple
factories (to allow KMS keys to work). On AEAD ciphers with a relevant
factory, an AssociatedData factory will be used to populate the
AdditionalData field of the SymmetricOpts struct, using it in the AEAD
Seal process.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add associated_data to Transit Encrypt/Decrypt API
This allows passing the associated_data (the last AD in AEAD) to
Transit's encrypt/decrypt when using an AEAD cipher (currently
aes128-gcm96, aes256-gcm96, and chacha20-poly1305). We err if this
parameter is passed on non-AEAD ciphers presently.
This associated data can be safely transited in plaintext, without risk
of modifications. In the event of tampering with either the ciphertext
or the associated data, decryption will fail.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add to documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
The SSH secrets engine previously split the `validPrincipals` field
on comma, then if user templating is enabled, evaluated the
templates on each substring. This meant the identity template was only
ever allowed to return a single principal. There are use cases
where it would be helpful for identity metadata to contain a list
of valid principals and for the identity template to be able to inject
all of those as valid principals.
This change inverts the order of processing. First the template
is evaluated, and then the resulting string is split on commas.
This allows the identity template to return a single comma-separated
string with multiple permitted principals.
There is a potential security implication here, that if a user is
allowed to update their own identity metadata, they may be able to
elevate privileges where previously this was not possible.
Fixes#11038
* Add tests for zlint-clean CA building
This test ensures that we can consistently pass ZLint's CA linting
tests on a root certificate generated by Vault. In particular, nominal
requirements are placed on the structure on the issuer's Subject, which
we supply, and the remaining requirements pass.
The one exception is we include both RFC and CA/BF BR lints in the
default zlint checks; this means ECDSA P-521 (which isn't accepted by
Mozilla's root store policies) is rejected, so we ignore to lints
related to that.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add options to copy to/from container, fix stopping
Stopping the container takes a bit of time for some unknown reason so
I've instead opted to shorten the sleep in the zlint tests to avoid
consuming resources too long after the test finish.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Make zlint tests execute in parallel
This improves the overall test time of the zlint tests, making the
container build up front once (provisioning zlint), and then copying the
cert into the new container image later.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* make fmt
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor Docker command execution
This refactor will allow others to interact with containers more easily,
providing two interfaces (RunCmdWithOutput and RunCmdInBackground) for
executing commands in running containers if they don't wish to do so
manually.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow building containerfiles in tests
By adding image building capabilities to testhelpers (and coupled with
the better command execution support), we can begin to build better,
more reliable integration tests on top of public base images without
needing to maintain separate forks of these images out-of-tree for any
shortcomings they might have.
In particular, rather than doing the rather messy echo hack for writing
clients.conf, it is far better to provision this via a slim
Containerfile overlay on top of the stock jumanjiman/radiusd:latest
image.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly parse stdout/stderr in RunCmdWithOutput
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* ctx -> bCtx for BuildContext
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update errors to use %w instead of %v
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix tidy-status, tidy-cancel on PR Secondaries
PKI's tidy-status included a bug that prevented PR secondary nodes from
responding with the status of the running tidy operation: while the
operation constructor correctly forwarded the node on PR standby
instances, the handler itself forwarded also on PR secondary nodes.
This is incorrect as the PR secondary nodes are the active node in the
local PR cluster, and run tidy operations otherwise.
This meant that while auto-tidy and tidy operations would run, there was
no insight into the process.
When implementing tidy-cancel, tidy-status's handler logic was reused,
duplicating the bug there as well.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
When revoking an issuer, we immediately force a full rebuild of all CRLs
(complete and delta). However, we had forgotten to guard the delta CRL's
inclusion of augmented issuers, resulting in double-listing the issuer's
serial number on both the complete and the delta CRL. This isn't
necessary as the delta's referenced complete CRL number has incremented
to the point where the issuer itself was included on the complete CRL.
Avoid this double reference and don't include issuers on delta CRLs;
they should always appear only on the complete CRL.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- Add some unit tests around the OCSP response validation that we
are using the proper signature algorithms.
- Add in test cases as well to validate SHA384 and SHA512 requested hash support
* Fix for duplicate SANs in signed certificates when othernames are present in the CSR SAN extension and UseCSRValues is true.
When UseCSRValues is true (as is the case on the sign-verbatim endpoint), all extensions including Subject Alternative Names are copied from the CSR to the final certificate.
If the Subject Alternative Name in question contains any othernames (such as a Microsoft UPN) the SAN extension is added again as a workaround for an encoding issue (in function HandleOtherSANs).
Having duplicate x509v3 extensions is invalid and is rejected by openssl on Ubuntu 20.04, and also by Go since https://github.com/golang/go/issues/50988 (including in Go 1.19).
In this fix I do not add the extension from the CSR if it will be added during HandleOtherSANs.
* Added unittest and changelog entry.
* Fix RevocationSigAlg provisioning in GCP
GCP restricts keys to a certain type of signature, including hash
algorithm, so we must provision our RevocationSigAlg from the root
itself unconditionally in order for GCP to work.
This does change the default, but only for newly created certificates.
Additionally, we clarify that CRL building is not fatal to the import
process.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add inverse mapping for SignatureAlgorithm
By default we'd use .String() on x509.SignatureAlgorithm, but this
doesn't round-trip. Switch to a custom map that is round-trippable
and matches the constant name as there is no other way to get this info
presently.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test to ensure root creation sets rev_sig_alg
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Test round-tripping of SigAlgoNames, InvSigAlgoNames
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix failing Default Update test
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
When requesting a SSH certificate with default_extension templating
enabled, if the request lacks entity information and a particular
extension requires templating, just these extensions will be elided.
Other extensions (if present) will still be on the final certificate.
Add a warning in the event of missing entity information and at least
one extension that was skipped as a result.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow OCSP to use issuer's RevocationSigAlgo
When an issuer specifies a RevocationSigAlgo, we should largely follow
this for both CRLs and OCSP. However, x/crypto/ocsp lacks support for
PSS signatures, so we drop these down to PKCS#1v1.5 instead.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add warning when issuer has PSS-based RevSigAlgo
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add note about OCSP and PSS support
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* PKI: Add support for signature_bits param to the intermediate/generate api
- Mainly to work properly with GCP backed managed keys, we need to
issue signatures that would match the GCP key algorithm.
- At this time due to https://github.com/golang/go/issues/45990 we
can't issue PSS signed CSRs, as the libraries in Go always request
a PKCS1v15.
- Add an extra check in intermediate/generate that validates the CSR's
signature before providing it back to the client in case we generated
a bad signature such as if an end-user used a GCP backed managed key
with a RSA PSS algorithm.
- GCP ignores the requested signature type and always signs with the
key's algorithm which can lead to a CSR that says it is signed with
a PKCS1v15 algorithm but is actually a RSA PSS signature
* Add cl
* PR feedback
* PKI: Do not load revoked certificates if CRL has been disabled
- Restore the prior behavior of not reading in all revoked certificates
if the CRL has been disabled as there might be performance issues
if a customer had or is still revoking a lot of certificates.
* Add cl
When adding delta CRL support, we unconditionally added the delta
indicator extension to the main CRL. We shouldn't have done this, and
instead only added it conditionally when we were building delta CRLs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
The periodic function only runs every 50ms, so waiting 60ms means we
might not be done fetching the CRL on slower CI systems or with high
test parallelism.
Tested with:
> untilfail -parallel=-9 ../../../cert.test -test.run=TestCRLFetch -test.count=1 -test.v
And shown to reliably fail before, fixed after.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Check if plugin version matches running version
When registering a plugin, we check if the request version matches the
self-reported version from the plugin. If these do not match, we log a
warning.
This uncovered a few missing pieces for getting the database version
code fully working.
We added an environment variable that helps us unit test the running
version behavior as well, but only for approle, postgresql, and consul
plugins.
Return 400 on plugin not found or version mismatch
Populate the running SHA256 of plugins in the mount and auth tables (#17217)
* Fix interoperability concerns with PSS
When Go parses a certificate with rsaPSS OID, it will accept this
certificate but not parse the SubjectPublicKeyInfo, leaving the
PublicKeyAlgorithm and PublicKey fields blank, but otherwise not erring.
The same behavior occurs with rsaPSS OID CSRs.
On the other hand, when Go parses rsaPSS OID PKCS8 private keys, these
keys will fail to parse completely.
Thus, detect and fail on any empty PublicKey certs and CSRs, warning the
user that we cannot parse these correctly and thus refuse to operate.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Run more PKI tests in parallel
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add notes about PSS shortcomings to considerations
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Basics of Cert-Count Telemetry, changelog, "best attempt" slice to capture (and test for) duplicates, Move sorting of possibleDoubleCountedRevokedSerials to after compare of entries. Add values to counter when still initializing.
Set lists to nil after use, Fix atomic2 import, Delay reporting metrics until after deduplication has completed,
The test works now, Move string slice to helper function; Add backendUUID to gauge name.
* Don't race for CRL rebuilding capability check
Core has recently seen some data races during SystemView/replication
updates between them and the PKI subsystem. This is because this
SystemView access occurs outside of a request (during invalidation
handling) and thus the proper lock isn't held.
Because replication status cannot change within the lifetime of a plugin
(and instead, if a node switches replication status, the entire plugin
instance will be torn down and recreated), it is safe to cache this
once, at plugin startup, and use it throughout its lifetime.
Thus, we replace this SystemView access with a stored boolean variable
computed ahead of time.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update builtin/logical/pki/backend.go
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
There were two races here:
1. Tests racing against periodic func on updating the backend.
2. Tests racing internally to itself, to access the http-served
CRL data.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fetch CRLs from a user defined CDP (PoC)
* Handle no param sent
* Move CRL fetch to a periodFunc. Use configured CA certs + system root as trusted certs for CRL fetch
* comments
* changelog
* Just use root trust
* cdp->url in api
* Store CRL and populate it initially in cdlWrite
* Update docs
* Update builtin/credential/cert/path_crls.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Handle pre-verification of a CRL url better
* just in case
* Fix crl write locking
* Add a CRL fetch unit test
* Remove unnecessary validity clear
* Better func name
* Don't exit early updating CRLs
* lock in updateCRLs
* gofumpt
* err-
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
Add plugin version to GRPC interface
Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems.
I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion:
If a plugin has a PluginVersion() method implemented, then that is used
If not, and the plugin is built into the Vault binary, then the go.mod version is used
Otherwise, the it will be the empty string.
My apologies for the length of this PR.
* Placeholder backend should be external
We use a placeholder backend (previously a framework.Backend) before a
GRPC plugin is lazy-loaded. This makes us later think the plugin is a
builtin plugin.
So we added a `placeholderBackend` type that overrides the
`IsExternal()` method so that later we know that the plugin is external,
and don't give it a default builtin version.
This option was elided from the default value for the usage field. This
results in issuers "losing" ocsp-signing when they're POST updated. Most
issuers will want OCSP signing by default, so it makes sense to add this
as the default.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Support version selection for database plugins
* Don't consider unversioned plugins for version selection algorithm
* Added version to 'plugin not found' error
* Add PluginFactoryVersion function to avoid changing sdk/ API
* Allow exposing access to the underlying container
This exposes the Container response from the Docker API, allowing
consumers of the testhelper to interact with the newly started running
container instance. This will be useful for two reasons:
1. Allowing radiusd container to start its own daemon after modifying
its configuration.
2. For loading certificates into a future similar integration test
using the PKI secrets engine.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow any client to connect to test radiusd daemon
This fixes test failures of the following form:
> 2022-09-07T10:46:19.332-0400 [TRACE] core: adding local paths: paths=[]
> 2022-09-07T10:46:19.333-0400 [INFO] core: enabled credential backend: path=mnt/ type=test
> 2022-09-07T10:46:19.334-0400 [WARN] Executing test step: step_number=1
> 2022-09-07T10:46:19.334-0400 [WARN] Executing test step: step_number=2
> 2022-09-07T10:46:29.334-0400 [WARN] Executing test step: step_number=3
> 2022-09-07T10:46:29.335-0400 [WARN] Executing test step: step_number=4
> 2022-09-07T10:46:39.336-0400 [WARN] Requesting RollbackOperation
> --- FAIL: TestBackend_acceptance (28.56s)
> testing.go:364: Failed step 4: erroneous response:
>
> &logical.Response{Secret:<nil>, Auth:<nil>, Data:map[string]interface {}{"error":"context deadline exceeded"}, Redirect:"", Warnings:[]string(nil), WrapInfo:(*wrapping.ResponseWrapInfo)(nil), Headers:map[string][]string(nil)}
> FAIL
> FAIL github.com/hashicorp/vault/builtin/credential/radius 29.238s
In particular, radiusd container ships with a default clients.conf which
restricts connections to ranges associated with the Docker daemon. When
creating new networks (such as in CircleCI) or when running via Podman
(which has its own set of network ranges), this initial config will no
longer be applicable. We thus need to write a new config into the image;
while we could do this by rebuilding a new image on top of the existing
layers (provisioning our config), we then need to manage these changes
and give hooks for the service setup to build it.
Thus, post-startup modification is probably easier to execute in our
case.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- When we added new tests that validate the RSA PSS feature, they
work properly on normal Go builds, but tests underneath the Boring
Crypto fips implementations fail due to a lack of SHA3 support in
FIPS 140-2.
* Get import correct
* limits, docs
* changelog
* unit tests
* And fix import for hmac unit test
* typo
* Update website/content/api-docs/secret/transit.mdx
Co-authored-by: Matt Schultz <975680+schultz-is@users.noreply.github.com>
* Update builtin/logical/transit/path_keys.go
Co-authored-by: Matt Schultz <975680+schultz-is@users.noreply.github.com>
* Validate key sizes a bit more carefully
* Update sdk/helper/keysutil/policy.go
Co-authored-by: Matt Schultz <975680+schultz-is@users.noreply.github.com>
Co-authored-by: Matt Schultz <975680+schultz-is@users.noreply.github.com>
* Add fields 'ttl' and 'num_uses' to SecretID generation.
Add fields 'ttl' and 'num_uses' when generating/obtaining a SecretID.
Rather than just being able to use the Role's SecretID ttl and num uses. #14390
* Add secret_id_num_uses response field to generating SecretID
Add the response field secret_id_num_uses to the endpoints for generating
SecretIDs. Used in testing but also to supply the vendor with this variable.
* Add tests for new ttl and num_uses SecretID generation fields
Add tests to assert the new TTL and NumUses option in the SecretID entry.
Separate test for testing with just parameters vs a -force example.
* Patch up test for ttl and num_uses fields
* Add changelog entry for auth/approle 'ttl' and 'num_uses' fields
* Add fields to API Docs and AppRole Auth Docs example
* Correct error message for failing test on missing field.
Change the error message produced when a test fails due to a missing field.
Previous values did not map to correct fields.
* Remove unnecessary int cast to int "secret_id_num_uses" field.
Unnecessary cast to int where type already is int.
* Move numUses field check to after assignment.
* Remove metadata entry in sample payload to limit change to changes made.
Remove metadata entry in sample payload for custom-secret-id. The metadata was not
changed in the features pull request.
* Bind fields 'ttl' and 'num_uses' to role's configuration.
Rather than implicitly overriding, error when the ttl is lower than and the num
uses higher than the role's configuration. #14390
* Update changelog 14474 with a more detailed description.
More elaborate description for the changelog. Specifying the per-request based fields.
* Elaborate more on the bounds of the 'ttl' and 'num_uses' field.
Specify in both the api-docs and the CLI the limits of the fields.
Specify that the role's configuration is still the leading factor.
* Upper bound ttl with role secret id ttl
Upper bound ttl with role secret id ttl when creating a secret id
Adding test cases for infinite ttl and num uses
Adding test cases for negative ttl and num uses
Validation on infinite ttl and num uses
* Formatting issues. Removed unnecessary newline
* Update documentation for AppRole Secret ID and Role
Changed that TTL is not allowed to be shorter to longer
* Cleanup approle secret ID test and impl
* Define ttl and num_uses in every test
Define ttl and num_uses in every test despite them not being tested.
This is to ensure that no unexpected behaviour comes to mind.
* Rename test RoleSecretID -> RoleSecretIDWithoutFields
* Test secret id generation defaults to Role's config
Test secret id generation defaults to Role's configuration entries.
* Change finit -> finite
Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
* Rephrase comments to the correct validation check
* Rephrase role-secret-id option description
* Remove "default" incorrect statement about ttl
* Remove "default" incorrect statement about ttl for custom secret id
* Touch up approle.mdx to align more with path_role documentation
Co-authored-by: Remco Buddelmeijer <r.buddelmeijer@fullstaq.com>
Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
* Add path to manually rebuild delta CRLs
The crl/rotate-delta path behaves like crl/rotate, triggering a
cluster-local rebuild of just the delta CRL. This is useful for when
delta CRLs are enabled with a longer-than-desired auto-rebuild period
after some high-profile revocations occur.
In the event delta CRLs are not enabled, this becomes a no-op.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for Delta CRL rebuilding
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update documentation about Delta CRLs
Also fixes a omission in the If-Modified-Since docs to mention that the
response header should probably also be passed through.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow tidy operations to be cancelled
When tidy operations take a long time to execute (and especially when
executing them automatically), having the ability to cancel them becomes
useful to reduce strain on Vault clusters (and let them be rescheduled
at a later time).
To this end, we add the /tidy-cancel write endpoint.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing auto-tidy synopsis / description
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add a pause duration between tidying certificates
By setting pause_duration, operators can have a little control over the
resource utilization of a tidy operation. While the list of certificates
remain in memory throughout the entire operation, a pause is added
between processing certificates and the revocation lock is released.
This allows other operations to occur during this gap and potentially
allows the tidy operation to consume less resources per unit of time
(due to the sleep -- though obviously consumes the same resources over
the time of the operation).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for cancellation, pause
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add API docs on pause_duration, /tidy-cancel
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add lock releasing around tidy pause
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Reset cancel guard, return errors
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* accommodate salt lengths for RSA PSS
* address feedback
* generalise salt length to an int
* fix error reporting
* Revert "fix error reporting"
This reverts commit 8adfc15fe3303b8fdf9f094ea246945ab1364077.
* fix a faulty check
* check for min/max salt lengths
* stringly-typed HTTP param
* unit tests for sign/verify HTTP requests
also, add marshaling for both SDK and HTTP requests
* randomly sample valid salt length
* add changelog
* add documentation
* Add remove_roots_from_chain flag to sign and issue pki apis
- Add a new flag to allow end-users to control if we return the
root/self-signed CA certificate within the list of certificates in
ca_chain field on issue and sign api calls.
* Add cl
* PR feedback
We switch these fields to use the explicit default value (computing the
time in seconds appropriately).
As reported by @beornf, thanks!
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add ability to perform automatic tidy operations
This enables the PKI secrets engine to allow tidy to be started
periodically by the engine itself, avoiding the need for interaction.
This operation is disabled by default (to avoid load on clusters which
don't need tidy to be run) but can be enabled.
In particular, a default tidy configuration is written (via
/config/auto-tidy) which mirrors the options passed to /tidy. Two
additional parameters, enabled and interval, are accepted, allowing
auto-tidy to be enabled or disabled and controlling the interval
(between successful tidy runs) to attempt auto-tidy.
Notably, a manual execution of tidy will delay additional auto-tidy
operations. Status is reported via the existing /tidy-status endpoint.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation on auto-tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for auto-tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Prevent race during parallel testing
We modified the RollbackManager's execution window to allow more
faithful testing of the periodicFunc. However, the TestAutoRebuild and
the new TestAutoTidy would then race against each other for modifying
the period and creating their clusters (before resetting to the old
value).
This changeset adds a lock around this, preventing the races.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use tidyStatusLock to gate lastTidy time
This prevents a data race between the periodic func and the execution of
the running tidy.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add read lock around tidyStatus gauges
When reading from tidyStatus for computing gauges, since the underlying
values aren't atomics, we really should be gating these with a read lock
around the status access.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
