* POC of Okta Auth Number Challenge verification
* switch from callbacks to operations, forward validate to primary
* cleanup and nonce description update
* add changelog
* error on empty nonce, no forwarding, return correct_answer instead
* properly clean up verify goroutine
* add docs on new endpoint and parameters
* change polling frequency when WAITING to 1s
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* Add CRL checking to chain building tests
This should ensure that, with our complex issuer setups, we can revoke
the issued certificates correctly and they'll show up on the correct
CRLs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix related issuer detection in CRL building
When building our mapping of issuers, we incorrectly used the issuer's
RawIssuer field to construct the mapping, rather than the issuer's
RawSubject. This caused us to not correctly detect the cross-signed
issuers as having the same CRLs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address issues with revoke operations pre-migration of PKI issuers
- Leverage the legacyBundleShimID though out the path of CRL building
when legacy storage mode is active.
- Instead of having multiple locations without a lock checking for the
useLegacyBundleCaStorage flag is set, check it once and then use the
same issuerId everywhere
- Address some locking issues that might lead to a bad read/write when
switching from legacy to non-legacy mode on startup and post-migration
* Add test suite for PKI apis pre-migration to new issuer storage format
- Add tests that validate all apis work as expected in pre-migration mode
- Add tests for apis that we don't expect to work, they should return a
migration related error message
- Add some missing validations on various new apis.
* Refactor chain building test cases to be shared
This will allow us to execute these test cases and then benchmark just
the chain building, separate from the certificate creation (and without
the consistency tests).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Benchmark chain building code
Using the existing test cases (and a few special ones), generate some
simple chains and benchmark how long chain building takes. We switch
from generating a cluster (slow) to directly calling
createBackendWithStorage(), which improves test execution time too:
$ go test -count=1 -run=Test_CAChainBuilding github.com/hashicorp/vault/builtin/logical/pki
ok github.com/hashicorp/vault/builtin/logical/pki 0.764s
(previously it was 5-10 seconds, for fewer tests).
Additionally, we now have benchmarks:
$ go test -v -run=BenchmarkChainBuilding -bench=. github.com/hashicorp/vault/builtin/logical/pki
goos: linux
goarch: amd64
pkg: github.com/hashicorp/vault/builtin/logical/pki
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
BenchmarkChainBuilding
BenchmarkChainBuilding/test-case-0
BenchmarkChainBuilding/test-case-0-16 616 1921783 ns/op
BenchmarkChainBuilding/test-case-1
BenchmarkChainBuilding/test-case-1-16 1191 998201 ns/op
BenchmarkChainBuilding/test-case-2
BenchmarkChainBuilding/test-case-2-16 547 2229810 ns/op
BenchmarkChainBuilding/test-case-3
BenchmarkChainBuilding/test-case-3-16 525 2264951 ns/op
BenchmarkChainBuilding/test-case-4
BenchmarkChainBuilding/test-case-4-16 1732 693686 ns/op
BenchmarkChainBuilding/test-case-5
BenchmarkChainBuilding/test-case-5-16 51700 23230 ns/op
BenchmarkChainBuilding/test-case-6
BenchmarkChainBuilding/test-case-6-16 9343 124523 ns/op
BenchmarkChainBuilding/test-case-7
BenchmarkChainBuilding/test-case-7-16 5106 234902 ns/op
BenchmarkChainBuilding/test-case-8
BenchmarkChainBuilding/test-case-8-16 2334 494382 ns/op
PASS
ok github.com/hashicorp/vault/builtin/logical/pki 12.707s
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- Do not grab a lock within the requestRebuildIfActiveNode function
to avoid issues being called from the invalidate function
- Leverage more atmoic operations, and only grab the lock if we are
going to perform the rebuild.
Previously we'd return the raw enum value, which the entity accessing
the API wouldn't have any easy way of translating back into string
values. Return the string value directly instead.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- Leverage a Get lookup operation to see if our reference field is a UUID
instead of listing all key/issuers and iterating over the list.
- This should be faster and we get a cached lookup possibly if it was a
UUID entry that we previously loaded.
- Address some small feedback about migration wording as well.
* Return the ca_chain response from root issued cert api
* Fix parent selection in cert chain building
When building chains, we'd choose the next neighbor from Go's
unordered map. However, this doesn't necessarily result in the most
optimal path: we want to prefer to visit roots over other
intermediates, as this allows us to have a more consistent chain,
putting roots before their cross-signed equivalents rather than
potentially at the end.
We additionally now ensure chains are stable.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Steve Clark <steven.clark@hashicorp.com>
- Validate the key_type and key_bits arguments that were provided and
perform the same default processing of 0 as we used to do for the
generateRoot/generateIntermediate apis
- Add a test that validates the behaviour
- Update the field description blurbs.
* Move existing test helpers into a new test_helpers.go file within PKI
* Compare issuer certificates by cert, signature algo and signature
- Instead of comparing the strings of a certificate, instead leverage
the Go Raw attribute within a parsed certificate to compare. The Raw
attribute is a byte array of an ASN.1 DER containing the cert,
signature algo and signature.
- Rework a bit of the importIssuers function as well to fail checks on the
inbound issuer earlier as well as load keys/issuers just before we need
them
* Prevent revocation of issuers using revokeCert
Similar to the existing behavior, we'll prevent the revocation of
existing issuer certificates from the existing /revoke/:serial endpoint
for now. This is because a serial number alone is not enough information
(in the worst case) to precisely identify an issuer (as intermediates
signed by two separate external (e.g., OpenSSL) CAs using incremental
serial numbers might have the same serial number).
Additionally, we fix revoking certs on performance secondary clusters,
when they've not yet been migrated.
In a separate change, we'll open up a separate code path to revoke
issuers, ensuring we know exactly which issuer is revoked (and which CRL
it should belong on at time of revocation).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add warning when revoking expired cert
This prevents confusion when a nil response (with no revocation info) is
returned; requesters are informed that the specified certificate has
already expired.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Starter PKI CA Storage API (#14796)
* Simple starting PKI storage api for CA rotation
* Add key and issuer storage apis
* Add listKeys and listIssuers storage implementations
* Add simple keys and issuers configuration storage api methods
* Handle resolving key, issuer references
The API context will usually have a user-specified reference to the key.
This is either the literal string "default" to select the default key,
an identifier of the key, or a slug name for the key. Here, we wish to
resolve this reference to an actual identifier that can be understood by
storage.
Also adds the missing Name field to keys.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add method to fetch an issuer's cert bundle
This adds a method to construct a certutil.CertBundle from the specified
issuer identifier, optionally loading its corresponding key for signing.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor certutil PrivateKey PEM handling
This refactors the parsing of PrivateKeys from PEM blobs into shared
methods (ParsePEMKey, ParseDERKey) that can be reused by the existing
Bundle parsing logic (ParsePEMBundle) or independently in the new
issuers/key-based PKI storage code.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add importKey, importCert to PKI storage
importKey is generally preferable to the low-level writeKey for adding
new entries. This takes only the contents of the private key (as a
string -- so a PEM bundle or a managed key handle) and checks if it
already exists in the storage.
If it does, it returns the existing key instance.
Otherwise, we create a new one. In the process, we detect any issuers
using this key and link them back to the new key entry.
The same holds for importCert over importKey, with the note that keys
are not modified when importing certificates.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for importing issuers, keys
This adds tests for importing keys and issuers into the new storage
layout, ensuring that identifiers are correctly inferred and linked.
Note that directly writing entries to storage (writeKey/writeissuer)
will take KeyID links from the parent entry and should not be used for
import; only existing entries should be updated with this info.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Implement PKI storage migration.
- Hook into the backend::initialize function, calling the migration on a primary only.
- Migrate an existing certificate bundle to the new issuers and key layout
* Make fetchCAInfo aware of new storage layout
This allows fetchCAInfo to fetch a specified issuer, via a reference
parameter provided by the user. We pass that into the storage layer and
have it return a cert bundle for us. Finally, we need to validate that
it truly has the key desired.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Begin /issuers API endpoints
This implements the fetch operations around issuers in the PKI Secrets
Engine. We implement the following operations:
- LIST /issuers - returns a list of known issuers' IDs and names.
- GET /issuer/:ref - returns a JSON blob with information about this
issuer.
- POST /issuer/:ref - allows configuring information about issuers,
presently just its name.
- DELETE /issuer/:ref - allows deleting the specified issuer.
- GET /issuer/:ref/{der,pem} - returns a raw API response with just
the DER (or PEM) of the issuer's certificate.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add import to PKI Issuers API
This adds the two core import code paths to the API:
/issuers/import/cert and /issuers/import/bundle. The former differs from
the latter in that the latter allows the import of keys. This allows
operators to restrict importing of keys to privileged roles, while
allowing more operators permission to import additional certificates
(not used for signing, but instead for path/chain building).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add /issuer/:ref/sign-intermediate endpoint
This endpoint allows existing issuers to be used to sign intermediate
CA certificates. In the process, we've updated the existing
/root/sign-intermediate endpoint to be equivalent to a call to
/issuer/default/sign-intermediate.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add /issuer/:ref/sign-self-issued endpoint
This endpoint allows existing issuers to be used to sign self-signed
certificates. In the process, we've updated the existing
/root/sign-self-issued endpoint to be equivalent to a call to
/issuer/default/sign-self-issued.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add /issuer/:ref/sign-verbatim endpoint
This endpoint allows existing issuers to be used to directly sign CSRs.
In the process, we've updated the existing /sign-verbatim endpoint to be
equivalent to a call to /issuer/:ref/sign-verbatim.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow configuration of default issuers
Using the new updateDefaultIssuerId(...) from the storage migration PR
allows for easy implementation of configuring the default issuer. We
restrict callers from setting blank defaults and setting default to
default.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix fetching default issuers
After setting a default issuer, one should be able to use the old /ca,
/ca_chain, and /cert/{ca,ca_chain} endpoints to fetch the default issuer
(and its chain). Update the fetchCertBySerial helper to no longer
support fetching the ca and prefer fetchCAInfo for that instead (as
we've already updated that to support fetching the new issuer location).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add /issuer/:ref/{sign,issue}/:role
This updates the /sign and /issue endpoints, allowing them to take the
default issuer (if none is provided by a role) and adding
issuer-specific versions of them.
Note that at this point in time, the behavior isn't yet ideal (as
/sign/:role allows adding the ref=... parameter to override the default
issuer); a later change adding role-based issuer specification will fix
this incorrect behavior.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support root issuer generation
* Add support for issuer generate intermediate end-point
* Update issuer and key arguments to consistent values
- Update all new API endpoints to use the new agreed upon argument names.
- issuer_ref & key_ref to refer to existing
- issuer_name & key_name for new definitions
- Update returned values to always user issuer_id and key_id
* Add utility methods to fetch common ref and name arguments
- Add utility methods to fetch the issuer_name, issuer_ref, key_name and key_ref arguments from data fields.
- Centralize the logic to clean up these inputs and apply various validations to all of them.
* Rename common PKI backend handlers
- Use the buildPath convention for the function name instead of common...
* Move setting PKI defaults from writeCaBundle to proper import{keys,issuer} methods
- PR feedback, move setting up the default configuration references within
the import methods instead of within the writeCaBundle method. This should
now cover all use cases of us setting up the defaults properly.
* Introduce constants for issuer_ref, rename isKeyDefaultSet...
* Fix legacy PKI sign-verbatim api path
- Addresses some test failures due to an incorrect refactoring of a legacy api
path /sign-verbatim within PKI
* Use import code to handle intermediate, config/ca
The existing bundle import code will satisfy the intermediate import;
use it instead of the old ca_bundle import logic. Additionally, update
/config/ca to use the new import code as well.
While testing, a panic was discovered:
> reflect.Value.SetMapIndex: value of type string is not assignable to type pki.keyId
This was caused by returning a map with type issuerId->keyId; instead
switch to returning string->string maps so the audit log can properly
HMAC them.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify error message on missing defaults
When the default issuer and key are missing (and haven't yet been
specified), we should clarify that error message.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update test semantics for new changes
This makes two minor changes to the existing test suite:
1. Importing partial bundles should now succeed, where they'd
previously error.
2. fetchCertBySerial no longer handles CA certificates.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support for deleting all keys, issuers
The old DELETE /root code must now delete all keys and issuers for
backwards compatibility. We strongly suggest calling individual delete
methods (DELETE /key/:key_ref or DELETE /issuer/:issuer_ref) instead,
for finer control.
In the process, we detect whether the deleted key/issuers was set as the
default. This will allow us to warn (from the single key/deletion issuer
code) whether or not the default was deleted (while allowing the
operation to succeed).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Introduce defaultRef constant within PKI
- Replace hardcoded "default" references with a constant to easily identify various usages.
- Use the addIssuerRefField function instead of redefining the field in various locations.
* Rework PKI test TestBackend_Root_Idempotency
- Validate that generate/root calls are no longer idempotent, but the bundle importing
does not generate new keys/issuers
- As before make sure that the delete root api resets everything
- Address a bug within the storage that we bombed when we had multiple different
key types within storage.
* Assign Name=current to migrated key and issuer
- Detail I missed from the RFC was to assign the Name field as "current" for migrated key and issuer.
* Build CRL upon PKI intermediary set-signed api called
- Add a call to buildCRL if we created an issuer within pathImportIssuers
- Augment existing FullCAChain to verify we have a proper CRL post set-signed api call
- Remove a code block writing out "ca" storage entry that is no longer used.
* Identify which certificate or key failed
When importing complex chains, we should identify in which certificate
or key the failure occurred.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* PKI migration writes out empty migration log entry
- Since the elements of the struct were not exported we serialized an empty
migration log to disk and would re-run the migration
* Add chain-building logic to PKI issuers path
With the one-entry-per-issuer approach, CA Chains become implicitly
constructed from the pool of issuers. This roughly matches the existing
expectations from /config/ca (wherein a chain could be provided) and
/intemediate/set-signed (where a chain may be provided). However, in
both of those cases, we simply accepted a chain. Here, we need to be
able to reconstruct the chain from parts on disk.
However, with potential rotation of roots, we need to be aware of
disparate chains. Simply concating together all issuers isn't
sufficient. Thus we need to be able to parse a certificate's Issuer and
Subject field and reconstruct valid (and potentially parallel)
parent<->child mappings.
This attempts to handle roots, intermediates, cross-signed
intermediates, cross-signed roots, and rotated keys (wherein one might
not have a valid signature due to changed key material with the same
subject).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Return CA Chain when fetching issuers
This returns the CA Chain attribute of an issuer, showing its computed
chain based on other issuers in the database, when fetching a specific
issuer.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add testing for chain building
Using the issuance infrastructure, we generate new certificates (either
roots or intermediates), positing that this is roughly equivalent to
importing an external bundle (minus error handling during partial
imports). This allows us to incrementally construct complex chains,
creating reissuance cliques and cross-signing cycles.
By using ECDSA certificates, we avoid high signature verification and
key generation times.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow manual construction of issuer chain
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix handling of duplicate names
With the new issuer field (manual_chain), we can no longer err when a
name already exists: we might be updating the existing issuer (with the
same name), but changing its manual_chain field. Detect this error and
correctly handle it.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for manual chain building
We break the clique, instead building these chains manually, ensuring
that the remaining chains do not change and only the modified certs
change. We then reset them (back to implicit chain building) and ensure
we get the same results as earlier.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add stricter verification of issuers PEM format
This ensures each issuer is only a single certificate entry (as
validated by count and parsing) without any trailing data.
We further ensure that each certificate PEM has leading and trailing
spaces removed with only a single trailing new line remaining.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix full chain building
Don't set the legacy IssuingCA field on the certificate bundle, as we
prefer the CAChain field over it.
Additionally, building the full chain could result in duplicate
certificates when the CAChain included the leaf certificate itself. When
building the full chain, ensure we don't include the bundle's
certificate twice.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add stricter tests for full chain construction
We wish to ensure that each desired certificate in the chain is only
present once.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Rename PKI types to avoid constant variable name collisions
keyId -> keyID
issuerId -> issuerID
key -> keyEntry
issuer -> issuerEntry
keyConfig -> keyConfigEntry
issuerConfig -> issuerConfigEntry
* Update CRL handling for multiple issuers
When building CRLs, we've gotta make sure certs issued by that issuer
land up on that issuer's CRL and not some other CRL. If no CRL is
found (matching a cert), we'll place it on the default CRL.
However, in the event of equivalent issuers (those with the same subject
AND the same key material) -- perhaps due to reissuance -- we'll only
create a single (unified) CRL for them.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow fetching updated CRL locations
This updates fetchCertBySerial to support querying the default issuer's
CRL.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove legacy CRL storage location test case
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update to CRLv2 Format to copy RawIssuer
When using the older Certificate.CreateCRL(...) call, Go's x509 library
copies the parsed pkix.Name version of the CRL Issuer's Subject field.
For certain constructed CAs, this fails since pkix.Name is not suitable
for round-tripping. This also builds a CRLv1 (per RFC 5280) CRL.
In updating to the newer x509.CreateRevocationList(...) call, we can
construct the CRL in the CRLv2 format and correctly copy the issuer's
name. However, this requires holding an additional field per-CRL, the
CRLNumber field, which is required in Go's implementation of CRLv2
(though OPTIONAL in the spec). We store this on the new
LocalCRLConfigEntry object, per-CRL.
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add comment regarding CRL non-assignment in GOTO
In previous versions of Vault, it was possible to sign an empty CRL
(when the CRL was disabled and a force-rebuild was requested). Add a
comment about this case.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow fetching the specified issuer's CRL
We add a new API endpoint to fetch the specified issuer's CRL directly
(rather than the default issuer's CRL at /crl and /certs/crl). We also
add a new test to validate the CRL in a multi-root scenario and ensure
it is signed with the correct keys.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new PKI key prefix to seal wrapped storage (#15126)
* Refactor common backend initialization within backend_test
- Leverage an existing helper method within the PKI backend tests to setup a PKI backend with storage.
* Add ability to read legacy cert bundle if the migration has not occurred on secondaries.
- Track the migration state forbidding an issuer/key writing api call if we have not migrated
- For operations that just need to read the CA bundle, use the same tracking variable to
switch between reading the legacy bundle or use the new key/issuer storage.
- Add an invalidation function that will listen for updates to our log path to refresh the state
on secondary clusters.
* Always write migration entry to trigger secondary clusters to wake up
- Some PR feedback and handle a case in which the primary cluster does
not have a CA bundle within storage but somehow a secondary does.
* Update CA Chain to report entire chain
This merges the ca_chain JSON field (of the /certs/ca_chain path) with
the regular certificate field, returning the root of trust always. This
also affects the non-JSON (raw) endpoints as well.
We return the default issuer's chain here, rather than all known issuers
(as that may not form a strict chain).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow explicit issuer override on roles
When a role is used to generate a certificate (such as with the sign/
and issue/ legacy paths or the legacy sign-verbatim/ paths), we prefer
that issuer to the one on the request. This allows operators to set an
issuer (other than default) for requests to be issued against,
effectively making the change no different from the users' perspective
as it is "just" a different role name.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for role-based issuer selection
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Expand NotAfter limit enforcement behavior
Vault previously strictly enforced NotAfter/ttl values on certificate
requests, erring if the requested TTL extended past the NotAfter date of
the issuer. In the event of issuing an intermediate, this behavior was
ignored, instead permitting the issuance.
Users generally do not think to check their issuer's NotAfter date when
requesting a certificate; thus this behavior was generally surprising.
Per RFC 5280 however, issuers need to maintain status information
throughout the life cycle of the issued cert. If this leaf cert were to
be issued for a longer duration than the parent issuer, the CA must
still maintain revocation information past its expiration.
Thus, we add an option to the issuer to change the desired behavior:
- err, to err out,
- permit, to permit the longer NotAfter date, or
- truncate, to silently truncate the expiration to the issuer's
NotAfter date.
Since expiration of certificates in the system's trust store are not
generally validated (when validating an arbitrary leaf, e.g., during TLS
validation), permit should generally only be used in that case. However,
browsers usually validate intermediate's validity periods, and thus
truncate should likely be used (as with permit, the leaf's chain will
not validate towards the end of the issuance period).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for expanded issuance behaviors
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add warning on keyless default issuer (#15178)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update PKI to new Operations framework (#15180)
The backend Framework has updated Callbacks (used extensively in PKI) to
become deprecated; Operations takes their place and clarifies forwarding
of requests.
We switch to the new format everywhere, updating some bad assumptions
about forwarding along the way. Anywhere writes are handled (that should
be propagated to all nodes in all clusters), we choose to forward the
request all the way up to the performance primary cluster's primary
node. This holds for issuers/keys, roles, and configs (such as CRL
config, which is globally set for all clusters despite all clusters
having their own separate CRL).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Kitography/vault 5474 rebase (#15150)
* These parts work (put in signature so that backend wouldn't break, but missing fields, desc, etc.)
* Import and Generate API calls w/ needed additions to SDK.
* make fmt
* Add Help/Sync Text, fix some of internal/exported/kms code.
* Fix PEM/DER Encoding issue.
* make fmt
* Standardize keyIdParam, keyNameParam, keyTypeParam
* Add error response if key to be deleted is in use.
* replaces all instances of "default" in code with defaultRef
* Updates from Callbacks to Operations Function with explicit forwarding.
* Fixes a panic with names not being updated everywhere.
* add a logged error in addition to warning on deleting default key.
* Normalize whitespace upon importing keys.
Authored-by: Alexander Scheel <alexander.m.scheel@gmail.com>
* Fix isKeyInUse functionality.
* Fixes tests associated with newline at end of key pem.
* Add alternative proposal PKI aliased paths (#15211)
* Add aliased path for root/rotate/:exported
This adds a user-friendly path name for generating a rotated root. We
automatically choose the name "next" for the newly generated root at
this path if it doesn't already exist.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add aliased path for intermediate/cross-sign
This allows cross-signatures to work.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add path for replacing the current root
This updates default to point to the value of the issuer with name
"next" rather than its current value.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove plural issuers/ in signing paths
These paths use a single issuer and thus shouldn't include the plural
issuers/ as a path prefix, instead using the singular issuer/ path
prefix.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Only warn if default issuer was imported
When the default issuer was not (re-)imported, we'd fail to find it,
causing an extraneous warning about missing keys, even though this
issuer indeed had a key.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing issuer sign/issue paths
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clean up various warnings within the PKI package (#15230)
* Rebuild CRLs on secondary performance clusters post migration and on new/updated issuers
- Hook into the backend invalidation function so that secondaries are notified of
new/updated issuer or migrations occuring on the primary cluster. Upon notification
schedule a CRL rebuild to take place upon the next process to read/update the CRL
or within the periodic function if no request comes in.
* Schedule rebuilding PKI CRLs on active nodes only
- Address an issue that we were scheduling the rebuilding of a CRL on standby
nodes, which would not be able to write to storage.
- Fix an issue with standby nodes not correctly determining that a migration previously
occurred.
* Return legacy CRL storage path when no migration has occurred.
* Handle issuer, keys locking (#15227)
* Handle locking of issuers during writes
We need a write lock around writes to ensure serialization of
modifications. We use a single lock for both issuer and key
updates, in part because certain operations (like deletion) will
potentially affect both.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing b.useLegacyBundleCaStorage guards
Several locations needed to guard against early usage of the new issuers
endpoint pre-migration.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Address PKI to properly support managed keys (#15256)
* Address codebase for managed key fixes
* Add proper public key comparison for better managed key support to importKeys
* Remove redundant public key fetching within PKI importKeys
* Correctly handle rebuilding remaining chains
When deleting a specific issuer, we might impact the chains. From a
consistency perspective, we need to ensure the remaining chains are
correct and don't refer to the since-deleted issuer, so trigger a full
rebuild here.
We don't need to call this in the delete-the-world (DELETE /root) code
path, as there shouldn't be any remaining issuers or chains to build.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove legacy CRL bundle on world deletion
When calling DELETE /root, we should remove the legacy CRL bundle, since
we're deleting the legacy CA issuer bundle as well.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove deleted issuers' CRL entries
Since CRLs are no longer resolvable after deletion (due to missing
issuer ID, which will cause resolution to fail regardless of if an ID or
a name/default reference was used), we should delete these CRLs from
storage to avoid leaking them.
In the event that this issuer comes back (with key material), we can
simply rebuild the CRL at that time (from the remaining revoked storage
entries).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add unauthed JSON fetching of CRLs, Issuers (#15253)
Default to fetching JSON CRL for consistency
This makes the bare issuer-specific CRL fetching endpoint return the
JSON-wrapped CRL by default, moving the DER CRL to a specific endpoint.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Add JSON-specific endpoint for fetching issuers
Unlike the unqualified /issuer/:ref endpoint (which also returns JSON),
we have a separate /issuer/:ref/json endpoint to return _only_ the
PEM-encoded certificate and the chain, mirroring the existing /cert/ca
endpoint but for a specific issuer. This allows us to make the endpoint
unauthenticated, whereas the bare endpoint would remain authenticated
and usually privileged.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Add tests for raw JSON endpoints
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add unauthenticated issuers endpoints to PKI table
This adds the unauthenticated issuers endpoints?
- LIST /issuers,
- Fetching _just_ the issuer certificates (in JSON/DER/PEM form), and
- Fetching the CRL of this issuer (in JSON/DER/PEM form).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add issuer usage restrictions bitset
This allows issuers to have usage restrictions, limiting whether they
can be used to issue certificates or if they can generate CRLs. This
allows certain issuers to not generate a CRL (if the global config is
with the CRL enabled) or allows the issuer to not issue new certificates
(but potentially letting the CRL generation continue).
Setting both fields to false effectively forms a soft delete capability.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* PKI Pod rotation Add Base Changelog (#15283)
* PKI Pod rotation changelog.
* Use feature release-note formatting of changelog.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
Co-authored-by: Kit Haines <kit.haines@hashicorp.com>
Co-authored-by: kitography <khaines@mit.edu>
* add wrapping key endpoint
* change how wrapping key is stored
* move wrapping key func to backend
* refactor wrapping key generation
* Initial unit tests for Transit wrapping key endpoint
* Wire up wrapping key unit tests to actual implementation.
* Clean up Transit BYOK wrapping key tests and imports.
* Fix Transit wrapping key endpoint formatting.
* Update transit wrapping key to use lock manager for safe concurrent use.
* Rename some Transit wrapping key variables. Ensure the Transit wrapping key is correctly typed and formatted in a unit test.
* Fix spacing issue in Transit wrapping key endpoint help string.
Co-authored-by: rculpepper <rculpepper@hashicorp.com>
* Allow callers to choose the entropy source for the random endpoints
* Put source in the URL for sys as well
* changelog
* docs
* Fix unit tests, and add coverage
* refactor to use a single common implementation
* Update documentation
* one more tweak
* more cleanup
* Readd lost test expected code
* fmt
VAULT-5827 Don't prepare SQL queries before executing them
We don't support proper prepared statements, i.e., preparing once and
executing many times since we do our own templating. So preparing our
queries does not really accomplish anything, and can have severe
performance impacts (see
https://github.com/hashicorp/vault-plugin-database-snowflake/issues/13
for example).
This behavior seems to have been copy-pasted for many years but not for
any particular reason that we have been able to find. First use was in
https://github.com/hashicorp/vault/pull/15
So here we switch to new methods suffixed with `Direct` to indicate
that they don't `Prepare` before running `Exec`, and switch everything
here to use those. We maintain the older methods with the existing
behavior (with `Prepare`) for backwards compatibility.
* Bootstrap Nomad ACL system if no token is given
Similar to the [Bootstrap the Consul ACL system if no token is given][boostrap-consul]
it would be very useful to bootstrap Nomads ACL system and manage it in
Vault.
[boostrap-consul]:https://github.com/hashicorp/vault/pull/10751
* Add changelog entry
* Remove debug log line
* Remove redundant else
* Rename Nomad acl bootstrap param
* Replace sleep with attempt to list nomad leader, setup will retry until successful
* fmt
- As part of the PKI rotation project we need to hook into some of the functions
that were factored out for managed keys in regards to key handling within the
CA bundles.
- Refactor the codebase so that we only extract managed key stuff from oss/ent
and not additional business logic.
VAULT-5827 Update mongodb, brotli
Closes https://github.com/hashicorp/vault-plugin-secrets-mongodbatlas/issues/11
* `brotli` 1.0.1 was withdrawn
* `go-client-mongodb-atlas` has an old dependency on a renamed repo, and
has been renamed twice. This caused issues in
https://github.com/hashicorp/vault-plugin-secrets-mongodbatlas/issues/11
for example.
* VAULT-5827 Set unwrap token during database tests
The unwrap token is necessary for the plugins to start correctly when
running when running acceptance tests locally, e.g.,
```
$ VAULT_MONGODBATLAS_PROJECT_ID=... VAULT_MONGODBATLAS_PRIVATE_KEY=... VAULT_MONGODBATLAS_PUBLIC_KEY=... TEST='-run TestBackend_StaticRole_Rotations_MongoDBAtlas github.com/hashicorp/vault/builtin/logical/database' make test
--- FAIL: TestBackend_StaticRole_Rotations_MongoDBAtlas (5.33s)
rotation_test.go:818: err:%!s(<nil>) resp:&logical.Response{Secret:<nil>, Auth:<nil>, Data:map[string]interface {}{"error":"error creating database object: invalid database version: 2 errors occurred:\n\t* Unrecognized remote plugin message: PASS\n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.\n\t* Incompatible API version with plugin. Plugin version: 5, Client versions: [3 4]\n\n"}, Redirect:"", Warnings:[]string(nil), WrapInfo:(*wrapping.ResponseWrapInfo)(nil), Headers:map[string][]string(nil)}
```
Note the `PASS` message there, which indicates that the plugin exited
before starting the RPC server.
* Warnings indicating ignored and replaced parameters
* Avoid additional var creation
* Add warnings only if the response is non-nil
* Return the response even when error is non-nil
* Fix tests
* Rearrange comments
* Print warning in the log
* Fix another test
* Add CL
When adding SignatureBits control logic, we incorrectly allowed
specification of SignatureBits in the case of an ECDSA issuer. As noted
in the original request, NIST and Mozilla (and others) are fairly
prescriptive in the choice of signatures (matching the size of the
NIST P-curve), and we shouldn't usually use a smaller (or worse, larger
and truncate!) hash.
Ignore the configuration of signature bits and always use autodetection
for ECDSA like ed25519.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add PKI test for delete role
- Create a role, validate that defaults are what we expect
and delete the role, verifying it is gone on subsequent read
attempts.
* Add PKI test for crl/rotate command
- Missing a unit test that validates the crl/rotate command works. The test validates the rotate command was successful
by checking if we have a different/new update time on the CRL.
* Rework PKI TestBackend_PathFetchValidRaw test to not write directly to storage
- Rework the existing test to not write directly to storage as we might change that in the future.
- Add tests that validate the ca_chain behaviour of not returning the root authority cert
* PR Feedback
* Additional PR feedback
* Correctly handle minimums, default SignatureBits
When using KeyType = "any" on a role (whether explicitly or implicitly
via a sign-verbatim like operation), we need to update the value of
SignatureBits from its new value 0 to a per-key-type default value. This
will allow sign operations on these paths to function correctly, having
the correctly inferred default signature bit length.
Additionally, this allows the computed default value for key type to be
used for minimum size validation in the RSA/ECDSA paths. We additionally
enforce the 2048-minimum in this case as well.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix defaults and validation of "any" KeyType
When certutil is given the placeholder any keytype, it attempts to
validate and update the default zero value. However, in lacking a
default value for SignatureBits, it cannot update the value from the
zero value, thus causing validation to fail.
Add more awareness to the placeholder "any" value to certutil.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add role-based regression tests for key bits
This adds regression tests for Key Type, Key Bits, and Signature Bits
parameters on the role. We test several values, including the "any"
value to ensure it correctly restricts key sizes.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add sign-verbatim test for key type
This ensures that we test sign-verbatim against a variety of key types.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Fix semgrep 0.86.5 parsing failures
- semgrep https://github.com/returntocorp/semgrep/pull/4671 seems to have
introduce this parsing failure within version 0.86.0 and higher
- Workaround parsing failure by breaking out the if error check.
* Pin semgrep version to 0.86.5
* Fix formatting issues
The URL password redaction operation did not handle the case where the
database connection URL was provided as a percent-encoded string, and
its password component contained reserved characters. It attempted to
redact the password by replacing the unescaped password in the
percent-encoded URL. This resulted in the password being revealed when
reading the configuration from Vault.
* Misc PKI code fixes.
- Harden the code base a bit adding default's to switch statements
to various error handlers and processing statements.
- Fixup some error messages to include proper values we support.
* Additional default case missing within PKI
* Fix typo in PKI error message
* Add error check when looking up public key info for a managed key within PKI
* Trap use case that an unknown public key is returned to PKI through managed keys
Instead of using the field FieldData.Raw, use method GetOkError() which does
type conversion but still allows to check whether a value for the parameter was
provided. Note that GetOkError() converts nil values to default or zero values,
so, for example, a nil plaintext value will result in the empty string being
encrypted.
* Update description of certificate fetch API
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify /config/crl and /config/url PKI are empty
GET-ing these URLs will return 404 until such time as a config is posted
to them, even though (in the case of CRL), default values will be used.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify usage of /pki/crl/rotate
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update documentation around PKI key_bits
This unifies the description of key_bits to match the API description
(which is consistent across all usages).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix indented field descriptions in PKI paths
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify documentation around serial_number
Note that this field has no impact on the actual Serial Number field and
only an attribute in the requested certificate's Subject.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix spelling of localdomain
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
As reported by Steve Clark, building an intermediate mount in PKI (and
calling /intermediate/set-signed) results in a duplicate intermediate CA
certificate in the full chain output (ca_chain field of the
/cert/ca_chain API endpoint response).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add warning when generate_lease=no_store=true
When no_store=true, the value of generate_lease is ignored completely
(and set to false). This means that when generate_lease=true is
specified by the caller of the API, it is silently swallowed. While
changing the behavior could break callers, setting a warning on the
response (changing from a 204->200 in the process) seems to make the
most sense.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
As pointed out internally, a lot of the API docs and FrameworkField
descriptions of parameters were out of date. This syncs a number of
them, updating their descriptions where relevant.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new AllowWildcardCertificate field to PKI role
This field allows the PKI role to control whether or not issuance of
wildcard certificates are allowed. We default (both on migration and
new role creation) to the less secure true value for backwards
compatibility with existing Vault versions.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor sanitizedName to reducedName
Per comment, this variable name was confusing during the reproduction
and subsequent fix of the earlier vulnerability and associated bug
report. Because the common name isn't necessarily _sanitized_ in any way
(and indeed must be considered in relation to other parts or the whole),
but portions of the entire name are removed, reducedName appears to make
the most sense.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Enforce AllowWildcardCertificates during issuance
This commit adds the bulk of correctly validating wildcard certificate
Common Names during issuance according to RFC 6125 Section 6.4.3
semantics. As part of this, support for RFC 2818-conforming wildcard
certificates (wherein there are almost no restrictions on issuance) has
been removed.
Note that this flag does take precedence over AllowAnyName, giving a
little more safety in wildcard issuance in this case.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update test cases to conform with RFC 6125
Test cases 19, 70+71, and 83+84 didn't conform with the RFC 6125, and so
should've been rejected under strict conformance. For 70+71 and 83+84,
we previously conditioned around the value of AllowSubdomains (allowing
issuance when true), but they likely should've been rejected either way.
Additionally, update the notes about globs matching wildcard
certificates to notate this is indeed the case.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Check AllowWildcardCertifciates in issuance tests
This allows for regression tests to cover the new
AllowWildcardCertificate conditional. We add additional test cases
ensuring that wildcard issuance is properly forbidden in all relevant
scenarios, while allowing the existing test cases to validate that
wildcard status doesn't affect non-wildcard certificates.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add Wildcard allowance during signing operations
When using sign-verbatim, sign-intermediate, or getting certificate
generation parameters, set AllowWildcardCertificates to mirror existing
policies.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow issuance of wildcard via glob match
From Vault v1.8.0 onwards, we would incorrectly disallow issuance of a
wildcard certificate when allow_glob_domain was enabled with a
multi-part glob domain in allowed_domains (such as *.*.foo) when
attempting to issue a wildcard for a subdomain (such as *.bar.foo).
This fixes that by reverting an errant change in the case insensitivity
patch. Here, when validating against a very powerful glob construct, we
leave the wildcard prefix (*.) from the raw common_name element, to
allow multi-part globs to match wildcard entries.
It is important to note that "sanitizedName" is an incorrect variable
naming here. Wildcard parsing (per RFC 6125 which supercedes RFC 2818)
must be in the left-most segment of the domain, but we lack validation
to ensure no internal wildcards exist. Additionally per item 3 of
section 6.4.3 of RFC 6125, wildcards MAY be internal to a domain
segment, in which case sanitizedName again leaves the wildcard in place.
Resolves: #13530
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove duplicate email address check
As pointed out by Steven Clark (author of the removed conditional in
70012cd865b3dcdab376dba0c0e0abc88c48f508), this is duplicate from the
now-reintroduced comparison against name (versus the erroneous
sanitizedName at the time of his commit).
This is a reversion of the changes to builtin/logical/pki/cert_util.go,
but keeping the additional valuable test cases.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add multi-dimensional PKI role issuance tests
This commit introduces multi-dimensional testing of PKI secrets engine's
role-based certificate issuance with the intent of preventing future
regressions.
Here, dimensions of testing include:
- AllowedDomains to decide which domains are approved for issuance,
- AllowBareDomains to decide if raw entries of AllowedDomains are
permitted,
- AllowGlobDomains to decide if glob patterns in AllowedDomains are
parsed,
- AllowSubdomains to decide if subdomains of AllowedDomains are
permitted,
- AllowLocalhost to decide if localhost identifiers are permitted, and
- CommonName of the certificate to request.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Add checks for other error types within the PKI plugin
- The PKI plugin assumes the code it is calling always returns an error
of type errutil.UserError or errutil.InternalError. While I believe
so far this is still true, it would be easy to add a code path that
just returns a generic error and we would completely ignore it.
- This was found within some managed key testing where I forgot to wrap
an error within one of the expected types
* Add changelog
* Allow OpenSSH-style key type identifiers
To bring better parity with the changes of #14008, wherein we allowed
OpenSSH-style key identifiers during generation. When specifying a list
of allowed keys, validate against both OpenSSH-style key identifiers
and the usual simplified names as well ("rsa" or "ecdsa"). Notably, the
PKI secrets engine prefers "ec" over "ecdsa", so we permit both as well.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix missing quote in docs
* Explicitly call out SSH algorithm_signer default
Related: #11608
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Use rsa-sha2-256 as the default SSH CA hash algo
As mentioned in the OpenSSH 8.2 release notes, OpenSSH will no longer be
accepting ssh-rsa signatures by default as these use the insecure SHA-1
algorithm.
For roles in which an explicit signature type wasn't specified, we
should change the default from SHA-1 to SHA-256 for security and
compatibility with modern OpenSSH releases.
See also: https://www.openssh.com/txt/release-8.2
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update docs mentioning new algorithm change
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix missing parenthesis, clarify new default value
* Add to side bar
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* plugin/catalog: support plugin registration whe type is explicitly provided
* don't use database type on plugin backend test; mock doesn't satisfy the DB interface
* check multiplexing support from plugin directly on newPluginClient
* do not return mutiplexed bool on catalog helper funcs
* Allow specifying multiple allowed SSH key lengths
In the ssh secrets engine, only a single allowed key length was allowed
for each algorithm type. However, many algorithms have multiple safe
values (such as RSA and ECDSA); allowing a single role to have multiple
values for a single algorithm is thus helpful.
On creation or update, roles can now specify multiple types using a list
or comma separated string of allowed values:
allowed_user_key_lengths: map[string][]int{"rsa": []int{2048, 4096}}
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Break out ssh upgrade logic into separate function
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update parseutil for optional lists of integers
go get -u github.com/hashicorp/go-secure-stdlib/parseutil
go mod tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Simplify parse logic using new parseutil
The newly introduced parseutil.ParseIntSlice handles the more
complicated optional int-like slice logic for us.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* adds development workflow to mirage config
* adds mirage handler and factory for mfa workflow
* adds mfa handling to auth service and cluster adapter
* moves auth success logic from form to controller
* adds mfa form component
* shows delayed auth message for all methods
* adds new code delay to mfa form
* adds error views
* fixes merge conflict
* adds integration tests for mfa-form component
* fixes auth tests
* updates mfa response handling to align with backend
* updates mfa-form to handle multiple methods and constraints
* adds noDefault arg to Select component
* updates mirage mfa handler to align with backend and adds generator for various mfa scenarios
* adds tests
* flaky test fix attempt
* reverts test fix attempt
* adds changelog entry
* updates comments for todo items
* removes faker from mfa mirage factory and handler
* adds number to word helper
* fixes tests
* Revert "Merge branch 'main' into ui/mfa"
This reverts commit 8ee6a6aaa1b6c9ec16b985c10d91c3806819ec40, reversing
changes made to 2428dd6cca07bb41cda3f453619646ca3a88bfd0.
* format-ttl helper fix from main
* feat: DB plugin multiplexing (#13734)
* WIP: start from main and get a plugin runner from core
* move MultiplexedClient map to plugin catalog
- call sys.NewPluginClient from PluginFactory
- updates to getPluginClient
- thread through isMetadataMode
* use go-plugin ClientProtocol interface
- call sys.NewPluginClient from dbplugin.NewPluginClient
* move PluginSets to dbplugin package
- export dbplugin HandshakeConfig
- small refactor of PluginCatalog.getPluginClient
* add removeMultiplexedClient; clean up on Close()
- call client.Kill from plugin catalog
- set rpcClient when muxed client exists
* add ID to dbplugin.DatabasePluginClient struct
* only create one plugin process per plugin type
* update NewPluginClient to return connection ID to sdk
- wrap grpc.ClientConn so we can inject the ID into context
- get ID from context on grpc server
* add v6 multiplexing protocol version
* WIP: backwards compat for db plugins
* Ensure locking on plugin catalog access
- Create public GetPluginClient method for plugin catalog
- rename postgres db plugin
* use the New constructor for db plugins
* grpc server: use write lock for Close and rlock for CRUD
* cleanup MultiplexedClients on Close
* remove TODO
* fix multiplexing regression with grpc server connection
* cleanup grpc server instances on close
* embed ClientProtocol in Multiplexer interface
* use PluginClientConfig arg to make NewPluginClient plugin type agnostic
* create a new plugin process for non-muxed plugins
* feat: plugin multiplexing: handle plugin client cleanup (#13896)
* use closure for plugin client cleanup
* log and return errors; add comments
* move rpcClient wrapping to core for ID injection
* refactor core plugin client and sdk
* remove unused ID method
* refactor and only wrap clientConn on multiplexed plugins
* rename structs and do not export types
* Slight refactor of system view interface
* Revert "Slight refactor of system view interface"
This reverts commit 73d420e5cd2f0415e000c5a9284ea72a58016dd6.
* Revert "Revert "Slight refactor of system view interface""
This reverts commit f75527008a1db06d04a23e04c3059674be8adb5f.
* only provide pluginRunner arg to the internal newPluginClient method
* embed ClientProtocol in pluginClient and name logger
* Add back MLock support
* remove enableMlock arg from setupPluginCatalog
* rename plugin util interface to PluginClient
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* feature: multiplexing: fix unit tests (#14007)
* fix grpc_server tests and add coverage
* update run_config tests
* add happy path test case for grpc_server ID from context
* update test helpers
* feat: multiplexing: handle v5 plugin compiled with new sdk
* add mux supported flag and increase test coverage
* set multiplexingSupport field in plugin server
* remove multiplexingSupport field in sdk
* revert postgres to non-multiplexed
* add comments on grpc server fields
* use pointer receiver on grpc server methods
* add changelog
* use pointer for grpcserver instance
* Use a gRPC server to determine if a plugin should be multiplexed
* Apply suggestions from code review
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* add lock to removePluginClient
* add multiplexingSupport field to externalPlugin struct
* do not send nil to grpc MultiplexingSupport
* check err before logging
* handle locking scenario for cleanupFunc
* allow ServeConfigMultiplex to dispense v5 plugin
* reposition structs, add err check and comments
* add comment on locking for cleanupExternalPlugin
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Ensure we can issue against generated SSH CA keys
This adds a test to ensure that we can issue leaf SSH certificates using
the newly generated SSH CA keys. Presently this fails because the
ed25519 key private is stored using PKIX's PKCS8 PrivateKey object
format rather than using OpenSSH's desired private key format:
> path_config_ca_test.go:211: bad case 12: err: failed to parse stored CA private key: ssh: invalid openssh private key format, resp: <nil>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add dependency on edkey for OpenSSH ed25519 keys
As mentioned in various terraform-provider-tls discussions, OpenSSH
doesn't understand the standard OpenSSL/PKIX ed25519 key structure (as
generated by PKCS8 marshalling). Instead, we need to place it into the
OpenSSH RFC 8709 format. As mentioned in this dependency's README,
support in golang.org/x/crypto/ssh is presently lacking for this.
When the associated CL is merged, we should be able to remove this dep
and rely on the (extended) standard library, however, no review progress
appears to have been made since the CL was opened by the author.
See also: https://go-review.googlesource.com/c/crypto/+/218620/
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add generation support for other SSH CA key types
This adds two new arguments to config/ca, mirroring the values of PKI
secrets engine but tailored towards SSH mounts. Key types are specified
as x/crypto/ssh KeyAlgo identifiers (e.g., ssh-rsa or ssh-ed25519)
and respect current defaults (ssh-rsa/4096). Key bits defaults to 0,
which for ssh-rsa then takes a value of 4096.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation on key_type, key_bits for ssh/config/ca
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow all other_sans in sign-intermediate and sign-verbatim
/sign-verbatim and /sign-intermediate are more dangerous endpoints in
that they (usually) do not have an associated role. In this case, a
permissive role is constructed during execution of these tests. However,
the AllowedOtherSANs field was missing from this, prohibiting its use
when issuing certificates.
Resolves: #13157
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add duration/count metrics to PKI issue and revoke flows
* docs, changelog
* tidy
* last tidy
* remove err
* Update callsites
* Simple returns
* Handle the fact that test cases don't have namespaces
* Add mount point to the request
* fmt
* Handle empty mount point, and add it to unit tests
* improvement
* Turns out sign-verbatim is tricky, it can take a role but doesn't have to
* Get around the field schema problem
* Use application/pem-certificate-chain for PEMs
As mentioned in #10948, it appears we're incorrectly using the
`application/pkix-cert` media type for PEM blobs, when
`application/x-pem-file` is more appropriate. Per RFC 5280 Section
4.2.1.13, `application/pkix-crl` is only appropriate when the CRL is in
DER form. Likewise, Section 4.2.2.1 states that `application/pkix-cert`
is only applicable when a single DER certificate is used.
Per recommendation in RFC 8555 ("ACME"), Section 7.4.2 and 9.1, we use
the newer `application/pem-certificate-chain` media type for
certificates. However, this is not applicable for CRLs, so we use fall
back to `application/x-pem-file` for these. Notably, no official IETF
source is present for the latter. On the OpenSSL PKI tutorial
(https://pki-tutorial.readthedocs.io/en/latest/mime.html), this type is
cited as coming from S/MIME's predecessor, PEM, but neither of the main
PEM RFCs (RFC 934, 1421, 1422, 1423, or 1424) mention this type.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Include full chain in /cert/ca_chain response
This allows callers to get the full chain (including issuing
certificates) from a call to /cert/ca_chain. Previously, most endpoints
(including during issuance) do not include the root authority, requiring
an explicit call to /cert/ca to fetch. This allows full chains to be
constructed without without needing multiple calls to the API.
Resolves: #13489
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test case for full CA issuance
We test three main scenarios:
1. A root-only CA's `/cert/ca_chain`'s `.data.ca_chain` field should
contain only the root,
2. An intermediate CA (with root provide) should contain both the root
and the intermediate.
3. An external (e.g., `/config/ca`-provided) CA with both root and
intermediate should contain both certs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation for new ca_chain field
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add note about where to find the entire chain
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Support Y10K value in notAfter field when signing non-CA certificates
* Add changelog entry for 13736
* Add test for using not_after parameter for non-CA certificates that are being signed
* Fix CA value for test for not_after value when signing non-CA certs
* Address formatting
* Add changelog file
* Revert changelog entry commit f28b54e7b5ad21144c8a2da942d766e64a332caf
* Fix a transit deadlock caused by indefinite lock holding in key autorotation.
* Move down manual policy locking in transit autorotation to avoid NPE.
* Wrap conditional transit key autorotation in a function to allow for cleaner policy lock management.
* Remove a dnagling continue statement from transit key autorotation.
* Fix upndomain bug causing alias name to change
* Fix nil map
* Add changelog
* revert
* Update changelog
* Add test for alias metadata name
* Fix code comment
* Add auto_rotate_interval field to transit key creation path.
* Add auto_rotate_interval field to transit key config update path.
* Implement transit automatic key rotation on an hourly interval.
* Fixes transit key autorotation key listing typo.
* Add unit tests for transit key autorotation.
* Add unit tests for transit key creation with autorotation interval.
* Add unit tests for transit key config update with autorotation interval.
* Document new auto_rotate_interval fields in key creation and key config update endpoints.
* Add changelog for transit key autorotation.
* Wrap individual transit key autorotation in a policy lock.
* Add a safeguard to transit key autorotation to ensure only one execution happens simultaneously.
* auth/cert: Add certificate extensions as metadata
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
* Add changelog for #13348
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
* CLI changes for new mount tune config parameter allowed_managed_keys
* Correct allowed_managed_keys description in auth and secrets
* Documentation update for secrets and removed changes for auth
* Add changelog and remove documentation changes for auth
* removed changelog
* Correct the field description
* Replace - with : when listing certificate serials
* Add allowed_uri_sans_template
Enables identity templating for the allowed_uri_sans field in PKI cert roles.
Implemented as suggested in #8509
* changelog++
* Update docs with URI SAN templating
* github auth: use org id to verify creds
* add check for required org param; add test case
* update UTs
* add nil check for org
* add changelog
* fix typo in ut
* set org ID if it is unset; add more ut coverage
* add optional organization_id
* move client instantiation
* refactor parse URL; add UT for setting org ID
* fix comment in UT
* add nil check
* don't update org name on change; return warning
* refactor verifyCredentials
* error when unable to fetch org ID on config write; add warnings
* fix bug in log message
* update UT and small refactor
* update comments and log msg
* use getter for org ID
* Allow universal default for key_bits
This allows the key_bits field to take a universal default value, 0,
which, depending on key_type, gets adjusted appropriately into a
specific default value (rsa->2048, ec->256, ignored under ed25519).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Handle universal default key size in certutil
Also move RSA < 2048 error message into certutil directly, instead of in
ca_util/path_roles.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing RSA key sizes to pki/backend_test.go
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to returning updated values
When determining the default, don't pass in pointer types, but instead
return the newly updated value.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Re-add fix for ed25519 from #13254
Ed25519 internally specifies a hash length; by changing the default from
256 to 0, we fail validation in ValidateSignatureLength(...) unless we
specify the key algorithm.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- The test was attempting to test the convergent encryption behaviour
with several key types but the common function never used the passed
in key type. So we ran the test with the default aes256-gcm96 only.
* Warn user supplying nonce values in FIPS mode for transit encryption requests
- Send back a warning within the response if an end-user supplies nonce
values that we use within the various transit encrypt apis.
- We do not send a warning if an end-user supplies a nonce value but we
don't use it.
- Affected api methods are encrypt, rewrap and datakey
- The warning is only sent when we are operating in FIPS mode.
* Adding support for SHA3 in the transit backend.
* Adds SHA-3 tests for transit sign/verify path. Adds SHA-3 tests for logical system tools path hash functionality. Updates documentation to include SHA-3 algorithms in system tools path hashing.
* Adds changelog entry.
Co-authored-by: robison jacka <robison@packetized.io>
Update the AWS auth backend acceptance tests to account for the new
`iam_tags` field that comes back on responses.
* marked only tests requiring creds as acceptance
Renamed TestBackend_* to TestAcceptanceBackend_* if the test requires
AWS credentials. Otherwise left the name as TestBackend_* and set
`AcceptanceTest: false`.
* ensure generated names aren't too long
IAM roles and users have a 64 character limit, and adding Acceptance
to the test names was putting some over the length limit, so modified
generateUniqueName() to take a max length parameter and added
functions for each type of name generation (user, role, group).
* return pkcs8 format for ed25519 curve
convertRespToPKCS8 does not recognize the ed25519 key. Changes
to recognize ed25519 key and return its PKCS8 format
* Restrict ECDSA signatures with NIST P-Curve hashes
When using an ECDSA signature with a NIST P-Curve, we should follow
recommendations from BIS (Section 4.2) and Mozilla's root store policy
(section 5.1.2) to ensure that arbitrary selection of signature_bits
does not exceed what the curve is capable of signing.
Related: #11245
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to certutil.ValidateKeyTypeSignatureLength(...)
Replaces previous calls to certutil.ValidateKeyTypeLength(...) and
certutil.ValidateSignatureLength(...) with a single call, allowing for
curve<->hash validation.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to autodetection of signature_bits
This enables detection of whether the caller manually specified a value
for signature_bits or not; when not manually specified, we can provision
a value that complies with new NIST P-Curve policy.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Select hash function length automatically
Due to our change in behavior (to default to -1 as the value to
signature_bits to allow for automatic hash selection), switch
ValidateKeyTypeSignatureLength(...) to accept a pointer to hashBits and
provision it with valid default values.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Prevent invalid Curve size lookups
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch from -1 to 0 as default SignatureBits
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
VAULT-444: Add PKI tidy-status endpoint.
Add metrics so that the PKI tidy status can be monitored using telemetry as well.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Document allow_different_signature_algorithm param
* Flip the semantics of different key types for sign self issued
* More language tweaks
* Fix the field definition description
* Rework differenttype test for the new flag
* typo
* Native Login method, userpass and approle interfaces to implement it
* Add AWS auth interface for Login, unexported struct fields for now
* Add Kubernetes client login
* Add changelog
* Add a test for approle client login
* Return errors from LoginOptions, use limited reader for secret ID
* Fix auth comment length
* Return actual type not interface, check for client token in tests
* Require specification of secret ID location using SecretID struct as AppRole arg
* Allow password from env, file, or plaintext
* Add flexibility in how to fetch k8s service token, but still with default
* Avoid passing strings that need to be validated by just having different login options
* Try a couple real tests with approle and userpass login
* Fix method name in comment
* Add context to Login methods, remove comments about certain sources being inherently insecure
* Perform read of secret ID at login time
* Read password from file at login time
* Pass context in integ tests
* Read env var values in at login time, add extra tests
* Update api version
* Revert "Update api version"
This reverts commit 1ef3949497dcf878c47e0e5ffcbc8cac1c3c1679.
* Update api version in all go.mod files
* Modernize SSH key lengths
No default change was made in this commit; note that the code already
enforced a default of 2048 bits. ssh-keygen and Go's RSA key generation
allows for key sizes including 3072, 4096, 8192; update the values of
SSH key generation to match PKI's allowed RSA key sizes (from
certutil.ValidateKeyTypeLength(...)). We still allow the legacy SSH key
size of 1024; in the near future we should likely remove it.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Ensure minimum of 2048-bit PKI RSA keys
While the stated path is a false-positive, verifying all paths is
non-trivial. We largely validate API call lengths using
certutil.ValidateKeyTypeLength(...), but ensuring no other path calls
certutil.generatePrivateKey(...) --- directly or indirectly --- is
non-trivial. Thus enforcing a minimum in this method sounds like a sane
compromise.
Resolves: https://github.com/hashicorp/vault/security/code-scanning/55
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Let allowed_users template mix templated and non-templated parts (#10388)
* Add documentation
* Change test function names
* Add documentation
* Add changelog entry
* Forbid ssh key signing with specified extensions when role allowed_extensions is not set
- This is a behaviour change on how we process the allowed_extensions role
parameter when it does not contain a value. The previous handling allowed
a client to override and specify any extension they requested.
- We now require a role to explicitly set this behaviour by setting the parameter
to a '*' value which matches the behaviour of other keys such as allowed_users
within the role.
- No migration of existing roles is provided either, so operators if they truly
want this behaviour will need to update existing roles appropriately.
* update azure instructions
Update instructions in regards to azure AD Authentication and OIDC
* Initial pass of ed25519
* Fix typos on marshal function
* test wip
* typo
* fix tests
* missef changelog
* fix mismatch between signature and algo
* added test coverage for ed25519
* remove pkcs1 since does not exist for ed25519
* add ed25519 support to getsigner
* pull request feedback
Signed-off-by: Anner J. Bonilla <abonilla@hoyosintegrity.com>
* typo on key
Signed-off-by: Anner J. Bonilla <abonilla@hoyosintegrity.com>
* cast mistake
Signed-off-by: Anner J. Bonilla <abonilla@hoyosintegrity.com>
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* core: set namespace from the sysview's mount entry on GeneratePasswordFromPolicy
* test: update TestDynamicSystemView to be ns-aware, update tests
* add changelog entry
* port of ldap fix for early cred rotation
* some more porting
* another couple lines to port
* final commits before report
* remove deadlock
* needs testing
* updates
* Sync with OpenLDAP PR
* Update the update error handling for items not found in the queue
* WIP unit tests
* Need to configure DB mount correctly, with db type mockv5
* Need to find a way to inject errors into that mock db
* throw error on role creation failure
* do not swallow error on role creation
* comment out wip tests and add in a test for disallowed role
* Use newly generated password in WAL
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* return err on popFromRotationQueueByKey error; cleanup on setStaticAccount
* test: fix TestPlugin_lifecycle
* Uncomment and fix unit tests
* Use mock database plugin to inject errors
* Tidy test code to rely less on code internals where possible
* Some stronger test assertions
* Undo logging updates
* Add changelog
* Remove ticker and background threads from WAL tests
* Keep pre-existing API behaviour of allowing update static role to act as a create
* Switch test back to update operation
* Revert my revert, and fix some test bugs
* Fix TestBackend_StaticRole_LockRegression
* clean up defer on TestPlugin_lifecycle
* unwrap reqs on cleanup
* setStaticAccount: don't hold a write lock
* TestStoredWALsCorrectlyProcessed: set replication state to unknown
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
* WIP: Unset the certificate's SignatureAlgorithm to allown cross-signing of different key types
* Allow signing self issued certs with a different public key algorithm
* Remove cruft
* Remove stale import
* changelog
* eliminate errwrap
* Add a test to cover the lack of opt-in flag
* Better comment
Co-authored-by: catsby <clint@ctshryock.com>
* Enforce Minimum cache size for transit backend
* enfore minimum cache size and log a warning during backend construction
* Update documentation for transit backend cache configuration
* Added changelog
* Addressed review feedback and added unit test
* Modify code in pathCacheConfigWrite to make use of the updated cache size
* Updated code to refresh cache size on transit backend without restart
* Update code to acquire read and write locks appropriately
This change adds the ability to set the signature algorithm of the
CAs that Vault generates and any certificates it signs. This is a
potentially useful stepping stone for a SHA3 transition down the line.
Summary:
* Adds the field "signature_bits" to CA and Sign endpoints
* Adds support for SHA256, SHA384 and SHA512 signatures on EC and RSA
keytypes.
* Fix pkcs7 parsing in some cases
brings in https://github.com/mozilla-services/pkcs7/pull/61 from upstream
In some cases but not all, aws includes a certificate in the pkcs7 response,
and currently vault fails to parse those certificates:
```
URL: PUT https://vault.example.com/v1/auth/aws/login
Code: 500. Errors
* failed to parse the BER encoded PKCS#7 signature: ber2der: Invalid BER format
```
This fixes logins on those instances. Note we could not readily ascertain why
some instances have those certificates and others don't.
* Add changelog entry
* Correct missed line
* Send x-forwarded-for in Okta Push Factor request
Why:
In order for Okta to properly report the location of the authentication
attempt, the X-Forwarded-For header must be included in the request to
Okta (if it exists).
This change addresses the need by:
* Duplicating the value of X-Forwarded-For if it's passed through to the
auth backend
* Add changelog entry for 12320
* update genUsername to cap STS usernames at 64 chars
* add changelog
* refactor tests into t.Run block
* patch: remove warningExpected bool and include expected string
* patch: revert sts to cap at 32 chars and add assume_role case in genUsername
* update changelog
* update genUsername to return error if username generated exceeds length limits
* update changelog
* add conditional default username template to provide custom STS usernames
* update changelog
* include test for failing STS length case
* update comments for more clarity
* Work around rabbitmq regression with UserInfo.Tags in rabbitmq 3.9: use v3.8 docker image in tests.
* Also pin cassandra docker image version to 3.11 (4.00 was making tests fail)
Unlike the other libraries that were migrated, there are no usages of
this lib in any of our plugins, and the only other known usage was in
go-kms-wrapping, which has been updated. Aliasing it like the other libs
would still keep the aws-sdk-go dep in the sdk module because of the
function signatures. So I've simply removed it entirely here.
* add ability to customize IAM usernames based on templates
* add changelog
* remove unnecessary logs
* patch: add test for readConfig
* patch: add default STS Template
* patch: remove unnecessary if cases
* patch: add regex checks in username test
* patch: update genUsername to return an error instead of warnings
* patch: separate tests for default and custom templates
* patch: return truncate warning from genUsername and trigger a 400 response on errors
* patch: truncate midString to 42 chars in default template
* docs: add new username_template field to aws docs
* fix: cap token TTL at login time based on default lease TTL
* add changelog file
* patch: update warning messages to not include 'at login'
* patch: remove default lease capping and test
* update changelog
* patch: revert warning message
* initializing resp variable with aa *logical.Response before using it to add warning for default-service or default-batch token type. Also adding guard around code that sets resp to a new logical.Response further on in the function.
* adding changelog entry
* renaming changelog file to match PR number
* add username customization for rabbitmq
* add changelog for rabbitmq
* Update builtin/logical/rabbitmq/path_config_connection.go
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* updating API docs
* moved to changelog folder
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* approle: convert Callbacks to Operations
The usage of oldstyle "Callbacks" is causing the `cannot write to readonly
storage` error message when `login` is attempted against a performance standby.
Use the newstyle "Operations" and additionally set the Forward
parameters to forward the request to the Active vault node.
* add changelog
* do not forward for alias lookahead operation
* remove forward fields and remove changelog
- Because this request is an UpdateOperation, it should have automatically been
routed to the primary/active by the router before it reaches the backend.
- changelog should not be needed as this change is only a refactor with
no user-facing behavior changes.
* feat(aws): add ability to provide a sessionName to sts credentials
Co-authored-by: Brad Vernon <bvernon@nvidia.com>
Co-authored-by: Jim Kalafut <jim@kalafut.net>
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Add support for templated values in SSH CA DefaultExtensions.
* Reworking the logic per feedback, adding basic test.
* Adding test, so we cover both default extension templating & ignoring default when user-provided extensions are present.
* Fixed up an unintentional extension handling defect, added test to cover the case.
* Refactor Default Extension tests into `enabled` and `disabled`.
* Replace deprecated terms in AWS Auth
This PR is part of an effort to remove non-inclusive language throughout
Vault. The AWS Auth backend uses the "whitelist" and "blacklist" term
extensively, and these are the focus of the PR:
* Add new API endpoints that use the preferred terminology, while
deprecating the old endpoints. These endpoints offer identical
functionality and are basically aliases. This is the only functional
change in the PR except for terms in error messages.
* Replace "whitelist" -> "access list", "blacklist" -> "deny list" in
variable names, comments, etc.
Note that storage locations were *not* changed at this time, as that is
a more complex process involving versioning that we may tackle in a future
revision. We have reduced the occurrences of non-inclusive language,
however.
Reviewers should be sure to "Ignore Whitespace" in diffs, especially for
the tests, which were basically indented one level as part of looping
over the tests with both the old and new names.
* Added support for iam_tags for AWS secret roles
This change allows iam_users generated by the secrets engine
to add custom tags in the form of key-value pairs to users
that are created.
* upgrade vault dependency set
* etcd and grpc issues:
* better for tests
* testing
* all upgrades for hashicorp deps
* kubernetes plugin upgrade seems to work
* kubernetes plugin upgrade seems to work
* etcd and a bunch of other stuff
* all vulnerable packages upgraded
* k8s is broken in linux env but not locally
* test fixes
* fix testing
* fix etcd and grpc
* fix etcd and grpc
* use master branch of go-testing-interface
* roll back etcd upgrade
* have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd
* rolling back in the replace directives
* a few more testing dependencies to clean up
* fix go mod vendor
* Send a test message before committing a new audit device.
Also, lower timeout on connection attempts in socket device.
* added changelog
* go mod vendor (picked up some unrelated changes.)
* Skip audit device check in integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
* Update go version to 1.15.3
* Fix OU ordering for go1.15.x testing
* Fix CI version
* Update docker image
* Fix test
* packagespec upgrade -version 0.1.8
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
* Improve errors for aws login with an unbound ARN
* Factor hasWildcardBind into its own function
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
This change allows people who are using templated policies to use the
role_name in their templates through {{
identity.entity.aliases.approle.metadata.role_name }}.
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.
* Support salt in DeriveKey
* Revert "Support salt in DeriveKey"
This reverts commit b295ae42673308a2d66d66b53527c6f9aba92ac9.
* Refactor out key derivation, symmetric encryption, and symmetric decryption into generic functions
* comments
* comments
* go mod vendor
* bump both go.mods
* This one too
* bump
* bump
* bump
* Make the lesser used params of symmetric ops a struct
* go fmt
* Call GetKey instead of DeriveKey
* Address feedback
* Wrong rv
* Rename calls
* Assign the nonce field
* trivial change
* Check nonce len instead
* go mod vendor
Adds debug and warn logging around AWS credential chain generation,
specifically to help users debugging auto-unseal problems on AWS, by
logging which role is being used in the case of a webidentity token.
Adds a deferred call to flush the log output as well, to ensure logs
are output in the event of an initialization failure.
* Use us-gov-west-1 for global APIs in aws-us-gov
Certain partition-global AWS services, like IAM, seem to require
specific regions. In the regular 'aws' partition, this is us-east-1. In
the 'aws-us-gov' partition, this is us-gov-west-1. Providing
us-gov-east-1 returns an error from AWS:
SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-gov-east-1'.
This resolves a problem where AWS authentication could randomly fail
depending on the value cached by Vault at startup.
Decrypting an ciphertext where its corresponding value equals empty, the payload property "plaintext" is missing in the response object. This fixes the problem by adding a new, distinct struct for decrypt batch response items where "omitempty" is not set.
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Lower the interval for rotation during tests, to make it more likely
that our five second grace period is sufficient.
* Rewrite to make the rotateCredentials ticker a configurable value.
* 'go mod vendor' for SDK changes.
* secrets/ssh: allow algorithm_signer to use the key's default algo
* add test for ed25519 key signing
* test: add role upgrade test case
* test: rename and add more test cases
* test: clean up tests cases, fix broken test case on expected error
* test: fix broken test case on expected error
* Bring over PSIRT-37 changes from ENT
* Add additional allowed headers
* Already had this one
* Change to string slice comma separated parsing
* Add allowed_sts_header_values to read output
* Only validate AWS related request headers
* one per line
* Import ordering
* Update test
* Add X-Amz-Credential
* Reorder imports
* pki: use revocationInfo.RevocationTimeUTC when revoking certs with tidy_revoked_certs set to true
* update comment
* tidy: use same time snapshot for OR comparison
We've noticed some leftover processes from vault plugins on our boxes. Some
of them were even left over from previous instances of the service and reparented
to init. This could cause issues if too many of them accumulate.
When running with TRACE logging the logs showed that there was an error return by
the call to Setup() the plugin. Looking through the code it looks like we do not
call Cleanup() in that case.
Co-authored-by: Sorin Dumitru <sorindumitru@users.noreply.github.com>
* Remove two tests from TestTransit_SignVerify_P256. These verify that
you can't verify something signed using JWS marshaling when claiming
it's ASN1, and vice versa. The problem is that we rely on unmarshaling
failing, and it doesn't always. Both encodings use base64, one with
padding, one without, so depending on the data sometimes unmarshaling
will work when we expect it to fail. It would be nice to preserve
these tests if they could be made reliable, but I didn't see an easy way,
and I don't think they add enough value to warrant greater effort.
* Restore the tests I removed, and improve the verify function to broaden the notion of errExpected.
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* add first stepwise test env, Docker, with example transit test
* update transit stepwise test
* add other tests that use stepwise
* cleanup test, make names different than just 'transit'
* return the stderr if compile fails with error
* minor cleanups
* minor cleanups
* go mod vendor
* cleanups
* remove some extra code, and un-export some fields/methods
* update vendor
* remove reference to vault.CoreConfig, which really wasn't used anyway
* update with go mod vendor
* restore Precheck method to test cases
* clean up some networking things; create networks with UUID, clean up during teardown
* vendor stepwise
* Update sdk/testing/stepwise/environments/docker/environment.go
haha thanks :D
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/environments/docker/environment.go
Great catch, thanks
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* fix redundant name
* update error message in test
* Update builtin/credential/userpass/stepwise_test.go
More explicit error checking and responding
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/logical/aws/stepwise_test.go
`test` -> `testFunc`
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/logical/transit/stepwise_test.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* fix typos
* update error messages to provide clarity
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* update error handling / collection in Teardown
* panic if GenerateUUID returns an error
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/credential/userpass/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update builtin/logical/aws/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update builtin/logical/transit/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* import ordering
* standardize on dc from rc for cluster
* lowercase name
* CreateAPIClient -> NewAPIClient
* testWait -> ensure
* go mod cleanup
* cleanups
* move fields and method around
* make start and dockerclusternode private; use better random serial number
* use better random for SerialNumber
* add a timeout to the context used for terminating the docker container
* Use a constant for the Docker client version
* rearrange import statements
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Resolve merge conflicts and updates from running a test
* move testing/_test.go over to legacy
* updates
* Add core of plugin test framework Stepwise (#9166)
* adding stepwise testing, but there are protocol buff error :/
* move file and update sdk/go.mo
* update/sync modules
* update from other branch
* update sdk/go.mod
* some cleanups after feedback
* remove enviornments from this PR
* update vendor
* change from running go mod tidy
* change from go mod tidy
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* change panic to error
* Update sdk/testing/stepwise/helpers.go
return `nil` and not `err` at the end
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Defer close() on successful Open of a file
* document the re-creation of steps
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* remove unused BarrierKeys()
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* updates from feedback
* fix return with bad arguments
* Rename things:
- StepOperation -> Operation
- StepwiseEnvironment -> Environment
- StepCheckFunc -> AssertionFunc
- step.Check -> step.Assert
* document the environment interface methods
* rename EnvironmentOptions to MountOptions
* rename Name to RegistryName
* remove ExpectError because it's redundant
* minor doc update
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* add checkShouldRun function
* remove redundant return
* remove vestigial PreCheck function
* add tt.Helper() to makeRequest
* minor code formatting and document 1-based index for log output of Steps
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* minor updates
* update sdk
* use local reference for api, vault dep
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* cleanup some defer functions
* call fatal if environment setup fails, and don't call teardown
* defer re-setting client token in makeRequest
* Move legacy logicaltest back to testhelpers
* update mods and test files with go mod tidy
* go mod vendor
* remove relative replace directives
* restore old logical test location
* move declaration to main stepwise file
* remove index var and use i+1
* add testing for write, delete paths of makeRequest
* update stepwise core testing to do request counting
* remove unused methods
* Update sdk/testing/stepwise/stepwise.go
remove dead line
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update sdk/testing/stepwise/stepwise.go
fix capitalization in code comment
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* update code comments for SkipTeardown to clarify its use
* update stepwise
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
Allows vault roles to be associated with IAM groups in the AWS
secrets engine, since IAM groups are a recommended way to manage
IAM user policies. IAM users generated against a vault role will
be added to the IAM Groups. For a credential type of
`assumed_role` or `federation_token`, the policies sent to the
corresponding AWS call (sts:AssumeRole or sts:GetFederation) will
be the policies from each group in `iam_groups` combined with the
`policy_document` and `policy_arns` parameters.
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
* Don't use string formatting to prepare queries.
We should, when possible, use the built-in params and ? format when
preparing and executing a query. This is done to prevent SQL Injection
attacks.
* Revert some changes due to failing tests, update mssql go driver
* Add docker container startup for some MSSQL tests
* Remove acceptance test flagging, add more SQL injection protection
* Refactor MSSQL prepareTestContainer to a test helper
Also, remove all ? references and convert them to @p*
* helpful errors: print not only CN but also exactly what we are comparing
* helpful errors: return different errors for non-existent and unknown keys
* helpful errors: print error about encrypted key instead of "private key not found"
* add aliasmetadata sdk helper and add to aws auth
* split into ec2_metadata and iam_metadata fields
* fix tests
* strip pointer
* add test of default metadata
* more test <3
* switch from interface to custom marshallers
* add tests for marshalling
* store nil when selected fields are default
* separate loop into pieces
* separate acc test into multiple
* Update builtin/credential/aws/path_login.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* changes from feedback
* update aws test
* refactor to also populate auth metadata
* update how jsonification is tested
* only add populated metadata values
* add auth_type to ec2 logins
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* fix: rotate root credentials for database plugins using WAL
* test: adds a test for WAL rollback logic
* fix: progress on wal rollback
* docs: updates some comments
* docs: updates some comments
* test: adds additional test coverage for WAL rollback
* chore: remove unneeded log
* style: error handling, imports, signature line wraps
* fix: always close db plugin connection
Makes "ldap operation failed" error messages a little more useful. Also
makes the errors unique so it's easier to debug where an error is coming
from when one occurs.
