Commit Graph

2357 Commits

Author SHA1 Message Date
Alexander Scheel 15ae00d147
Add unified crl building (#18792)
* Add unified CRL config storage helpers

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add support to build unified CRLs

This allows us to build unified versions of both the complete and delta
CRLs. This mostly involved creating a new variant of the
unified-specific CRL builder, fetching certs from each cluster's storage
space.

Unlike OCSP, here we do not unify the node's local storage with the
cross-cluster storage: this node is the active of the performance
primary, so writes to unified storage happen exactly the same as
writes to cluster-local storage, meaning the two are always in
sync. Other performance secondaries do not rebuild the CRL, and hence
the out-of-sync avoidance that we'd like to solve with the OCSP
responder is not necessary to solve here.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add ability to fetch unified CRLs

This adds to the path-fetch APIs the ability to return the unified CRLs.
We update the If-Modified-Since infrastructure to support querying the
unified CRL specific data and fetchCertBySerial to support all unified
variants. This works for both the default/global fetch APIs and the
issuer-specific fetch APIs.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rebuild CRLs on unified status changes

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Handle rebuilding CRLs due to either changing

This allows detecting if the Delta CRL needs to be rebuilt because
either the local or the unified CRL needs to be rebuilt. We never
trigger rebuilding the unified delta on a non-primary cluster.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Ensure serials aren't added to unified CRL twice

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 19:17:34 +00:00
Alexander Scheel 1c85d611e2
Write delta WAL entries for unified CRLs (#18785)
* Write delta WAL entries for unified CRLs

When we'd ordinarily write delta WALs for local CRLs, we also need to
populate the cross-cluster delta WAL. This could cause revocation to
appear to fail if the two clusters are disconnected, but notably regular
cross-cluster revocation would also fail.

Notably, this commit also changes us to not write Delta WALs when Delta
CRLs is disabled (versus previously doing it when auto rebuild is
enabled in case Delta CRLs were later asked for), and instead,
triggering rebuilding a complete CRL so we don't need up-to-date Delta
WAL info.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Update IMS test for forced CRL rebuilds

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 16:56:08 +00:00
Alexander Scheel ec7502aa44
More cross cluster queue tweaks (#18789)
* Move comment about perf-primary only invalidation

Also remove noisy debug log.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove more noisy log statements during queue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Skip revocation entries from our current cluster

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add locking and comment about tidying revoke queue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Switch to time.Since for tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor tidyStatuses into path_tidy.go

Leaving these in backend.go often causes us to miss adding useful values
to tidyStatus when we add a new config parameter.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Track the number of deleted revocation request

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow tidy to remove confirmed revocation requests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add missing field to tidy test

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 16:52:38 +00:00
Steven Clark d0453ed40b
Add unified storage support to OCSP handler (#18788) 2023-01-23 15:49:07 +00:00
Steven Clark f3ce351e01
Add support for revoke by serial number to update the unified CRL (#18786) 2023-01-23 10:22:10 -05:00
Alexander Scheel b3dc380c82
Add cross-cluster revocation queues for PKI (#18784)
* Add global, cross-cluster revocation queue to PKI

This adds a global, cross-cluster replicated revocation queue, allowing
operators to revoke certificates by serial number across any cluster. We
don't support revoking with private key (PoP) in the initial
implementation.

In particular, building on the PBPWF work, we add a special storage
location for handling non-local revocations which gets replicated up to
the active, primary cluster node and back down to all secondary PR
clusters. These then check the pending revocation entry and revoke the
serial locally if it exists, writing a cross-cluster confirmation entry.

Listing capabilities are present under pki/certs/revocation-queue,
allowing operators to see which certs are present. However, a future
improvement to the tidy subsystem will allow automatic cleanup of stale
entries.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow tidying revocation queue entries

No manual operator control of revocation queue entries are allowed.
However, entries are stored with their request time, allowing tidy to,
after a suitable safety buffer, remove these unconfirmed and presumably
invalid requests.

Notably, when a cluster goes offline, it will be unable to process
cross-cluster revocations for certificates it holds. If tidy runs,
potentially valid revocations may be removed. However, it is up to the
administrator to ensure the tidy window is sufficiently long that any
required maintenance is done (or, prior to maintenance when an issue is
first noticed, tidy is temporarily disabled).

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Only allow enabling global revocation queue on Vault Enterprise

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use a locking queue to handle revocation requests

This queue attempts to guarantee that PKI's invalidateFunc won't have
to wait long to execute: by locking only around access to the queue
proper, and internally using a list, we minimize the time spent locked,
waiting for queue accesses.

Previously, we held a lock during tidy and processing that would've
prevented us from processing invalidateFunc calls.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* use_global_queue->cross_cluster_revocation

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Grab revocation storage lock when processing queue

We need to grab the storage lock as we'll actively be revoking new
certificates in the revocation queue. This ensures nobody else is
competing for storage access, across periodic funcs, new revocations,
and tidy operations.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Fix expected tidy status test

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow probing RollbackManager directly in tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Address review feedback on revocationQueue

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add more cancel checks, fix starting manual tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-23 09:29:27 -05:00
Alexander Scheel 3adfed1af8
Add missing space in PKI error (#18778)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-20 11:02:17 -05:00
Alexander Scheel 6b4f770de9
Refactor CRL Building for unified CRLs (#18754)
* Refactor CRL building into separate functions

This will allow us to add the ability to add and build a unified CRL
across all clusters, reusing logic that is common to both, but letting
each have their own certificate lists.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rename localCRLConfigEntry->internalCRLConfigEntry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rename Delta WALs to Local Delta WALs

This adds clarity that we'll have a separate local and remote Delta CRL
and WALs for each.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-18 15:05:14 -05:00
Anton Averchenkov 6ae09f3074
Add AppRole response schema validation tests (#18636)
This PR modifies every test in `builtin/credentials/approle/path_role_test.go` with new validation checks to ensure that approle/path_role  successful responses align with the declared response schema.

It also introduces a test helper in `sdk/helper/testhelpers`:

```go
func FindResponseSchema(t *testing.T, ...)
```

This test helper will be useful for all plugins that require similar response schema validation in tests.

### Background

This PR is part of the ongoing work to add structured responses in Vault OpenAPI (VLT-234)
2023-01-13 15:23:36 -05:00
Steven Clark e0e957731b
Refactor the PKI revocation handler to prep for unified revocation (#18685)
* Rename revokeCert variable to identify serial number formatting

* Refactor out lease specific behavior out of revokeCert

 - Isolate the specific behavior regarding revoking lease specific
   certificates outside of the revokeCert function and into the only
   caller that leveraged used it.
 - This allows us to simplify revokeCert a little bit and keeps the
   function purely about revoking a certificate

* Within revokeCert short circuit the already revoked use-case

 - Make the function a little easier to process by exiting early
   if the certificate has already been revoked.

* Do not load certificates from storage multiple times during revocation

 - Isolate the loading of a certificate and parsing of a certificate
   into a single attempt, either when provided the certificate for BYOC
   revocation or strictly from storage for the other revocation types.

* With BYOC write certificate entry using dashes not the legacy colon char
2023-01-13 10:31:03 -05:00
Max Bowsher d1f2b101b5
Add option 'elide_list_responses' to audit backends (#18128)
This PR relates to a feature request logged through HashiCorp commercial
support.

Vault lacks pagination in its APIs. As a result, certain list operations
can return **very** large responses.  The user's chosen audit sinks may
experience difficulty consuming audit records that swell to tens of
megabytes of JSON.

In our case, one of the systems consuming audit log data could not cope,
and failed.

The responses of list operations are typically not very interesting, as
they are mostly lists of keys, or, even when they include a "key_info"
field, are not returning confidential information. They become even less
interesting once HMAC-ed by the audit system.

Some example Vault "list" operations that are prone to becoming very
large in an active Vault installation are:

    auth/token/accessors/
    identity/entity/id/
    identity/entity-alias/id/
    pki/certs/

In response, I've coded a new option that can be applied to audit
backends, `elide_list_responses`. When enabled, response data is elided
from audit logs, only when the operation type is "list".

For added safety, the elision only applies to the "keys" and "key_info"
fields within the response data - these are conventionally the only
fields present in a list response - see logical.ListResponse, and
logical.ListResponseWithInfo. However, other fields are technically
possible if a plugin author writes unusual code, and these will be
preserved in the audit log even with this option enabled.

The elision replaces the values of the "keys" and "key_info" fields with
an integer count of the number of entries. This allows even the elided
audit logs to still be useful for answering questions like "Was any data
returned?" or "How many records were listed?".
2023-01-11 16:15:52 -05:00
Anton Averchenkov 5f7e95fcb9
Fix AppRole / path_role response schema (#18637) 2023-01-11 12:15:29 -05:00
Alexander Scheel 44c3b736bf
Allow tidy to backup legacy CA bundles (#18645)
* Allow tidy to backup legacy CA bundles

With the new tidy_move_legacy_ca_bundle option, we'll use tidy to move
the legacy CA bundle from /config/ca_bundle to /config/ca_bundle.bak.
This does two things:

 1. Removes ca_bundle from the hot-path of initialization after initial
    migration has completed. Because this entry is seal wrapped, this
    may result in performance improvements.
 2. Allows recovery of this value in the event of some other failure
    with migration.

Notably, this cannot occur during migration in the unlikely (and largely
unsupported) case that the operator immediately downgrades to Vault
<1.11.x. Thus, we reuse issuer_safety_buffer; while potentially long,
tidy can always be run manually with a shorter buffer (and only this
flag) to manually move the bundle if necessary.

In the event of needing to recover or undo this operation, it is
sufficient to use sys/raw to read the backed up value and subsequently
write it to its old path (/config/ca_bundle).

The new entry remains seal wrapped, but otherwise isn't used within the
code and so has better performance characteristics.

Performing a fat deletion (DELETE /root) will again remove the backup
like the old legacy bundle, preserving its wipe characteristics.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation about new tidy parameter

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests for migration scenarios

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Clean up time comparisons

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-11 12:12:53 -05:00
Alexander Scheel a18187c643
Correctly distinguish empty issuer names in PKI (#18466)
* Correctly distinguish empty issuer names

When using client.Logical().JSONMergePatch(...) with an empty issuer
name, patch incorrectly reports:

> issuer name contained invalid characters

In this case, both the error in getIssuerName(...) is incorrect and
patch should allow setting an empty issuer name explicitly.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-01-10 10:04:30 -05:00
Alexander Scheel 38de21468e
Add cluster_aia_path templating variable (#18493)
* Add cluster_aia_path templating variable

Per discussion with maxb, allow using a non-Vault distribution point
which may use an insecure transport for RFC 5280 compliance.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Address feedback from Max

Co-authored-by: Max Bowsher <maxbowsher@gmail.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Max Bowsher <maxbowsher@gmail.com>
2023-01-10 09:51:37 -05:00
Mike Palmiotto 43a78c85f4
Mark deprecated builtins Removed (#18039)
* Remove logical database builtins

* Drop removed builtins from registry keys

* Update plugin prediction test

* Remove app-id builtin

* Add changelog
2023-01-09 09:16:35 -05:00
Steven Clark cfd5b8a933
Resolve unrecognized parameter warnings on batch_input parameter in transit (#18299)
* Resolve unused warnings on batch_input parameter in transit

* Add cl

* Fix text in hmac batch_input parameter description
2023-01-04 09:15:48 -05:00
Christopher Swenson 7395249046
Add LICENSE back to pkcs7 (#18527)
This was not copied over when the this code was
copied in https://github.com/hashicorp/vault/pull/12340.

Also adds a stub for the `.copywrite.hcl` file (for when
Vault is onboarded to Copywrite) and adds the `pkcs7` and
`ui/node_modules` to the ignore pattern.
2022-12-22 09:54:43 -08:00
Alexander Scheel 3ccbddab0e
Add issuer reference info on JSON endpoint (#18482)
* Add issuer reference info on JSON endpoint

This endpoint is unauthenticated and shouldn't contain sensitive
information. However, listing the issuers (LIST /issuers) already
returns both the issuer ID and the issuer name (if any) so this
information is safe to return here.

When fetching /pki/issuer/default/json, it would be nice to know exactly
which issuer ID and name it corresponds to, without having to fetch the
authenticated endpoint as well.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-12-19 21:39:01 +00:00
davidadeleon 51b1b6d446
Approle: Fix CIDR validation for /32 masks on Token Bound CIDRs (#18145)
* Fix CIDR validation for /32 masks

* run go fmt

* add changelog
2022-12-16 12:09:05 -05:00
Mike Palmiotto 9d5f021792
Fix SHA1 patch for Go 1.19.4; patch test (#18405)
Bad news: the hot patch we were using breaks in Go 1.19.4: 6109c07ec4

Good news: we can now patch with an environment variable at runtime.

Co-authored-by: Christopher Swenson <christopher.swenson@hashicorp.com>
2022-12-15 12:52:45 -05:00
Alexander Scheel 3a5b48afe4
Correctly handle issuer tidying in auto-tidy config (#18347)
* Correctly handle issuer tidying in auto-tidy config

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add missing parameters to auto-tidy docs

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-12-14 15:35:21 -05:00
Scott Miller c9531431a4
Add the batch reference field, as in Transform, to Transit operations (#18243)
* Add the batch reference field, as in Transform, to Transit operations

* changelog

* docs

* More mapstructure tags
2022-12-13 12:03:40 -06:00
Scott Miller c1cfc11a51
Return the partial success code override for all batch error types (#18310)
* Return the partial success code override for all batch error types

* changelog

* docs

* Lost the actual override logic. :)

* And don't hardcode 400

* gate on success
2022-12-12 17:08:22 -06:00
Steven Clark 3bf683b872
Document adding metadata to entity alias within cert auth (#18308)
* Document adding metadata to entity alias within cert auth

* Update website/content/api-docs/auth/cert.mdx

Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>

Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
2022-12-12 13:08:00 -05:00
Alexander Scheel f3911cce66
Add transit key config to disable upserting (#18272)
* Rename path_config -> path_keys_config

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add config/keys to disable upserting

Transit would allow anyone with Create permissions on the encryption
endpoint to automatically create new encryption keys. This becomes hard
to reason about for operators, especially if typos are subtly
introduced (e.g., my-key vs my_key) -- there is no way to merge these
two keys afterwards.

Add the ability to globally disable upserting, so that if the
applications using Transit do not need the capability, it can be
globally disallowed even under permissive policies.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation on disabling upsert

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Update website/content/api-docs/secret/transit.mdx

Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>

* Update website/content/api-docs/secret/transit.mdx

Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
2022-12-08 15:45:18 -05:00
Alexander Scheel a8764e0cf1
Refactor PKI to use shared storage context (#18266)
A lot of places took a (context, backend, request) tuple, ignoring the
request proper and only using it for its storage. This (modified) tuple
is exactly the set of elements in the shared storage context, so we
should be using that instead of manually passing all three elements
around.

This simplifies a few places where we'd generate a storage context at
the request level and then split it apart only to recreate it again
later (e.g., CRL building).

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-12-08 09:27:02 -05:00
Steven Clark 735a8d4b31
Address race in PKI test case (#18267)
- Nick brought this to our attention, one of the PKI test suites
   is overwriting the production code's value leading to a data race
   issue.
 - Remove the setting of the variable with the same value from the test
   suite.
2022-12-08 09:11:05 -05:00
Anton Averchenkov 545ee098ab
Add openapi response definitions to approle/path_role.go (#18198)
This PR modifies the path schema of `approle/path_role.go`, switching the old `Callbacks` to the equivalent `Operations` objects with a list of response fields for the 200 responses. This will allow us to generate a response structures in openapi.json. This PR is split out from #18055 along with #18192.

### Example

For `GET "/auth/approle/role/{role_name}/bind-secret-id"` path, it will update the response as follows:

```diff
        "responses": {
          "200": {
            "description": "OK",
++            "content": {
++              "application/json": {
++                "schema": {
++                  "$ref": "#/components/schemas/ApproleRoleBindSecretIdResponse"
++                }
++             }
            }
          }
        }
```

And will add the actual response structure:

```diff
++      "ApproleRoleBindSecretIdResponse": {
++        "type": "object",
++        "properties": {
++          "bind_secret_id": {
++            "type": "boolean",
++            "description": "Impose secret_id to be presented when logging in using this role. Defaults to 'true'."
++          }
++        }
++      },
```
2022-12-05 16:55:13 -05:00
Alexander Scheel 2398634862
Respond with data to all writes in PKI engine (#18222)
* Respond with data to all writes in PKI engine

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-12-05 10:40:39 -05:00
Alexander Scheel f86fdf530f
Allow templating cluster-local AIA URIs (#18199)
* Allow templating of cluster-local AIA URIs

This adds a new configuration path, /config/cluster, which retains
cluster-local configuration. By extending /config/urls and its issuer
counterpart to include an enable_templating parameter, we can allow
operators to correctly identify the particular cluster a cert was
issued on, and tie its AIA information to this (cluster, issuer) pair
dynamically.

Notably, this does not solve all usage issues around AIA URIs: the CRL
and OCSP responder remain local, meaning that some merge capability is
required prior to passing it to other systems if they use CRL files and
must validate requests with certs from any arbitrary PR cluster.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation about templated AIAs

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* AIA URIs -> AIA URLs

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* issuer.AIAURIs might be nil

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow non-nil response to config/urls

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Always validate URLs on config update

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Ensure URLs lack templating parameters

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Review feedback

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-12-05 10:38:26 -05:00
Steven Clark 9bf3b4c582
Do not use possibly nil HttpRequest object in default OCSP handler (#18190) 2022-12-01 13:23:41 -05:00
Steven Clark 826e87884e
Address a nil panic when writing an empty POST request to the ocsp handler (#18184)
* Address a nil panic when writing an empty POST request to the ocsp handler

 - Seems when no JSON body is sent with a POST request Vault will not
   populate the HTTPRequest member variable which caused the nil panic
   - vault write -force pki/ocsp
 - Add a check for it and the Body member variable to be nil before use.

* Add cl
2022-12-01 15:10:12 +00:00
Alexander Scheel 347cdf811c
Disable nginx integration test in pki test suites (#18141)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-29 13:30:25 -05:00
Alexander Scheel a04855c98d
Add crl integraiton to tests (#17447)
* Add tests using client certificates

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor Go TLS client tests

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests for CRLs

Note that Delta CRL support isn't present in nginx or apache, so we lack
a server-side test presently. Wget2 does appear to support it however,
if we wanted to add a client-side OpenSSL test.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add checks for delta CRL with wget2

This ensures the delta CRL is properly formatted and accepted by
OpenSSL.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Re-add missing test helpers

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Rename clientFullChain->clientWireChain

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-28 10:32:22 -05:00
Alexander Scheel a8faa543e6
Add pki nginx/wget/curl/Go integration tests (#17320)
* Rename integation_test.go->integration_test.go

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add ability to fetch container's network addresses

This lets us return the on-network container address, allowing us to
spawn client containers which contact server containers.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add integration tests with nginx, curl, wget, Go

We build new integration tests, spawning a test instance on nginx and
ensuring we can connect with a variety of clients against a variety of
CA and leaf certificate types. This will ultimately let us detect issues
with compatibility as we expand the matrix of supported servers and
clients.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Make runner reference unique

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Attempt to fix CI with longer wait

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Finish moving nginx tests to pkiext package

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* make fmt

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add more debugging, work on CircleCI

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-23 15:00:18 -05:00
Tom Proctor 853643d02b
Remove pinned builtin plugin versions from storage (#18051)
* Removes _builtin_ versions from mount storage where it already exists
* Stops new builtin versions being put into storage on mount creation/tuning
* Stops the plugin catalog from returning a builtin plugin that has been overridden, so it more accurately reflects the plugins that are available to actually run
2022-11-23 18:36:25 +00:00
Alexander Scheel 2a801895e3
Create CSR in Transit, sign with PKI (#17630)
Execute with:

$ go test -v -run=TestTransitPKICSR github.com/hashicorp/vault/builtin/logical/transit
...
    backend_test.go:1843: csr: -----BEGIN CERTIFICATE REQUEST-----
        MIICXjCCAUYCAQAwGTEXMBUGA1UEAxMOZGFkZ2FyY29ycC5jb20wggEiMA0GCSqG
        SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDD8GUy2Rut9ILPXH/Ef7lEaYijuBB9wogd
        hKD3uJyfK5PqBqM8166UsrP7Y+bWkwDrMke3aDxXRNybys33kIc8KfGwS3omNYd3
        17KN1D4ZgQ+oW6xISa3ISOl4D7XeFtHeTP0U1plVXBd9kqTlo4YPlUF/kTfqmxDu
        2a41BIS5HlORdLLG+jQ3shRgwHANONBhlaUnIqEeykdW8/iEBlqoYlMzty9W724R
        2mKk0FzrVAZ/X5ZO992dAMrQDvc3Nofl+ddzbElBJLumrcDSwALFVge+ag1N48kE
        CCfxjizEykGdCrR+VELb8b33IgFf6EOVRnS5Qy8whmw943v5Oru5AgMBAAGgADAN
        BgkqhkiG9w0BAQsFAAOCAQEAdg9SwbrWszMmz60JWQPPfwW+XhzR0MdY82adK8P6
        9xpWyJU+U649tAFQb+PCT1OcU7ETd59QcEV38VLndBPWhotTXl5oB5XAqg2bkKHV
        nLc0cGwyxBSs77LALc//m2f5v2otO9fLOmuM2RMfD02ZUliBmZUzeaUIJYEfT+cS
        M60uLKJvnNBu5xH1q0oG9P0uNkpEX+QGx6SwhR1/41pmygiUR+uwJxxuRGMvECoN
        dsHZtzi7ftEHBJ9tk94hd/RFnDsvWlHGyfRWhALNtbo6QjHxjBJIFKh+GHlI8Tnf
        6YWvD0VIodE609+RlCrhFlGd+3NUSt0b/f0bgkMJLzLqEw==
        -----END CERTIFICATE REQUEST-----
    backend_test.go:1878: root: -----BEGIN CERTIFICATE-----
        MIIDHTCCAgWgAwIBAgIUIwCzCdrsgkcNOi5liRNHeH+n+tUwDQYJKoZIhvcNAQEL
        BQAwFjEUMBIGA1UEAxMLUEtJIFJvb3QgWDEwHhcNMjIxMDIxMTQ1NjQyWhcNMjIx
        MTIyMTQ1NzExWjAWMRQwEgYDVQQDEwtQS0kgUm9vdCBYMTCCASIwDQYJKoZIhvcN
        AQEBBQADggEPADCCAQoCggEBAMdM65f5p3fLwQP1sezBRFqAxUZhOQwnnnp8mFXp
        3fIF9pqLMzNvyd+bCUXv+aFalX4KY1iOoKVHJWwtpXMoKn40U+DZkapR9CsVQt5Q
        9xzIcuPPuI+/oNwU4qB9mAuwG+U7KLosGnQOR3NI02A4dnl5I0z8Y/DJLz29GP/P
        1zPYMBRBpkMz4F2Xr0w6tTXWDsmqZ9j7ukBDoizmnB2xfKzSjCVmQvXa71UlqbG9
        td75LCgpiQh/50mHFHs6RKtqrlFUY5BtPPs+tHUf4nklieIzbAEwA8Fbq4d/Xpq1
        HRoRvWj3nelX/h/IRlj/VKJssd1ZL+1kdzxKB4N6AFC1nusCAwEAAaNjMGEwDgYD
        VR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFCniHbjHy/UG
        ShROBOiikqxCe+OkMB8GA1UdIwQYMBaAFCniHbjHy/UGShROBOiikqxCe+OkMA0G
        CSqGSIb3DQEBCwUAA4IBAQBSKnl154oyc1Rncm4sr/1f1QM5rCaw/lqgISfvki8m
        t6yyxQL+9lDpebFjBDTL5teRzuMYyqN4pRkIhpITDGmFXRpEOv14mcbASX7nPBEN
        bYXhOh1UC8a0CLzT2ll0ERFNnUEPRi0s0ONRm3lIZAV3Mzf4sOdwfRwmP33hBe/1
        V9D7Lcx5N84EPrvGC/r8F/PsVKHyKFS46qB1MvhMppRG6fJ2cFmg5UGwdKdmxuvz
        FoT+RaTLkgcQgkDuYClNco5OVIM7Bd4JTNK3WbqvtGklOHslrz+ND0eMYM/LK+ZS
        zNM35nzK6QaN2M4IO4Wuy3y2yu8xllEfmssXwAtTi4wk
        -----END CERTIFICATE-----
    backend_test.go:1879: leaf: -----BEGIN CERTIFICATE-----
        MIIDDzCCAfegAwIBAgIUBGeUSi0p3ffndZqgvlBvMvn8qgMwDQYJKoZIhvcNAQEL
        BQAwFjEUMBIGA1UEAxMLUEtJIFJvb3QgWDEwHhcNMjIxMDIxMTQ1NjQyWhcNMjIx
        MDIxMTUwNzEyWjAZMRcwFQYDVQQDEw5kYWRnYXJjb3JwLmNvbTCCASIwDQYJKoZI
        hvcNAQEBBQADggEPADCCAQoCggEBAMPwZTLZG630gs9cf8R/uURpiKO4EH3CiB2E
        oPe4nJ8rk+oGozzXrpSys/tj5taTAOsyR7doPFdE3JvKzfeQhzwp8bBLeiY1h3fX
        so3UPhmBD6hbrEhJrchI6XgPtd4W0d5M/RTWmVVcF32SpOWjhg+VQX+RN+qbEO7Z
        rjUEhLkeU5F0ssb6NDeyFGDAcA040GGVpScioR7KR1bz+IQGWqhiUzO3L1bvbhHa
        YqTQXOtUBn9flk733Z0AytAO9zc2h+X513NsSUEku6atwNLAAsVWB75qDU3jyQQI
        J/GOLMTKQZ0KtH5UQtvxvfciAV/oQ5VGdLlDLzCGbD3je/k6u7kCAwEAAaNSMFAw
        DgYDVR0PAQH/BAQDAgOoMB0GA1UdDgQWBBSRCRR/62DjS1kjWHrVQ0Y58leUbDAf
        BgNVHSMEGDAWgBQp4h24x8v1BkoUTgToopKsQnvjpDANBgkqhkiG9w0BAQsFAAOC
        AQEAvWPLGqtC1SRy61Y17HtJ0giDUwpCZbOUkAwtdDAnKIhR1v4wrlY3sKUBLuhK
        xOJIWfVlCnPUt5uTnPaWyVyUfry6YNerish1k7ny/R1n58PjsPhUg8GJB9HHsME+
        gQQ22z6D/87n0bEE8PaTzIU6+cVHoIBJ0rqzjZVkBs0cEjf+l40RPP1h+ZiTw27u
        CR2iXmHJ9TQ8ZBWygIhxB9JOMbk5jpH6w6wJqq8XK9zuC1hlYbXH1K5KvZJxAPlh
        CJkoq2KxaIwByTHjRdGjDogSibsyY+CxQUnktefXb6tYKvFTpUFsh1fjQRCwUrlD
        SExMRHhFJBHfyPD1w26N3IjRlg==
        -----END CERTIFICATE-----

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-23 17:57:23 +00:00
Steven Clark 92c1a2bd0a
New PKI API to generate and sign a CRL based on input data (#18040)
* New PKI API to generate and sign a CRL based on input data

 - Add a new PKI API that allows an end-user to feed in all the
   information required to generate and sign a CRL by a given issuer.
 - This is pretty powerful API allowing an escape hatch for 3rd parties
   to craft customized CRLs with extensions based on their individual
   needs

* Add api-docs and error if reserved extension is provided as input

* Fix copy/paste error in Object Identifier constants

* Return nil on errors instead of partially filled slices

* Add cl
2022-11-22 11:41:04 -05:00
Scott Miller b51b2a7027
Add cached OCSP client support to Cert Auth (#17093)
* wip

* Add cached OCSP client support to Cert Auth

* ->pointer

* Code cleanup

* Fix unit tests

* Use an LRU cache, and only persist up to 1000 of the most recently used values to stay under the storage entry limit

* Fix caching, add fail open mode parameter to cert auth roles

* reduce logging

* Add the retry client and GET then POST logic

* Drop persisted cache, make cache size configurable, allow for parallel testing of multiple servers

* dead code

* Update builtin/credential/cert/path_certs.go

Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Hook invalidate to reinit the ocsp cache size

* locking

* Conditionally init the ocsp client

* Remove cache size config from cert configs, it's a backend global

* Add field

* Remove strangely complex validity logic

* Address more feedback

* Rework error returning logic

* More edge cases

* MORE edge cases

* Add a test matrix with a builtin responder

* changelog

* Use an atomic for configUpdated

* Actually use ocsp_enabled, and bind to a random port for testing

* Update builtin/credential/cert/path_login.go

Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Refactor unit tests

* Add status to cache

* Make some functions private

* Rename for testing, and attribute

* Up to date gofumpt

* remove hash from key, and disable the vault dependent unit test

* Comment out TestMultiOCSP

* imports

* more imports

* Address semgrep results

* Attempt to pass some sort of logging to test_responder

* fix overzealous search&replace

Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-21 10:39:24 -06:00
Alexander Scheel 75b70d84e6
Add list to cert auth's CRLs (#18043)
* Add crl list capabilities to cert auth

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add docs on cert auth CRL listing

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add test for cert auth listing

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-18 11:39:17 -05:00
Hamid Ghaf 9543067ffe
fix auth renew panic (#18011)
* fix auth renew panic

* CL

* adding a test step to a cert test for pathLoginRenew
2022-11-18 10:38:18 -05:00
Steven Clark 01e87c481c
Add new PKI api to combine and sign different CRLs from the same issuer (#17813)
* Add new PKI api to combine and sign different CRLs from the same issuer

 - Add a new PKI api /issuer/<issuer ref>/resign-crls that will allow
   combining and signing different CRLs that were signed by the same
   issuer.
 - This allows external actors to combine CRLs into a single CRL across
   different Vault clusters that share the CA certificate and key material
   such as performance replica clusters and the primary cluster

* Update API docs

* PR Feedback - Delta CRL rename

* Update to latest version of main

* PR Feedback - Get rid of the new caEntry struct

* Address PR feedback in api-docs and PEM encoded response
2022-11-17 16:53:05 -05:00
Michael Anthony aa74bd7ed7
[QT-309] Resolve AWS config before testing ACC (#17949)
* Check to resolve AWS config before testing ACC

* Adjust wording of error to be more clear
2022-11-17 11:55:27 -07:00
Michael Anthony 0624d8f36e
[QT-309] Ensure environment variables are populated before proceeding (#17915)
* Ensure environment variables are populated before proceeding

* DRY up credNames var
2022-11-17 11:55:17 -07:00
Alexander Scheel c25b90831e
Move pki docker tests to pkiext (#17928)
* Export CreateBackendWithStorage for pkiext

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Move zlint_test.go to pkiext

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Fix mount all test to ignore pkiext

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-14 18:26:26 -05:00
Alexander Scheel a0136f5f19
Handle removed default issuers (#17930)
Credit to Steve for finding this one.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-14 15:13:39 -05:00
divyaac 036bd45ca7
Added error message (#17904)
* Added error message

* Added changelog

* Grammar Chagne

* Changed wording
2022-11-11 11:04:10 -08:00
Michael Dempsey fd032831cb
Expose ssh algorithm_signer in web interface (#10114) (#10299)
* Expose ssh algorithm_signer in web interface (#10114)

* Adds allowed values for algorithm_signer to ssh plugin API
* Adds algorithm_signer as field in UI

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-10 14:24:53 -08:00
Alexander Scheel 5a2ee4ca7a
Add automatic tidy of expired issuers (#17823)
* Add automatic tidy of expired issuers

To aid PKI users like Consul, which periodically rotate intermediates,
and provided a little more consistency with older versions of Vault
which would silently (and dangerously!) replace the configured CA on
root/intermediate generation, we introduce an automatic tidy of expired
issuers.

This includes a longer safety buffer (1 year) and logging of the
relevant issuer information prior to deletion (certificate contents, key
ID, and issuer ID/name) to allow admins to recover this value if
desired, or perform further cleanup of keys.

From my PoV, removal of the issuer is thus a relatively safe operation
compared to keys (which I do not feel comfortable removing) as they can
always be re-imported if desired. Additionally, this is an opt-in tidy
operation, not enabled by default. Lastly, most major performance
penalties comes with lots of issuers within the mount, not as much
large numbers of keys (as only new issuer creation/import operations are
affected, unlike LIST /issuers which is a public, unauthenticated
endpoint).

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add test for tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add docs on tidy of issuers

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Restructure logging

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add missing fields to expected tidy output

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-11-10 10:53:26 -05:00