luxolus/open-vault
2
0
Fork 0
Code Issues 1 Pull requests Releases Activity
open-vault/website/content/docs/secrets/pki/rotation-primitives.mdx
Alexander Scheel 3f8aaedc2a
Add suggested root rotation procedure (#19033) 
* Add suggested root rotation procedure

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Clarify docs heading

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

---------

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-02-07 13:51:33 -05:00

499 lines
26 KiB
Plaintext
Raw Blame History

---
layout: docs
page_title: 'PKI - Secrets Engine: Rotation Primitives'
description: The PKI secrets engine for Vault generates TLS certificates.
---
# PKI Secrets Engine - Rotation Primitives
Since Vault 1.11.0, Vault's PKI Secrets Engine supports multiple issuers in a
single mount point. By using the certificate types below, rotation can be
accomplished in various situations involving both root and intermediate CAs
managed by Vault.
## X.509 Certificate Fields
X.509 is a complex specification; modern implementations tend to refer to
[RFC 5280](https://datatracker.ietf.org/doc/html/rfc5280) for specific
details. For validation of certificates, both RFC 5280 and the TLS
validation [RFC 6125](https://datatracker.ietf.org/doc/html/rfc6125) are
important for understanding how to achieve rotation.
The following is a simplification of these standards for the purpose of
this document.
Every X.509 certificate begins with an asymmetric key pair, using an algorithm
like RSA or ECDSA. This key pair is used to create a Certificate Signing
Request (CSR), which contains a set of fields the requester would like in the
final certificate (but, it is up to the Certificate Authority (CA) to decide what
fields to take from the CSR and which to override). The CSR also contains the
public key of the pair, which is signed by the private key of the key pair to
prove possession. Usually, the requester would ask for attributes in the
Subject field of the CSR or in the Subject Alternative Name extension CSR to
be respected in the final certificate. It is up to the CA if these values are
trusted or not. When approved by the issuing authority (which may be backed by
this asymmetric key itself in the case of a root self-signed certificate), the
authority attaches the Subject of _its_ certificate to the issued certificate in
the Issuer field, assigns a unique serial number to the issued certificate, and
signs the set of fields with its private key, thus creating the certificate.
There are some important restrictions here:
- One certificate can only have one Issuer, but this issuer is identified by
the Subject on the issuing certificate and its public key.
- One key pair can be used for multiple certificates, but one certificate can
only have one backing key material.
The following fields on the final certificate are relevant to rotation:
- The backing [public](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.7)
and private key material (Subject Public Key Info).
- Note that the private key is not included in the certificate but is
uniquely determined by the public key material.
- The [Subject](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.6) of the certificate.
- This identifies the entity to which the certificate was issued. While the
SAN values (in the [Subject Alternative Name](https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.6)
extension) is useful when validating TLS Server certificates against the
negotiated hostname and URI, it isn't generally relevant for the purposes
of validating intermediate certificate chains or in rotation.
- The [Validity](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.5)
period of this certificate.
- Notably, RFC 5280 does not place any requirements around the issued
certificate's validity period relative to the validity period of the
issuing certificate. However, it [does state](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.5)
that certificates ought to be revoked if their status cannot be maintained
up to their notAfter date. This is why Vault 1.11's `/pki/issuer/:issuer_ref`
configuration endpoint maintains the `leaf_not_after_behavior` per-issuer
rather than per-role.
- Additionally, some browsers will place ultimate trust in the certificates
in their trust stores, even when these certificates are expired.
- Note that this only applies to certificates in the trust store; validity
periods will still be enforced for certificates not in the store (such
as intermediates).
- The [Issuer](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.4) and
[signatureValue](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.1.3)
of this certificate.
- In the issued certificate's Issuer field, the issuing certificate places
its own Subject value. This allows the issuer to be identified later
(without having to try signature validation against every known local
certificate), when validating the presented certificate and chain.
- The signature over the entire certificate (by the issuer's private key)
is then placed in the signatureValue field.
- The optional [Authority Key Identifier](https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.1)
field.
- This field can contain either (or both) of two values:
- The hash of the issuer's public key. This extension is set and this
value is filled in by Vault.
- The Issuer's Subject and Serial Number. This value is not set by Vault.
- The latter is a dangerous restriction for the purposes of rotation: it
prevents cross-signing and reissuance as the new issuing certificates
(while having the same backing key material) will have different serial
numbers. See the [Limitations of Primitives](#limitations-of-primitives)
section below for more information on this restriction.
- The [Serial Number](https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.2)
of this certificate.
- This field is unique to a specific issuer; when a certificate is
reissued by its parent authority, it will always have a different serial
number field.
- The [CRL distribution](https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.13)
point field.
- This is a field detailing where a CRL is expected to exist for this
certificate and under which CRL issuers (defaulting to the issuing
certificate itself) the CRL is expected to be signed by. This is mostly
informational and for server software like nginx, Vault's Cert Auth method,
and Apache, CRLs are provided to the server, rather than having the
server fetch CRLs for certificates automatically.
- Note that root certificates (in browsers trust stores) are generally not
considered revocable. However, if an intermediate is revoked by serial,
it will appear on its parent's CRL, and may prevent rotation from
happening.
## X.509 Rotation Primitives
Rotation (from an organizational standpoint) can only safely happen with
certain intermediate X.509 certificates being issued. To distinguish the two
types of certificates used to achieve rotation, this document notates them
as _primitives_.
Rotation of an end-entity certificate is trivial from an X.509 trust chain
perspective; this process happens every day and should only depend on what is
in the trust store and not the end-entity certificate itself. In Vault, the
requester would hit the various issuance endpoints (`/pki/issue/:name` or
`/pki/sign/:name` -- or use the unsafe `/pki/sign-verbatim`) and swap out the
old certificate with the new certificate and reload the configuration or
restart the service. Other parts of the organizations might use
[ACME](https://datatracker.ietf.org/doc/html/rfc8555) for certificate issuance
and rotation, especially if the service is public-facing (and thus needs to
be issued by a Public CA). Given it was signed by a trusted root, any devices
connecting to the service would not know the difference.
Rotation of intermediate certificates is almost as easy. Assuming a decent
operational setup (wherein during end-entity issuance, the full certificate
chain is updated in the service's configuration), this should be as easy as
creating a new intermediate CA, signing it against the root CA, and then
beginning issuance against the new intermediate certificate. In Vault, if
the intermediate is generated in an existing mount path (or is moved into
such), the requesting entity shouldn't care much. Under ACME, Let's Encrypt
has successfully rotated intermediates to present a cross-signed chain
([for older Android devices](https://letsencrypt.org/2020/12/21/extending-android-compatibility.html)).
Assuming the old intermediate's parent(s) are still valid and trusted,
certificates issued under old intermediates should continue to validate.
The hard part of rotation--calling for the use of these primitives--is
rotating root certificates. These live in every device's trust store and
are hard to update from an organization-wide operational perspective.
Unless the organization can swap out roots almost instantaneously and
simultaneously (e.g., via an agent) with no missed devices, this process
will likely span months.
To make this process lower risk, there are various primitive certificate
types that use the [above certificate fields](#x-509-certificate-fields).
Key to their success is the following note:
~> Note: While certificates are added to the trust store, it is ultimately
the associated key material that determines trust: two issuer certificates
with the same subject but different public keys cannot validate the same
leaf certificate; only if the keys are the same can this occur.
### Cross-Signed Primitive
This is the most common type of rotation primitive. A common CSR is signed by
two CAs, resulting in two certificates. These certificates must have the same
Subject (but may have different Issuers and will have different Serial Numbers)
and the same backing key material, to allow certificates they sign to be
trusted by either variant.
Note that, due to restrictions in how end-entity certificates are used and
validated (services and validation libraries expect only one), cross-signing
most typically only applies to intermediate.
#### A Note on Cross-Signed Roots
Technically, cross-signing can occur between two roots, allowing trust bundles
with either root to validate certs issued through the other. However, this
process creates a certificate that is effectively an intermediate (as it is
no longer self-signed) and usually must be served alongside the trust chain.
Given this restriction, it's preferable to instead cross-sign the top-level
intermediates under the root unless strictly necessary when the old root
certificate has been used to directly issue leaf certificates.
So, the rest of this process flow assumes an intermediate is being
cross-signed as this is more common.
##### Process Flow
```
-------------------
| generate key pair | -------------> ...
------------------- ...
| | ...
-------------- -------------- ...
| generate CSR | | generate CSR | ...
-------------- -------------- ...
| | ...
----------- ----------- ...
| signed by | | signed by | ...
| root A | | root B | ...
----------- ----------- ...
```
Here, a key pair was generated at some point in time. Two CSRs are created and
sent to two different root authorities (Root A and Root B). These result in two
separate certificates (potentially with different validity periods) with the
same Subject and same backing key material.
Note that this cross-signing need not happen simultaneously; there could be a
gap of several years between the first and second certificate. Additionally,
there's no limit on the number of cross-signed "duplicate" (used loosely--with
the same subject and key material) certificates: this could be cross-signed
by many different root certificates if necessary and desired.
##### Certificate Hierarchy
```
-------- --------
| root A | | root B |
-------- --------
| |
---------------- ----------------
| intermediate C | <- same key material -> | intermediate D |
---------------- | ----------------
|
-------------------
| leaf certificates |
-------------------
```
The above process results in two trust paths: either of root A or root B (or
both) could exist in the client's trust stores and the leaf certificate would
validate correctly. Because the same key material is used for both intermediate
certificates (C and D), the issued leaf certificate's signature field would
be the same regardless of which intermediate was contacted.
Cross-signing is thus a unifying primitive; two separate trust paths now join
into a single one, by having leaf certificate's issuer field to point to two
separate paths (via duplication of the certificate in the chain) and would be
conditionally validated based on which root is present in the trust store.
This construct is documented and used in several places:
- https://letsencrypt.org/certificates/
- https://scotthelme.co.uk/cross-signing-alternate-trust-paths-how-they-work/
- https://security.stackexchange.com/questions/14043/what-is-the-use-of-cross-signing-certificates-in-x-509
#### Execution in Vault
To create a cross-signed certificate in Vault, use the [`/intermediate/cross-sign`
endpoint](/vault/api-docs/secret/pki#generate-intermediate-csr). Here, when creating
a cross-signature to all `cert B` to be validated by `cert A`, provide the values
(`key_ref`, all Subject parts, &c) for `cert B` during intermediate generation.
Then sign this CSR (using the [`/issuer/:issuer_ref/sign-intermediate`
endpoint](/vault/api-docs/secret/pki#sign-intermediate)) with `cert A`'s reference
and provide necessary values from `cert B` (e.g., Subject parts). `cert A` may
live outside Vault. Finally, import the cross-signed certificate into Vault
[using the `/issuers/import/cert` endpoint](/vault/api-docs/secret/pki#import-ca-certificates-and-keys).
If this process succeeded, and both `cert A` and `cert B` and their key
material lives in Vault, the newly imported cross-signed certificate
will have a `ca_chain` response field [during read](/vault/api-docs/secret/pki#read-issuer)
containing `cert A`, and `cert B`'s `ca_chain` will contain the cross-signed
cert and its `ca_chain` value.
~> Note: Regardless of issuer type, is important to provide all relevant
parameters as they were originally; Vault does not infer e.g., the Subject
name parameters from the existing issuer; it merely reuses the same key
material.
##### Notes on `manual_chain`
If an intermediate is cross-signed and imported into the same mount as its
pair, Vault will not detect the cross-signed pairs during automatic chain
building. As a result, leaf issuance will have a chain that only includes
one of these pairs of chains. This is because the leaf issuance's `ca_chain`
parameter copies the value from signing issuer directly, rather than computing
its own copy of the chain.
To fix this, update the `manual_chain` field on the [issuers](/vault/api-docs/secret/pki#update-issuer)
to include the chains of both pairs. For instance, given `intA` signed by
`rootA` and `intB` signed by `rootB` as its cross-signed version, one
could do the following:
```
$ vault patch pki/issuer/intA manual_chain=self,rootA,intB,rootB
$ vault patch pki/issuer/intB manual_chain=self,rootB,intA,rootA
```
This will ensure that issuance with either copy of the intermediate reports
the full cross-signed chain when signing leaf certs.
### Reissuance Primitive
The second most common type of rotation primitive. In this scheme, the existing
key material is used to generate a new certificate, usually at a much later
point in time from the existing issuance.
While similar to the cross-signed primitive, this one differs in that usually
the reissuance happens after the original certificate expires or is close to
expiration and is reissued by the original root CA. In the event of a
self-signed certificate (e.g., a root certificate), this parent certificate
would be itself. In both cases, this changes the contents of the certificate
(due to the new serial number) but allows all existing leaf signatures to
still validate.
Unlike the cross-signed primitive, this primitive type can be used on all
types of certificates (including leaves, intermediates, and roots).
#### Process Flow
```
-------------------
| generate key pair | ---------------> ...
------------------- ...
| | ...
-------------- -------------- ...
| generate CSR | <-> | generate CSR | ...
-------------- -------------- ...
| | ...
------------------ ------------------ ...
| signed by issuer | -> | signed by issuer | -> ...
------------------ ------------------ ...
```
In this process flow, a single key pair is generated at some point in time
and stored. The CSR (with same requested fields) is generated from this
common key material and signed by the same issuer at multiple points in
time, preserving all critical fields (Subject, Issuer, &c). While there is
strictly no limit on the number of times a key can be reissued, at some point
safety would dictate the key material should be rotated instead of being
continually reissued.
#### Certificate Hierarchy
```
------
-----------| root |-------------
/ ------ \
| |
--------------- ---------------
| original cert | <- same key material -> | reissued cert |
--------------- | ---------------
|
-------------------
| leaf certificates |
-------------------
```
Note that while this again results in two trust paths, depending on which
intermediate certificate is presented and is still valid, only a root need be
trusted. When a reissued certificate is a root certificate, the issuance link is
simply self-loop. But, in this case, note that both certificates are
(technically) valid issuers of each other. This means it should be possible to
provide a reissued root certificate in the TLS certificate chain and have it
chain back to an existing root certificate in a trust store.
This primitive type is thus an incrementing primitive; the life cycle of an
existing key is extended into the future by issuing a new certificate with the
same key material from the existing authority.
#### Execution in Vault
To create a reissued root certificate in Vault, use [`/issuers/generate/root/existing`
endpoint](/vault/api-docs/secret/pki#generate-root). This allows the generation of a new
root certificate with the existing key material (via the `key_ref` request parameter).
If this process succeeded, when [reading the issuer](/vault/api-docs/secret/pki#read-issuer)
(via `GET /issuer/:issuer_ref`), both issuers (old and reissued) will appear in
each others' `ca_chain` response field (unless prevented so by a `manual_chain`
value).
To create a reissued intermediate certificate in Vault, this is a three step
process:
1. Use the [`/issuers/generate/intermediate/existing`
endpoint](/vault/api-docs/secret/pki#generate-intermediate-csr)
to generate a new CSR with the existing key material with the `key_ref`
request parameter.
2. Sign this CSR via the same signing process under the same issuer. This
step is specific to the parent CA, which may or may not be Vault.
3. Finally, use the [`/intermediate/set-signed` endpoint](/vault/api-docs/secret/pki#import-ca-certificates-and-keys)
to import the signed certificate from step 2.
If the process to reissue an intermediate certificate succeeded, when
[reading the issuer](/vault/api-docs/secret/pki#read-issuer) (via
`GET /issuer/:issuer_ref`), both issuers (old and reissued) will have
the same `ca_chain` response field, except for the first entry (unless
prevented so by a `manual_chain` value).
~> Note: Regardless of issuer type, is important to provide all relevant
parameters as they were originally; Vault does not infer e.g., the Subject
name parameters from the existing issuer; it merely reuses the same key
material.
### Temporal Primitives
We can use the above primitive types to rotate roots and intermediates to new
keys and extend their lifetimes. This time-based rotation is what ultimately
allows us to rotate root certificates.
There's two main variants of this: a **forward** primitive, wherein an old
certificate is used to bless new key material, and a **backwards** primitive,
wherein a new certificate is used to bless old key material. Both of these
primitives are independently used by Let's Encrypt in the aforementioned
chain of trust document:
- The link from DST Root CA X3 to ISRG Root X1 is an example of a forward
primitive.
- The link from ISRG Root X1 to R3 (which was originally signed by DST Root
CA X3) is an example of a backwards primitive.
For most organizations with a hierarchical structured CA setup, cross-signing
all intermediates with both the new and old root CAs is sufficient for root
rotation.
However, for organizations which have directly issued leaf certificates from a
root, the old root will need to be reissued under the new root (with shorter
duration) to allow these certificates to continue to validate. This combines
both of the above primitives (cross-signing and reissuance) into a single
backwards primitive step. In the future, these organizations should probably
move to a more standard, hierarchical setup.
### Limitations of Primitives
The certificate's [Authority Key Identifier](https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.1)
extension field may contain either or both of the issuer's keyIdentifier
(a hash of the public key) or both the issuer's Subject and Serial Number
fields. Generating certificates with the latter enabled (luckily not possible
in Vault, especially so since Vault uses strictly random serial numbers)
prevents building a proper cross-signed chain without re-issuing the same
serial number, which will not work with most browsers' trust stores and
validation engines, due to [caching of
certificates](https://support.mozilla.org/en-US/kb/Certificate-contains-the-same-serial-number-as-another-certificate)
used in successful validations. In the strictest sense, when using a
cross-signing primitive (from a different CA), the intermediate could be reissued
with the same serial number, assuming no previous certificate was issued by that
CA with that serial. This does not work when using a reissuance primitive as these
are technically the same authority and thus this authority must issue
certificates with unique serial numbers.
## Suggested Root Rotation Procedure
The following is a suggested process for achieving root rotation easily and
without (outage) impact to the broader organization, assuming [best
practices](/vault/docs/secrets/pki/considerations#use-a-ca-hierarchy) are
being followed. Some adaption will be necessary.
Note that this process takes time. How much time is dependent on the
automation level and operational awareness of the organization.
1. [Generate](/vault/api-docs/secret/pki#generate-root) the new root
certificate. For clarity, it is suggested to use a new common name
to distinguish it from the old root certificate. Key material need
not be the same.
2. [Cross-sign](#cross-signed-primitive) all existing intermediates.
It is important to update the manual chain on the issuers as discussed
in that section, as we assume servers are configured to combine the
`certificate` field with the `ca_chain` field on renewal and issuance,
thus getting the cross-signed intermediates.
3. Encourage rotation to pickup the new cross-signed intermediates. With
short-lived certificates, this should [happen
automatically](/vault/docs/secrets/pki/considerations#automate-leaf-certificate-renewal).
However, for some long-lived certs, it is suggested to rotate them
manually and proactively. This step takes time, and depends on the
types of certificates issued (e.g., server certs, code signing, or client
auth).
4. Once _all_ chains have been updated, new systems can be brought online
with only the new root certificate, and connect to all existing systems.
5. Existing systems can now be migrated with a one-shot root switch: the
new root can be added and the old root can be removed at the same time.
Assuming the above step 3 can be achieved in a reasonable amount of time,
this decreases the time it takes to move the majority of systems over to
fully using the new root and no longer trusting the old root. This step
also takes time, depending on how quickly the organization can migrate
roots and ensure all such systems are migrated. If some systems are
offline and only infrequently online (or, if they have hard-coded
certificate stores and need to reach obsolescence first), the organization
might not be ready to move on to future steps.
6. At this point, since all systems now use the new root, it is safe to remove
or archive the old root and intermediates, updating the manual chain to
point strictly to the new intermediate+root.
At this point, rotation is fully completed.
## Tutorial
Refer to the [Build Your Own Certificate Authority (CA)](/vault/tutorials/secrets-management/pki-engine)
guide for a step-by-step tutorial.
Have a look at the [PKI Secrets Engine with Managed Keys](/vault/tutorials/enterprise/managed-key-pki)
for more about how to use externally managed keys with PKI.
## API
The PKI secrets engine has a full HTTP API. Please see the
[PKI secrets engine API](/vault/api-docs/secret/pki) for more
details.
Reference in a new issue View git blame Copy permalink