2020-04-07 16:59:49 +00:00
|
|
|
---
|
|
|
|
layout: docs
|
|
|
|
page_title: Transform - Secrets Engines
|
|
|
|
description: >-
|
|
|
|
The Transform secrets engine for Vault performs secure data transformation.
|
|
|
|
---
|
|
|
|
|
|
|
|
# Transform Secrets Engine
|
|
|
|
|
|
|
|
-> **Note**: This secret engine requires [Vault
|
|
|
|
Enterprise](https://www.hashicorp.com/products/vault/) with the Advanced Data
|
2021-09-08 15:59:25 +00:00
|
|
|
Protection Transform Module.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
The Transform secrets engine handles secure data transformation and tokenization
|
|
|
|
against provided input value. Transformation methods may encompass NIST vetted
|
|
|
|
cryptographic standards such as [format-preserving encryption
|
|
|
|
(FPE)](https://en.wikipedia.org/wiki/Format-preserving_encryption) via
|
|
|
|
[FF3-1](https://csrc.nist.gov/publications/detail/sp/800-38g/rev-1/draft), but
|
|
|
|
can also be pseudonymous transformations of the data through other means, such
|
|
|
|
as masking.
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
The secret engine currently supports `fpe`, `masking`, and `tokenization` as
|
2020-11-11 18:54:11 +00:00
|
|
|
data transformation types.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
## Setup
|
|
|
|
|
|
|
|
Most secrets engines must be configured in advance before they can perform their
|
|
|
|
functions. These steps are usually completed by an operator or configuration
|
|
|
|
management tool.
|
|
|
|
|
|
|
|
1. Enable the Transform secrets engine:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault secrets enable transform
|
2020-04-07 16:59:49 +00:00
|
|
|
Success! Enabled the transform secrets engine at: transform/
|
2020-12-17 21:53:33 +00:00
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
By default, the secrets engine will mount at the name of the engine. To enable
|
|
|
|
the secrets engine at a different path, use the -path argument.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
1. Create a named role:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault write transform/role/payments transformations=ccn-fpe
|
|
|
|
Success! Data written to: transform/role/payments
|
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
1. Create a transformation:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault write transform/transformation/ccn-fpe \
|
|
|
|
type=fpe \
|
|
|
|
template=ccn \
|
|
|
|
tweak_source=internal \
|
|
|
|
allowed_roles=payments
|
|
|
|
Success! Data written to: transform/transformation/ccn-fpe
|
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
1. Optionally, create a template:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault write transform/template/ccn \
|
|
|
|
type=regex \
|
|
|
|
pattern='(\d{4})-(\d{4})-(\d{4})-(\d{4})' \
|
|
|
|
alphabet=numerics
|
|
|
|
Success! Data written to: transform/template/ccn
|
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
1. Optionally, create an alphabet:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault write transform/alphabet/numerics \
|
|
|
|
alphabet="0123456789"
|
|
|
|
Success! Data written to: transform/alphabet/numerics
|
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
## Usage
|
|
|
|
|
|
|
|
After the secrets engine is configured and a user/machine has a Vault token with
|
|
|
|
the proper permission, it can use this secrets engine to encode and decode input
|
|
|
|
values.
|
|
|
|
|
|
|
|
1. Encode some input value using the `/encode` endpoint with a named role:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault write transform/encode/payments value=1111-2222-3333-4444
|
|
|
|
Key Value
|
|
|
|
--- -----
|
|
|
|
encoded_value 9300-3376-4943-8903
|
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
A transformation must be provided if the role contains more than one
|
|
|
|
transformation. A tweak must be provided if the tweak source for the
|
|
|
|
transformation is "supplied".
|
|
|
|
|
|
|
|
1. Decode some input value using the `/decode` endpoint with a named role:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
```text
|
|
|
|
$ vault write transform/decode/payments value=9300-3376-4943-8903
|
|
|
|
Key Value
|
|
|
|
--- -----
|
|
|
|
decoded_value 1111-2222-3333-4444
|
|
|
|
```
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
A transformation must be provided if the role contains more than one
|
|
|
|
transformation. A tweak must be provided if the tweak source for the
|
|
|
|
transformation is "supplied" or "generated".
|
|
|
|
|
|
|
|
## Roles, Transformations, Templates, and Alphabets
|
|
|
|
|
|
|
|
The Transform secrets engine contains several types of resources that
|
|
|
|
encapsulate different aspects of the information required in order to perform
|
|
|
|
data transformation.
|
|
|
|
|
|
|
|
- **Roles** are the basic high-level construct that holds the set of
|
2020-12-17 21:53:33 +00:00
|
|
|
transformation that it is allowed to performed. The role name is provided when
|
|
|
|
performing encode and decode operations.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
- **Transformations** hold information about a particular transformation. It
|
2020-12-17 21:53:33 +00:00
|
|
|
contains information about the type of transformation that we want to perform,
|
|
|
|
the template that it should use for value detection, and other
|
|
|
|
transformation-specific values such as the tweak source or the masking character
|
|
|
|
to use.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
- **Templates** allow us to determine what and how to capture the value that we
|
2020-12-17 21:53:33 +00:00
|
|
|
want to transform.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
- **Alphabets** provide the set of valid UTF-8 character contained within both
|
2020-12-17 21:53:33 +00:00
|
|
|
the input and transformed value on FPE transformations.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
2020-04-11 00:00:48 +00:00
|
|
|
## Transformations
|
|
|
|
|
|
|
|
### Format Preserving Encryption
|
|
|
|
|
|
|
|
Format Preserving Encryption (FPE) performs cryptographically secure
|
|
|
|
transformation via FF3-1 to encode input values while maintaining its data
|
|
|
|
format and length.
|
|
|
|
|
2020-06-23 17:32:54 +00:00
|
|
|
#### Tweak and Tweak Source
|
2020-04-11 00:00:48 +00:00
|
|
|
|
2020-06-23 17:32:54 +00:00
|
|
|
FF3-1 uses a non-confidential parameter called the tweak along with the
|
|
|
|
ciphertext when performing encryption and decryption operations. The tweak
|
|
|
|
is precisely a 7-byte value. The secret engine consumes a base64 encoded string
|
|
|
|
of this value for its encode and decode operation whenever this input is
|
|
|
|
required.
|
2020-04-11 00:00:48 +00:00
|
|
|
|
2020-10-29 19:04:48 +00:00
|
|
|
In order to simplify the flow of encoding and decoding operations, transformation
|
2020-04-11 00:00:48 +00:00
|
|
|
creation can take care of generating and associating a tweak value. This allows
|
|
|
|
applications to provide a single value without having the need to generate or
|
|
|
|
store any other metadata.
|
|
|
|
|
|
|
|
In cases where more granularity is required, a tweak value can be generated by
|
|
|
|
Vault and returned, or it may be independently generated and provided.
|
|
|
|
|
|
|
|
In summary, there are three ways in which the tweak value may be sourced:
|
|
|
|
|
|
|
|
- `supplied`: This is the default behavior for FPE transformations. The tweak
|
|
|
|
value must be generated externally, and supplied into the on encode and decode
|
|
|
|
operations.
|
|
|
|
- `generated`: The secret engine will take care of generating the tweak value
|
|
|
|
on encode operations and return this back as part of the response along
|
|
|
|
with the encoded value. It is up to the application to store this value
|
|
|
|
so that it can be provided back when decoding the encoded value.
|
|
|
|
- `internal`: The secret engine will generate an internal tweak value per
|
|
|
|
transformation. This value is not returned on encode or decode operations
|
|
|
|
since it gets re-used for all encode and decode operations for the
|
|
|
|
transformation. Depending on the uniqueness of the dataset, this mode may
|
|
|
|
introduce higher risks, but provides the most convenience since the value does
|
|
|
|
not need to be stored separately. This mode should only be used if the values
|
|
|
|
being encoded are sufficiently unique.
|
|
|
|
|
2020-06-15 03:53:36 +00:00
|
|
|
Your team and organization should weigh the trade-offs when it comes to
|
2021-04-06 17:49:04 +00:00
|
|
|
choosing the proper tweak source to use. For `supplied` and `internal`
|
2021-05-04 16:47:13 +00:00
|
|
|
sourcing, please see [FF3-1 Tweak Usage Details](transform/ff3-tweak-details)
|
2020-04-11 00:00:48 +00:00
|
|
|
|
2020-06-05 12:45:18 +00:00
|
|
|
#### Input Limits
|
|
|
|
|
2020-06-23 17:32:54 +00:00
|
|
|
FF3-1 specifies both minimum and maximum limits on the length of an input.
|
|
|
|
These limits are driven by the security goals, making sure that for a given
|
|
|
|
alphabet the input size does not leave the input guessable by brute force.
|
2020-06-05 12:45:18 +00:00
|
|
|
|
2020-06-23 17:32:54 +00:00
|
|
|
Given an alphabet of length A, an input length L is valid if:
|
2020-06-05 12:45:18 +00:00
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
- L >= 2,
|
|
|
|
- A<sup>L</sup> >= 1,000,000
|
|
|
|
- and L <= 2 \* floor(log<sub>A</sub>(2<sup>96</sup>)).
|
2020-06-05 12:45:18 +00:00
|
|
|
|
|
|
|
As a concrete example, for handling credit card numbers, A is 10, L is 16, so
|
2020-12-17 21:53:33 +00:00
|
|
|
valid input lengths would be between 6 and 56 characters. This is because
|
|
|
|
10<sup>6</sup>=1,000,000 (already greater than 2), and 2 \* floor(log<sub>10</sub>(2<sup>96</sup>)) = 56.
|
2020-06-05 12:45:18 +00:00
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
Of course, in the case of credit card numbers valid input would always be
|
2020-11-09 16:58:54 +00:00
|
|
|
between 12 and 19 decimal digits.
|
2020-06-05 12:45:18 +00:00
|
|
|
|
|
|
|
#### Output Limitations
|
|
|
|
|
|
|
|
After transformation and formatting by the template, the value is an encrypted
|
2020-12-17 21:53:33 +00:00
|
|
|
version of the input with the format preserved. However, the value itself may
|
|
|
|
be _invalid_ with respect to other standards. For example the output credit card
|
2020-06-23 17:32:54 +00:00
|
|
|
number may not validate (it likely won't create a valid check digit).
|
2020-06-05 12:45:18 +00:00
|
|
|
|
|
|
|
So one must consider when the outputs are stored whether validation in storage
|
|
|
|
may reject them.
|
|
|
|
|
2020-04-11 00:00:48 +00:00
|
|
|
### Masking
|
|
|
|
|
|
|
|
Masking performs replacement of matched characters on the input value with a
|
|
|
|
desired character. This form of transformation is non-reversible and thus does
|
2020-05-11 15:12:48 +00:00
|
|
|
not support retrieving the original value back using the decode operation.
|
2020-04-11 00:00:48 +00:00
|
|
|
|
2020-11-09 16:58:54 +00:00
|
|
|
### Tokenization
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
[Tokenization](transform/tokenization) exchanges a
|
|
|
|
sensitive value for an unrelated value called a _token_. The original sensitive
|
2020-11-09 16:58:54 +00:00
|
|
|
value cannot be recovered from a token alone, they are irreversible.
|
|
|
|
|
|
|
|
#### Inputs
|
|
|
|
|
|
|
|
Tokenization inputs are not processed by templates or alphabets, as they do not
|
|
|
|
preserve any of the contents or format of the input.
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
#### Outputs
|
2020-11-09 16:58:54 +00:00
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
Tokenization is not format preserving. The token output is a Base58 encoded
|
2020-11-09 16:58:54 +00:00
|
|
|
string value of unrelated length, and is not rendered by a template.
|
|
|
|
|
2020-11-11 18:54:11 +00:00
|
|
|
The decoded value is returned verbatim as it was before encoding.
|
2020-11-09 16:58:54 +00:00
|
|
|
|
|
|
|
#### Metadata
|
|
|
|
|
|
|
|
As tokenization isn't format preserving and is stateful, the input values can be
|
2020-12-17 21:53:33 +00:00
|
|
|
any length, subject to other limits in Vault's request processing. In addition,
|
|
|
|
non-sensitive _metadata_ can be encoded alongside the value, and retrieved either
|
|
|
|
with or independently of the original value.
|
2020-11-09 16:58:54 +00:00
|
|
|
|
|
|
|
#### Operations
|
|
|
|
|
|
|
|
In addition to encode and decode, as tokenization is stateful, it provides two
|
|
|
|
additional operations:
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
- Retrieve metadata given a token.
|
|
|
|
- Check whether an input value has a valid, unexpired token.
|
2020-11-09 16:58:54 +00:00
|
|
|
|
2020-11-11 18:54:11 +00:00
|
|
|
#### Stores
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
Tokenization is stateful. Tokenized state can be stored internally (the
|
|
|
|
default) or in an external store. Currently only PostgreSQL is supported
|
2020-11-11 18:54:11 +00:00
|
|
|
for external storage.
|
|
|
|
|
2020-11-09 16:58:54 +00:00
|
|
|
#### Mapping Modes
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
[Tokenization](transform/tokenization) stores the results of an encode operation
|
2020-11-11 18:54:11 +00:00
|
|
|
in storage using a cryptographic construct that enhances the safety of its values.
|
2020-12-17 21:53:33 +00:00
|
|
|
In the `default` mapping mode, the token itself is transformed via a one way
|
|
|
|
function involving the transform key and elements of the token. As Vault does
|
|
|
|
not store the token, the values in Vault storage themselves cannot be used to
|
2020-11-09 16:58:54 +00:00
|
|
|
retrieve original input.
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
A second mapping mode, `exportable` is provided for cases where
|
|
|
|
operators may need to recover the full set of decoded inputs in an emergency via
|
|
|
|
the export operation. It is strongly recommended that one use the `default` mode if
|
2020-11-11 18:54:11 +00:00
|
|
|
possible, as it is resistant to more types of attack.
|
2020-11-09 16:58:54 +00:00
|
|
|
|
2020-04-07 16:59:49 +00:00
|
|
|
## Deletion Behavior
|
|
|
|
|
|
|
|
The deletion of resources, aside from roles, is guarded by checking whether any
|
|
|
|
other related resources are currently using it in order to avoid accidental data
|
|
|
|
loss of any encoded value that may depend on these bits of information to
|
|
|
|
decode and reconstruct the original value. Role deletion can be safely done
|
|
|
|
since the information related to the transformation itself is contained within
|
|
|
|
transformation object and its related resources.
|
|
|
|
|
|
|
|
The following rules applies when it comes to deleting a resource:
|
|
|
|
|
2020-11-09 16:58:54 +00:00
|
|
|
- A transformation cannot be deleted if it's in use by a role.
|
|
|
|
- A template or store cannot be deleted if it's in use by a transformation.
|
|
|
|
- An alphabet cannot be deleted if it's in use by a template.
|
2020-04-07 16:59:49 +00:00
|
|
|
|
|
|
|
## Provided Builtin Resources
|
|
|
|
|
|
|
|
The secret engine provides a set of builtin templates and alphabets that are
|
|
|
|
considered common. Builtin templates cannot be deleted, and the prefix
|
|
|
|
"builtin/" on template and alphabet names is a reserved keyword.
|
|
|
|
|
|
|
|
### Templates
|
|
|
|
|
|
|
|
The following builtin templates are available for use in the secret engine:
|
|
|
|
|
|
|
|
- builtin/creditcardnumber
|
|
|
|
- builtin/socialsecuritynumber
|
|
|
|
|
|
|
|
Note that these templates only check for the matching pattern(s), and not the
|
|
|
|
validity of the value itself. For instance, the builtin credit card number
|
|
|
|
template can determine whether the provided value is in the format of commonly
|
|
|
|
issued credit cards, but not whether the credit card is a valid number from a
|
|
|
|
particular issuer.
|
|
|
|
|
2020-06-23 17:32:54 +00:00
|
|
|
Templates currently only accept regular expressions as the matching pattern
|
|
|
|
type. It uses Go's standard library for the regexp engine, which supports
|
|
|
|
[the RE2 syntax](https://github.com/google/re2/wiki/Syntax).
|
|
|
|
|
2020-11-09 16:58:54 +00:00
|
|
|
**Note**: The `builtin/any` template is only valid and is the default for the tokenization
|
2020-12-17 21:53:33 +00:00
|
|
|
transform.
|
2020-11-09 16:58:54 +00:00
|
|
|
|
2020-04-07 16:59:49 +00:00
|
|
|
### Alphabets
|
|
|
|
|
|
|
|
The following builtin alphabets are available for use in the secret engine:
|
|
|
|
|
|
|
|
- builtin/numeric
|
|
|
|
- builtin/alphalower
|
|
|
|
- builtin/alphaupper
|
|
|
|
- builtin/alphanumericlower
|
|
|
|
- builtin/alphanumericupper
|
|
|
|
- builtin/alphanumeric
|
|
|
|
|
2020-12-17 21:53:33 +00:00
|
|
|
Custom alphabets must contain between 2 and 65536 unique characters.
|
2020-11-09 16:58:54 +00:00
|
|
|
|
|
|
|
### Stores
|
|
|
|
|
|
|
|
The following builtin store is available (and is the default) for tokenization
|
|
|
|
transformations:
|
|
|
|
|
|
|
|
- builtin/internal
|
2020-06-05 12:45:18 +00:00
|
|
|
|
2020-04-07 16:59:49 +00:00
|
|
|
## Learn
|
|
|
|
|
|
|
|
Refer to the [Transform Secrets Engine](https://learn.hashicorp.com/vault/adp/transform) guide for a step-by-step tutorial.
|
|
|
|
|
|
|
|
## API
|
|
|
|
|
|
|
|
The Transform secrets engine has a full HTTP API. Please see the
|
|
|
|
[Transform secrets engine API](/api/secret/transform) for more
|
|
|
|
details.
|