open-vault/website/source/docs/secrets/gcp/index.html.md

layout	page_title	sidebar_current	description
docs	GCP - Secrets Engines	docs-secrets-gcp	The GCP secrets engine for Vault generates service account keys and OAuth tokens dynamically based on IAM policies.

GCP Secrets Engine

The Google Cloud Platform secrets engine dynamically generates IAM service account credentials based on IAM policies, allowing users to have access to GCP resources without having to manage a new identity (service account).

Quick Links:

• Things to Note: If you are running into unexpected errors (or wish to avoid doing so), please read this section! In particular, if you have used other Vault secrets engines (i.e. the AWS engine), this covers important differences in behavior.
• Contributing/Issues: This engine was written as a Vault plugin and thus exists outside the main Vault repo. Please report issues, request features, or submit contributions to the plugin-specific repo on Github (all are welcome!)
• Docs Quick Links:
  • GCP secrets engine HTTP API
  • Expected roleset bindings format

Background

Benefits of using this Vault secrets engine to manage IAM service accounts and credentials:

• Automatic cleanup of long-lived IAM service account keys
  • Vault associates leases with generated IAM service account keys s.t. they will be automatically revoked when the lease expires.
• Users don't need to create a new service-account per one-off or short-term access.
• Multi-cloud applications:
  • Users authenticate to Vault using some central identity (LDAP, AppRole, etc) and can generate GCP credentials without having to create and manage a new service account for that user.

Overview

Note: The following docs assume that the secrets engine has been mounted at gcp, the default mount path. Adjust your calls accordingly if you mount the engine at a different path. All example calls in this doc will use the Vault CL tool - see HTTP API for possible API calls.

Setup

Initially, the secrets engine must be enabled in Vault.

$ vault secrets enable gcp
Success! Enabled the gcp secrets engine at: gcp/

You will need to set up the secrets engine with:

• Config: General config, including credentials that Vault will need to make calls to GCP APIs (either explicitly or using Application Default Credentials), lease defaults, etc.

• Rolesets: Generated credentials will need to be associated with sets of IAM roles on specific GCP resources. Rolesets have an associated secret_type that determines a secret type that can be generated under this role set.

Usage - Secret Generation

Once the secrets engine has been set up, you can generate two types of secrets in Vault:

• Access Tokens (secret_type: access_token): This endpoint generates non-renewable OAuth2 access tokens with a lifetime of an hour.

• Service Account Keys (secret_type: service_account_key): This endpoint generates long-lived IAM service account keys.

Each secret is associated with a Vault lease that can be revoked and possibly renewed (see lease docs for how to do so). On revoking a lease, the secret is no longer guaranteed to work.

Each secret is generated under a specified role set, which determine which permissions (IAM roles) the generated credentials have on specific GCP resources. Note that a role set can only generate one type of secret, specified at role set creation as secret_type

Config

The config/ endpoint is used to configure any information shared by the entire GCP secrets engine. You can

Allowed Operations: write, read

Examples:

$ vault write gcp/config credentials="..." ttl=100 max_ttl=1000
$ vault read gcp/config

Parameters For Write:

• credentials (string:"") - JSON credentials (either file contents or '@path/to/file'). See next sections to learn more about required permissions and alternative ways to provide these credentials.
• ttl (int: 0 || string:"0s") – Specifies default config TTL for long-lived credentials (i.e. service account keys). Accepts integer number of seconds or Go duration format string.
• max_ttl (int: 0 || string:"0s")– Specifies default config TTL for long-lived credentials (i.e. service account keys). Accepts integer number of seconds or Go duration format string.

Passing Credentials To Vault

If you would rather not pass the IAM credentials using the payload credentials parameter, there are multiple ways to pass IAM credentials to the Vault server. You can specify credentials in the following ways (given in order of evaluation):

1. Static credential JSON provided to the API as a payload (i.e. credentials parameter)
2. Credentials in the environment variables GOOGLE_CREDENTIALS or GOOGLE_CLOUD_KEYFILE_JSON
3. Parse JSON file ~/.gcp/credentials
4. Google Application Default Credentials

Required Permissions

At present, this endpoint does not confirm that the provided GCP credentials are for existing IAM users or have valid permissions. However, they will need the following permissions:

// Service Account + Key Admin
    iam.serviceAccounts.create
    iam.serviceAccounts.delete
    iam.serviceAccounts.get
    iam.serviceAccounts.list
    iam.serviceAccountKeys.create
    iam.serviceAccountKeys.delete
    iam.serviceAccountKeys.get
    iam.serviceAccountKeys.list
    iam.serviceAccounts.update

// IAM Policy Changes
    $service.$resource.getIamPolicy
    $service.$resource.setIamPolicy

where $service.$resource refers to the GCP service/resources you wish to assign credentials permissions on, e.g. cloudresourcemanager.projects.get/setIamPolicy for GCP projects

You can either create a custom role with these permissions and assign it at a project-level to an IAM service account for Vault to use, or assign roles/iam.serviceAccountAdmin and roles/iam.serviceAccountKeyAdmin to get the permissions other than get/setIamPolicy permissions, and assign custom roles or global roles that include get/setIamPolicy permissions for the resources that Vault will change policies for.

Rolesets

Rolesets determine the permissions that service account credentials (secrets) generated by Vault will have on given GCP resources.

Endpoint: roleset/ (rolesets/ for list)

Allowed Operations: write, read, list, delete, rotate, rotate-key

Creating/Updating Rolesets

Each roleset will have an associated service account that is created when the role set is created or updated.

(see things to note for background)

Parameters For Write:

• name (string: <required>): Required. Name of the role. Cannot be updated. Given as part of path.
• secret_type (string: "access_token"): Type of secret generated for this role set. Accepted values: access_token, service_account_key. Cannot be updated.
• project (string: <required>): Name of the GCP project that this roleset's service account will belong to. Cannot be updated.
• bindings (string: <required>): Bindings configuration string (expected HCL or JSON string, raw or base64-encoded)
• token_scopes (array: []): List of OAuth scopes to assign to access_token secrets generated under this role set (access_token role sets only)

Update is the same call but will error if non-updatable fields are given with different values. If you update a roleset's bindings, note this will effectively revoke any secrets generated under this roleset.

Examples:

# secret type `access_token`
$ vault write gcp/roleset/my-token-roleset \
    project="mygcpproject" \
    secret_type="access_token"  \
    token_scopes="https://www.googleapis.com/auth/cloud-platform"
    bindings=@binds.hcl \
         
# secret type `service_account_key`       
$ vault write gcp/roleset/my-key-roleset \
    project="mygcpproject" \
    secret_type="service_account_key"  \
    bindings="<base64-encoded-hcl-string>"

Roleset Bindings

The rolesets binding argument accepts bindings in the following format:

resource "path/to/my/resource" {
  roles = [
    "roles/viewer",
    "projects/X/roles/myprojCustomRole"
  ] 
}

resource "//service.googleapis.com/path/to/another/resource" {
  roles = [
    "organizations/Y/roles/myprojCustomRole"
  ] 
}

The top-level blocks are resource blocks, defined as resource "a/resource/name" {...}. Each block define IAM policy information to be bound to this resource.

The following resource path formats are supported:

1. Project-level Self-Link (Specify Service and Version) A URI with scheme and host. Generally the self_link attribute of some resource. Must be resource with parent project (i.e. relative resource name projects/$PROJECT/...). Examples: * Compute alpha zone:

   ```
   https://www.googleapis.com/compute/alpha/projects/my-project/zones/us-central1-c
   ```
   

2. Full resource name (Specify Service): A scheme-less URI consisting of a DNS-compatible API service name and a resource path. The resource path is the relative resource name (see next). Use to specify service but use either the preferred service version or the only version for which this resource is IAM-enabled.

   Examples:

   • Compute snapshot:

   //compute.googleapis.com/project/X/snapshots/Y
   

   • Pubsub snapshot:

   //pubsub.googleapis.com/project/X/snapshots/Y
   

3. Relative Resource Name: A path-noscheme URI path. Use if version/service are apparent from resource type (or you want to use only the preferred version of the service). General format is resource name as accepted by the corresponding REST API, but we've added some exceptions (namely Storage). Examples:

   • Storage bucket object:

   b/bucketname/o/objectname
   buckets/bucketname/o/objectname 
   

   • Pubsub Topic:

   projects/X/topics/Y
   

Each resource block accepts the following arguments:

• roles: An array of string names for IAM roles. Each string must be a global role name (roles/roleFoo), a project-level custom role (projects/myproj/roles/roleFoo) or an organization-level custom role (organizations/myorg/roles/roleFoo)

You can provide this as a plaintext string blob, the base64-encoded version of this string, or using syntax @path/to/bindings.hcl to pass in a filename

Creation Workflow:

When an admin user creates a roleset or updates the role set bindings, Vault does the following:

1. Parses given bindings configuration file as described above.

2. Add WAL entries to clean-up:

   • Current service account, key, and bindings in roleset if updating and update succeeds
   • New service account, key, and bindings if create/update fails.

3. Attempt to create a new IAM service account.

   • The new service account email is "vault$rolesetName-$timestamp@...", where the roleset name might be truncated to fit the IAM service account character limit)
   • The new service account display name will contain the Vault roleset it belongs to.

4. For each resource in the bindings

   a. call getIamPolicy method on the resource.

   b. For each role in the resource's role list, add a binding to this policy with the email of the new service account.

    { "role": "roles/theRoleToAssign", "member": "serviceAccount:$EMAIL" }
   

   c. call setIamPolicy method on the resource

   See this section to learn more about how we call IAM methods on various resources.

   Note: Because the service account and policies are assigned now, the resources given must exist at roleset creation time.

5. If the roleset generates access tokens (secret_type = "access_token"), create a new service account key for this service account.

6. Save the new role set.

   Note: If steps 2-6 fail, we return a error response and the new service account will not be saved. WAL entries will have been added s.t. any entities will eventually get cleaned up, but you may need to manually delete these if rollback fails and you want to free up quota immediately.

   After step 6, the create/update operation will return a success, but we have a last cleanup step:

7. Try to delete the old service account (and any keys) and remove its old IAM policy bindings. If any of these calls to GCP APIs fail, we added the WAL entries in step 2 so eventually these resources will get cleaned up. We will return a warning in the response. In addition, we will try to delete the WAL entries for the now in-use service account that we just created s.t. it does not attempt to clean up these entities later.

WAL Cleanup

As mentioned, Vault will create WAL entries, both for newly created service accounts/bindings, which may not get used if update fails, and for old service account bindings, which may need to be deleted if immediate cleanup fail.

Each WAL entry contains the name of the role set that was under update/creation. The WAL cleanup functions work as follows:

1. Vault will attempt to get the role set (which may be missing because it was deleted or was never created).

2. For each type of WAL entry:

   • Service Account: We try to delete the service account saved in the WAL entry.
     • If the roleset is still using the service account: We do nothing and remove the WAL entry. This happens if we failed to delete a WAL entry for a successful operation, or we preemptively added a WAL entry to delete old service accounts for an update that ended up failing.
   • Service Account Key (for access_token rolesets)
     • If a key name is included in the WAL entry:
       • We are attempting to clean up a previously created and saved key after a role set update.
       • If the roleset still exists and uses the key, we do nothing and remove the WAL entry. Otherwise, we delete the key.
     • If no key name is included:
       • We are attempting to clean up newly created keys for failed roleset updates. Because the key name is generated by GCP, we do not know the key name when we create the WAL entry.
       • Instead of deleting one key, we list all the user-managed keys under this service account and delete any that are not being used by the current roleset, if it exists. Since this service account is being controlled by Vault and the secrets are access tokens, there should not be any other user-managed keys for this service account.
   • IAM Resource-Policy Bindings:
     • The entry contains the GCP resource name that may need policy bindings cleaned up, and the bound roles and service account email that needs to be cleaned up. We get the IAM policy, remove the necessary bindings, and set the IAM policy without these bindings.
     • If the roleset exists and is still using this service account email, we remove only IAM policy bindings not included in the current roleset's bindings.

Other Roleset Operations

Rotation

If you want to reset the service account for a given roleset, or rotate the key used for access_token rolesets, the GCP secrets engine has two endpoints for this:

Examples:

$ vault write gcp/roleset/my-key-roleset/rotate \
      
$ vault write gcp/roleset/my-token-roleset/rotate
$ vault write gcp/roleset/my-token-roleset/rotate-key # `access_token` only

The rotate/ endpoint works similar to roleset update (i.e. see the create/update workflow above). It replaces the roleset's service account with a new service account and replaceds any bindings with the new email as the IAM policy binding member. This will in effect revoke any secrets generated under this roleset.

The rotate-key/ endpoint only applies to access_token rolesets and simply rotates the key saved in the roleset used to generate access tokens, while keeping the same service account.

Note that rotating the service account (/rotate) for an access_token roleset will also effectively rotate the key.

Read and Delete

Examples:

$ vault read gcp/roleset/my-token-roleset
         
$ vault delete gcp/roleset/my-key-roleset

On read, you can get the associated service account, token scopes for access_token rolesets, and other information.

On delete, the service account, bindings, and any keys associated with the roleset service account will be deleted.

Generating Secrets

Access Tokens

You can only generate tokens under access_token rolesets.

Example Call:

$ vault read gcp/token/my-token-roleset

Key                Value
---                -----
lease_id           gcp/token/my-token-roleset/some-uuid
lease_duration     59m59s
lease_renewable    false
token              ya29.c.Ell9...

This endpoint generates non-renewable OAuth2 access tokens with a lifetime of an hour. Vault will simply return an access token (as token in output) that can be used in calls to GCP APIs as part of an "Authorization: Bearer" header:

$ curl -H "Authorization: Bearer $TOKEN" ...

Service Account Keys

You can only generate tokens under service_account_key rolesets.

Example Call:

$ vault read gcp/key/my-key-roleset

lease_id            gcp/key/my-key-roleset/some-uuid
lease_duration      768h
lease_renewable     true
key_algorithm       KEY_ALG_RSA_2048
key_type            TYPE_GOOGLE_CREDENTIALS_FILE
private_key_data    <base-64 encoded private key data>...

This endpoint generates IAM service account keys associated with the role set's service account. These keys are by default long-lived in GCP, but Vault associates a lease (see output lease_* information). This lease can be renewed to extend the key lifetime, or revoked. When the lease is revoked, the service account key will be deleted (thus freeing up quota).

As mentioned in things to note, there is a limit of 10 service account keys per account and thus a limit of 10 secrets per service_account_key roleset. Read this section to learn about how to get around this limit if you are running into issues.

To learn how to use service account keys, see calling Google APIs, how to authenticate in gcloud, or how to authenticate in Google Cloud client libraries.

Things to Note

Access Tokens vs Service Account Keys

While in general the IAM team prefers that you use short-term access tokens over long-lived, hard-to-manage service account keys to authenticate calls to GCP APIs, this is currently impractical as several Google tools require service account keys for authenticating calls (i.e. gcloud CLI tool, client libraries). Thus, while we default to roleset secret_type = access_token, we also offer the option to generate keys (set role set secret_type to service_account_key), but we caution that these keys should still be carefully monitored even if they have been associated with Vault leases.

NOTE: access_token role sets will generate a service account key that is saved in Vault. Vault will be the only place this key can be accessed (i.e. console or API users cannot access the private key data) and this data will not be returned in output for reading the role set. However, this key can and should be rotated as necessary to avoid possible theft. Note that rotation does effectively invalidate all secrets previously generated under the roleset.

Service account are created on roleset update/creation rather than per secret.

(NOTE: This is different than AWS!!)

There are a couple of reasons why we want service account creation to happen during role-set creation/updates (i.e. during server set up) rather than on the fly, per secret generated:

• IAM service account creation and permissions propagation can take up to a minute. Because the service account is created and permissions are assigned during set-up, secrets can be used immediately instead of after a possible delay. This can make automated workflows slightly less complicated/flaky.
• Service Account Quotas: GCP projects by default have a limit on the number of IAM service accounts you can create (currently 100, including system-managed service accounts). You can request additional quota, which is better done during a set-up step.
  • If service accounts are created per secret, this limit would instead be on the number of issued secrets.

However, there are a couple of caveats that come from generating per-role-set as well:

• Service account key limit: Because GCP IAM has set a hard limit (currently 10) on the number of service account keys, attempts to generate more keys than the limit will result in an error - Vault is notified on the fly when a user tries to create a new secret and the GCP IAM service returns an error. Thus, if you are generating service account keys, for each service_account_key role set, there is a hard limit of of 10 secrets (keys)* If you find yourself running into this limit, consider:

  • Having shorter TTLs or revoking access earlier: If you're not using service account keys created earlier, consider rotating and freeing quota earlier.
  • Creating more role sets with the same set of permissions: Additional role-sets can always be created with the same set of permissions, which adds service accounts and thus effectively increases the number of key secrets you can generate.
  • Using access_token role sets: If you need several, very short-term accesses to GCP resources, consider instead requesting access_token secrets which have no limit and are naturally short-lived.

• Resources must exist at role set creation time. Because we set the bindings for the service account at this point, if the resource does not exist, calls to getIamPolicy will fail on the resource and the role set creation will fail.

• Role-set creation might take a while and partially fail. Because every service account creation, key creation, and IAM policy change per resource is a GCP API call and we can't do transactions through GCP, if one call fails, the roleset creation fails and Vault will attempt to rollback all changes made. However, these rollback calls themselves are API calls that also might fail. We add WAL entries to ensure that any unused entities or bindings get cleaned up eventually, but you may run into possible situations where you hit quota limits or have unused IAM service accounts that have not been cleaned up. If you manually do this cleanup, the WAL entry will just succeed without doing anything, so feel free to do this (carefully) if you need to immediately clean up issues.

External Changes to Vault-owned IAM Accounts

While Vault will initially create and assign permissions to IAM service accounts, it is possible that an external user deletes this service account and/or Vault-managed keys (for access_token role sets), or adjusts this service account's permissions. We will deny secret generation in the first case until the role set has been rotated or updated, but the second case is hard to detect on the fly.

• If your credentials has unexpected permissions, consider the cases that the service account has been either assigned new external permissions or has possibly inherited permissions from a parent resource policy.
  • Consider just rotating this service account periodically anyways to avoid tampering.
• In general, you should not be changing these service accounts via console/API if you are using Vault. Please warn your GCP project owners to avoid accidentally changing these Vault roleset service accounts.
  • Vault role set accounts have emails with the format vault<roleset-prefix>-<creation UNIX timestamp>@..., where roleset-prefix is the roleset name, possibly truncated to fit the character limit on rolesets. The display name (description) will also have the full Vault roleset name.
• If the service account/key has been deleted, you will need to regenerate the role set account/key using the gcp/$roleset/rotate or gcp/$roleset/rotate-key endpoints. Updates to the role set bindings will also trigger service account recreation.

Calling IAM Methods

An IAM-enabled resource (under an arbitrary GCP service) supports the following three IAM methods:

• getIamPolicy
• setIamPolicy
• testIamPermissions

In the case of this secrets engine, we need to call getIamPolicy and setIamPolicy on an arbitrary resource under an arbitrary service, which would be difficult using the generated Go google APIs. Instead, we autogenerated a library, using the Google API Discovery Service to find IAM-enabled resources and configure HTTP calls on arbitrary services/resources for IAM.

For each binding config resource block (with a resource name), we attempt to find the resource type based on the relative resource name and match it to a service config as seen in this autogenerated config file

To re-generate this file, run:

go generate github.com/hashicorp/vault-plugin-secrets-gcp/plugin/iamutil

In general, we try to make it so you can specify the resource as given in the HTTP API URL (between base API URL and get/setIamPolicy suffix). For some possibly non-standard APIs, we have also added exceptions to try to reach something more standard; a notable current example is the Google Cloud Storage API, whose methods look like https://www.googleapis.com/storage/v1/b/bucket/o/object where we accept either b/bucket/o/object or buckets/bucket/objects/object as valid relative resource names.

If you are having trouble during role set creation with errors suggesting the resource format is invalid or API calls are failing for a resource you know exists, please report any issues you run into. It could be that the API is a non-standard form or we need to re-generate our config file.

API

The GCP secrets engine has a full HTTP API. Please see the GCP secrets engine API docs for more details.