open-nomad/website/source/guides/security/vault-pki-integration.html.md

layout	page_title	sidebar_current	description
guides	Vault PKI Secrets Engine Integration	guides-security-vault-pki	Securing Nomad's cluster communication with TLS is important for both security and easing operations. Nomad can use mutual TLS (mTLS) for authenticating for all HTTP and RPC communication. This guide will leverage Vault's PKI secrets engine to accomplish this task.

Vault PKI Secrets Engine Integration

You can use Consul Template in your Nomad cluster to integrate with Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates. By using this method, you enable each node to have a unique certificate with a relatively short ttl. This feature, along with automatic certificate rotation, allows you to safely and securely scale your cluster while using mutual TLS (mTLS).

Reference Material

• Vault PKI Secrets Engine
• Consul Template
• Build Your Own Certificate Authority (CA)

Estimated Time to Complete

25 minutes

Challenge

Secure your existing Nomad cluster with mTLS. Configure a root and intermediate CA in Vault and ensure (with the help of Consul Template) that you are periodically renewing your X.509 certificates on all nodes to maintain a healthy cluster state.

Solution

Enable TLS in your Nomad cluster configuration. Additionally, configure Consul Template on all nodes along with the appropriate templates to communicate with Vault and ensure all nodes are dynamically generating/renewing their X.509 certificates.

Prerequisites

To perform the tasks described in this guide, you need to have a Nomad environment with Consul and Vault installed. You can use this repo to easily provision a sandbox environment. This guide will assume a cluster with one server node and three client nodes.

~> Please Note: This guide is for demo purposes and is only using a single Nomad server with a Vault server configured alongside it. In a production cluster, 3 or 5 Nomad server nodes are recommended along with a separate Vault cluster. Please see Vault Reference Architecture to learn how to securely deploy a Vault cluster.

Steps

Step 1: Initialize Vault Server

Run the following command to initialize Vault server and receive an unseal key and initial root token (if you are running the environment provided in this guide, the Vault server is co-located with the Nomad server). Be sure to note the unseal key and initial root token as you will need these two pieces of information.

$ vault operator init -key-shares=1 -key-threshold=1

The vault operator init command above creates a single Vault unseal key for convenience. For a production environment, it is recommended that you create at least five unseal key shares and securely distribute them to independent operators. The vault operator init command defaults to five key shares and a key threshold of three. If you provisioned more than one server, the others will become standby nodes but should still be unsealed.

Step 2: Unseal Vault

Run the following command and then provide your unseal key to Vault.

$ vault operator unseal

The output of unsealing Vault will look similar to the following:

Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           1
Threshold              1
Version                1.0.3
Cluster Name           vault-cluster-d1b6513f
Cluster ID             87d6d13f-4b92-60ce-1f70-41a66412b0f1
HA Enabled             true
HA Cluster             n/a
HA Mode                standby
Active Node Address    <none>

Step 3: Log in to Vault

Use the login command to authenticate yourself against Vault using the initial root token you received earlier. You will need to authenticate to run the necessary commands to write policies, create roles, and configure your root and intermediate CAs.

$ vault login <your initial root token>

If your login is successful, you will see output similar to what is shown below:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
...

Step 4: Generate the Root CA

Enable the PKI secrets engine at the pki path:

$ vault secrets enable pki

Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours:

$ vault secrets tune -max-lease-ttl=87600h pki

• Please note: we are using a common and recommended pattern which is to have one mount act as the root CA and to use this CA only to sign intermediate CA CSRs from other PKI secrets engines (which we will create in the next few steps). For tighter security, you can store your CA outside of Vault and use the PKI engine only as an intermediate CA.

Generate the root certificate and save the certificate as CA_cert.crt:

$ vault write -field=certificate pki/root/generate/internal \
    common_name="global.nomad" ttl=87600h > CA_cert.crt

Step 5: Generate the Intermediate CA and CSR

Enable the PKI secrets engine at the pki_int path:

$ vault secrets enable -path=pki_int pki

Tune the PKI secrets engine at the pki_int path to issue certificates with a maximum time-to-live (TTL) of 43800 hours:

$ vault secrets tune -max-lease-ttl=43800h pki_int

Generate a CSR from your intermediate CA and save it as pki_intermediate.csr:

$ vault write -format=json pki_int/intermediate/generate/internal \
    common_name="global.nomad Intermediate Authority" \
    ttl="43800h" | jq -r '.data.csr' > pki_intermediate.csr

Step 6: Sign the CSR and Configure Intermediate CA Certificate

Sign the intermediate CA CSR with the root certificate and save the generated certificate as intermediate.cert.pem:

$ vault write -format=json pki/root/sign-intermediate \
    csr=@pki_intermediate.csr format=pem_bundle \
    ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem

Once the CSR is signed and the root CA returns a certificate, it can be imported back into Vault:

vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Step 7: Create a Role

A role is a logical name that maps to a policy used to generate credentials. In our example, it will allow you to use configuration parameters that specify certificate common names, designate alternate names, and enable subdomains along with a few other key settings.

Create a role named nomad-cluster that specifies the allowed domains, enables you to create certificates for subdomains, and generates certificates with a TTL of 86400 seconds (24 hours).

$ vault write pki_int/roles/nomad-cluster allowed_domains=global.nomad \
    allow_subdomains=true max_ttl=86400s require_cn=false generate_lease=true

You should see the following output if the command you issues was successful:

Success! Data written to: pki_int/roles/nomad-cluster

Step 8: Create a Policy to Access the Role Endpoint

Recall from Step 1 that we generated a root token that we used to log in to Vault. Although we could use that token in our next steps to generate our TLS certs, the recommended security approach is to create a new token based on a specific policy with limited privileges.

Create a policy file named tls-policy.hcl and provide it the following contents:

path "pki_int/issue/nomad-cluster" {
  capabilities = ["update"]
}

Note that we have are specifying the update capability on the path pki_int/issue/nomad-cluster. All other privileges will be denied. You can read more about Vault policies here.

Write the policy we just created into Vault:

$ vault policy write tls-policy tls-policy.hcl
Success! Uploaded policy: tls-policy

Step 9: Generate a Token based on tls-policy

Create a token based on tls-policy with the following command:

$ vault token create -policy="tls-policy" -ttl=24h

If the command is successful, you will see output similar to the following:

Key                  Value
---                  -----
token                s.xafiYzh7MCMotHLu2d35hepR
token_accessor       9vj7q5nnF53JAcTyxvccpAZK
token_duration       24h
token_renewable      true
token_policies       ["default" "tls-policy"]
identity_policies    []
policies             ["default" "tls-policy"]

Make a note of this token as you will need it in the upcoming steps.

Step 10: Create and Populate the Templates Directory

We need to create templates that Consul Template can use to render the actual certificates and keys on the nodes in our cluster. In this guide, we will place these templates in /opt/nomad/templates.

Create a directory called templates in /opt/nomad:

$ sudo mkdir /opt/nomad/templates

Below are the templates that the Consul Template configuration will use. We will provide different templates to the nodes depending on whether they are server nodes or client nodes. All of the nodes will get the CLI templates (since we want to use the CLI on any of the nodes).

For Nomad Servers:

agent.crt.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.certificate }}
{{ end }}

agent.key.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.private_key }}
{{ end }}

ca.crt.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h"}}
{{ .Data.issuing_ca }}
{{ end }}

For Nomad Clients:

Replace the word server in the common_name option in each template with the word client.

agent.crt.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.certificate }}
{{ end }}

agent.key.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.private_key }}
{{ end }}

ca.crt.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h"}}
{{ .Data.issuing_ca }}
{{ end }}

For Nomad CLI (provide this on all nodes):

cli.crt.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "ttl=24h" }}
{{ .Data.certificate }}
{{ end }}

cli.key.tpl:

{{ with secret "pki_int/issue/nomad-cluster" "ttl=24h" }}
{{ .Data.private_key }}
{{ end }}

Step 11: Configure Consul Template on All Nodes

If you are using the AWS environment provided in this guide, you already have Consul Template installed on all nodes. If you are using your own environment, please make sure Consul Template is installed. You can download it here.

Provide the token you created in Step 9 to the Consul Template configuration file located at /etc/consul-template.d/consul-template.hcl. You will also need to specify the template stanza so you can render each of the following on your nodes at the specified location from the templates you created in the previous step:

• Node certificate
• Node private key
• CA public certificate

We will also specify the template stanza to create certs and keys from the templates we previously created for the Nomad CLI (which defaults to HTTP but will need to use HTTPS once once TLS is enabled in our cluster).

Your consul-template.hcl configuration file should look similar to the following (you will need to provide this to each node in the cluster):

# This denotes the start of the configuration section for Vault. All values
# contained in this section pertain to Vault.
vault {
  # This is the address of the Vault leader. The protocol (http(s)) portion
  # of the address is required.
  address      = "http://active.vault.service.consul:8200"

  # This value can also be specified via the environment variable VAULT_TOKEN.
  token        = "s.xafiYzh7MCMotHLu2d35hepR"

  # This should also be less than or around 1/3 of your TTL for a predictable
  # behaviour. See https://github.com/hashicorp/vault/issues/3414
  grace        = "1s"

  # This tells Consul Template that the provided token is actually a wrapped
  # token that should be unwrapped using Vault's cubbyhole response wrapping
  # before being used. Please see Vault's cubbyhole response wrapping
  # documentation for more information.
  unwrap_token = false

  # This option tells Consul Template to automatically renew the Vault token
  # given. If you are unfamiliar with Vault's architecture, Vault requires
  # tokens be renewed at some regular interval or they will be revoked. Consul
  # Template will automatically renew the token at half the lease duration of
  # the token. The default value is true, but this option can be disabled if
  # you want to renew the Vault token using an out-of-band process.
  renew_token  = true
}

# This block defines the configuration for connecting to a syslog server for
# logging.
syslog {
  enabled  = true

  # This is the name of the syslog facility to log to.
  facility = "LOCAL5"
}

# This block defines the configuration for a template. Unlike other blocks,
# this block may be specified multiple times to configure multiple templates.
template {
  # This is the source file on disk to use as the input template. This is often
  # called the "Consul Template template". 
  source      = "/opt/nomad/templates/agent.crt.tpl"

  # This is the destination path on disk where the source template will render.
  # If the parent directories do not exist, Consul Template will attempt to
  # create them, unless create_dest_dirs is false.
  destination = "/opt/nomad/agent-certs/agent.crt"

  # This is the permission to render the file. If this option is left
  # unspecified, Consul Template will attempt to match the permissions of the
  # file that already exists at the destination path. If no file exists at that
  # path, the permissions are 0644.
  perms       = 0700

  # This is the optional command to run when the template is rendered. The
  # command will only run if the resulting template changes. 
  command     = "systemctl reload nomad"
}

template {
  source      = "/opt/nomad/templates/agent.key.tpl"
  destination = "/opt/nomad/agent-certs/agent.key"
  perms       = 0700
  command     = "systemctl reload nomad"
}

template {
  source      = "/opt/nomad/templates/ca.crt.tpl"
  destination = "/opt/nomad/agent-certs/ca.crt"
  command     = "systemctl reload nomad"
}

# The following template stanzas are for the CLI certs

template {
  source      = "/opt/nomad/templates/cli.crt.tpl"
  destination = "/opt/nomad/cli-certs/cli.crt"
}

template {
  source      = "/opt/nomad/templates/cli.key.tpl"
  destination = "/opt/nomad/cli-certs/cli.key"
}

!> Note: we have hard-coded the token we created into the Consul Template configuration file. Although we can avoid this by assigning it to the environment variable VAULT_TOKEN, this method can still pose a security concern. The recommended approach is to securely introduce this token to Consul Template. To learn how to accomplish this, see Secure Introduction.

• Please also note we have applied file permissions 0700 to the agent.crt and agent.key since only the root user should be able to read those files. Any other user using the Nomad CLI will be able to read the CLI certs and key that we have created for them along with intermediate CA cert.

Step 12: Start the Consul Template Service

Start the Consul Template service on each node:

$ sudo systemctl start consul-template

You can quickly confirm the appropriate certs and private keys were generated in the destination directory you specified in your Consul Template configuration by listing them out:

$ ls /opt/nomad/agent-certs/ /opt/nomad/cli-certs/
/opt/nomad/agent-certs/:
agent.crt  agent.key  ca.crt

/opt/nomad/cli-certs/:
cli.crt  cli.key

Step 13: Configure Nomad to Use TLS

Add the following tls stanza to the configuration of all Nomad agents (servers and clients) in the cluster (configuration file located at /etc/nomad.d/nomad.hcl in this example):

tls {
  http = true
  rpc  = true

  ca_file   = "/opt/nomad/agent-certs/ca.crt"
  cert_file = "/opt/nomad/agent-certs/agent.crt"
  key_file  = "/opt/nomad/agent-certs/agent.key"

  verify_server_hostname = true
  verify_https_client    = true
}

Additionally, ensure the rpc_upgrade_mode option is set to true on your server nodes (this is to ensure the Nomad servers will accept both TLS and non-TLS connections during the upgrade):

rpc_upgrade_mode       = true

Reload Nomad's configuration on all nodes:

$ systemctl reload nomad

Once Nomad has been reloaded on all nodes, go back to your server nodes and change the rpc_upgrade_mode option to false (or remove the line since the option defaults to false) so that your Nomad servers will only accept TLS connections:

rpc_upgrade_mode       = false

You will need to reload Nomad on your servers after changing this setting. You can read more about RPC Upgrade Mode here.

If you run nomad status, you will now receive the following error:

Error querying jobs: Get http://172.31.52.215:4646/v1/jobs: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

This is because the Nomad CLI defaults to communicating via HTTP instead of HTTPS. We can configure the local Nomad client to connect using TLS and specify our custom key and certificates by setting the following environments variables:

export NOMAD_ADDR=https://localhost:4646
export NOMAD_CACERT="/opt/nomad/agent-certs/ca.crt"
export NOMAD_CLIENT_CERT="/opt/nomad/cli-certs/cli.crt"
export NOMAD_CLIENT_KEY="/opt/nomad/cli-certs/cli.key"

After these environment variables are correctly configured, the CLI will respond as expected:

$ nomad status
No running jobs

Encrypt Server Gossip

At this point all of Nomad's RPC and HTTP communication is secured with mTLS. However, Nomad servers also communicate with a gossip protocol, Serf, that does not use TLS:

• HTTP - Used to communicate between CLI and Nomad agents. Secured by mTLS.
• RPC - Used to communicate between Nomad agents. Secured by mTLS.
• Serf - Used to communicate between Nomad servers. Secured by a shared key.

Nomad server's gossip protocol use a shared key instead of TLS for encryption. This encryption key must be added to every server's configuration using the encrypt parameter or with the -encrypt command line option.

The Nomad CLI includes a operator keygen command for generating a new secure gossip encryption key:

$ nomad operator keygen
cg8StVXbQJ0gPvMd9o7yrg==

Alternatively, you can use any method that base64 encodes 16 random bytes:

$ openssl rand -base64 16
raZjciP8vikXng2S5X0m9w==
$ dd if=/dev/urandom bs=16 count=1 status=none | base64
LsuYyj93KVfT3pAJPMMCgA==

Put the same generated key into every server's configuration file or command line arguments:

server {
  enabled = true

  # Self-elect, should be 3 or 5 for production
  bootstrap_expect = 1

  # Encrypt gossip communication
  encrypt = "cg8StVXbQJ0gPvMd9o7yrg=="
}

Unlike with TLS, reloading Nomad will not be enough to initiate encryption of gossip traffic. At this point, you may restart each Nomad server with systemctl restart nomad.