open-vault/website/content/docs/secrets/ssh/signed-ssh-certificates.mdx

---
layout: docs
page_title: Signed SSH Certificates - SSH - Secrets Engines
description: >-
  The signed SSH certificates is the simplest and most powerful in terms of

  setup complexity and in terms of being platform agnostic. When using this

  type, an SSH CA signing key is generated or configured at the secrets engine's
  mount.

  This key will be used to sign other SSH keys.
---

# Signed SSH Certificates

The signed SSH certificates is the simplest and most powerful in terms of setup
complexity and in terms of being platform agnostic. By leveraging Vault's
powerful CA capabilities and functionality built into OpenSSH, clients can SSH
into target hosts using their own local SSH keys.

In this section, the term "**client**" refers to the person or machine
performing the SSH operation. The "**host**" refers to the target machine. If
this is confusing, substitute "client" with "user".

This page will show a quick start for this secrets engine. For detailed documentation
on every path, use `vault path-help` after mounting the secrets engine.

## Client Key Signing

Before a client can request their SSH key be signed, the Vault SSH secrets engine must
be configured. Usually a Vault administrator or security team performs these
steps. It is also possible to automate these actions using a configuration
management tool like Chef, Puppet, Ansible, or Salt.

### Signing Key & Role Configuration

The following steps are performed in advance by a Vault administrator, security
team, or configuration management tooling.

1.  Mount the secrets engine. Like all secrets engines in Vault, the SSH secrets engine
    must be mounted before use.

    ```text
    $ vault secrets enable -path=ssh-client-signer ssh
    Successfully mounted 'ssh' at 'ssh-client-signer'!
    ```

    This enables the SSH secrets engine at the path "ssh-client-signer". It is
    possible to mount the same secrets engine multiple times using different
    `-path` arguments. The name "ssh-client-signer" is not special - it can be
    any name, but this documentation will assume "ssh-client-signer".

1.  Configure Vault with a CA for signing client keys using the `/config/ca`
    endpoint. If you do not have an internal CA, Vault can generate a keypair for
    you.

    ```text
    $ vault write ssh-client-signer/config/ca generate_signing_key=true
    Key             Value
    ---             -----
    public_key      ssh-rsa AAAAB3NzaC1yc2EA...
    ```

    If you already have a keypair, specify the public and private key parts as
    part of the payload:

    ```text
    $ vault write ssh-client-signer/config/ca \
        private_key="..." \
        public_key="..."
    ```

    Regardless of whether it is generated or uploaded, the client signer public
    key is accessible via the API at the `/public_key` endpoint.

1.  Add the public key to all target host's SSH configuration. This process can
    be manual or automated using a configuration management tool. The public key is
    accessible via the API and does not require authentication.

    ```text
    $ curl -o /etc/ssh/trusted-user-ca-keys.pem http://127.0.0.1:8200/v1/ssh-client-signer/public_key
    ```

    ```text
    $ vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem
    ```

    Add the path where the public key contents are stored to the SSH
    configuration file as the `TrustedUserCAKeys` option.

    ```text
    # /etc/ssh/sshd_config
    # ...
    TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
    ```

    Restart the SSH service to pick up the changes.

1.  Create a named Vault role for signing client keys.

    ~> **IMPORTANT NOTE:** Prior to Vault-1.9, if `"allowed_extensions"` is either empty or not specified in the role,
    Vault will assume permissive defaults: any user assigned to the role may specify any arbitrary
    extension values as part of the certificate request to the Vault server.
    This may have significant impact on third-party systems that rely on an `extensions` field for security-critical information.
    In those cases, consider using a template to specify default extensions, and explicitly setting
    `"allowed_extensions"` to an arbitrary, non-empty string if the field is empty or not set.

    Because of the way some SSH certificate features are implemented, options
    are passed as a map. The following example adds the `permit-pty` extension
    to the certificate, and allows the user to specify their own values for `permit-pty` and `permit-port-forwarding`
    when requesting the certificate.

    ```text
    $ vault write ssh-client-signer/roles/my-role -<<"EOH"
    {
      "algorithm_signer": "rsa-sha2-256",
      "allow_user_certificates": true,
      "allowed_users": "*",
      "allowed_extensions": "permit-pty,permit-port-forwarding",
      "default_extensions": {
        "permit-pty": ""
      },
      "key_type": "ca",
      "default_user": "ubuntu",
      "ttl": "30m0s"
    }
    EOH
    ```

### Client SSH Authentication

The following steps are performed by the client (user) that wants to
authenticate to machines managed by Vault. These commands are usually run from
the client's local workstation.

1.  Locate or generate the SSH public key. Usually this is `~/.ssh/id_rsa.pub`.
    If you do not have an SSH keypair, generate one:

    ```text
    $ ssh-keygen -t rsa -C "user@example.com"
    ```

1.  Ask Vault to sign your **public key**. This file usually ends in `.pub` and
    the contents begin with `ssh-rsa ...`.

    ```text
    $ vault write ssh-client-signer/sign/my-role \
        public_key=@$HOME/.ssh/id_rsa.pub

    Key             Value
    ---             -----
    serial_number   c73f26d2340276aa
    signed_key      ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1...
    ```

    The result will include the serial and the signed key. This signed key is
    another public key.

    To customize the signing options, use a JSON payload:

    ```text
    $ vault write ssh-client-signer/sign/my-role -<<"EOH"
    {
      "public_key": "ssh-rsa AAA...",
      "valid_principals": "my-user",
      "key_id": "custom-prefix",
      "extensions": {
        "permit-pty": "",
        "permit-port-forwarding": ""
      }
    }
    EOH
    ```

1.  Save the resulting signed, public key to disk. Limit permissions as needed.

    ```text
    $ vault write -field=signed_key ssh-client-signer/sign/my-role \
        public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
    ```

    If you are saving the certificate directly beside your SSH keypair, suffix
    the name with `-cert.pub` (`~/.ssh/id_rsa-cert.pub`). With this naming
    scheme, OpenSSH will automatically use it during authentication.

1.  (Optional) View enabled extensions, principals, and metadata of the signed
    key.

    ```text
    $ ssh-keygen -Lf ~/.ssh/signed-cert.pub
    ```

1.  SSH into the host machine using the signed key. You must supply both the
    signed public key from Vault **and** the corresponding private key as
    authentication to the SSH call.

    ```text
    $ ssh -i signed-cert.pub -i ~/.ssh/id_rsa username@10.0.23.5
    ```

## Host Key Signing

For an added layers of security, we recommend enabling host key signing. This is
used in conjunction with client key signing to provide an additional integrity
layer. When enabled, the SSH agent will verify the target host is valid and
trusted before attempting to SSH. This will reduce the probability of a user
accidentally SSHing into an unmanaged or malicious machine.

### Signing Key Configuration

1.  Mount the secrets engine. For the most security, mount at a different path from the
    client signer.

    ```text
    $ vault secrets enable -path=ssh-host-signer ssh
    Successfully mounted 'ssh' at 'ssh-host-signer'!
    ```

1.  Configure Vault with a CA for signing host keys using the `/config/ca`
    endpoint. If you do not have an internal CA, Vault can generate a keypair for
    you.

    ```text
    $ vault write ssh-host-signer/config/ca generate_signing_key=true
    Key             Value
    ---             -----
    public_key      ssh-rsa AAAAB3NzaC1yc2EA...
    ```

    If you already have a keypair, specify the public and private key parts as
    part of the payload:

    ```text
    $ vault write ssh-host-signer/config/ca \
        private_key="..." \
        public_key="..."
    ```

    Regardless of whether it is generated or uploaded, the host signer public
    key is accessible via the API at the `/public_key` endpoint.

1.  Extend host key certificate TTLs.

    ```text
    $ vault secrets tune -max-lease-ttl=87600h ssh-host-signer
    ```

1.  Create a role for signing host keys. Be sure to fill in the list of allowed
    domains, set `allow_bare_domains`, or both.

    ```text
    $ vault write ssh-host-signer/roles/hostrole \
        key_type=ca \
        algorithm_signer=rsa-sha2-256 \
        ttl=87600h \
        allow_host_certificates=true \
        allowed_domains="localdomain,example.com" \
        allow_subdomains=true
    ```

1.  Sign the host's SSH public key.

    ```text
    $ vault write ssh-host-signer/sign/hostrole \
        cert_type=host \
        public_key=@/etc/ssh/ssh_host_rsa_key.pub
    Key             Value
    ---             -----
    serial_number   3746eb17371540d9
    signed_key      ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1y...
    ```

1.  Set the resulting signed certificate as `HostCertificate` in the SSH
    configuration on the host machine.

    ```text
    $ vault write -field=signed_key ssh-host-signer/sign/hostrole \
        cert_type=host \
        public_key=@/etc/ssh/ssh_host_rsa_key.pub > /etc/ssh/ssh_host_rsa_key-cert.pub
    ```

    Set permissions on the certificate to be `0640`:

    ```text
    $ chmod 0640 /etc/ssh/ssh_host_rsa_key-cert.pub
    ```

    Add host key and host certificate to the SSH configuration file.

    ```text
    # /etc/ssh/sshd_config
    # ...

    # For client keys
    TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem

    # For host keys
    HostKey /etc/ssh/ssh_host_rsa_key
    HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
    ```

    Restart the SSH service to pick up the changes.

### Client-Side Host Verification

1.  Retrieve the host signing CA public key to validate the host signature of
    target machines.

    ```text
    $ curl http://127.0.0.1:8200/v1/ssh-host-signer/public_key
    ```

    ```text
    $ vault read -field=public_key ssh-host-signer/config/ca
    ```

1.  Add the resulting public key to the `known_hosts` file with authority.

    ```text
    # ~/.ssh/known_hosts
    @cert-authority *.example.com ssh-rsa AAAAB3NzaC1yc2EAAA...
    ```

1.  SSH into target machines as usual.

## Troubleshooting

When initially configuring this type of key signing, enable `VERBOSE` SSH
logging to help annotate any errors in the log.

```text
# /etc/ssh/sshd_config
# ...
LogLevel VERBOSE
```

Restart SSH after making these changes.

By default, SSH logs to `/var/log/auth.log`, but so do many other things. To
extract just the SSH logs, use the following:

```shell-session
$ tail -f /var/log/auth.log | grep --line-buffered "sshd"
```

If you are unable to make a connection to the host, the SSH server logs may
provide guidance and insights.

### Name is not a listed principal

If the `auth.log` displays the following messages:

```text
# /var/log/auth.log
key_cert_check_authority: invalid certificate
Certificate invalid: name is not a listed principal
```

The certificate does not permit the username as a listed principal for
authenticating to the system. This is most likely due to an OpenSSH bug (see
[known issues](#known-issues) for more information). This bug does not respect
the `allowed_users` option value of "\*". Here are ways to work around this
issue:

1.  Set `default_user` in the role. If you are always authenticating as the same
    user, set the `default_user` in the role to the username you are SSHing into the
    target machine:

    ```text
    $ vault write ssh/roles/my-role -<<"EOH"
    {
      "default_user": "YOUR_USER",
      // ...
    }
    EOH
    ```

1.  Set `valid_principals` during signing. In situations where multiple users may
    be authenticating to SSH via Vault, set the list of valid principles during key
    signing to include the current username:

    ```text
    $ vault write ssh-client-signer/sign/my-role -<<"EOH"
    {
      "valid_principals": "my-user"
      // ...
    }
    EOH
    ```

### No Prompt After Login

If you do not see a prompt after authenticating to the host machine, the signed
certificate may not have the `permit-pty` extension. There are two ways to add
this extension to the signed certificate.

- As part of the role creation

  ```text
  $ vault write ssh-client-signer/roles/my-role -<<"EOH"
  {
    "default_extensions": {
      "permit-pty": ""
    }
    // ...
  }
  EOH
  ```

- As part of the signing operation itself:

  ```text
  $ vault write ssh-client-signer/sign/my-role -<<"EOH"
  {
    "extensions": {
      "permit-pty": ""
    }
    // ...
  }
  EOH
  ```

### No Port Forwarding

If port forwarding from the guest to the host is not working, the signed
certificate may not have the `permit-port-forwarding` extension. Add the
extension as part of the role creation or signing process to enable port
forwarding. See [no prompt after login](#no-prompt-after-login) for examples.

```json
{
  "default_extensions": {
    "permit-port-forwarding": ""
  }
}
```

### No X11 Forwarding

If X11 forwarding from the guest to the host is not working, the signed
certificate may not have the `permit-X11-forwarding` extension. Add the
extension as part of the role creation or signing process to enable X11
forwarding. See [no prompt after login](#no-prompt-after-login) for examples.

```json
{
  "default_extensions": {
    "permit-X11-forwarding": ""
  }
}
```

### No Agent Forwarding

If agent forwarding from the guest to the host is not working, the signed
certificate may not have the `permit-agent-forwarding` extension. Add the
extension as part of the role creation or signing process to enable agent
forwarding. See [no prompt after login](#no-prompt-after-login) for examples.

```json
{
  "default_extensions": {
    "permit-agent-forwarding": ""
  }
}
```

### Key Comments
There are additional steps needed to preserve [comment attributes](https://www.rfc-editor.org/rfc/rfc4716#section-3.3.2)
in keys which ought to be considered if they are required. Private and public 
key may have comments applied to them and for example where `ssh-keygen` is used 
with its `-C` parameter - similar to:

```shell-session
ssh-keygen -C "...Comments" -N "" -t rsa -b 4096 -f host-ca
```

Adapted key values containing comments must be provided with the key related 
parameters as per the Vault CLI and API steps demonstrated below. 

```shell-extension
# Using CLI:
vault secrets enable -path=hosts-ca ssh
KEY_PRI=$(cat ~/.ssh/id_rsa | sed -z 's/\n/\\n/g')
KEY_PUB=$(cat ~/.ssh/id_rsa.pub | sed -z 's/\n/\\n/g')
# Create / Update keypair in Vault
vault write ssh-client-signer/config/ca \
  generate_signing_key=false \
  private_key="${KEY_PRI}" \
  public_key="${KEY_PUB}"
```

```shell-extension
# Using API:
curl -X POST -H "X-Vault-Token: ..." -d '{"type":"ssh"}' http://127.0.0.1:8200/v1/sys/mounts/hosts-ca
KEY_PRI=$(cat ~/.ssh/id_rsa | sed -z 's/\n/\\n/g')
KEY_PUB=$(cat ~/.ssh/id_rsa.pub | sed -z 's/\n/\\n/g')
tee payload.json <<EOF
{
  "generate_signing_key" : false,
  "private_key"          : "${KEY_PRI}",
  "public_key"           : "${KEY_PUB}"
}
EOF
# Create / Update keypair in Vault
curl -X POST -H "X-Vault-Token: ..." -d @payload.json http://127.0.0.1:8200/v1/hosts-ca/config/ca
```

~> **IMPORTANT:** Do NOT add a private key password since Vault can't decrypt it.
Destroy the keypair and `payload.json` from your hosts immediately after they have been confirmed as successfully uploaded.

### Known Issues
- On SELinux-enforcing systems, you may need to adjust related types so that the
  SSH daemon is able to read it. For example, adjust the signed host certificate
  to be an `sshd_key_t` type.

- On some versions of SSH, you may get the following error:

  ```text
  no separate private key for certificate
  ```

  This is a bug introduced in OpenSSH version 7.2 and fixed in 7.5. See
  [OpenSSH bug 2617](https://bugzilla.mindrot.org/show_bug.cgi?id=2617) for
  details.

- On some versions of SSH, you may get the following error on target host:

  ```text
  userauth_pubkey: certificate signature algorithm ssh-rsa: signature algorithm not supported [preauth]
  ```
  Fix is to add below line to /etc/ssh/sshd_config
  ```text
  CASignatureAlgorithms ^ssh-rsa
  ```
  The ssh-rsa algorithm is no longer supported in [OpenSSH 8.2](https://www.openssh.com/txt/release-8.2)

## API

The SSH secrets engine has a full HTTP API. Please see the
[SSH secrets engine API](/api-docs/secret/ssh) for more
details.