From f64baec276c3d6678ba72e2a213c629dfc47a933 Mon Sep 17 00:00:00 2001 From: Seth Hoenig Date: Sat, 15 May 2021 17:19:23 -0600 Subject: [PATCH] docs: update docs for linux capabilities in exec/java/docker drivers Update docs for allow_caps, cap_add, cap_drop in exec/java/docker driver pages. Also update upgrade guide with guidance on new default linux capabilities for exec and java drivers. --- drivers/shared/capabilities/set.go | 1 + website/content/docs/drivers/docker.mdx | 61 +++++++++++-------- website/content/docs/drivers/exec.mdx | 43 +++++++++++++ website/content/docs/drivers/java.mdx | 43 +++++++++++++ .../content/docs/upgrade/upgrade-specific.mdx | 40 +++++++++++- 5 files changed, 158 insertions(+), 30 deletions(-) diff --git a/drivers/shared/capabilities/set.go b/drivers/shared/capabilities/set.go index 916e3d4d3..d6fe527d9 100644 --- a/drivers/shared/capabilities/set.go +++ b/drivers/shared/capabilities/set.go @@ -23,6 +23,7 @@ type Set struct { data map[string]nothing } +// New creates a new Set setting caps as the initial elements. func New(caps []string) *Set { m := make(map[string]nothing, len(caps)) for _, c := range caps { diff --git a/website/content/docs/drivers/docker.mdx b/website/content/docs/drivers/docker.mdx index 68e2bf4f4..3c202e107 100644 --- a/website/content/docs/drivers/docker.mdx +++ b/website/content/docs/drivers/docker.mdx @@ -452,30 +452,26 @@ config { - `cap_add` - (Optional) A list of Linux capabilities as strings to pass directly to [`--cap-add`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). Effective capabilities (computed from `cap_add` and `cap_drop`) have to match the configured allowlist. - The allowlist can be customized using the [`allow_caps`](#plugin_caps) plugin option key in the client node's configuration. + The allowlist can be customized using the [`allow_caps`][allow_caps] plugin option key in the client node's configuration. For example: - ```hcl - config { - cap_add = [ - "SYS_TIME", - ] - } - ``` +```hcl +config { + cap_add = ["net_raw", sys_time"] +} +``` - `cap_drop` - (Optional) A list of Linux capabilities as strings to pass directly to [`--cap-drop`](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities). Effective capabilities (computed from `cap_add` and `cap_drop`) have to match the configured allowlist. - The allowlist can be customized using the [`allow_caps`](#plugin_caps) plugin option key in the client node's configuration. + The allowlist can be customized using the [`allow_caps`][allow_caps] plugin option key in the client node's configuration. For example: - ```hcl - config { - cap_drop = [ - "MKNOD", - ] - } - ``` +```hcl +config { + cap_drop = ["mknod"] +} +``` - `cpu_hard_limit` - (Optional) `true` or `false` (default). Use hard CPU limiting instead of soft limiting. By default this is `false` which means @@ -797,10 +793,7 @@ plugin "docker" { } allow_privileged = false - allow_caps = ["CHOWN", "NET_RAW"] - - # allow_caps can also be set to "ALL" - # allow_caps = ["ALL"] + allow_caps = ["chown", "net_raw"] } } ``` @@ -823,13 +816,22 @@ plugin "docker" { from the Docker engine during an image pull within this timeframe, Nomad will timeout the request that initiated the pull command. (Minimum of `1m`) -- `allow_caps` - A list of allowed Linux capabilities. - Defaults to - `CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP, NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE` which is the list of - capabilities allowed by docker by default, as defined here. Allows the - operator to control which capabilities can be obtained by tasks using cap_add - and cap_drop options. Supports the value "ALL" as a shortcut for allowlisting - all capabilities. +- `allow_caps` - A list of allowed Linux capabilities. Defaults to + +```hcl +["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod", + "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"] +``` + + which is the same list of capabilities allowed by [docker by default][docker_caps] + (sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities can be obtained + by tasks using [`cap_add`][cap_add] and [`cap_drop`][cap_drop] options. Supports + the value `"all"` as a shortcut for allow-listing all capabilities supported by + the operating system. + +!> **Warning:** Allowing more capabilities beyond the default may lead to +undesirable consequences, including untrusted tasks being able to compromise the +host system. - `allow_runtimes` - defaults to `["runc", "nvidia"]` - A list of the allowed docker runtimes a task may use. @@ -1136,3 +1138,8 @@ Windows is relatively new and rapidly evolving you may want to consult the [plugin-stanza]: /docs/configuration/plugin [allocation working directory]: /docs/runtime/environment#task-directories 'Task Directories' [`auth_soft_fail=true`]: #auth_soft_fail +[cap_add]: /docs/drivers/docker#cap_add +[cap_drop]: /docs/drivers/docker#cap_drop +[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12 +[docker_caps]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities +[allow_caps]: /docs/drivers/docker#allow_caps diff --git a/website/content/docs/drivers/exec.mdx b/website/content/docs/drivers/exec.mdx index 71e491c99..df557b7c7 100644 --- a/website/content/docs/drivers/exec.mdx +++ b/website/content/docs/drivers/exec.mdx @@ -54,6 +54,27 @@ be able to access sensitive process information like environment variables. !> **Warning:** If set to `"host"`, other processes running as the same user will be able to make use of IPC features, like sending unexpected POSIX signals. +- `cap_add` - (Optional) A list of Linux capabilities to enable for the task. + Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset + of the allowed capabilities configured with [`allow_caps`][allow_caps]. + +```hcl +config { + cap_add = ["net_raw", "sys_time"] +} +``` + +- `cap_drop` - (Optional) A list of Linux capabilities to disable for the task. + Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset + of the allowed capabilities configured with [`allow_caps`][allow_caps]. + +```hcl +config { + cap_drop = ["all"] + cap_add = ["chown", "sys_chroot", "mknod"] +} +``` + ## Examples To run a binary present on the Node: @@ -138,6 +159,23 @@ able to make use of IPC features, like sending unexpected POSIX signals. for file system isolation without `pivot_root`. This is useful for systems where the root is on a ramdisk. +- `allow_caps` - A list of allowed Linux capabilities. Defaults to + +```hcl +["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod", + "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"] +``` + + which is modeled after the capabilities allowed by [docker by default][docker_caps] + (sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities + can be obtained by tasks using [`cap_add`][cap_add] and [`cap_drop`][cap_drop] options. + Supports the value `"all"` as a shortcut for allow-listing all capabilities supported + by the operating system. + +!> **Warning:** Allowing more capabilities beyond the default may lead to +undesirable consequences, including untrusted tasks being able to compromise the +host system. + ## Client Attributes The `exec` driver will set the following client attributes: @@ -200,3 +238,8 @@ This list is configurable through the agent client [default_pid_mode]: /docs/drivers/exec#default_pid_mode [default_ipc_mode]: /docs/drivers/exec#default_ipc_mode +[cap_add]: /docs/drivers/exec#cap_add +[cap_drop]: /docs/drivers/exec#cap_drop +[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12 +[allow_caps]: /docs/drivers/exec#allow_caps +[docker_caps]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities diff --git a/website/content/docs/drivers/java.mdx b/website/content/docs/drivers/java.mdx index 8124d491c..f7e180a52 100644 --- a/website/content/docs/drivers/java.mdx +++ b/website/content/docs/drivers/java.mdx @@ -61,6 +61,27 @@ be able to access sensitive process information like environment variables. !> **Warning:** If set to `"host"`, other processes running as the same user will be able to make use of IPC features, like sending unexpected POSIX signals. +- `cap_add` - (Optional) A list of Linux capabilities to enable for the task. + Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset + of the allowed capabilities configured with [`allow_caps`][allow_caps]. + +```hcl +config { + cap_add = ["net_raw", "sys_time"] +} +``` + +- `cap_drop` - (Optional) A list of Linux capabilities to disable for the task. + Effective capabilities (computed from `cap_add` and `cap_drop`) must be a subset + of the allowed capabilities configured with [`allow_caps`][allow_caps]. + +```hcl +config { + cap_drop = ["all"] + cap_add = ["chown", "sys_chroot", "mknod"] +} +``` + ## Examples A simple config block to run a Java Jar: @@ -138,6 +159,23 @@ be able to access sensitive process information like environment variables. !> **Warning:** If set to `"host"`, other processes running as the same user will be able to make use of IPC features, like sending unexpected POSIX signals. +- `allow_caps` - A list of allowed Linux capabilities. Defaults to + +```hcl +["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod", + "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"] +``` + + which is modeled after the capabilities allowed by [docker by default][docker_caps] + (sans [`NET_RAW`][no_net_raw]). Allows the operator to control which capabilities + can be obtained by tasks using [`cap_add`][cap_add] and [`cap_drop`][cap_drop] options. + Supports the value `"all"` as a shortcut for allow-listing all capabilities supported + by the operating system. + +!> **Warning:** Allowing more capabilities beyond the default may lead to +undesirable consequences, including untrusted tasks being able to compromise the +host system. + ## Client Requirements The `java` driver requires Java to be installed and in your system's `$PATH`. On @@ -208,3 +246,8 @@ This list is configurable through the agent client [default_pid_mode]: /docs/drivers/java#default_pid_mode [default_ipc_mode]: /docs/drivers/java#default_ipc_mode +[cap_add]: /docs/drivers/java#cap_add +[cap_drop]: /docs/drivers/java#cap_drop +[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12 +[allow_caps]: /docs/drivers/java#allow_caps +[docker_caps]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities diff --git a/website/content/docs/upgrade/upgrade-specific.mdx b/website/content/docs/upgrade/upgrade-specific.mdx index 2ff1ade19..e74d16f3c 100644 --- a/website/content/docs/upgrade/upgrade-specific.mdx +++ b/website/content/docs/upgrade/upgrade-specific.mdx @@ -54,7 +54,36 @@ these fields. Connect native tasks running in host networking mode will now have `CONSUL_HTTP_ADDR` set automatically. Before this was only the case for bridge networking. If an operator -already explicitly set `CONSUL_HTTP_ADDR` then it will not get overriden. +already explicitly set `CONSUL_HTTP_ADDR` then it will not get overridden. + +#### Linux capabilities in exec/java + +Following the security [remediation][no_net_raw] in Nomad versions 0.12.12, 1.0.5, +and 1.1.0-rc1, the `exec` and `java` task drivers will additionally no longer enable +the following linux capabilities by default: + +``` +AUDIT_CONTROL AUDIT_READ BLOCK_SUSPEND DAC_READ_SEARCH IPC_LOCK IPC_OWNER LEASE +LINUX_IMMUTABLE MAC_ADMIN MAC_OVERRIDE NET_ADMIN NET_BROADCAST SYS_ADMIN +SYS_BOOT SYSLOG SYS_MODULE SYS_NICE SYS_PACCT SYS_PTRACE SYS_RAWIO SYS_RESOURCE +SYS_TIME SYS_TTY_CONFIG WAKE_ALARM +``` + +The capabilities now enabled by default are modeled after Docker default [`linux capabilities`]: + +``` +AUDIT_WRITE CHOWN DAC_OVERRIDE FOWNER FSETID KILL MKNOD NET_BIND_SERVICE +NET_RAW SETFCAP SETGID SETPCAP SETUID SYS_CHROOT +``` + +A new `allow_caps` plugin configuration parameter for [`exec`][allow_caps_exec] +and [`java`][allow_caps_java] task drivers can be used to restrict the set of +capabilities allowed for use by tasks. + +Tasks using the `exec` or `java` task drivers can add or remove desired linux +capabilities using the [`cap_add`][cap_add_exec] and [`cap_drop`][cap_drop_exec] +task configuration options. + #### iptables @@ -63,9 +92,9 @@ inserting them as the first rule. This allows better control for user-defined iptables rules but users who append rules currently should verify that their rules are being appended in the correct order. -## Nomad 1.1.0, 1.0.5, 0.12.12 +## Nomad 1.1.0-rc1, 1.0.5, 0.12.12 -Nomad versions 1.1.0, 1.0.5 and 0.12.12 change the behavior of the `docker`, `exec`, +Nomad versions 1.1.0-rc1, 1.0.5 and 0.12.12 change the behavior of the `docker`, `exec`, and `java` task drivers so that the [`CAP_NET_RAW`] linux capability is disabled by default. This is one of the [`linux capabilities`] that Docker itself enables by default, as this capability enables the generation of ICMP packets - used by @@ -1111,3 +1140,8 @@ deleted and then Nomad 0.3.0 can be launched. [`CAP_NET_RAW`]: https://security.stackexchange.com/a/128988 [`linux capabilities`]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities [`allow_caps`]: /docs/drivers/docker#allow_caps +[no_net_raw]: /docs/upgrade/upgrade-specific#nomad-1-1-0-rc1-1-0-5-0-12-12 +[allow_caps_exec]: /docs/drivers/exec#allow_caps +[allow_caps_java]: /docs/drivers/java#allow_caps +[cap_add_exec]: /docs/drivers/exec#cap_add +[cap_drop_exec]: /docs/drivers/exec#cap_drop