10c3bad652
Fixes #2522 Skip embedding client.alloc_dir when building chroot. If a user configures a Nomad client agent so that the chroot_env will embed the client.alloc_dir, Nomad will happily infinitely recurse while building the chroot until something horrible happens. The best case scenario is the filesystem's path length limit is hit. The worst case scenario is disk space is exhausted. A bad agent configuration will look something like this: ```hcl data_dir = "/tmp/nomad-badagent" client { enabled = true chroot_env { # Note that the source matches the data_dir "/tmp/nomad-badagent" = "/ohno" # ... } } ``` Note that `/ohno/client` (the state_dir) will still be created but not `/ohno/alloc` (the alloc_dir). While I cannot think of a good reason why someone would want to embed Nomad's client (and possibly server) directories in chroots, there should be no cause for harm. chroots are only built when Nomad runs as root, and Nomad disables running exec jobs as root by default. Therefore even if client state is copied into chroots, it will be inaccessible to tasks. Skipping the `data_dir` and `{client,server}.state_dir` is possible, but this PR attempts to implement the minimum viable solution to reduce risk of unintended side effects or bugs. When running tests as root in a vm without the fix, the following error occurs: ``` === RUN TestAllocDir_SkipAllocDir alloc_dir_test.go:520: Error Trace: alloc_dir_test.go:520 Error: Received unexpected error: Couldn't create destination file /tmp/TestAllocDir_SkipAllocDir1457747331/001/nomad/test/testtask/nomad/test/testtask/.../nomad/test/testtask/secrets/.nomad-mount: open /tmp/TestAllocDir_SkipAllocDir1457747331/001/nomad/test/.../testtask/secrets/.nomad-mount: file name too long Test: TestAllocDir_SkipAllocDir --- FAIL: TestAllocDir_SkipAllocDir (22.76s) ``` Also removed unused Copy methods on AllocDir and TaskDir structs. Thanks to @eveld for not letting me forget about this!
476 lines
16 KiB
Plaintext
476 lines
16 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: client Stanza - Agent Configuration
|
|
description: |-
|
|
The "client" stanza configures the Nomad agent to accept jobs as assigned by
|
|
the Nomad server, join the cluster, and specify driver-specific configuration.
|
|
---
|
|
|
|
# `client` Stanza
|
|
|
|
<Placement groups={['client']} />
|
|
|
|
The `client` stanza configures the Nomad agent to accept jobs as assigned by
|
|
the Nomad server, join the cluster, and specify driver-specific configuration.
|
|
|
|
```hcl
|
|
client {
|
|
enabled = true
|
|
servers = ["1.2.3.4:4647", "5.6.7.8:4647"]
|
|
}
|
|
```
|
|
|
|
## `client` Parameters
|
|
|
|
- `alloc_dir` `(string: "[data_dir]/alloc")` - Specifies the directory to use
|
|
for allocation data. By default, this is the top-level
|
|
[data_dir](/docs/configuration#data_dir) suffixed with
|
|
"alloc", like `"/opt/nomad/alloc"`. This must be an absolute path.
|
|
|
|
- `chroot_env` <code>([ChrootEnv](#chroot_env-parameters): nil)</code> -
|
|
Specifies a key-value mapping that defines the chroot environment for jobs
|
|
using the Exec and Java drivers.
|
|
|
|
- `enabled` `(bool: false)` - Specifies if client mode is enabled. All other
|
|
client configuration options depend on this value.
|
|
|
|
- `max_kill_timeout` `(string: "30s")` - Specifies the maximum amount of time a
|
|
job is allowed to wait to exit. Individual jobs may customize their own kill
|
|
timeout, but it may not exceed this value.
|
|
|
|
- `disable_remote_exec` `(bool: false)` - Specifies if the client should disable
|
|
remote task execution to tasks running on this client.
|
|
|
|
- `meta` `(map[string]string: nil)` - Specifies a key-value map that annotates
|
|
with user-defined metadata.
|
|
|
|
- `network_interface` `(string: varied)` - Specifies the name of the interface
|
|
to force network fingerprinting on. When run in dev mode, this defaults to the
|
|
loopback interface. When not in dev mode, the interface attached to the
|
|
default route is used. The scheduler chooses from these fingerprinted IP
|
|
addresses when allocating ports for tasks. This value support [go-sockaddr/template
|
|
format][go-sockaddr/template].
|
|
|
|
If no non-local IP addresses are found, Nomad could fingerprint link-local IPv6
|
|
addresses depending on the client's
|
|
[`"fingerprint.network.disallow_link_local"`](#fingerprint-network-disallow_link_local)
|
|
configuration value.
|
|
|
|
- `cpu_total_compute` `(int: 0)` - Specifies an override for the total CPU
|
|
compute. This value should be set to `# Cores * Core MHz`. For example, a
|
|
quad-core running at 2 GHz would have a total compute of 8000 (4 \* 2000). Most
|
|
clients can determine their total CPU compute automatically, and thus in most
|
|
cases this should be left unset.
|
|
|
|
- `memory_total_mb` `(int:0)` - Specifies an override for the total memory. If set,
|
|
this value overrides any detected memory.
|
|
|
|
- `min_dynamic_port` `(int:20000)` - Specifies the minimum dynamic port to be
|
|
assigned. Individual ports and ranges of ports may be excluded from dynamic
|
|
port assignment via [`reserved`](#reserved-parameters) parameters.
|
|
|
|
- `max_dynamic_port` `(int:32000)` - Specifies the maximum dynamic port to be
|
|
assigned. Individual ports and ranges of ports may be excluded from dynamic
|
|
port assignment via [`reserved`](#reserved-parameters) parameters.
|
|
|
|
- `node_class` `(string: "")` - Specifies an arbitrary string used to logically
|
|
group client nodes by user-defined class. This can be used during job
|
|
placement as a filter.
|
|
|
|
- `options` <code>([Options](#options-parameters): nil)</code> - Specifies a
|
|
key-value mapping of internal configuration for clients, such as for driver
|
|
configuration.
|
|
|
|
- `reserved` <code>([Reserved](#reserved-parameters): nil)</code> - Specifies
|
|
that Nomad should reserve a portion of the node's resources from receiving
|
|
tasks. This can be used to target a certain capacity usage for the node. For
|
|
example, a value equal to 20% of the node's CPU could be reserved to target
|
|
a CPU utilization of 80%.
|
|
|
|
- `servers` `(array<string>: [])` - Specifies an array of addresses to the Nomad
|
|
servers this client should join. This list is used to register the client with
|
|
the server nodes and advertise the available resources so that the agent can
|
|
receive work. This may be specified as an IP address or DNS, with or without
|
|
the port. If the port is omitted, the default port of `4647` is used.
|
|
|
|
- `server_join` <code>([server_join][server-join]: nil)</code> - Specifies
|
|
how the Nomad client will connect to Nomad servers. The `start_join` field
|
|
is not supported on the client. The retry_join fields may directly specify
|
|
the server address or use go-discover syntax for auto-discovery. See the
|
|
documentation for more detail.
|
|
|
|
- `state_dir` `(string: "[data_dir]/client")` - Specifies the directory to use
|
|
to store client state. By default, this is - the top-level
|
|
[data_dir](/docs/configuration#data_dir) suffixed with
|
|
"client", like `"/opt/nomad/client"`. This must be an absolute path.
|
|
|
|
- `gc_interval` `(string: "1m")` - Specifies the interval at which Nomad
|
|
attempts to garbage collect terminal allocation directories.
|
|
|
|
- `gc_disk_usage_threshold` `(float: 80)` - Specifies the disk usage percent which
|
|
Nomad tries to maintain by garbage collecting terminal allocations.
|
|
|
|
- `gc_inode_usage_threshold` `(float: 70)` - Specifies the inode usage percent
|
|
which Nomad tries to maintain by garbage collecting terminal allocations.
|
|
|
|
- `gc_max_allocs` `(int: 50)` - Specifies the maximum number of allocations
|
|
which a client will track before triggering a garbage collection of terminal
|
|
allocations. This will _not_ limit the number of allocations a node can run at
|
|
a time, however after `gc_max_allocs` every new allocation will cause terminal
|
|
allocations to be GC'd.
|
|
|
|
- `gc_parallel_destroys` `(int: 2)` - Specifies the maximum number of
|
|
parallel destroys allowed by the garbage collector. This value should be
|
|
relatively low to avoid high resource usage during garbage collections.
|
|
|
|
- `no_host_uuid` `(bool: true)` - By default a random node UUID will be
|
|
generated, but setting this to `false` will use the system's UUID. Before
|
|
Nomad 0.6 the default was to use the system UUID.
|
|
|
|
- `cni_path` `(string: "/opt/cni/bin")` - Sets the search path that is used for
|
|
CNI plugin discovery. Multiple paths can be searched using colon delimited
|
|
paths
|
|
|
|
- `cni_config_dir` `(string: "/opt/cni/config")` - Sets the directory where CNI
|
|
network configuration is located. The client will use this path when fingerprinting
|
|
CNI networks. Filenames should use the `.conflist` extension.
|
|
|
|
- `bridge_network_name` `(string: "nomad")` - Sets the name of the bridge to be
|
|
created by nomad for allocations running with bridge networking mode on the
|
|
client.
|
|
|
|
- `bridge_network_subnet` `(string: "172.26.64.0/20")` - Specifies the subnet
|
|
which the client will use to allocate IP addresses from.
|
|
|
|
- `template` <code>([Template](#template-parameters): nil)</code> - Specifies
|
|
controls on the behavior of task
|
|
[`template`](/docs/job-specification/template) stanzas.
|
|
|
|
- `host_volume` <code>([host_volume](#host_volume-stanza): nil)</code> - Exposes
|
|
paths from the host as volumes that can be mounted into jobs.
|
|
|
|
- `host_network` <code>([host_network](#host_network-stanza): nil)</code> - Registers
|
|
additional host networks with the node that can be selected when port mapping.
|
|
|
|
- `cgroup_parent` `(string: "/nomad")` - Specifies the cgroup parent for which cgroup
|
|
subsystems managed by Nomad will be mounted under. Currently this only applies to the
|
|
`cpuset` subsystems. This field is ignored on non Linux platforms.
|
|
|
|
### `chroot_env` Parameters
|
|
|
|
Drivers based on [isolated fork/exec](/docs/drivers/exec) implement file
|
|
system isolation using chroot on Linux. The `chroot_env` map allows the chroot
|
|
environment to be configured using source paths on the host operating system.
|
|
The mapping format is:
|
|
|
|
```text
|
|
source_path -> dest_path
|
|
```
|
|
|
|
The following example specifies a chroot which contains just enough to run the
|
|
`ls` utility:
|
|
|
|
```hcl
|
|
client {
|
|
chroot_env {
|
|
"/bin/ls" = "/bin/ls"
|
|
"/etc/ld.so.cache" = "/etc/ld.so.cache"
|
|
"/etc/ld.so.conf" = "/etc/ld.so.conf"
|
|
"/etc/ld.so.conf.d" = "/etc/ld.so.conf.d"
|
|
"/etc/passwd" = "/etc/passwd"
|
|
"/lib" = "/lib"
|
|
"/lib64" = "/lib64"
|
|
}
|
|
}
|
|
```
|
|
|
|
When `chroot_env` is unspecified, the `exec` driver will use a default chroot
|
|
environment with the most commonly used parts of the operating system. Please
|
|
see the [Nomad `exec` driver documentation](/docs/drivers/exec#chroot) for
|
|
the full list.
|
|
|
|
As of Nomad 1.2, Nomad will never attempt to embed the `alloc_dir` in the
|
|
chroot as doing so would cause infinite recursion.
|
|
|
|
### `options` Parameters
|
|
|
|
~> Note: In Nomad 0.9 client configuration options for drivers were deprecated.
|
|
See the [plugin stanza][plugin-stanza] documentation for more information.
|
|
|
|
The following is not an exhaustive list of options for only the Nomad
|
|
client. To find the options supported by each individual Nomad driver, please
|
|
see the [drivers documentation](/docs/drivers).
|
|
|
|
- `"driver.allowlist"` `(string: "")` - Specifies a comma-separated list of
|
|
allowlisted drivers . If specified, drivers not in the allowlist will be
|
|
disabled. If the allowlist is empty, all drivers are fingerprinted and enabled
|
|
where applicable.
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"driver.allowlist" = "docker,qemu"
|
|
}
|
|
}
|
|
```
|
|
|
|
- `"driver.denylist"` `(string: "")` - Specifies a comma-separated list of
|
|
denylisted drivers . If specified, drivers in the denylist will be
|
|
disabled.
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"driver.denylist" = "docker,qemu"
|
|
}
|
|
}
|
|
```
|
|
|
|
- `"env.denylist"` `(string: see below)` - Specifies a comma-separated list of
|
|
environment variable keys not to pass to these tasks. Nomad passes the host
|
|
environment variables to `exec`, `raw_exec` and `java` tasks. If specified,
|
|
the defaults are overridden. If a value is provided, **all** defaults are
|
|
overridden (they are not merged).
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"env.denylist" = "MY_CUSTOM_ENVVAR"
|
|
}
|
|
}
|
|
```
|
|
|
|
The default list is:
|
|
|
|
```text
|
|
CONSUL_TOKEN
|
|
CONSUL_HTTP_TOKEN
|
|
VAULT_TOKEN
|
|
NOMAD_LICENSE
|
|
AWS_ACCESS_KEY_ID
|
|
AWS_SECRET_ACCESS_KEY
|
|
AWS_SESSION_TOKEN
|
|
GOOGLE_APPLICATION_CREDENTIALS
|
|
```
|
|
|
|
- `"user.denylist"` `(string: see below)` - Specifies a comma-separated
|
|
denylist of usernames for which a task is not allowed to run. This only
|
|
applies if the driver is included in `"user.checked_drivers"`. If a value is
|
|
provided, **all** defaults are overridden (they are not merged).
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"user.denylist" = "root,ubuntu"
|
|
}
|
|
}
|
|
```
|
|
|
|
The default list is:
|
|
|
|
```text
|
|
root
|
|
Administrator
|
|
```
|
|
|
|
- `"user.checked_drivers"` `(string: see below)` - Specifies a comma-separated
|
|
list of drivers for which to enforce the `"user.denylist"`. For drivers using
|
|
containers, this enforcement is usually unnecessary. If a value is provided,
|
|
**all** defaults are overridden (they are not merged).
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"user.checked_drivers" = "exec,raw_exec"
|
|
}
|
|
}
|
|
```
|
|
|
|
The default list is:
|
|
|
|
```text
|
|
exec
|
|
qemu
|
|
java
|
|
```
|
|
|
|
- `"fingerprint.allowlist"` `(string: "")` - Specifies a comma-separated list of
|
|
allowlisted fingerprinters. If specified, any fingerprinters not in the
|
|
allowlist will be disabled. If the allowlist is empty, all fingerprinters are
|
|
used.
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"fingerprint.allowlist" = "network"
|
|
}
|
|
}
|
|
```
|
|
|
|
- `"fingerprint.denylist"` `(string: "")` - Specifies a comma-separated list of
|
|
denylisted fingerprinters. If specified, any fingerprinters in the denylist
|
|
will be disabled.
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"fingerprint.denylist" = "network"
|
|
}
|
|
}
|
|
```
|
|
|
|
- `"fingerprint.network.disallow_link_local"` `(string: "false")` - Specifies
|
|
whether the network fingerprinter should ignore link-local addresses in the
|
|
case that no globally routable address is found. The fingerprinter will always
|
|
prefer globally routable addresses.
|
|
|
|
```hcl
|
|
client {
|
|
options = {
|
|
"fingerprint.network.disallow_link_local" = "true"
|
|
}
|
|
}
|
|
```
|
|
|
|
### `reserved` Parameters
|
|
|
|
- `cpu` `(int: 0)` - Specifies the amount of CPU to reserve, in MHz.
|
|
|
|
- `memory` `(int: 0)` - Specifies the amount of memory to reserve, in MB.
|
|
|
|
- `disk` `(int: 0)` - Specifies the amount of disk to reserve, in MB.
|
|
|
|
- `reserved_ports` `(string: "")` - Specifies a comma-separated list of ports to
|
|
reserve on all fingerprinted network devices. Ranges can be specified by using
|
|
a hyphen separated the two inclusive ends.
|
|
|
|
### `template` Parameters
|
|
|
|
- `function_denylist` `([]string: ["plugin"])` - Specifies a list of template
|
|
rendering functions that should be disallowed in job specs. By default the
|
|
`plugin` function is disallowed as it allows running arbitrary commands on
|
|
the host as root (unless Nomad is configured to run as a non-root user).
|
|
|
|
- `disable_file_sandbox` `(bool: false)` - Allows templates access to arbitrary
|
|
files on the client host via the `file` function. By default templates can
|
|
access files only within the [task working directory].
|
|
|
|
### `host_volume` Stanza
|
|
|
|
The `host_volume` stanza is used to make volumes available to jobs.
|
|
|
|
The key of the stanza corresponds to the name of the volume for use in the
|
|
`source` parameter of a `"host"` type [`volume`](/docs/job-specification/volume)
|
|
and ACLs.
|
|
|
|
```hcl
|
|
client {
|
|
host_volume "ca-certificates" {
|
|
path = "/etc/ssl/certs"
|
|
read_only = true
|
|
}
|
|
}
|
|
```
|
|
|
|
#### `host_volume` Parameters
|
|
|
|
- `path` `(string: "", required)` - Specifies the path on the host that should
|
|
be used as the source when this volume is mounted into a task. The path must
|
|
exist on client startup.
|
|
|
|
- `read_only` `(bool: false)` - Specifies whether the volume should only ever be
|
|
allowed to be mounted `read_only`, or if it should be writeable.
|
|
|
|
### `host_network` Stanza
|
|
|
|
The `host_network` stanza is used to register additional host networks with
|
|
the node that can be used when port mapping.
|
|
|
|
The key of the stanza corresponds to the name of the network used in the
|
|
[`host_network`](/docs/job-specification/network#host-networks).
|
|
|
|
```hcl
|
|
client {
|
|
host_network "public" {
|
|
cidr = "203.0.113.0/24"
|
|
reserved_ports = "22,80"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### `host_network` Parameters
|
|
|
|
- `cidr` `(string: "")` - Specifies a cidr block of addresses to match against.
|
|
If an address is found on the node that is contained by this cidr block, the
|
|
host network will be registered with it.
|
|
|
|
- `interface` `(string: "")` - Filters searching of addresses to a specific interface.
|
|
|
|
- `reserved_ports` `(string: "")` - Specifies a comma-separated list of ports to
|
|
reserve on all fingerprinted network devices. Ranges can be specified by using
|
|
a hyphen separating the two inclusive ends.
|
|
|
|
## `client` Examples
|
|
|
|
### Common Setup
|
|
|
|
This example shows the most basic configuration for a Nomad client joined to a
|
|
cluster.
|
|
|
|
```hcl
|
|
client {
|
|
enabled = true
|
|
server_join {
|
|
retry_join = [ "1.1.1.1", "2.2.2.2" ]
|
|
retry_max = 3
|
|
retry_interval = "15s"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Reserved Resources
|
|
|
|
This example shows a sample configuration for reserving resources to the client.
|
|
This is useful if you want to allocate only a portion of the client's resources
|
|
to jobs.
|
|
|
|
```hcl
|
|
client {
|
|
enabled = true
|
|
|
|
reserved {
|
|
cpu = 500
|
|
memory = 512
|
|
disk = 1024
|
|
reserved_ports = "22,80,8500-8600"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Custom Metadata, Network Speed, and Node Class
|
|
|
|
This example shows a client configuration which customizes the metadata, network
|
|
speed, and node class. The scheduler can use this information while processing
|
|
[constraints][metadata_constraint]. The metadata is completely user configurable;
|
|
the values below are for illustrative purposes only.
|
|
|
|
```hcl
|
|
client {
|
|
enabled = true
|
|
node_class = "prod"
|
|
|
|
meta {
|
|
owner = "ops"
|
|
cached_binaries = "redis,apache,nginx,jq,cypress,nodejs"
|
|
rack = "rack-12-1"
|
|
}
|
|
}
|
|
```
|
|
|
|
[plugin-options]: #plugin-options
|
|
[plugin-stanza]: /docs/configuration/plugin
|
|
[server-join]: /docs/configuration/server_join 'Server Join'
|
|
[metadata_constraint]: /docs/job-specification/constraint#user-specified-metadata 'Nomad User-Specified Metadata Constraint Example'
|
|
[task working directory]: /docs/runtime/environment#task-directories 'Task directories'
|
|
[go-sockaddr/template]: https://godoc.org/github.com/hashicorp/go-sockaddr/template
|