2a5c423ae0
Disallowing per_alloc for host volumes in some cases makes life of a nomad user much harder. When we rely on the NOMAD_ALLOC_INDEX for any configuration that needs to be re-used across restarts we need to make sure allocation placement is consistent. With CSI volumes we can use the `per_alloc` feature but for some reason this is explicitly disabled for host volumes. Ensure host volumes understand the concept of per_alloc
207 lines
6.7 KiB
Plaintext
207 lines
6.7 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: volume Stanza - Job Specification
|
|
description: >-
|
|
The "volume" stanza allows the group to specify that it requires a given
|
|
volume from the cluster. Nomad will automatically handle ensuring that the
|
|
volume is available and mounted into the task.
|
|
---
|
|
|
|
# `volume` Stanza
|
|
|
|
<Placement groups={['job', 'group', 'volume']} />
|
|
|
|
The `volume` stanza allows the group to specify that it requires a
|
|
given volume from the cluster.
|
|
|
|
The key of the stanza is the name of the volume as it will be exposed
|
|
to task configuration.
|
|
|
|
```hcl
|
|
job "docs" {
|
|
group "example" {
|
|
volume "certs" {
|
|
type = "host"
|
|
source = "ca-certificates"
|
|
read_only = true
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
```hcl
|
|
job "docs" {
|
|
group "example" {
|
|
volume "data" {
|
|
type = "csi"
|
|
source = "csi-volume"
|
|
read_only = true
|
|
attachment_mode = "file-system"
|
|
access_mode = "single-node-writer"
|
|
per_alloc = true
|
|
|
|
mount_options {
|
|
fs_type = "ext4"
|
|
mount_flags = ["noatime"]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
|
|
The Nomad server will ensure that the allocations are only scheduled
|
|
on hosts that have a set of volumes that meet the criteria specified
|
|
in the `volume` stanzas. These may be [host volumes][host_volume]
|
|
configured on the client, or [CSI volumes][csi_volume] dynamically
|
|
mounted by [CSI plugins][csi_plugin].
|
|
|
|
The Nomad client will make the volumes available to tasks according to
|
|
the [volume_mount][volume_mount] stanza in the `task` configuration.
|
|
|
|
## `volume` Parameters
|
|
|
|
- `type` `(string: "")` - Specifies the type of a given volume. The
|
|
valid volume types are `"host"` and `"csi"`.
|
|
|
|
- `source` `(string: <required>)` - The name of the volume to
|
|
request. When using `host_volume`'s this should match the published
|
|
name of the host volume. When using `csi` volumes, this should match
|
|
the ID of the registered volume.
|
|
|
|
- `read_only` `(bool: false)` - Specifies that the group only requires
|
|
read only access to a volume and is used as the default value for
|
|
the `volume_mount -> read_only` configuration. This value is also
|
|
used for validating `host_volume` ACLs and for scheduling when a
|
|
matching `host_volume` requires `read_only` usage.
|
|
|
|
- `per_alloc` `(bool: false)` - Specifies that the `source` of the volume
|
|
should have the suffix `[n]`, where `n` is the allocation index. This allows
|
|
mounting a unique volume per allocation, so long as the volume's source is
|
|
named appropriately. For example, with the source `myvolume` and `per_alloc
|
|
= true`, the allocation named `myjob.mygroup.mytask[0]` will require a
|
|
volume ID `myvolume[0]`.
|
|
|
|
The following fields are only valid for volumes with `type = "csi"`:
|
|
|
|
- `access_mode` `(string: <required>)` - Defines whether a volume should be
|
|
available concurrently. The `access_mode` and `attachment_mode` together
|
|
must exactly match one of the volume's `capability` blocks. Can be one of
|
|
`"single-node-reader-only"`, `"single-node-writer"`,
|
|
`"multi-node-reader-only"`, `"multi-node-single-writer"`, or
|
|
`"multi-node-multi-writer"`. Most CSI plugins support only single-node
|
|
modes. Consult the documentation of the storage provider and CSI plugin.
|
|
|
|
- `attachment_mode` `(string: <required>)` - The storage API that will be used
|
|
by the volume. The `access_mode` and `attachment_mode` together must exactly
|
|
match one of the volume's `capability` blocks. Most storage providers will
|
|
support `"file-system"`, to mount volumes using the CSI filesystem API. Some
|
|
storage providers will support `"block-device"`, which will mount the volume
|
|
with the CSI block device API within the container.
|
|
|
|
- `mount_options` - Options for mounting CSI volumes that have the
|
|
`file-system` [attachment mode]. These options override the `mount_options`
|
|
field from [volume registration]. Consult the documentation for your storage
|
|
provider and CSI plugin as to whether these options are required or
|
|
necessary.
|
|
|
|
- `fs_type`: file system type (ex. `"ext4"`)
|
|
- `mount_flags`: the flags passed to `mount` (ex. `["ro", "noatime"]`)
|
|
|
|
## Volume Interpolation
|
|
|
|
Because volumes represent state, many workloads with multiple allocations will
|
|
want to mount specific volumes to specific tasks. The `volume` block is used
|
|
to schedule workloads, so `${NOMAD_ALLOC_INDEX}` can't be used directly in the
|
|
`volume.source` field. The following job specification demonstrates how to use
|
|
multiple volumes with multiple allocations, using the `per_alloc` field. This
|
|
job specification also shows using HCL2 -variables interpolation to expose
|
|
information to the task's environment.
|
|
|
|
```hcl
|
|
variables {
|
|
path = "test"
|
|
}
|
|
|
|
job "example" {
|
|
datacenters = ["dc1"]
|
|
|
|
group "cache" {
|
|
|
|
count = 2
|
|
|
|
volume "cache-volume" {
|
|
type = "csi"
|
|
source = "test-volume"
|
|
attachment_mode = "file-system"
|
|
access_mode = "single-node-writer"
|
|
per_alloc = true
|
|
}
|
|
|
|
network {
|
|
port "db" {
|
|
to = 6379
|
|
}
|
|
}
|
|
|
|
task "redis" {
|
|
driver = "docker"
|
|
config {
|
|
image = "redis:7"
|
|
ports = ["db"]
|
|
}
|
|
resources {
|
|
cpu = 500
|
|
memory = 256
|
|
}
|
|
|
|
env {
|
|
# this will be available as the MOUNT_PATH environment
|
|
# variable in the task
|
|
MOUNT_PATH = "${NOMAD_ALLOC_DIR}/${var.path}"
|
|
}
|
|
|
|
volume_mount {
|
|
volume = "cache-volume"
|
|
destination = "${NOMAD_ALLOC_DIR}/${var.path}"
|
|
}
|
|
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
The job that results from this job specification has two task groups, each one
|
|
named for each of the two volumes. Each allocation has its own volume.
|
|
|
|
```shell-session
|
|
$ nomad job status example
|
|
ID = example
|
|
...
|
|
|
|
Allocations
|
|
ID Node ID Task Group Version Desired Status Created Modified
|
|
81d32909 352c6926 cache-1 0 run running 4s ago 3s ago
|
|
ce6fbfc8 352c6926 cache-0 0 run running 4s ago 3s ago
|
|
|
|
$ nomad volume status 'test-volume[0]'
|
|
...
|
|
Allocations
|
|
ID Node ID Task Group Version Desired Status Created Modified
|
|
ce6fbfc8 352c6926 cache-0 0 run running 29s ago 18s ago
|
|
|
|
$ nomad volume status 'test-volume[1]'
|
|
...
|
|
Allocations
|
|
ID Node ID Task Group Version Desired Status Created Modified
|
|
81d32909 352c6926 cache-0 0 run running 29s ago 18s ago
|
|
```
|
|
|
|
[volume_mount]: /nomad/docs/job-specification/volume_mount 'Nomad volume_mount Job Specification'
|
|
[host_volume]: /nomad/docs/configuration/client#host_volume-stanza
|
|
[csi_volume]: /nomad/docs/commands/volume/register
|
|
[csi_plugin]: /nomad/docs/job-specification/csi_plugin
|
|
[csi_volume]: /nomad/docs/commands/volume/register
|
|
[attachment mode]: /nomad/docs/commands/volume/register#attachment_mode
|
|
[volume registration]: /nomad/docs/commands/volume/register#mount_options
|