luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/website/content/api-docs/volumes.mdx
Tim Gross 3fc7482ecd
CSI: failed allocation should not block its own controller unpublish (#14484) 
A Nomad user reported problems with CSI volumes associated with failed
allocations, where the Nomad server did not send a controller unpublish RPC.

The controller unpublish is skipped if other non-terminal allocations on the
same node claim the volume. The check has a bug where the allocation belonging
to the claim being freed was included in the check incorrectly. During a normal
allocation stop for job stop or a new version of the job, the allocation is
terminal. But allocations that fail are not yet marked terminal at the point in
time when the client sends the unpublish RPC to the server.

For CSI plugins that support controller attach/detach, this means that the
controller will not be able to detach the volume from the allocation's host and
the replacement claim will fail until a GC is run. This changeset fixes the
conditional so that the claim's own allocation is not included, and makes the
logic easier to read. Include a test case covering this path.

Also includes two minor extra bugfixes:

* Entities we get from the state store should always be copied before
altering. Ensure that we copy the volume in the top-level unpublish workflow
before handing off to the steps.

* The list stub object for volumes in `nomad/structs` did not match the stub
object in `api`. The `api` package also did not include the current
readers/writers fields that are expected by the UI. True up the two objects and
add the previously undocumented fields to the docs.
2022-09-08 13:30:05 -04:00

788 lines
23 KiB
Plaintext
Raw Blame History

---
layout: api
page_title: Volumes - HTTP API
description: The `/volume` endpoints are used to query for and interact with volumes.
---
# Volumes HTTP API
The `/volume` and `/volumes` endpoints are used to query for and interact with
volumes.
## List Volumes
This endpoint lists all volumes.
| Method | Path | Produces |
| ------ | ------------- | ------------------ |
| `GET` | `/v1/volumes` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | --------------------------- |
| `YES` | `namespace:csi-list-volume` |
### Parameters
- `type` `(string: "")` - Specifies the type of volume to
query. Currently only supports `csi`. This is specified as a query
string parameter. Returns an empty list if omitted.
- `node_id` `(string: "")` - Specifies a string to filter volumes
based on an Node ID prefix. Because the value is decoded to bytes,
the prefix must have an even number of hexadecimal characters
(0-9a-f). This is specified as a query string parameter.
- `plugin_id` `(string: "")` - Specifies a string to filter volumes
based on a plugin ID prefix. Because the value is decoded to bytes,
the prefix must have an even number of hexadecimal characters
(0-9a-f). This is specified as a query string parameter.
- `next_token` `(string: "")` - This endpoint supports paging. The `next_token`
parameter accepts a string which identifies the next expected volume. This
value can be obtained from the `X-Nomad-NextToken` header from the previous
response.
- `per_page` `(int: 0)` - Specifies a maximum number of volumes to return for
this request. If omitted, the response is not paginated. The value of the
`X-Nomad-NextToken` header of the last response can be used as the
`next_token` of the next request to fetch additional pages.
- `filter` `(string: "")` - Specifies the [expression](/api-docs#filtering)
used to filter the results. Consider using pagination or a query parameter to
reduce resource used to serve the request.
### Sample Request
```shell-session
$ curl \
https://localhost:4646/v1/volumes?type=csi&node_id=foo&plugin_id=plugin-id1
```
### Sample Response
```json
[
{
"ID": "volume-id1",
"ExternalID": "vol-abcdef",
"Namespace": "default",
"Name": "volume id1",
"Topologies": [
{
"foo": "bar"
}
],
"AccessMode": "multi-node-single-writer",
"AttachmentMode": "file-system",
"CurrentReaders": 2,
"CurrentWriters": 1,
"Schedulable": true,
"PluginID": "plugin-id1",
"Provider": "ebs",
"ControllerRequired": true,
"ControllersHealthy": 3,
"ControllersExpected": 3,
"NodesHealthy": 15,
"NodesExpected": 18,
"ResourceExhausted": 0,
"CreateIndex": 42,
"ModifyIndex": 64
}
]
```
## Read Volume
This endpoint reads information about a specific volume.
| Method | Path | Produces |
| ------ | --------------------------- | ------------------ |
| `GET` | `/v1/volume/csi/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | --------------------------- |
| `YES` | `namespace:csi-read-volume` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
path.
### Sample Request
```shell-session
$ curl \
https://localhost:4646/v1/volume/csi/volume-id1
```
### Sample Response
```json
{
"ID": "volume-id1",
"Name": "volume id1",
"Namespace": "default",
"ExternalID": "vol-abcdef",
"Topologies": [{ "foo": "bar" }],
"AccessMode": "multi-node-single-writer",
"AttachmentMode": "file-system",
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"Allocations": [
{
"ID": "a8198d79-cfdb-6593-a999-1e9adabcba2e",
"EvalID": "5456bd7a-9fc0-c0dd-6131-cbee77f57577",
"Name": "example.cache[0]",
"NodeID": "fb2170a8-257d-3c64-b14d-bc06cc94e34c",
"PreviousAllocation": "516d2753-0513-cfc7-57ac-2d6fac18b9dc",
"NextAllocation": "cd13d9b9-4f97-7184-c88b-7b451981616b",
"RescheduleTracker": {
"Events": [
{
"PrevAllocID": "516d2753-0513-cfc7-57ac-2d6fac18b9dc",
"PrevNodeID": "9230cd3b-3bda-9a3f-82f9-b2ea8dedb20e",
"RescheduleTime": 1517434161192946200,
"Delay": "5000000000"
}
]
},
"JobID": "example",
"TaskGroup": "cache",
"DesiredStatus": "run",
"DesiredDescription": "",
"ClientStatus": "running",
"ClientDescription": "",
"TaskStates": {
"redis": {
"State": "running",
"FinishedAt": "0001-01-01T00:00:00Z",
"LastRestart": "0001-01-01T00:00:00Z",
"Restarts": 0,
"StartedAt": "2017-07-25T23:36:26.106431265Z",
"Failed": false,
"Events": [
{
"Type": "Received",
"Time": 1495747371795703800,
"FailsTask": false,
"RestartReason": "",
"SetupError": "",
"DriverError": "",
"ExitCode": 0,
"Signal": 0,
"Message": "",
"KillTimeout": 0,
"KillError": "",
"KillReason": "",
"StartDelay": 0,
"DownloadError": "",
"ValidationError": "",
"DiskLimit": 0,
"FailedSibling": "",
"VaultError": "",
"TaskSignalReason": "",
"TaskSignal": "",
"DriverMessage": ""
},
{
"Type": "Driver",
"Time": 1495747371798867200,
"FailsTask": false,
"RestartReason": "",
"SetupError": "",
"DriverError": "",
"ExitCode": 0,
"Signal": 0,
"Message": "",
"KillTimeout": 0,
"KillError": "",
"KillReason": "",
"StartDelay": 0,
"DownloadError": "",
"ValidationError": "",
"DiskLimit": 0,
"FailedSibling": "",
"VaultError": "",
"TaskSignalReason": "",
"TaskSignal": "",
"DriverMessage": "Downloading image redis:7"
},
{
"Type": "Started",
"Time": 1495747379525667800,
"FailsTask": false,
"RestartReason": "",
"SetupError": "",
"DriverError": "",
"ExitCode": 0,
"Signal": 0,
"Message": "",
"KillTimeout": 0,
"KillError": "",
"KillReason": "",
"StartDelay": 0,
"DownloadError": "",
"ValidationError": "",
"DiskLimit": 0,
"FailedSibling": "",
"VaultError": "",
"TaskSignalReason": "",
"TaskSignal": "",
"DriverMessage": ""
}
]
}
},
"CreateIndex": 54,
"ModifyIndex": 57,
"CreateTime": 1495747371794276400,
"ModifyTime": 1495747371794276400
}
],
"ReadAllocs": {
"a8198d79-cfdb-6593-a999-1e9adabcba2e": null
},
"WriteAllocs": {},
"Schedulable": true,
"PluginID": "plugin-id1",
"Provider": "ebs",
"Version": "1.0.1",
"ControllerRequired": true,
"ControllersHealthy": 3,
"ControllersExpected": 3,
"NodesHealthy": 15,
"NodesExpected": 18,
"ResourceExhausted": 0,
"CreateIndex": 42,
"ModifyIndex": 64
}
```
## Register Volume
This endpoint registers an external volume with Nomad. The volume must exist
in the external storage provider (see [Create Volume] below). It is an error
to register an already registered volume.
| Method | Path | Produces |
| ------ | --------------------------- | ------------------ |
| `PUT` | `/v1/volume/csi/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
path.
### Sample Payload
The payload must include a JSON document that describes the volume's
parameters.
```json
{
"Volumes": [
{
"ExternalID": "vol-abcdef",
"ID": "volume-id1",
"Name": "volume one",
"Namespace": "default",
"PluginID": "plugin-id1",
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"Secrets": {
"password": "xyzzy"
},
"Topologies": [{ "foo": "bar" }]
}
]
}
```
### Sample Request
```shell-session
$ curl \
--request PUT \
--data @payload.json \
https://localhost:4646/v1/volume/csi/volume-id1
```
## Create Volume
This endpoint creates a volume in an external storage provider and registers
it with Nomad. It is an error to create an already-registered volume. Only CSI
plugins that implement the [Controller][csi_plugins_internals] interface with
the `CREATE_DELETE_VOLUME` capability support this endpoint.
| Method | Path | Produces |
| ------ | ---------------------------------- | ------------------ |
| `PUT` | `/v1/volume/csi/:volume_id/create` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
path.
### Sample Payload
The payload must include a JSON document that describes the volume's
parameters.
```json
{
"Volumes": [
{
"ID": "volume-id1",
"Name": "volume one",
"Namespace": "default",
"PluginID": "plugin-id1",
"MountOptions": {
"FsType": "ext4",
"MountFlags": ["ro", "noatime"],
},
"RequestedCapacityMin": 10737418240,
"RequestedCapacityMax": 21474836480,
"RequestedCapabilities": [
{
"AccessMode": "single-node-writer",
"AttachmentMode": "file-system"
},
{
"AccessMode": "single-node-writer",
"AttachmentMode": "block-device"
}
],
"Secrets": {
"password": "xyzzy"
},
"SnapshotID": "snap-12345",
"Topologies": [{ "foo": "bar" }]
}
]
}
```
### Sample Request
```shell-session
$ curl \
--request PUT \
--data @payload.json \
https://localhost:4646/v1/volume/csi/volume-id1
```
## Deregister Volume
This endpoint deregisters an external volume with Nomad. It is an error to
deregister a volume that is in use.
| Method | Path | Produces |
| -------- | --------------------------- | ------------------ |
| `DELETE` | `/v1/volume/csi/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
path.
- `force` `(bool: false)` - Force deregistration of the volume and immediately
drop claims for terminal allocations. Returns an error if the volume has
running allocations. This does not detach the volume from client nodes.
This is specified as a query string parameter.
### Sample Request
```shell-session
$ curl \
--request DELETE \
https://localhost:4646/v1/volume/csi/volume-id1?force=false
```
## Delete Volume
This endpoint deletes an external volume from the storage provider, and
deregisters it from Nomad. It is an error to delete a volume that is in
use. Only CSI plugins that implement the [Controller][csi_plugins_internals]
interface with the `CREATE_DELETE_VOLUME` capability support this endpoint.
| Method | Path | Produces |
| -------- | ---------------------------------- | ------------------ |
| `DELETE` | `/v1/volume/csi/:volume_id/delete` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
This endpoint accepts a `X-Nomad-CSI-Secrets` header to set secrets
for deleting the volume as comma-separated key-value pairs (see the
example below). These secrets will be merged with any secrets already
stored when the CSI volume was created.
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
path.
### Sample Request
```shell-session
$ curl \
--request DELETE \
-H "X-Nomad-CSI-Secrets: secret-key-1=value-1,secret-key-2=value-2" \
https://localhost:4646/v1/volume/csi/volume-id1/delete
```
## Detach Volume
This endpoint detaches an external volume from a Nomad client node. It is an
error to detach a volume that is in use.
| Method | Path | Produces |
| -------- | ---------------------------------- | ------------------ |
| `DELETE` | `/v1/volume/csi/:volume_id/detach` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the
volume. This must be the full ID. This is specified as part of the
path.
- `node` `(string: <required>)` - The node to detach the volume from.
This is specified as a query string parameter.
### Sample Request
```shell-session
$ curl \
--request DELETE \
https://localhost:4646/v1/volume/csi/volume-id/detach?node=00000000-0000-0000-0000-000000000000
```
## List External Volumes
This endpoint lists storage volumes that are known to the external storage
provider but may not be registered with Nomad. Only CSI plugins that
implement the [Controller][csi_plugins_internals] interface with the
`LIST_VOLUMES` capability support this endpoint.
| Method | Path | Produces |
|--------|------------------------|--------------------|
| `GET` | `/v1/volumes/external` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `YES` | `namespace:csi-write-volume` |
### Parameters
- `plugin_id` `(string: "")` - Specifies a string to filter volumes
based on a plugin ID prefix. Because the value is decoded to bytes,
the prefix must have an even number of hexadecimal characters
(0-9a-f). This is specified as a query string parameter.
- `next_token` `(string: "")` - This endpoint supports paging. The
`next_token` parameter accepts a string returned in a previous response's
`NextToken` field to request the next page of results.
- `per_page` `(int: <required>)` - Specifies a maximum number of snapshots to
return for this request. The response will include a `NextToken` field that
can be passed to the next request to fetch additional pages.
### Sample Request
```shell-session
$ curl \
https://localhost:4646/v1/volumes/external?&plugin_id=plugin-id1&per_page=2
```
### Sample Response
```json
{
"NextToken": "eyJ2IjoiMiIsImMiOiJLZ",
"Volumes": [
{
"ExternalID": "vol-37ac485e",
"CapacityBytes": 1000000,
"SnapshotID": "snap-12345",
"PublishedExternalNodeIDs": ["i-12345", "i-abcde"],
"IsAbnormal": false,
"Status": ""
},
{
"ExternalID": "vol-10ac4879",
"CapacityBytes": 1000000,
"SnapshotID": "snap-abcdef",
"PublishedExternalNodeIDs": ["i-12345", "i-abcde"],
"IsAbnormal": true,
"Status": "example error message from provider"
}
]
}
```
## Create Snapshot
This endpoint creates a snapshot of a volume on the external storage
provider. Only CSI plugins that implement the
[Controller][csi_plugins_internals] interface with the
`CREATE_DELETE_SNAPSHOT` capability support this endpoint.
| Method | Path | Produces |
| -------- | --------------------------------- | ------------------ |
| `PUT` | `/v1/volumes/snapshot/:volume_id` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
### Sample Payload
The payload must include a JSON document that describes the snapshot's
parameters.
```json
{
"Snapshots": [
{
"SourceVolumeID": "volume-id1",
"PluginID": "plugin-id1",
"Name": "mysnap",
"Secrets": {
"password": "xyzzy"
},
"Parameters": {
"example": "infrequent_access"
}
}
]
}
```
### Parameters
- `:volume_id` `(string: <required>)` - Specifies the ID of the volume to
snapshot. This must be the full ID. This is specified as part of the path.
### Sample Request
```shell-session
$ curl \
--request PUT \
--data @payload.json \
https://localhost:4646/v1/volume/csi/volume-id1/snapshot
```
### Sample Response
```json
{
"Snapshots": [
{
"ID": "snap-031f5f7e3406d594a",
"SizeBytes": 10737418240,
"CreateTime": 1617909982,
"IsReady": false,
"SourceVolumeID": "volume-id1"
}
]
}
```
## Delete Snapshot
This endpoint deletes a volume snapshot from the external storage
provider. Only CSI plugins that implement the
[Controller][csi_plugins_internals] interface with the
`CREATE_DELETE_SNAPSHOT` capability support this endpoint.
| Method | Path | Produces |
| -------- | ---------------------- | ------------------ |
| `DELETE` | `/v1/volumes/snapshot` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------------- |
| `NO` | `namespace:csi-write-volume` |
This endpoint accepts a `X-Nomad-CSI-Secrets` header to set secrets
for deleting the snapshot as comma-separated key-value pairs (see the
example below). These secrets will be merged with any secrets already
stored when the CSI snapshot was created.
### Parameters
- `plugin_id` `(string: <required>)` - Specifies the prefix of a CSI plugin ID
to perform the delete. Because the value is decoded to bytes, the prefix
must have an even number of hexadecimal characters (0-9a-f). This is
specified as a query string parameter.
- `snapshot_id` `(string: <required>)` - Specifies the snapshot ID to
delete. This is specified as a query parameter.
### Sample Request
```shell-session
$ curl \
--request DELETE \
-H "X-Nomad-CSI-Secrets: secret-key-1=value-1,secret-key-2=value-2" \
https://localhost:4646/v1/volumes/snapshot
```
## List Snapshots
This endpoint lists volume snapshots on the external storage provider. Only
CSI plugins that implement the [Controller][csi_plugins_internals] interface
with the `LIST_SNAPSHOTS` capability support this endpoint.
| Method | Path | Produces |
| ------ | ---------------------- | ------------------ |
| `GET` | `/v1/volumes/snapshot` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/api-docs#blocking-queries) and
[required ACLs](/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | --------------------------- |
| `YES` | `namespace:csi-list-volume` |
This endpoint accepts a `X-Nomad-CSI-Secrets` header to set secrets
for deleting the snapshot as comma-separated key-value pairs (see the
example below). These secrets will be merged with any secrets already
stored when the CSI snapshot was created.
### Parameters
- `plugin_id` `(string: <required>)` - Specifies the prefix of a CSI plugin ID
to perform the list. Because the value is decoded to bytes, the prefix must
have an even number of hexadecimal characters (0-9a-f). This is specified as
a query string parameter.
- `next_token` `(string: "")` - This endpoint supports paging. The
`next_token` parameter accepts a string returned in a previous response's
`NextToken` field to request the next page of results.
- `per_page` `(int: <required>)` - Specifies a maximum number of snapshots to
return for this request. The response will include a `NextToken` field that
can be passed to the next request to fetch additional pages.
### Sample Request
```shell-session
$ curl \
-H "X-Nomad-CSI-Secrets: secret-key-1=value-1,secret-key-2=value-2" \
https://localhost:4646/v1/volumes/snapshot?plugin_id=plugin-id1&per_page=2
```
### Sample Response
```json
{
"NextToken": "eyJ2IjoiMiIsImMiOiJLZ",
"Snapshots": [
{
"CreateTime": 1227088980,
"ExternalSourceVolumeID": "vol-37ac485e",
"ID": "snap-9df717f4",
"IsReady": true,
"Parameters": null,
"PluginID": "aws-ebs0",
"SizeBytes": 2147483648
},
{
"CreateTime": 1227091232,
"ExternalSourceVolumeID": "vol-10ac4879",
"ID": "snap-5cf81835",
"IsReady": true,
"Parameters": null,
"PluginID": "aws-ebs0",
"SizeBytes": 11811160064
}
]
}
```
[csi]: https://github.com/container-storage-interface/spec
[csi_plugin]: /docs/job-specification/csi_plugin
[csi_plugins_internals]: /docs/concepts/plugins/csi#csi-plugins
[Create Volume]: #create-volume
Reference in a new issue View git blame Copy permalink