luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/website/content/api-docs/client.mdx
Tim Gross 17bd930ca9
logs: fix missing allocation logs after update to Nomad 1.5.4 (#17087) 
When the server restarts for the upgrade, it loads the `structs.Job` from the
Raft snapshot/logs. The jobspec has long since been parsed, so none of the
guards around the default value are in play. The empty field value for `Enabled`
is the zero value, which is false.

This doesn't impact any running allocation because we don't replace running
allocations when either the client or server restart. But as soon as any
allocation gets rescheduled (ex. you drain all your clients during upgrades),
it'll be using the `structs.Job` that the server has, which has `Enabled =
false`, and logs will not be collected.

This changeset fixes the bug by adding a new field `Disabled` which defaults to
false (so that the zero value works), and deprecates the old field.

Fixes #17076
2023-05-04 16:01:18 -04:00

837 lines
24 KiB
Plaintext
Raw Blame History

---
layout: api
page_title: Client - HTTP API
description: |-
The /client endpoints are used to access client information and inspect
allocations running on a particular client.
---
# Client HTTP API
The `/v1/client` endpoints are used to interact with the Nomad clients.
Since Nomad 0.8.0, both a client and server can handle client endpoints. This is
particularly useful for when a direct connection to a client is not possible due
to the network configuration. For high volume access to the client endpoints,
particularly endpoints streaming file contents, direct access to the node should
be preferred as it avoids adding additional load to the servers.
When accessing the endpoints via the server, if the desired node is ambiguous
based on the URL, an additional `?node_id` query parameter must be provided to
disambiguate.
## Read Node Metadata
This endpoint queries Node metadata on a specific Client agent and responds
with the following fields:
- `Meta` `(object)` - The Node metadata that will be registered with the Nomad
servers and used by the scheduler (after up to 10 seconds of delay for
batching). This is the merged version of the `Static` and `Dynamic` fields.
- `Static` `(object)` - The Node metadata set in the Client agent's
configuration file. Only loaded when an agent starts.
- `Dynamic` `(object)` - The Node metadata set via the API (see below). Unlike
`Meta` and `Static`, this object may contain `null` values to differentiate
"unset" keys from keys with an empty string value (`""`).
Note that [`/v1/node/:node_id`][api-node-read] only contains the `Meta` object.
It may take up to 10 seconds for dynamic Node metadata to be sent to Servers
and visible through the Node API. Use the Node API to see the version of Node
metadata the scheduler uses.
| Method | Path | Produces |
| ------ | --------------------- | ------------------ |
| `GET` | `/v1/client/metadata` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------- |
| `NO` | `node:read` |
### Parameters
- `:node_id` `(string: <optional>)` - Specifies the node to query.
This is required when the endpoint is being accessed via a server. Defaults
to the node recieving the request otherwise. This is specified as part of
the URL. Note, this must be the _full_ node ID, not the short 8-character
one. This must be specified as part of the path (`?node_id=...`).
### Sample Request
```shell-session
$ nomad operator api /v1/client/metadata
```
### Sample Response
Formatted by appending `?pretty` above.
```json
{
"Meta": {
"connect.proxy_concurrency": "1",
"connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.log_level": "debug",
"foo": "bar"
},
"Dynamic": {
"key_to_unset": null,
"foo": "bar",
"connect.log_level": "debug"
},
"Static": {
"connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.log_level": "info",
"connect.proxy_concurrency": "1"
}
}
```
### Sample Request
## Update Node Metadata
This endpoint updates dynamic Node metadata on a specific Client agent. Since
dynamic Node metadata is only periodically synchronized to Nomad Servers, the
`Meta` returned in this API may not be reflected in the
[`/v1/node/:node_id`][api-node-read] API for up to 10 seconds. Scheduling uses
the Node API version of `Meta`.
For convenience this endpoint returns the same response as a GET.
| Method | Path | Produces |
| ------ | --------------------- | ------------------ |
| `POST` | `/v1/client/metadata` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------- |
| `NO` | `node:write` |
### Parameters
- `NodeID` or `:node_id` `(string: <optional>)` - Specifies the node to query.
This is required when the endpoint is being accessed via a server. Defaults
to the node recieving the request otherwise. This is specified as part of
the URL. Note, this must be the _full_ node ID, not the short 8-character
one. This may be specified as part of the path (`?node_id=...`) or request
(`NodeID: "..."`).
- `Meta` `(object: <required>)` - Specifies the Node metadata keys to update.
Only specified keys are updated.
- `<key>` `(string: <optional>)` - Specifies a metadata key to update to a
particular value. Since `""` is a valid value and distinct from unset,
the `null` value is used to mark a key as unset. Keys must be valid
dotted HCL identifiers. For example `connect.log_level` is a valid key
while `some/path` is not.
### Sample Payload
```json
{
"Meta": {
"connect.log_level": "debug",
"key_to_unset": null,
"foo": "bar"
}
}
```
### Sample Request
Assuming the above payload is in a file called `meta.json`.
```shell-session
$ nomad operator api /v1/client/metadata < meta.json
```
### Sample Response
Formatted by appending `?pretty` above.
```json
{
"Meta": {
"connect.proxy_concurrency": "1",
"connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.log_level": "debug",
"foo": "bar"
},
"Dynamic": {
"key_to_unset": null,
"foo": "bar",
"connect.log_level": "debug"
},
"Static": {
"connect.sidecar_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.gateway_image": "docker.io/envoyproxy/envoy:v${NOMAD_envoy_version}",
"connect.log_level": "info",
"connect.proxy_concurrency": "1"
}
}
```
## Read Stats
This endpoint queries the actual resources consumed on a node. The API endpoint
is hosted by the Nomad client and requests have to be made to the nomad client
whose resource usage metrics are of interest.
| Method | Path | Produces |
| ------ | ------------------ | ------------------ |
| `GET` | `/v1/client/stats` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------ |
| `NO` | `node:read` |
### Parameters
- `node_id` `(string: <optional>)` - Specifies the node to query. This is
required when the endpoint is being accessed via a server. This is specified as
part of the URL. Note, this must be the _full_ node ID, not the short
8-character one. This is specified as part of the path.
### Sample Request
```shell-session
$ nomad operator api /v1/client/stats
```
### Sample Response
```json
{
"AllocDirStats": {
"Available": 142943150080,
"Device": "",
"InodesUsedPercent": 0.05312946180421879,
"Mountpoint": "",
"Size": 249783500800,
"Used": 106578206720,
"UsedPercent": 42.668233241448746
},
"CPU": [
{
"CPU": "cpu0",
"Idle": 80,
"System": 11,
"Total": 20,
"User": 9
},
{
"CPU": "cpu1",
"Idle": 99,
"System": 0,
"Total": 1,
"User": 1
},
{
"CPU": "cpu2",
"Idle": 89,
"System": 7.000000000000001,
"Total": 11,
"User": 4
},
{
"CPU": "cpu3",
"Idle": 100,
"System": 0,
"Total": 0,
"User": 0
},
{
"CPU": "cpu4",
"Idle": 92.92929292929293,
"System": 4.040404040404041,
"Total": 7.07070707070707,
"User": 3.0303030303030303
},
{
"CPU": "cpu5",
"Idle": 99,
"System": 1,
"Total": 1,
"User": 0
},
{
"CPU": "cpu6",
"Idle": 92.07920792079209,
"System": 4.9504950495049505,
"Total": 7.920792079207921,
"User": 2.9702970297029703
},
{
"CPU": "cpu7",
"Idle": 99,
"System": 0,
"Total": 1,
"User": 1
}
],
"CPUTicksConsumed": 1126.8044804480448,
"DeviceStats": [
{
"InstanceStats": {
"6a61929e-d572-092d-5921-156a913f8e56": {
"Stats": {
"Attributes": {
"Used Memory": {
"Desc": "Memory in use by the device",
"IntNumeratorVal": 128,
"Unit": "MiB"
}
},
"Nested": {}
},
"Summary": {
"Desc": "Memory in use by the device",
"IntNumeratorVal": 128,
"Unit": "MiB"
},
"Timestamp": "2020-12-18T17:15:08.949806Z"
}
},
"Name": "modelA",
"Type": "skeleton",
"Vendor": "hashicorp"
},
{
"InstanceStats": {
"73af5d3e-00f9-0786-9bc1-8f5ffa953f15": {
"Stats": {
"Attributes": {
"Used Memory": {
"Desc": "Memory in use by the device",
"IntNumeratorVal": 128,
"Unit": "MiB"
}
},
"Nested": {}
},
"Summary": {
"Desc": "Memory in use by the device",
"IntNumeratorVal": 128,
"Unit": "MiB"
},
"Timestamp": "2020-12-18T17:15:08.949806Z"
}
},
"Name": "modelB",
"Type": "skeleton",
"Vendor": "hashicorp"
}
],
"DiskStats": [
{
"Available": 142943150080,
"Device": "/dev/disk1",
"InodesUsedPercent": 0.05312946180421879,
"Mountpoint": "/",
"Size": 249783500800,
"Used": 106578206720,
"UsedPercent": 42.668233241448746
}
],
"Memory": {
"Available": 6232244224,
"Free": 470618112,
"Total": 17179869184,
"Used": 10947624960
},
"Timestamp": 1495743032992498200,
"Uptime": 193520
}
```
## Read Allocation Statistics
The client `allocation` endpoint is used to query the actual resources consumed
by an allocation.
| Method | Path | Produces |
| ------ | --------------------------------------- | ------------------ |
| `GET` | `/v1/client/allocation/:alloc_id/stats` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | -------------------- |
| `NO` | `namespace:read-job` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/stats
```
### Sample Response
```json
{
"ResourceUsage": {
"CpuStats": {
"Measured": ["Throttled Periods", "Throttled Time", "Percent"],
"Percent": 0.14159538847117795,
"SystemMode": 0,
"ThrottledPeriods": 0,
"ThrottledTime": 0,
"TotalTicks": 3.256693934837093,
"UserMode": 0
},
"MemoryStats": {
"Cache": 1744896,
"KernelMaxUsage": 0,
"KernelUsage": 0,
"MaxUsage": 4710400,
"Measured": ["RSS", "Cache", "Swap", "Max Usage"],
"RSS": 1486848,
"Swap": 0
}
},
"Tasks": {
"redis": {
"Pids": null,
"ResourceUsage": {
"CpuStats": {
"Measured": ["Throttled Periods", "Throttled Time", "Percent"],
"Percent": 0.14159538847117795,
"SystemMode": 0,
"ThrottledPeriods": 0,
"ThrottledTime": 0,
"TotalTicks": 3.256693934837093,
"UserMode": 0
},
"MemoryStats": {
"Cache": 1744896,
"KernelMaxUsage": 0,
"KernelUsage": 0,
"MaxUsage": 4710400,
"Measured": ["RSS", "Cache", "Swap", "Max Usage"],
"RSS": 1486848,
"Swap": 0
}
},
"Timestamp": 1495743243970720000
}
},
"Timestamp": 1495743243970720000
}
```
## Read File
This endpoint reads the contents of a file in an allocation directory.
| Method | Path | Produces |
| ------ | ----------------------------- | ------------ |
| `GET` | `/v1/client/fs/cat/:alloc_id` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------------- |
| `NO` | `namespace:read-fs` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
- `path` `(string: "/")` - Specifies the path of the file to read, relative to
the root of the allocation directory.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99
```
```shell-session
$ nomad operator api \
/v1/client/fs/cat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=alloc/file.json
```
### Sample Response
```text
(whatever was in the file...)
```
## Read File at Offset
This endpoint reads the contents of a file in an allocation directory at a
particular offset and limit.
| Method | Path | Produces |
| ------ | -------------------------------- | ------------ |
| `GET` | `/v1/client/fs/readat/:alloc_id` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------------- |
| `NO` | `namespace:read-fs` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
- `path` `(string: "/")` - Specifies the path of the file to read, relative to
the root of the allocation directory.
- `offset` `(int: <required>)` - Specifies the byte offset from where content
will be read.
- `limit` `(int: <required>)` - Specifies the number of bytes to read from the
offset.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/fs/readat/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/foo&offset=1323&limit=19303
```
### Sample Response
```text
(whatever was in the file, starting from offset, up to limit bytes...)
```
## Stream File
This endpoint streams the contents of a file in an allocation directory.
| Method | Path | Produces |
| ------ | -------------------------------- | ------------ |
| `GET` | `/v1/client/fs/stream/:alloc_id` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------------- |
| `NO` | `namespace:read-fs` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
- `path` `(string: "/")` - Specifies the path of the file to read, relative to
the root of the allocation directory.
- `follow` `(bool: true)`- Specifies whether to tail the file.
- `offset` `(int: <required>)` - Specifies the byte offset from where content
will be read.
- `origin` `(string: "start|end")` - Applies the relative offset to either the
start or end of the file.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/fs/stream/5fc98185-17ff-26bc-a802-0c74fa471c99?path=/alloc/logs/redis.log
```
### Sample Response
```json
({
"File": "alloc/logs/redis.log",
"Offset": 3604480,
"Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..."
},
{
"File": "alloc/logs/redis.log",
"FileEvent": "file deleted"
})
```
#### Field Reference
The return value is a stream of frames. These frames contain the following
fields:
- `Data` - A base64 encoding of the bytes being streamed.
- `FileEvent` - An event that could cause a change in the streams position. The
possible values are "file deleted" and "file truncated".
- `Offset` - Offset is the offset into the stream.
- `File` - The name of the file being streamed.
## Stream Logs
This endpoint streams a task's stderr/stdout logs. Note that if logging is set
to [disabled=true][] for the task, this endpoint will return a 404 error.
| Method | Path | Produces |
| ------ | ------------------------------ | ------------ |
| `GET` | `/v1/client/fs/logs/:alloc_id` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | -------------------------------------------- |
| `NO` | `namespace:read-logs` or `namespace:read-fs` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
- `task` `(string: <required>)` - Specifies the name of the task inside the
allocation to stream logs from.
- `follow` `(bool: false)`- Specifies whether to tail the logs.
- `type` `(string: "stderr|stdout")` - Specifies the stream to stream.
- `offset` `(int: 0)` - Specifies the offset to start streaming from.
- `origin` `(string: "start|end")` - Specifies either "start" or "end" and
applies the offset relative to either the start or end of the logs
respectively. Defaults to "start".
- `plain` `(bool: false)` - Return just the plain text without framing. This can
be useful when viewing logs in a browser.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/fs/logs/5fc98185-17ff-26bc-a802-0c74fa471c99
```
### Sample Response
```json
({
"File": "alloc/logs/redis.stdout.0",
"Offset": 3604480,
"Data": "NTMxOTMyCjUzMTkzMwo1MzE5MzQKNTMx..."
},
{
"File": "alloc/logs/redis.stdout.0",
"FileEvent": "file deleted"
})
```
#### Field Reference
The return value is a stream of frames. These frames contain the following
fields:
- `Data` - A base64 encoding of the bytes being streamed.
- `FileEvent` - An event that could cause a change in the streams position. The
possible values are "file deleted" and "file truncated".
- `Offset` - Offset is the offset into the stream.
- `File` - The name of the file being streamed.
## List Files
This endpoint lists files in an allocation directory.
| Method | Path | Produces |
| ------ | ---------------------------- | ------------ |
| `GET` | `/v1/client/fs/ls/:alloc_id` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------------- |
| `NO` | `namespace:read-fs` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
- `path` `(string: "/")` - Specifies the path of the file to read, relative to
the root of the allocation directory.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/fs/ls/5fc98185-17ff-26bc-a802-0c74fa471c99
```
### Sample Response
```json
[
{
"Name": "alloc",
"IsDir": true,
"Size": 4096,
"FileMode": "drwxrwxr-x",
"ModTime": "2016-03-15T15:40:00.414236712-07:00"
},
{
"Name": "redis",
"IsDir": true,
"Size": 4096,
"FileMode": "drwxrwxr-x",
"ModTime": "2016-03-15T15:40:56.810238153-07:00"
}
]
```
## Stat File
This endpoint stats a file in an allocation.
| Method | Path | Produces |
| ------ | ------------------------------ | ------------ |
| `GET` | `/v1/client/fs/stat/:alloc_id` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------------- |
| `NO` | `namespace:read-fs` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
- `path` `(string: "/")` - Specifies the path of the file to read, relative to
the root of the allocation directory.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/fs/stat/5fc98185-17ff-26bc-a802-0c74fa471c99
```
### Sample Response
```json
{
"Name": "redis-syslog-collector.out",
"IsDir": false,
"Size": 96,
"FileMode": "-rw-rw-r--",
"ModTime": "2016-03-15T15:40:56.822238153-07:00"
}
```
## GC Allocation
This endpoint forces a garbage collection of a particular, stopped allocation
on a node. Note that the allocation will still exist on the server and appear
in server responses.
| Method | Path | Produces |
| ------ | ------------------------------------ | ------------------ |
| `GET` | `/v1/client/allocation/:alloc_id/gc` | `application/json` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ---------------------- |
| `NO` | `namespace:submit-job` |
### Parameters
- `:alloc_id` `(string: <required>)` - Specifies the allocation ID to query.
This is specified as part of the URL. Note, this must be the _full_ allocation
ID, not the short 8-character one. This is specified as part of the path.
### Sample Request
```shell-session
$ nomad operator api \
/v1/client/allocation/5fc98185-17ff-26bc-a802-0c74fa471c99/gc
```
## GC All Allocation
This endpoint forces a garbage collection of all stopped allocations on a node.
| Method | Path | Produces |
| ------ | --------------- | ------------ |
| `GET` | `/v1/client/gc` | `text/plain` |
The table below shows this endpoint's support for
[blocking queries](/nomad/api-docs#blocking-queries) and
[required ACLs](/nomad/api-docs#acls).
| Blocking Queries | ACL Required |
| ---------------- | ------------ |
| `NO` | `node:write` |
### Parameters
- `node_id` `(string: <optional>)` - Specifies the node to target. This is
required when the endpoint is being accessed via a server. This is specified as
part of the URL. Note, this must be the _full_ node ID, not the short
8-character one. This is specified as part of the path.
### Sample Request
```shell-session
$ nomad operator api /v1/client/gc
```
[api-node-read]: /nomad/api-docs/nodes
[disabled=true]: /nomad/docs/job-specification/logs#disabled
Reference in a new issue View git blame Copy permalink