open-nomad/website/pages/docs/commands/node/drain.mdx

138 lines
5 KiB
Plaintext
Raw Normal View History

---
2020-02-06 23:45:31 +00:00
layout: docs
page_title: 'Commands: node drain'
sidebar_title: drain
description: |
2018-03-22 17:56:06 +00:00
The node drain command is used to configure a node's drain strategy.
---
2018-03-22 17:56:06 +00:00
# Command: node drain
2018-03-22 17:56:06 +00:00
The `node drain` command is used to toggle drain mode on a given node. Drain
2015-09-15 20:52:02 +00:00
mode prevents any new tasks from being allocated to the node, and begins
migrating all existing allocations away. Allocations will be migrated according
to their [`migrate`][migrate] stanza until the drain's deadline is reached.
By default the `node drain` command blocks until a node is done draining and
2020-02-06 23:45:31 +00:00
all allocations have terminated. Canceling the `node drain` command _will not_
cancel the drain. Drains may be canceled by using the `-disable` parameter
below.
When draining more than one node at a time, it is recommended you first disable
2018-03-30 23:24:19 +00:00
[scheduling eligibility][eligibility] on all nodes that will be drained. For
2020-02-06 23:45:31 +00:00
example if you are decommissioning an entire class of nodes, first run `node eligibility -disable` on all of their node IDs, and then run `node drain -enable`. This will ensure allocations drained from the first node are not
placed on another node about to be drained.
The [node status] command compliments this nicely by providing the current drain
status of a given node.
2015-09-15 03:24:04 +00:00
See the [Workload Migration guide] for detailed examples of node draining.
## Usage
```plaintext
2018-03-22 17:56:06 +00:00
nomad node drain [options] <node>
```
2016-04-11 22:20:49 +00:00
A `-self` flag can be used to drain the local node. If this is not supplied, a
node ID or prefix must be provided. If there is an exact match, the drain mode
will be adjusted for that node. Otherwise, a list of matching nodes and
information will be displayed.
It is also required to pass one of `-enable` or `-disable`, depending on which
operation is desired.
## General Options
2020-02-06 23:45:31 +00:00
@include 'general_options.mdx'
2018-03-22 17:56:06 +00:00
## Drain Options
- `-enable`: Enable node drain mode.
- `-disable`: Disable node drain mode.
- `-deadline`: Set the deadline by which all allocations must be moved off the
node. Remaining allocations after the deadline are force removed from the
node. Defaults to 1 hour.
- `-detach`: Return immediately instead of entering monitor mode.
- `-monitor`: Enter monitor mode directly without modifying the drain status.
- `-force`: Force remove allocations off the node immediately.
- `-no-deadline`: No deadline allows the allocations to drain off the node
without being force stopped after a certain deadline.
- `-ignore-system`: Ignore system allows the drain to complete without stopping
system job allocations. By default system jobs are stopped last.
- `-keep-ineligible`: Keep ineligible will maintain the node's scheduling
ineligibility even if the drain is being disabled. This is useful when an
existing drain is being cancelled but additional scheduling on the node is not
desired.
- `-self`: Drain the local node.
- `-yes`: Automatic yes to prompts.
2015-09-15 03:24:04 +00:00
## Examples
2016-04-11 22:20:49 +00:00
Enable drain mode on node with ID prefix "4d2ba53b":
2015-09-15 03:24:04 +00:00
2020-05-18 20:53:06 +00:00
```shell-session
$ nomad node drain -enable f4e8a9e5
Are you sure you want to enable drain mode for node "f4e8a9e5-30d8-3536-1e6f-cda5c869c35e"? [y/N] y
2018-03-30T23:13:16Z: Ctrl-C to stop monitoring: will not cancel the node drain
2018-03-30T23:13:16Z: Node "f4e8a9e5-30d8-3536-1e6f-cda5c869c35e" drain strategy set
2018-03-30T23:13:17Z: Alloc "1877230b-64d3-a7dd-9c31-dc5ad3c93e9a" marked for migration
2018-03-30T23:13:17Z: Alloc "1877230b-64d3-a7dd-9c31-dc5ad3c93e9a" draining
2018-03-30T23:13:17Z: Alloc "1877230b-64d3-a7dd-9c31-dc5ad3c93e9a" status running -> complete
2018-03-30T23:13:29Z: Alloc "3fce5308-818c-369e-0bb7-f61f0a1be9ed" marked for migration
2018-03-30T23:13:29Z: Alloc "3fce5308-818c-369e-0bb7-f61f0a1be9ed" draining
2018-03-30T23:13:30Z: Alloc "3fce5308-818c-369e-0bb7-f61f0a1be9ed" status running -> complete
2018-03-30T23:13:41Z: Alloc "9a98c5aa-a719-2f34-ecfc-0e6268b5d537" marked for migration
2018-03-30T23:13:41Z: Alloc "9a98c5aa-a719-2f34-ecfc-0e6268b5d537" draining
2018-03-30T23:13:41Z: Node "f4e8a9e5-30d8-3536-1e6f-cda5c869c35e" has marked all allocations for migration
2018-03-30T23:13:42Z: Alloc "9a98c5aa-a719-2f34-ecfc-0e6268b5d537" status running -> complete
2018-03-30T23:13:42Z: All allocations on node "f4e8a9e5-30d8-3536-1e6f-cda5c869c35e" have stopped.
2016-04-11 22:20:49 +00:00
```
Enable drain mode on the local node:
2020-05-18 20:53:06 +00:00
```shell-session
$ nomad node drain -enable -self
...
```
Enable drain mode but do not stop system jobs:
2020-05-18 20:53:06 +00:00
```shell-session
$ nomad node drain -enable -ignore-system 4d2ba53b
...
2015-09-15 03:24:04 +00:00
```
Disable drain mode but keep the node ineligible for scheduling. Useful for
inspecting the current state of a misbehaving node without Nomad trying to
start or migrate allocations:
2020-05-18 20:53:06 +00:00
```shell-session
$ nomad node drain -disable -keep-ineligible 4d2ba53b
...
```
Enable drain mode and detach from monitoring, then reattach later:
2020-05-18 20:53:06 +00:00
```shell-session
$ nomad node drain -enable -detach -self
...
$ nomad node drain -self -monitor
...
```
2020-02-06 23:45:31 +00:00
[eligibility]: /docs/commands/node/eligibility
[migrate]: /docs/job-specification/migrate
[node status]: /docs/commands/node/status
[workload migration guide]: https://learn.hashicorp.com/nomad/operating-nomad/node-draining