luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/website/source/guides/operating-a-job/failure-handling-strategies/restart.html.md
Michael Schurter 5f1033263b docs: move operating-a-job to guides 
Add redirects from /docs/ -> /guides/ and update old redirects to point
to the new location.
2018-04-18 16:21:16 -07:00

3.7 KiB
Raw Blame History

layout page_title sidebar_current description
guides Restart Stanza - Operating a Job guides-operating-a-job-failure-handling-strategies-local-restarts Nomad can restart a task on the node it is running on to recover from failures. Task restarts can be configured to be limited by number of attempts within a specific interval.

Restart Stanza

To enable restarting a failed task on the node it is running on, the task group can be annotated with configurable options using the restart stanza. Nomad will restart the failed task up to attempts times within a provided interval. Operators can also choose whether to keep attempting restarts on the same node, or to fail the task so that it can be rescheduled on another node, via the mode parameter.

We recommend setting mode to fail in the restart stanza to allow rescheduling the task on another node.

Example

The following CLI example shows job status and allocation status for a failed task that is being restarted by Nomad. Allocations are in the pending state while restarts are attempted. The Recent Events section in the CLI shows ongoing restart attempts.

$nomad job status demo
ID            = demo
Name          = demo
Submit Date   = 2018-04-12T14:37:18-05:00
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
demo        0       3         0        0       0         0

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
ce5bf1d1  8a184f31  demo        0        run      pending  27s ago  5s ago
d5dee7c8  8a184f31  demo        0        run      pending  27s ago  5s ago
ed815997  8a184f31  demo        0        run      pending  27s ago  5s ago

In the following example, the allocation ce5bf1d1 is restarted by Nomad approximately every ten seconds, with a small random jitter. It eventually reaches its limit of three attempts and transitions into a failed state, after which it becomes eligible for rescheduling.

$nomad alloc-status ce5bf1d1
ID                     = ce5bf1d1
Eval ID                = 64e45d11
Name                   = demo.demo[1]
Node ID                = a0ccdd8b
Job ID                 = demo
Job Version            = 0
Client Status          = failed
Client Description     = <none>
Desired Status         = run
Desired Description    = <none>
Created                = 56s ago
Modified               = 22s ago

Task "demo" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
100 MHz  300 MiB  300 MiB  0

Task Events:
Started At     = 2018-04-12T22:29:08Z
Finished At    = 2018-04-12T22:29:08Z
Total Restarts = 3
Last Restart   = 2018-04-12T17:28:57-05:00

Recent Events:
Time                       Type            Description
2018-04-12T17:29:08-05:00  Not Restarting  Exceeded allowed attempts 3 in interval 5m0s and mode is "fail"
2018-04-12T17:29:08-05:00  Terminated      Exit Code: 127
2018-04-12T17:29:08-05:00  Started         Task started by client
2018-04-12T17:28:57-05:00  Restarting      Task restarting in 10.364602876s
2018-04-12T17:28:57-05:00  Terminated      Exit Code: 127
2018-04-12T17:28:57-05:00  Started         Task started by client
2018-04-12T17:28:47-05:00  Restarting      Task restarting in 10.666963769s
2018-04-12T17:28:47-05:00  Terminated      Exit Code: 127
2018-04-12T17:28:47-05:00  Started         Task started by client
2018-04-12T17:28:35-05:00  Restarting      Task restarting in 11.777324721s