142 lines
3.8 KiB
Plaintext
142 lines
3.8 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: check_restart Stanza - Job Specification
|
|
sidebar_title: check_restart
|
|
description: |-
|
|
The "check_restart" stanza instructs Nomad when to restart tasks with
|
|
unhealthy service checks.
|
|
---
|
|
|
|
# `check_restart` Stanza
|
|
|
|
<Placement
|
|
groups={[
|
|
['job', 'group', 'task', 'service', 'check_restart'],
|
|
['job', 'group', 'task', 'service', 'check', 'check_restart']
|
|
]}
|
|
/>
|
|
|
|
As of Nomad 0.7 the `check_restart` stanza instructs Nomad when to restart
|
|
tasks with unhealthy service checks. When a health check in Consul has been
|
|
unhealthy for the `limit` specified in a `check_restart` stanza, it is
|
|
restarted according to the task group's [`restart` policy][restart_stanza]. The
|
|
`check_restart` settings apply to [`check`s][check_stanza], but may also be
|
|
placed on [`service`s][service_stanza] to apply to all checks on a service.
|
|
If `check_restart` is set on both the check and service, the stanzas are
|
|
merged with the check values taking precedence.
|
|
|
|
```hcl
|
|
job "mysql" {
|
|
group "mysqld" {
|
|
|
|
restart {
|
|
attempts = 3
|
|
delay = "10s"
|
|
interval = "10m"
|
|
mode = "fail"
|
|
}
|
|
|
|
task "server" {
|
|
service {
|
|
tags = ["leader", "mysql"]
|
|
|
|
port = "db"
|
|
|
|
check {
|
|
type = "tcp"
|
|
port = "db"
|
|
interval = "10s"
|
|
timeout = "2s"
|
|
}
|
|
|
|
check {
|
|
type = "script"
|
|
name = "check_table"
|
|
command = "/usr/local/bin/check_mysql_table_status"
|
|
args = ["--verbose"]
|
|
interval = "60s"
|
|
timeout = "5s"
|
|
|
|
check_restart {
|
|
limit = 3
|
|
grace = "90s"
|
|
ignore_warnings = false
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
- `limit` `(int: 0)` - Restart task when a health check has failed `limit`
|
|
times. For example 1 causes a restart on the first failure. The default,
|
|
`0`, disables health check based restarts. Failures must be consecutive. A
|
|
single passing check will reset the count, so flapping services may not be
|
|
restarted.
|
|
|
|
- `grace` `(string: "1s")` - Duration to wait after a task starts or restarts
|
|
before checking its health.
|
|
|
|
- `ignore_warnings` `(bool: false)` - By default checks with both `critical`
|
|
and `warning` statuses are considered unhealthy. Setting `ignore_warnings = true` treats a `warning` status like `passing` and will not trigger a restart.
|
|
|
|
## Example Behavior
|
|
|
|
Using the example `mysql` above would have the following behavior:
|
|
|
|
```hcl
|
|
check_restart {
|
|
# ...
|
|
grace = "90s"
|
|
# ...
|
|
}
|
|
```
|
|
|
|
When the `server` task first starts and is registered in Consul, its health
|
|
will not be checked for 90 seconds. This gives the server time to startup.
|
|
|
|
```hcl
|
|
check_restart {
|
|
limit = 3
|
|
# ...
|
|
}
|
|
```
|
|
|
|
After the grace period if the script check fails, it has 180 seconds (`60s interval * 3 limit`) to pass before a restart is triggered. Once a restart is
|
|
triggered the task group's [`restart` policy][restart_stanza] takes control:
|
|
|
|
```hcl
|
|
restart {
|
|
# ...
|
|
delay = "10s"
|
|
# ...
|
|
}
|
|
```
|
|
|
|
The [`restart` stanza][restart_stanza] controls the restart behavior of the
|
|
task. In this case it will stop the task and then wait 10 seconds before
|
|
starting it again.
|
|
|
|
Once the task restarts Nomad waits the `grace` period again before starting to
|
|
check the task's health.
|
|
|
|
```hcl
|
|
restart {
|
|
attempts = 3
|
|
# ...
|
|
interval = "10m"
|
|
mode = "fail"
|
|
}
|
|
```
|
|
|
|
If the check continues to fail, the task will be restarted up to `attempts`
|
|
times within an `interval`. If the `restart` attempts are reached within the
|
|
`limit` then the `mode` controls the behavior. In this case the task would fail
|
|
and not be restarted again. See the [`restart` stanza][restart_stanza] for
|
|
details.
|
|
|
|
[check_stanza]: /docs/job-specification/service#check-parameters 'check stanza'
|
|
[restart_stanza]: /docs/job-specification/restart 'restart stanza'
|
|
[service_stanza]: /docs/job-specification/service 'service stanza'
|