luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/drivers/docker/config.go

778 lines
28 KiB
Go
Raw Normal View History

docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
package docker
import (
cleanup driver eventor goroutines This fixes few cases where driver eventor goroutines are leaked during normal operations, but especially so in tests. This change makes few modifications: First, it switches drivers to use `Context`s to manage shutdown events. Previously, it relied on callers invoking `.Shutdown()` function that is specific to internal drivers only and require casting. Using `Contexts` provide a consistent idiomatic way to manage lifecycle for both internal and external drivers. Also, I discovered few places where we don't clean up a temporary driver instance in the plugin catalog code, where we dispense a driver to inspect and validate the schema config without properly cleaning it up.
2020-05-26 13:44:26 +00:00
"context"
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
"fmt"
drivers/docker: detect arch for default infra_image The 'docker.config.infra_image' would default to an amd64 container. It is possible to reference the correct image for a platform using the `runtime.GOARCH` variable, eliminating the need to explicitly set the `infra_image` on non-amd64 platforms. Also upgrade to Google's pause container version 3.1 from 3.0, which includes some enhancements around process management. Fixes #8926
2020-09-23 18:44:27 +00:00
"runtime"
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"strconv"
"strings"
"time"
Support docker bind mounts
2018-11-26 21:45:01 +00:00
docker "github.com/fsouza/go-dockerclient"
Making pull activity timeout configurable in Docker * Making pull activity timeout configurable in Docker plugin config, first pass * Fixing broken function call * Fixing broken tests * Fixing linter suggestion * Adding documentation on new parameter in Docker plugin config * Adding unit test * Setting min value for pull_activity_timeout, making pull activity duration a private var
2019-12-18 11:58:53 +00:00
"github.com/hashicorp/go-hclog"
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
"github.com/hashicorp/nomad/helper/pluginutils/hclutils"
goimports until make check is happy
2019-01-23 14:27:14 +00:00
"github.com/hashicorp/nomad/helper/pluginutils/loader"
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"github.com/hashicorp/nomad/plugins/base"
"github.com/hashicorp/nomad/plugins/drivers"
"github.com/hashicorp/nomad/plugins/shared/hclspec"
)
docker: moved fingerprint code to it's own file
2018-11-16 18:52:54 +00:00
const (
// NoSuchContainerError is returned by the docker daemon if the container
// does not exist.
NoSuchContainerError = "No such container"
driver/docker: stopping a dead container not error
2018-12-15 19:30:29 +00:00
// ContainerNotRunningError is returned by the docker daemon if the container
// is not running, yet we requested it to stop
ContainerNotRunningError = "Container not running"
docker: moved fingerprint code to it's own file
2018-11-16 18:52:54 +00:00
// pluginName is the name of the plugin
pluginName = "docker"
// fingerprintPeriod is the interval at which the driver will send fingerprint responses
fingerprintPeriod = 30 * time.Second
// dockerTimeout is the length of time a request can be outstanding before
// it is timed out.
dockerTimeout = 5 * time.Minute
// dockerBasicCaps is comma-separated list of Linux capabilities that are
// allowed by docker by default, as documented in
// https://docs.docker.com/engine/reference/run/#block-io-bandwidth-blkio-constraint
dockerBasicCaps = "CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID," +
"SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE"
// dockerAuthHelperPrefix is the prefix to attach to the credential helper
// and should be found in the $PATH. Example: ${prefix-}${helper-name}
dockerAuthHelperPrefix = "docker-credential-"
)
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
func PluginLoader(opts map[string]string) (map[string]interface{}, error) {
conf := map[string]interface{}{}
if v, ok := opts["docker.endpoint"]; ok {
conf["endpoint"] = v
}
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
// dockerd auth
authConf := map[string]interface{}{}
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
if v, ok := opts["docker.auth.config"]; ok {
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
authConf["config"] = v
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
if v, ok := opts["docker.auth.helper"]; ok {
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
authConf["helper"] = v
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
conf["auth"] = authConf
// dockerd tls
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
if _, ok := opts["docker.tls.cert"]; ok {
conf["tls"] = map[string]interface{}{
"cert": opts["docker.tls.cert"],
"key": opts["docker.tls.key"],
"ca": opts["docker.tls.ca"],
}
}
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
// garbage collection
gcConf := map[string]interface{}{}
if v, err := strconv.ParseBool(opts["docker.cleanup.image"]); err == nil {
gcConf["image"] = v
}
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
if v, ok := opts["docker.cleanup.image.delay"]; ok {
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
gcConf["image_delay"] = v
}
if v, err := strconv.ParseBool(opts["docker.cleanup.container"]); err == nil {
gcConf["container"] = v
}
conf["gc"] = gcConf
// volume options
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
volConf := map[string]interface{}{}
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
if v, err := strconv.ParseBool(opts["docker.volumes.enabled"]); err == nil {
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
volConf["enabled"] = v
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
if v, ok := opts["docker.volumes.selinuxlabel"]; ok {
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
volConf["selinuxlabel"] = v
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
conf["volumes"] = volConf
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
// capabilities
s/0.13/1.0/g 1.0 here we come!
2020-10-14 22:17:47 +00:00
// COMPAT(1.0) uses inclusive language. whitelist is used for backward compatibility.
use allow/deny instead of the colored alternatives (#9019) Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2020-10-12 12:47:05 +00:00
if v, ok := opts["docker.caps.allowlist"]; ok {
conf["allow_caps"] = strings.Split(v, ",")
} else if v, ok := opts["docker.caps.whitelist"]; ok {
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
conf["allow_caps"] = strings.Split(v, ",")
}
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
// privileged containers
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
if v, err := strconv.ParseBool(opts["docker.privileged.enabled"]); err == nil {
conf["allow_privileged"] = v
}
Add support for docker runtimes - docker fingerprint issues a docker api system info call to get the list of supported OCI runtimes. - OCI runtimes are reported as comma separated list of names - docker driver is aware of GPU runtime presence - docker driver throws an error when user tries to run container with GPU, when GPU runtime is not present - docker GPU runtime name is configurable
2018-12-18 01:03:43 +00:00
// nvidia_runtime
if v, ok := opts["docker.nvidia_runtime"]; ok {
conf["nvidia_runtime"] = v
}
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
return conf, nil
}
var (
// PluginID is the rawexec plugin metadata registered in the plugin
// catalog.
PluginID = loader.PluginID{
Name: pluginName,
PluginType: base.PluginTypeDriver,
}
// PluginConfig is the rawexec factory function registered in the
// plugin catalog.
PluginConfig = &loader.InternalPluginConfig{
Config: map[string]interface{}{},
cleanup driver eventor goroutines This fixes few cases where driver eventor goroutines are leaked during normal operations, but especially so in tests. This change makes few modifications: First, it switches drivers to use `Context`s to manage shutdown events. Previously, it relied on callers invoking `.Shutdown()` function that is specific to internal drivers only and require casting. Using `Contexts` provide a consistent idiomatic way to manage lifecycle for both internal and external drivers. Also, I discovered few places where we don't clean up a temporary driver instance in the plugin catalog code, where we dispense a driver to inspect and validate the schema config without properly cleaning it up.
2020-05-26 13:44:26 +00:00
Factory: func(ctx context.Context, l hclog.Logger) interface{} { return NewDockerDriver(ctx, l) },
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
// pluginInfo is the response returned for the PluginInfo RPC
pluginInfo = &base.PluginInfoResponse{
Add plugin API versioning to plugin loader and plugins
2018-12-18 00:40:58 +00:00
Type: base.PluginTypeDriver,
PluginApiVersions: []string{drivers.ApiVersion010},
PluginVersion: "0.1.0",
Name: pluginName,
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
danglingContainersBlock = hclspec.NewObject(map[string]*hclspec.Spec{
"enabled": hclspec.NewDefault(
hclspec.NewAttr("enabled", "bool", false),
hclspec.NewLiteral(`true`),
),
"period": hclspec.NewDefault(
hclspec.NewAttr("period", "string", false),
hclspec.NewLiteral(`"5m"`),
),
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
"creation_grace": hclspec.NewDefault(
hclspec.NewAttr("creation_grace", "string", false),
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
hclspec.NewLiteral(`"5m"`),
),
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
"dry_run": hclspec.NewDefault(
hclspec.NewAttr("dry_run", "bool", false),
hclspec.NewLiteral(`false`),
),
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
})
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
// configSpec is the hcl specification returned by the ConfigSchema RPC
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
// and is used to parse the contents of the 'plugin "docker" {...}' block.
// Example:
// plugin "docker" {
executor: fix failing stats related test
2019-01-10 19:20:18 +00:00
// config {
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
// endpoint = "unix:///var/run/docker.sock"
// auth {
// config = "/etc/docker-auth.json"
// helper = "docker-credential-aws"
// }
// tls {
// cert = "/etc/nomad/nomad.pub"
// key = "/etc/nomad/nomad.pem"
// ca = "/etc/nomad/nomad.cert"
// }
// gc {
// image = true
// image_delay = "5m"
// container = false
// }
// volumes {
// enabled = true
// selinuxlabel = "z"
// }
// allow_privileged = false
// allow_caps = ["CHOWN", "NET_RAW" ... ]
Add support for docker runtimes - docker fingerprint issues a docker api system info call to get the list of supported OCI runtimes. - OCI runtimes are reported as comma separated list of names - docker driver is aware of GPU runtime presence - docker driver throws an error when user tries to run container with GPU, when GPU runtime is not present - docker GPU runtime name is configurable
2018-12-18 01:03:43 +00:00
// nvidia_runtime = "nvidia"
executor: fix failing stats related test
2019-01-10 19:20:18 +00:00
// }
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
// }
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
configSpec = hclspec.NewObject(map[string]*hclspec.Spec{
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
"endpoint": hclspec.NewAttr("endpoint", "string", false),
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
// docker daemon auth option for image registry
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
"auth": hclspec.NewBlock("auth", false, hclspec.NewObject(map[string]*hclspec.Spec{
"config": hclspec.NewAttr("config", "string", false),
"helper": hclspec.NewAttr("helper", "string", false),
})),
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
// client tls options
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"tls": hclspec.NewBlock("tls", false, hclspec.NewObject(map[string]*hclspec.Spec{
"cert": hclspec.NewAttr("cert", "string", false),
"key": hclspec.NewAttr("key", "string", false),
"ca": hclspec.NewAttr("ca", "string", false),
})),
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
driver/docker: add extra labels ( job name, task and task group name)
2021-03-08 13:59:52 +00:00
// extra docker labels, globs supported
"extra_labels": hclspec.NewAttr("extra_labels", "list(string)", false),
docker: support configuring default log driver in plugin options
2021-03-12 21:04:33 +00:00
// logging options
"logging": hclspec.NewDefault(hclspec.NewBlock("logging", false, hclspec.NewObject(map[string]*hclspec.Spec{
"type": hclspec.NewAttr("type", "string", false),
"config": hclspec.NewBlockAttrs("config", "string", false),
})), hclspec.NewLiteral(`{
type = "json-file"
config = {
max-file = "2"
max-size = "2m"
}
}`)),
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
// garbage collection options
// default needed for both if the gc {...} block is not set and
// if the default fields are missing
"gc": hclspec.NewDefault(hclspec.NewBlock("gc", false, hclspec.NewObject(map[string]*hclspec.Spec{
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
"image": hclspec.NewDefault(
hclspec.NewAttr("image", "bool", false),
hclspec.NewLiteral("true"),
),
Add default value "3m" to image_delay, making it consistent with docs.
2019-08-06 06:01:51 +00:00
"image_delay": hclspec.NewDefault(
hclspec.NewAttr("image_delay", "string", false),
docker: Fix driver spec hclspec.NewLiteral does not quote its values, which caused `3m` to be parsed as a nonsensical literal which broke the plugin loader during initialization. By quoting the value here, it starts correctly.
2019-09-03 06:53:37 +00:00
hclspec.NewLiteral("\"3m\""),
Add default value "3m" to image_delay, making it consistent with docs.
2019-08-06 06:01:51 +00:00
),
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
"container": hclspec.NewDefault(
hclspec.NewAttr("container", "bool", false),
hclspec.NewLiteral("true"),
),
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
"dangling_containers": hclspec.NewDefault(
hclspec.NewBlock("dangling_containers", false, danglingContainersBlock),
driver/docker: ensure that defaults are populated Looks like we may need to pass default literal at each layer to be able, so defaults are set properly.
2019-10-18 22:27:28 +00:00
hclspec.NewLiteral(`{
enabled = true
period = "5m"
creation_grace = "5m"
}`),
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
),
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
})), hclspec.NewLiteral(`{
image = true
docker: image_delay default missing without gc stanza (#9101) In the Docker driver plugin config for garbage collection, the `image_delay` field was missing from the default we set if the entire `gc` stanza is missing. This results in a default of 0s and immediate GC of Docker images. Expanded docker gc config test fields.
2020-10-15 16:36:01 +00:00
image_delay = "3m"
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
container = true
driver/docker: ensure that defaults are populated Looks like we may need to pass default literal at each layer to be able, so defaults are set properly.
2019-10-18 22:27:28 +00:00
dangling_containers = {
enabled = true
period = "5m"
creation_grace = "5m"
}
docker: add default blocks for driver plugin config schema
2018-11-20 03:58:05 +00:00
}`)),
// docker volume options
// defaulted needed for both if the volumes {...} block is not set and
// if the default fields are missing
"volumes": hclspec.NewDefault(hclspec.NewBlock("volumes", false, hclspec.NewObject(map[string]*hclspec.Spec{
docker: disable host volume binding by default
2020-06-23 17:43:37 +00:00
"enabled": hclspec.NewAttr("enabled", "bool", false),
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
"selinuxlabel": hclspec.NewAttr("selinuxlabel", "string", false),
docker: disallow volume mounts from host by default (#9321) The default behavior for `docker.volumes.enabled` is intended to be `false`, but the HCL schema defaults to `true` if the value is unset. Set the default literal value to `true`. Additionally, Docker driver mounts of type "volume" (but not "bind") are not being properly sandboxed with that setting. Disable Docker mounts with type "volume" entirely whenever the `docker.volumes.enabled` flag is set to false. Note this is unrelated to the `volume_mount` feature, which is constrained to preconfigured host volumes or whatever is mounted by a CSI plugin. This changeset includes updates to unit tests that should have been failing under the documented behavior but were not.
2020-11-11 15:03:46 +00:00
})), hclspec.NewLiteral("{ enabled = false }")),
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
"allow_privileged": hclspec.NewAttr("allow_privileged", "bool", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"allow_caps": hclspec.NewDefault(
hclspec.NewAttr("allow_caps", "list(string)", false),
hclspec.NewLiteral(`["CHOWN","DAC_OVERRIDE","FSETID","FOWNER","MKNOD","NET_RAW","SETGID","SETUID","SETFCAP","SETPCAP","NET_BIND_SERVICE","SYS_CHROOT","KILL","AUDIT_WRITE"]`),
),
Add support for docker runtimes - docker fingerprint issues a docker api system info call to get the list of supported OCI runtimes. - OCI runtimes are reported as comma separated list of names - docker driver is aware of GPU runtime presence - docker driver throws an error when user tries to run container with GPU, when GPU runtime is not present - docker GPU runtime name is configurable
2018-12-18 01:03:43 +00:00
"nvidia_runtime": hclspec.NewDefault(
hclspec.NewAttr("nvidia_runtime", "string", false),
hclspec.NewLiteral(`"nvidia"`),
),
Add a knob to restrict docker runtimes
2020-05-12 14:13:50 +00:00
// list of docker runtimes allowed to be used
use allow_runtimes for consistency Other allow lists use allow_ prefix (e.g. allow_caps, allow_privileged).
2020-05-12 15:03:08 +00:00
"allow_runtimes": hclspec.NewDefault(
hclspec.NewAttr("allow_runtimes", "list(string)", false),
Add a knob to restrict docker runtimes
2020-05-12 14:13:50 +00:00
hclspec.NewLiteral(`["runc", "nvidia"]`),
),
docker: allow configuration of infra image
2019-06-14 15:42:32 +00:00
// image to use when creating a network namespace parent container
"infra_image": hclspec.NewDefault(
hclspec.NewAttr("infra_image", "string", false),
drivers/docker: detect arch for default infra_image The 'docker.config.infra_image' would default to an amd64 container. It is possible to reference the correct image for a platform using the `runtime.GOARCH` variable, eliminating the need to explicitly set the `infra_image` on non-amd64 platforms. Also upgrade to Google's pause container version 3.1 from 3.0, which includes some enhancements around process management. Fixes #8926
2020-09-23 18:44:27 +00:00
hclspec.NewLiteral(fmt.Sprintf(
`"gcr.io/google_containers/pause-%s:3.1"`,
runtime.GOARCH,
)),
docker: allow configuration of infra image
2019-06-14 15:42:32 +00:00
),
driver/docker: allow configurable pull context timeout setting. Pulling large docker containers can take longer than the default context timeout. Without a way to change this it is very hard for users to utilise Nomad properly without hacky work arounds. This change adds an optional pull_timeout config parameter which gives operators the possibility to account for increase pull times where needed. The infra docker image also has the option to set a custom timeout to keep consistency.
2020-08-12 07:58:07 +00:00
// timeout to use when pulling the infra image.
"infra_image_pull_timeout": hclspec.NewDefault(
hclspec.NewAttr("infra_image_pull_timeout", "string", false),
hclspec.NewLiteral(`"5m"`),
),
driver: allow disabling log collection Operators commonly have docker logs aggregated using various tools and don't need nomad to manage their docker logs. Worse, Nomad uses a somewhat heavy docker api call to collect them and it seems to cause problems when a client runs hundreds of log collections. Here we add a knob to disable log aggregation completely for nomad. When log collection is disabled, we avoid running logmon and docker_logger for the docker tasks in this implementation. The downside here is once disabled, `nomad logs ...` commands and API no longer return logs and operators must corrolate alloc-ids with their aggregated log info. This is meant as a stop gap measure. Ideally, we'd follow up with at least two changes: First, we should optimize behavior when we can such that operators don't need to disable docker log collection. Potentially by reverting to using pre-0.9 syslog aggregation in linux environments, though with different trade-offs. Second, when/if logs are disabled, nomad logs endpoints should lookup docker logs api on demand. This ensures that the cost of log collection is paid sparingly.
2019-12-07 03:11:41 +00:00
Making pull activity timeout configurable in Docker * Making pull activity timeout configurable in Docker plugin config, first pass * Fixing broken function call * Fixing broken tests * Fixing linter suggestion * Adding documentation on new parameter in Docker plugin config * Adding unit test * Setting min value for pull_activity_timeout, making pull activity duration a private var
2019-12-18 11:58:53 +00:00
// the duration that the driver will wait for activity from the Docker engine during an image pull
// before canceling the request
"pull_activity_timeout": hclspec.NewDefault(
hclspec.NewAttr("pull_activity_timeout", "string", false),
hclspec.NewLiteral(`"2m"`),
),
address review comments
2019-12-13 16:08:12 +00:00
// disable_log_collection indicates whether docker driver should collect logs of docker
// task containers. If true, nomad doesn't start docker_logger/logmon processes
driver: allow disabling log collection Operators commonly have docker logs aggregated using various tools and don't need nomad to manage their docker logs. Worse, Nomad uses a somewhat heavy docker api call to collect them and it seems to cause problems when a client runs hundreds of log collections. Here we add a knob to disable log aggregation completely for nomad. When log collection is disabled, we avoid running logmon and docker_logger for the docker tasks in this implementation. The downside here is once disabled, `nomad logs ...` commands and API no longer return logs and operators must corrolate alloc-ids with their aggregated log info. This is meant as a stop gap measure. Ideally, we'd follow up with at least two changes: First, we should optimize behavior when we can such that operators don't need to disable docker log collection. Potentially by reverting to using pre-0.9 syslog aggregation in linux environments, though with different trade-offs. Second, when/if logs are disabled, nomad logs endpoints should lookup docker logs api on demand. This ensures that the cost of log collection is paid sparingly.
2019-12-07 03:11:41 +00:00
"disable_log_collection": hclspec.NewAttr("disable_log_collection", "bool", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
})
docker: introduce a new hcl2-friendly `mount` syntax (#9635) Introduce a new more-block friendly syntax for specifying mounts with a new `mount` block type with the target as label: ```hcl config { image = "..." mount { type = "..." target = "target-path" volume_options { ... } } } ``` The main benefit here is that by `mount` being a block, it can nest blocks and avoids the compatibility problems noted in https://github.com/hashicorp/nomad/pull/9634/files#diff-2161d829655a3a36ba2d916023e4eec125b9bd22873493c1c2e5e3f7ba92c691R128-R155 . The intention is for us to promote this `mount` blocks and quietly deprecate the `mounts` type, while still honoring to preserve compatibility as much as we could. This addresses the issue in https://github.com/hashicorp/nomad/issues/9604 .
2020-12-15 19:13:50 +00:00
// mountBodySpec is the hcl specification for the `mount` block
mountBodySpec = hclspec.NewObject(map[string]*hclspec.Spec{
"type": hclspec.NewDefault(
hclspec.NewAttr("type", "string", false),
hclspec.NewLiteral("\"volume\""),
),
"target": hclspec.NewAttr("target", "string", false),
"source": hclspec.NewAttr("source", "string", false),
"readonly": hclspec.NewAttr("readonly", "bool", false),
"bind_options": hclspec.NewBlock("bind_options", false, hclspec.NewObject(map[string]*hclspec.Spec{
"propagation": hclspec.NewAttr("propagation", "string", false),
})),
"tmpfs_options": hclspec.NewBlock("tmpfs_options", false, hclspec.NewObject(map[string]*hclspec.Spec{
"size": hclspec.NewAttr("size", "number", false),
"mode": hclspec.NewAttr("mode", "number", false),
})),
"volume_options": hclspec.NewBlock("volume_options", false, hclspec.NewObject(map[string]*hclspec.Spec{
"no_copy": hclspec.NewAttr("no_copy", "bool", false),
"labels": hclspec.NewAttr("labels", "list(map(string))", false),
"driver_config": hclspec.NewBlock("driver_config", false, hclspec.NewObject(map[string]*hclspec.Spec{
"name": hclspec.NewAttr("name", "string", false),
"options": hclspec.NewAttr("options", "list(map(string))", false),
})),
})),
})
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
// taskConfigSpec is the hcl specification for the driver config section of
// a task within a job. It is returned in the TaskConfigSchema RPC
taskConfigSpec = hclspec.NewObject(map[string]*hclspec.Spec{
"image": hclspec.NewAttr("image", "string", true),
"advertise_ipv6_address": hclspec.NewAttr("advertise_ipv6_address", "bool", false),
"args": hclspec.NewAttr("args", "list(string)", false),
"auth": hclspec.NewBlock("auth", false, hclspec.NewObject(map[string]*hclspec.Spec{
"username": hclspec.NewAttr("username", "string", false),
"password": hclspec.NewAttr("password", "string", false),
"email": hclspec.NewAttr("email", "string", false),
"server_address": hclspec.NewAttr("server_address", "string", false),
})),
"auth_soft_fail": hclspec.NewAttr("auth_soft_fail", "bool", false),
"cap_add": hclspec.NewAttr("cap_add", "list(string)", false),
"cap_drop": hclspec.NewAttr("cap_drop", "list(string)", false),
"command": hclspec.NewAttr("command", "string", false),
Add cpuset_cpus to docker driver.
2020-06-25 16:30:16 +00:00
"cpuset_cpus": hclspec.NewAttr("cpuset_cpus", "string", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"cpu_hard_limit": hclspec.NewAttr("cpu_hard_limit", "bool", false),
docker: set default cpu cfs period (#6737) * docker: set default cpu cfs period Co-Authored-By: Michael Schurter <mschurter@hashicorp.com>
2019-11-20 00:05:15 +00:00
"cpu_cfs_period": hclspec.NewDefault(
hclspec.NewAttr("cpu_cfs_period", "number", false),
hclspec.NewLiteral(`100000`),
),
tests: add hcl task driver config parsing tests (#5314) * drivers: add config parsing tests Add basic tests for parsing and encoding task config. * drivers/docker: fix some config declarations * refactor and document config parse helpers
2019-02-12 19:46:37 +00:00
"devices": hclspec.NewBlockList("devices", hclspec.NewObject(map[string]*hclspec.Spec{
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"host_path": hclspec.NewAttr("host_path", "string", false),
"container_path": hclspec.NewAttr("container_path", "string", false),
"cgroup_permissions": hclspec.NewAttr("cgroup_permissions", "string", false),
})),
"dns_search_domains": hclspec.NewAttr("dns_search_domains", "list(string)", false),
"dns_options": hclspec.NewAttr("dns_options", "list(string)", false),
"dns_servers": hclspec.NewAttr("dns_servers", "list(string)", false),
"entrypoint": hclspec.NewAttr("entrypoint", "list(string)", false),
"extra_hosts": hclspec.NewAttr("extra_hosts", "list(string)", false),
"force_pull": hclspec.NewAttr("force_pull", "bool", false),
"hostname": hclspec.NewAttr("hostname", "string", false),
"interactive": hclspec.NewAttr("interactive", "bool", false),
"ipc_mode": hclspec.NewAttr("ipc_mode", "string", false),
"ipv4_address": hclspec.NewAttr("ipv4_address", "string", false),
"ipv6_address": hclspec.NewAttr("ipv6_address", "string", false),
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
"labels": hclspec.NewAttr("labels", "list(map(string))", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"load": hclspec.NewAttr("load", "string", false),
docker: fix logging config parsing Fixes https://groups.google.com/d/topic/nomad-tool/B3Uo6Kns2BI/discussion
2019-02-04 18:56:05 +00:00
"logging": hclspec.NewBlock("logging", false, hclspec.NewObject(map[string]*hclspec.Spec{
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
"type": hclspec.NewAttr("type", "string", false),
drivers/docker: rename logging `type` to `driver` Docker uses the term logging `driver` in its public documentations: in `docker` daemon config[1], `docker run` arguments [2] and in docker compose file[3]. Interestingly, docker used `type` in its API [4] instead of everywhere else. It's unfortunate that Nomad used `type` modeling after the Docker API rather than the user facing documents. Nomad using `type` feels very non-user friendly as it's disconnected from how Docker markets the flag and shows internal representation instead. Here, we rectify the situation by introducing `driver` field and prefering it over `type` in logging. [1] https://docs.docker.com/config/containers/logging/configure/ [2] https://docs.docker.com/engine/reference/run/#logging-drivers---log-driver [3] https://docs.docker.com/compose/compose-file/#logging [4] https://docs.docker.com/engine/api/v1.39/#operation/ContainerCreate
2019-02-28 20:25:17 +00:00
"driver": hclspec.NewAttr("driver", "string", false),
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
"config": hclspec.NewAttr("config", "list(map(string))", false),
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
})),
driver/docker: enable setting hard/soft memory limits Fixes #2093 Enable configuring `memory_hard_limit` in the docker config stanza for tasks. If set, this field will be passed to the container runtime as `--memory`, and the `memory` configuration from the task resource configuration will be passed as `--memory_reservation`, creating hard and soft memory limits for tasks using the docker task driver.
2020-05-31 17:38:27 +00:00
"mac_address": hclspec.NewAttr("mac_address", "string", false),
"memory_hard_limit": hclspec.NewAttr("memory_hard_limit", "number", false),
docker: introduce a new hcl2-friendly `mount` syntax (#9635) Introduce a new more-block friendly syntax for specifying mounts with a new `mount` block type with the target as label: ```hcl config { image = "..." mount { type = "..." target = "target-path" volume_options { ... } } } ``` The main benefit here is that by `mount` being a block, it can nest blocks and avoids the compatibility problems noted in https://github.com/hashicorp/nomad/pull/9634/files#diff-2161d829655a3a36ba2d916023e4eec125b9bd22873493c1c2e5e3f7ba92c691R128-R155 . The intention is for us to promote this `mount` blocks and quietly deprecate the `mounts` type, while still honoring to preserve compatibility as much as we could. This addresses the issue in https://github.com/hashicorp/nomad/issues/9604 .
2020-12-15 19:13:50 +00:00
// mount and mounts are effectively aliases, but `mounts` is meant for pre-1.0
// assignment syntax `mounts = [{type="..." ..."}]` while
// `mount` is 1.0 repeated block syntax `mount { type = "..." }`
"mount": hclspec.NewBlockList("mount", mountBodySpec),
"mounts": hclspec.NewBlockList("mounts", mountBodySpec),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"network_aliases": hclspec.NewAttr("network_aliases", "list(string)", false),
"network_mode": hclspec.NewAttr("network_mode", "string", false),
Rename OCIRuntime to Runtime; allow gpu conflicts is they are the same runtime; add conflict test
2020-04-03 18:40:58 +00:00
"runtime": hclspec.NewAttr("runtime", "string", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"pids_limit": hclspec.NewAttr("pids_limit", "number", false),
"pid_mode": hclspec.NewAttr("pid_mode", "string", false),
docker: support group allocated ports and host_networks (#8623) * docker: support group allocated ports * docker: add new ports driver config to specify which group ports are mapped * docker: update port mapping docs
2020-08-11 22:30:22 +00:00
"ports": hclspec.NewAttr("ports", "list(string)", false),
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
"port_map": hclspec.NewAttr("port_map", "list(map(number))", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"privileged": hclspec.NewAttr("privileged", "bool", false),
driver/docker: allow configurable pull context timeout setting. Pulling large docker containers can take longer than the default context timeout. Without a way to change this it is very hard for users to utilise Nomad properly without hacky work arounds. This change adds an optional pull_timeout config parameter which gives operators the possibility to account for increase pull times where needed. The infra docker image also has the option to set a custom timeout to keep consistency.
2020-08-12 07:58:07 +00:00
"image_pull_timeout": hclspec.NewDefault(
hclspec.NewAttr("image_pull_timeout", "string", false),
hclspec.NewLiteral(`"5m"`),
),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"readonly_rootfs": hclspec.NewAttr("readonly_rootfs", "bool", false),
"security_opt": hclspec.NewAttr("security_opt", "list(string)", false),
"shm_size": hclspec.NewAttr("shm_size", "number", false),
driver/docker: use BlockAttrs for storage_opts storage_opts is a new field in 0.9 cycle and doesn't have backward compatibility constraints.
2019-02-20 01:22:59 +00:00
"storage_opt": hclspec.NewBlockAttrs("storage_opt", "string", false),
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
"sysctl": hclspec.NewAttr("sysctl", "list(map(string))", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"tty": hclspec.NewAttr("tty", "bool", false),
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
"ulimit": hclspec.NewAttr("ulimit", "list(map(string))", false),
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
"uts_mode": hclspec.NewAttr("uts_mode", "string", false),
"userns_mode": hclspec.NewAttr("userns_mode", "string", false),
"volumes": hclspec.NewAttr("volumes", "list(string)", false),
"volume_driver": hclspec.NewAttr("volume_driver", "string", false),
"work_dir": hclspec.NewAttr("work_dir", "string", false),
})
// capabilities is returned by the Capabilities RPC and indicates what
// optional features this driver supports
capabilities = &drivers.Capabilities{
SendSignals: true,
Exec: true,
use drivers.FSIsolation
2019-01-04 21:11:25 +00:00
FSIsolation: drivers.FSIsolationImage,
docker: support shared network namespaces
2019-05-14 00:59:31 +00:00
NetIsolationModes: []drivers.NetIsolationMode{
drivers.NetIsolationModeHost,
drivers.NetIsolationModeGroup,
drivers.NetIsolationModeTask,
},
MustInitiateNetwork: true,
volumes: return better error messages for unsupported task drivers (#8030) When an allocation runs for a task driver that can't support volume mounts, the mounting will fail in a way that can be hard to understand. With host volumes this usually means failing silently, whereas with CSI the operator gets inscrutable internals exposed in the `nomad alloc status`. This changeset adds a MountConfig field to the task driver Capabilities response. We validate this when the `csi_hook` or `volume_hook` fires and return a user-friendly error. Note that we don't currently have a way to get driver capabilities up to the server, except through attributes. Validating this when the user initially submits the jobspec would be even better than what we're doing here (and could be useful for all our other capabilities), but that's out of scope for this changeset. Also note that the MountConfig enum starts with "supports all" in order to support community plugins in a backwards compatible way, rather than cutting them off from volume mounting unexpectedly.
2020-05-21 13:18:02 +00:00
MountConfigs: drivers.MountConfigSupportAll,
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
)
type TaskConfig struct {
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Image string `codec:"image"`
AdvertiseIPv6Addr bool `codec:"advertise_ipv6_address"`
Args []string `codec:"args"`
Auth DockerAuth `codec:"auth"`
AuthSoftFail bool `codec:"auth_soft_fail"`
CapAdd []string `codec:"cap_add"`
CapDrop []string `codec:"cap_drop"`
Command string `codec:"command"`
CPUCFSPeriod int64 `codec:"cpu_cfs_period"`
CPUHardLimit bool `codec:"cpu_hard_limit"`
Add cpuset_cpus to docker driver.
2020-06-25 16:30:16 +00:00
CPUSetCPUs string `codec:"cpuset_cpus"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Devices []DockerDevice `codec:"devices"`
DNSSearchDomains []string `codec:"dns_search_domains"`
DNSOptions []string `codec:"dns_options"`
DNSServers []string `codec:"dns_servers"`
Entrypoint []string `codec:"entrypoint"`
ExtraHosts []string `codec:"extra_hosts"`
ForcePull bool `codec:"force_pull"`
Hostname string `codec:"hostname"`
Interactive bool `codec:"interactive"`
IPCMode string `codec:"ipc_mode"`
IPv4Address string `codec:"ipv4_address"`
IPv6Address string `codec:"ipv6_address"`
Labels hclutils.MapStrStr `codec:"labels"`
LoadImage string `codec:"load"`
Logging DockerLogging `codec:"logging"`
MacAddress string `codec:"mac_address"`
driver/docker: enable setting hard/soft memory limits Fixes #2093 Enable configuring `memory_hard_limit` in the docker config stanza for tasks. If set, this field will be passed to the container runtime as `--memory`, and the `memory` configuration from the task resource configuration will be passed as `--memory_reservation`, creating hard and soft memory limits for tasks using the docker task driver.
2020-05-31 17:38:27 +00:00
MemoryHardLimit int64 `codec:"memory_hard_limit"`
docker: introduce a new hcl2-friendly `mount` syntax (#9635) Introduce a new more-block friendly syntax for specifying mounts with a new `mount` block type with the target as label: ```hcl config { image = "..." mount { type = "..." target = "target-path" volume_options { ... } } } ``` The main benefit here is that by `mount` being a block, it can nest blocks and avoids the compatibility problems noted in https://github.com/hashicorp/nomad/pull/9634/files#diff-2161d829655a3a36ba2d916023e4eec125b9bd22873493c1c2e5e3f7ba92c691R128-R155 . The intention is for us to promote this `mount` blocks and quietly deprecate the `mounts` type, while still honoring to preserve compatibility as much as we could. This addresses the issue in https://github.com/hashicorp/nomad/issues/9604 .
2020-12-15 19:13:50 +00:00
Mounts []DockerMount `codec:"mount"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
NetworkAliases []string `codec:"network_aliases"`
NetworkMode string `codec:"network_mode"`
Rename OCIRuntime to Runtime; allow gpu conflicts is they are the same runtime; add conflict test
2020-04-03 18:40:58 +00:00
Runtime string `codec:"runtime"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
PidsLimit int64 `codec:"pids_limit"`
PidMode string `codec:"pid_mode"`
docker: support group allocated ports and host_networks (#8623) * docker: support group allocated ports * docker: add new ports driver config to specify which group ports are mapped * docker: update port mapping docs
2020-08-11 22:30:22 +00:00
Ports []string `codec:"ports"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
PortMap hclutils.MapStrInt `codec:"port_map"`
Privileged bool `codec:"privileged"`
driver/docker: allow configurable pull context timeout setting. Pulling large docker containers can take longer than the default context timeout. Without a way to change this it is very hard for users to utilise Nomad properly without hacky work arounds. This change adds an optional pull_timeout config parameter which gives operators the possibility to account for increase pull times where needed. The infra docker image also has the option to set a custom timeout to keep consistency.
2020-08-12 07:58:07 +00:00
ImagePullTimeout string `codec:"image_pull_timeout"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
ReadonlyRootfs bool `codec:"readonly_rootfs"`
SecurityOpt []string `codec:"security_opt"`
ShmSize int64 `codec:"shm_size"`
driver/docker: use BlockAttrs for storage_opts storage_opts is a new field in 0.9 cycle and doesn't have backward compatibility constraints.
2019-02-20 01:22:59 +00:00
StorageOpt map[string]string `codec:"storage_opt"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Sysctl hclutils.MapStrStr `codec:"sysctl"`
TTY bool `codec:"tty"`
Ulimit hclutils.MapStrStr `codec:"ulimit"`
UTSMode string `codec:"uts_mode"`
UsernsMode string `codec:"userns_mode"`
Volumes []string `codec:"volumes"`
VolumeDriver string `codec:"volume_driver"`
WorkDir string `codec:"work_dir"`
docker: introduce a new hcl2-friendly `mount` syntax (#9635) Introduce a new more-block friendly syntax for specifying mounts with a new `mount` block type with the target as label: ```hcl config { image = "..." mount { type = "..." target = "target-path" volume_options { ... } } } ``` The main benefit here is that by `mount` being a block, it can nest blocks and avoids the compatibility problems noted in https://github.com/hashicorp/nomad/pull/9634/files#diff-2161d829655a3a36ba2d916023e4eec125b9bd22873493c1c2e5e3f7ba92c691R128-R155 . The intention is for us to promote this `mount` blocks and quietly deprecate the `mounts` type, while still honoring to preserve compatibility as much as we could. This addresses the issue in https://github.com/hashicorp/nomad/issues/9604 .
2020-12-15 19:13:50 +00:00
// MountsList supports the pre-1.0 mounts array syntax
MountsList []DockerMount `codec:"mounts"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
type DockerAuth struct {
Username string `codec:"username"`
Password string `codec:"password"`
Email string `codec:"email"`
ServerAddr string `codec:"server_address"`
}
type DockerDevice struct {
HostPath string `codec:"host_path"`
ContainerPath string `codec:"container_path"`
CgroupPermissions string `codec:"cgroup_permissions"`
}
refactor device manipulation
2018-12-04 19:50:59 +00:00
func (d DockerDevice) toDockerDevice() (docker.Device, error) {
dd := docker.Device{
PathOnHost: d.HostPath,
PathInContainer: d.ContainerPath,
CgroupPermissions: d.CgroupPermissions,
}
if d.HostPath == "" {
return dd, fmt.Errorf("host path must be set in configuration for devices")
}
if dd.CgroupPermissions == "" {
dd.CgroupPermissions = "rwm"
}
if !validateCgroupPermission(dd.CgroupPermissions) {
return dd, fmt.Errorf("invalid cgroup permission string: %q", dd.CgroupPermissions)
}
return dd, nil
}
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
type DockerLogging struct {
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Type string `codec:"type"`
drivers/docker: rename logging `type` to `driver` Docker uses the term logging `driver` in its public documentations: in `docker` daemon config[1], `docker run` arguments [2] and in docker compose file[3]. Interestingly, docker used `type` in its API [4] instead of everywhere else. It's unfortunate that Nomad used `type` modeling after the Docker API rather than the user facing documents. Nomad using `type` feels very non-user friendly as it's disconnected from how Docker markets the flag and shows internal representation instead. Here, we rectify the situation by introducing `driver` field and prefering it over `type` in logging. [1] https://docs.docker.com/config/containers/logging/configure/ [2] https://docs.docker.com/engine/reference/run/#logging-drivers---log-driver [3] https://docs.docker.com/compose/compose-file/#logging [4] https://docs.docker.com/engine/api/v1.39/#operation/ContainerCreate
2019-02-28 20:25:17 +00:00
Driver string `codec:"driver"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Config hclutils.MapStrStr `codec:"config"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
type DockerMount struct {
Support docker bind mounts
2018-11-26 21:45:01 +00:00
Type string `codec:"type"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
Target string `codec:"target"`
Source string `codec:"source"`
ReadOnly bool `codec:"readonly"`
Support docker bind mounts
2018-11-26 21:45:01 +00:00
BindOptions DockerBindOptions `codec:"bind_options"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
VolumeOptions DockerVolumeOptions `codec:"volume_options"`
add support for tmpfs
2018-11-26 22:22:57 +00:00
TmpfsOptions DockerTmpfsOptions `codec:"tmpfs_options"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
Support docker bind mounts
2018-11-26 21:45:01 +00:00
func (m DockerMount) toDockerHostMount() (docker.HostMount, error) {
if m.Type == "" {
use allow/deny instead of the colored alternatives (#9019) Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2020-10-12 12:47:05 +00:00
// for backward compatibility, as type is optional
Support docker bind mounts
2018-11-26 21:45:01 +00:00
m.Type = "volume"
}
hm := docker.HostMount{
Target: m.Target,
Source: m.Source,
Type: m.Type,
ReadOnly: m.ReadOnly,
}
switch m.Type {
case "volume":
vo := m.VolumeOptions
hm.VolumeOptions = &docker.VolumeOptions{
NoCopy: vo.NoCopy,
Labels: vo.Labels,
DriverConfig: docker.VolumeDriverConfig{
Name: vo.DriverConfig.Name,
Options: vo.DriverConfig.Options,
},
}
case "bind":
hm.BindOptions = &docker.BindOptions{
Propagation: m.BindOptions.Propagation,
}
add support for tmpfs
2018-11-26 22:22:57 +00:00
case "tmpfs":
if m.Source != "" {
return hm, fmt.Errorf(`invalid source, must be "" for tmpfs`)
}
hm.TempfsOptions = &docker.TempfsOptions{
SizeBytes: m.TmpfsOptions.SizeBytes,
Mode: m.TmpfsOptions.Mode,
}
Support docker bind mounts
2018-11-26 21:45:01 +00:00
default:
add support for tmpfs
2018-11-26 22:22:57 +00:00
return hm, fmt.Errorf(`invalid mount type, must be "bind", "volume", "tmpfs": %q`, m.Type)
Support docker bind mounts
2018-11-26 21:45:01 +00:00
}
return hm, nil
}
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
type DockerVolumeOptions struct {
NoCopy bool `codec:"no_copy"`
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Labels hclutils.MapStrStr `codec:"labels"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
DriverConfig DockerVolumeDriverConfig `codec:"driver_config"`
}
Support docker bind mounts
2018-11-26 21:45:01 +00:00
type DockerBindOptions struct {
Propagation string `codec:"propagation"`
}
add support for tmpfs
2018-11-26 22:22:57 +00:00
type DockerTmpfsOptions struct {
SizeBytes int64 `codec:"size"`
Mode int `codec:"mode"`
}
Support docker bind mounts
2018-11-26 21:45:01 +00:00
// DockerVolumeDriverConfig holds a map of volume driver specific options
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
type DockerVolumeDriverConfig struct {
drivers: restore port_map old json support This ensures that `port_map` along with other block like attribute declarations (e.g. ulimit, labels, etc) can handle various hcl and json syntax that was supported in 0.8. In 0.8.7, the following declarations are effectively equivalent: ``` // hcl block port_map { http = 80 https = 443 } // hcl assignment port_map = { http = 80 https = 443 } // json single element array of map (default in API response) {"port_map": [{"http": 80, "https": 443}]} // json array of individual maps (supported accidentally iiuc) {"port_map: [{"http": 80}, {"https": 443}]} ``` We achieve compatbility by using `NewAttr("...", "list(map(string))", false)` to be serialized to a `map[string]string` wrapper, instead of using `BlockAttrs` declaration. The wrapper merges the list of maps automatically, to ease driver development. This approach is closer to how v0.8.7 implemented the fields [1][2], and despite its verbosity, seems to perserve 0.8.7 behavior in hcl2. This is only required for built-in types that have backward compatibility constraints. External drivers should use `BlockAttrs` instead, as they see fit. [1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216 [2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-13 17:55:48 +00:00
Name string `codec:"name"`
Options hclutils.MapStrStr `codec:"options"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
// ContainerGCConfig controls the behavior of the GC reconciler to detects
// dangling nomad containers that aren't tracked due to docker/nomad bugs
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
type ContainerGCConfig struct {
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
// Enabled controls whether container reconciler is enabled
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
Enabled bool `codec:"enabled"`
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
// DryRun indicates that reconciler should log unexpectedly running containers
// if found without actually killing them
DryRun bool `codec:"dry_run"`
// PeriodStr controls the frequency of scanning containers
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
PeriodStr string `codec:"period"`
period time.Duration `codec:"-"`
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
// CreationGraceStr is the duration allowed for a newly created container
// to live without being registered as a running task in nomad.
// A container is treated as leaked if it lived more than grace duration
// and haven't been registered in tasks.
CreationGraceStr string `codec:"creation_grace"`
CreationGrace time.Duration `codec:"-"`
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
}
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
type DriverConfig struct {
driver/docker: allow configurable pull context timeout setting. Pulling large docker containers can take longer than the default context timeout. Without a way to change this it is very hard for users to utilise Nomad properly without hacky work arounds. This change adds an optional pull_timeout config parameter which gives operators the possibility to account for increase pull times where needed. The infra docker image also has the option to set a custom timeout to keep consistency.
2020-08-12 07:58:07 +00:00
Endpoint string `codec:"endpoint"`
Auth AuthConfig `codec:"auth"`
TLS TLSConfig `codec:"tls"`
GC GCConfig `codec:"gc"`
Volumes VolumeConfig `codec:"volumes"`
AllowPrivileged bool `codec:"allow_privileged"`
AllowCaps []string `codec:"allow_caps"`
GPURuntimeName string `codec:"nvidia_runtime"`
InfraImage string `codec:"infra_image"`
InfraImagePullTimeout string `codec:"infra_image_pull_timeout"`
infraImagePullTimeoutDuration time.Duration `codec:"-"`
DisableLogCollection bool `codec:"disable_log_collection"`
PullActivityTimeout string `codec:"pull_activity_timeout"`
pullActivityTimeoutDuration time.Duration `codec:"-"`
driver/docker: add extra labels ( job name, task and task group name)
2021-03-08 13:59:52 +00:00
ExtraLabels []string `codec:"extra_labels"`
docker: support configuring default log driver in plugin options
2021-03-12 21:04:33 +00:00
Logging LoggingConfig `codec:"logging"`
Add a knob to restrict docker runtimes
2020-05-12 14:13:50 +00:00
use allow_runtimes for consistency Other allow lists use allow_ prefix (e.g. allow_caps, allow_privileged).
2020-05-12 15:03:08 +00:00
AllowRuntimesList []string `codec:"allow_runtimes"`
allowRuntimes map[string]struct{} `codec:"-"`
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
}
type AuthConfig struct {
Config string `codec:"config"`
Helper string `codec:"helper"`
docker: move recoverable error proto to shared structs
2018-11-14 11:20:35 +00:00
}
type TLSConfig struct {
Cert string `codec:"cert"`
Key string `codec:"key"`
CA string `codec:"ca"`
}
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
type GCConfig struct {
Image bool `codec:"image"`
ImageDelay string `codec:"image_delay"`
imageDelayDuration time.Duration `codec:"-"`
Container bool `codec:"container"`
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
DanglingContainers ContainerGCConfig `codec:"dangling_containers"`
docker: group common config into blocks
2018-11-20 02:07:30 +00:00
}
docker: move volume driver options to seperate block
2018-11-20 02:32:08 +00:00
type VolumeConfig struct {
Enabled bool `codec:"enabled"`
SelinuxLabel string `codec:"selinuxlabel"`
}
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
docker: support configuring default log driver in plugin options
2021-03-12 21:04:33 +00:00
type LoggingConfig struct {
Type string `codec:"type"`
Config map[string]string `codec:"config"`
}
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
func (d *Driver) PluginInfo() (*base.PluginInfoResponse, error) {
return pluginInfo, nil
}
func (d *Driver) ConfigSchema() (*hclspec.Spec, error) {
return configSpec, nil
}
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
const danglingContainersCreationGraceMinimum = 1 * time.Minute
Making pull activity timeout configurable in Docker * Making pull activity timeout configurable in Docker plugin config, first pass * Fixing broken function call * Fixing broken tests * Fixing linter suggestion * Adding documentation on new parameter in Docker plugin config * Adding unit test * Setting min value for pull_activity_timeout, making pull activity duration a private var
2019-12-18 11:58:53 +00:00
const pullActivityTimeoutMinimum = 1 * time.Minute
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
Add plugin API versioning to plugin loader and plugins
2018-12-18 00:40:58 +00:00
func (d *Driver) SetConfig(c *base.Config) error {
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
var config DriverConfig
Add plugin API versioning to plugin loader and plugins
2018-12-18 00:40:58 +00:00
if len(c.PluginConfig) != 0 {
if err := base.MsgPackDecode(c.PluginConfig, &config); err != nil {
return err
}
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
}
d.config = &config
prefer TrimPrefix to checking HasPrefix first
2021-01-22 13:36:09 +00:00
d.config.InfraImage = strings.TrimPrefix(d.config.InfraImage, "https://")
fix the inconsistency handling between infra image and normal task image
2020-12-09 03:33:34 +00:00
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
if len(d.config.GC.ImageDelay) > 0 {
dur, err := time.ParseDuration(d.config.GC.ImageDelay)
if err != nil {
return fmt.Errorf("failed to parse 'image_delay' duration: %v", err)
}
d.config.GC.imageDelayDuration = dur
}
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
if len(d.config.GC.DanglingContainers.PeriodStr) > 0 {
dur, err := time.ParseDuration(d.config.GC.DanglingContainers.PeriodStr)
if err != nil {
return fmt.Errorf("failed to parse 'period' duration: %v", err)
}
d.config.GC.DanglingContainers.period = dur
}
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
if len(d.config.GC.DanglingContainers.CreationGraceStr) > 0 {
dur, err := time.ParseDuration(d.config.GC.DanglingContainers.CreationGraceStr)
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
if err != nil {
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
return fmt.Errorf("failed to parse 'creation_grace' duration: %v", err)
}
if dur < danglingContainersCreationGraceMinimum {
return fmt.Errorf("creation_grace is less than minimum, %v", danglingContainersCreationGraceMinimum)
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
}
refactor reconciler code and address comments
2019-10-17 12:37:18 +00:00
d.config.GC.DanglingContainers.CreationGrace = dur
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
}
Making pull activity timeout configurable in Docker * Making pull activity timeout configurable in Docker plugin config, first pass * Fixing broken function call * Fixing broken tests * Fixing linter suggestion * Adding documentation on new parameter in Docker plugin config * Adding unit test * Setting min value for pull_activity_timeout, making pull activity duration a private var
2019-12-18 11:58:53 +00:00
if len(d.config.PullActivityTimeout) > 0 {
dur, err := time.ParseDuration(d.config.PullActivityTimeout)
if err != nil {
return fmt.Errorf("failed to parse 'pull_activity_timeout' duaration: %v", err)
}
if dur < pullActivityTimeoutMinimum {
return fmt.Errorf("pull_activity_timeout is less than minimum, %v", pullActivityTimeoutMinimum)
}
d.config.pullActivityTimeoutDuration = dur
}
driver/docker: allow configurable pull context timeout setting. Pulling large docker containers can take longer than the default context timeout. Without a way to change this it is very hard for users to utilise Nomad properly without hacky work arounds. This change adds an optional pull_timeout config parameter which gives operators the possibility to account for increase pull times where needed. The infra docker image also has the option to set a custom timeout to keep consistency.
2020-08-12 07:58:07 +00:00
if d.config.InfraImagePullTimeout != "" {
dur, err := time.ParseDuration(d.config.InfraImagePullTimeout)
if err != nil {
return fmt.Errorf("failed to parse 'infra_image_pull_timeout' duaration: %v", err)
}
d.config.infraImagePullTimeoutDuration = dur
}
use allow_runtimes for consistency Other allow lists use allow_ prefix (e.g. allow_caps, allow_privileged).
2020-05-12 15:03:08 +00:00
d.config.allowRuntimes = make(map[string]struct{}, len(d.config.AllowRuntimesList))
for _, r := range d.config.AllowRuntimesList {
d.config.allowRuntimes[r] = struct{}{}
Add a knob to restrict docker runtimes
2020-05-12 14:13:50 +00:00
}
Add plugin API versioning to plugin loader and plugins
2018-12-18 00:40:58 +00:00
if c.AgentConfig != nil {
d.clientConfig = c.AgentConfig.Driver
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
}
dockerClient, _, err := d.dockerClients()
if err != nil {
return fmt.Errorf("failed to get docker client: %v", err)
}
coordinatorConfig := &dockerCoordinatorConfig{
tests: don't delete images after tests complete Fix some docker test flakiness where image cleanup process may contaminate other tests. A clean up process may attempt to delete an image while it's used by another test.
2020-05-26 22:19:18 +00:00
ctx: d.ctx,
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
client: dockerClient,
cleanup: d.config.GC.Image,
logger: d.logger,
removeDelay: d.config.GC.imageDelayDuration,
}
docker: unexport new coordinator func
2018-11-20 04:07:07 +00:00
d.coordinator = newDockerCoordinator(coordinatorConfig)
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
Only start reconciler once in main driver driver.SetConfig is not appropriate for starting up reconciler goroutine. Some ephemeral driver instances are created for validating config and we ought not to side-effecting goroutines for those. We currently lack a lifecycle hook to inject these, so I picked the `Fingerprinter` function for now, and reconciler should only run after fingerprinter started. Use `sync.Once` to ensure that we only start reconciler loop once.
2019-10-18 18:35:15 +00:00
d.reconciler = newReconciler(d)
docker: periodically reconcile containers When running at scale, it's possible that Docker Engine starts containers successfully but gets wedged in a way where API call fails. The Docker Engine may remain unavailable for arbitrary long time. Here, we introduce a periodic reconcilation process that ensures that any container started by nomad is tracked, and killed if is running unexpectedly. Basically, the periodic job inspects any container that isn't tracked in its handlers. A creation grace period is used to prevent killing newly created containers that aren't registered yet. Also, we aim to avoid killing unrelated containters started by host or through raw_exec drivers. The logic is to pattern against containers environment variables and mounts to infer if they are an alloc docker container. Lastly, the periodic job can be disabled to avoid any interference if need be.
2019-09-13 15:24:58 +00:00
docker: move config RPCs to config.go
2018-11-20 03:41:14 +00:00
return nil
}
func (d *Driver) TaskConfigSchema() (*hclspec.Spec, error) {
return taskConfigSpec, nil
}
func (d *Driver) Capabilities() (*drivers.Capabilities, error) {
return capabilities, nil
}
driver: allow disabling log collection Operators commonly have docker logs aggregated using various tools and don't need nomad to manage their docker logs. Worse, Nomad uses a somewhat heavy docker api call to collect them and it seems to cause problems when a client runs hundreds of log collections. Here we add a knob to disable log aggregation completely for nomad. When log collection is disabled, we avoid running logmon and docker_logger for the docker tasks in this implementation. The downside here is once disabled, `nomad logs ...` commands and API no longer return logs and operators must corrolate alloc-ids with their aggregated log info. This is meant as a stop gap measure. Ideally, we'd follow up with at least two changes: First, we should optimize behavior when we can such that operators don't need to disable docker log collection. Potentially by reverting to using pre-0.9 syslog aggregation in linux environments, though with different trade-offs. Second, when/if logs are disabled, nomad logs endpoints should lookup docker logs api on demand. This ensures that the cost of log collection is paid sparingly.
2019-12-07 03:11:41 +00:00
var _ drivers.InternalCapabilitiesDriver = (*Driver)(nil)
func (d *Driver) InternalCapabilities() drivers.InternalCapabilities {
return drivers.InternalCapabilities{
DisableLogCollection: d.config != nil && d.config.DisableLogCollection,
}
}
Reference in a new issue Copy permalink