Commit graph

35 commits

Author SHA1 Message Date
Nick Ethier 729dd9018c
docker: set default cpu cfs period (#6737)
* docker: set default cpu cfs period

Co-Authored-By: Michael Schurter <mschurter@hashicorp.com>
2019-11-19 19:05:15 -05:00
Mahmood Ali 977b86f924 driver/docker: ensure that defaults are populated
Looks like we may need to pass default literal at each layer to be able,
so defaults are set properly.
2019-10-18 18:27:28 -04:00
Mahmood Ali 3aec7b56ea Only start reconciler once in main driver
driver.SetConfig is not appropriate for starting up reconciler
goroutine.  Some ephemeral driver instances are created for validating
config and we ought not to side-effecting goroutines for those.

We currently lack a lifecycle hook to inject these, so I picked the
`Fingerprinter` function for now, and reconciler should only run after
fingerprinter started.

Use `sync.Once` to ensure that we only start reconciler loop once.
2019-10-18 14:43:23 -04:00
Mahmood Ali 8739cc2a62 refactor reconciler code and address comments 2019-10-17 09:42:23 -04:00
Mahmood Ali aa59280edc docker: periodically reconcile containers
When running at scale, it's possible that Docker Engine starts
containers successfully but gets wedged in a way where API call fails.
The Docker Engine may remain unavailable for arbitrary long time.

Here, we introduce a periodic reconcilation process that ensures that any
container started by nomad is tracked, and killed if is running
unexpectedly.

Basically, the periodic job inspects any container that isn't tracked in
its handlers.  A creation grace period is used to prevent killing newly
created containers that aren't registered yet.

Also, we aim to avoid killing unrelated containters started by host or
through raw_exec drivers.  The logic is to pattern against containers
environment variables and mounts to infer if they are an alloc docker
container.

Lastly, the periodic job can be disabled to avoid any interference if
need be.
2019-10-17 08:36:01 -04:00
Danielle Lancashire 724586ba1d
docker: Fix driver spec
hclspec.NewLiteral does not quote its values, which caused `3m` to be
parsed as a nonsensical literal which broke the plugin loader during
initialization. By quoting the value here, it starts correctly.
2019-09-03 08:53:37 +02:00
Zhiguang Wang 832df1091b Add default value "3m" to image_delay, making it consistent with docs. 2019-09-02 16:40:00 +08:00
Nick Ethier 1dae42ab81
docker: allow configuration of infra image 2019-07-31 01:04:07 -04:00
Nick Ethier 1fc5f86a7c
docker: support shared network namespaces 2019-07-31 01:03:20 -04:00
Mahmood Ali 104869c0e1 drivers/docker: rename logging type to driver
Docker uses the term logging `driver` in its public documentations: in
`docker` daemon config[1], `docker run` arguments [2] and in docker compose file[3].
Interestingly, docker used `type` in its API [4] instead of everywhere
else.

It's unfortunate that Nomad used `type` modeling after the Docker API
rather than the user facing documents.  Nomad using `type` feels very
non-user friendly as it's disconnected from how Docker markets the flag
and shows internal representation instead.

Here, we rectify the situation by introducing `driver` field and
prefering it over `type` in logging.

[1] https://docs.docker.com/config/containers/logging/configure/
[2] https://docs.docker.com/engine/reference/run/#logging-drivers---log-driver
[3] https://docs.docker.com/compose/compose-file/#logging
[4] https://docs.docker.com/engine/api/v1.39/#operation/ContainerCreate
2019-02-28 16:04:03 -05:00
Mahmood Ali 4def8529db driver/docker: use BlockAttrs for storage_opts
storage_opts is a new field in 0.9 cycle and doesn't have backward
compatibility constraints.
2019-02-19 20:35:28 -05:00
Mahmood Ali 46cd3c3f55 drivers: restore port_map old json support
This ensures that `port_map` along with other block like attribute
declarations (e.g. ulimit, labels, etc) can handle various hcl and json
syntax that was supported in 0.8.

In 0.8.7, the following declarations are effectively equivalent:

```
// hcl block
port_map {
  http = 80
  https = 443
}

// hcl assignment
port_map = {
  http  = 80
  https = 443
}

// json single element array of map (default in API response)
{"port_map": [{"http": 80, "https": 443}]}

// json array of individual maps (supported accidentally iiuc)
{"port_map: [{"http": 80}, {"https": 443}]}
```

We achieve compatbility by using `NewAttr("...", "list(map(string))",
false)` to be serialized to a `map[string]string` wrapper, instead of using
`BlockAttrs` declaration.  The wrapper merges the list of maps
automatically, to ease driver development.

This approach is closer to how v0.8.7 implemented the fields [1][2], and
despite its verbosity, seems to perserve 0.8.7 behavior in hcl2.

This is only required for built-in types that have backward
compatibility constraints.  External drivers should use `BlockAttrs`
instead, as they see fit.

[1] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L216
[2] https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L698-L700
2019-02-16 11:37:33 -05:00
Mahmood Ali f7102cd01d
tests: add hcl task driver config parsing tests (#5314)
* drivers: add config parsing tests

Add basic tests for parsing and encoding task config.

* drivers/docker: fix some config declarations

* refactor and document config parse helpers
2019-02-12 14:46:37 -05:00
Michael Schurter 3b84e08fa4
Merge pull request #5297 from hashicorp/b-docker-logging
Docker: Fix logging config parsing
2019-02-11 06:57:52 -08:00
Gertjan Roggemans 94ca78354b docker: Fix volume driver_config options spec (#5309)
Fixes #5308
2019-02-11 09:18:44 -05:00
Michael Schurter e1e4b10884 docker: fix logging config parsing
Fixes
https://groups.google.com/d/topic/nomad-tool/B3Uo6Kns2BI/discussion
2019-02-04 11:07:57 -08:00
Michael Schurter 32daa7b47b goimports until make check is happy 2019-01-23 06:27:14 -08:00
Michael Schurter be0bab7c3f move pluginutils -> helper/pluginutils
I wanted a different color bikeshed, so I get to paint it
2019-01-22 15:50:08 -08:00
Alex Dadgar cdcd3c929c loader and singleton 2019-01-22 15:11:57 -08:00
oleksii.shyman e41fbf7577 Add support for docker runtimes
- docker fingerprint issues a docker api system info call to get the
  list of supported OCI runtimes.
  - OCI runtimes are reported as comma separated list of names
  - docker driver is aware of GPU runtime presence
  - docker driver throws an error when user tries to run container with
  GPU, when GPU runtime is not present
  - docker GPU runtime name is configurable
2019-01-15 11:34:47 -08:00
Nick Ethier 7e306afde3
executor: fix failing stats related test 2019-01-12 12:18:23 -05:00
Mahmood Ali 9369b123de use drivers.FSIsolation 2019-01-08 09:11:47 -05:00
Alex Dadgar 4c57d2ec4d Add plugin API versioning to plugin loader and plugins 2018-12-18 16:48:00 -08:00
Mahmood Ali 990a7d6776 driver/docker: stopping a dead container not error 2018-12-15 15:03:56 -05:00
Mahmood Ali a580cef986 refactor device manipulation 2018-12-04 20:55:59 -05:00
Mahmood Ali f6d6a50c39 add support for tmpfs 2018-11-27 07:20:17 -05:00
Mahmood Ali 0a09f5521d Support docker bind mounts 2018-11-27 07:20:17 -05:00
Mahmood Ali e9e415f186 Add support for storage opt 2018-11-20 16:11:02 -05:00
Nick Ethier 3ccd359735
docker: unexport new coordinator func 2018-11-19 23:07:07 -05:00
Nick Ethier 8b9b2b476e
docker: add default blocks for driver plugin config schema 2018-11-19 22:59:18 -05:00
Nick Ethier 2667f48a5d
docker: move config RPCs to config.go 2018-11-19 22:59:18 -05:00
Nick Ethier ce4b867d21
docker: move volume driver options to seperate block 2018-11-19 22:59:18 -05:00
Nick Ethier fca2df3c79
docker: group common config into blocks 2018-11-19 22:59:17 -05:00
Nick Ethier 8ef73e63ce
docker: moved fingerprint code to it's own file 2018-11-19 22:59:17 -05:00
Nick Ethier ced5d5c445
docker: move recoverable error proto to shared structs 2018-11-19 22:59:16 -05:00