Commit graph

106 commits

Author SHA1 Message Date
Michael Schurter 32daa7b47b goimports until make check is happy 2019-01-23 06:27:14 -08:00
Michael Schurter be0bab7c3f move pluginutils -> helper/pluginutils
I wanted a different color bikeshed, so I get to paint it
2019-01-22 15:50:08 -08:00
Alex Dadgar b2c7268843 move reattach config 2019-01-22 15:11:58 -08:00
Alex Dadgar cdcd3c929c loader and singleton 2019-01-22 15:11:57 -08:00
Nick Ethier e3c6f89b9a
drivers: use consts for task handle version 2019-01-18 18:31:01 -05:00
Nick Ethier 6804450c69
cleanup code comments and small fixes from refactor 2019-01-18 18:31:01 -05:00
Mahmood Ali 5df63fda7c
Merge pull request #5190 from hashicorp/f-memory-usage
Track Basic Memory Usage as reported by cgroups
2019-01-18 16:46:02 -05:00
Danielle Tomlinson b65bf78513 docker: Fix missing import 2019-01-17 18:44:27 +01:00
Danielle Tomlinson 7fca934509 chore: General Cleanup 2019-01-17 18:43:14 +01:00
Danielle Tomlinson e73962d8d6 docker: Only run Cleanup test on unix os' 2019-01-17 18:43:14 +01:00
Danielle Tomlinson 30a5e25d94 fixup: Typo in docker test 2019-01-17 18:43:14 +01:00
Danielle Tomlinson 82018cd030 chore: Fix docklog linting 2019-01-17 18:43:14 +01:00
Danielle Tomlinson 3b2ff2005b chore: Fix docker test linting
Due to https://github.com/tsenart/deadcode/issues/3 we can't specify
these consts on their own. This moves them into the _platform_test.go
files to avoid creating a package that only exposes a couple of values.
2019-01-17 18:43:14 +01:00
Danielle Tomlinson de86435cf8 docker: Test cleanup for windows
* Docker for Windows does not support ulimits
* Use filepath.ToSlash to test workdir
* Convert expected mount paths to system style
* Skip security-opt test on windows
  - Windows does not support seccomp, and it's unclear which options are
    available.
* Skip StartN due to lack of sigint
* docker: Use api to get image info on windows
* No bridge on windows
* Stop hardcoding /bin/
2019-01-17 18:43:14 +01:00
Danielle Tomlinson 3d2f0bb7da docker: ExpandPath tests validate slashpath 2019-01-17 18:43:13 +01:00
Danielle Tomlinson 37190a5595 dockerlogger: Fix tests on windows
Uses the home directory and windows path expansion, as c:\tmp doesn't
necessarily exist, and mktemp would involve unnecessarily complicating
the commands.
2019-01-17 18:43:13 +01:00
Danielle Tomlinson e6c0738b65 Expand unix build definition 2019-01-17 18:43:13 +01:00
Preetha Appan 55319b05d1
clean up read access 2019-01-16 11:04:11 -06:00
Preetha Appan 469a286b1b
Refactor logging in drivers to use a tri-state boolean
Changes logging warnings/errors only if the state changes
from healthy to unhealthy
2019-01-16 10:19:31 -06:00
Preetha Appan 0c2c0a2d43
Make docker driver logging less redundant 2019-01-16 10:16:57 -06:00
oleksii.shyman e41fbf7577 Add support for docker runtimes
- docker fingerprint issues a docker api system info call to get the
  list of supported OCI runtimes.
  - OCI runtimes are reported as comma separated list of names
  - docker driver is aware of GPU runtime presence
  - docker driver throws an error when user tries to run container with
  GPU, when GPU runtime is not present
  - docker GPU runtime name is configurable
2019-01-15 11:34:47 -08:00
Danielle Tomlinson 272a8726d7 docker: Terminate dockerlogger
Previously, we did not attempt to stop Docker Logger processes until
DestroyTask, which means that under many circumstances, we will never
successfully close the plugin client.

This commit terminates the plugin process when `run` terminates, or when
`DestroyTask` is called.

Steps to repro:

```
$ nomad agent -dev
$ nomad init
$ nomad run example.nomad
$ nomad stop example
$ ps aux | grep nomad # See docker logger process running
$ signal the dev agent
$ ps aux | grep nomad # See docker logger process running
```
2019-01-15 14:58:05 +01:00
Alex Dadgar 471fdb3ccf
Merge pull request #5173 from hashicorp/b-log-levels
Plugins use parent loggers
2019-01-14 16:14:30 -08:00
Mahmood Ali 9909d98bee Track Basic Memory Usage as reported by cgroups
Track current memory usage, `memory.usage_in_bytes`, in addition to
`memory.max_memory_usage_in_bytes` and friends.  This number is closer
what Docker reports.

Related to https://github.com/hashicorp/nomad/issues/5165 .
2019-01-14 18:47:52 -05:00
Nick Ethier 7e306afde3
executor: fix failing stats related test 2019-01-12 12:18:23 -05:00
Nick Ethier b0d9440474
docker: add test for stats collection 2019-01-12 12:18:22 -05:00
Nick Ethier 9fea54e0dc
executor: implement streaming stats API
plugins/driver: update driver interface to support streaming stats

client/tr: use streaming stats api

TODO:
 * how to handle errors and closed channel during stats streaming
 * prevent tight loop if Stats(ctx) returns an error

drivers: update drivers TaskStats RPC to handle streaming results

executor: better error handling in stats rpc

docker: better control and error handling of stats rpc

driver: allow stats to return a recoverable error
2019-01-12 12:18:22 -05:00
Alex Dadgar 14ed757a56 Plugins use parent loggers
This PR fixes various instances of plugins being launched without using
the parent loggers. This meant that logs would not all go to the same
output, break formatting etc.
2019-01-11 11:36:37 -08:00
Mahmood Ali 5389ebae41
Merge pull request #5166 from hashicorp/b-docker-tests-mac
tests: run docker tests in macOS out of box
2019-01-09 13:07:37 -05:00
Mahmood Ali 90f3cea187
Merge pull request #5157 from hashicorp/r-drivers-no-cstructs
drivers: avoid referencing client/structs package
2019-01-09 13:06:46 -05:00
Mahmood Ali 4952f2a182
Merge pull request #5159 from hashicorp/r-macos-tests
Fix Travis MacOS job
2019-01-09 08:22:30 -05:00
Mahmood Ali c78ed7246f tests: run docker tests in macOS out of box
Use `/tmp` as temporary directory for docker driver tests, so tests can
run out of the box without any intervention.

macOS sets tempdir as `/var`, which Docker does not whitelist as a path
that can be bind-mounted.
2019-01-08 14:35:40 -05:00
Mahmood Ali 64f80343fc drivers: re-export ResourceUsage structs
Re-export the ResourceUsage structs in drivers package to avoid drivers
directly depending on the internal client/structs package directly.

I attempted moving the structs to drivers, but that caused some import
cycles that was a bit hard to disentagle.  Alternatively, I added an
alias here that's sufficient for our purposes of avoiding external
drivers depend on internal packages, while allowing us to restructure
packages in future without breaking source compatibility.
2019-01-08 09:11:47 -05:00
Mahmood Ali 916a40bb9e move cstructs.DeviceNetwork to drivers pkg 2019-01-08 09:11:47 -05:00
Mahmood Ali 9369b123de use drivers.FSIsolation 2019-01-08 09:11:47 -05:00
Danielle Tomlinson a9b9ad34dc drivers: Implement InternalPluginDriver interface
This implements the InternalPluginDriver interface in each driver, and
calls the cancellation fn for their respective eventers.

This fixes a per task goroutine leak during test suite execution.
2019-01-08 13:49:31 +01:00
Alex Dadgar 0106f23aaa Review comments 2019-01-07 14:50:28 -08:00
Alex Dadgar 6c6e035dba add docker logger to separate main 2019-01-07 14:49:40 -08:00
Alex Dadgar a6b36df4de remove nil logger 2019-01-07 14:48:01 -08:00
Mahmood Ali 0ba7b0c132 tests: helper function for checking docker presense 2019-01-07 08:27:06 -05:00
Mahmood Ali 796d625ab6 Skip tests requiring Docker deamon if not found. 2019-01-07 07:59:13 -05:00
Preetha Appan 2fb2de3cef
Standardize driver health description messages for all drivers 2019-01-06 22:06:38 -06:00
Nick Ethier ce1a5cba0e
drivermanager: use allocID and task name to route task events 2018-12-18 23:01:51 -05:00
Alex Dadgar bc55ec81b5 fix docker launching plugins 2018-12-18 16:48:01 -08:00
Alex Dadgar 4c57d2ec4d Add plugin API versioning to plugin loader and plugins 2018-12-18 16:48:00 -08:00
Alex Dadgar 327b551b39 Drivers 2018-12-18 15:50:11 -08:00
Danielle Tomlinson b61da13c20 docker: Delete Task on Destroy
Currently the docker driver does not remove tasks from its state map
when destroying the task, which leads to issues when restarting tasks in
place, and leaks expired handles over time.
2018-12-18 15:53:31 +01:00
Mahmood Ali 168749ffd1
Merge pull request #5008 from hashicorp/b-docker-test-20181214
Fix flakiness in docker tests
2018-12-15 16:03:00 -05:00
Mahmood Ali e4f44b9be5 testes: remove TestDockerDriver_Kill
We already have two other Kill tests (e.g.
TestDockerDriver_Start_Kill_Wait and
TestDockerDriver_Start_KillTimeout), so don't need yet another flaky
test.
2018-12-15 15:03:56 -05:00
Mahmood Ali 990a7d6776 driver/docker: stopping a dead container not error 2018-12-15 15:03:56 -05:00