47df1dde10
Merge branch 'master' into f-grpc-executor
2018-12-06 21:42:38 -05:00
19a695308f
executor: fix tests
2018-12-06 21:39:53 -05:00
913efed9f5
executor: fix broken non-linux build
2018-12-06 21:33:20 -05:00
2283cb2c39
executor: use drivers.Resources as resource model
2018-12-06 21:22:02 -05:00
29ef54c0ee
executor: merge plugin shim with executor package
2018-12-06 21:13:45 -05:00
71353a88d4
executor: remove structs package
2018-12-06 20:54:14 -05:00
1e3c3cb287
Deprecate IOPS
...
IOPS have been modelled as a resource since Nomad 0.1 but has never
actually been detected and there is no plan in the short term to add
detection. This is because IOPS is a bit simplistic of a unit to define
the performance requirements from the underlying storage system. In its
current state it adds unnecessary confusion and can be removed without
impacting any users. This PR leaves IOPS defined at the jobspec parsing
level and in the api/ resources since these are the two public uses of
the field. These should be considered deprecated and only exist to allow
users to stop using them during the Nomad 0.9.x release. In the future,
there should be no expectation that the field will exist.
2018-12-06 15:09:26 -08:00
a7b205daf2
Merge pull request #4955 from hashicorp/fix-docker-tests-20181203
...
Fix docker driver tests
2018-12-06 16:41:33 -05:00
bdc53b1d8e
driver/rkt: mount plugin devices
2018-12-06 15:46:35 -05:00
2c0fd2a902
driver/lxc: mount plugin devices
...
Also, LXC requires target paths to be relative. Container paths in LXC
binds should never be absolute paths, so we strip any preceeding `/`,
even if a user sets one.
2018-12-06 15:46:35 -05:00
699875eb1c
fixup: add missed docker utils test
2018-12-06 15:46:35 -05:00
e9557ae596
tests: ensure image is loaded as test setup
2018-12-06 15:36:43 -05:00
57ffece7f8
executor: update test references
2018-12-05 11:07:48 -05:00
02f4b0fac5
executor: update driver references
2018-12-05 11:04:18 -05:00
8b20de4801
executor: use grpc instead of netrpc as plugin protocol
...
* Added protobuf spec for executor
* Seperated executor structs into their own package
2018-12-05 11:03:56 -05:00
b55fb642f1
driver/docker: honor plugin devices
2018-12-04 21:31:28 -05:00
a580cef986
refactor device manipulation
2018-12-04 20:55:59 -05:00
3a18105d06
drivers/exec: refactor stop/kill tests
...
Simplify the tests to do all assertions within the main goroutine and
account for status propagation delay.
2018-12-04 20:34:43 -05:00
428d35a5a9
executor: Keep 0.8.6 exit code for wait() failures
...
0.8.6 uses exit code 1 when `proc.Wait()` fails: https://github.com/hashicorp/nomad/blob/v0.8.6/client/driver/executor/executor.go#L442
2018-12-04 19:38:25 -05:00
8df9de6fd5
driver/rkt: use rkt environment
...
The rkt command itself needs an environment with PATH set to find iptables.
2018-12-04 14:00:45 -05:00
06a5cadf35
drivers/rkt: use image isolation for rkt
2018-12-04 11:40:10 -05:00
178365848e
tests: don't assert in WaitForResult
...
WaitForResult expects body to fail and retries few times before giving
up. Assertions inside the testfn body causes it to terminate abruptly
without retrying.
2018-12-04 11:40:10 -05:00
f8ceeebf11
no t.Parallel() in excutor table driven tests ( #4948 )
...
When `t.Parallel()` is used inside a `t.Run()` sub-set, the closure
doesn't behave as expected, and some cases effectively get skipped.
More details can be found in
https://gist.github.com/posener/92a55c4cd441fc5e5e85f27bca008721
2018-12-04 09:04:04 -05:00
216a2566c7
Update LXC with drivers/testutils changes ( #4951 )
2018-12-04 08:57:54 -05:00
c88e3723eb
Fix docker tests
...
Some tests have containers that die almost immediately, and may die
and cleaned up before `driver.WaitUntilStarted` runs.
The causes for container dying seems special for each test:
* TestDockerDriver_Cleanup: `hello-world` image just emits a message and exits immediately
* TestDockerDriver_ForcePull_RepoDigest: the busybox image in `TestDockerDriver_ForcePull_RepoDigest` test didn't support `-p 0` argument
* TestDockerDriver_Entrypoint: with the entrypoint being `/bin/sh -c`, the command needs to be the entire string; otherwise, it ignores the comments
2018-12-03 23:08:52 -05:00
2516cb16b9
Kill all container processes on shutdown
...
Currently, libcontainer-based executor, upon shutdown, kills the
container initial process. The children of the killed process remain
running, and the executor is never marked as terminated until they do.
Also, fix a case where we treat processes as successful, when
`proc.Wait()` fails. In some attempts, I was getting "waitid no child
processes" errors and such error shouldn't get process to be considered
successful.
2018-12-03 20:40:49 -05:00
bd8e4f1c15
Test Stopping a multi-process exec
...
Ensure that exec children processes get killed as well.
2018-12-03 20:40:19 -05:00
10b3e68a6d
Merge pull request #4925 from hashicorp/f-driver-plugins-dani
...
Third Party Driver Plugins Support
2018-12-03 20:48:19 +01:00
88622b97bd
libcontainer to manage /dev and /proc ( #4945 )
...
libcontainer already manages `/dev`, overriding task_dir - so let's use it for `/proc` as well and remove deadcode.
2018-12-03 10:41:01 -05:00
393b76ed7f
plugins: Move driver testing support to subpackage
...
this allows us to drop a cyclical import, but is subobptimal as it
requires BaseDriver tests to move. This falls firmly into the realm of
being a hack. Alternatives welcome.
2018-12-01 17:29:39 +01:00
66c521ca17
client: Move fingerprint structs to pkg
...
This removes a cyclical dependency when importing client/structs from
dependencies of the plugin_loader, specifically, drivers. Due to
client/config also depending on the plugin_loader.
It also better reflects the ownership of fingerprint structs, as they
are fairly internal to the fingerprint manager.
2018-12-01 17:10:39 +01:00
51a9f7369e
Merge pull request #4936 from hashicorp/f-legacy-refactor
...
Refactor and repackage client/driver
2018-11-30 13:38:06 +01:00
84e04cfa40
Merge pull request #4926 from hashicorp/f-docker-image-ref
...
Use user provided image name to launch container
2018-11-30 07:27:39 -05:00
94d43b8003
Merge pull request #4924 from hashicorp/f-docker-mounts
...
Support bind and tmpfs docker mounts
2018-11-30 07:27:17 -05:00
2db5ae38d8
client: Rename drivers/shared/env => client/taskenv
2018-11-30 12:18:39 +01:00
f3a77b8084
client: Merge driver/shared/structs and client/structs
2018-11-30 10:56:45 +01:00
fdfe93aa25
fixup: executorplugin: fix rkt build
2018-11-30 10:47:08 +01:00
04c8851b4c
client: Migrate DriverStats optout to drivers/shared/structs
2018-11-30 10:46:13 +01:00
d582ea1d8b
drivers: Create drivers/shared/structs
...
This creates a drivers/shared/structs package and moves the buffer size
checks into it.
2018-11-30 10:46:13 +01:00
0544a57abe
drivers: Move client/drivers/executor to drivers/shared/executor
2018-11-30 10:46:13 +01:00
1a29811169
drivers: Move client/drivers/env to drivers/shared/env
...
As part of deprecating legacy drivers, we're moving the env package to a
new drivers/shared tree, as it is used by the modern docker and rkt
driver packages, and is useful for 3rd party plugins.
2018-11-30 10:46:13 +01:00
0d90ba392e
Fix lxc test panic
2018-11-28 13:56:17 -06:00
924f1b69e9
Fix failing lxc test
2018-11-28 11:05:35 -06:00
bf58c65ef7
Fix LXC driver fingerprint to use typedattributes
2018-11-28 10:09:10 -06:00
9f4439243b
Fix docker driver to use new fingerprint typed attributes
2018-11-28 10:01:03 -06:00
f89dbcd9cc
modify fingerprint interface to use typed attribute struct
2018-11-28 10:01:03 -06:00
9af8deabbf
address review comments
2018-11-27 21:40:43 -05:00
6d34d2fade
Add Driver Plugin for LXC
2018-11-27 21:40:43 -05:00
e565c63eed
gofmt -s -w drivers/rkt/driver_test.go
2018-11-27 17:24:23 -08:00
844fd47acc
Use user provided image name to launch container
...
This allows the container to be tagged with a user friendly image name
(e.g. `redis:3.2`) rather than the image ID (e.g.
`sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916`).
Useful for human debugging, as well as some debugging and image scanning
tools.
This risks two bad changes:
1. Discrepancy in image resolution between docker and Nomad's image
loader.
* I checked the image creation paths in Nomad, and noticed that we
either pulled the image or inspect the image with the user provided
name.
2. A race in image tagging where the tag is modified between image
loading and container creation.
* I, personally, don't think this case is cause for concern, as it is
analogous to the task running a bit later. As long as the image is
still present, creating the container should be good.
2018-11-27 16:12:15 -05:00
f6d6a50c39
add support for tmpfs
2018-11-27 07:20:17 -05:00
0a09f5521d
Support docker bind mounts
2018-11-27 07:20:17 -05:00
3dd6ba514a
drivers/rkt: updated test to include new AllocID field in TaskConfig
2018-11-26 21:37:58 +00:00
9bd4317139
modified TaskConfig to include AllocID
...
use this for volume names in drivers/rkt to address #1150
2018-11-26 18:54:26 +00:00
141092e46d
Formatting and typo fixes
2018-11-25 11:53:21 -05:00
c61d99b525
Merge pull request #4908 from hashicorp/f-docker-opts-storageopt
...
Add support for docker storage options
2018-11-20 21:08:27 -05:00
1f3fe02e62
docker: sync access to exit result within a handle
2018-11-20 20:41:32 -05:00
2275153875
Apply suggestions from code review
...
Co-Authored-By: nickethier <ncethier@gmail.com>
2018-11-20 20:33:31 -05:00
e9e415f186
Add support for storage opt
2018-11-20 16:11:02 -05:00
3ccd359735
docker: unexport new coordinator func
2018-11-19 23:07:07 -05:00
8b9b2b476e
docker: add default blocks for driver plugin config schema
2018-11-19 22:59:18 -05:00
2667f48a5d
docker: move config RPCs to config.go
2018-11-19 22:59:18 -05:00
aa9f45ae47
docker: fix tests
2018-11-19 22:59:18 -05:00
0f03e8f520
docker: remove container pointer from task handle
2018-11-19 22:59:18 -05:00
ce4b867d21
docker: move volume driver options to seperate block
2018-11-19 22:59:18 -05:00
fca2df3c79
docker: group common config into blocks
2018-11-19 22:59:17 -05:00
813341dd59
Apply suggestions from code review
...
Co-Authored-By: nickethier <ncethier@gmail.com>
2018-11-19 22:59:17 -05:00
b7bd36db30
docker: remove global pull coordinator
2018-11-19 22:59:17 -05:00
f0a86859a0
docker: remove call to global metrics instance
2018-11-19 22:59:17 -05:00
8ef73e63ce
docker: moved fingerprint code to it's own file
2018-11-19 22:59:17 -05:00
4be8a86ef9
plugins/driver: remove NodeResources from task Resources and use PercentTicks field for docker driver
2018-11-19 22:59:17 -05:00
ced5d5c445
docker: move recoverable error proto to shared structs
2018-11-19 22:59:16 -05:00
585e468085
docker: implement recover task logic
2018-11-19 22:59:16 -05:00
ee51cb6a93
docker: finished porting tests
2018-11-19 22:59:16 -05:00
3d7cdea19e
drivers/docker: more work porting tests from old driver plugin
2018-11-19 22:59:16 -05:00
8f8698b3e1
docker: started work on porting docker driver to new plugin framework
2018-11-19 22:59:15 -05:00
9e64ce7d73
docker: properly launch docker logger process
2018-11-19 22:59:12 -05:00
f8cdd561f0
client: interpolate driver configurations
...
Also add missing SetDriverNetwork calls.
2018-11-15 16:25:57 -08:00
1f4db08f42
Regenerate proto files with protoc-gen-go@v1.2.0
2018-11-14 09:01:26 -05:00
1e92161f14
Merge pull request #4858 from hashicorp/b-fix-master-20181109
...
Fix some tests in master
2018-11-13 16:08:26 -05:00
17e8446484
Merge pull request #4868 from hashicorp/b-plugin-ctx
...
Plugin client's handle plugin dying
2018-11-13 10:26:53 -08:00
356c194acc
Use materialized duration fields for driver config
2018-11-13 10:21:40 -05:00
470d20cdf3
Avoid downloading image if present locally
2018-11-13 10:21:40 -05:00
7690f389a0
Prioritize checking consumer context cancellation
...
Tests expect that as soon as eventer shuts down immediately on context
cancellations; but golang does not guarantee priority when multiple
pending channels are ready in a select statement.
2018-11-13 10:21:40 -05:00
e9067e52b4
pull alpine image needed for test
...
The test requires the image to be present locally, so importing it as
part of setup.
2018-11-13 10:21:40 -05:00
8923ea4663
Handle time.Duration in mock
...
Mock driver config uses `time.Duration` fields but we initialize them
inconsistently, as time.Duration sometimes and as duration strings other
times. Previously, `mapstructure` handles it and does the right thing.
This is no longer the case with MsgPack. I could not find a good way to
bring back old behavior without too much complexity. `MsgPack` extended
types weren't ideal here as we lose type information (e.g. int64 vs
string), and the input is a generic map and not a MsgPack serialization
of duration.
As such, I went with the simple solution of declaring the config field
as duration string, and panicing if the test doesn't pass a valid
string.
I found this to cause the smallest change in tests, but we can
alternatively force all to be int64 instead.
2018-11-13 10:21:40 -05:00
fb56dd699d
distinguish java driver tests from others
2018-11-13 10:21:40 -05:00
343df28165
Fix java driver tests
2018-11-13 10:21:40 -05:00
e506ebbc24
Merge pull request #4845 from hashicorp/r-exec-refactor
...
Update exec driver to match rawexec
2018-11-12 20:59:32 -05:00
693f244cce
Plugin client's handle plugin dying
...
This PR plumbs the plugins done ctx through the base and driver plugin
clients (device already had it). Further, it adds generic handling of
gRPC stream errors.
2018-11-12 17:09:27 -08:00
880e5015f2
rawexec: Only use cgroups when running as root.
...
If Nomad is not running as root, we should not try to use cgroups for pid
freezing.
This originally was implemented pre-driver-support in
https://github.com/hashicorp/nomad/blob/v0.8.6/client/driver/raw_exec.go#L120-L130
2018-11-10 06:45:11 -08:00
9c76a7d9b8
handle nil resources limits
...
Config may not express resource constraints on the job!
2018-11-09 15:52:44 -05:00
1ed53ca478
shorter health descriptions
2018-11-08 10:00:49 -08:00
032f86bc78
Add a helper functions for checking unix root
2018-11-08 10:00:49 -08:00
21d126cd68
Update exec driver to match rawexec
...
Update exec driver with latest patterns used by rawexec, and eliminate
accidental diffs.
2018-11-08 10:00:49 -08:00
a7ca737fb6
review comments
2018-11-07 11:31:52 -08:00
204ca8230c
Device manager
...
Introduce a device manager that manages the lifecycle of device plugins
on the client. It fingerprints, collects stats, and forwards Reserve
requests to the correct plugin. The manager, also handles device plugins
failing and validates their output.
2018-11-07 10:43:15 -08:00
32283b2a66
reconcile java with latest patterns
2018-11-06 12:49:29 -08:00
467360954a
driver/java: support non-linux targets
...
Java driver supports non-linux targets, so LinuxResources fields isn't
guaranteed to be not-nil.
2018-11-06 12:41:39 -08:00
e64ef839b8
Reconcile with latest master patterns
2018-11-06 12:41:39 -08:00
c3881bd8d1
add java driver tests
2018-11-06 12:41:39 -08:00
4f525d0a17
Implement java driver
2018-11-06 12:41:39 -08:00
392d548b85
Merge pull request #4828 from hashicorp/b-restore
...
Implement client agent restarting
2018-11-05 18:50:15 -06:00
2bbd88888c
client: first pass at implementing task restoring
...
Task restoring works but dead tasks may be restarted
2018-11-05 12:32:05 -08:00
a17521475d
Merge pull request #4826 from hashicorp/b-driver-exec-tweaks-20181031
...
Register exec driver plugin among some fixes
2018-11-02 10:11:05 -04:00
dc0fc442ab
add plugin config
2018-11-02 08:51:23 -04:00
83db3e6a50
rkt: standardize names/code
2018-11-01 15:53:07 -07:00
1a7fb51285
rawexec: standardize names/code
2018-11-01 15:53:07 -07:00
5a5d52023d
qemu: standardize names/code
2018-11-01 15:53:07 -07:00
3c5e606ef6
drivers/mock: standardize names/code
2018-10-31 11:52:51 -07:00
9a45837c4e
Register exec driver plugin among some fixes
...
Namely, remove the `enabled` configuration flag, as it's specific to
`raw_exec` driver. Also, pass resource limits to underlying call.
2018-10-31 13:55:59 -04:00
d3dafc9761
mock_driver: match other's fingerprint message
2018-10-30 17:38:23 -07:00
2a38f575c0
rkt: fix race, register by default, fix fingerprint
2018-10-30 17:37:00 -07:00
5b0b75e1ee
qemu: fix fingerprint description and handle race
2018-10-30 17:34:08 -07:00
6f4a28a7c2
rawexec: fix fingerprint results and tests
...
In tests:
* always cleanup (Kill) the harness
* don't call SetConfig more than once (a race)
* avoid cgroups to allow running without being root
2018-10-30 17:16:22 -07:00
616801417c
rawexec: fix race around h.procState
2018-10-30 17:15:47 -07:00
9fa53565ad
drivers: remove stutter from exported driver names
...
Also fix a comment on the logger that got copy/pasted around.
2018-10-30 14:05:31 -07:00
fc320906aa
qemu: fix build and register by default
...
API changes had broken the build.
2018-10-29 22:10:37 -07:00
3fcf8ba7e6
Merge pull request #4795 from hashicorp/f-plugin-config
...
Pass client configuration to plugins through loader
2018-10-29 18:42:27 -07:00
bda3b1d3b3
rename NomadConfig to ClientAgentConfig
2018-10-29 21:34:34 -04:00
68991d8acb
Merge branch 'f-driver-plugin-qemu' of github.com:hashicorp/nomad into f-driver-plugin-qemu
2018-10-25 13:51:58 -05:00
bfff771416
remove driver config struct from qemu driver since its not used
2018-10-25 13:51:06 -05:00
48d25dbef1
Fix hclog log line formatting
...
Co-Authored-By: preetapan <preetha@hashicorp.com>
2018-10-25 13:47:17 -05:00
95a1637741
Fix imports
2018-10-25 13:45:44 -05:00
ca151a7828
Fix copy pasted comment
...
Co-Authored-By: preetapan <preetha@hashicorp.com>
2018-10-25 13:43:29 -05:00
c85b90f1e4
remove newline
...
Co-Authored-By: preetapan <preetha@hashicorp.com>
2018-10-25 13:42:20 -05:00
66c15e3a67
update comment
...
Co-Authored-By: preetapan <preetha@hashicorp.com>
2018-10-25 13:41:24 -05:00
9ca6c998c7
Remove unnecessary return
...
Co-Authored-By: preetapan <preetha@hashicorp.com>
2018-10-25 13:40:28 -05:00
fc7ba5e79d
remove unnecessary return
...
Co-Authored-By: preetapan <preetha@hashicorp.com>
2018-10-25 13:40:15 -05:00
9916b9e0ce
drivers: update SetConfig to handle driver client config
2018-10-18 23:32:17 -04:00
af3e9adf2f
Update drivers/qemu/driver.go
2018-10-17 17:42:01 -05:00
93eb048b87
QEMU driver plugin
2018-10-17 11:43:15 -05:00
4b694b2dec
Merge pull request #4793 from hashicorp/b-no-cty
...
Do not use cty in drivers
2018-10-16 20:04:41 -07:00
823a6c8d6c
update rkt and exec setconfig funcs
2018-10-16 23:00:26 -04:00
65adb80ebf
plumb NomadConfig into plugins
2018-10-16 22:47:22 -04:00
b657c3909a
drivers/exec: better fingerprinting and detection on non linux
2018-10-16 22:45:33 -04:00
d94b631b6b
drivers/exec: add exec implementation
2018-10-16 22:45:28 -04:00
88c1885044
Do not use cty in drivers
2018-10-16 17:17:07 -07:00
838ddf4d4a
fix linter errors
2018-10-16 16:56:57 -07:00
448a709603
driver/docker: rename container logger
2018-10-16 16:56:56 -07:00
64605a3957
drivers/mock: fix plugin name
...
Was mock_driver before plugins, so keep the name.
2018-10-16 16:56:56 -07:00
d634157158
Address review comments around logging task properly
2018-10-16 16:56:56 -07:00
3280727c4e
Address review comments
2018-10-16 16:56:56 -07:00
02b1f26d34
Switch back to using map[string]string for port map
2018-10-16 16:56:56 -07:00
ed5641055f
fixup comments, logging, and missing method impls
...
from #4777 comments
2018-10-16 16:56:56 -07:00
1a29337e48
register drivers by default
...
Do not register mock_driver on release builds.
2018-10-16 16:56:56 -07:00
118ce7c782
drivers/mock: complete plugin impl
2018-10-16 16:56:56 -07:00
9f564781ce
drivers/mock: start mock driver implementation
2018-10-16 16:56:56 -07:00
9b3e82a4e7
make port map a slice of maps to match existing rkt driver
2018-10-16 16:56:56 -07:00
4f4777d6a6
Review comments
2018-10-16 16:56:56 -07:00
edad5e9e6d
Stats collection test
2018-10-16 16:56:56 -07:00
678072ecd1
RKT driver plugin and unit tests
2018-10-16 16:56:56 -07:00
c7ac1186c9
client: add test for driverfailure during fingerprinting
2018-10-16 16:56:56 -07:00
15e8477c4e
rkt: start rkt driver plugin
2018-10-16 16:56:56 -07:00
c2d2bfa06c
Get raw exec tests compiling and passing again
2018-10-16 16:56:56 -07:00
3183b33d24
client: review comments and fixup/skip tests
2018-10-16 16:56:56 -07:00
64e9fc3798
docklog: add go-plugin for forwarding of docker logs
2018-10-16 16:56:56 -07:00
4a4c7dbbfc
client: begin driver plugin integration
...
client: fingerprint driver plugins
2018-10-16 16:56:56 -07:00
7946a14aa8
Fix lints
2018-10-16 16:56:56 -07:00
5fb95c1a00
drivers/shared: added func comment to eventer
2018-10-16 16:56:55 -07:00
951a084198
drivers/shared: move eventer to subpackage under drivers shared package
2018-10-16 16:56:55 -07:00
969ecea6f0
drivers/utils: better handling of consumer cleanup in eventer
2018-10-16 16:56:55 -07:00
8cf32eb9d2
drivers/rawexec: PR comments and feedback
2018-10-16 16:56:55 -07:00
a95cbc38ba
drivers/raw_exec: sync access to task state
2018-10-16 16:53:31 -07:00
28e8e5852c
drivers/raw_exec: added unix specific tests
2018-10-16 16:53:31 -07:00
1f6873806e
raw_exec: move package outside of plugins dir
2018-10-16 16:53:31 -07:00
Nick Ethier