luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
15549 commits 1 branch 0 tags 311 MiB
Commit graph

439 commits

Author SHA1 Message Date
Danielle Tomlinson 10b3e68a6d
Merge pull request #4925 from hashicorp/f-driver-plugins-dani 
Third Party Driver Plugins Support
 2018-12-03 20:48:19 +01:00
Mahmood Ali 88622b97bd
libcontainer to manage /dev and /proc (#4945) 
libcontainer already manages `/dev`, overriding task_dir - so let's use it for `/proc` as well and remove deadcode.
 2018-12-03 10:41:01 -05:00
Danielle Tomlinson 393b76ed7f plugins: Move driver testing support to subpackage 
this allows us to drop a cyclical import, but is subobptimal as it
requires BaseDriver tests to move. This falls firmly into the realm of
being a hack. Alternatives welcome.
 2018-12-01 17:29:39 +01:00
Danielle Tomlinson 66c521ca17 client: Move fingerprint structs to pkg 
This removes a cyclical dependency when importing client/structs from
dependencies of the plugin_loader, specifically, drivers. Due to
client/config also depending on the plugin_loader.

It also better reflects the ownership of fingerprint structs, as they
are fairly internal to the fingerprint manager.
 2018-12-01 17:10:39 +01:00
Danielle Tomlinson 51a9f7369e
Merge pull request #4936 from hashicorp/f-legacy-refactor 
Refactor and repackage client/driver
 2018-11-30 13:38:06 +01:00
Mahmood Ali 84e04cfa40
Merge pull request #4926 from hashicorp/f-docker-image-ref 
Use user provided image name to launch container
 2018-11-30 07:27:39 -05:00
Mahmood Ali 94d43b8003
Merge pull request #4924 from hashicorp/f-docker-mounts 
Support bind and tmpfs docker mounts
 2018-11-30 07:27:17 -05:00
Danielle Tomlinson 2db5ae38d8 client: Rename drivers/shared/env => client/taskenv 2018-11-30 12:18:39 +01:00
Danielle Tomlinson f3a77b8084 client: Merge driver/shared/structs and client/structs 2018-11-30 10:56:45 +01:00
Danielle Tomlinson fdfe93aa25 fixup: executorplugin: fix rkt build 2018-11-30 10:47:08 +01:00
Danielle Tomlinson 04c8851b4c client: Migrate DriverStats optout to drivers/shared/structs 2018-11-30 10:46:13 +01:00
Danielle Tomlinson d582ea1d8b drivers: Create drivers/shared/structs 
This creates a drivers/shared/structs package and moves the buffer size
checks into it.
 2018-11-30 10:46:13 +01:00
Danielle Tomlinson 0544a57abe drivers: Move client/drivers/executor to drivers/shared/executor 2018-11-30 10:46:13 +01:00
Danielle Tomlinson 1a29811169 drivers: Move client/drivers/env to drivers/shared/env 
As part of deprecating legacy drivers, we're moving the env package to a
new drivers/shared tree, as it is used by the modern docker and rkt
driver packages, and is useful for 3rd party plugins.
 2018-11-30 10:46:13 +01:00
Preetha Appan 0d90ba392e
Fix lxc test panic 2018-11-28 13:56:17 -06:00
Preetha Appan 924f1b69e9
Fix failing lxc test 2018-11-28 11:05:35 -06:00
Preetha Appan bf58c65ef7
Fix LXC driver fingerprint to use typedattributes 2018-11-28 10:09:10 -06:00
Preetha Appan 9f4439243b
Fix docker driver to use new fingerprint typed attributes 2018-11-28 10:01:03 -06:00
Preetha Appan f89dbcd9cc
modify fingerprint interface to use typed attribute struct 2018-11-28 10:01:03 -06:00
Mahmood Ali 9af8deabbf address review comments 2018-11-27 21:40:43 -05:00
Mahmood Ali 6d34d2fade Add Driver Plugin for LXC 2018-11-27 21:40:43 -05:00
Michael Schurter e565c63eed gofmt -s -w drivers/rkt/driver_test.go 2018-11-27 17:24:23 -08:00
Mahmood Ali 844fd47acc Use user provided image name to launch container 
This allows the container to be tagged with a user friendly image name
(e.g. `redis:3.2`) rather than the image ID (e.g.
`sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916`).

Useful for human debugging, as well as some debugging and image scanning
tools.

This risks two bad changes:
1. Discrepancy in image resolution between docker and Nomad's image
loader.
  * I checked the image creation paths in Nomad, and noticed that we
either pulled the image or inspect the image with the user provided
name.

2. A race in image tagging where the tag is modified between image
loading and container creation.
  * I, personally, don't think this case is cause for concern, as it is
analogous to the task running a bit later.  As long as the image is
still present, creating the container should be good.
 2018-11-27 16:12:15 -05:00
Mahmood Ali f6d6a50c39 add support for tmpfs 2018-11-27 07:20:17 -05:00
Mahmood Ali 0a09f5521d Support docker bind mounts 2018-11-27 07:20:17 -05:00
Chris Baker 3dd6ba514a drivers/rkt: updated test to include new AllocID field in TaskConfig 2018-11-26 21:37:58 +00:00
Chris Baker 9bd4317139 modified TaskConfig to include AllocID 
use this for volume names in drivers/rkt to address #1150
 2018-11-26 18:54:26 +00:00
Mahmood Ali 141092e46d Formatting and typo fixes 2018-11-25 11:53:21 -05:00
Mahmood Ali c61d99b525
Merge pull request #4908 from hashicorp/f-docker-opts-storageopt 
Add support for docker storage options
 2018-11-20 21:08:27 -05:00
Nick Ethier 1f3fe02e62
docker: sync access to exit result within a handle 2018-11-20 20:41:32 -05:00
Michael Schurter 2275153875
Apply suggestions from code review 
Co-Authored-By: nickethier <ncethier@gmail.com>
 2018-11-20 20:33:31 -05:00
Mahmood Ali e9e415f186 Add support for storage opt 2018-11-20 16:11:02 -05:00
Nick Ethier 3ccd359735
docker: unexport new coordinator func 2018-11-19 23:07:07 -05:00
Nick Ethier 8b9b2b476e
docker: add default blocks for driver plugin config schema 2018-11-19 22:59:18 -05:00
Nick Ethier 2667f48a5d
docker: move config RPCs to config.go 2018-11-19 22:59:18 -05:00
Nick Ethier aa9f45ae47
docker: fix tests 2018-11-19 22:59:18 -05:00
Nick Ethier 0f03e8f520
docker: remove container pointer from task handle 2018-11-19 22:59:18 -05:00
Nick Ethier ce4b867d21
docker: move volume driver options to seperate block 2018-11-19 22:59:18 -05:00
Nick Ethier fca2df3c79
docker: group common config into blocks 2018-11-19 22:59:17 -05:00
Michael Schurter 813341dd59
Apply suggestions from code review 
Co-Authored-By: nickethier <ncethier@gmail.com>
 2018-11-19 22:59:17 -05:00
Nick Ethier b7bd36db30
docker: remove global pull coordinator 2018-11-19 22:59:17 -05:00
Nick Ethier f0a86859a0
docker: remove call to global metrics instance 2018-11-19 22:59:17 -05:00
Nick Ethier 8ef73e63ce
docker: moved fingerprint code to it's own file 2018-11-19 22:59:17 -05:00
Nick Ethier 4be8a86ef9
plugins/driver: remove NodeResources from task Resources and use PercentTicks field for docker driver 2018-11-19 22:59:17 -05:00
Nick Ethier ced5d5c445
docker: move recoverable error proto to shared structs 2018-11-19 22:59:16 -05:00
Nick Ethier 585e468085
docker: implement recover task logic 2018-11-19 22:59:16 -05:00
Nick Ethier ee51cb6a93
docker: finished porting tests 2018-11-19 22:59:16 -05:00
Nick Ethier 3d7cdea19e
drivers/docker: more work porting tests from old driver plugin 2018-11-19 22:59:16 -05:00
Nick Ethier 8f8698b3e1
docker: started work on porting docker driver to new plugin framework 2018-11-19 22:59:15 -05:00
Nick Ethier 9e64ce7d73
docker: properly launch docker logger process 2018-11-19 22:59:12 -05:00
Michael Schurter f8cdd561f0 client: interpolate driver configurations 
Also add missing SetDriverNetwork calls.
 2018-11-15 16:25:57 -08:00
Mahmood Ali 1f4db08f42 Regenerate proto files with protoc-gen-go@v1.2.0 2018-11-14 09:01:26 -05:00
Mahmood Ali 1e92161f14
Merge pull request #4858 from hashicorp/b-fix-master-20181109 
Fix some tests in master
 2018-11-13 16:08:26 -05:00
Alex Dadgar 17e8446484
Merge pull request #4868 from hashicorp/b-plugin-ctx 
Plugin client's handle plugin dying
 2018-11-13 10:26:53 -08:00
Mahmood Ali 356c194acc Use materialized duration fields for driver config 2018-11-13 10:21:40 -05:00
Mahmood Ali 470d20cdf3 Avoid downloading image if present locally 2018-11-13 10:21:40 -05:00
Mahmood Ali 7690f389a0 Prioritize checking consumer context cancellation 
Tests expect that as soon as eventer shuts down immediately on context
cancellations; but golang does not guarantee priority when multiple
pending channels are ready in a select statement.
 2018-11-13 10:21:40 -05:00
Mahmood Ali e9067e52b4 pull alpine image needed for test 
The test requires the image to be present locally, so importing it as
part of setup.
 2018-11-13 10:21:40 -05:00
Mahmood Ali 8923ea4663 Handle time.Duration in mock 
Mock driver config uses `time.Duration` fields but we initialize them
inconsistently, as time.Duration sometimes and as duration strings other
times.  Previously, `mapstructure` handles it and does the right thing.

This is no longer the case with MsgPack.  I could not find a good way to
bring back old behavior without too much complexity.  `MsgPack` extended
types weren't ideal here as we lose type information (e.g. int64 vs
string), and the input is a generic map and not a MsgPack serialization
of duration.

As such, I went with the simple solution of declaring the config field
as duration string, and panicing if the test doesn't pass a valid
string.

I found this to cause the smallest change in tests, but we can
alternatively force all to be int64 instead.
 2018-11-13 10:21:40 -05:00
Mahmood Ali fb56dd699d distinguish java driver tests from others 2018-11-13 10:21:40 -05:00
Mahmood Ali 343df28165 Fix java driver tests 2018-11-13 10:21:40 -05:00
Mahmood Ali e506ebbc24
Merge pull request #4845 from hashicorp/r-exec-refactor 
Update exec driver to match rawexec
 2018-11-12 20:59:32 -05:00
Alex Dadgar 693f244cce Plugin client's handle plugin dying 
This PR plumbs the plugins done ctx through the base and driver plugin
clients (device already had it). Further, it adds generic handling of
gRPC stream errors.
 2018-11-12 17:09:27 -08:00
Danielle Tomlinson 880e5015f2 rawexec: Only use cgroups when running as root. 
If Nomad is not running as root, we should not try to use cgroups for pid
freezing.

This originally was implemented pre-driver-support in
https://github.com/hashicorp/nomad/blob/v0.8.6/client/driver/raw_exec.go#L120-L130
 2018-11-10 06:45:11 -08:00
Mahmood Ali 9c76a7d9b8 handle nil resources limits 
Config may not express resource constraints on the job!
 2018-11-09 15:52:44 -05:00
Mahmood Ali 1ed53ca478 shorter health descriptions 2018-11-08 10:00:49 -08:00
Mahmood Ali 032f86bc78 Add a helper functions for checking unix root 2018-11-08 10:00:49 -08:00
Mahmood Ali 21d126cd68 Update exec driver to match rawexec 
Update exec driver with latest patterns used by rawexec, and eliminate
accidental diffs.
 2018-11-08 10:00:49 -08:00
Alex Dadgar a7ca737fb6 review comments 2018-11-07 11:31:52 -08:00
Alex Dadgar 204ca8230c Device manager 
Introduce a device manager that manages the lifecycle of device plugins
on the client. It fingerprints, collects stats, and forwards Reserve
requests to the correct plugin. The manager, also handles device plugins
failing and validates their output.
 2018-11-07 10:43:15 -08:00
Mahmood Ali 32283b2a66 reconcile java with latest patterns 2018-11-06 12:49:29 -08:00
Mahmood Ali 467360954a driver/java: support non-linux targets 
Java driver supports non-linux targets, so LinuxResources fields isn't
guaranteed to be not-nil.
 2018-11-06 12:41:39 -08:00
Mahmood Ali e64ef839b8 Reconcile with latest master patterns 2018-11-06 12:41:39 -08:00
Mahmood Ali c3881bd8d1 add java driver tests 2018-11-06 12:41:39 -08:00
Mahmood Ali 4f525d0a17 Implement java driver 2018-11-06 12:41:39 -08:00
Michael Schurter 392d548b85
Merge pull request #4828 from hashicorp/b-restore 
Implement client agent restarting
 2018-11-05 18:50:15 -06:00
Michael Schurter 2bbd88888c client: first pass at implementing task restoring 
Task restoring works but dead tasks may be restarted
 2018-11-05 12:32:05 -08:00
Mahmood Ali a17521475d
Merge pull request #4826 from hashicorp/b-driver-exec-tweaks-20181031 
Register exec driver plugin among some fixes
 2018-11-02 10:11:05 -04:00
Mahmood Ali dc0fc442ab add plugin config 2018-11-02 08:51:23 -04:00
Michael Schurter 83db3e6a50 rkt: standardize names/code 2018-11-01 15:53:07 -07:00
Michael Schurter 1a7fb51285 rawexec: standardize names/code 2018-11-01 15:53:07 -07:00
Michael Schurter 5a5d52023d qemu: standardize names/code 2018-11-01 15:53:07 -07:00
Michael Schurter 3c5e606ef6 drivers/mock: standardize names/code 2018-10-31 11:52:51 -07:00
Mahmood Ali 9a45837c4e Register exec driver plugin among some fixes 
Namely, remove the `enabled` configuration flag, as it's specific to
`raw_exec` driver.  Also, pass resource limits to underlying call.
 2018-10-31 13:55:59 -04:00
Michael Schurter d3dafc9761 mock_driver: match other's fingerprint message 2018-10-30 17:38:23 -07:00
Michael Schurter 2a38f575c0 rkt: fix race, register by default, fix fingerprint 2018-10-30 17:37:00 -07:00
Michael Schurter 5b0b75e1ee qemu: fix fingerprint description and handle race 2018-10-30 17:34:08 -07:00
Michael Schurter 6f4a28a7c2 rawexec: fix fingerprint results and tests 
In tests:
* always cleanup (Kill) the harness
* don't call SetConfig more than once (a race)
* avoid cgroups to allow running without being root
 2018-10-30 17:16:22 -07:00
Michael Schurter 616801417c rawexec: fix race around h.procState 2018-10-30 17:15:47 -07:00
Michael Schurter 9fa53565ad drivers: remove stutter from exported driver names 
Also fix a comment on the logger that got copy/pasted around.
 2018-10-30 14:05:31 -07:00
Michael Schurter fc320906aa qemu: fix build and register by default 
API changes had broken the build.
 2018-10-29 22:10:37 -07:00
Nick Ethier 3fcf8ba7e6
Merge pull request #4795 from hashicorp/f-plugin-config 
Pass client configuration to plugins through loader
 2018-10-29 18:42:27 -07:00
Nick Ethier bda3b1d3b3
rename NomadConfig to ClientAgentConfig 2018-10-29 21:34:34 -04:00
Preetha Appan 68991d8acb
Merge branch 'f-driver-plugin-qemu' of github.com:hashicorp/nomad into f-driver-plugin-qemu 2018-10-25 13:51:58 -05:00
Preetha Appan bfff771416
remove driver config struct from qemu driver since its not used 2018-10-25 13:51:06 -05:00
Michael Schurter 48d25dbef1
Fix hclog log line formatting 
Co-Authored-By: preetapan <preetha@hashicorp.com>
 2018-10-25 13:47:17 -05:00
Preetha Appan 95a1637741
Fix imports 2018-10-25 13:45:44 -05:00
Michael Schurter ca151a7828
Fix copy pasted comment 
Co-Authored-By: preetapan <preetha@hashicorp.com>
 2018-10-25 13:43:29 -05:00
Michael Schurter c85b90f1e4
remove newline 
Co-Authored-By: preetapan <preetha@hashicorp.com>
 2018-10-25 13:42:20 -05:00
Michael Schurter 66c15e3a67
update comment 
Co-Authored-By: preetapan <preetha@hashicorp.com>
 2018-10-25 13:41:24 -05:00
Michael Schurter 9ca6c998c7
Remove unnecessary return 
Co-Authored-By: preetapan <preetha@hashicorp.com>
 2018-10-25 13:40:28 -05:00
Michael Schurter fc7ba5e79d
remove unnecessary return 
Co-Authored-By: preetapan <preetha@hashicorp.com>
 2018-10-25 13:40:15 -05:00
Nick Ethier 9916b9e0ce
drivers: update SetConfig to handle driver client config 2018-10-18 23:32:17 -04:00
Michael Schurter af3e9adf2f Update drivers/qemu/driver.go 2018-10-17 17:42:01 -05:00
Preetha Appan 93eb048b87
QEMU driver plugin 2018-10-17 11:43:15 -05:00
Alex Dadgar 4b694b2dec
Merge pull request #4793 from hashicorp/b-no-cty 
Do not use cty in drivers
 2018-10-16 20:04:41 -07:00
Nick Ethier 823a6c8d6c
update rkt and exec setconfig funcs 2018-10-16 23:00:26 -04:00
Nick Ethier 65adb80ebf
plumb NomadConfig into plugins 2018-10-16 22:47:22 -04:00
Nick Ethier b657c3909a
drivers/exec: better fingerprinting and detection on non linux 2018-10-16 22:45:33 -04:00
Nick Ethier d94b631b6b
drivers/exec: add exec implementation 2018-10-16 22:45:28 -04:00
Alex Dadgar 88c1885044 Do not use cty in drivers 2018-10-16 17:17:07 -07:00
Michael Schurter 838ddf4d4a fix linter errors 2018-10-16 16:56:57 -07:00
Nick Ethier 448a709603 driver/docker: rename container logger 2018-10-16 16:56:56 -07:00
Michael Schurter 64605a3957 drivers/mock: fix plugin name 
Was mock_driver before plugins, so keep the name.
 2018-10-16 16:56:56 -07:00
Preetha Appan d634157158 Address review comments around logging task properly 2018-10-16 16:56:56 -07:00
Preetha Appan 3280727c4e Address review comments 2018-10-16 16:56:56 -07:00
Preetha Appan 02b1f26d34 Switch back to using map[string]string for port map 2018-10-16 16:56:56 -07:00
Michael Schurter ed5641055f fixup comments, logging, and missing method impls 
from #4777 comments
 2018-10-16 16:56:56 -07:00
Michael Schurter 1a29337e48 register drivers by default 
Do not register mock_driver on release builds.
 2018-10-16 16:56:56 -07:00
Michael Schurter 118ce7c782 drivers/mock: complete plugin impl 2018-10-16 16:56:56 -07:00
Nick Ethier 9f564781ce drivers/mock: start mock driver implementation 2018-10-16 16:56:56 -07:00
Preetha Appan 9b3e82a4e7 make port map a slice of maps to match existing rkt driver 2018-10-16 16:56:56 -07:00
Preetha Appan 4f4777d6a6 Review comments 2018-10-16 16:56:56 -07:00
Preetha Appan edad5e9e6d Stats collection test 2018-10-16 16:56:56 -07:00
Preetha Appan 678072ecd1 RKT driver plugin and unit tests 2018-10-16 16:56:56 -07:00
Nick Ethier c7ac1186c9 client: add test for driverfailure during fingerprinting 2018-10-16 16:56:56 -07:00
Nick Ethier 15e8477c4e rkt: start rkt driver plugin 2018-10-16 16:56:56 -07:00
Preetha Appan c2d2bfa06c Get raw exec tests compiling and passing again 2018-10-16 16:56:56 -07:00
Nick Ethier 3183b33d24 client: review comments and fixup/skip tests 2018-10-16 16:56:56 -07:00
Nick Ethier 64e9fc3798 docklog: add go-plugin for forwarding of docker logs 2018-10-16 16:56:56 -07:00
Nick Ethier 4a4c7dbbfc client: begin driver plugin integration 
client: fingerprint driver plugins
 2018-10-16 16:56:56 -07:00
Alex Dadgar 7946a14aa8 Fix lints 2018-10-16 16:56:56 -07:00
Nick Ethier 5fb95c1a00 drivers/shared: added func comment to eventer 2018-10-16 16:56:55 -07:00
Nick Ethier 951a084198 drivers/shared: move eventer to subpackage under drivers shared package 2018-10-16 16:56:55 -07:00
Nick Ethier 969ecea6f0 drivers/utils: better handling of consumer cleanup in eventer 2018-10-16 16:56:55 -07:00
Nick Ethier 8cf32eb9d2 drivers/rawexec: PR comments and feedback 2018-10-16 16:56:55 -07:00
Nick Ethier a95cbc38ba drivers/raw_exec: sync access to task state 2018-10-16 16:53:31 -07:00
Nick Ethier 28e8e5852c drivers/raw_exec: added unix specific tests 2018-10-16 16:53:31 -07:00
Nick Ethier 1f6873806e raw_exec: move package outside of plugins dir 2018-10-16 16:53:31 -07:00