Commit graph

13716 commits

Author SHA1 Message Date
Michael Schurter fd2db90569 plugins: msgpack codec should use cty tags 2018-10-16 16:56:56 -07:00
Preetha Appan d634157158 Address review comments around logging task properly 2018-10-16 16:56:56 -07:00
Preetha Appan 3280727c4e Address review comments 2018-10-16 16:56:56 -07:00
Preetha Appan 02b1f26d34 Switch back to using map[string]string for port map 2018-10-16 16:56:56 -07:00
Michael Schurter ed5641055f fixup comments, logging, and missing method impls
from #4777 comments
2018-10-16 16:56:56 -07:00
Michael Schurter 1a29337e48 register drivers by default
Do not register mock_driver on release builds.
2018-10-16 16:56:56 -07:00
Michael Schurter 118ce7c782 drivers/mock: complete plugin impl 2018-10-16 16:56:56 -07:00
Nick Ethier 9f564781ce drivers/mock: start mock driver implementation 2018-10-16 16:56:56 -07:00
Michael Schurter 528c426c53 Port client portion of #4392 to new taskrunner
PR #4392 was merged to master *after* allocrunnerv2 was branched, so the
client-specific portions must be ported from master to arv2.
2018-10-16 16:56:56 -07:00
Michael Schurter f12501d4c3 tr: implement dispatch payload hook
Now passing the TaskDir struct to prestart hooks instead of just the
root task dir itself as dispatch needs local/.
2018-10-16 16:56:56 -07:00
Preetha Appan 9b3e82a4e7 make port map a slice of maps to match existing rkt driver 2018-10-16 16:56:56 -07:00
Preetha Appan 4f4777d6a6 Review comments 2018-10-16 16:56:56 -07:00
Preetha Appan edad5e9e6d Stats collection test 2018-10-16 16:56:56 -07:00
Preetha Appan 678072ecd1 RKT driver plugin and unit tests 2018-10-16 16:56:56 -07:00
Nick Ethier d9f0cbf4a9 client: log retry during driver fingerprint redispense 2018-10-16 16:56:56 -07:00
Nick Ethier c7ac1186c9 client: add test for driverfailure during fingerprinting 2018-10-16 16:56:56 -07:00
Nick Ethier 15e8477c4e rkt: start rkt driver plugin 2018-10-16 16:56:56 -07:00
Preetha Appan c2d2bfa06c Get raw exec tests compiling and passing again 2018-10-16 16:56:56 -07:00
Nick Ethier 8cf669b5aa taskrunner: return error on waitCh 2018-10-16 16:56:56 -07:00
Nick Ethier 047fad2953 client: simplify driver plugin logic from review comments 2018-10-16 16:56:56 -07:00
Nick Ethier ed3cdaf3d1 plugin/driver: add Copy funcs 2018-10-16 16:56:56 -07:00
Nick Ethier 9686e1b258 client: fix broked tests from refactoring 2018-10-16 16:56:56 -07:00
Nick Ethier 3183b33d24 client: review comments and fixup/skip tests 2018-10-16 16:56:56 -07:00
Nick Ethier 64e9fc3798 docklog: add go-plugin for forwarding of docker logs 2018-10-16 16:56:56 -07:00
Nick Ethier f192c3752a client: refactor post allocrunnerv2 finalization 2018-10-16 16:56:56 -07:00
Nick Ethier 4a4c7dbbfc client: begin driver plugin integration
client: fingerprint driver plugins
2018-10-16 16:56:56 -07:00
Alex Dadgar 7946a14aa8 Fix lints 2018-10-16 16:56:56 -07:00
Alex Dadgar 89dafaaea9 compile on windows 2018-10-16 16:56:56 -07:00
Alex Dadgar ad4fac526c more test fixes 2018-10-16 16:56:56 -07:00
Alex Dadgar 45e41cca03 allocrunnerv2 -> allocrunner 2018-10-16 16:56:56 -07:00
Alex Dadgar 9baa7402ef fix test compiling 2018-10-16 16:56:55 -07:00
Alex Dadgar 7d9c069f09 skip building deprecated files 2018-10-16 16:56:55 -07:00
Alex Dadgar 6c9d9d5173 move files around 2018-10-16 16:56:55 -07:00
Nick Ethier 5fb95c1a00 drivers/shared: added func comment to eventer 2018-10-16 16:56:55 -07:00
Nick Ethier 951a084198 drivers/shared: move eventer to subpackage under drivers shared package 2018-10-16 16:56:55 -07:00
Nick Ethier 969ecea6f0 drivers/utils: better handling of consumer cleanup in eventer 2018-10-16 16:56:55 -07:00
Nick Ethier 8d04ae860d plugins/drivers: remove bool to track if eventLoop shutdown and use context instead 2018-10-16 16:56:55 -07:00
Nick Ethier 8cf32eb9d2 drivers/rawexec: PR comments and feedback 2018-10-16 16:56:55 -07:00
Nick Ethier 5742a6b932 plugin/drivers: rework eventer and change naming stream -> consumer 2018-10-16 16:56:55 -07:00
Michael Schurter 5f696608a6 tests: fix missing logger caused by bad merge 2018-10-16 16:56:55 -07:00
Michael Schurter 048510b13e tr: properly comment handle fields 2018-10-16 16:56:55 -07:00
Michael Schurter 9e49ed3464 ar: AllocState should not mutate ar.state
If ar.state.TaskStates has not been set, set it on the copy of ar.state.
That keeps ar.state manipulations in one location and allows AllocState
to only acquire read-locks.
2018-10-16 16:56:55 -07:00
Michael Schurter f279b1d1b1 tests: test logs endpoint against pending task
Although the really exciting change is making WaitForRunning return the
allocations that it started. This should cut down test boilerplate
significantly.
2018-10-16 16:56:55 -07:00
Michael Schurter dd4227f84a tests: make a test client/config easier to generate
Sadly can't move the fingerprint timeout tweak into the helper due to
circular imports.
2018-10-16 16:56:55 -07:00
Michael Schurter 1d747048ea tests: ensure task state is initialized in NewAR
Also expose NoopDB for use in tests.
2018-10-16 16:56:55 -07:00
Michael Schurter 6bcf772f3c tests: test via ServeMux so http codes are set 2018-10-16 16:56:55 -07:00
Michael Schurter 960f3be76c client: expose task state to client
The interesting decision in this commit was to expose AR's state and not
a fully materialized Allocation struct. AR.clientAlloc builds an Alloc
that contains the task state, so I considered simply memoizing and
exposing that method.

However, that would lead to AR having two awkwardly similar methods:
 - Alloc() - which returns the server-sent alloc
 - ClientAlloc() - which returns the fully materialized client alloc

Since ClientAlloc() could be memoized it would be just as cheap to call
as Alloc(), so why not replace Alloc() entirely?

Replacing Alloc() entirely would require Update() to immediately
materialize the task states on server-sent Allocs as there may have been
local task state changes since the server received an Alloc update.

This quickly becomes difficult to reason about: should Update hooks use
the TaskStates? Are state changes caused by TR Update hooks immediately
reflected in the Alloc? Should AR persist its copy of the Alloc? If so,
are its TaskStates canonical or the TaskStates on TR?

So! Forget that. Let's separate the static Allocation from the dynamic
AR & TR state!

 - AR.Alloc() is for static Allocation access (often for the Job)
 - AR.AllocState() is for the dynamic AR & TR runtime state (deployment
   status, task states, etc).

If code needs to know the status of a task: AllocState()
If code needs to know the names of tasks: Alloc()

It should be very easy for a developer to reason about which method they
should call and what they can do with the return values.
2018-10-16 16:56:55 -07:00
Michael Schurter fb4aa74153 client: add comment 2018-10-16 16:56:55 -07:00
Michael Schurter 9a7e6be2b6 client: fix potentially dropped streaming errors 2018-10-16 16:56:55 -07:00
Michael Schurter 4b44b9039b tr: remove unneeded lock; chan synchronizes access 2018-10-16 16:56:55 -07:00