luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
9468 commits 1 branch 0 tags 311 MiB
Commit graph

2622 commits

Author SHA1 Message Date
Alex Dadgar d760e68774 darwin test fixes 2017-07-22 19:48:47 -07:00
Alex Dadgar 553bc91725 Parallel client tests (#2890) 
* alloc_runner

* Random tests

* parallel task_runner and no exec compatible check

* Parallel client

* Fail fast and use random ports

* Fix docker port mapping

* Make concurrent pull less timing dependant

* up parallel

* Fixes

* don't build chroots in parallel on travis

* Reduce parallelism on travis with lxc/rkt

* make java test app not run forever

* drop parallelism a little

* use docker ports that are out of the os's ephemeral port range

* Limit even more on travis

* rkt deadline
 2017-07-22 19:04:36 -07:00
Alex Dadgar b6f0782732 typo 2017-07-22 12:55:30 -07:00
Alex Dadgar 8cf9d15b01 typo 2017-07-22 12:33:07 -07:00
Alex Dadgar 9e9c20ca77 small fixes 2017-07-22 12:25:02 -07:00
Alex Dadgar 5a3df2ed89 Merge pull request #2888 from hashicorp/b-fix-allocrunner-test 
Fix TestAllocRunner_TaskLeader_StopTG and unrelated races
 2017-07-22 11:44:04 -07:00
Alex Dadgar 46c8bec9b0 faster vaultclient 2017-07-21 19:38:37 -07:00
Michael Schurter d840fc8c95 Fix tr race by not sharing alloc/task 
prestart only needs the original alloc/task so pass their pointers in.
Task updates may concurrently replace the pointer on tr.
 2017-07-21 16:17:42 -07:00
Michael Schurter a22cfa8387 Minor test race fix 2017-07-21 16:17:23 -07:00
Michael Schurter 9a7a1d8c13 Fix race by not accessing tr.task from ar 2017-07-21 16:16:53 -07:00
Michael Schurter 2e9a1e3fa6 Remove unneeded saveTaskRunnerState method 
Collapse it into the one place it's called
 2017-07-21 16:16:02 -07:00
Michael Schurter 996ce9286e Fix test race by locking around ar.tasks access 2017-07-21 14:25:51 -07:00
Michael Schurter 8d1d8eac46 Fix handle race 2017-07-21 14:00:32 -07:00
Michael Schurter 5f40901422 Fix more test races 2017-07-21 14:00:21 -07:00
Michael Schurter b9ba447399 Fixup a few more even rarer test races 2017-07-21 13:43:32 -07:00
Michael Schurter 38cb2021dd Always interpolate task before calling with Consul 
Also switch to returning a copy of the task to avoid races between
altering the Task and persitence.
 2017-07-21 13:37:16 -07:00
Michael Schurter 6e80a8ee39 Fix TestAllocRunner_TaskLeader_StopTG 
Also make alloc runner tests less racy. Basically every alloc runner
test used to have races with `upd.{Count,Allocs}`
 2017-07-21 13:37:16 -07:00
Alex Dadgar e509661cf9 executor and logging pkg 2017-07-21 12:14:54 -07:00
Alex Dadgar 7c433a1767 Parallel 2017-07-21 12:06:39 -07:00
Karel Malec 4b98f94a88 Allow rkt driver to mount volumes read-only 2017-07-21 13:05:15 +02:00
Alex Dadgar 56f9cf86df Speed up client startup 2017-07-20 22:34:24 -07:00
Michael Schurter 0d7f7e2b9d Merge pull request #2878 from hashicorp/b-save-state 
Fix state handling on restart
 2017-07-20 17:16:46 -07:00
Karel Malec cf985f011c Pass task group name as NOMAD_GROUP_NAME environment variable 2017-07-21 01:22:54 +02:00
Alex Dadgar 09c8ee621b Destroy tasks that are part of terminal alloc 2017-07-20 12:02:04 -07:00
Michael Schurter 9a7f649e56 Don't save task runner state if it is destroyed 2017-07-20 10:17:41 -07:00
Alex Dadgar 64776b1370 Should not persist state after alloc_runner is garbage collected 2017-07-19 17:31:30 -07:00
Michael Schurter c1b8bef813 Use broadcast send retry logic everywhere 2017-07-18 14:36:32 -07:00
Alex Dadgar d2381c9263 Merge pull request #2853 from hashicorp/b-watcher 
Improve alloc health watcher
 2017-07-18 14:12:28 -07:00
Alex Dadgar bd43bd509c Save deployment status 2017-07-18 12:37:52 -07:00
Alex Dadgar 41f67e3535 Small fixes 2017-07-18 12:19:57 -07:00
Michael Schurter c24e73ede7 Fix deadlock caused by syncing during destroy 
When replacing an alloc the new alloc is blocked until the old alloc is
destroyed. This could cause a deadlock:

1. Destroying the old alloc includes a final sync of its status
2. Syncing status causes a GC
3. A GC looks for terminal allocs to cleanup
4. The GC waits for an alloc to stop completely before GC'ing

If the GC chooses the currently-being-destroyed-alloc to GC, the GC
deadlocks. If `client.max_parallel` deadlocks happen the GC is wedged
until the Nomad process is restarted.

Performing the final sync asynchronously is an ugly hack but prevents
the deadlock by allowing the final sync to occur after the alloc runner
has shutdown and been destroyed.
 2017-07-18 11:12:56 -07:00
Michael Schurter 420be86e39 Test AllocDir.Copy 2017-07-17 15:46:54 -07:00
Michael Schurter cdb2e96d99 Add AllocRunner.allocID for ease-of-use 
Since the AllocRunner.alloc struct can be mutated, most of AllocRunner
needs to acquire a lock to get the alloc's ID. Log lines always need to
include the alloc ID, so we often skipped acquiring a lock just to grab
the ID and accepted the race.

Let's make the race detector a little happier by storing the ID in a
single assignment field.
 2017-07-17 15:46:54 -07:00
Michael Schurter 181fda825a Fix log level 2017-07-17 15:46:54 -07:00
Michael Schurter 98f6e7f10f Don't fail if task dirs don't exist on creation 
Task dir metadata is created in AllocRunner.Run which may not run before
an alloc is sync'd and Nomad exits. There's no reason not to just create
task dir metadata on restore if it doesn't exist.
 2017-07-17 15:46:54 -07:00
Michael Schurter 51515cbe0c Ensure allocDir is never nil and persisted safely 
Fixes #2834
 2017-07-17 15:46:54 -07:00
Alex Dadgar 0821ee67f5 Fix alloc broadcaster panic on double close 2017-07-17 14:09:05 -07:00
Michael Schurter 0a6bf87365 Fix nil panic in Docker error condition 
Fixes #2835

Yet another bug caused by overwriting container and then trying to
reference container.ID in the err handling block. Did a quick audit of
docker.go and it seems to be the last offender. See #2804 for previous
bug.
 2017-07-14 10:48:19 -07:00
Michael Schurter e9a416b731 Merge branch 'master' into fix-pending-state 2017-07-10 10:43:23 -07:00
unknown 26b16fa3ce #2563 fixed pending state for allocations with terminal status 2017-07-09 16:18:06 +03:00
Alex Dadgar 05894f4611 Small fixes 2017-07-07 17:34:50 -07:00
Michael Schurter fecb16cfb2 Merge pull request #2793 from hashicorp/b-2776-ct-vault-servername 
Propagate vault.tls_server_name to consul-template
 2017-07-07 16:44:19 -07:00
Michael Schurter 95a9a5da71 Merge pull request #2787 from hashicorp/f-docker-test-mac 
Test #2652 - Docker MAC Address option
 2017-07-07 16:22:10 -07:00
Michael Schurter 4be4df21c9 Merge pull request #2797 from hashicorp/f-2785-docker-bridge-ip 
Add driver.docker.bridge_ip node attribute
 2017-07-07 16:20:20 -07:00
Michael Schurter 94389c3ecc Remove debug logging 2017-07-07 16:19:42 -07:00
Michael Schurter 5e3e3818db Merge pull request #2804 from hashicorp/b-2802-docker-panic 
Don't panic in container list/remove/inspect race
 2017-07-07 15:35:51 -07:00
Michael Schurter 67a7b0eac9 Don't panic in container list/remove/inspect race 
Fixes #2802

While it's hard to reproduce the theoretical race is:

1. This goroutine calls ListContainers()
2. Another goroutine removes a container X
3. This goroutine attempts to InspectContainer(X)

However, this bug could be hit in the much simpler case of
InspectContainer() timing out.

In those cases an error is returned and the old code attempted to wrap
the error with the now-nil container.ID. Storing the container ID fixes
that panic.
 2017-07-07 15:10:59 -07:00
Alex Dadgar bf97a2455c Vet and small improvement on watcher failure detection 2017-07-07 14:53:01 -07:00
Alex Dadgar 45712c6ca3 test fixes 2017-07-07 14:11:27 -07:00
Alex Dadgar ade9a7c768 @jippi Changed my mind! Good suggestion 2017-07-07 12:12:48 -07:00
Alex Dadgar c063eba836 Warn log 2017-07-07 12:10:04 -07:00
Alex Dadgar 067ed86a47 Client watches for allocation health using task state and Consul checks 
This PR adds watching of allocation health at the client. The client can
watch for health based on the tasks running on time and also based on
the consul checks passing.
 2017-07-07 12:10:04 -07:00
Alex Dadgar 001058227e watcher per alloc 2017-07-07 12:07:08 -07:00
Alex Dadgar 2e2fd26bed Update index 2017-07-07 12:07:08 -07:00
Alex Dadgar ecee5e370e initial watcher 2017-07-07 12:07:08 -07:00
Alex Dadgar c77944ed29 assign names 2017-07-07 12:03:11 -07:00
Michael Schurter 084dd384c1 Add driver.docker.bridge_ip node attribute 
Fixes #2785
 2017-07-07 10:14:10 -07:00
Michael Schurter d38d48151a Propagate vault.tls_server_name to consul-template 
Fixes #2776
 2017-07-06 16:56:50 -07:00
Michael Schurter 39edf23fd5 Merge pull request #2786 from hashicorp/f-docker-auth-soft-fail 
Default to auth hard fail but optionally soft fail
 2017-07-06 13:25:56 -07:00
Michael Schurter bae1b7db2d Test #2652 
Also cleanup docker config opts docs
 2017-07-06 12:46:25 -07:00
Michael Schurter 8f4353779a Merge branch 'master' into master 2017-07-06 12:09:36 -07:00
Michael Schurter 2900f941b5 Default to auth hard fail but optionally soft fail 2017-07-06 11:35:34 -07:00
Michael Schurter 08b452adf5 Merge pull request #2781 from hashicorp/f-2678-getter-mode 
Add support for go-getter modes
 2017-07-06 11:06:40 -07:00
Michael Schurter b000bb8598 Merge pull request #2744 from aep/master 
Do not fail when no docker registry auth is available
 2017-07-06 11:04:11 -07:00
Michael Schurter 0d3bdf7210 Add support for go-getter modes 
Fixes #2678
 2017-07-06 10:45:44 -07:00
Michael Schurter 644f0cfaa4 Consistently quote alloc ids in client logs 2017-07-06 10:24:52 -07:00
Michael Schurter 4fd9ef6a8c Tiny client race condition fix 
Plus some logging improvements that may help with #2563
 2017-07-05 16:15:19 -07:00
Michael Schurter 8e2e26c607 rkt: use %s instead of %q when interpolating env 
Fixes #2686
 2017-07-05 09:36:17 -07:00
Michael Schurter b2382f99f2 0 compute == error 2017-07-03 14:51:02 -07:00
Michael Schurter ecf090e980 Fix cpu_total_compute override 2017-07-03 14:51:02 -07:00
Michael Schurter 2d741c770b Merge pull request #2732 from hashicorp/b-persist-alloc-updates 
Persist Alloc when EvalID changes
 2017-07-03 14:46:43 -07:00
Michael Schurter 56a6f8ca8a Merge pull request #2763 from hashicorp/f-bad-state-help 
Add more logging to restore state errors
 2017-07-03 14:45:03 -07:00
Michael Schurter 9d4b0651ef Merge pull request #2753 from hashicorp/b-leader-dies-first 
Destroy task group leader first
 2017-07-03 14:38:04 -07:00
Michael Schurter 6e7cc3964e Merge pull request #2709 from hashicorp/f-advertise-docker-ips 
Advertise driver-specific addresses
 2017-07-03 14:04:12 -07:00
Michael Schurter 5ec52ec24a Destroy task group leader first 
Before this commit all tasks in a task group were destroyed
concurrently. This meant logging sidecars might be stopped before the
leader task whose logs still need to be shipped.

This commit blocks on the leader shutting down before signalling to
followers to shutdown.
 2017-07-03 13:56:56 -07:00
Michael Schurter 596727230b Suggest wiping out alloc dir too 2017-07-03 12:29:21 -07:00
Michael Schurter 11f68bfca2 Add more logging to restore state errors 2017-07-03 11:58:41 -07:00
Arvid E. Picciani aa4f029f10 Do not fail when no docker registry auth is available 
this amends the behaviour introduced with #2651
and allows pulling public images when docker.auth.helper is set
 2017-06-27 11:11:18 +02:00
Michael Schurter 8fcf866a7d Fix some tests still expecting reverted behavior 2017-06-23 16:51:38 -07:00
Michael Schurter e81252ba45 Default no_host_uuid to true instead of false 
The host UUID isn't unique in many virtualized cases and of dubious
value even when it is univerally unique. Default to a random UUID.
 2017-06-23 16:23:01 -07:00
Michael Schurter 5a274e6683 Style and comments 2017-06-23 15:20:04 -07:00
Michael Schurter cff8546035 Fix spelling & re-add immutable state struct 2017-06-23 13:01:39 -07:00
Michael Schurter d359d3b554 Rename immutable -> alloc 
meh; naming is hard
 2017-06-23 10:58:36 -07:00
Michael Schurter af2fc0f1bc Persist Alloc when EvalID changes 2017-06-22 17:33:12 -07:00
Michael Schurter f3a6ddc57d Remove DRIVER env vars 
Also make NOMAD_ADDR_* use host ip:port for consistency. NOMAD_PORT_*
varies based on port map and the driver IP isn't exposed as an env var
as the only place it can be used is in script checks anyway.
 2017-06-21 17:19:08 -07:00
Michael Schurter 0633d0c286 Have Qemu return PortMap 2017-06-21 17:19:08 -07:00
Michael Schurter 38a0695687 Simplify Docker Networks processing 2017-06-21 17:19:08 -07:00
Michael Schurter fec83b271a Bump error log level 2017-06-21 17:19:08 -07:00
Michael Schurter 8d677bc6b9 Fix lxc tests 2017-06-21 17:19:08 -07:00
Michael Schurter 8d440b1675 Skip DRIVER env vars for labels without a port mapping 2017-06-21 17:19:08 -07:00
Michael Schurter c0eff81383 Fix Service.AddressMode changes during task updates 2017-06-21 17:19:08 -07:00
Michael Schurter 67d154a274 Test driver network advertisement and checks 2017-06-21 17:19:08 -07:00
Michael Schurter b9bfb84b53 Implement DriverNetwork and Service.AddressMode 
Ideally DriverNetwork would be fully populated in Driver.Prestart, but
Docker doesn't assign the container's IP until you start the container.

However, it's important to setup the port env vars before calling
Driver.Start, so Prestart should populate that.
 2017-06-21 17:19:08 -07:00
Hynek Schlawack 59ab34c264 Fix typos 2017-06-16 16:10:12 +02:00
Michael Schurter b69e060071 Log PID when sending signals 2017-06-12 11:11:36 -07:00
Michael Schurter ffb417a300 Merge pull request #2697 from hashicorp/b-port-map 
Fix port map interpolation for docker
 2017-06-09 13:29:36 -07:00
Michael Schurter a3827d2cc6 Fix bad merge conflict resolution 2017-06-09 10:40:47 -07:00
Michael Schurter eabd6759c6 Merge branch 'master' into add-no-overlay-option 2017-06-09 09:59:35 -07:00
Alex Dadgar 5ba2662b30 Merge pull request #2687 from mmickan/issue-2685 
Include symlinks in snapshots when migrating disks
 2017-06-08 13:35:46 -07:00
Michael Schurter 784d69789e Merge branch 'master' into add-no-overlay-option 2017-06-08 13:15:56 -07:00
Alex Dadgar 7695e636d5 Fix port map interpolation for docker 
This PR fixes an issue in which the value of the portmap could not be
interpolated.

Fixes https://github.com/hashicorp/nomad/issues/2680
 2017-06-08 13:12:32 -07:00
Karel Malec b55f4bf601 Fix backticks in docs; refine --debug comment 2017-06-07 21:11:22 +02:00
Karel Malec a258a803f2 Added insecure_options config list 2017-06-07 09:58:42 +02:00
Karel Malec 1957e9dfa6 Add a no_overlay option for the rkt task config. 2017-06-07 00:17:33 +02:00
Mark Mickan c196d320f8 Add tests for migrating symlinks in alloc and local directories 2017-06-04 15:56:22 +09:30
Mark Mickan 236f24c9a4 Include symlinks in snapshots when migrating disks 
Fixes #2685
 2017-06-04 00:36:18 +09:30
Michael Schurter d1dd380890 Switch to hashicorp/go-envparse 2017-06-02 15:58:52 -07:00
Michael Schurter a552bcdb55 Move env file parsing to a library 2017-06-02 15:03:27 -07:00
Alex Dadgar 3b46fe136f small cleanup 2017-05-31 15:56:54 -07:00
Alex Dadgar 8d6e28ace8 Merge branch 'master' into feature/2334 2017-05-31 14:27:07 -07:00
Alex Dadgar 044f1da5ff Merge pull request #2681 from hashicorp/b-deadlock 
Fix a deadlock relating to blocked allocations
 2017-05-31 14:26:54 -07:00
Alex Dadgar ec9cb2c751 Merge pull request #2672 from eyberg/master 
dont throw away errors in log rotation
 2017-05-31 14:14:22 -07:00
Alex Dadgar b1eea2269a Fix deadlock 2017-05-31 14:05:47 -07:00
Michael Schurter cb568a5cf6 Cleanup lots of leaked alloc runners in tests 2017-05-31 11:39:50 -07:00
Ulrik Mikaelsson 6138564f00 Implement support for docker-credential-helpers 
Solves: #2334
 2017-05-31 12:45:02 +02:00
Michael Schurter ffc2b36dc7 Merge pull request #2636 from hashicorp/f-gc-alloc-limit 
Add new gc_max_allocs tuneable
 2017-05-30 16:14:09 -07:00
Michael Schurter dd51aa1cb9 Merge pull request #2654 from hashicorp/f-env-consul 
Add envconsul-like support and refactor environment handling
 2017-05-30 14:40:14 -07:00
Michael Schurter e1a7c2d6d7 Fix Error -> Errorf 2017-05-30 12:08:59 -07:00
Michael Schurter 53d713bacb Fix getter tests 
And use an interface for ReplaceEnv since its all getter needs.
 2017-05-26 16:52:47 -07:00
Michael Schurter 51d8231911 Fix executor tests 2017-05-26 16:46:03 -07:00
Michael Schurter 3184616936 Always use PATH-only env for rkt commands 2017-05-26 15:41:26 -07:00
Michael Schurter 83543b1d80 Use custom TaskEnv to provide PATH for rkt 2017-05-26 15:24:14 -07:00
Michael Schurter 3afade9675 Let's pretend I never committed this 2017-05-26 15:06:59 -07:00
Michael Schurter 3eb0827c19 Fix and test multi-env-template loading 2017-05-25 17:13:33 -07:00
Alex Dadgar d279c1f5e6 Merge pull request #2675 from hashicorp/b-perms 
Fix perms to just set exec bit
 2017-05-25 14:45:08 -07:00
Alex Dadgar 28aef447e9 Fix perms to just set exec bit 2017-05-25 14:44:13 -07:00
Ian Eyberg 3fbc58997b dont throw away errors in log rotation 2017-05-25 11:49:33 -07:00
Michael Schurter 6d67d4dfbf Fix formatting 2017-05-25 09:38:49 -07:00
Michael Schurter 796d3250a5 Comment and correct formatting 2017-05-25 09:30:58 -07:00
Michael Schurter 5f9cb4c514 Switch tests to mock_driver 2017-05-25 09:28:10 -07:00
Michael Schurter ce8c4fa520 Fail fast on env template failures 2017-05-24 13:44:52 -07:00
Michael Schurter 4963cf07d7 Add env file test 2017-05-23 17:07:25 -07:00
Michael Schurter a233b0401d Add env testing 2017-05-23 16:46:29 -07:00
Michael Schurter d793dde4e9 Shrink chroot to avoid timing test failure 2017-05-23 16:11:24 -07:00
Michael Schurter 15ef740ab6 Add env.Builder.UpdateTask for alloc updates 2017-05-23 16:00:57 -07:00
Michael Schurter f324ca4683 Fix test data 2017-05-23 13:53:34 -07:00
Michael Schurter e7db2c9b0e Handle Driver.Prestart returning nil, nil 2017-05-23 13:53:34 -07:00
Michael Schurter fd9bef768f Move task env into execcontext 
Also inject PATH into rkt commands since we're no longer appending host
env vars for it.
 2017-05-23 13:53:34 -07:00
Michael Schurter 59ad200d1a Fix env var interpolation and env tests 2017-05-23 13:53:34 -07:00
Michael Schurter 37b148fb60 Add PortMap to struct returned by Driver.Prestart 
Moves env.Builder out of drivers entirely so one less thing to worry
about when implementing driver plugins.
 2017-05-23 13:53:34 -07:00
Michael Schurter 83c641ced0 Move env template handling into consul_template.go 2017-05-23 13:53:34 -07:00
Michael Schurter a2eb7d17c9 Improve PortMap handling and simplify Builder creation 2017-05-23 13:53:34 -07:00
Michael Schurter 8da110855e Move path building to task dir initialization 2017-05-23 13:53:34 -07:00
Michael Schurter d2c08ff24b Refactor TaskEnvironment into Builder and TaskEnv 2017-05-23 13:53:33 -07:00
Michael Schurter 10b6610e56 Functional consul template env file support 2017-05-23 13:45:14 -07:00
Brandon Fulljames 6d7caa5f6b Fix for test 2017-05-19 16:59:07 -07:00
Brandon Fulljames 6b2d5bd4fd Add SecurityOpt as a config field in Docker driver 2017-05-19 16:18:49 -07:00
Alex Dadgar 6d76fb2f17 Merge branch 'master' of github.com:hashicorp/nomad 2017-05-17 14:46:03 -07:00
Alex Dadgar ee8dd84965 Fix nil job on allocation 
The way the copying was happening on the alloc_runner was by temporarily
setting the alloc.Job to nil, copying and then restoring it. This
created an issue in which when the alloc was shared (which it is in
server/client mode and between alloc_runner/task_runner) there were race
conditions that could create a panic.

Fixes https://github.com/hashicorp/nomad/issues/2605
 2017-05-17 14:07:06 -04:00
Michael Schurter 06f937bf28 Merge pull request #2591 from hashicorp/b-2180-script-updates 
Properly interpolate services on updated tasks
 2017-05-17 09:09:01 -07:00
johannesa 323a0a78f3 Fixed wrong newline 2017-05-17 16:51:22 +02:00
johannesa ea644237cf mac address pinning in docker driver 
This commit adds mac address pining to the docker driver.
 2017-05-17 16:41:00 +02:00
Michael Schurter 3841692138 gc_max_allocs should include blocked & migrating 2017-05-12 16:03:22 -07:00
Michael Schurter 49ce86ee0a Lower default gc_max_allocs to 50 2017-05-12 15:57:27 -07:00
Michael Schurter 0453c2709c Add new gc_max_allocs tuneable 
More than gc_max_allocs may be running on a node, but terminal allocs
will be garbage collected to try to keep the total number below the
limit.
 2017-05-11 17:18:02 -07:00
Alex Dadgar 68c3a2bd98 Fix vet errors 2017-05-11 13:08:08 -07:00
Alex Dadgar 23685d65d0 Merge pull request #2610 from hashicorp/f-bolt-db 
Client persist state using bolt-db and more efficient write patterns
 2017-05-09 13:01:36 -07:00
Alex Dadgar 3cd7e06fba Fix test 2017-05-09 11:35:48 -07:00
Michael Schurter 3b15db1ac4 Ignore Consul deregister errors on executors 
Errors here only occur if Consul is not running when Nomad is restarted.

Errors here are only an issue if:
 * Consul is being used but is down or misbehaving
 * The executor is old (<0.6)
 * The task has services
 * The services hit a pre-0.6 consul.Syncer bug

If all of those conditions are met the pre-0.6 bugs will persist for
this task until Nomad is restarted.
 2017-05-09 11:28:27 -07:00
Alex Dadgar ba70cc4f01 Merge branch 'master' into f-bolt-db 2017-05-09 11:11:55 -07:00
Alex Dadgar 843bc26e5d Respond to comments 2017-05-09 10:50:24 -07:00
Michael Schurter 85210eb92f Update consul/api to support unix socket addrs 
Fixes #2594
 2017-05-08 11:57:04 -07:00
Michael Schurter 5b8415df2c Merge pull request #2585 from hashicorp/b-2554-container-exec 
Execute exec/java script checks in containers
 2017-05-05 10:31:18 -07:00
Michael Schurter 28e17b7a09 Test pre06ScriptCheck 2017-05-04 16:49:00 -07:00
Michael Schurter b11d1ad1ce Oops, remove dev logging 2017-05-04 16:27:04 -07:00
Michael Schurter 897b516117 Reuse ExecScript implementation 2017-05-04 16:21:40 -07:00
Michael Schurter f286b6b798 Build new env from new alloc before interpolating 2017-05-04 15:06:15 -07:00
Alex Dadgar 2d54ee2925 Fix tests 2017-05-03 15:14:19 -07:00
Kate Taggart 706e09748b fix test. 2017-05-03 12:45:59 -07:00
Kate Taggart 2fb6301b37 responding to feedback on PR: remove Region from Node struct, some grammatical niceties. 2017-05-03 12:45:59 -07:00
Kate Taggart af22cb722e I think I did it. 2017-05-03 12:45:59 -07:00
Kate Taggart 277d5ddb36 allow region to also be interpolated, like datacenter. 2017-05-03 12:45:59 -07:00
Kate Taggart 9fa97c38a0 just found out dc and region are going to be a bit different, as Region didn't previously exist in the Node struct. 2017-05-03 12:45:59 -07:00
Kate Taggart 1a5ebace03 add helper functions to set and clear dc, region names. 2017-05-03 12:45:59 -07:00
Kate Taggart 2062d5d1dc add dc and region to task env building function. flipped order of a couple functions to be consistent with the ordering of the rest of that section. 2017-05-03 12:45:59 -07:00
Kate Taggart e572ec8997 starting the jelly bean trail for two new env vars 2017-05-03 12:45:59 -07:00
Alex Dadgar 730e49a598 Helpful comment 2017-05-03 11:27:33 -07:00
Alex Dadgar 1d8444bc1e Fix tests 2017-05-03 11:15:30 -07:00
Alex Dadgar e00f9c9413 Restore state + upgrade path 2017-05-02 18:21:49 -07:00
Alex Dadgar ec101b4760 Revert "metrics" 
This reverts commit 4d6a012c6fb6f1fba6c62985d091b1a20c3198e7.
 2017-05-02 09:28:11 -07:00
Alex Dadgar 8e516b5dc2 Async and sync saving of client state 2017-05-01 16:16:53 -07:00
Alex Dadgar a7fd08d42a perf 2017-05-01 16:01:50 -07:00
Alex Dadgar e010fdf8c0 metrics 2017-05-01 14:51:27 -07:00
Alex Dadgar d779defe65 Use batching 2017-05-01 14:50:34 -07:00
Alex Dadgar b94f855326 boltDB database for client state 2017-05-01 14:50:34 -07:00
Alex Dadgar bddedd7aba Don't deepcopy job when retrieving copy of Alloc 
This PR removes deepcopying of the job attached to the allocation in the
alloc runner. This operation is called very often so removing reflect
from the code path and the potentially large number of mallocs need to
create a job reduced memory and cpu pressure.
 2017-05-01 14:50:34 -07:00
Pete Wildsmith 6a09d5300f address feedback 2017-04-28 10:27:37 +01:00
Pete Wildsmith 418d59fd00 clean up consul earlier when destroying a task 2017-04-27 23:29:30 +01:00
Michael Schurter cafefa049b Properly interpolate services on updated tasks 
Previously was interpolating the original task's services again.

Fixes #2180

Also fixes a slight memory leak in the new consul agent. Script check
handles weren't being deleted after cancellation.
 2017-04-26 11:22:01 -07:00
Michael Schurter 20322a5e92 Test pre-0.6 script check upgrade path 2017-04-25 11:41:03 -07:00
Michael Schurter b965708c51 Test env+cgroups for exec driver checks 2017-04-25 11:13:06 -07:00
Michael Schurter c069ca0bfe Change raw_exec to use simplified exec wrapper 2017-04-21 16:50:20 -07:00
Michael Schurter 095d2ee340 Switch java/exec to use Exec in Executor 2017-04-21 16:25:49 -07:00
Michael Schurter a305b68159 Restart tasks on upgrade with script checks and old executors 2017-04-21 16:25:49 -07:00
Michael Schurter a63162c7ff Fix Windows build. 2017-04-19 13:16:48 -07:00
Michael Schurter 83f9591d75 Thanks go vet! 2017-04-19 13:05:41 -07:00
Michael Schurter c8d3e869c6 Unregister from Consul when waiting for restart 2017-04-19 12:42:48 -07:00
Michael Schurter 16ac08ac8c Remove stale comment 2017-04-19 12:42:47 -07:00
Michael Schurter a3fc157233 Fix circular test imports 2017-04-19 12:42:47 -07:00
Michael Schurter caf317e3f2 Use a DriverAbility to expose Exec functionality 2017-04-19 12:42:47 -07:00
Michael Schurter 0a59982936 Move removal from Consul into TaskRunner cleanup 2017-04-19 12:42:47 -07:00
Michael Schurter 745ad9521f Move ScriptExecutor to driver 2017-04-19 12:42:47 -07:00
Michael Schurter 244251490a Add UpdateTask method instead of Remove/Add 2017-04-19 12:42:47 -07:00
Michael Schurter e204a287ed Refactor Consul Syncer into new ServiceClient 
Fixes #2478 #2474 #1995 #2294

The new client only handles agent and task service advertisement. Server
discovery is mostly unchanged.

The Nomad client agent now handles all Consul operations instead of the
executor handling task related operations. When upgrading from an
earlier version of Nomad existing executors will be told to deregister
from Consul so that the Nomad agent can re-register the task's services
and checks.

Drivers - other than qemu - now support an Exec method for executing
abritrary commands in a task's environment. This is used to implement
script checks.

Interfaces are used extensively to avoid interacting with Consul in
tests that don't assert any Consul related behavior.
 2017-04-19 12:42:47 -07:00
Michael Schurter 4b18f916d9 Set ownership on directories in chroot 
Also support getOwner on all Unixes as they all have `Stat_t.{U,G}id`
 2017-04-17 12:41:33 -07:00
Michael Schurter 83fb374d0b Don't disable hardlinking! 2017-04-17 11:03:15 -07:00
Michael Schurter 547d5a81b4 Chown files when copying into chroot 
Fixes #2552

Not needed when hardlinking. Only adds Linux support but other OS's may
be easy.
 2017-04-17 11:03:15 -07:00
Alex Dadgar 61f4a2dac6 Sync allocation state before waiting for a destroy 
This change ensures that the client syncs allocation state with the
servers before entering its wait loop for the allocation to be
destroyed.

Fixes https://github.com/hashicorp/nomad/issues/2563
 2017-04-14 13:09:54 -07:00
Alex Dadgar 67d29f684d Merge pull request #2519 from hashicorp/b-sticky-tmp 
Add sticky bit to temp directory
 2017-04-12 14:59:14 -07:00
Alex Dadgar ad5fea9875 Merge pull request #2541 from hashicorp/f-stable-distributed-id 
Hash host ID so its stable and well distributed
 2017-04-11 11:27:53 -07:00
Alex Dadgar cdd624ff5b Add ExtraHosts to Docker driver 
This PR allows job submitters to add extra hosts to the containers
/etc/hosts file.

Fixes https://github.com/hashicorp/nomad/issues/2546
 2017-04-11 10:52:41 -07:00
Diptanu Choudhury a96018fccc Fixed typo 2017-04-10 11:45:11 -07:00
Alex Dadgar 2321e8a4a0 Hash host ID so its stable and well distributed 
This PR takes the host ID and runs it through a hash so that it is well
distributed. This makes it so that machines that report similar host IDs
are easily distinguished.

Instances of similar IDs occur on EC2 where the ID is prefixed and on
motherboards created in the same batch.

Fixes https://github.com/hashicorp/nomad/issues/2534
 2017-04-10 11:44:51 -07:00
Diptanu Choudhury 45ad1298e6 Added a test 2017-04-08 13:44:21 -07:00
Alex Dadgar eed4a9124e Merge branch 'f-ipv6-fingerprint' of github.com:hashicorp/nomad into f-ipv6-fingerprint 2017-04-07 18:37:19 -07:00
Diptanu Choudhury 353f193983 Removed redundant code 2017-04-07 18:28:22 -07:00
Alex Dadgar 4e3e008864 Merge branch 'master' into f-ipv6-fingerprint 2017-04-07 17:11:24 -07:00
Diptanu Choudhury 963b9baf14 Ignoring link local addresses 2017-04-07 16:04:36 -07:00
Alex Dadgar 81188906a5 Merge pull request #2535 from clinta/docker-ip 
Allow specifying container IP with docker driver
 2017-04-07 12:59:12 -07:00
Clint Armstrong d71ddcb756 Allow specifying container IP with docker driver 2017-04-07 11:56:07 -04:00
Michael Schurter aede1478db Create AssertUntil helper func 2017-04-06 17:05:09 -07:00
Diptanu Choudhury b9e71f4349 Finding the appropriate cidr block 2017-04-06 16:04:59 -07:00
Michael Schurter 521354ee99 Improve test timings 
1234ms was far longer than needed and not sleeping between iterations
was just mean.
 2017-04-06 11:10:36 -07:00
Michael Schurter a81c387adf Require TLS for server RPC when enabled 
Fixes #2525

We used to be checking a RequireTLS field that was never set. Instead we
can just check the TLSConfig.EnableRPC field and require TLS if it's
enabled.

Added a few unfortunately slow integration tests to assert the intended
behavior of misconfigured RPC TLS.

Also disable a lot of noisy test logging when -v isn't specified.
 2017-04-06 09:34:36 -07:00
Diptanu Choudhury 0944d9d640 Making the fingerprinter support ipv6 ips 2017-04-06 01:06:32 -07:00
Alex Dadgar d0da30324e Split driver tests 2017-04-04 15:12:14 -07:00
Alex Dadgar f95af65c75 Split task runner tests 2017-04-04 15:02:09 -07:00
Alex Dadgar f2ba6db607 Add sticky bit to temp directory 
Fixes an issue where the Ruby runtime expects the sticky bit to be set
on the temp directory. The sticky bit is commonly set on the temp
directory since it is usually shared by many users. This change brings
ours in line with that assumption.
 2017-04-04 10:48:29 -07:00
Alex Dadgar c52000f792 FinishedAt only records when the task has actually started 2017-03-31 17:06:05 -07:00
Alex Dadgar 81b78f77e1 Track task start/finish time & improve logs errors 
This PR adds tracking to when a task starts and finishes and the logs
API takes advantage of this and returns better errors when asking for
logs that do not exist.
 2017-03-31 16:14:11 -07:00
Alex Dadgar 6bee23047a Fix variable capture and add tests 
This PR fixes token revocation and adds tests to make sure it is
working. The 0.5.6 RC1's token revocation does not work becasue the
token's value is captured at the instantiation of the deferred
stoprenewal statement rather than its exectution.
 2017-03-29 13:17:50 -07:00
Robert Neumayer 804f83f1fb Fix typo and be consistent in naming 2017-03-29 09:33:54 +02:00
Michael Schurter ae3810052d Merge pull request #2482 from hashicorp/f-2289-better-artifact-err 
Improve artifact download error message
 2017-03-28 12:48:22 -07:00
Michael Schurter 2d07b75308 Recoverable struct field must be exported 2017-03-28 12:18:57 -07:00
Michael Schurter 1653b42400 Merge branch 'master' into b-2491-task-env 2017-03-28 11:28:00 -07:00
Alex Dadgar 1e95ae7e6a Merge pull request #2495 from hashicorp/b-vault-stop-renew 
Stop Vault token renew on task exit
 2017-03-28 11:14:18 -07:00
Alex Dadgar 5e7e19de4b Merge pull request #2461 from hashicorp/b-groups 
Various fixes for setting user/group of task
 2017-03-28 11:13:27 -07:00
Alex Dadgar 3b493b29b0 Merge pull request #2488 from hashicorp/f-node-values-template 
Access Node Meta and Attrs in template
 2017-03-28 11:10:29 -07:00
Alex Dadgar 5b36a90723 feedback 2017-03-28 11:10:11 -07:00
Michael Schurter d58bcde183 Make test more accurate and add changelog entry 2017-03-28 10:53:26 -07:00
Alex Dadgar d1645f47b1 Stop Vault token renew on task exit 
This PR fixes an oversight in which the client would attempt to renew a
token even after the task exits.

Fixes https://github.com/hashicorp/nomad/issues/2475
 2017-03-28 10:53:15 -07:00
Michael Schurter 9a0c66b55b Don't take a reference a var in a loop 
Fixes #2491
 2017-03-28 10:16:27 -07:00
Michael Schurter 41dcc72033 Recoverable is a method now, not a field 2017-03-27 15:41:35 -07:00
Michael Schurter 507862ade3 Add WrapRecoverable helper 2017-03-27 15:37:15 -07:00
Alex Dadgar 23562d77c9 Access Node Meta and Attrs in template 
This PR allows accessing the Node's attributes and metadata as in a
template.

```
template {
    data = "{{ env \"attr.unique.network.ip-address\" }}"
    destination = "local/out"
}
```
 2017-03-27 15:15:33 -07:00
Alex Dadgar 4ecebe7d8c Proper reference counting through task restarts 
This PR fixes an issue in which the reference count on a Docker image
would become inflated through task restarts.
 2017-03-25 17:05:53 -07:00
Michael Schurter 0e6c564406 Improve artifact download error message 
Fixes #2289

Unfortunately took more RecoverableError hijinx than I would have liked.
There might be a better way.
 2017-03-24 15:26:05 -07:00
Alex Dadgar 7e6c08191d Fix vet 2017-03-24 12:24:47 -07:00
Alex Dadgar c3551c761e Fix panic when restarting non-running task 
This PR fixes an issue that is hit when running templates with restart
mode in which the client could panic when the handle is not running.

Fixes https://github.com/hashicorp/nomad/issues/2479
 2017-03-24 12:04:22 -07:00
Alex Dadgar a171a014b3 Various fixes for setting user/group of task 
This PR fixes two issues:
* Folder permissions in -dev mode were incorrect and not suitable for
running as a particular user.
* Was not setting the group membership properly for the launched
process.

Fixes https://github.com/hashicorp/nomad/issues/2160
 2017-03-20 14:21:13 -07:00