Commit Graph

13853 Commits

Author SHA1 Message Date
Michael Schurter d686ad51fb
Merge pull request #5043 from hashicorp/b-taskenv-conflicts
taskenv: have maps take precedence over primitives
2019-01-07 12:34:48 -08:00
Chris Baker f99e18aaf4 gofmt to make check happy 2019-01-07 18:01:59 +00:00
Chris Baker 6d279f57ad updated CHANGELOG to note backward incompatibility in node metadata validation 2019-01-07 17:32:45 +00:00
Chris Baker a61afad5bb added validation on client metadata keys 2019-01-07 17:16:38 +00:00
Danielle Tomlinson 023223c782
Merge pull request #5148 from hashicorp/dani/f-e2e-consul
e2e: Add consultemplate test
2019-01-07 17:54:20 +01:00
Danielle Tomlinson d195680ec1 e2e: Add consultemplate test
This adds a basic test for consul template, that verifies the behaviour
of consul-template with task blocking and restarting of tasks
2019-01-07 17:53:55 +01:00
Preetha ec39dd2383
Merge pull request #5156 from hashicorp/f-standard-driver-health-messages
Standardize driver health description messages for all drivers
2019-01-07 09:49:28 -06:00
Mahmood Ali 58fb6812db tests: busybox only depends on arch
Busybox is compiled for linux only.  Making the file used in executor
tests even for non-linux targets, as having the file present has no
side-effects.
2019-01-07 08:36:32 -05:00
Mahmood Ali 0ba7b0c132 tests: helper function for checking docker presense 2019-01-07 08:27:06 -05:00
Mahmood Ali 796d625ab6 Skip tests requiring Docker deamon if not found. 2019-01-07 07:59:13 -05:00
Preetha Appan 2fb2de3cef
Standardize driver health description messages for all drivers 2019-01-06 22:06:38 -06:00
Preetha f5b71ee5c9
Merge pull request #5155 from hashicorp/b-remove-rkt-logging
remove unnecessary logging in rkt driver fingerprint method
2019-01-06 21:37:58 -06:00
Preetha Appan 76c09c7cbf
remove unnecessary logging in rkt driver fingerprint method 2019-01-06 20:59:20 -06:00
Mahmood Ali 8797a4f0ea drivers/exec: restrict devices exposed to tasks
We ultimately decided to provide a limited set of devices in exec/java
drivers instead of all of host ones.  Pre-0.9, we made all host devices
available to exec tasks accidentally, yet most applications only use a
small subset, and this choice limits our ability to restrict/isolate GPU
and other devices.

Starting with 0.9, by default, we only provide the same subset of
devices Docker provides, and allow users to provide more devices as
needed on case-by-case basis.

This reverts commit 5805c64a9f1c3b409693493dfa30e7136b9f547b.
This reverts commit ff9a4a17e59388dcab067949e0664f645b2f5bcf.
2019-01-06 17:03:19 -05:00
Danielle Tomlinson c13dc7f110
Merge pull request #5149 from hashicorp/dani/e2e-friendly
e2e: Output setup instructions after terraform
2019-01-04 22:14:03 +01:00
Mahmood Ali 91d761e832
Merge pull request #5150 from hashicorp/b-task-received-event
taskrunner: emit TaskReceived event
2019-01-04 14:58:39 -05:00
Mahmood Ali cd3c6cf60b taskrunner: emit TaskReceived event
Preserve pre-0.9, where task runner emits `Received: Task received by
client` event on task runner creation.
2019-01-04 14:32:29 -05:00
Chris Baker e0d8b96b5b
Merge pull request #5146 from hashicorp/b-1173-log-spam
updated to latest hashicorp/raft and hashicorp/memberlist to pull
2019-01-04 12:55:18 -05:00
Chris Baker bd0f2c854a Merge branch 'master' of github.com:hashicorp/nomad into b-1173-log-spam 2019-01-04 16:56:34 +00:00
Chris Baker 4109306fff updated serf along with raft 2019-01-04 16:56:26 +00:00
Danielle Tomlinson 33547c99e7 e2e: Output setup instructions after terraform
This adds a message that provides environment setup instructions for
running e2e tests after running terraform apply.

This allows copy/pasting exports, rather than manually constructing
them.
2019-01-04 16:55:14 +01:00
Mahmood Ali 56e3171310
driver/exec: use dedicated /dev mount (#5147)
Use a dedicated /dev mount so we can inject more devices if necessary,
and avoid allowing a container to contaminate host /dev.

Follow up to https://github.com/hashicorp/nomad/pull/5143 - and fixes master.
2019-01-04 10:36:05 -05:00
Chris Baker e2e83feb5d Merge branch 'master' of github.com:hashicorp/nomad into b-1173-log-spam 2019-01-04 14:59:26 +00:00
Chris Baker 11c4496a4d updated to latest hashicorp/raft and hashicorp/memberlist to pull
upstream fixes for accept-loop error handling
2019-01-04 14:01:36 +00:00
Mahmood Ali a9f6288fe3
Merge pull request #5143 from hashicorp/b-exec-regressions
Restore some `exec` behavior
2019-01-04 08:58:40 -05:00
Mahmood Ali 606ab23235 goimport file 2019-01-04 08:53:50 -05:00
Preetha Appan 378dd74d2a
Added waiting on client node ready state before running e2e tests 2019-01-03 16:16:20 -06:00
Preetha 1e69a6645f
Update README.md 2019-01-03 16:15:59 -06:00
Preetha 5501ff42c9
Update README.md 2019-01-03 15:31:19 -06:00
Michael Schurter 875e231511
Merge pull request #5038 from hashicorp/b-drivermanager-tests
WIP: fix failing tests caused by async driver manager
2019-01-03 12:32:18 -08:00
Preetha Appan f458cb63dd
Increase alloc wait timeout in e2e test 2019-01-03 14:02:02 -06:00
Mahmood Ali 5b0702c9eb drivers/exec: bind mount /dev into rootfs
Restores pre-0.9 behavior, where Nomad makes /dev available to exec
task.  Switching to libcontainer, we accidentally made only a small
subset available.

Here, we err on the side of preserving behavior of 0.8, instead of going
for the sensible route, where only a reasonable subset of devices is
mounted by default and user can opt to request more.
2019-01-03 14:29:18 -05:00
Mahmood Ali d23f47736c drivers/exec: run as `nobody` by default
libcontainer based drivers (e.g. exec, java) should default to running
processes as `nobody` [1]; but libcontainer treats empty user as `root`
in our case (either because of default or due to `root` being current
user).

[1] 94c28a4c6c/website/source/docs/job-specification/task.html.md (task-parameters)
2019-01-03 14:29:18 -05:00
Preetha 9e235f4cb6
Update e2e readme 2019-01-03 13:24:58 -06:00
Preetha 0071307414
Update README.md 2019-01-03 13:19:04 -06:00
Preetha 758ae0ca7c
Update README.md 2019-01-03 12:12:43 -06:00
Danielle Tomlinson 35a4790740
Merge pull request #5142 from hashicorp/dani/cleanup-allocrunner-logs
allocrunner: Standardised discard logs
2019-01-03 18:40:48 +01:00
Preetha Appan d182c0f5cd
Increase timeout in e2e test 2019-01-03 11:22:21 -06:00
Preetha 8078cb79f0
Merge pull request #5140 from hashicorp/dani/b-taskrunner
taskrunner: Persist environment from hooks
2019-01-03 09:30:52 -06:00
Danielle Tomlinson 4a944e114e
Merge pull request #5141 from hashicorp/dani/fix-static-checks
Fix `make check`
2019-01-03 14:40:25 +01:00
Danielle Tomlinson 29196ca70e allocrunner: Standardised discard logs
Follow up from https://github.com/hashicorp/nomad/pull/5007#pullrequestreview-186739124
2019-01-03 14:04:31 +01:00
Danielle Tomlinson d3b41a26c4 e2e: goimports e2eutil/utils.go 2019-01-03 13:31:49 +01:00
Danielle Tomlinson 1c8baf7db7 chore: Fix environement->environment typo 2019-01-03 13:31:30 +01:00
Danielle Tomlinson 43f2dc0c36 chore: Fix environement->environment typo 2019-01-03 13:31:26 +01:00
Danielle Tomlinson 45174ac3e9
Merge pull request #5041 from hashicorp/dani/b-driver-healt
drivers: Cleanup root user fingerprinting
2019-01-03 13:16:28 +01:00
Danielle Tomlinson 28aa34ea78 taskrunner: Persist environment from hooks
https://github.com/hashicorp/nomad/pull/5032 introduced a regression
where the origHookState was used in place of the response from the hook.
2019-01-03 13:13:57 +01:00
Preetha ef3dfa36e3
Merge pull request #5037 from hashicorp/f-e2e-spread
e2e tests for spread stanza
2018-12-20 15:39:08 -06:00
Alex Dadgar d7d32c2f61
Merge pull request #5032 from hashicorp/f-driver-env
Store device envs separately and pass to drivers
2018-12-20 13:38:27 -08:00
Preetha Appan 2845a556a3
Clean up map update code 2018-12-20 15:12:48 -06:00
Michael Schurter e47a3ceed6 taskenv: have maps take precedence over primitives
**The Bug:**

You may have seen log lines like this when running 0.9.0-dev:

```
... client.alloc_runner.task_runner: some environment variables not available for rendering: ... keys="attr.driver.docker.volumes.enabled, attr.driver.docker.version, attr.driver.docker.bridge_ip, attr.driver.qemu.version"
```

Not only should we not be erroring on builtin driver attributes, but the
results were nondeterministic due to map iteration order!

The root cause is that we have an old root attribute for all drivers
like:

```
attr.driver.docker = "1"
```

When attributes were opaque variable names it was fine to also have
"nested" attributes like:

```
attr.driver.docker.version = "1.2.3"
```

However in the HCLv2 world the variable names are no longer opaque: they
form an object tree. The `docker` object can no longer both hold a value
(`"1"`) *and* nested attributes (`version = "1.2.3"`).

**The Fix:**

Since the old `attr.driver.<name> = "1"` attribues are useless for task
config interpolation, create a new precedence rule for creating the task
config evaluation context:

*Maps take precedence over primitives.*

This means `attr.driver.docker.version` will always take precedence over
`attr.driver.docker`. The results are determinstic and give users access
to the more useful metadata.

I made this a general precedence rule instead of special-casing driver
attrs because it seemed like better default behavior than spamming
WARNings to logs that were likely unactionable by users.
2018-12-20 11:37:46 -08:00