Commit graph

382 commits

Author SHA1 Message Date
Mahmood Ali df2b579c6b driver/docker: collect tty container logs
Fixes https://github.com/hashicorp/nomad/issues/5475

When container is a tty container, we need to get raw terminal output
without any additional processing.
2019-04-24 22:01:51 -04:00
Mahmood Ali 6014a884be comment on using init() for libcontainer handling 2019-04-19 09:49:04 -04:00
Mahmood Ali 4322055301 comment what refer to 2019-04-19 09:49:04 -04:00
Mahmood Ali 18993421f2 Move libcontainer helper to executor package 2019-04-19 09:49:04 -04:00
Danielle Lancashire a096a7f112 Switch to pre-0.9 behaviour for handling volumes
In Nomad 0.9, we made volume driver handling the same for `""`, and
`"local"` volumes. Prior to Nomad 0.9 however these had slightly different
behaviour for relative paths and named volumes.

Prior to 0.9 the empty string would expand relative paths within the task
dir, and `"local"` volumes that are not absolute paths would be treated
as docker named volumes.

This commit reverts to the previous behaviour as follows:

| Nomad Version | Driver  |   Volume Spec    | Behaviour                 |
|-------------------------------------------------------------------------
| all           | ""      | testing:/testing | allocdir/testing          |
| 0.8.7         | "local" | testing:/testing | "testing" as named volume |
| 0.9.0         | "local" | testing:/testing | allocdir/testing          |
| 0.9.1         | "local" | testing:/testing | "testing" as named volume |
2019-04-18 14:28:45 +02:00
Mahmood Ali 01a13a0947 locking and opening streams in goroutine comment 2019-04-16 11:02:19 -04:00
Mahmood Ali 357b86adc3 open fifo on background goroutine 2019-04-15 21:20:09 -04:00
Mahmood Ali 9e48fab225
Merge pull request #5537 from hashicorp/b-nonvidia-flag
Allow compiling without nvidia integration
2019-04-10 13:40:55 -04:00
Mahmood Ali b4d84fd6a9 Allow compiling without nvidia integration
nvidia library use of dynamic library seems to conflict with alpine and
musl based OSes.  This adds a `nonvidia` tag to allow compiling nomad
for alpine images.

The nomad releases currently only support glibc based OS environments,
so we default to compiling with nvidia.
2019-04-10 09:19:12 -04:00
Mahmood Ali f7d39d6186 ci: move docker unix-y test
Fix AppVeyor failing builds, by moving docker image url test to run on unix
systems only.  The used paused image is a linux image only, not
available on Windows.
2019-04-09 19:59:58 -04:00
Nick Ethier 4bbdb80b73
drivers/docker: fix image name handleing when prefixed with https:// 2019-04-04 22:10:18 -04:00
Mahmood Ali 77a5edd3ae an alternative order 2019-04-02 20:00:54 -04:00
Mahmood Ali 334c6e9f5f try not without checking stat first 2019-04-02 19:55:44 -04:00
Mahmood Ali 17df86acda basic test for #4809 2019-04-02 19:50:35 -04:00
Michael Schurter 923cd91850
Merge pull request #5504 from hashicorp/b-exec-path
executor/linux: make chroot binary paths absolute
2019-04-02 14:09:50 -07:00
Michael Schurter 47bed4316f executor/linux: comment this bizarre code 2019-04-02 11:25:45 -07:00
Michael Schurter 1d569a27dc Revert "executor/linux: add defensive checks to binary path"
This reverts commit cb36f4537e63d53b198c2a87d1e03880895631bd.
2019-04-02 11:17:12 -07:00
Michael Schurter fc5487dbbc executor/linux: add defensive checks to binary path 2019-04-02 09:40:53 -07:00
Michael Schurter ff639f9ccc docker: improve stats names and comments 2019-04-02 09:18:38 -07:00
Mahmood Ali e2723399f3
Update drivers/docker/stats.go comment
Co-Authored-By: schmichael <michael.schurter@gmail.com>
2019-04-02 09:09:17 -07:00
Michael Schurter df3467a6ce docker: fix send after close panic in stats
destCh was being written to by one goroutine and closed by another
goroutine. This panic occurred in Travis:

```
=== FAIL: drivers/docker TestDockerCoordinator_ConcurrentPulls (117.66s)
=== PAUSE TestDockerCoordinator_ConcurrentPulls
=== CONT  TestDockerCoordinator_ConcurrentPulls

panic: send on closed channel

goroutine 5358 [running]:
github.com/hashicorp/nomad/drivers/docker.dockerStatsCollector(0xc0003a4a20, 0xc0003a49c0, 0x3b9aca00)
	/home/travis/gopath/src/github.com/hashicorp/nomad/drivers/docker/stats.go:108 +0x167

created by
github.com/hashicorp/nomad/drivers/docker.TestDriver_DockerStatsCollector
	/home/travis/gopath/src/github.com/hashicorp/nomad/drivers/docker/stats_test.go:33 +0x1ab
```

The 2 ways to fix this kind of error are to either (1) add extra
coordination around multiple goroutines writing to a chan or (2) make it
so only one goroutines writes to a chan.

I implemented (2) first as it's simpler, but @notnoop pointed out since
the same destCh in reused in the stats loop there's now a double close
panic possible!

So this implements (1) by adding a *usageSender struct for handling
concurrent senders and closing.
2019-04-02 08:28:08 -07:00
Michael Schurter 7d49bc4c71 executor/linux: make chroot binary paths absolute
Avoid libcontainer.Process trying to lookup the binary via $PATH as the
executor has already found where the binary is located.
2019-04-01 15:45:31 -07:00
Mahmood Ali cb16ad7e3f comment configureTLogging 2019-04-01 16:52:58 -04:00
Mahmood Ali 81f4f07ed7 rename fifo methods for clarity 2019-04-01 16:52:58 -04:00
Mahmood Ali 88dc4a255a avoid opening files just to close them 2019-04-01 13:24:18 -04:00
Mahmood Ali dac2cd3df3 Add test cases for waiting on children
Also, make the test use files just like in the non-test case.
2019-04-01 13:24:18 -04:00
Mahmood Ali 2648d4176c
Merge pull request #5468 from hashicorp/b-test-exec-kill-wait
Adding tests for killing task without grace
2019-03-31 20:37:15 -04:00
Michael Schurter b8d1dd95a0
Update drivers/shared/executor/executor_test.go
Co-Authored-By: notnoop <mahmood@notnoop.com>
2019-03-31 20:34:24 -04:00
Michael Schurter 294d405397 docker: restore pre-0.9 container names
As far as I can tell Nomad itself does not use the container name after
container creation, so this should be safe.

OP: https://groups.google.com/d/topic/nomad-tool/kYkyERfVRXE/discussion
v0.8.7 code: https://github.com/hashicorp/nomad/blob/v0.8.7/client/driver/docker.go#L1530-L1531
2019-03-29 13:55:43 -07:00
Mahmood Ali df5d7ba50d fix test setup 2019-03-26 09:15:22 -04:00
Mahmood Ali d737a9836a test kill wait 2019-03-26 09:15:22 -04:00
Mahmood Ali b2b7618a1c
clarify unknown signal log line (#5466) 2019-03-25 17:19:43 -04:00
Mahmood Ali 2a7b18aec4
Revert "executor: synchronize exitState accesses" (#5449)
Reverts hashicorp/nomad#5433

Apparently, channel communications can constitute Happens-Before even for proximate variables, so this syncing isn't necessary.

> _The closing of a channel happens before a receive that returns a zero value because the channel is closed._
https://golang.org/ref/mem#tmp_7
2019-03-20 07:33:05 -04:00
Nick Ethier 505e36ff7a
Merge pull request #5429 from hashicorp/b-blocking-executor-shutdown
executor: block shutdown on process exiting
2019-03-19 15:18:01 -04:00
Mahmood Ali a1776dba34 executor: synchronize exitState accesses
exitState is set in `wait()` goroutine but accessed in a different
`Wait()` goroutine, so accesses must be synchronized by a lock.
2019-03-17 11:56:58 -04:00
Nick Ethier 7418d09cf0
executor: block shutdown on process exiting 2019-03-15 23:50:17 -04:00
Mahmood Ali 8ec49fc133
Handle when cannot fetch docker logs (#5420)
Fix #5418

When using a docker logger that doesn't support log streaming through
API, currently docker logger runs a tight loop of Docker API calls
unexpectedly. This change ensures we stop fetching logs early.

Also, this adds some basic backoff strategy when Docker API logging
fails unexpectedly, to avoid accidentally DoSing the docker daemon.
2019-03-14 16:23:11 -04:00
Mahmood Ali fb55717b0c
Regenerate Proto files (#5421)
Noticed that the protobuf files are out of sync with ones generated by 1.2.0 protoc go plugin.

The cause for these files seem to be related to release processes, e.g. [0.9.0-beta1 preperation](ecec3d38de (diff-da4da188ee496377d456025c2eab4e87)), and [0.9.0-beta3 preperation](b849d84f2f).

This restores the changes to that of the pinned protoc version and fails build if protobuf files are out of sync.  Sample failing Travis job is that of the first commit change: https://travis-ci.org/hashicorp/nomad/jobs/506285085
2019-03-14 10:56:27 -04:00
Preetha Appan 7f0d9e0c8e
minor review feedback 2019-03-13 13:27:28 -05:00
Preetha Appan 273f1e993d
Validate all auth fields being empty rather than just email
This fixes a regression in 0.9 beta3 compared to 0.8.7 in validating
docker auth config
2019-03-13 11:47:37 -05:00
Preetha Appan 549ae657f0
Dont require email address for docker auth 2019-03-13 11:08:56 -05:00
Preetha 7759166b0d
Merge pull request #5380 from quasilyte/patch-1
drivers/shared/executor: fix strings.Replace call
2019-03-06 11:47:01 -06:00
Mahmood Ali bb32ba8784
Support driver config fields being set to nil (#5391)
To pick up https://github.com/hashicorp/hcl2/pull/90
2019-03-05 21:47:06 -05:00
Iskander (Alex) Sharipov e69909fbd3
drivers/shared/executor: fix strings.Replace call
strings.Replace call with n=0 argument makes no sense
as it will do nothing. Probably -1 is intended.

Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
2019-03-02 00:33:17 +03:00
Mahmood Ali 4726cb2207 logging.Type over logging.Driver 2019-02-28 16:40:18 -05:00
Mahmood Ali 104869c0e1 drivers/docker: rename logging type to driver
Docker uses the term logging `driver` in its public documentations: in
`docker` daemon config[1], `docker run` arguments [2] and in docker compose file[3].
Interestingly, docker used `type` in its API [4] instead of everywhere
else.

It's unfortunate that Nomad used `type` modeling after the Docker API
rather than the user facing documents.  Nomad using `type` feels very
non-user friendly as it's disconnected from how Docker markets the flag
and shows internal representation instead.

Here, we rectify the situation by introducing `driver` field and
prefering it over `type` in logging.

[1] https://docs.docker.com/config/containers/logging/configure/
[2] https://docs.docker.com/engine/reference/run/#logging-drivers---log-driver
[3] https://docs.docker.com/compose/compose-file/#logging
[4] https://docs.docker.com/engine/api/v1.39/#operation/ContainerCreate
2019-02-28 16:04:03 -05:00
Mahmood Ali 67e2a0ac05
docker: report unhealthy in unsupported Windows (#5356)
On Windows, Nomad only supports Windows containers, so report as
unhealthy otherwise.
2019-02-27 08:10:23 -05:00
Michael Schurter 812f1679e2
Merge pull request #5352 from hashicorp/b-leaked-logmon
logmon fixes
2019-02-26 08:35:46 -08:00
Danielle Tomlinson e250aad31b
Merge pull request #5355 from hashicorp/dani/windows-dockerstats
docker: Support Stats on Windows
2019-02-26 16:39:48 +01:00
Danielle Tomlinson e3dc80bea3 docker: Return undetected before first detection
This commit causes the docker driver to return undetected before it
first establishes a connection to the docker daemon.

This fixes a bug where hosts without docker installed would return as
unhealthy, rather than undetected.
2019-02-25 11:02:42 +01:00