Commit Graph

637 Commits

Author SHA1 Message Date
Nick Ethier b34db8b3b6 nit: code cleanup/organization 2021-04-16 15:14:29 -04:00
Nick Ethier 07dca26f0d qemu: set the number of cores equal to the number of reserved cores if set 2021-04-15 13:32:33 -04:00
Nick Ethier c9216ba7d9 drivers/docker: move cgroups logic to linux build file 2021-04-15 10:39:11 -04:00
Nick Ethier 390c4c5119 docker: add support for cpuset cgroup management 2021-04-15 10:24:31 -04:00
Nick Ethier fe283c5a8f executor: add support for cpuset cgroup 2021-04-15 10:24:31 -04:00
Nick Ethier b6b74a98a9 client/fingerprint: move existing cgroup concerns to cgutil 2021-04-13 13:28:36 -04:00
Yoan Blanc ac0d5d8bd3
chore: bump golangci-lint from v1.24 to v1.39
Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2021-04-03 09:50:23 +02:00
Mahmood Ali 9ff7220588 reuse existing function and typo fix 2021-04-02 11:56:27 -04:00
Mahmood Ali 565496e6ba drivers/docker: account for cgroup-v2 memory stats
If the docker engine is running on cgroup-v2 host, then RSS and Max
Usage doesn't get reported.

Using a heauristic here to avoid adding more API calls to the Docker
Engine to infer cgroups version. Also, opted to avoid coordinating stats
collection with fingerprinting, which adds concurrency complexities.
2021-04-01 12:23:57 -04:00
Mahmood Ali edec658e50 drivers/exec: Account for cgroup-v2 memory stats
If the host is running with cgroup-v2, RSS and Max Usage doesn't get
reported anymore.
2021-04-01 12:13:21 -04:00
Tim Gross e76eeeb848 drivers/docker: fix flaky image coordinator test
The test assertion that we don't have a delete future remaining races with the
code its testing, because the removal of the image and the removal of the
future are not atomic. Move this assertion into a `WaitForResult` to avoid
test flakes which we're seeing on CI on Windows in particular.
2021-03-31 15:59:01 -04:00
zhsj 5a182e1d03
deps: update runc to v1.0.0-rc93
includes updates for breaking changes in runc v1.0.0-rc93
2021-03-31 10:57:02 -04:00
Mahmood Ali bf1c0dcf17 driver/exec: set soft memory limit
Linux offers soft memory limit:
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#soft-limits
, and
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html?highlight=memory.low
.

We can set soft memory limits through libcontainer
`Resources.MemoryReservation`: https://pkg.go.dev/github.com/opencontainers/runc@v0.1.1/libcontainer/configs#Resources
2021-03-30 16:55:58 -04:00
Mahmood Ali f44a04454d oversubscription: driver/exec to honor MemoryMaxMB 2021-03-30 16:55:58 -04:00
Mahmood Ali 275feb5bec oversubscription: docker to honor MemoryMaxMB values 2021-03-30 16:55:58 -04:00
Tim Gross f820021f9e deps: bump gopsutil to v3.21.2 2021-03-30 16:02:51 -04:00
Charlie Voiselle 0473f35003
Fixup uses of `sanity` (#10187)
* Fixup uses of `sanity`
* Remove unnecessary comments.

These checks are better explained by earlier comments about
the context of the test. Per @tgross, moved the tests together
to better reinforce the overall shared context.

* Update nomad/fsm_test.go
2021-03-16 18:05:08 -04:00
Florian Apolloner a0873d5da4
docker: support configuring default log driver in plugin options 2021-03-12 16:04:33 -05:00
Adrian Todorov 47e1cb11df
driver/docker: add extra labels ( job name, task and task group name) 2021-03-08 08:59:52 -05:00
Seth Hoenig 8ee9835923 drivers/exec+java: Add task configuration to restore previous PID/IPC isolation behavior
This PR adds pid_mode and ipc_mode options to the exec and java task
driver config options. By default these will defer to the default_pid_mode
and default_ipc_mode agent plugin options created in #9969. Setting
these values to "host" mode disables isolation for the task. Doing so
is not recommended, but may be necessary to support legacy job configurations.

Closes #9970
2021-02-08 14:26:35 -06:00
Seth Hoenig 152534fe21 docs: fixup comments, var names 2021-02-08 10:58:44 -06:00
Seth Hoenig 4bc6e5a215 drivers/exec+java: Add configuration to restore previous PID/IPC namespace behavior.
This PR adds default_pid_mode and default_ipc_mode options to the exec and java
task drivers. By default these will default to "private" mode, enabling PID and
IPC isolation for tasks. Setting them to "host" mode disables isolation. Doing
so is not recommended, but may be necessary to support legacy job configurations.

Closes #9969
2021-02-05 15:52:11 -06:00
Nick Ethier d2f192821e drivers/docker: support mapping multiple host ports to the same container port 2021-02-02 22:54:23 -05:00
Chris Baker ce68ee164b Version 1.0.3
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1
 
 iQEcBAABAgAGBQJgEuOKAAoJEFGFLYc0j/xMxF8H/3TTU6Tu+Xm0YvcsDaYDphZ/
 X7KQBV0aFiuL5VkTw4PzKEsgryIy9/sqEPyxxyKRowAmos9qhiusjNAIfqdP4TF8
 tdZmTedkfWir9uPD+hyv/LXpwbQ2T8kTwS3xHTYvaOmaCxZr710FEn+imnMk1AUn
 Xs5itkd/CYGr0nBLm+I5GutWSDPmL7Uw8J5Z30fFyoaxoCPAbCWQQNk793SCRUc5
 f/uo18V2tFInmQ+3sAdnM4gPewyStK/a5VvzWavL9fVDtYK83wlqWSchTXY5jpVz
 zNEzt/rYhbBzakPQQKb5zieblh2iGI8aHWpD5w4WduqO2Sg6B/5lAeNZIlW0UJg=
 =2g3c
 -----END PGP SIGNATURE-----

Merge tag 'v1.0.3' into post-release-1.0.3

Version 1.0.3
2021-01-29 19:30:08 +00:00
vagrant bff87e3d0c attempting to fix flaky tests caused by pid isolation 2021-01-28 12:03:20 +00:00
Kris Hicks f5527aea48 Backfill unit test for NEWIPC 2021-01-28 12:03:19 +00:00
Chris Baker 3b580d9474 modify exec driver test TestExecDriver_StartWaitStop in light of the fact that signaling sleep with SIGINT doesn't work if it's PID1 2021-01-28 12:03:19 +00:00
Chris Baker 5e4be990ff modify exec driver test TestExecDriver_DestroyKills all in light of the fact that PID namespacing means that the kernel does this now 2021-01-28 12:03:19 +00:00
Chris Baker ac1b9655ce put exec process in a new IPC namespace 2021-01-28 12:03:19 +00:00
Kris Hicks a5298ea4ba Add unit test for container namespacing 2021-01-28 12:03:19 +00:00
Kris Hicks c13f75d9e1 Always check that resource constraints were applied 2021-01-28 12:03:19 +00:00
Kris Hicks 87188f04de Add PID namespacing and e2e test 2021-01-28 12:03:19 +00:00
Tim Gross 987cdb3a69 prefer TrimPrefix to checking HasPrefix first 2021-01-22 13:41:28 -05:00
Huan Wang ba8b2297b1 fix the inconsistency handling between infra image and normal task image 2021-01-22 13:41:28 -05:00
Kris Hicks 7694a66414
Don't prepend https to docker cred helper call (#9852)
Some credential helpers, like the ECR helper, will strip the protocol if
given. Others, like the linux "pass" helper, do not.
2021-01-21 11:46:59 -08:00
Charlie Voiselle 4f4d6e6c37
Enable network namespaces for QEMU driver (#9861)
* Enable network namespaces for QEMU driver
* Add CHANGELOG entry
2021-01-21 14:05:46 -05:00
Kris Hicks 8a8b95a119
executor_linux: Remove unreachable PATH= code (#9778)
This has to have been unused because the HasPrefix operation is
backwards, meaning a Command.Env that includes PATH= never would have
worked; the default path was always used.
2021-01-15 11:19:09 -08:00
Mahmood Ali de954da350
docker: introduce a new hcl2-friendly `mount` syntax (#9635)
Introduce a new more-block friendly syntax for specifying mounts with a new `mount` block type with the target as label:

```hcl
config {
  image = "..."

  mount {
    type = "..."
    target = "target-path"
    volume_options { ... }
  }
}
```

The main benefit here is that by `mount` being a block, it can nest blocks and avoids the compatibility problems noted in https://github.com/hashicorp/nomad/pull/9634/files#diff-2161d829655a3a36ba2d916023e4eec125b9bd22873493c1c2e5e3f7ba92c691R128-R155 .

The intention is for us to promote this `mount` blocks and quietly deprecate the `mounts` type, while still honoring to preserve compatibility as much as we could.

This addresses the issue in https://github.com/hashicorp/nomad/issues/9604 .
2020-12-15 14:13:50 -05:00
Kris Hicks 0cf9cae656
Apply some suggested fixes from staticcheck (#9598) 2020-12-10 07:29:18 -08:00
Kris Hicks 0a3a748053
Add gosimple linter (#9590) 2020-12-09 11:05:18 -08:00
Kris Hicks 93155ba3da
Add gocritic to golangci-lint config (#9556) 2020-12-08 12:47:04 -08:00
Tim Gross d286d941dc docker: kill signal API should include timeout context
When the Docker driver kills as task, we send a request via the Docker API for
dockerd to fire the signal. We send that signal and then block for the
`kill_timeout` waiting for the container to exit. But if the Docker API
blocks, we will block indefinitely because we haven't configured the API call
with the same timeout.

This changeset is a minimal intervention to add the timeout to the Docker API
call _only_ when we have the `kill_timeout` set. Future work should examine
whether we should be threading contexts through other `go-dockerclient` API
calls.
2020-12-02 16:51:57 -05:00
Mahmood Ali 98c02851c8
use comment ignores (#9448)
Use targetted ignore comments for the cases where we are bound by
backward compatibility.

I've left some file based linters, especially when the file is riddled
with linter voilations (e.g. enum names), or if it's a property of the
file (e.g. package and file names).

I encountered an odd behavior related to RPC_REQUEST_RESPONSE_UNIQUE and
RPC_REQUEST_STANDARD_NAME.  Apparently, if they target a `stream` type,
we must separate them into separate lines so that the ignore comment
targets the type specifically.
2020-11-25 16:03:01 -05:00
Mahmood Ali b2a8752c5f
honor task user when execing into raw_exec task (#9439)
Fix #9210 .

This update the executor so it honors the User when using nomad alloc exec. The bug was that the exec task didn't honor the init command when execing.
2020-11-25 09:34:10 -05:00
Nick Ethier c9bd7e89ca command: use correct port mapping syntax in examples 2020-11-23 10:25:30 -06:00
Mahmood Ali d92d413ffd
Merge pull request #8291 from shishir-a412ed/cpusets
Add cpuset_cpus to docker driver.
2020-11-11 17:13:27 -05:00
Mahmood Ali a89da9982d
raw_exec: don't use cgroups when no_cgroup is set (#9328)
When raw_exec is configured with [`no_cgroups`](https://www.nomadproject.io/docs/drivers/raw_exec#no_cgroups), raw_exec shouldn't attempt to create a cgroup.

Prior to this change, we accidentally always required freezer cgroup to do stats PID tracking. We already have the proper fallback in place for metrics, so only need to ensure that we don't create a cgroup for the task.

Fixes https://github.com/hashicorp/nomad/issues/8565
2020-11-11 16:20:34 -05:00
Shishir Mahajan 572c398187 Fix review comments. 2020-11-11 12:30:00 -08:00
Shishir Mahajan 9192100d4e Fix circleci. 2020-11-11 12:30:00 -08:00
Shishir Mahajan c30fea5cd3 Add cpuset_cpus to docker driver. 2020-11-11 12:30:00 -08:00