This upgrades hcl2 library dependency to pick up
https://github.com/hashicorp/hcl2/pull/113 .
Prior to this change, parsing and decoding array attributes containing
invalid errors (e.g. references to unknown variables) are silently
dropped, with `cty.Unknown` being assigned to the bad element. Rather
than showing a type/meaningful error from hcl2, we get a very decrypted
error message from msgpack layer trying to handle `cty.unknown`.
This ensures that we propagate diagnostics correctly and report
meaningful errors to users.
Fixes https://github.com/hashicorp/nomad/issues/5694
Fixes https://github.com/hashicorp/nomad/issues/5680
Our testing so far indicates that ugorji/go/codec maintains backward
compatiblity with the version we are using now, for purposes of Nomad
serialization.
Using latest ugorji/go allows us to get back to using upstream library,
get get the optimizations benefits in RPC paths (including code
generation optimizations).
ugorji/go introduced two significant changes:
* time binary format in debb8e2d2e. Setting `h.BasicHandle.TimeNotBuiltin = true` restores old behavior
* ugorji/go started honoring `json` tag as well:
v1.1.4 is the latest but has a bug in handling RawString that's fixed in
d09a80c1e0
.
* client/metrics: modified metrics to use (updated) client copy of allocation instead of (unupdated) server copy
* updated armon/go-metrics to address race condition in DisplayMetrics
* Divest api/ package of deps elsewhere in the nomad repo.
This will allow making api/ a module without then pulling in the
external repo, leading to a package name conflict.
This required some migration of tests to an apitests/ folder (can be
moved anywhere as it has no deps on it). It also required some
duplication of code, notably some test helpers from api/ -> apitests/
and part (but not all) of testutil/ -> api/testutil/.
Once there's more separation and an e.g. sdk/ folder those can be
removed in favor of a dep on the sdk/ folder, provided the sdk/ folder
doesn't depend on api/ or /.
* Also remove consul dep from api/ package
* Fix stupid linters
* Some restructuring
* CVE-2019-5736: Update libcontainer depedencies
Libcontainer is vulnerable to a runc container breakout, that was
reported as CVE-2019-5736[1]. Upgrading vendored libcontainer with the fix.
The runc changes are captured in 369b920277 .
[1] https://seclists.org/oss-sec/2019/q1/119
Previously used `github.com/shirou/gopsutil`[1], used some GPL code [2].
This was somewhat unintentional, and was addressed later [3].
Due to being late in the cycle of Nomad release when this is noticed,
and time elapsed since we updated the dependency, we want to be
conservative in our package updates.
As such, we opted to go with forking the repo to use the previously used
version with the GPL removal code commit, done in [4].
[1] 5776ff9c7c
[2] 5776ff9c7c/host/include/smc.c
[3] c95755e4bc
[4] 62d5761ddb
This fixes a bug related to shutting down of GRPC plugin interfaces
(more info: https://github.com/hashicorp/go-plugin/pull/88)
This does not yet fix all test cases for subprocess leaking, but is a
useful independant change.
Introduce a device manager that manages the lifecycle of device plugins
on the client. It fingerprints, collects stats, and forwards Reserve
requests to the correct plugin. The manager, also handles device plugins
failing and validates their output.
* client/executor: refactor client to remove interpolation
* executor: POC libcontainer based executor
* vendor: use hashicorp libcontainer fork
* vendor: add libcontainer/nsenter dep
* executor: updated executor interface to simplify operations
* executor: implement logging pipe
* logmon: new logmon plugin to manage task logs
* driver/executor: use logmon for log management
* executor: fix tests and windows build
* executor: fix logging key names
* executor: fix test failures
* executor: add config field to toggle between using libcontainer and standard executors
* logmon: use discover utility to discover nomad executable
* executor: only call libcontainer-shim on main in linux
* logmon: use seperate path configs for stdout/stderr fifos
* executor: windows fixes
* executor: created reusable pid stats collection utility that can be used in an executor
* executor: update fifo.Open calls
* executor: fix build
* remove executor from docker driver
* executor: Shutdown func to kill and cleanup executor and its children
* executor: move linux specific universal executor funcs to seperate file
* move logmon initialization to a task runner hook
* client: doc fixes and renaming from code review
* taskrunner: use shared config struct for logmon fifo fields
* taskrunner: logmon only needs to be started once per task
* vendor: bump libcontainer and docker to remove Sirupsen imports
* vendor: fix bad vendoring of archive package
* vendor: fix api changes to cgroups in executor
* vendor: fix docker api changes
* vendor: update github.com/Azure/go-ansiterm to use non capitalized logrus import
The blake2b package only includes an asm implementation for amd64. Other
architectures use a generic Go implementation. Unfortunately appengine
is one of those architectures and we explicitly ignore appengine when
vendoring because it is an unsupported platform for Nomad.
Commit 925ec07a4 started ignoring appengine to help trim down the vendor
package. We now need to include appengine again - despite not building
for it - just to avoid accidently breaking other archs we do support.
Mahmood Ali