Introduce limits to prevent unauthorized users from exhausting all
ephemeral ports on agents:
* `{https,rpc}_handshake_timeout`
* `{http,rpc}_max_conns_per_client`
The handshake timeout closes connections that have not completed the TLS
handshake by the deadline (5s by default). For RPC connections this
timeout also separately applies to first byte being read so RPC
connections with TLS enabled have `rpc_handshake_time * 2` as their
deadline.
The connection limit per client prevents a single remote TCP peer from
exhausting all ephemeral ports. The default is 100, but can be lowered
to a minimum of 26. Since streaming RPC connections create a new TCP
connection (until MultiplexV2 is used), 20 connections are reserved for
Raft and non-streaming RPCs to prevent connection exhaustion due to
streaming RPCs.
All limits are configurable and may be disabled by setting them to `0`.
This also includes a fix that closes connections that attempt to create
TLS RPC connections recursively. While only users with valid mTLS
certificates could perform such an operation, it was added as a
safeguard to prevent programming errors before they could cause resource
exhaustion.
Update github.com/aws/aws-sdk-go and github.com/hashicorp/go-discover to
pick up support for EC2 Metadata Instance Service v2 changes.
Follow up to https://github.com/hashicorp/go-discover/pull/128 .
This is the basic code to add the Windows Service Manager hooks to Nomad.
Includes vendoring golang.org/x/sys/windows/svc and added Docs:
* guide for installing as a windows service.
* configuration for logging to file from PR #6429
Adds new package that can be used by client and server RPC endpoints to
facilitate monitoring based off of a logger
clean up old code
small comment about write
rm old comment about minsize
rename to Monitor
Removes connection logic from monitor command
Keep connection logic in endpoints, use a channel to send results from
monitoring
use new multisink logger and interfaces
small test for dropped messages
update go-hclogger and update sink/intercept logger interfaces
Adds nomad monitor command. Like consul monitor, this command allows you
to stream logs from a nomad agent in real time with a a specified log
level
add endpoint tests
Upgrade go-hclog to latest version
The current version of go-hclog pads log prefixes to equal lengths
so info becomes [INFO ] and debug becomes [DEBUG]. This breaks
hashicorp/logutils/level.go Check function. Upgrading to the latest
version removes this padding and fixes log filtering that uses logutils
Check
AgentMonitor is an endpoint to stream logs for a given agent. It allows
callers to pass in a supplied log level, which may be different than the
agents config allowing for temporary debugging with lower log levels.
Pass in logWriter when setting up Agent
Update multierror to latest as of now. Our version is very old and
dates back to Sep 2015[1]. Here, we aim to pick up a panic fix found in
n https://github.com/hashicorp/go-multierror/pull/11 (Dec 2016).
This is a purely hygiene maintenance change. I am unaware of any causes
of the panic in our current dependencies. Though, some private internal
libraries did rely on the "recent" behavior of go-multierror, and I
aimed to update here to ease our adoption of other libraries later.
[1] d30f09973e
Golang 1.13 is pickier with importpaths and aliasing and fails
compilation currently.
Here, for go-msgpack dependency, we use upstream ugorji/go with a single
change
23165f7bc3
.
For consistency and to ease noticing descripency, I made ugorji/go and
hashicorp/go-msgpack reference the same sha.
This is a dependency management update and has no functional change to
product.
* ar: refactor network bridge config to use go-cni lib
* ar: use eth as the iface prefix for bridged network namespaces
* vendor: update containerd/go-cni package
* ar: update network hook to use TODO contexts when calling configurator
* unnecessary conversion
This upgrades hcl2 library dependency to pick up
https://github.com/hashicorp/hcl2/pull/113 .
Prior to this change, parsing and decoding array attributes containing
invalid errors (e.g. references to unknown variables) are silently
dropped, with `cty.Unknown` being assigned to the bad element. Rather
than showing a type/meaningful error from hcl2, we get a very decrypted
error message from msgpack layer trying to handle `cty.unknown`.
This ensures that we propagate diagnostics correctly and report
meaningful errors to users.
Fixes https://github.com/hashicorp/nomad/issues/5694
Fixes https://github.com/hashicorp/nomad/issues/5680
Our testing so far indicates that ugorji/go/codec maintains backward
compatiblity with the version we are using now, for purposes of Nomad
serialization.
Using latest ugorji/go allows us to get back to using upstream library,
get get the optimizations benefits in RPC paths (including code
generation optimizations).
ugorji/go introduced two significant changes:
* time binary format in debb8e2d2e. Setting `h.BasicHandle.TimeNotBuiltin = true` restores old behavior
* ugorji/go started honoring `json` tag as well:
v1.1.4 is the latest but has a bug in handling RawString that's fixed in
d09a80c1e0
.
* client/metrics: modified metrics to use (updated) client copy of allocation instead of (unupdated) server copy
* updated armon/go-metrics to address race condition in DisplayMetrics
* Divest api/ package of deps elsewhere in the nomad repo.
This will allow making api/ a module without then pulling in the
external repo, leading to a package name conflict.
This required some migration of tests to an apitests/ folder (can be
moved anywhere as it has no deps on it). It also required some
duplication of code, notably some test helpers from api/ -> apitests/
and part (but not all) of testutil/ -> api/testutil/.
Once there's more separation and an e.g. sdk/ folder those can be
removed in favor of a dep on the sdk/ folder, provided the sdk/ folder
doesn't depend on api/ or /.
* Also remove consul dep from api/ package
* Fix stupid linters
* Some restructuring
* CVE-2019-5736: Update libcontainer depedencies
Libcontainer is vulnerable to a runc container breakout, that was
reported as CVE-2019-5736[1]. Upgrading vendored libcontainer with the fix.
The runc changes are captured in 369b920277 .
[1] https://seclists.org/oss-sec/2019/q1/119
Previously used `github.com/shirou/gopsutil`[1], used some GPL code [2].
This was somewhat unintentional, and was addressed later [3].
Due to being late in the cycle of Nomad release when this is noticed,
and time elapsed since we updated the dependency, we want to be
conservative in our package updates.
As such, we opted to go with forking the repo to use the previously used
version with the GPL removal code commit, done in [4].
[1] 5776ff9c7c
[2] 5776ff9c7c/host/include/smc.c
[3] c95755e4bc
[4] 62d5761ddb
This fixes a bug related to shutting down of GRPC plugin interfaces
(more info: https://github.com/hashicorp/go-plugin/pull/88)
This does not yet fix all test cases for subprocess leaking, but is a
useful independant change.
Introduce a device manager that manages the lifecycle of device plugins
on the client. It fingerprints, collects stats, and forwards Reserve
requests to the correct plugin. The manager, also handles device plugins
failing and validates their output.
* client/executor: refactor client to remove interpolation
* executor: POC libcontainer based executor
* vendor: use hashicorp libcontainer fork
* vendor: add libcontainer/nsenter dep
* executor: updated executor interface to simplify operations
* executor: implement logging pipe
* logmon: new logmon plugin to manage task logs
* driver/executor: use logmon for log management
* executor: fix tests and windows build
* executor: fix logging key names
* executor: fix test failures
* executor: add config field to toggle between using libcontainer and standard executors
* logmon: use discover utility to discover nomad executable
* executor: only call libcontainer-shim on main in linux
* logmon: use seperate path configs for stdout/stderr fifos
* executor: windows fixes
* executor: created reusable pid stats collection utility that can be used in an executor
* executor: update fifo.Open calls
* executor: fix build
* remove executor from docker driver
* executor: Shutdown func to kill and cleanup executor and its children
* executor: move linux specific universal executor funcs to seperate file
* move logmon initialization to a task runner hook
* client: doc fixes and renaming from code review
* taskrunner: use shared config struct for logmon fifo fields
* taskrunner: logmon only needs to be started once per task
* vendor: bump libcontainer and docker to remove Sirupsen imports
* vendor: fix bad vendoring of archive package
* vendor: fix api changes to cgroups in executor
* vendor: fix docker api changes
* vendor: update github.com/Azure/go-ansiterm to use non capitalized logrus import
