open-nomad/client/stats
Danielle Lancashire 4f2343e1c0
client: Return empty values when host stats fail
Currently, there is an issue when running on Windows whereby under some
circumstances the Windows stats API's will begin to return errors (such
as internal timeouts) when a client is under high load, and potentially
other forms of resource contention / system states (and other unknown
cases).

When an error occurs during this collection, we then short circuit
further metrics emission from the client until the next interval.

This can be problematic if it happens for a sustained number of
intervals, as our metrics aggregator will begin to age out older
metrics, and we will eventually stop emitting various types of metrics
including `nomad.client.unallocated.*` metrics.

However, when metrics collection fails on Linux, gopsutil will in many cases
(e.g cpu.Times) silently return 0 values, rather than an error.

Here, we switch to returning empty metrics in these failures, and
logging the error at the source. This brings the behaviour into line
with Linux/Unix platforms, and although making aggregation a little
sadder on intermittent failures, will result in more desireable overall
behaviour of keeping metrics available for further investigation if
things look unusual.
2019-09-19 01:22:07 +02:00
..
cpu.go Enable more linters 2017-09-26 15:26:33 -07:00
cpu_test.go Add Client Device Stats structs in `api` package 2018-11-14 14:41:19 -05:00
cpu_unix.go Fix invalid CPU stats on Windows 2017-09-10 15:30:48 -07:00
cpu_windows.go Fix invalid CPU stats on Windows 2017-09-10 15:30:48 -07:00
host.go client: Return empty values when host stats fail 2019-09-19 01:22:07 +02:00
host_test.go client: Guard against "NaN" values from floats 2017-09-08 16:21:07 -05:00