open-nomad/devices/gpu/nvidia
Mahmood Ali 2588b3bc98 cleanup driver eventor goroutines
This fixes few cases where driver eventor goroutines are leaked during
normal operations, but especially so in tests.

This change makes few modifications:

First, it switches drivers to use `Context`s to manage shutdown events.
Previously, it relied on callers invoking `.Shutdown()` function that is
specific to internal drivers only and require casting.  Using `Contexts`
provide a consistent idiomatic way to manage lifecycle for both internal
and external drivers.

Also, I discovered few places where we don't clean up a temporary driver
instance in the plugin catalog code, where we dispense a driver to
inspect and validate the schema config without properly cleaning it up.
2020-05-26 11:04:04 -04:00
..
cmd cleanup driver eventor goroutines 2020-05-26 11:04:04 -04:00
nvml nvidia package restructue + build non-linux 2018-10-05 13:56:04 -07:00
device.go cleanup driver eventor goroutines 2020-05-26 11:04:04 -04:00
device_test.go Add support for docker runtimes 2019-01-15 11:34:47 -08:00
fingerprint.go Device manager 2018-11-07 10:43:15 -08:00
fingerprint_test.go Use Attribute in device fingerprinting 2018-10-13 11:43:06 -07:00
README.md Update devices/gpu/nvidia/README.md 2019-01-23 17:44:24 -08:00
stats.go devices/nvidia: memory state as the summary stat 2018-12-10 12:18:24 -05:00
stats_test.go devices/nvidia: memory state as the summary stat 2018-12-10 12:18:24 -05:00

This package provides an implementation of nvidia device plugin

Behavior

Nvidia device plugin uses NVML bindings to get data regarding available nvidia devices and will expose them via Fingerprint RPC. GPUs can be excluded from fingerprinting by setting the ignored_gpu_ids field. Plugin sends statistics for fingerprinted devices every stats_period period.

Config

The configuration should be passed via an HCL file that begins with a top level config stanza:

config {
  ignored_gpu_ids = ["uuid1", "uuid2"]
  fingerprint_period = "5s"
}

The valid configuration options are:

  • ignored_gpu_ids (list(string): []): list of GPU UUIDs strings that should not be exposed to nomad
  • fingerprint_period (string: "1m"): interval to repeat the fingerprint process to identify possible changes.