Nick Ethier
GitHub
commit
2c9c9dbd67
|
|
@ -0,0 +1,82 @@
|
||||||
|
---
|
||||||
|
layout: "docs"
|
||||||
|
page_title: "Base Plugin"
|
||||||
|
sidebar_current: "docs-internals-plugins-base"
|
||||||
|
description: |-
|
||||||
|
Learn about how to author a Nomad plugin.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Base Plugin
|
||||||
|
|
||||||
|
The base plugin is a special plugin type implemented by all plugins. It allows
|
||||||
|
for common plugin operations such as defining a configuration schema and
|
||||||
|
version information.
|
||||||
|
|
||||||
|
## Plugin API
|
||||||
|
|
||||||
|
#### `PluginInfo() (*PluginInfoResponse, error)`
|
||||||
|
|
||||||
|
A `PluginInfoResponse` contains meta data about the plugin.
|
||||||
|
|
||||||
|
```go
|
||||||
|
PluginInfoResponse{
|
||||||
|
// Type is the plugin type which is implemented
|
||||||
|
Type: PluginTypeDriver,
|
||||||
|
// Plugin API versions supported by the plugin
|
||||||
|
PluginApiVersions: []string{drivers.ApiVersion010},
|
||||||
|
// Version of the plugin
|
||||||
|
PluginVersion: "0.1.0",
|
||||||
|
// Name of the plugin
|
||||||
|
Name: "foodriver",
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `ConfigSchema() (*hclspec.Spec, error)`
|
||||||
|
|
||||||
|
The `ConfigSchema` function allows a plugin to tell Nomad the schema for it's
|
||||||
|
configuration. This configuration is given in a [plugin block][pluginblock] of
|
||||||
|
the client configuration. The schema is defined with the [hclspec][hclspec]
|
||||||
|
package.
|
||||||
|
|
||||||
|
#### `SetConfig(config *Config) error`
|
||||||
|
|
||||||
|
The `SetConfig` function is called when starting the plugin for the first
|
||||||
|
time. The `Config` given has two different configuration fields. The first
|
||||||
|
`PluginConfig`, is an encoded configuration from the `plugin` block of the
|
||||||
|
client config. The second, `AgentConfig`, is the Nomad agent's configuration
|
||||||
|
which is given to all plugins.
|
||||||
|
|
||||||
|
## HCL Specifications
|
||||||
|
|
||||||
|
`*hclspec.Spec` is a struct that defines the schema to validate an HCL entity
|
||||||
|
against. The full documentation of the different hcl attribute types can be
|
||||||
|
found on the [hclspec godoc][hclspec].
|
||||||
|
|
||||||
|
For a basic example, lets look at the driver configuration for the raw_exec
|
||||||
|
driver:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
job "example" {
|
||||||
|
...
|
||||||
|
driver = "raw_exec"
|
||||||
|
config {
|
||||||
|
command = "/bin/sleep"
|
||||||
|
args = ["100"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The `config` block is what is validated against the `hclspec.Spec`. It has two
|
||||||
|
keys, command which takes a string attribute and args which takes an array
|
||||||
|
attribute. The corresponding `*hclspec.Spec` would be:
|
||||||
|
|
||||||
|
```go
|
||||||
|
spec := hclspec.NewObject(map[string]*hclspec.Spec{
|
||||||
|
"command": hclspec.NewAttr("command", "string", true),
|
||||||
|
"args": hclspec.NewAttr("args", "list(string)", false),
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
[hclspec]: https://godoc.org/github.com/hashicorp/nomad/plugins/shared/hclspec
|
||||||
|
[pluginblock]: /docs/configuration/plugin.html
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
---
|
||||||
|
layout: "docs"
|
||||||
|
page_title: "Device Plugins"
|
||||||
|
sidebar_current: "docs-internals-plugins-devices"
|
||||||
|
description: |-
|
||||||
|
Learn about how to author a Nomad device plugin.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Devices
|
||||||
|
|
||||||
|
Device plugin documentation is currently a work in progress. Until there is
|
||||||
|
documentation, the [Nvidia GPU plugin](https://github.com/hashicorp/nomad/tree/master/devices/gpu/nvidia) is a useful example.
|
||||||
|
|
@ -0,0 +1,33 @@
|
||||||
|
---
|
||||||
|
layout: "docs"
|
||||||
|
page_title: "Plugins"
|
||||||
|
sidebar_current: "docs-internals-plugins"
|
||||||
|
description: |-
|
||||||
|
Learn about how external plugins work in Nomad.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Plugins
|
||||||
|
|
||||||
|
Nomad 0.9 introduced a plugin framework which allows users to extend the
|
||||||
|
functionality of some components within Nomad. The design of the plugin system
|
||||||
|
is inspired by the lessons learned from plugin systems implemented in other
|
||||||
|
HashiCorp products such as Terraform and Vault.
|
||||||
|
|
||||||
|
The following components are currently plugable within Nomad:
|
||||||
|
|
||||||
|
- [Task Drivers](/docs/internals/plugins/task-drivers.html)
|
||||||
|
- [Devices](/docs/internals/plugins/devices.html)
|
||||||
|
|
||||||
|
# Architecture
|
||||||
|
|
||||||
|
The Nomad plugin framework uses the [go-plugin][goplugin] project to expose
|
||||||
|
a language independent plugin interface. Plugins implement a set of GRPC
|
||||||
|
services and methods which Nomad manages by running the plugin and calling the
|
||||||
|
implemented RPCs. This means that plugins are free to be implemented in the
|
||||||
|
author's language of choice.
|
||||||
|
|
||||||
|
To make plugin development easier, a set of go interfaces and structs exist for
|
||||||
|
each plugin type that abstract away go-plugin and the GRPC interface. The
|
||||||
|
guides in this documentation reference these abstractions for ease of use.
|
||||||
|
|
||||||
|
[goplugin]: https://github.com/hashicorp/go-plugin
|
||||||
|
|
@ -0,0 +1,195 @@
|
||||||
|
---
|
||||||
|
layout: "docs"
|
||||||
|
page_title: "Task Driver Plugins"
|
||||||
|
sidebar_current: "docs-internals-plugins-task-drivers"
|
||||||
|
description: |-
|
||||||
|
Learn about how to author a Nomad plugin.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Task Drivers
|
||||||
|
|
||||||
|
Task drivers in Nomad are the runtime components that execute workloads. For
|
||||||
|
a real world example of a Nomad task driver plugin implementation see the [LXC
|
||||||
|
driver source][lxcdriver].
|
||||||
|
|
||||||
|
## Authoring Task Driver Plugins
|
||||||
|
|
||||||
|
Authoring a task driver (shortened to driver in this documentation) in Nomad
|
||||||
|
consists of implementing the [DriverPlugin][driverplugin] interface and adding
|
||||||
|
a main package to launch the plugin. A driver plugin is long lived and it's
|
||||||
|
lifetime is not bound to the Nomad client. This means that the Nomad client can
|
||||||
|
be restarted without the restarting the driver. Nomad will ensure that one
|
||||||
|
instance of the driver is running, meaning if the driver crashes or otherwise
|
||||||
|
terminates, Nomad will launch another instance of it.
|
||||||
|
|
||||||
|
Drivers should maintain as little state as possible. State for a task is stored
|
||||||
|
by the Nomad client on task creation. This enables a pattern where the driver
|
||||||
|
can maintain an in-memory state of the running tasks, and if necessary the
|
||||||
|
Nomad client can recover tasks into the driver state.
|
||||||
|
|
||||||
|
## Task Driver Plugin API
|
||||||
|
|
||||||
|
The [base plugin][baseplugin] must be implement in addition to the following
|
||||||
|
functions.
|
||||||
|
|
||||||
|
### `TaskConfigSchema() (*hclspec.Spec, error)`
|
||||||
|
|
||||||
|
This function returns the schema for the driver configuration of the task. For
|
||||||
|
more information on `hclspec.Spec` see the HCL section in the [base
|
||||||
|
plugin][baseplugin] documentation.
|
||||||
|
|
||||||
|
### `Capabilities() (*Capabilities, error)`
|
||||||
|
|
||||||
|
Capabilities define what features the driver implements. Example:
|
||||||
|
|
||||||
|
```go
|
||||||
|
Capabilities {
|
||||||
|
// Does the driver support sending OS signals to the task?
|
||||||
|
SendSignals: true,
|
||||||
|
// Does the driver support executing a command within the task execution
|
||||||
|
// environment?
|
||||||
|
Exec: true,
|
||||||
|
// What filesystem isolation is supported by the driver. Options include
|
||||||
|
// FSIsolationImage, FSIsolationChroot, and FSIsolationNone
|
||||||
|
FSIsolation: FSIsolationImage,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `Fingerprint(context.Context) (<-chan *Fingerprint, error)`
|
||||||
|
|
||||||
|
This function is called by the client when the plugin is started. It allows the
|
||||||
|
driver to indicate its health to the client. The channel returned should
|
||||||
|
immediately send an initial Fingerprint, then send periodic updates at an
|
||||||
|
interval that is appropriate for the driver until the context is canceled.
|
||||||
|
|
||||||
|
The fingerprint consists of a `HealthState` and `HealthDescription` to inform
|
||||||
|
the client about its health. Additionally an `Attributes` field is available
|
||||||
|
for the driver to add additional attributes to the client node. The fingerprint
|
||||||
|
`HealthState` can be one of three states.
|
||||||
|
|
||||||
|
- `HealthStateUndetected`: Indicates that the necessary dependencies for the
|
||||||
|
driver are not detected on the system. Ex. java runtime for the java driver
|
||||||
|
- `HealthStateUnhealthy`: Indicates that something is wrong with the driver
|
||||||
|
runtime. Ex. docker daemon stopped for the Docker driver
|
||||||
|
- `HealthStateHealthy`: All systems go
|
||||||
|
|
||||||
|
### `StartTask(*TaskConfig) (*TaskHandle, *DriverNetwork, error)`
|
||||||
|
|
||||||
|
This function takes a [`TaskConfig`][taskconfig] which includes all of the configuration
|
||||||
|
needed to launch the task. Additionally the driver configuration can be decoded
|
||||||
|
from the `TaskConfig` by calling `*TaskConfig.DecodeDriverConfig(t interface{})`
|
||||||
|
passing in a pointer to the driver specific configuration struct. The
|
||||||
|
`TaskConfig` includes an `ID` field which future operations on the task will be
|
||||||
|
referenced by.
|
||||||
|
|
||||||
|
Drivers return a [`*TaskHandle`][taskhandle] which contains
|
||||||
|
the required information for the driver to reattach to the running task in the
|
||||||
|
case of plugin crashes or restarts. Some of this required state
|
||||||
|
will be specific to the driver implementation, thus a `DriverState` field
|
||||||
|
exists to allow the driver to encode custom state into the struct. Helper
|
||||||
|
fields exist on the `TaskHandle` to `GetDriverState` and `SetDriverState`
|
||||||
|
removing the need for the driver to handle serialization.
|
||||||
|
|
||||||
|
A `*DriverNetwork` can optionally be returned to describe the network of the
|
||||||
|
task if it is modified by the driver. An example of this is in the Docker
|
||||||
|
driver where tasks can be attached to a specific Docker network.
|
||||||
|
|
||||||
|
If an error occurs, it is expected that the driver will cleanup any created
|
||||||
|
resources prior to returning the error.
|
||||||
|
|
||||||
|
#### Logging
|
||||||
|
|
||||||
|
Nomad handles all rotation and plumbing of task logs. In order for task stdout
|
||||||
|
and stderr to be received by Nomad, they must be written to the correct
|
||||||
|
location. Prior to starting the task through the driver, the Nomad client
|
||||||
|
creates FIFOs for stdout and stderr. These paths are given to the driver in the
|
||||||
|
`TaskConfig`. The [`fifo` package][fifopackage] can be used to support
|
||||||
|
cross platform writing to these paths.
|
||||||
|
|
||||||
|
#### TaskHandle Schema Versioning
|
||||||
|
|
||||||
|
A `Version` field is available on the TaskHandle struct to facilitate backwards
|
||||||
|
compatible recovery of tasks. This field is opaque to Nomad, but allows the
|
||||||
|
driver to handle recover tasks that were created by an older version of the
|
||||||
|
plugin.
|
||||||
|
|
||||||
|
### `RecoverTask(*TaskHandle) error`
|
||||||
|
|
||||||
|
When a driver is restarted it is not expected to persist any internal state to
|
||||||
|
disk. To support this, Nomad will attempt to recover a task that was
|
||||||
|
previously started if the driver does not recognize the task ID. During task
|
||||||
|
recovery, Nomad calls `RecoverTask` passing the `TaskHandle` that was
|
||||||
|
returned by the `StartTask` function. If no error was returned, it is
|
||||||
|
expected that the driver can now operate on the task by referencing the task
|
||||||
|
ID. If an error occurs, the Nomad client will mark the task as `lost`.
|
||||||
|
|
||||||
|
### `WaitTask(ctx context.Context, id string) (<-chan *ExitResult, error)`
|
||||||
|
|
||||||
|
The `WaitTask` function is expected to return a channel that will send an
|
||||||
|
`*ExitResult` when the task exits or close the channel when the context is
|
||||||
|
canceled. It is also expected that calling `WaitTask` on an exited task will
|
||||||
|
immediately send an `*ExitResult` on the returned channel.
|
||||||
|
|
||||||
|
### `StopTask(taskID string, timeout time.Duration, signal string) error`
|
||||||
|
|
||||||
|
The `StopTask` function is expected to stop a running task by sending the given
|
||||||
|
signal to it. If the task does not stop during the given timeout, the driver
|
||||||
|
must forcefully kill the task.
|
||||||
|
|
||||||
|
`StopTask` does not clean up resources of the task or remove it from the
|
||||||
|
driver's internal state. A call to `WaitTask` after `StopTask` is valid and
|
||||||
|
should be handled.
|
||||||
|
|
||||||
|
### `DestroyTask(taskID string, force bool) error`
|
||||||
|
|
||||||
|
The `DestroyTask` function cleans up and removes a task that has terminated. If
|
||||||
|
force is set to true, the driver must destroy the task even if it is still
|
||||||
|
running. If `WaitTask` is called after `DestroyTask`, it should return
|
||||||
|
`drivers.ErrTaskNotFound` as no task state should exist after `DestroyTask` is
|
||||||
|
called.
|
||||||
|
|
||||||
|
### `InspectTask(taskID string) (*TaskStatus, error)`
|
||||||
|
|
||||||
|
The `InspectTask` function returns detailed status information for the
|
||||||
|
referenced `taskID`.
|
||||||
|
|
||||||
|
### `TaskStats(ctx context.Context, id string, i time.Duration) (<-chan *cstructs.TaskResourceUsage, error)`
|
||||||
|
|
||||||
|
The `TaskStats` function returns a channel which the driver should send stats
|
||||||
|
to at the given interval. The driver must send stats at the given interval
|
||||||
|
until the given context is canceled or the task terminates.
|
||||||
|
|
||||||
|
### `TaskEvents(context.Context) (<-chan *TaskEvent, error)`
|
||||||
|
|
||||||
|
The Nomad client publishes events associated with an allocation. The
|
||||||
|
`TaskEvents` function allows the driver to publish driver specific events about
|
||||||
|
tasks and the Nomad client will associate them with the correct allocation.
|
||||||
|
|
||||||
|
An `Eventer` utility is available in the
|
||||||
|
`github.com/hashicorp/nomad/drivers/shared/eventer` package implements an
|
||||||
|
event loop and publishing mechanism for use in the `TaskEvents` function.
|
||||||
|
|
||||||
|
### `SignalTask(taskID string, signal string) error`
|
||||||
|
|
||||||
|
> Optional - can be skipped by embedding `drivers.DriverSignalTaskNotSupported`
|
||||||
|
|
||||||
|
The `SignalTask` function is used by drivers which support sending OS signals
|
||||||
|
(`SIGHUP`, `SIGKILL`, `SIGUSR1` etc.) to the task. It is an optional function
|
||||||
|
and is listed as a capability in the driver `Capabilities` struct.
|
||||||
|
|
||||||
|
### `ExecTask(taskID string, cmd []string, timeout time.Duration) (*ExecTaskResult, error)`
|
||||||
|
|
||||||
|
> Optional - can be skipped by embedding `drivers.DriverExecTaskNotSupported`
|
||||||
|
|
||||||
|
The `ExecTask` function is used by the Nomad client to execute commands inside
|
||||||
|
the task execution context. For example, the Docker driver executes commands
|
||||||
|
inside the running container. `ExecTask` is called for Consul script checks.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
[lxcdriver]: https://github.com/hashicorp/nomad-driver-lxc
|
||||||
|
[DriverPlugin]: https://github.com/hashicorp/nomad/blob/v0.9.0-beta2/plugins/drivers/driver.go#L39-L57
|
||||||
|
[baseplugin]: /docs/internals/plugins/base.html
|
||||||
|
[taskconfig]: https://godoc.org/github.com/hashicorp/nomad/plugins/drivers#TaskConfig
|
||||||
|
[taskhandle]: https://godoc.org/github.com/hashicorp/nomad/plugins/drivers#TaskHandle
|
||||||
|
[fifopackage]: https://godoc.org/github.com/hashicorp/nomad/client/lib/fifo
|
||||||
|
|
@ -9,6 +9,21 @@
|
||||||
<a href="/docs/internals/architecture.html">Architecture</a>
|
<a href="/docs/internals/architecture.html">Architecture</a>
|
||||||
</li>
|
</li>
|
||||||
|
|
||||||
|
<li<%= sidebar_current("docs-internals-plugins") %>>
|
||||||
|
<a href="/docs/internals/plugins/index.html">Plugins</a>
|
||||||
|
<ul class="nav">
|
||||||
|
<li <%= sidebar_current("docs-internals-plugins-base") %>>
|
||||||
|
<a href="/docs/internals/plugins/base.html">Base</a>
|
||||||
|
</li>
|
||||||
|
<li <%= sidebar_current("docs-internals-plugins-task-driver") %>>
|
||||||
|
<a href="/docs/internals/plugins/task-drivers.html">Task Drivers</a>
|
||||||
|
</li>
|
||||||
|
<li <%= sidebar_current("docs-internals-plugins-device") %>>
|
||||||
|
<a href="/docs/internals/plugins/devices.html">Devices</a>
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
</li>
|
||||||
|
|
||||||
<li<%= sidebar_current("docs-internals-scheduling") %>>
|
<li<%= sidebar_current("docs-internals-scheduling") %>>
|
||||||
<a href="/docs/internals/scheduling/index.html">Scheduling</a>
|
<a href="/docs/internals/scheduling/index.html">Scheduling</a>
|
||||||
<ul class="nav">
|
<ul class="nav">
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue