docs: migration of Nomad Autoscaler docs with cluster updates.
Co-authored-by: Chris Baker <1675087+cgbaker@users.noreply.github.com>
This commit is contained in:
James Rasell
parent
07b5acb3cc
commit
9cc5540926
|
|
@ -223,6 +223,29 @@ export default [
|
||||||
},
|
},
|
||||||
'schedulers',
|
'schedulers',
|
||||||
{ category: 'runtime', content: ['environment', 'interpolation'] },
|
{ category: 'runtime', content: ['environment', 'interpolation'] },
|
||||||
|
{
|
||||||
|
category: 'autoscaling',
|
||||||
|
content: [
|
||||||
|
'agent',
|
||||||
|
'api',
|
||||||
|
'cli',
|
||||||
|
'policy',
|
||||||
|
{
|
||||||
|
category: 'plugins',
|
||||||
|
content: [
|
||||||
|
'apm',
|
||||||
|
'strategy',
|
||||||
|
'target'
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
category: 'internals',
|
||||||
|
content: [
|
||||||
|
'checks'
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
{ category: 'telemetry', content: ['metrics'] },
|
{ category: 'telemetry', content: ['metrics'] },
|
||||||
{ category: 'vault-integration' },
|
{ category: 'vault-integration' },
|
||||||
'------------',
|
'------------',
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,240 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Agent
|
||||||
|
sidebar_title: Agent
|
||||||
|
description: The Nomad Autoscaler is a long lived process which coordinates scaling activates.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Agent
|
||||||
|
|
||||||
|
The Nomad Autoscaler agent has a variety of parameters that can be specified
|
||||||
|
via configuration files or command-line flags. Configuration files are written
|
||||||
|
in [HCL][hcl_v2]. The Nomad Autoscaler can read and combine parameters from
|
||||||
|
multiple configuration files or directories to configure the agent.
|
||||||
|
|
||||||
|
## Nomad Namespaces
|
||||||
|
|
||||||
|
The Nomad Autoscaler currently has limited support for
|
||||||
|
[Nomad Namespaces][nomad_namespaces]. The `nomad` configuration below supports
|
||||||
|
specifying a namespace; if configured with a namespace, the Autoscaler will
|
||||||
|
retrieve scaling policies and perform autoscaling only for jobs in that
|
||||||
|
namespace. A future version will include support for multiple namespaces.
|
||||||
|
|
||||||
|
## Nomad ACLs
|
||||||
|
|
||||||
|
The Nomad Autoscaler can be configured to interact with an ACL-enabled Nomad
|
||||||
|
cluster. Nomad 0.11 includes the `scale` ACL policy disposition specifically for
|
||||||
|
supporting the operations of the Nomad Autoscaler. Therefore, the
|
||||||
|
following policy is sufficient for creating an ACL token that can be used by
|
||||||
|
the autoscaler for fetching scaling policies and scaling jobs:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
namespace "default" {
|
||||||
|
policy = "scale"
|
||||||
|
}
|
||||||
|
|
||||||
|
Other APM and target plugins may require additional ACLs; see the plugin documentation for more information.
|
||||||
|
|
||||||
|
## Load Order and Merging
|
||||||
|
|
||||||
|
The Nomad Autoscaler agent supports multiple configuration files, which can be
|
||||||
|
provided using the [-config][autoscaler_cli_config] CLI flag. The flag can
|
||||||
|
accept either a file or folder. In the case of a folder, any `.hcl` and `.json`
|
||||||
|
files in the folder will be loaded and merged in lexicographical order. Directories
|
||||||
|
are not loaded recursively.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ nomad-autoscaler agent -config=autoscaler.conf -config=/etc/nomad-autoscaler -config=extra.json
|
||||||
|
```
|
||||||
|
|
||||||
|
This will load configuration from autoscaler.conf, from `.hcl` and `.json` files
|
||||||
|
under `/etc/nomad-autoscaler`, and finally from `extra.json`. As each file is
|
||||||
|
processed, its contents are merged into the existing configuration. When merging,
|
||||||
|
any non-empty values from the latest config file will append or replace
|
||||||
|
parameters in the current configuration. An empty value means `""` for strings,
|
||||||
|
`0` for integer or float values, and `false` for booleans.
|
||||||
|
|
||||||
|
## General Parameters
|
||||||
|
|
||||||
|
- `log_level` `(string: "INFO")` - Specify the verbosity level of Nomad
|
||||||
|
Autoscaler's logs. Valid values include DEBUG, INFO, and WARN, in decreasing
|
||||||
|
order of verbosity.
|
||||||
|
|
||||||
|
- `log_json` `(bool: false)` - Output logs in a JSON format.
|
||||||
|
|
||||||
|
- `plugin_dir` `(string: "./plugins")` - The plugin directory is used to
|
||||||
|
discover Nomad Autoscaler plugins.
|
||||||
|
|
||||||
|
## `http` Block
|
||||||
|
|
||||||
|
The `http` block configures the Nomad Autoscaler's HTTP endpoint.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
http {
|
||||||
|
bind_address = "10.0.0.10"
|
||||||
|
bind_port = 9999
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `http` Parameters
|
||||||
|
|
||||||
|
- `bind_address` `(string "127.0.0.1")` - The HTTP address that the server will
|
||||||
|
bind to.
|
||||||
|
|
||||||
|
- `bind_port` `(int 8080)` - The port that the server will bind to.
|
||||||
|
|
||||||
|
## `nomad` Block
|
||||||
|
|
||||||
|
The `nomad` block configures the Nomad Autoscaler's Nomad client.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
nomad {
|
||||||
|
address = "http://my-nomad.systems:4646"
|
||||||
|
region = "esp-vlc-1"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `nomad` Parameters
|
||||||
|
|
||||||
|
- `address` `(string "http://127.0.0.1:4646")` - The address of the Nomad server
|
||||||
|
in the form of protocol://addr:port.
|
||||||
|
|
||||||
|
- `region` `(string "global")` - The region of the Nomad servers to connect with.
|
||||||
|
|
||||||
|
- `namespace` `(string "")` - The target namespace for queries and actions bound
|
||||||
|
to a namespace.
|
||||||
|
|
||||||
|
- `token` `(string "")` - The SecretID of an ACL token to use to authenticate
|
||||||
|
API requests with.
|
||||||
|
|
||||||
|
- `http_auth` `(string "")` - The authentication information to use when connecting
|
||||||
|
to a Nomad API which is using HTTP authentication.
|
||||||
|
|
||||||
|
- `ca_cert` `(string "")` - Path to a PEM encoded CA cert file to use to verify
|
||||||
|
the Nomad server SSL certificate.
|
||||||
|
|
||||||
|
- `ca_path` `(string "")` - Path to a directory of PEM encoded CA cert files to
|
||||||
|
verify the Nomad server SSL certificate.
|
||||||
|
|
||||||
|
- `client_cert` `(string "")` - Path to a PEM encoded client certificate for TLS
|
||||||
|
authentication to the Nomad server.
|
||||||
|
|
||||||
|
- `client_key` `(string "")` - Path to an unencrypted PEM encoded private key
|
||||||
|
matching the client certificate.
|
||||||
|
|
||||||
|
- `tls_server_name` `(string "")` - The server name to use as the SNI host when
|
||||||
|
connecting via TLS.
|
||||||
|
|
||||||
|
- `skip_verify` `(bool false)` - Do not verify TLS certificates. This is strongly
|
||||||
|
discouraged.
|
||||||
|
|
||||||
|
## `policy` Block
|
||||||
|
|
||||||
|
The `policy` block configures the Nomad Autoscaler's policy handling.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
policy {
|
||||||
|
dir = "/opt/nomad-autoscaler/plugins"
|
||||||
|
default_cooldown = "2m"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `policy` Parameters
|
||||||
|
|
||||||
|
- `dir` `(string "./plugins")` - The path to a directory used to load scaling
|
||||||
|
policies.
|
||||||
|
|
||||||
|
- `default_cooldown` `(string "5m")` - The default cooldown that will be applied
|
||||||
|
to all scaling policies which do not specify a cooldown period.
|
||||||
|
|
||||||
|
- `default_evaluation_interval` `(string "10s")` - The default evaluation interval
|
||||||
|
that will be applied to all scaling policies which do not specify an evaluation
|
||||||
|
interval.
|
||||||
|
|
||||||
|
## `apm` Block
|
||||||
|
|
||||||
|
The `apm` block is used to configure application performance metric (APM) plugins.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
apm "example-apm-plugin" {
|
||||||
|
driver = "example-apm-plugin"
|
||||||
|
args = ["-my-flag"]
|
||||||
|
|
||||||
|
config = {
|
||||||
|
address = "http://127.0.0.1:9090"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `apm` Parameters
|
||||||
|
|
||||||
|
- `args` `(array<string>: [])` - Specifies a set of arguments to pass to the
|
||||||
|
plugin binary when it is executed.
|
||||||
|
|
||||||
|
- `driver` `(string: "")` - The plugin's executable name relative to to the
|
||||||
|
plugin_dir. If the plugin has a suffix, such as .exe, this should be omitted.
|
||||||
|
|
||||||
|
- `config` `(map<string><string>: nil)` - Specifies configuration values for
|
||||||
|
the plugin either as HCL or JSON. The accepted values are plugin specific.
|
||||||
|
Please refer to the individual plugin's documentation.
|
||||||
|
|
||||||
|
## `target` Block
|
||||||
|
|
||||||
|
The `target` block is used to configure scaling target plugins.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
target "example-target-plugin" {
|
||||||
|
driver = "example-target-plugin"
|
||||||
|
args = ["-my-flag"]
|
||||||
|
|
||||||
|
config = {
|
||||||
|
region = "esp-vlc-1"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `target` Parameters
|
||||||
|
|
||||||
|
- `args` `(array<string>: [])` - Specifies a set of arguments to pass to the
|
||||||
|
plugin binary when it is executed.
|
||||||
|
|
||||||
|
- `driver` `(string: "")` - The plugin's executable name relative to to the
|
||||||
|
plugin_dir. If the plugin has a suffix, such as .exe, this should be omitted.
|
||||||
|
|
||||||
|
- `config` `(map<string><string>: nil)` - Specifies configuration values for
|
||||||
|
the plugin either as HCL or JSON. The accepted values are plugin specific.
|
||||||
|
Please refer to the individual plugin's documentation.
|
||||||
|
|
||||||
|
## `strategy` Block
|
||||||
|
|
||||||
|
The `strategy` block is used to configure scaling strategy plugins.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
strategy "example-strategy-plugin" {
|
||||||
|
driver = "example-strategy-plugin"
|
||||||
|
args = ["-my-flag"]
|
||||||
|
|
||||||
|
config = {
|
||||||
|
algorithm = "complex"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `strategy` Parameters
|
||||||
|
|
||||||
|
- `args` `(array<string>: [])` - Specifies a set of arguments to pass to the
|
||||||
|
plugin binary when it is executed.
|
||||||
|
|
||||||
|
- `driver` `(string: "")` - The plugin's executable name relative to to the
|
||||||
|
plugin_dir. If the plugin has a suffix, such as .exe, this should be omitted.
|
||||||
|
|
||||||
|
- `config` `(map<string><string>: nil)` - Specifies configuration values for
|
||||||
|
the plugin either as HCL or JSON. The accepted values are plugin specific.
|
||||||
|
Please refer to the individual plugin's documentation.
|
||||||
|
|
||||||
|
[hcl_v2]: https://github.com/hashicorp/hcl/tree/hcl2
|
||||||
|
[nomad_namespaces]: https://learn.hashicorp.com/nomad/governance-and-policy/namespaces
|
||||||
|
[nomad_acls]: https://learn.hashicorp.com/nomad?track=acls#acls
|
||||||
|
[autoscaler_cli_config]: /docs/autoscaling/cli#config
|
||||||
|
|
@ -0,0 +1,29 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: HTTP API
|
||||||
|
sidebar_title: API
|
||||||
|
description: Learn about the Nomad Autoscaler HTTP API.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler HTTP API
|
||||||
|
|
||||||
|
The Nomad Autoscaler exposes a small, simple API to be used for health checking
|
||||||
|
the agent.
|
||||||
|
|
||||||
|
## Health API
|
||||||
|
|
||||||
|
This endpoint can be used to query the Nomad Autoscaler agent aliveness. If the
|
||||||
|
agent is alive, the request will return a 200 OK, otherwise it will return a
|
||||||
|
503 ServiceUnavailable.
|
||||||
|
|
||||||
|
| Method | Path | Produces |
|
||||||
|
| ------ | ------------ | ------------------ |
|
||||||
|
| `GET` | `/v1/health` | `application/json` |
|
||||||
|
|
||||||
|
### Sample Request
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ curl \
|
||||||
|
--request PUT \
|
||||||
|
https://localhost:8080/v1/health
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,108 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: CLI
|
||||||
|
sidebar_title: CLI
|
||||||
|
description: >
|
||||||
|
The Nomad Autoscaler can be controlled via a command-line interface. This
|
||||||
|
page documents all the commands the Nomad Autoscaler accepts.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Command: agent
|
||||||
|
|
||||||
|
The agent command is used to start the Nomad Autoscaler which runs until an
|
||||||
|
interrupt signal is received. The Nomad Autoscaler agent's configuration
|
||||||
|
primarily comes from the config files used, but a subset of the options may
|
||||||
|
also be passed directly as CLI arguments. See the
|
||||||
|
[Nomad Autoscaler Agent guide][nomad_autoscaler_agent_guide] for more information
|
||||||
|
on how to use this command and the options it has.
|
||||||
|
|
||||||
|
## Command-line Options
|
||||||
|
|
||||||
|
A subset of the available Nomad Autoscaler agent configuration can optionally be
|
||||||
|
passed in via CLI arguments. The `agent` command accepts the following arguments:
|
||||||
|
|
||||||
|
- `-config=<path>`: The path to either a single config file or a directory of
|
||||||
|
config files to use for configuring the Nomad Autoscaler agent.
|
||||||
|
|
||||||
|
- `-log-level=<level>`: Specify the verbosity level of Nomad Autoscaler's logs.
|
||||||
|
Valid values include DEBUG, INFO, and WARN, in decreasing order of verbosity.
|
||||||
|
The default is `INFO`.
|
||||||
|
|
||||||
|
- `-log-json`: Output logs in a JSON format. The default is false.
|
||||||
|
|
||||||
|
- `-plugin-dir=<path>`: The plugin directory is used to discover Nomad Autoscaler
|
||||||
|
plugins. If not specified, the plugin directory defaults to be that of
|
||||||
|
`<current-dir>/plugins/`.
|
||||||
|
|
||||||
|
- `-http-bind-address=<addr>`: The HTTP address that the health server will bind
|
||||||
|
to. The default is `127.0.0.1`.
|
||||||
|
|
||||||
|
- `-http-bind-port=<port>`: The port that the health server will bind to. The
|
||||||
|
default is `8080`.
|
||||||
|
|
||||||
|
- `-nomad-address=<addr>`: The address of the Nomad server in the form of
|
||||||
|
protocol://addr:port. The default is `http://127.0.0.1:4646`.
|
||||||
|
|
||||||
|
- `-nomad-region=<region>`: The region of the Nomad servers to connect with.
|
||||||
|
|
||||||
|
- `-nomad-namespace=<namespace>`: The target namespace for queries and actions
|
||||||
|
bound to a namespace.
|
||||||
|
|
||||||
|
- `-nomad-token=<token>`: The SecretID of an ACL token to use to authenticate
|
||||||
|
API requests with.
|
||||||
|
|
||||||
|
- `-nomad-http-auth=<username:password>`: The authentication information to use
|
||||||
|
when connecting to a Nomad API which is using HTTP authentication.
|
||||||
|
|
||||||
|
- `-nomad-ca-cert=<path>`: Path to a PEM encoded CA cert file to use to verify
|
||||||
|
the Nomad server SSL certificate.
|
||||||
|
|
||||||
|
- `-nomad-ca-path=<path>`: Path to a directory of PEM encoded CA cert files to
|
||||||
|
verify the Nomad server SSL certificate. If both `-nomad-ca-cert` and
|
||||||
|
`-nomad-ca-path` are specified, `-nomad-ca-cert` is used.
|
||||||
|
|
||||||
|
- `-nomad-client-cert=<path>`: Path to a PEM encoded client certificate for TLS
|
||||||
|
authentication to the Nomad server. Must also specify `-nomad-client-key`.
|
||||||
|
|
||||||
|
- `-nomad-client-key=<path>`: Path to an unencrypted PEM encoded private key
|
||||||
|
matching the client certificate from `-nomad-client-cert`.
|
||||||
|
|
||||||
|
- `-nomad-tls-server-name=<name>`: The server name to use as the SNI host when
|
||||||
|
connecting via TLS.
|
||||||
|
|
||||||
|
- `-nomad-skip-verify`: Do not verify TLS certificates. This is strongly discouraged.
|
||||||
|
|
||||||
|
- `-policy-dir=<path>`: The path to a directory used to load scaling policies.
|
||||||
|
|
||||||
|
- `-policy-default-cooldown=<dur>`: The default cooldown that will be applied to
|
||||||
|
all scaling policies which do not specify a cooldown period. The default is `5m`.
|
||||||
|
|
||||||
|
- `-policy-default-evaluation-interval=<dur>`: The default evaluation interval
|
||||||
|
that will be applied to all scaling policies which do not specify an evaluation
|
||||||
|
interval. The default is `10s`.
|
||||||
|
|
||||||
|
# Nomad Autoscaler Command: version
|
||||||
|
|
||||||
|
The `version` command displays build information about the running binary,
|
||||||
|
including the release version and the exact revision.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```plaintext
|
||||||
|
nomad-autoscaler version
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
This command prints both the version number as well as the exact commit SHA used
|
||||||
|
during the build. The SHA may also have the string `+CHANGES` appended to the
|
||||||
|
end, indicating that local, uncommitted changes were detected at build time.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
```shell-session
|
||||||
|
$ nomad-autoscaler version
|
||||||
|
Nomad Autoscaler v0.0.3-dev (da91fa9)
|
||||||
|
```
|
||||||
|
|
||||||
|
[nomad_autoscaler_agent_guide]: /docs/autoscaling/agent
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Autoscaling
|
||||||
|
sidebar_title: Autoscaling
|
||||||
|
description: |-
|
||||||
|
Overview of the Nomad Autoscaler that provides horizontal application and
|
||||||
|
cluster scaling.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Overview
|
||||||
|
|
||||||
|
This section details the Nomad Autoscaler, a horizontal application and cluster
|
||||||
|
autoscaler for Nomad. The Nomad Autoscaler is built and released separately to
|
||||||
|
Nomad. The source code can be viewed on [GitHub][autoscaler_github] and releases
|
||||||
|
are available on the [HashiCorp releases page][autoscaler_releases] or via
|
||||||
|
[Docker Hub][autoscaler_dockerhub].
|
||||||
|
|
||||||
|
The Nomad Autoscaler repository includes a number of [demos][autoscaler_demo]
|
||||||
|
which provide guided learning on running the autoscaler.
|
||||||
|
|
||||||
|
## Horizontal Application Autoscaling
|
||||||
|
|
||||||
|
Horizontal application autoscaling is the process of automatically controlling the number of instances of an application
|
||||||
|
to have sufficient work throughput to meet service-level agreements (SLA). In
|
||||||
|
Nomad, horizontal application autoscaling can be achieved by modifying the number
|
||||||
|
of allocations in a task group based on the value of a relevant metric, such as
|
||||||
|
CPU and memory utilization or number of open connections. This is enabled by configuring
|
||||||
|
[autoscaling policies][autoscaling_policy] on individual Nomad jobs using the [scaling block][scaling_block].
|
||||||
|
## Horizontal Cluster Autoscaling
|
||||||
|
|
||||||
|
Horizontal cluster autoscaling is the process of adding or removing Nomad clients from a cluster to ensure there
|
||||||
|
is an appropriate amount of cluster resource for the scheduled applications.
|
||||||
|
This is achieved by interacting with remote providers to start or terminate new
|
||||||
|
Nomad clients based on metrics such as the remaining free schedulable CPU or memory.
|
||||||
|
Cluster scaling is enabled by configuring the [autoscaler agent][/docs/autoscaling/agent#dir]
|
||||||
|
with policies targeting the Nomad cluster.
|
||||||
|
|
||||||
|
[scaling_block]: /docs/job-specification/scaling
|
||||||
|
[autoscaling_policy]: /docs/autoscaling/policy
|
||||||
|
[autoscaler_github]: https://github.com/hashicorp/nomad-autoscaler
|
||||||
|
[autoscaler_releases]: https://releases.hashicorp.com/nomad-autoscaler/
|
||||||
|
[autoscaler_dockerhub]: https://hub.docker.com/repository/docker/hashicorp/nomad-autoscaler
|
||||||
|
[autoscaler_demo]: https://github.com/hashicorp/nomad-autoscaler/tree/master/demo
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Checks
|
||||||
|
sidebar_title: Checks
|
||||||
|
description: Learn about how the Autoscaler deals with policy checks.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Check Calculations
|
||||||
|
|
||||||
|
A scaling policy can include several checks all of which produce a scaling
|
||||||
|
suggesting. The checks are executed at the same time during a policy evaluation
|
||||||
|
and the results can conflict with each other. In a scenario like this, the
|
||||||
|
autoscaler iterates the results the chooses the safest result which results in
|
||||||
|
retaining the most capacity of the resource.
|
||||||
|
|
||||||
|
In a scenario where two checks return different desired directions, the following
|
||||||
|
logic is applied.
|
||||||
|
|
||||||
|
- `ScaleOut and ScaleIn => ScaleOut`
|
||||||
|
- `ScaleOut and ScaleNone => ScaleOut`
|
||||||
|
- `ScaleIn and ScaleNone => ScaleNone`
|
||||||
|
|
||||||
|
In situations where the two same actions are suggested, but with different counts the
|
||||||
|
following logic is applied, where the count is the absolute desired value.
|
||||||
|
|
||||||
|
- `ScaleOut(10) and ScaleOut(9) => ScaleOut(10)`
|
||||||
|
- `ScaleIn(3) and ScaleIn(4) => ScaleIn(4)`
|
||||||
|
|
@ -0,0 +1,13 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Internals
|
||||||
|
sidebar_title: Internals
|
||||||
|
description: >
|
||||||
|
This section covers the internals of the Nomad Autoscaler and explains
|
||||||
|
technical details of its operation.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Internals
|
||||||
|
|
||||||
|
This section covers the internals of the Nomad Autoscaler and explains the
|
||||||
|
technical details of how it functions, its architecture, and sub-systems.
|
||||||
|
|
@ -0,0 +1,126 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: APM
|
||||||
|
sidebar_title: APM
|
||||||
|
description: APM plugins provide metric data points describing the resources current state.
|
||||||
|
---
|
||||||
|
|
||||||
|
# APM Plugins
|
||||||
|
|
||||||
|
APMs are used to store metrics about an applications performance and current
|
||||||
|
state. The APM (Application Performance Management) plugin is responsible for
|
||||||
|
querying the APM and returning a value which will be used to determine if
|
||||||
|
scaling should occur.
|
||||||
|
|
||||||
|
## Prometheus APM Plugin
|
||||||
|
|
||||||
|
Use [Prometheus][prometheus_io] metrics to scale your Nomad job task groups or
|
||||||
|
cluster. The query performed on Prometheus should return a single value. You can
|
||||||
|
use the [scalar][prometheus_scaler_function] function in your query to achieve
|
||||||
|
this.
|
||||||
|
|
||||||
|
### Agent Configuration Options
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
apm "prometheus" {
|
||||||
|
driver = "prometheus"
|
||||||
|
|
||||||
|
config = {
|
||||||
|
address = "http://prometheus.my.endpoint.io:9090"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `address` `(string: "http://127.0.0.1:9090")` - The address of the Prometheus
|
||||||
|
endpoint used to perform queries.
|
||||||
|
|
||||||
|
### Policy Configuration Options
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
check {
|
||||||
|
source = "prometheus"
|
||||||
|
query = "scalar(avg((haproxy_server_current_sessions{backend=\"http_back\"}) and (haproxy_server_up{backend=\"http_back\"} == 1)))"
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Nomad APM Plugin
|
||||||
|
|
||||||
|
The Nomad APM plugin allows querying the Nomad API for metric data. This provides
|
||||||
|
an immediate starting point without addition applications but comes at the price
|
||||||
|
of efficiency. When using this APM, it is advised to monitor Nomad carefully
|
||||||
|
ensuring it is not put under excessive load pressure.
|
||||||
|
|
||||||
|
### Agent Configuration Options
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
target "nomad-apm" {
|
||||||
|
driver = "nomad-apm"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Policy Configuration Options - Task Groups
|
||||||
|
|
||||||
|
The Nomad APM allows querying Nomad to understand the current resource usage of
|
||||||
|
a task group.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
check {
|
||||||
|
source = "nomad-apm"
|
||||||
|
query = "avg_cpu"
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Querying Nomad task group metrics is be done using the `operation_metric` syntax,
|
||||||
|
where valid operations are:
|
||||||
|
|
||||||
|
- `avg` - returns the average of the metric value across allocations in the task
|
||||||
|
group.
|
||||||
|
|
||||||
|
- `min` - returns the lowest metric value among the allocations in the task group.
|
||||||
|
|
||||||
|
- `max` - returns the highest metric value among the allocations in the task
|
||||||
|
group.
|
||||||
|
|
||||||
|
- `sum` - returns the sum of all the metric values for the allocations in the
|
||||||
|
task group.
|
||||||
|
|
||||||
|
The metric value can be:
|
||||||
|
|
||||||
|
- `cpu` - CPU usage as reported by the `nomad.client.allocs.cpu.total_percent`
|
||||||
|
metric.
|
||||||
|
|
||||||
|
- `memory` - Memory usage as reported by the `nomad.client.allocs.memory.usage`
|
||||||
|
metric.
|
||||||
|
|
||||||
|
### Policy Configuration Options - Client Nodes
|
||||||
|
|
||||||
|
The Nomad APM allows querying Nomad to understand the current allocated resource
|
||||||
|
as a percentage of the total available.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
check {
|
||||||
|
source = "nomad-apm"
|
||||||
|
query = "percentage-allocated_cpu"
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Querying Nomad client node metrics is be done using the `operation_metric` syntax,
|
||||||
|
where valid operations are:
|
||||||
|
|
||||||
|
- `percentage-allocated` - returns the allocated percentage of the desired
|
||||||
|
resource.
|
||||||
|
|
||||||
|
The metric value can be:
|
||||||
|
|
||||||
|
- `cpu` - allocated CPU as reported by calculating total allocatable against the
|
||||||
|
total allocated by the scheduler.
|
||||||
|
|
||||||
|
- `memory` - allocated memory as reported by calculating total allocatable against
|
||||||
|
the total allocated by the scheduler.
|
||||||
|
|
||||||
|
[prometheus_io]: https://prometheus.io/
|
||||||
|
[prometheus_scaler_function]: https://prometheus.io/docs/prometheus/latest/querying/functions/#scalar
|
||||||
|
[nomad_telemetry_stanza]: /docs/configuration/telemetry#inlinecode-publish_allocation_metrics
|
||||||
|
|
@ -0,0 +1,65 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Plugins
|
||||||
|
sidebar_title: Plugins
|
||||||
|
description: Plugins are used to architect the Nomad Autoscaler into distinct areas.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Plugins
|
||||||
|
|
||||||
|
Plugins are an essential part of the Nomad Autoscaler architecture. The Autoscaler
|
||||||
|
uses the [go-plugin][go_plugin_github] library to implement an ecosystem of
|
||||||
|
different types of plugins. Each plugin type is responsible for a specific task;
|
||||||
|
APM plugins retrieve metrics about the workloads being monitored and Strategy
|
||||||
|
plugins decide which actions Nomad should execute to keep the policy valid. The
|
||||||
|
flexibility of plugins allows the Nomad Autoscaler to be extended to meet specific
|
||||||
|
business requirements or technology use cases.
|
||||||
|
|
||||||
|
The Nomad Autoscaler currently ships with a number of built-in plugins to ease
|
||||||
|
the learning curve. Details of these can be found below, under the specific
|
||||||
|
plugin type sections.
|
||||||
|
|
||||||
|
# General Options
|
||||||
|
|
||||||
|
All plugins which require Nomad API connectivity support the parameters detailed
|
||||||
|
below. These plugins include Nomad APM, Nomad Target and all cluster scaling
|
||||||
|
targets.
|
||||||
|
|
||||||
|
- `nomad_config_inherit` `(bool: true)` - A boolean flag which indicates whether
|
||||||
|
the plugin should inherit the agents Nomad configuration parameters. Plugins
|
||||||
|
can override individual parameters and have their Nomad configuration merged
|
||||||
|
with that of the agent.
|
||||||
|
|
||||||
|
- `nomad_address` `(string: "")` - The address of the Nomad server in the form
|
||||||
|
of protocol://addr:port.
|
||||||
|
|
||||||
|
- `nomad_region` `(string: "")` - The region of the Nomad servers to connect with.
|
||||||
|
|
||||||
|
- `nomad_namespace` `(string: "")` - The target namespace for queries and actions
|
||||||
|
bound to a namespace.
|
||||||
|
|
||||||
|
- `nomad_token` `(string: "")` - The SecretID of an ACL token to use to authenticate
|
||||||
|
API requests with.
|
||||||
|
|
||||||
|
- `nomad_http-auth` `(string: "")` - The authentication information to use when
|
||||||
|
connecting to a Nomad API which is using HTTP authentication.
|
||||||
|
|
||||||
|
- `nomad_ca-cert` `(string: "")` - Path to a PEM encoded CA cert file to use to
|
||||||
|
verify the Nomad server SSL certificate.
|
||||||
|
|
||||||
|
- `nomad_ca-path` `(string: "")` - Path to a directory of PEM encoded CA cert
|
||||||
|
files to verify the Nomad server SSL certificate.
|
||||||
|
|
||||||
|
- `nomad_client-cert` `(string: "")` - Path to a PEM encoded client certificate
|
||||||
|
for TLS authentication to the Nomad server.
|
||||||
|
|
||||||
|
- `nomad-client-key` `(string: "")` - Path to an unencrypted PEM encoded private
|
||||||
|
key matching the client certificate.
|
||||||
|
|
||||||
|
- `nomad_tls-server-name` `(string: "")` - The server name to use as the SNI
|
||||||
|
host when connecting via TLS.
|
||||||
|
|
||||||
|
- `nomad_skip-verify` `(string: "")` - Do not verify TLS certificates. This is
|
||||||
|
strongly discouraged.
|
||||||
|
|
||||||
|
[go_plugin_github]: https://github.com/hashicorp/go-plugin
|
||||||
|
|
@ -0,0 +1,46 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Strategy
|
||||||
|
sidebar_title: Strategy
|
||||||
|
description: Strategy plugins compare the current state of the system against the desired state.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Strategy Plugins
|
||||||
|
|
||||||
|
Strategy plugins compare the current state of the system against the desired state
|
||||||
|
defined by the operator in the scaling policy and generate an action that will
|
||||||
|
bring the system closer to the desired state. In practical terms, strategies
|
||||||
|
receive the current count and a metric value for a task group and output what
|
||||||
|
the new task group count should be.
|
||||||
|
|
||||||
|
## Target Value Strategy Plugin
|
||||||
|
|
||||||
|
The target value strategy plugin will perform count calculations in order to keep
|
||||||
|
the value resulting from the APM query at or around a specified target.
|
||||||
|
|
||||||
|
### Agent Configuration Options
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
strategy "target-value" {
|
||||||
|
driver = "target-value"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Policy Configuration Options
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
check {
|
||||||
|
...
|
||||||
|
strategy "target-value" {
|
||||||
|
target = 20
|
||||||
|
threshold = 0.0001
|
||||||
|
}
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
- `target` `(float: <required>)` - Specifies the metric value the Autscaler
|
||||||
|
should try to meet.
|
||||||
|
|
||||||
|
- `threshold` `(float: 0.01)` - Specifies how significant a change in the input
|
||||||
|
metric should be considered. Small threshold values can lead to output
|
||||||
|
fluctuation.
|
||||||
|
|
@ -0,0 +1,145 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Target
|
||||||
|
sidebar_title: Target
|
||||||
|
description: Target plugins determine where the resource to be autoscaled is located.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Target Plugins
|
||||||
|
|
||||||
|
Target Plugins determine where the resource to be autoscaled is located. All
|
||||||
|
target plugins support the `dry-run` policy config parameter which allows a policy
|
||||||
|
to be evaluated, but will noop any suggested changes.
|
||||||
|
|
||||||
|
## Nomad Task Group Target
|
||||||
|
|
||||||
|
The Nomad task group target indicates the scalable resource is a Nomad job
|
||||||
|
running on a Nomad cluster.
|
||||||
|
|
||||||
|
### Agent Configuration Options
|
||||||
|
|
||||||
|
The Nomad target is automatically launched by the Nomad Autoscaler and so the
|
||||||
|
following setup is optional.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
target "nomad" {
|
||||||
|
driver = "nomad"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Policy Configuration Options
|
||||||
|
|
||||||
|
If using the [Nomad job specification scaling stanza][nomad_scaling_stanza] to
|
||||||
|
configure the scaling policy, the following section can be omitted as Nomad will
|
||||||
|
populate them on job submission.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
check {
|
||||||
|
...
|
||||||
|
target "nomad" {
|
||||||
|
Job = "example"
|
||||||
|
Group = "cache"
|
||||||
|
}
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
- `job` `(string: "") ` - The job identifier which contains the task group to
|
||||||
|
scale as defined within the job specification [job stanza][nomad_job_stanza].
|
||||||
|
|
||||||
|
- `group` `(string: "")` - The name of the task group to scale as defined in the
|
||||||
|
job specification [group stanza][nomad_group_stanza].
|
||||||
|
|
||||||
|
## AWS AutoScaling Group Target
|
||||||
|
|
||||||
|
The AWS ASG target plugin allows for the scaling of the Nomad cluster clients
|
||||||
|
via manipulating [AWS AutoScaling Groups][aws_autoscaling].
|
||||||
|
|
||||||
|
### Agent Configuration Options
|
||||||
|
|
||||||
|
To use the AWS ASG target plugin, the agent configuration needs to be populated
|
||||||
|
with the appropriate target block. Authentication to the AWS API can be supplied
|
||||||
|
in a number of ways including EC2 instance roles. It is recommended, if possible
|
||||||
|
to use the [Vault AWS Secrets engine][vault_aws_backend] for supplying access
|
||||||
|
credentials to the plugin. Credentials should be injected into the configuration
|
||||||
|
via a template rather than as environment variables. This ensures the credentials
|
||||||
|
are passed only to the plugin, rather than being available for all plugins and
|
||||||
|
the agent process.
|
||||||
|
|
||||||
|
The IAM policy required for the AWS ASG plugin to function properly is detailed
|
||||||
|
below.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Action": [
|
||||||
|
"ec2:TerminateInstances",
|
||||||
|
"ec2:DescribeInstanceStatus",
|
||||||
|
"autoscaling:UpdateAutoScalingGroup",
|
||||||
|
"autoscaling:DetachInstances",
|
||||||
|
"autoscaling:DescribeScalingActivities",
|
||||||
|
"autoscaling:DescribeAutoScalingGroups",
|
||||||
|
"autoscaling:CreateOrUpdateTags"
|
||||||
|
],
|
||||||
|
"Resource": "*"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
target "aws-asg" {
|
||||||
|
driver = "aws-asg"
|
||||||
|
config = {
|
||||||
|
aws_region = "eu-west-3"
|
||||||
|
aws_access_key_id = "AKIAIOSFODNN7EXAMPLE"
|
||||||
|
aws_secret_key_id = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `aws_region` `(string: "us-east-1")` - The [AWS region][aws_region] identifier
|
||||||
|
to connect to and where resources should be managed.
|
||||||
|
|
||||||
|
- `aws_access_key_id` `(string: "")` - The AWS access key ID used to authenticate
|
||||||
|
with the AWS API.
|
||||||
|
|
||||||
|
- `aws_secret_key_id` `(string: "")` - The AWS secret key ID used to authenticate
|
||||||
|
with the AWS API.
|
||||||
|
|
||||||
|
- `aws_session_token` `(string: "")` - The AWS session token used to authenticate
|
||||||
|
with the AWS API.
|
||||||
|
|
||||||
|
### Policy Configuration Options
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
check {
|
||||||
|
...
|
||||||
|
target "aws-asg" {
|
||||||
|
asg_name = "hashistack-client-asg"
|
||||||
|
class = "hashistack"
|
||||||
|
drain_deadline = "5m"
|
||||||
|
}
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
- `asg_name` `(string: <required>)` - The name of the AWS AutoScaling Group to
|
||||||
|
interact with when performing scaling actions.
|
||||||
|
|
||||||
|
- `class` `(string: <required>)` - The Nomad [client node class][nomad_node_class]
|
||||||
|
identifier used to group nodes into a pool of resource.
|
||||||
|
|
||||||
|
- `drain_deadline` `(duration: "15m")` The Nomad [drain deadline][nomad_node_drain_deadline]
|
||||||
|
to use when performing node draining actions.
|
||||||
|
|
||||||
|
[nomad_node_class]: https://www.nomadproject.io/docs/configuration/client#node_class
|
||||||
|
[nomad_node_drain_deadline]: https://www.nomadproject.io/api-docs/nodes#deadline
|
||||||
|
[nomad_scaling_stanza]: /docs/job-specification/scaling
|
||||||
|
[nomad_group_stanza]: docs/job-specification/group#group-stanza
|
||||||
|
[nomad_job_stanza]: /docs/job-specification/job#job-stanza
|
||||||
|
[aws_region]: https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
|
||||||
|
[aws_autoscaling]: https://aws.amazon.com/autoscaling/
|
||||||
|
[vault_aws_backend]: https://www.vaultproject.io/docs/secrets/aws
|
||||||
|
|
@ -0,0 +1,96 @@
|
||||||
|
---
|
||||||
|
layout: docs
|
||||||
|
page_title: Scaling Policies
|
||||||
|
sidebar_title: Policy
|
||||||
|
description: >
|
||||||
|
Scaling policies describe the target resource desired state and how to
|
||||||
|
perform calculations to ensure the current state reaches the desired.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Nomad Autoscaler Scaling Policies
|
||||||
|
|
||||||
|
Nomad Autoscaler scaling policies can be configured via the
|
||||||
|
[task group scaling stanza][jobspec_scaling_stanza] or by configuration
|
||||||
|
files stored on disk.
|
||||||
|
|
||||||
|
## Top Level Options
|
||||||
|
|
||||||
|
- `enabled` - A boolean flag that allows operators to administratively disable a
|
||||||
|
policy from active evaluation.
|
||||||
|
|
||||||
|
- `min` - The minimum running count of the targeted resource. This can be 0 or any
|
||||||
|
positive integer.
|
||||||
|
|
||||||
|
- `max` - The maximum running count of the targeted resource. This can be 0 or any
|
||||||
|
positive integer.
|
||||||
|
|
||||||
|
## `policy` Options
|
||||||
|
|
||||||
|
- `cooldown` - A time interval after a scaling action during which no additional
|
||||||
|
scaling will be performed on the resource. It should be provided as a duration
|
||||||
|
(e.g.: "5s", "1m"). If omitted the configuration value
|
||||||
|
[policy_default_cooldown][policy_default_cooldown_agent] from the agent will
|
||||||
|
be used.
|
||||||
|
|
||||||
|
- `evaluation_interval` - Defines how often the policy is evaluated by the
|
||||||
|
Autoscaler. It should be provided as a duration (e.g.: "5s", "1m"). If
|
||||||
|
omitted the configuration value [default_evaluation_interval][eval_interval_agent]
|
||||||
|
from the agent will be used.
|
||||||
|
|
||||||
|
- `target` - Defines where the autoscaling target is running. Detailed information
|
||||||
|
on the configuration options can be found on the [target plugin][target_plugin_docs]
|
||||||
|
page.
|
||||||
|
|
||||||
|
- `check` - Specifies one or more checks to be executed when determining if a
|
||||||
|
scaling action is required.
|
||||||
|
|
||||||
|
## `check` Options
|
||||||
|
|
||||||
|
- `source` - The APM plugin that should handle the metric query. If omitted,
|
||||||
|
this defaults to using the Nomad APM.
|
||||||
|
|
||||||
|
- `query` - The query to run against the specified APM. Currently this query
|
||||||
|
should return a single value. Detailed information on the configuration options
|
||||||
|
can be found on the [apm plugin][apm_plugin_docs] page.
|
||||||
|
|
||||||
|
- `strategy` - The strategy to use, and it's configuration when calculating the
|
||||||
|
desired state based on the current count and the metric returned by the APM.
|
||||||
|
Detailed information on the configuration options can be found on the
|
||||||
|
[strategy plugin][strategy_plugin_docs] page.
|
||||||
|
|
||||||
|
### Example
|
||||||
|
|
||||||
|
A full example of a policy document that can be written into the Nomad task group
|
||||||
|
scaling stanza or via a file within the policy dir can be seen below.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
min = 2
|
||||||
|
max = 10
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
policy {
|
||||||
|
evaluation_interval = "5s"
|
||||||
|
cooldown = "1m"
|
||||||
|
|
||||||
|
target "target" {
|
||||||
|
Job = "example"
|
||||||
|
Group = "example"
|
||||||
|
}
|
||||||
|
|
||||||
|
check "active_connections" {
|
||||||
|
source = "prometheus"
|
||||||
|
query = "scalar(open_connections_example_cache)"
|
||||||
|
|
||||||
|
strategy "target_value" {
|
||||||
|
target = 10
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
[policy_default_cooldown_agent]: /docs/autoscaling/agent#default_cooldown
|
||||||
|
[eval_interval_agent]: /docs/autoscaling/agent#default_evaluation_interval
|
||||||
|
[target_plugin_docs]: /docs/autoscaling/plugins/target
|
||||||
|
[strategy_plugin_docs]: /docs/autoscaling/plugins/strategy
|
||||||
|
[apm_plugin_docs]: /docs/autoscaling/plugins/apm
|
||||||
|
[jobspec_scaling_stanza]: /docs/job-specification/scaling
|
||||||
Loading…
Reference in New Issue