guide
This commit is contained in:
Alex Dadgar
parent
8540be6964
commit
49248b11ef
|
|
@ -26,10 +26,10 @@ General Options:
|
|||
List Options:
|
||||
|
||||
-json
|
||||
Output the quota specifications in a JSON format.
|
||||
Output the namespaces in a JSON format.
|
||||
|
||||
-t
|
||||
Format and display the quota specifications using a Go template.
|
||||
Format and display the namespaces using a Go template.
|
||||
`
|
||||
return strings.TrimSpace(helpText)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,10 +26,10 @@ General Options:
|
|||
List Options:
|
||||
|
||||
-json
|
||||
Output the namespaces in a JSON format.
|
||||
Output the quota specifications in a JSON format.
|
||||
|
||||
-t
|
||||
Format and display the namespaces using a Go template.
|
||||
Format and display the quota specifications using a Go template.
|
||||
`
|
||||
return strings.TrimSpace(helpText)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,7 +9,8 @@ description: |-
|
|||
# Search HTTP API
|
||||
|
||||
The `/search` endpoint returns matches for a given prefix and context, where a
|
||||
context can be jobs, allocations, evaluations, nodes, or deployments.
|
||||
context can be jobs, allocations, evaluations, nodes, or deployments. When using
|
||||
Nomad Enterprise, the allowed contexts include quotas and namespaces.
|
||||
Additionally, a prefix can be searched for within every context.
|
||||
|
||||
| Method | Path | Produces |
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ The `acl policy delete` command requires the policy name as an argument.
|
|||
|
||||
## Examples
|
||||
|
||||
Delete a ACL Policy:
|
||||
Delete an ACL Policy:
|
||||
|
||||
```
|
||||
$ nomad acl policy delete my-policy
|
||||
|
|
|
|||
|
|
@ -11,9 +11,9 @@ description: |-
|
|||
# Nomad Enterprise Resource Quotas
|
||||
|
||||
In [Nomad Enterprise](https://www.hashicorp.com/products/nomad/), operators can
|
||||
define quota specifications and apply them to namespaces. When a quota is
|
||||
attached to a namespace, the jobs within the namespace may not consume more
|
||||
resources than the quota specification allows.
|
||||
define [quota specifications](/guides/quotas.html) and apply them to namespaces.
|
||||
When a quota is attached to a namespace, the jobs within the namespace may not
|
||||
consume more resources than the quota specification allows.
|
||||
|
||||
This allows operators to partition a shared cluster and ensure that no single
|
||||
actor can consume the whole resources of the cluster.
|
||||
|
|
|
|||
|
|
@ -48,6 +48,7 @@ The following table summarizes the ACL Rules that are available for constructing
|
|||
| [agent](#agent-rules) | Utility operations in the Agent API |
|
||||
| [node](#node-rules) | Node-level catalog operations |
|
||||
| [operator](#operator-rules) | Cluster-level operations in the Operator API |
|
||||
| [quota](#quota-rules) | Quota specification related operations |
|
||||
|
||||
Constructing rules from these policies is covered in detail in the Rule Specification section below.
|
||||
|
||||
|
|
@ -178,6 +179,10 @@ agent {
|
|||
node {
|
||||
policy = "read"
|
||||
}
|
||||
|
||||
quota {
|
||||
policy = "read"
|
||||
}
|
||||
```
|
||||
|
||||
This is equivalent to the following JSON input:
|
||||
|
|
@ -197,6 +202,9 @@ This is equivalent to the following JSON input:
|
|||
},
|
||||
"node": {
|
||||
"policy": "read"
|
||||
},
|
||||
"quota": {
|
||||
"policy": "read"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -284,6 +292,19 @@ operator {
|
|||
|
||||
There's only one operator policy allowed per rule set, and its value is set to one of the policy dispositions. In the example above, the token could be used to query the operator endpoints for diagnostic purposes but not make any changes.
|
||||
|
||||
### Quota Rules
|
||||
|
||||
The `quota` policy controls access to the quota specification operations in the [Quota API](/api/quotas.html), such as quota creation and deletion.
|
||||
Quota rules are specified for all quotas using the `quota` key:
|
||||
|
||||
```
|
||||
agent {
|
||||
policy = "write"
|
||||
}
|
||||
```
|
||||
|
||||
There's only one quota policy allowed per rule set, and its value is set to one of the policy dispositions.
|
||||
|
||||
# Advanced Topics
|
||||
|
||||
### Outages and Mulit-Region Replication
|
||||
|
|
|
|||
|
|
@ -27,10 +27,10 @@ When combined with ACLs, the isolation of namespaces can be enforced, only
|
|||
allowing designated users access to read or modify the jobs and associated
|
||||
objects in a namespace.
|
||||
|
||||
When quotas are applied to a namespace they provide a means to limit resource
|
||||
consumption by the jobs in the namespace. This can prevent a single actor from
|
||||
consuming excessive cluster resources and negatively impacting other teams and
|
||||
applications sharing the cluster.
|
||||
When [resource quotas](/guides/qoutas.html) are applied to a namespace they
|
||||
provide a means to limit resource consumption by the jobs in the namespace. This
|
||||
can prevent a single actor from consuming excessive cluster resources and
|
||||
negatively impacting other teams and applications sharing the cluster.
|
||||
|
||||
## Namespaced Objects
|
||||
|
||||
|
|
@ -39,7 +39,8 @@ jobs, allocations, deployments, and evaluations.
|
|||
|
||||
Nomad does not namespace objects that are shared across multiple namespaces.
|
||||
This includes nodes, [ACL policies](/guides/acl.html), [Sentinel
|
||||
policies](/guides/sentinel-policy.html), and quota specifications.
|
||||
policies](/guides/sentinel-policy.html), and [quota
|
||||
specifications](/guides/quotas.html).
|
||||
|
||||
## Working with Namespaces
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,243 @@
|
|||
---
|
||||
layout: "guides"
|
||||
page_title: "Resource Quotas"
|
||||
sidebar_current: "guides-quotas"
|
||||
description: |-
|
||||
Nomad Enterprise provides support for resource quotas, which allows operators
|
||||
to restrict the aggregate resource usage of namespaces.
|
||||
---
|
||||
|
||||
# Resource Quotas
|
||||
|
||||
Nomad Enterprise provides support for resource quotas, which allows operators to
|
||||
restrict the aggregate resource usage of namespaces.
|
||||
|
||||
~> **Enterprise Only!** This functionality only exists in Nomad Enterprise.
|
||||
This is not present in the open source version of Nomad.
|
||||
|
||||
## Use Case
|
||||
|
||||
When many teams or users are sharing Nomad clusters, there is the concern that a
|
||||
single user could use more than their fair share of resources. Resource quotas
|
||||
provide a mechansim for cluster administrators to restrict the resources that a
|
||||
[namespace](/guides/namespace.html) has access to.
|
||||
|
||||
## Quotas Objects
|
||||
|
||||
Quota specifications are first class objects in Nomad. A quota specification
|
||||
has a unique name, an optional human readable description and a set of quota
|
||||
limits. The quota limits defines the allowed resource usage within a region.
|
||||
|
||||
Quota objects are shareable among namespaces. This allows an operator to define
|
||||
higher level quota specifications, such as a `prod-api` quota, and multiple
|
||||
namespaces can apply the `prod-api` quota specification.
|
||||
|
||||
When a quota specification is attached to a namespace, all resource usage by
|
||||
jobs in the namespaces are accounted toward the quota limits. If the resource is
|
||||
exhausted, allocations with the namespaces will be queued until resources become
|
||||
available by either other jobs finishing or the quota being expanded.
|
||||
|
||||
## Working with Quotas
|
||||
|
||||
For specific details about working with quotas, see the [quotas
|
||||
commands](/docs/commands/quotas.html) and [HTTP API](/api/quotas.html)
|
||||
documentation.
|
||||
|
||||
### Creating quotas:
|
||||
|
||||
Resource quotas can be interacted with using the `nomad quota` subcommand. To
|
||||
get started with creating a quota specification, run `nomad quota init` which
|
||||
produces an example quota specification:
|
||||
|
||||
```
|
||||
$ nomad quota init
|
||||
Example quota specification written to spec.hcl
|
||||
|
||||
$ cat spec.hcl
|
||||
name = "default-quota"
|
||||
description = "Limit the shared default namespace"
|
||||
|
||||
# Create a limit for the global region. Additional limits may
|
||||
# be specified in-order to limit other regions.
|
||||
limit {
|
||||
region = "global"
|
||||
region_limit {
|
||||
cpu = 2500
|
||||
memory = 1000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A quota specification is composed of one or more resource limits. Each limit
|
||||
applies to a particular Nomad region. Within the limit object, operators can
|
||||
specify the allowed cpu and memory usage.
|
||||
|
||||
To create the particular quota, it is as simple as running:
|
||||
|
||||
```
|
||||
$ nomad quota apply spec.hcl
|
||||
Successfully applied quota specification "default-quota"!
|
||||
|
||||
$ nomad quota list
|
||||
Name Description
|
||||
default-quota Limit the shared default namespace
|
||||
api-prod Production instances of backend API servers
|
||||
api-qa QA instances of backend API servers
|
||||
web-prod Production instances of webservers
|
||||
web-qa QA instances of webservers
|
||||
```
|
||||
|
||||
### Attaching Quotas to Namespaces
|
||||
|
||||
In order for a quota to be enforced, we have to attach the quota specification
|
||||
to a namespace. This can be done using the `nomad namespace apply` command.
|
||||
We could add the quota specification we just created to the `default` namespace
|
||||
as follows:
|
||||
|
||||
```
|
||||
$ nomad namespace apply -quota default-quota default
|
||||
Successfully applied namespace "default"!
|
||||
```
|
||||
|
||||
### Viewing Quotas
|
||||
|
||||
Lets now run a job in the default namespace now that we have attached a quota:
|
||||
|
||||
```
|
||||
$ nomad init
|
||||
Example job file written to example.nomad
|
||||
|
||||
$ nomad run -detach example.nomad
|
||||
Job registration successful
|
||||
Evaluation ID: 985a1df8-0221-b891-5dc1-4d31ad4e2dc3
|
||||
|
||||
$ nomad quota status default-quota
|
||||
Name = default-quota
|
||||
Description = Limit the shared default namespace
|
||||
Limits = 1
|
||||
|
||||
Quota Limits
|
||||
Region CPU Usage Memory Usage
|
||||
global 500 / 2500 256 / 1000
|
||||
```
|
||||
|
||||
We can see the newly created job is accounted against the quota specification
|
||||
since it is being run in a namespace that has attached the quota. Now let us
|
||||
scale up the job from `count = 1` to `count = 4`:
|
||||
|
||||
```
|
||||
# Change count
|
||||
$ nomad run -detach example.nomad
|
||||
Job registration successful
|
||||
Evaluation ID: ce8e1941-0189-b866-3dc4-7cd92dc38a69
|
||||
|
||||
$ nomad status example
|
||||
ID = example
|
||||
Name = example
|
||||
Submit Date = 10/16/17 10:51:32 PDT
|
||||
Type = service
|
||||
Priority = 50
|
||||
Datacenters = dc1
|
||||
Status = running
|
||||
Periodic = false
|
||||
Parameterized = false
|
||||
|
||||
Summary
|
||||
Task Group Queued Starting Running Failed Complete Lost
|
||||
cache 1 0 3 0 0 0
|
||||
|
||||
Placement Failure
|
||||
Task Group "cache":
|
||||
* Quota limit hit "memory exhausted (1024 needed > 1000 limit)"
|
||||
|
||||
Latest Deployment
|
||||
ID = 7cd98a69
|
||||
Status = running
|
||||
Description = Deployment is running
|
||||
|
||||
Deployed
|
||||
Task Group Desired Placed Healthy Unhealthy
|
||||
cache 4 3 0 0
|
||||
|
||||
Allocations
|
||||
ID Node ID Task Group Version Desired Status Created At
|
||||
6d735236 81f72d90 cache 1 run running 10/16/17 10:51:32 PDT
|
||||
ce8e1941 81f72d90 cache 1 run running 10/16/17 10:51:32 PDT
|
||||
9b8e185e 81f72d90 cache 1 run running 10/16/17 10:51:24 PDT
|
||||
```
|
||||
|
||||
Here we can see Nomad created two more allocations but did not place the fourth
|
||||
allocation since that would cause the quota to be oversubscribed on memory.
|
||||
|
||||
### ACLs
|
||||
|
||||
Access to quotas can be restricted using [ACLs](/guides/acl.html). As an
|
||||
example we could create an ACL policy that allows read-only access to quotas.
|
||||
|
||||
```
|
||||
# Allow read only access to quotas.
|
||||
quota {
|
||||
policy = "read"
|
||||
}
|
||||
```
|
||||
|
||||
## Resource Limits
|
||||
|
||||
When specifying resource limits the following enforcement behaviors are defined:
|
||||
|
||||
* `limit < 0`: A limit less than zero disallows any access to the resource.
|
||||
|
||||
* `limit == 0`: A limit of zero allows unlimited access to the resource.
|
||||
|
||||
* `limit > 0`: A limit greater than zero enforces that the consumation is less
|
||||
than or equal to the given limit.
|
||||
|
||||
## Federation
|
||||
|
||||
Nomad makes working with quotas in a federated cluster simple by replicating
|
||||
quota specifications from the [authoratative Nomad
|
||||
region](/docs/agent/configuration/server.html#authoritative_region). This allows
|
||||
operators to interact with a single cluster but create quota specifications that
|
||||
apply to all Nomad clusters.
|
||||
|
||||
As an example, we can create a quota specification that applies to two regions:
|
||||
|
||||
```
|
||||
name = "federated-example"
|
||||
description = "A single quota spec effecting multiple regions"
|
||||
|
||||
# Create a limits for two regions
|
||||
limit {
|
||||
region = "europe"
|
||||
region_limit {
|
||||
cpu = 20000
|
||||
memory = 10000
|
||||
}
|
||||
}
|
||||
|
||||
limit {
|
||||
region = "asia"
|
||||
region_limit {
|
||||
cpu = 10000
|
||||
memory = 5000
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If we apply this, and attach it to a namespace with jobs in each region, we can
|
||||
see how the enforcement applies across federated clusters.
|
||||
|
||||
```
|
||||
$ nomad quota apply spec.hcl
|
||||
Successfully applied quota specification "federated-example"!
|
||||
|
||||
$ nomad quota status federated example
|
||||
Name = federated-example
|
||||
Description = A single quota spec effecting multiple regions
|
||||
Limits = 2
|
||||
|
||||
Quota Limits
|
||||
Region CPU Usage Memory Usage
|
||||
asia 2500 / 10000 1000 / 5000
|
||||
europe 8800 / 20000 6000 / 10000
|
||||
```
|
||||
|
|
@ -62,6 +62,10 @@
|
|||
<a href="/guides/outage.html">Outage Recovery</a>
|
||||
</li>
|
||||
|
||||
<li<%= sidebar_current("guides-quotas") %>>
|
||||
<a href="/guides/quotas.html">Resource Quotas</a>
|
||||
</li>
|
||||
|
||||
<li<%= sidebar_current("guides-securing-nomad") %>>
|
||||
<a href="/guides/securing-nomad.html">Securing Nomad</a>
|
||||
</li>
|
||||
|
|
|
|||
Loading…
Reference in New Issue