Commit graph

20998 commits

Author SHA1 Message Date
Chris Baker b11a092d2d added missing import from command/agent 2021-04-02 13:53:28 +00:00
Chris Baker 21bc48ca29 json handles were moved to a new package in #10202
this was unecessary after refactoring, so this moves them back to their
original location in package structs
2021-04-02 13:31:10 +00:00
Mahmood Ali b7529217fe
Merge pull request #10285 from hashicorp/build-no-codegen-by-default
dev: ignore msgpack codegen code in dev by default
2021-04-01 22:03:01 -04:00
Chris Baker d9f3115606
Merge pull request #10202 from hashicorp/f-node-drain-api
remove deprecated `Drain` field from `structs.Node`
2021-04-01 16:48:53 -05:00
Mahmood Ali 24fe595de4 dev: ignore msgpack codegen code in dev by default
This change disables using msgpack generated serializers in dev by
default.

In released binaries, we use code-generated msgpack serializers to
improve performance.  However, in development, code generated
serializers are a pain. If a developer forgets to re-generate code, the
code generated gets out of sync with the go structs, and result into
subtle bugs where some values appear not to persist as expected.

The CI and release scripts will continue to use the msgpack
code-generation. Devs who want to test locally can set
`GO_TAGS=codegen_generated` as well.
2021-04-01 16:27:18 -04:00
Chris Baker 436d46bd19
Merge branch 'main' into f-node-drain-api 2021-04-01 15:22:57 -05:00
James Rasell 138adfdc68
docs: add fixed-value and pass-through autoscaler plugin docs. (#10278)
docs: add fixed-value and pass-through autoscaler plugin docs.
2021-04-01 16:11:37 -04:00
Luiz Aoqui 6bbbc19eb7
Merge pull request #10199 from hashicorp/docs-update-autoscaling-aws-permission
docs: Update AWS permission requirements for the Autoscaler
2021-04-01 16:10:28 -04:00
Luiz Aoqui f802fe3789
Merge pull request #10183 from hashicorp/docs-autoscaling-policy-eval-cli
docs: add Autoscaler CLI flags for policy eval configuration
2021-04-01 16:10:01 -04:00
Tim Gross 0856483115 CSI: fingerprint detailed node capabilities
In order to support new node RPCs, we need to fingerprint plugin capabilities
in more detail. This changeset mirrors recent work to fingerprint controller
capabilities, but is not yet in use by any Nomad RPC.
2021-04-01 16:00:58 -04:00
Buck Doyle 6d67e90763
Add exchange of one-time token on UI load (#10066)
This adds UI support for receiving the one-time token passed via query parameter, as in #10134
and related PRs, and exchanging it for its corresponding secret ID. When this works, it’s mostly
invisible, with a brief flash of the OTT onscreen.

The authentication failure message now suggests the -authenticate flag.

When OTT exchange fails, it shows a whole-page error.

This includes a known UX shortcoming in that the OTT will not disappear from the URL when an
identifier is specified on the command line, like nomad ui -authenticate jobname. The goal is to
address that shortcoming in a forthcoming pull request.
2021-04-01 13:21:30 -05:00
Tim Gross 466b620fa4
CSI: volume snapshot 2021-04-01 11:16:52 -04:00
James Rasell c5bb1cf69c
deps: run make sync to sync vendor dir 2021-04-01 11:06:57 -04:00
James Rasell d3dee04ef8
Merge pull request #10270 from hashicorp/f-docs-autoscaling-selector-strategy
docs: add autoscaling docs to detail node selector strategy feat.
2021-04-01 16:51:00 +02:00
Tim Gross 8fa919780b
CSI: ensure api package has godoc documentation 2021-04-01 09:50:07 -04:00
James Rasell d07d799ff1
docs: add autoscaling docs to detail node selector strategy feat. 2021-04-01 08:47:16 +02:00
Michael Lange c337155020
Merge pull request #10257 from hashicorp/f-ui/bigger-units
UI: Refactor all displayed units to use common utils (featuring bigger suffixes)
2021-03-31 15:18:05 -07:00
Tim Gross 0340d748b7 update volume init for new volume spec 2021-03-31 16:37:09 -04:00
Tim Gross 0d3e564633 CSI: CLI for create/delete/list
Add new commands for creating, deleting, and listing external storage
volumes. Includes HCL decoding update for volume spec so that we can humanize
capacity bytes input values.
2021-03-31 16:37:09 -04:00
Tim Gross 71b9daffb9 CSI: fix misleading HTTP test
The HTTP test to create CSI volumes depends on having a controller plugin to
talk to, but the test was using a node-only plugin, which allows it to
silently ignore the missing controller.
2021-03-31 16:37:09 -04:00
Tim Gross 9fc4cf1419 CSI: fingerprint detailed controller capabilities
In order to support new controller RPCs, we need to fingerprint volume
capabilities in more detail and perform controller RPCs only when the specific
capability is present. This fixes a bug in Ceph support where the plugin can
only suport create/delete but we assume that it also supports attach/detach.
2021-03-31 16:37:09 -04:00
Tim Gross f149abfa41 CSI: volume creation/registration should not validate attachment
The CSI specification requires that we validate a list of `Capability` (access
mode + accessibility) when we create volume, but the existing volume
registration workflow incorrectly validates a single capability. The
specific capability required by a volume claim is checked at the time we make
the claim, so remove the check for `AttachmentMode`/`AcccessMode`.
2021-03-31 16:37:09 -04:00
Tim Gross aec5337862 CSI: HTTP handlers for create/delete/list 2021-03-31 16:37:09 -04:00
Tim Gross d38008176e CSI: create/delete/list volume RPCs
This commit implements the RPC handlers on the client that talk to the CSI
plugins on that client for the Create/Delete/List RPC.
2021-03-31 16:37:09 -04:00
Tim Gross 43622680fa test infrastructure for mock client RPCs (#10193)
This commit includes a new test client that allows overriding the RPC
protocols. Only the RPCs that are passed in are registered, which lets you
implement a mock RPC in the server tests. This commit includes an example of
this for the ClientCSI RPC server.
2021-03-31 16:37:09 -04:00
Tim Gross d97401f60e CSI: protobuffer mappings for Create/Delete/List volume RPCs
Note that unset proto fields for volume create should be nil. The CSI spec
handles empty fields and nil fields in the protobuf differently, which may
result in validation failures for creating volumes with no prior source (and
does in testing with the AWS EBS plugin). Refactor the `CreateVolumeRequest`
mapping to the protobuf in the plugin client to avoid this bug.
2021-03-31 16:37:09 -04:00
Tim Gross bdb5b87a48 docs: CLI commands for volume create/delete 2021-03-31 16:37:09 -04:00
Tim Gross e76eeeb848 drivers/docker: fix flaky image coordinator test
The test assertion that we don't have a delete future remaining races with the
code its testing, because the removal of the image and the removal of the
future are not atomic. Move this assertion into a `WaitForResult` to avoid
test flakes which we're seeing on CI on Windows in particular.
2021-03-31 15:59:01 -04:00
Michael Lange b6af7a31e2 Aggregate the BANs in the cluster details panel 2021-03-31 09:29:14 -07:00
Michael Lange a0e59f81ab Use common helpers and utils for formatting hertz 2021-03-31 09:29:14 -07:00
Michael Lange bf6fb48ca6 Format all bytes using helpers, even the ones that are already MiBs 2021-03-31 09:29:02 -07:00
Michael Lange 2a614b18a2 Add startingUnit option to the unit formatting utils 2021-03-31 09:29:02 -07:00
Michael Lange dc41e731ce format-scheduled-bytes helper 2021-03-31 09:29:02 -07:00
Michael Lange 285c9a9b50 Refactor formatBytes helper to use core units util 2021-03-31 09:29:01 -07:00
Michael Lange f6a377cd12 Units utils
Consolidates all the bytes and hertz units logic in one place.
2021-03-31 09:29:01 -07:00
zhsj 5a182e1d03
deps: update runc to v1.0.0-rc93
includes updates for breaking changes in runc v1.0.0-rc93
2021-03-31 10:57:02 -04:00
Drew Bailey d8d3faef39
Docs/licensing reference docs (#10260)
* add a license docs page

* license reference docs

* update wording

* use new docs-nav

* rm file held over from rebase, fix path
2021-03-31 10:24:27 -04:00
Mahmood Ali b0ec8d3b80
Merge pull request #10247 from hashicorp/f-memory-oversubscription-2
Memory oversubscription
2021-03-31 09:57:19 -04:00
Bryce Kalow a6ca40fa4e
feat(website): migrates to new nav data format (#10264) 2021-03-31 08:43:17 -05:00
Mahmood Ali e24dac2f39 fixup! oversubscription: Add MemoryMaxMB to internal structs 2021-03-31 09:26:26 -04:00
Mahmood Ali 4ec2d8f5e4 fixup! oversubscription: Add MemoryMaxMB to internal structs 2021-03-31 08:52:20 -04:00
Mahmood Ali bf1c0dcf17 driver/exec: set soft memory limit
Linux offers soft memory limit:
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html#soft-limits
, and
https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html?highlight=memory.low
.

We can set soft memory limits through libcontainer
`Resources.MemoryReservation`: https://pkg.go.dev/github.com/opencontainers/runc@v0.1.1/libcontainer/configs#Resources
2021-03-30 16:55:58 -04:00
Mahmood Ali 85502c1739 oversubscription: e2e tests! 2021-03-30 16:55:58 -04:00
Mahmood Ali f44a04454d oversubscription: driver/exec to honor MemoryMaxMB 2021-03-30 16:55:58 -04:00
Mahmood Ali 275feb5bec oversubscription: docker to honor MemoryMaxMB values 2021-03-30 16:55:58 -04:00
Mahmood Ali 95d85b9cac oversubscription: set the linux memory limit
Use the MemoryMaxMB as the LinuxResources limit. This is intended to ease
drivers implementation and adoption of the features: drivers that use
`resources.LinuxResources.MemoryLimitBytes` don't need to be updated.

Drivers that use NomadResources will need to updated to track the new
field value. Given that tasks aren't guaranteed to use up the excess
memory limit, this is a reasonable compromise.
2021-03-30 16:55:58 -04:00
Mahmood Ali 04a96016e4 oversubscription: add memory_max to hclv1
Allow specifying the `memory_max` field in HCL under the resources block.

Though HCLv1 is deprecated, I've updated them to ease our testing.
2021-03-30 16:55:58 -04:00
Mahmood Ali 18b581656d oversubscription: adds CLI and API support
This commit updates the API to pass the MemoryMaxMB field, and the CLI to show
the max set for the task.

Also, start parsing the MemoryMaxMB in hcl2, as it's set by tags.

A sample CLI output; note the additional `Max: ` for "task":

```
$ nomad alloc status 96fbeb0b
ID                  = 96fbeb0b-a0b3-aa95-62bf-b8a39492fd5c
[...]

Task "cgroup-fetcher" is "running"
Task Resources
CPU        Memory         Disk     Addresses
0/500 MHz  32 MiB/20 MiB  300 MiB

Task Events:
[...]

Task "task" is "running"
Task Resources
CPU        Memory          Disk     Addresses
0/500 MHz  176 KiB/20 MiB  300 MiB
           Max: 30 MiB

Task Events:
[...]
```
2021-03-30 16:55:58 -04:00
Mahmood Ali 0c2551270a oversubscription: Add MemoryMaxMB to internal structs
Start tracking a new MemoryMaxMB field that represents the maximum memory a task
may use in the client. This allows tasks to specify a memory reservation (to be
used by scheduler when placing the task) but use excess memory used on the
client if the client has any.

This commit adds the server tracking for the value, and ensures that allocations
AllocatedResource fields include the value.
2021-03-30 16:55:58 -04:00
Tim Gross f820021f9e deps: bump gopsutil to v3.21.2 2021-03-30 16:02:51 -04:00