Commit Graph

184 Commits

Author SHA1 Message Date
Mahmood Ali 389e043129 drivers: pass logger through driver plugin client
This fixes a panic whenever driver plugin attempts to log a message.
2019-01-25 09:38:41 -05:00
Alex Dadgar 003aa2a69c nvidia device plugin docs 2019-01-23 10:58:46 -08:00
Michael Schurter 32daa7b47b goimports until make check is happy 2019-01-23 06:27:14 -08:00
Michael Schurter be0bab7c3f move pluginutils -> helper/pluginutils
I wanted a different color bikeshed, so I get to paint it
2019-01-22 15:50:08 -08:00
Alex Dadgar 4bdccab550 goimports 2019-01-22 15:44:31 -08:00
Mahmood Ali 7897dec9d9 api: move formatFloat function
`helpers.FormatFloat` function is only used in `api`.  Moving it and
marking it as private.  We can re-export it if we find value later.
2019-01-18 15:31:31 -05:00
Michael Schurter 40975633ae Update helper/testtask/testtask_windows.go
Co-Authored-By: dantoml <dani@tomlinson.io>
2019-01-17 18:43:14 +01:00
Danielle Tomlinson e6c0738b65 Expand unix build definition 2019-01-17 18:43:13 +01:00
Danielle Tomlinson 96b1d5d20c testtask: Build on windows 2019-01-17 18:43:13 +01:00
Chris Baker e9db2ae822 Merge branch 'master' of github.com:hashicorp/nomad into f-1157-validate-node-meta-variables 2019-01-09 18:56:49 +00:00
Mahmood Ali 0dfa93a3c1 appease linter 2019-01-08 10:58:49 -05:00
Chris Baker bf00f93d87 moved interp key regex out to a helper function 2019-01-08 00:11:47 +00:00
Michael Schurter 337d07fdd8 client/state: improve upgradeTaskBucket error handling
And add a test
2018-12-19 10:39:27 -08:00
Mahmood Ali fa9b9028a5 Use max 3 precision in displaying floats
When formating floats in `nomad node status`, use a maximum precision of
3.
2018-12-10 12:18:24 -05:00
Michael Schurter 4d92603340 boltdd: return error on use-after-Close
Return the same error as boltdb instead of panic'ing.
2018-11-15 14:15:37 -08:00
Mahmood Ali 9da19c6450 address review comments 2018-10-30 13:58:52 -04:00
Mahmood Ali 4937095389 Allow artifacts checksum interpolation
Fixes https://github.com/hashicorp/nomad/issues/4814
2018-10-30 13:24:30 -04:00
Michael Schurter e060174130 ar: fix leader handling, state restoring, and destroying unrun ARs
* Migrated all of the old leader task tests and got them passing
* Refactor and consolidate task killing code in AR to always kill leader
  tasks first
* Fixed lots of issues with state restoring
* Fixed deadlock in AR.Destroy if AR.Run had never been called
* Added a new in memory statedb for testing
2018-10-19 09:45:45 -07:00
Nick Ethier 8b876e1cce fix package references after drivers/base subpackage removed 2018-10-16 16:53:31 -07:00
Nick Ethier 0e3f85222a driver/raw_exec: port existing raw_exec tests and add some testing utilities 2018-10-16 16:53:31 -07:00
Michael Schurter 4236255686 lots of comment/log fixes 2018-10-16 16:53:30 -07:00
Michael Schurter 820af27171 wrap boltdb in a write deduplicator
Saves a tiny bit of cpu and some IO. Sadly doesn't prevent all IO on
duplicate writes as the transactions are still created and committed.

$ go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: github.com/hashicorp/nomad/helper/boltdd
BenchmarkWriteDeduplication_On-4             500           4059591 ns/op           23736 B/op         56 allocs/op
BenchmarkWriteDeduplication_Off-4            300           4115319 ns/op           25942 B/op         55 allocs/op
2018-10-16 16:53:30 -07:00
Michael Schurter ae89b7da95 reimplement success state for tr hooks and state persistence
splits apart local and remote persistence

removes some locking *for now*
2018-10-16 16:53:29 -07:00
Alex Dadgar cbb5f21112 New parser and comparison 2018-10-12 15:25:34 -07:00
oleksii.shyman b4a4b395e3 Introduce nvidia-plugin fingerprinting
- created go-nvml wrapper for fingerprinting
  - added fingerprinting feature to nvidia-plugin
2018-10-03 15:11:56 -07:00
Alex Dadgar 9971b3393f yamux 2018-09-17 14:22:40 -07:00
Alex Dadgar 7739ef51ce agent + consul 2018-09-13 10:43:40 -07:00
Michael Schurter 401ed92847 config: accept CA PEM files with extra whitespace
Previously we did a validation pass over CA PEM files before calling
Go's CertPool.AppendCertsFromPEM to provide more detailed error messages
than the stdlib provides.

Unfortunately our validation was overly strict and rejected valid CA
files. This is actually the reason the stdlib PEM parser doesn't return
meaningful errors: PEM files are extremely permissive and it's difficult
to tell the difference between invalid data and valid metadata.

This PR removes our custom validation as it would reject valid data and
the extra error messages were not useful in diagnosing the error
encountered.
2018-09-06 11:38:56 -07:00
Michael Schurter 6def5bc4f9 client: set host name when migrating over tls
Not setting the host name led the Go HTTP client to expect a certificate
with a DNS-resolvable name. Since Nomad uses `${role}.${region}.nomad`
names ephemeral dir migrations were broken when TLS was enabled.

Added an e2e test to ensure this doesn't break again as it's very
difficult to test and the TLS configuration is very easy to get wrong.
2018-09-05 17:24:17 -07:00
Alex Dadgar c6576ddac1 Fix make check errors 2018-09-04 16:03:52 -07:00
Chelsea Holland Komlo f5e631886f add signature algorithm to error message 2018-08-13 16:21:18 -04:00
Chelsea Holland Komlo ed21481ca1 rename signature algorithm type per code review feedback 2018-08-13 16:11:49 -04:00
Chelsea Holland Komlo 16ffb2e412 extract functionality for determining signature algorithm per code review feedback 2018-08-13 16:08:23 -04:00
Chelsea Holland Komlo 91edec5bf4 change string repr of signature algorithms to constants 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 4b228b1919 remove redundant nil check 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 3f1d54f628 add default case for empty TLS structs 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 4755a65978 add comments 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 86103d41d4 type safety for string keys 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 31d6d00381 add simple getter for certificate 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 568564f63f refactor to use golang built in api for certs 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo bb6c30ee3c add functionality to check if signature algorithm is supported in cipher suites 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo b92098fd08 change function signature to take entire tls config object 2018-08-10 12:37:21 -04:00
Nick Ethier a3be46b5ee
vendor: remove unused github.com/kardianos/osext 2018-07-05 11:04:12 -04:00
Charlie Voiselle 1560d0b893
Extend timeout based on user feedback
Closes https://github.com/hashicorp/nomad/issues/4439.
2018-06-21 15:27:56 -04:00
Chelsea Holland Komlo da712f4f47 fixup! more specific test assertion 2018-06-13 09:58:40 -04:00
Chelsea Holland Komlo dca7235ca5 add tests and improve should reload logic 2018-06-08 15:10:10 -04:00
Chelsea Holland Komlo de03ce8070 move logic to determine whether to reload tls configuration to tlsutil helper 2018-06-08 14:33:58 -04:00
Chelsea Holland Komlo 914d2257ef enable more tls 1.2 ciphers 2018-06-07 17:49:57 -04:00
Alex Dadgar de98774f2c Add test and docs 2018-05-31 18:05:03 -07:00
Alex Dadgar 446fc64850
Merge branch 'master' into f-tls-parse-certs 2018-05-30 17:25:50 +00:00
Chelsea Holland Komlo 3edf309096 fixup! clearify docs and group similar TLS fields 2018-05-29 21:30:49 -04:00
Chelsea Holland Komlo 498b57036d refactor to remove duplication 2018-05-29 18:47:25 -04:00
Chelsea Holland Komlo 1dc14d8e0d handle parsing multiple certificates in a pem file 2018-05-29 18:25:43 -04:00
Chelsea Holland Komlo 9156556555 remove unnecessary type conversation 2018-05-29 17:07:38 -04:00
Chelsea Holland Komlo 521f8d3fb4 parse CA certificate to catch more specific errors 2018-05-25 18:14:32 -04:00
Chelsea Holland Komlo 19e4a5489b add support for tls PreferServerCipherSuites
add further tests for tls configuration
2018-05-25 13:20:00 -04:00
Chelsea Holland Komlo 38f611a7f2 refactor NewTLSConfiguration to pass in verifyIncoming/verifyOutgoing
add missing fields to TLS merge method
2018-05-23 18:35:30 -04:00
Chelsea Komlo 687c26093c
Merge pull request #4269 from hashicorp/f-tls-remove-weak-standards
Configurable TLS cipher suites and versions; disallow weak ciphers
2018-05-11 08:11:46 -04:00
Charlie Voiselle fd952eefbc Added deferred cancel to prevent context leaks 2018-05-10 18:52:54 -04:00
Chelsea Holland Komlo 44f536f18e add support for configurable TLS minimum version 2018-05-09 18:07:12 -04:00
Chelsea Holland Komlo 796bae6f1b allow configurable cipher suites
disallow 3DES and RC4 ciphers

add documentation for tls_cipher_suites
2018-05-09 17:15:31 -04:00
Charlie Voiselle 6e58e1ff4b
Merge branch 'master' into b-extend-win-cpu-fingerprint-timeout 2018-05-09 16:23:14 -04:00
Charlie Voiselle 62f99cc629 Addressed review comments 2018-05-09 13:21:35 -04:00
Charlie Voiselle d64b02f07d Override 3 sec. WMI timeout in gopsutil
The default timeout is too short for some overburdened or resource
constrained machines to complete the WMI query before the context
deadline expires.  This causes them to be unable to fingerprint the CPU
properly.
2018-05-08 17:00:31 -04:00
Charlie Voiselle 893b01158c
Fix the CPU Information error message
The new version of gopsutil introduces a 3 second timeout that could come up as an error here; however, we are outputting the wrong variable and eating the error.
2018-05-08 14:11:29 -04:00
Seth Vargo df4fe7e76c
Set user-agent when talking to GCE metadata 2018-04-10 10:36:46 -04:00
Michael Schurter 187716944f testlog: override testlogger with envvar 2018-03-21 16:49:48 -07:00
Josh Soref 0790a58fb7 spelling: unknown 2018-03-11 19:07:31 +00:00
Alex Dadgar f9cf642436 Client tls 2018-02-15 15:22:57 -08:00
Alex Dadgar aa98f8ba7b Enhance API pkg to utilize Server's Client Tunnel
This PR enhances the API package by having client only RPCs route
through the server when they are low cost and for filesystem access to
first attempt a direct connection to the node and then falling back to
a server routed request.
2018-02-15 13:59:03 -08:00
Alex Dadgar 2c0ad26374 New RPC Modes and basic setup for streaming RPC handlers 2018-02-15 13:59:01 -08:00
Alex Dadgar 6dd1c9f49d Refactor 2018-02-15 13:59:00 -08:00
Alex Dadgar 940a2df8a1 Pull inmem codec to helper 2018-02-15 13:59:00 -08:00
Chelsea Komlo d09cc2a69f
Merge pull request #3492 from hashicorp/f-client-tls-reload
Client/Server TLS dynamic reload
2018-01-23 05:51:32 -05:00
Charlie Voiselle 0f782acfda Allow `.` in Environment Variable Names
From [https://github.com/appc/spec/blob/master/spec/aci.md](https://github.com/appc/spec/blob/master/spec/aci.md):

>environment (list of objects, optional) represents the app's environment variables (ACE can append). The listed objects must have two key-value pairs: name and value. The name must consist solely of letters, digits, and underscores '_' as outlined in IEEE Std 1003.1-2008, 2016 Edition, with practical considerations dictating that the name may also include periods '.' and hyphens '-'. The value is an arbitrary string. These values are not evaluated in any way, and no substitutions are made.

Dotted environment variables are frequently used as a part of the Spring Boot pattern. (re: ZD-6116)

This PR specifically doesn't address the conversion of hyphens (`-`) due to an issue with rkt [[Nomad GH # 2358]](https://github.com/hashicorp/nomad/issues/2358).
2018-01-22 13:59:37 -08:00
Chelsea Holland Komlo 649f86f094 refactor creating a new tls configuration 2018-01-16 08:02:39 -05:00
Michael Schurter 0baf168ed0 Improve naming and docs 2018-01-08 13:36:07 -08:00
Michael Schurter bc10061aa2 Logger backed by *testing.T
For capturing log output in tests and only displaying them on failure.

Pulled out of #3241
2018-01-08 12:53:58 -08:00
Preetha Appan d3110f21bd Changes service name validation logic to ignore any environment variables first. 2017-11-15 15:35:43 -06:00
Chelsea Komlo 2dfda33703 Nomad agent reload TLS configuration on SIGHUP (#3479)
* Allow server TLS configuration to be reloaded via SIGHUP

* dynamic tls reloading for nomad agents

* code cleanup and refactoring

* ensure keyloader is initialized, add comments

* allow downgrading from TLS

* initalize keyloader if necessary

* integration test for tls reload

* fix up test to assert success on reloaded TLS configuration

* failure in loading a new TLS config should remain at current

Reload only the config if agent is already using TLS

* reload agent configuration before specific server/client

lock keyloader before loading/caching a new certificate

* introduce a get-or-set method for keyloader

* fixups from code review

* fix up linting errors

* fixups from code review

* add lock for config updates; improve copy of tls config

* GetCertificate only reloads certificates dynamically for the server

* config updates/copies should be on agent

* improve http integration test

* simplify agent reloading storing a local copy of config

* reuse the same keyloader when reloading

* Test that server and client get reloaded but keep keyloader

* Keyloader exposes GetClientCertificate as well for outgoing connections

* Fix spelling

* correct changelog style
2017-11-14 17:53:23 -08:00
Alex Dadgar 7522c56014 skip running test executables 2017-10-19 16:49:57 -07:00
Alex Dadgar c1cc51dbee sync 2017-10-13 14:36:02 -07:00
Michael Schurter a66c53d45a Remove `structs` import from `api`
Goes a step further and removes structs import from api's tests as well
by moving GenerateUUID to its own package.
2017-09-29 10:36:08 -07:00
Alex Dadgar 4173834231 Enable more linters 2017-09-26 15:26:33 -07:00
Michael Schurter bb8d5689d8 Add Header and Method support for HTTP checks 2017-08-17 16:44:21 -07:00
Alex Dadgar d86b3977b9 Fix alloc health with checks using interpolation
Fixes an issue in which the allocation health watcher was checking for
allocations health based on un-interpolated services and checks. Change
the interface for retrieving check information from Consul to retrieving
all registered services and checks by allocation. In the future this
will allow us to output nicer messages.

Fixes https://github.com/hashicorp/nomad/issues/2969
2017-08-07 16:27:08 -07:00
Alex Dadgar 2650bb1d12 Distinct Property supports arbitrary limit
This PR enhances the distinct_property constraint such that a limit can
be specified in the RTarget/value parameter. This allows constraints
such as:

```
constraint {
  distinct_property = "${meta.rack}"
  value = "2"
}
```

This restricts any given rack from running more than 2 allocations from
the task group.

Fixes https://github.com/hashicorp/nomad/issues/1146
2017-07-31 16:52:13 -07:00
Alex Dadgar 2471b86dec Show submit time 2017-07-07 12:07:07 -07:00
Alex Dadgar 0d42b5d421 initial reconciler 2017-07-07 12:01:17 -07:00
Michael Schurter 89abaf5ef4 Don't fail on first error detecting cpu stats
Since cpu.Counts() never returns an error this doesn't functionally
change anything today.
2017-07-03 14:51:02 -07:00
Michael Schurter fd9bef768f Move task env into execcontext
Also inject PATH into rkt commands since we're no longer appending host
env vars for it.
2017-05-23 13:53:34 -07:00
Alex Dadgar 9b9d69f577 Add a comment 2017-04-10 12:07:57 -07:00
Alex Dadgar 2321e8a4a0 Hash host ID so its stable and well distributed
This PR takes the host ID and runs it through a hash so that it is well
distributed. This makes it so that machines that report similar host IDs
are easily distinguished.

Instances of similar IDs occur on EC2 where the ID is prefixed and on
motherboards created in the same batch.

Fixes https://github.com/hashicorp/nomad/issues/2534
2017-04-10 11:44:51 -07:00
Alex Dadgar abce18749d Fix tests that exec nomad 2017-03-14 16:04:33 -07:00
Alex Dadgar a1a7941dec Various fixes
This PR:
* Uses Go 1.8 executable lookup
* Stores any err message from stats init method
* Allows overriding of Cpu Compute for hosts where it can't be detected
2017-03-14 12:56:31 -07:00
Michael Schurter 16adc44358 Round two of env var cleaning
Should bring us into conformance with IEEE Std 1003.1, 2004 Edition:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html

1 alloc/op and ~80ns/op on my machine.
2017-03-08 16:46:13 -08:00
Alex Dadgar 5be806a3df Fix vet script and fix vet problems
This PR fixes our vet script and fixes all the missed vet changes.

It also fixes pointers being printed in `nomad stop <job>` and `nomad
node-status <node>`.
2017-02-27 16:00:19 -08:00
Diptanu Choudhury 7567209857 Making the job spec return api.Job 2017-02-16 13:52:39 -08:00
Sean Chittenden adb5be23ef
Add better verification of a host's HostID. 2017-02-02 16:24:32 -08:00
Diptanu Choudhury e927de02d2 Moved functions to helper from structs 2017-01-18 15:55:14 -08:00