Commit Graph

242 Commits

Author SHA1 Message Date
Mahmood Ali f7102cd01d
tests: add hcl task driver config parsing tests (#5314)
* drivers: add config parsing tests

Add basic tests for parsing and encoding task config.

* drivers/docker: fix some config declarations

* refactor and document config parse helpers
2019-02-12 14:46:37 -05:00
Michael Schurter 3b84e08fa4
Merge pull request #5297 from hashicorp/b-docker-logging
Docker: Fix logging config parsing
2019-02-11 06:57:52 -08:00
mnachury aef18aa4ca Fix executable check on windows 2019-02-08 16:05:14 -05:00
Michael Schurter e1e4b10884 docker: fix logging config parsing
Fixes
https://groups.google.com/d/topic/nomad-tool/B3Uo6Kns2BI/discussion
2019-02-04 11:07:57 -08:00
Michael Schurter 9bf4b38ab3 plugins: update hclutils test
The test used old local copies of Docker structs and appeared to be
testing an outdated approach to task config decoding.

Updated to use real Docker structs so we can do end-to-end unit testing
of real Docker task configs.
2019-02-04 11:07:57 -08:00
Michael Schurter 6c0cc65b2e simplify hcl2 parsing helper
No need to pass in the entire eval context
2019-02-04 11:07:57 -08:00
Alex Dadgar b2af34141e
Update comment on int8ToPtr 2019-01-30 12:23:43 -08:00
Alex Dadgar 41265d4d61 Change types of weights on spread/affinity 2019-01-30 12:20:38 -08:00
Mahmood Ali 389e043129 drivers: pass logger through driver plugin client
This fixes a panic whenever driver plugin attempts to log a message.
2019-01-25 09:38:41 -05:00
Alex Dadgar 003aa2a69c nvidia device plugin docs 2019-01-23 10:58:46 -08:00
Michael Schurter 32daa7b47b goimports until make check is happy 2019-01-23 06:27:14 -08:00
Michael Schurter be0bab7c3f move pluginutils -> helper/pluginutils
I wanted a different color bikeshed, so I get to paint it
2019-01-22 15:50:08 -08:00
Alex Dadgar 4bdccab550 goimports 2019-01-22 15:44:31 -08:00
Mahmood Ali 7897dec9d9 api: move formatFloat function
`helpers.FormatFloat` function is only used in `api`.  Moving it and
marking it as private.  We can re-export it if we find value later.
2019-01-18 15:31:31 -05:00
Michael Schurter 40975633ae Update helper/testtask/testtask_windows.go
Co-Authored-By: dantoml <dani@tomlinson.io>
2019-01-17 18:43:14 +01:00
Danielle Tomlinson e6c0738b65 Expand unix build definition 2019-01-17 18:43:13 +01:00
Danielle Tomlinson 96b1d5d20c testtask: Build on windows 2019-01-17 18:43:13 +01:00
Chris Baker e9db2ae822 Merge branch 'master' of github.com:hashicorp/nomad into f-1157-validate-node-meta-variables 2019-01-09 18:56:49 +00:00
Mahmood Ali 0dfa93a3c1 appease linter 2019-01-08 10:58:49 -05:00
Chris Baker bf00f93d87 moved interp key regex out to a helper function 2019-01-08 00:11:47 +00:00
Michael Schurter 337d07fdd8 client/state: improve upgradeTaskBucket error handling
And add a test
2018-12-19 10:39:27 -08:00
Mahmood Ali fa9b9028a5 Use max 3 precision in displaying floats
When formating floats in `nomad node status`, use a maximum precision of
3.
2018-12-10 12:18:24 -05:00
Michael Schurter 4d92603340 boltdd: return error on use-after-Close
Return the same error as boltdb instead of panic'ing.
2018-11-15 14:15:37 -08:00
Mahmood Ali 9da19c6450 address review comments 2018-10-30 13:58:52 -04:00
Mahmood Ali 4937095389 Allow artifacts checksum interpolation
Fixes https://github.com/hashicorp/nomad/issues/4814
2018-10-30 13:24:30 -04:00
Michael Schurter e060174130 ar: fix leader handling, state restoring, and destroying unrun ARs
* Migrated all of the old leader task tests and got them passing
* Refactor and consolidate task killing code in AR to always kill leader
  tasks first
* Fixed lots of issues with state restoring
* Fixed deadlock in AR.Destroy if AR.Run had never been called
* Added a new in memory statedb for testing
2018-10-19 09:45:45 -07:00
Nick Ethier 8b876e1cce fix package references after drivers/base subpackage removed 2018-10-16 16:53:31 -07:00
Nick Ethier 0e3f85222a driver/raw_exec: port existing raw_exec tests and add some testing utilities 2018-10-16 16:53:31 -07:00
Michael Schurter 4236255686 lots of comment/log fixes 2018-10-16 16:53:30 -07:00
Michael Schurter 820af27171 wrap boltdb in a write deduplicator
Saves a tiny bit of cpu and some IO. Sadly doesn't prevent all IO on
duplicate writes as the transactions are still created and committed.

$ go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: github.com/hashicorp/nomad/helper/boltdd
BenchmarkWriteDeduplication_On-4             500           4059591 ns/op           23736 B/op         56 allocs/op
BenchmarkWriteDeduplication_Off-4            300           4115319 ns/op           25942 B/op         55 allocs/op
2018-10-16 16:53:30 -07:00
Michael Schurter ae89b7da95 reimplement success state for tr hooks and state persistence
splits apart local and remote persistence

removes some locking *for now*
2018-10-16 16:53:29 -07:00
Alex Dadgar cbb5f21112 New parser and comparison 2018-10-12 15:25:34 -07:00
oleksii.shyman b4a4b395e3 Introduce nvidia-plugin fingerprinting
- created go-nvml wrapper for fingerprinting
  - added fingerprinting feature to nvidia-plugin
2018-10-03 15:11:56 -07:00
Alex Dadgar 9971b3393f yamux 2018-09-17 14:22:40 -07:00
Alex Dadgar 7739ef51ce agent + consul 2018-09-13 10:43:40 -07:00
Michael Schurter 401ed92847 config: accept CA PEM files with extra whitespace
Previously we did a validation pass over CA PEM files before calling
Go's CertPool.AppendCertsFromPEM to provide more detailed error messages
than the stdlib provides.

Unfortunately our validation was overly strict and rejected valid CA
files. This is actually the reason the stdlib PEM parser doesn't return
meaningful errors: PEM files are extremely permissive and it's difficult
to tell the difference between invalid data and valid metadata.

This PR removes our custom validation as it would reject valid data and
the extra error messages were not useful in diagnosing the error
encountered.
2018-09-06 11:38:56 -07:00
Michael Schurter 6def5bc4f9 client: set host name when migrating over tls
Not setting the host name led the Go HTTP client to expect a certificate
with a DNS-resolvable name. Since Nomad uses `${role}.${region}.nomad`
names ephemeral dir migrations were broken when TLS was enabled.

Added an e2e test to ensure this doesn't break again as it's very
difficult to test and the TLS configuration is very easy to get wrong.
2018-09-05 17:24:17 -07:00
Alex Dadgar c6576ddac1 Fix make check errors 2018-09-04 16:03:52 -07:00
Chelsea Holland Komlo f5e631886f add signature algorithm to error message 2018-08-13 16:21:18 -04:00
Chelsea Holland Komlo ed21481ca1 rename signature algorithm type per code review feedback 2018-08-13 16:11:49 -04:00
Chelsea Holland Komlo 16ffb2e412 extract functionality for determining signature algorithm per code review feedback 2018-08-13 16:08:23 -04:00
Chelsea Holland Komlo 91edec5bf4 change string repr of signature algorithms to constants 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 4b228b1919 remove redundant nil check 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 3f1d54f628 add default case for empty TLS structs 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 4755a65978 add comments 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 86103d41d4 type safety for string keys 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 31d6d00381 add simple getter for certificate 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo 568564f63f refactor to use golang built in api for certs 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo bb6c30ee3c add functionality to check if signature algorithm is supported in cipher suites 2018-08-10 12:37:21 -04:00
Chelsea Holland Komlo b92098fd08 change function signature to take entire tls config object 2018-08-10 12:37:21 -04:00
Nick Ethier a3be46b5ee
vendor: remove unused github.com/kardianos/osext 2018-07-05 11:04:12 -04:00
Charlie Voiselle 1560d0b893
Extend timeout based on user feedback
Closes https://github.com/hashicorp/nomad/issues/4439.
2018-06-21 15:27:56 -04:00
Chelsea Holland Komlo da712f4f47 fixup! more specific test assertion 2018-06-13 09:58:40 -04:00
Chelsea Holland Komlo dca7235ca5 add tests and improve should reload logic 2018-06-08 15:10:10 -04:00
Chelsea Holland Komlo de03ce8070 move logic to determine whether to reload tls configuration to tlsutil helper 2018-06-08 14:33:58 -04:00
Chelsea Holland Komlo 914d2257ef enable more tls 1.2 ciphers 2018-06-07 17:49:57 -04:00
Alex Dadgar de98774f2c Add test and docs 2018-05-31 18:05:03 -07:00
Alex Dadgar 446fc64850
Merge branch 'master' into f-tls-parse-certs 2018-05-30 17:25:50 +00:00
Chelsea Holland Komlo 3edf309096 fixup! clearify docs and group similar TLS fields 2018-05-29 21:30:49 -04:00
Chelsea Holland Komlo 498b57036d refactor to remove duplication 2018-05-29 18:47:25 -04:00
Chelsea Holland Komlo 1dc14d8e0d handle parsing multiple certificates in a pem file 2018-05-29 18:25:43 -04:00
Chelsea Holland Komlo 9156556555 remove unnecessary type conversation 2018-05-29 17:07:38 -04:00
Chelsea Holland Komlo 521f8d3fb4 parse CA certificate to catch more specific errors 2018-05-25 18:14:32 -04:00
Chelsea Holland Komlo 19e4a5489b add support for tls PreferServerCipherSuites
add further tests for tls configuration
2018-05-25 13:20:00 -04:00
Chelsea Holland Komlo 38f611a7f2 refactor NewTLSConfiguration to pass in verifyIncoming/verifyOutgoing
add missing fields to TLS merge method
2018-05-23 18:35:30 -04:00
Chelsea Komlo 687c26093c
Merge pull request #4269 from hashicorp/f-tls-remove-weak-standards
Configurable TLS cipher suites and versions; disallow weak ciphers
2018-05-11 08:11:46 -04:00
Charlie Voiselle fd952eefbc Added deferred cancel to prevent context leaks 2018-05-10 18:52:54 -04:00
Chelsea Holland Komlo 44f536f18e add support for configurable TLS minimum version 2018-05-09 18:07:12 -04:00
Chelsea Holland Komlo 796bae6f1b allow configurable cipher suites
disallow 3DES and RC4 ciphers

add documentation for tls_cipher_suites
2018-05-09 17:15:31 -04:00
Charlie Voiselle 6e58e1ff4b
Merge branch 'master' into b-extend-win-cpu-fingerprint-timeout 2018-05-09 16:23:14 -04:00
Charlie Voiselle 62f99cc629 Addressed review comments 2018-05-09 13:21:35 -04:00
Charlie Voiselle d64b02f07d Override 3 sec. WMI timeout in gopsutil
The default timeout is too short for some overburdened or resource
constrained machines to complete the WMI query before the context
deadline expires.  This causes them to be unable to fingerprint the CPU
properly.
2018-05-08 17:00:31 -04:00
Charlie Voiselle 893b01158c
Fix the CPU Information error message
The new version of gopsutil introduces a 3 second timeout that could come up as an error here; however, we are outputting the wrong variable and eating the error.
2018-05-08 14:11:29 -04:00
Seth Vargo df4fe7e76c
Set user-agent when talking to GCE metadata 2018-04-10 10:36:46 -04:00
Michael Schurter 187716944f testlog: override testlogger with envvar 2018-03-21 16:49:48 -07:00
Josh Soref 0790a58fb7 spelling: unknown 2018-03-11 19:07:31 +00:00
Alex Dadgar f9cf642436 Client tls 2018-02-15 15:22:57 -08:00
Alex Dadgar aa98f8ba7b Enhance API pkg to utilize Server's Client Tunnel
This PR enhances the API package by having client only RPCs route
through the server when they are low cost and for filesystem access to
first attempt a direct connection to the node and then falling back to
a server routed request.
2018-02-15 13:59:03 -08:00
Alex Dadgar 2c0ad26374 New RPC Modes and basic setup for streaming RPC handlers 2018-02-15 13:59:01 -08:00
Alex Dadgar 6dd1c9f49d Refactor 2018-02-15 13:59:00 -08:00
Alex Dadgar 940a2df8a1 Pull inmem codec to helper 2018-02-15 13:59:00 -08:00
Chelsea Komlo d09cc2a69f
Merge pull request #3492 from hashicorp/f-client-tls-reload
Client/Server TLS dynamic reload
2018-01-23 05:51:32 -05:00
Charlie Voiselle 0f782acfda Allow `.` in Environment Variable Names
From [https://github.com/appc/spec/blob/master/spec/aci.md](https://github.com/appc/spec/blob/master/spec/aci.md):

>environment (list of objects, optional) represents the app's environment variables (ACE can append). The listed objects must have two key-value pairs: name and value. The name must consist solely of letters, digits, and underscores '_' as outlined in IEEE Std 1003.1-2008, 2016 Edition, with practical considerations dictating that the name may also include periods '.' and hyphens '-'. The value is an arbitrary string. These values are not evaluated in any way, and no substitutions are made.

Dotted environment variables are frequently used as a part of the Spring Boot pattern. (re: ZD-6116)

This PR specifically doesn't address the conversion of hyphens (`-`) due to an issue with rkt [[Nomad GH # 2358]](https://github.com/hashicorp/nomad/issues/2358).
2018-01-22 13:59:37 -08:00
Chelsea Holland Komlo 649f86f094 refactor creating a new tls configuration 2018-01-16 08:02:39 -05:00
Michael Schurter 0baf168ed0 Improve naming and docs 2018-01-08 13:36:07 -08:00
Michael Schurter bc10061aa2 Logger backed by *testing.T
For capturing log output in tests and only displaying them on failure.

Pulled out of #3241
2018-01-08 12:53:58 -08:00
Preetha Appan d3110f21bd Changes service name validation logic to ignore any environment variables first. 2017-11-15 15:35:43 -06:00
Chelsea Komlo 2dfda33703 Nomad agent reload TLS configuration on SIGHUP (#3479)
* Allow server TLS configuration to be reloaded via SIGHUP

* dynamic tls reloading for nomad agents

* code cleanup and refactoring

* ensure keyloader is initialized, add comments

* allow downgrading from TLS

* initalize keyloader if necessary

* integration test for tls reload

* fix up test to assert success on reloaded TLS configuration

* failure in loading a new TLS config should remain at current

Reload only the config if agent is already using TLS

* reload agent configuration before specific server/client

lock keyloader before loading/caching a new certificate

* introduce a get-or-set method for keyloader

* fixups from code review

* fix up linting errors

* fixups from code review

* add lock for config updates; improve copy of tls config

* GetCertificate only reloads certificates dynamically for the server

* config updates/copies should be on agent

* improve http integration test

* simplify agent reloading storing a local copy of config

* reuse the same keyloader when reloading

* Test that server and client get reloaded but keep keyloader

* Keyloader exposes GetClientCertificate as well for outgoing connections

* Fix spelling

* correct changelog style
2017-11-14 17:53:23 -08:00
Alex Dadgar 7522c56014 skip running test executables 2017-10-19 16:49:57 -07:00
Alex Dadgar c1cc51dbee sync 2017-10-13 14:36:02 -07:00
Michael Schurter a66c53d45a Remove `structs` import from `api`
Goes a step further and removes structs import from api's tests as well
by moving GenerateUUID to its own package.
2017-09-29 10:36:08 -07:00
Alex Dadgar 4173834231 Enable more linters 2017-09-26 15:26:33 -07:00
Michael Schurter bb8d5689d8 Add Header and Method support for HTTP checks 2017-08-17 16:44:21 -07:00
Alex Dadgar d86b3977b9 Fix alloc health with checks using interpolation
Fixes an issue in which the allocation health watcher was checking for
allocations health based on un-interpolated services and checks. Change
the interface for retrieving check information from Consul to retrieving
all registered services and checks by allocation. In the future this
will allow us to output nicer messages.

Fixes https://github.com/hashicorp/nomad/issues/2969
2017-08-07 16:27:08 -07:00
Alex Dadgar 2650bb1d12 Distinct Property supports arbitrary limit
This PR enhances the distinct_property constraint such that a limit can
be specified in the RTarget/value parameter. This allows constraints
such as:

```
constraint {
  distinct_property = "${meta.rack}"
  value = "2"
}
```

This restricts any given rack from running more than 2 allocations from
the task group.

Fixes https://github.com/hashicorp/nomad/issues/1146
2017-07-31 16:52:13 -07:00
Alex Dadgar 2471b86dec Show submit time 2017-07-07 12:07:07 -07:00
Alex Dadgar 0d42b5d421 initial reconciler 2017-07-07 12:01:17 -07:00
Michael Schurter 89abaf5ef4 Don't fail on first error detecting cpu stats
Since cpu.Counts() never returns an error this doesn't functionally
change anything today.
2017-07-03 14:51:02 -07:00
Michael Schurter fd9bef768f Move task env into execcontext
Also inject PATH into rkt commands since we're no longer appending host
env vars for it.
2017-05-23 13:53:34 -07:00
Alex Dadgar 9b9d69f577 Add a comment 2017-04-10 12:07:57 -07:00
Alex Dadgar 2321e8a4a0 Hash host ID so its stable and well distributed
This PR takes the host ID and runs it through a hash so that it is well
distributed. This makes it so that machines that report similar host IDs
are easily distinguished.

Instances of similar IDs occur on EC2 where the ID is prefixed and on
motherboards created in the same batch.

Fixes https://github.com/hashicorp/nomad/issues/2534
2017-04-10 11:44:51 -07:00
Alex Dadgar abce18749d Fix tests that exec nomad 2017-03-14 16:04:33 -07:00
Alex Dadgar a1a7941dec Various fixes
This PR:
* Uses Go 1.8 executable lookup
* Stores any err message from stats init method
* Allows overriding of Cpu Compute for hosts where it can't be detected
2017-03-14 12:56:31 -07:00
Michael Schurter 16adc44358 Round two of env var cleaning
Should bring us into conformance with IEEE Std 1003.1, 2004 Edition:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html

1 alloc/op and ~80ns/op on my machine.
2017-03-08 16:46:13 -08:00
Alex Dadgar 5be806a3df Fix vet script and fix vet problems
This PR fixes our vet script and fixes all the missed vet changes.

It also fixes pointers being printed in `nomad stop <job>` and `nomad
node-status <node>`.
2017-02-27 16:00:19 -08:00
Diptanu Choudhury 7567209857 Making the job spec return api.Job 2017-02-16 13:52:39 -08:00
Sean Chittenden adb5be23ef
Add better verification of a host's HostID. 2017-02-02 16:24:32 -08:00
Diptanu Choudhury e927de02d2 Moved functions to helper from structs 2017-01-18 15:55:14 -08:00
Michael Schurter d36d716bf9 Add docs for generating example certificates 2016-11-15 17:22:54 -08:00
Michael Schurter 345a2640dc Fix tlsutil tests 2016-11-10 12:18:13 -08:00
Alex Dadgar 5fba85c092 get tlsutil tests to compile - need to regenerate the certificates 2016-11-09 14:41:08 -08:00
Michael Schurter ae680c9c81 Remove incorrectly committed line and wrong comment 2016-11-01 15:57:21 -07:00
Michael Schurter 536c2921e9 Remove ServerName because we verify based on region 2016-11-01 14:17:31 -07:00
Diptanu Choudhury 1a8fa8c8d5 Making Nomad TLS configs region aware 2016-11-01 11:55:29 -07:00
Diptanu Choudhury 7c61e115bd Moved tlsutil into helpers 2016-10-25 16:05:37 -07:00
Alex Dadgar 751aa114bf Fix Vault parsing of booleans 2016-10-10 18:04:39 -07:00
Alex Dadgar 4ff8edd2da Floor CPU MHz and total compute and mark hostname as unique 2016-06-22 15:01:36 -07:00
Sean Chittenden 21b84fc3e6
Memoize the CPU stats. Error if CPU fingerprinting fails. 2016-06-17 12:13:53 -07:00
Diptanu Choudhury 7fb507e810 Moving the clkspeed code to helper 2016-06-11 17:31:49 +02:00
Diptanu Choudhury ea65fd7925 Checking in the stats helper package 2016-06-10 23:46:33 +02:00
Sean Chittenden dc28ab0cb5
Speling police 2016-05-15 09:41:34 -07:00
Alex Dadgar b4bb28c425 Job diff using generic structures 2016-05-10 22:23:34 -07:00
Ivo Verberk cd301327ed Add comments and fix a typo 2016-04-11 23:09:09 +02:00
Ivo Verberk 542603dec6 Add helper to validate raw configuration data 2016-04-10 00:42:43 +02:00
Diptanu Choudhury 5439d4c23c Interpolating service tags 2016-03-28 15:02:00 -07:00
Aleksejs Sinicins 0ce9ea3bab Allow dashes in var names 2016-02-27 18:42:33 +02:00
Alex Dadgar 5018f5dd1e Only interpret vars wrapped in braces 2016-02-04 17:26:46 -08:00
Alex Dadgar 1ceb6f012a Fix a bunch of tests
Up timeouts

trusty travis beta

Increase timeouts
2016-01-20 16:03:53 -08:00
Alex Dadgar 3ba1c9b76b merge 2016-01-11 09:58:26 -08:00
Diptanu Choudhury f6fb42835e Using cgo dependencies to look up users 2015-12-15 11:12:13 -08:00
Chris Bednarski 1e3e120646 Added build flag to user-lookup so it does not build on windows 2015-12-01 14:28:12 -08:00
Chris Bednarski 9292a97062 Merge branch 'user-lookup-nocgo' of https://github.com/carlosdp/nomad into b-user-lookup 2015-12-01 13:44:56 -08:00
Carlos Diaz-Padron 55e49506f0 Refactor out userLookup to helper package
Also replaces user.Lookup in exec driver
2015-12-01 11:59:55 -08:00
Diptanu Choudhury 29915ddd16 Moving the args to helper 2015-11-26 14:13:19 -08:00
Chris Hines 5b2168bb12 Use package testtask and httptest.Server to make client/driver tests OS independent. 2015-11-25 15:56:20 -05:00
Chris Hines d7ebe099c1 Factor portable test task out of client/driver/executor. 2015-11-24 20:59:42 -05:00
Alex Dadgar 88391eedfb Search path 2015-11-03 13:26:09 -08:00
Alex Dadgar 5562fc7672 Create Spawn pkg that handles IPC with the spawn-daemon and update exec_linux to use that 2015-11-02 20:28:37 -08:00
Ivo Verberk c6e1b13b51 Fix vet warnings 2015-10-07 12:26:58 +02:00
Alex Dadgar 8230008114 Correctly scan the CWD for the nomad executable 2015-09-30 14:30:09 -07:00
Alex Dadgar 0e3f21b34f Linux executor with cgroup isolation support 2015-09-21 09:08:57 -07:00
Armon Dadgar adb771ed1d agent: moving functions into helpers 2015-08-23 16:57:54 -07:00