e1cbcf0b3c
rpc: give min rpc version variable a better name
2018-04-09 11:09:05 -07:00
88a9409f8e
rpc: only attempt NodeRpc for nodes>=0.8
...
Attempting NodeRpc (or streaming node rpc) for clients that do not
support it causes it to hang indefinitely because while the TCP
connection exists, the client will never respond.
2018-04-09 11:08:06 -07:00
6254d75eee
Merge pull request #4101 from hashicorp/b-rescheduling-edge-fixes
...
Fixes edge cases around timing/ task finish time being set more than once
2018-04-04 16:18:21 -05:00
5e4525bd30
Moves setting finishedAt to the right place and adds two unit tests.
2018-04-04 14:38:15 -05:00
b1a90462a8
Merge pull request #4094 from hashicorp/b-drain-panic
...
drain: fix double-close panic on drain future
2018-04-04 10:31:14 -07:00
4c9c6decd3
Merge pull request #4100 from hashicorp/b-vault-no-auth
...
Improve handling of Vault errors
2018-04-03 17:23:43 -07:00
af1b185ce4
Fix flaky deadline tests
2018-04-03 16:51:57 -07:00
ba6628a1b6
drain: return on first error
...
If one error is encountered it is unlikely any further attempts will
succeed, so fail fast.
2018-04-03 16:46:35 -07:00
2b14371db5
Fix spelling
2018-04-03 15:58:03 -07:00
9617a13a2b
Correctly handle the upgrade path of a node being drained when applying Raft logs
2018-04-03 15:32:44 -07:00
00537c739b
Fixes edge cases around timing and task finish time being set more than once
2018-04-03 16:34:59 -05:00
58a3ec3fb2
Improve Vault error handling
2018-04-03 14:29:22 -07:00
edc4891283
drain: improve tests and fix spelling
...
* transistion -> transition
* don't t.Fatal in goroutines
* don't mutate global state
2018-04-02 16:40:47 -07:00
6840becf46
drain: refactor batch_future into its own file
...
aka What If structs.go Wasn't So Big?
2018-04-02 16:40:06 -07:00
44a749a7cc
drain: fix double-close panic on drain future
2018-04-02 16:39:18 -07:00
86f9044676
remove generated files
2018-03-30 16:52:49 -07:00
af81349dbe
Generated files
2018-03-30 16:14:40 -07:00
23ec54a372
Merge pull request #4089 from hashicorp/tls-error-fix
...
Check for nil for RPC listener; prevent double closing of listener channel
2018-03-30 16:08:13 -07:00
7f28cfcdfe
small cleanup
2018-03-30 15:49:56 -07:00
a77dd08dd9
prevent double close due to error in creating listener
2018-03-30 17:15:56 -04:00
402a026c88
add further error handling for rpc connection handling
2018-03-30 17:03:36 -04:00
e8809f40dc
Test transistion from both infinite and a future deadline to force
2018-03-30 11:24:39 -07:00
32a673a7e1
Fix force deadline notification
2018-03-30 09:58:29 -07:00
1aa415b0d8
Integration test
2018-03-30 09:33:23 -07:00
dc03fab29b
Canonicalize migrate
2018-03-29 17:42:58 -07:00
e458ab9031
Merge branch 'master' into b-drain-batch
2018-03-29 17:10:34 -07:00
62e9553333
Merge pull request #4069 from hashicorp/f-hashealth
...
add HasHealth helper for nil checks
2018-03-29 17:03:20 -07:00
301704091b
Handle upgrade where Node doesn't have eligiblity
...
This PR handles upgrading a node that has no scheduling eligiblity set.
2018-03-29 16:52:23 -07:00
7d2aae2c11
test handleTaskGroup
2018-03-29 16:38:47 -07:00
049a9213d2
Watch batch jobs
2018-03-29 16:07:51 -07:00
9a732c4acb
Merge pull request #4071 from hashicorp/b-handle-missing-finishedat
...
handle missing finishedAt
2018-03-29 17:11:34 -05:00
f12194328c
Integration test for batch complete case
2018-03-29 13:51:04 -07:00
81d48fc7cf
Merge pull request #4079 from hashicorp/b-filter-desiredstop
...
Filter desired status stop allocs correctly
2018-03-29 15:36:22 -05:00
c8317532ff
Use time from task events if task state does not have FinishedAt set
2018-03-29 14:05:56 -05:00
b194f93f2f
Disallow Update stanza on Batch
2018-03-29 11:28:56 -07:00
91b5bb58d9
add HasHealth helper for nil checks
...
We performed the DeploymentStatus nil checks a couple different ways, so
hopefully this helper will consoldiate them and make it more clear what
the code is doing.
2018-03-29 09:29:19 -07:00
607e631714
Merge pull request #4046 from hashicorp/tls-same-file-reload
...
Check file contents when determining if agent should reload TLS confi…
2018-03-29 10:51:32 -04:00
5090fefe96
Filter out allocs with DesiredState = stop, and unit tests
2018-03-29 09:28:52 -05:00
8776f4b942
Fix failing test
2018-03-29 07:59:38 -05:00
2da661595d
If FinishedAt is not set use alloc's modify time for rescheduling logic
2018-03-29 07:42:58 -05:00
b18f789020
Unmark drain when nodes hit their deadline and only batch/system left and add all job type integration test
2018-03-28 17:25:58 -07:00
b33d909bf9
add test to assert invalid files return error
2018-03-28 18:31:35 -04:00
58ada9bc42
return error when setting checksum; don't reload
2018-03-28 18:15:50 -04:00
2d5af7ff4d
set TLS checksum when parsing config
...
Refactor checksum comparison, always set checksum if it is empty
2018-03-28 09:56:11 -04:00
65ddae86f8
Merge pull request #4054 from hashicorp/b-drainer-index-fix
...
drainer: reset index when new job registered
2018-03-27 16:28:25 -07:00
79a2781585
Merge pull request #4053 from hashicorp/b-drain-sys-jobs-2
...
drain: fix draining of system jobs
2018-03-27 16:26:45 -07:00
de4b3772f1
Create evals for system jobs when drain is unset
...
This PR creates evals for system jobs when:
* Drain is unset and mark eligible is true
* Eligibility is restored to the node
2018-03-27 15:53:24 -07:00
dd5f627feb
set server configuration checksum on reload
2018-03-27 18:03:52 -04:00
ec60a1d3e3
drain: improve comments
2018-03-27 14:27:09 -07:00
e5dfb7e487
drain: unittest draining node logic
2018-03-27 14:24:01 -07:00
a1ed305a24
test: add mock batch and system allocs
...
Since the BatchJob helper had a different task group than the Alloc
helper, it was difficult to create a valid batch alloc.
2018-03-27 14:24:01 -07:00
77bddc7941
drain: stop sys jobs after drain completes
...
System allocs should be drained when a node's deadline is hit or when
all other allocs on the node have stopped/migrated.
2018-03-27 14:24:01 -07:00
fae77b874b
drainer: reset index when new job registered
2018-03-27 14:12:59 -07:00
b522a0fadc
fix up to string to use time.Time
2018-03-27 15:43:57 -04:00
31557cc44f
move tests to use time.Time
2018-03-27 15:43:57 -04:00
003bc209b9
use time.Time for node events for compatibility
2018-03-27 15:43:57 -04:00
6e6d6b7e33
check file contents when determining if agent should reload TLS configuration
2018-03-27 15:42:20 -04:00
59005d1d26
Merge pull request #4049 from hashicorp/b-tunnel
...
Only track nodes if the conn is from the node
2018-03-27 12:39:34 -07:00
5dacb057b7
Only track nodes if the conn is from the node
...
Fixes a bug in which a connection to a Nomad server was treated as a
connection to a node because the server forwarded a node specific RPC.
2018-03-27 09:59:31 -07:00
57e2cd04bd
Merge pull request #4025 from hashicorp/reload-http-tls
...
Allow TLS configurations for HTTP and RPC connections to be reloaded …
2018-03-26 18:00:30 -04:00
539114124e
Fix too long token test case
2018-03-26 16:28:33 -05:00
33e170c15d
s/linear/constant/g
2018-03-26 14:45:09 -05:00
7db930b3c3
Extra test case and better error message for ambiguous config
2018-03-26 13:30:09 -05:00
c2a95f9d7d
add test for upgrading only RPC connections
2018-03-26 10:55:27 -04:00
fbd56c35a8
Adds additional validation for ambigous settings (having both unlimited and attempts set)
2018-03-24 10:29:20 -05:00
39987d5236
Merge branch 'master' into b-acl-name
2018-03-22 14:51:40 -07:00
a7f627e34c
eligbile -> eligible
2018-03-21 16:55:22 -07:00
a4f346abeb
remove spurious TODOs and FIXMEs
2018-03-21 16:55:22 -07:00
9f3086a268
test: must initialize jobResults with new func
2018-03-21 16:51:45 -07:00
e432c9af55
test: disable node drainer during tests
...
Node drainer would throw off the index checks
2018-03-21 16:51:45 -07:00
5c8c4bce2a
test: disable drain during fsm test
...
drainer was unsetting drain before fsm could read written value
2018-03-21 16:51:45 -07:00
341d87aa48
tests: use mock.BatchJob to fix tests
2018-03-21 16:51:45 -07:00
8b107acc06
mock: add BatchJob() helper
2018-03-21 16:51:45 -07:00
cb61a4bdc7
Fix linting errors
2018-03-21 16:51:45 -07:00
640ebdaef6
fix race in drain integration tests
2018-03-21 16:51:45 -07:00
c401d5a098
Refactor assertOps into a helper func
2018-03-21 16:51:45 -07:00
187b0e1a48
Remove debug prints
2018-03-21 16:51:45 -07:00
f67eca48ac
Deregister garbage collected jobs
2018-03-21 16:51:45 -07:00
922842546c
JobNs -> NamespacedID
...
Also drop the New func as it's easy to swap the order of arguments since
they're both strings.
2018-03-21 16:51:45 -07:00
8dc7d9fb6a
drainer: RegisterJob -> RegisterJobs
...
Test job watcher
2018-03-21 16:51:45 -07:00
3116897099
Fix deadline heap triggering
...
Chan must be buffered to avoid skipping triggering altogether
Also made timing in a test a bit more lenient
2018-03-21 16:51:45 -07:00
9d23c965da
fix comment
2018-03-21 16:51:45 -07:00
fb4badf1bc
sharding
2018-03-21 16:51:44 -07:00
2d91b9dfba
Batch drain update
2018-03-21 16:51:44 -07:00
92b636dd32
Fix deadline handling
2018-03-21 16:51:44 -07:00
9898edfa90
Switch to drainerv2 impl
2018-03-21 16:51:44 -07:00
7b2bad8c5e
Toggle Drain allows resetting eligibility
...
This PR allows marking a node as eligible for scheduling while toggling
drain. By default the `nomad node drain -disable` commmand will mark it
as eligible but the drainer will maintain in-eligibility.
2018-03-21 16:51:44 -07:00
ad80e655cc
code review
2018-03-21 16:51:44 -07:00
11f9fe4960
spelling fixes
2018-03-21 16:51:44 -07:00
bc7385812d
Comments
2018-03-21 16:51:44 -07:00
e87c677a42
handle empty node case
2018-03-21 16:51:44 -07:00
405dab2253
integration test and basic fixes
2018-03-21 16:51:44 -07:00
e63bcb474d
Drainer
2018-03-21 16:51:44 -07:00
4754366640
job watcher
2018-03-21 16:51:44 -07:00
504bfabb4d
Node's being untracked or having updated deadlines, updates the deadliner
2018-03-21 16:51:44 -07:00
66eaaa6a4d
node watcher
2018-03-21 16:51:44 -07:00
527ac0b39d
drain heap
2018-03-21 16:51:44 -07:00
2d4c193a0a
Initial design
2018-03-21 16:51:44 -07:00
33ca319080
System test runs on mac
2018-03-21 16:51:44 -07:00
f8d4a3a9e6
Fix file names
2018-03-21 16:51:44 -07:00
32a7649359
refactor main drainloop into 2 more methods
2018-03-21 16:51:44 -07:00
5e52f84bb7
drainer: refactor newStopAllocs, applyMigrations
2018-03-21 16:51:44 -07:00
62960ed7bd
client: don't monitor health of non-service jobs
...
Also fix system job draining; won't work without deadline fixes
2018-03-21 16:51:44 -07:00
a37329189a
Improve DeadlineTime helper
2018-03-21 16:51:44 -07:00
b7c993f0e5
drainer: convert fsm errors to go errors
2018-03-21 16:51:44 -07:00
ab0de41884
drainer: factor job & node watchers out of drainer.go
2018-03-21 16:51:44 -07:00
5922aef623
Restart every time SetEnabled(true) is called
2018-03-21 16:51:44 -07:00
959d447d38
Remove unused context
2018-03-21 16:51:44 -07:00
8b41e9b2e1
drainer: drainer should shutdown with server
2018-03-21 16:51:44 -07:00
0a17076ad2
refactor drainer into a subpkg
2018-03-21 16:51:44 -07:00
93871c18f8
Fix retaining the drain
2018-03-21 16:51:44 -07:00
010a6b8ca5
Unblock evals once eligible
2018-03-21 16:51:44 -07:00
8289cc3c6f
HTTP and API
2018-03-21 16:51:44 -07:00
0fba0101b6
RPC/FSM/State Store for Eligibility
2018-03-21 16:51:44 -07:00
b3d2346419
Upgrade path
2018-03-21 16:51:43 -07:00
2f5309d82a
Remove update time
2018-03-21 16:51:43 -07:00
0965c9ed28
Fix tests
2018-03-21 16:51:43 -07:00
010228577e
Drain cli, api, http
2018-03-21 16:51:43 -07:00
e459a666ed
Node.Drain takes strategy
2018-03-21 16:49:48 -07:00
03d0e5b8a0
improve drain fsm/statestore tests
2018-03-21 16:49:48 -07:00
d1ec65d765
switch to new raft DesiredTransition message
2018-03-21 16:49:48 -07:00
acf59ee75e
drainer: switch to job based watching
2018-03-21 16:49:48 -07:00
db4a634072
RPC, FSM, State Store for marking DesiredTransistion
...
fix build tag
2018-03-21 16:49:48 -07:00
c0542474db
drain: initial drainv2 structs and impl
2018-03-21 16:49:48 -07:00
6fc9231dac
Merge pull request #3856 from hashicorp/f-client-add-health-checks
...
Client driver health checks for Docker
2018-03-21 18:05:00 -04:00
66e44cdb73
Allow TLS configurations for HTTP and RPC connections to be reloaded separately
2018-03-21 17:51:08 -04:00
01898b2c25
Merge pull request #4007 from hashicorp/f-show-rescheduling-cli-job-status
...
Show a section on upcoming delayed evaluations when applicable
2018-03-21 14:37:38 -05:00
f801709a0a
fix issue when updating node events
2018-03-21 15:15:26 -04:00
86b7b3d2d9
fix up health check logic comparison; add node events to client driver checks
2018-03-21 15:15:26 -04:00
d8f68e5ef8
fix up codereview feedback
2018-03-21 15:15:26 -04:00
c7fd0bd8a1
fix up scheduler mocks
2018-03-21 15:15:25 -04:00
c50d02ae93
go style; update comments
2018-03-21 15:15:25 -04:00
a522da6994
fix up gofmt
2018-03-21 15:15:25 -04:00
3aa726baab
fix scheduler driver name; create node structs file
2018-03-21 15:15:25 -04:00
3cba95e8a7
allow nomad to schedule based on the status of a client driver health check
...
Slight updates for go style
2018-03-21 15:15:25 -04:00
0bde357731
add concept of health checks to fingerprinters and nodes
...
fix up feedback from code review
add driver info for all drivers to node
2018-03-21 15:15:25 -04:00
17f2f52f08
Merge pull request #3979 from hashicorp/b_update_compat_delete
...
Delete compatibility code for job level update stanza
2018-03-21 09:17:01 -05:00
70c370c6fe
Merge pull request #4003 from jrasell/f_gh_3988
...
Allow Nomads Consul health check names to be configurable.
2018-03-20 16:44:08 -07:00
121c3bc997
Update Consul check params from using health-check to check.
2018-03-20 16:03:58 +01:00
31a3c81c3b
Show a section on upcoming delayed evaluations when applicable
2018-03-19 21:42:37 -05:00
33a5a72323
Make suggested interval round to seconds, and more end to end test cases
2018-03-19 14:56:52 -05:00
15afef9b77
Allow Nomads Consul health checks to be configurable.
...
This change allows the client HTTP and the server HTTP, Serf and
RPC health check names within Consul to be configurable with the
defaults as previous. The configuration can be done via either a
config file or using CLI flags.
Closes #3988
2018-03-19 19:37:56 +01:00
9e05c9a50e
Merge pull request #3997 from hashicorp/b-serf-addr
...
RPC Advertise used exclusively for Clients
2018-03-19 09:30:20 -07:00
2baa1c38f2
clarify comment
2018-03-16 16:47:08 -07:00
b8607ad6d6
Heartbeat uses client rpc advertise and server defaults server rpc advertise addr
2018-03-16 16:47:08 -07:00
52b7fb5361
Separate client and server rpc advertise addresses
2018-03-16 16:47:08 -07:00
c3e8f6319c
gofmt -s (simplify) files
2018-03-16 16:31:16 -07:00
b3ab063132
Merge pull request #3992 from hashicorp/f-vault-orphan
...
Allow and recommend Orphaned Vault tokens
2018-03-16 10:59:54 -07:00
6a44e6092f
Pull snapshotting out of loop
2018-03-16 10:54:26 -07:00
7545c0053e
job gc uses batch endpoint
2018-03-16 10:53:03 -07:00
586ae36d13
Batch Deregister RPC
2018-03-16 10:53:03 -07:00
c152774997
Allow and recommend Orphaned Vault tokens
...
This PR removes enforcement that the Vault token role disallows orphaned
tokens and recommends orphaned tokens to simplify the
bootstrapping/upgrading of Nomad clusters. The requirement that Nomad's
Vault token never expire and be shared by all instances of Nomad servers
is not operationally friendly.
2018-03-15 15:32:08 -07:00
fc782d5942
List unblocks on summary changes
2018-03-15 10:22:03 -07:00
85be2d99b3
Drop ACL todo
2018-03-14 16:41:46 -07:00
3537c73289
Merge pull request #3978 from hashicorp/b-core-sched
...
Always add core scheduler
2018-03-14 16:13:15 -07:00
56e60e5840
Fix linting warning
2018-03-14 16:12:22 -05:00
9a5e6edf1f
Rename DelayCeiling to MaxDelay
2018-03-14 16:10:32 -05:00
4193015e48
Update comment
2018-03-14 16:10:32 -05:00
3e96c6c4e0
Address more code review feedback
2018-03-14 16:10:32 -05:00
9749b7a05c
Avoids unnecessary timer object creation
2018-03-14 16:10:32 -05:00
9fed0d2103
Get reschedule policy from the alloc directly
2018-03-14 16:10:32 -05:00
4d5e9bcb45
Extra comments, remove unnecessary if condition
2018-03-14 16:10:32 -05:00
c6f333c90f
Move delayheap to lib package
2018-03-14 16:10:32 -05:00
1ab8f2b57a
Address some code review comments
2018-03-14 16:10:32 -05:00
7887f39ff4
Added a delay heap to track evals with WaitUntil set, and use in eval broker
2018-03-14 16:10:32 -05:00
342c3fb961
Added FollowupEvalID field and helper methods to calculate reschedule eligibility based on delay
2018-03-14 16:10:32 -05:00
87538fc87d
Fix formatting
2018-03-14 16:10:32 -05:00
51ec6ec15e
Formatting and linting fixes
2018-03-14 16:10:32 -05:00
5f50c3d618
Add new reschedule options to API layer and unit tests
2018-03-14 16:10:32 -05:00
10c9662222
New delayed rescheduling options, validation function and unit tests
2018-03-14 16:10:32 -05:00
92cb552ff6
Always add core scheduler and detect invalid schedulers
2018-03-14 10:53:27 -07:00
55e4f5cdc4
Require core scheduler
2018-03-14 10:37:49 -07:00
948d917a60
lint warning fixed
2018-03-14 11:30:09 -05:00
a924183604
Remove compat code for upgrade stanza that copied state from job level update stanza
2018-03-14 10:21:46 -05:00
810eedfa2a
Merge pull request #3945 from hashicorp/f-add-node-events
...
Add node events
2018-03-14 08:42:55 -04:00
360d6e5a92
Merge pull request #3968 from hashicorp/f-nicer-vault-error
...
Make server side error messages from vault more clearer
2018-03-13 20:49:39 -05:00
7b5955826d
Fix lint warning
2018-03-13 20:49:01 -05:00
de6ebb6e6c
small cleanup
2018-03-13 18:08:22 -07:00
63e14b7d63
nodeevents -> events
2018-03-13 18:08:22 -07:00
d3c3deffad
fixes
2018-03-13 18:08:22 -07:00
b41501e442
code review feedback
2018-03-13 18:08:21 -07:00
8f109c344c
make check fixes
2018-03-13 18:08:21 -07:00
1488b076d1
code review feedback
2018-03-13 18:08:21 -07:00
19ef872769
keep state store functions in one file
2018-03-13 18:08:21 -07:00
a8655320fd
fix up go check warnings
2018-03-13 18:08:21 -07:00
a8bcbd81e6
batch submitting node events
2018-03-13 18:05:40 -07:00
d30c269fbe
code review feedback
2018-03-13 18:05:40 -07:00
0f306aa0dd
move all structs to structs file
2018-03-13 18:05:40 -07:00
311cb857ac
add node id to tests; check for node events in state store
2018-03-13 18:05:40 -07:00
00d9923454
Ensure node updates don't strip node events
...
Add node events to CLI
2018-03-13 18:05:40 -07:00
ae7a2d6eb1
code review feedback
2018-03-13 18:05:40 -07:00
93b732f97e
move adding node registration event to the state store
2018-03-13 18:05:40 -07:00
e7e4a31f5d
fix up error logging
2018-03-13 18:05:40 -07:00
4ede27a3c8
RPC, FSM, state store for Node.EmitEvent
...
add node event when registering a node for the first time
2018-03-13 18:05:40 -07:00
9e10e26387
NodeEvents struct
2018-03-13 18:05:40 -07:00
e08ecb7da2
Fix incorrect comment
2018-03-13 18:25:41 -05:00
be4c30112d
Code review comment
2018-03-13 17:52:33 -05:00
9618f52746
Remove error wrapping and make vault connection server side errors clearer.
2018-03-13 17:09:03 -05:00
9a4fcaaf34
Fix bug with not including namespace in indexing blocked evals
2018-03-13 13:23:11 -05:00
93b0e42e6c
Fix merge change
2018-03-12 16:25:49 -07:00
Michael Schurter