luxolus/open-nomad
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
open-nomad/nomad
History
Tim Gross a6652bffad
CSI: reorder controller volume detachment (#12387) 
In #12112 and #12113 we solved for the problem of races in releasing
volume claims, but there was a case that we missed. During a node
drain with a controller attach/detach, we can hit a race where we call
controller publish before the unpublish has completed. This is
discouraged in the spec but plugins are supposed to handle it
safely. But if the storage provider's API is slow enough and the
plugin doesn't handle the case safely, the volume can get "locked"
into a state where the provider's API won't detach it cleanly.

Check the claim before making any external controller publish RPC
calls so that Nomad is responsible for the canonical information about
whether a volume is currently claimed.

This has a couple side-effects that also had to get fixed here:

* Changing the order means that the volume will have a past claim
  without a valid external node ID because it came from the client, and
  this uncovered a separate bug where we didn't assert the external node
  ID was valid before returning it. Fallthrough to getting the ID from
  the plugins in the state store in this case. We avoided this
  originally because of concerns around plugins getting lost during node
  drain but now that we've fixed that we may want to revisit it in
  future work.
* We should make sure we're handling `FailedPrecondition` cases from
  the controller plugin the same way we handle other retryable cases.
* Several tests had to be updated because they were assuming we fail
  in a particular order that we're no longer doing.
2022-03-29 09:44:00 -04:00
..
deploymentwatcher ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
drainer drainer: defer CSI plugins until last (#12324) 2022-03-22 10:26:56 -04:00
mock Merge branch 'main' into f-1.3-boogie-nights 2022-03-23 09:41:25 +01:00
state test: move remaining tests to use ci.Parallel. 2022-03-24 08:45:13 +01:00
stream ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
structs Merge branch 'main' into f-1.3-boogie-nights 2022-03-25 16:40:32 +01:00
volumewatcher ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
acl.go
acl_endpoint.go api: apply consistent behaviour of the reverse query parameter (#12244) 2022-03-11 19:44:52 -05:00
acl_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
acl_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
alloc_endpoint.go Merge branch 'main' into f-1.3-boogie-nights 2022-03-23 09:41:25 +01:00
alloc_endpoint_test.go test: move remaining tests to use ci.Parallel. 2022-03-24 08:45:13 +01:00
autopilot.go
autopilot_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
blocked_evals.go cleanup: prevent leaks from time.After 2022-02-02 14:32:26 -06:00
blocked_evals_stats.go Add metrics for blocked eval resources (#10454) 2021-04-29 15:03:45 -04:00
blocked_evals_stats_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
blocked_evals_system.go
blocked_evals_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
client_agent_endpoint.go
client_agent_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
client_alloc_endpoint.go
client_alloc_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
client_csi_endpoint.go
client_csi_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
client_fs_endpoint.go
client_fs_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
client_rpc.go core: remove all traces of unused protocol version 2022-02-18 16:12:36 -08:00
client_rpc_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
client_stats_endpoint.go
client_stats_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
config.go set raft v3 as the default config (#12341) 2022-03-22 15:06:25 -04:00
consul.go consul: correctly check consul acl token namespace when using consul oss 2021-06-08 13:55:57 -05:00
consul_oss_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
consul_policy.go consul: correctly check consul acl token namespace when using consul oss 2021-06-08 13:55:57 -05:00
consul_policy_oss_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
consul_policy_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
consul_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
core_sched.go Add pagination, filtering and sort to more API endpoints (#12186) 2022-03-08 20:54:17 -05:00
core_sched_test.go CSI: reorder controller volume detachment (#12387) 2022-03-29 09:44:00 -04:00
csi_endpoint.go CSI: reorder controller volume detachment (#12387) 2022-03-29 09:44:00 -04:00
csi_endpoint_test.go CSI: reorder controller volume detachment (#12387) 2022-03-29 09:44:00 -04:00
deployment_endpoint.go api: apply consistent behaviour of the reverse query parameter (#12244) 2022-03-11 19:44:52 -05:00
deployment_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
deployment_watcher_shims.go consul: plubming for specifying consul namespace in job/group 2021-04-05 10:03:19 -06:00
drainer_int_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
drainer_shims.go
endpoints_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
eval_broker.go cleanup: prevent leaks from time.After 2022-02-02 14:32:26 -06:00
eval_broker_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
eval_endpoint.go api: add related evals to eval details (#12305) 2022-03-17 13:56:14 -04:00
eval_endpoint_test.go api: add related evals to eval details (#12305) 2022-03-17 13:56:14 -04:00
event_endpoint.go
event_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
fsm.go Merge branch 'main' into f-1.3-boogie-nights 2022-03-23 09:41:25 +01:00
fsm_oss.go chore: ensure consistent file naming for non-enterprise files. 2022-01-13 11:32:16 +01:00
fsm_registry_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
fsm_test.go test: move remaining tests to use ci.Parallel. 2022-03-24 08:45:13 +01:00
heartbeat.go
heartbeat_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
job_endpoint.go Merge branch 'main' into f-1.3-boogie-nights 2022-03-23 09:41:25 +01:00
job_endpoint_hook_connect.go fix panic when Connect mesh gateway doesn't have a proxy block (#11257) 2021-10-04 15:52:07 -04:00
job_endpoint_hook_connect_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
job_endpoint_hook_expose_check.go connect: use deterministic injected dynamic exposed port 2021-04-30 15:18:22 -06:00
job_endpoint_hook_expose_check_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
job_endpoint_hooks.go job: add native service discovery job constraint mutator. 2022-03-14 12:42:12 +01:00
job_endpoint_hooks_test.go test: move remaining tests to use ci.Parallel. 2022-03-24 08:45:13 +01:00
job_endpoint_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
job_endpoint_oss_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
job_endpoint_test.go Merge branch 'main' into f-1.3-boogie-nights 2022-03-25 16:40:32 +01:00
job_endpoint_validators.go namespaces: allow enabling/disabling allowed drivers per namespace 2022-02-24 09:27:32 -05:00
job_endpoint_validators_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
leader.go core: use the new Raft API when removing peers (#12340) 2022-03-22 15:07:31 -04:00
leader_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
leader_test.go core: use the new Raft API when removing peers (#12340) 2022-03-22 15:07:31 -04:00
merge.go
namespace_endpoint.go
namespace_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
node_endpoint.go core: remove node service registrations when node is down. 2022-03-23 09:42:46 +01:00
node_endpoint_test.go test: move remaining tests to use ci.Parallel. 2022-03-24 08:45:13 +01:00
operator_endpoint.go
operator_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
periodic.go
periodic_endpoint.go
periodic_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
periodic_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
plan_apply.go scheduler: detect and log unexpected scheduling collisions (#11793) 2022-01-14 20:09:14 -05:00
plan_apply_oss.go chore: ensure consistent file naming for non-enterprise files. 2022-01-13 11:32:16 +01:00
plan_apply_pool.go
plan_apply_pool_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
plan_apply_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
plan_endpoint.go fix mTLS certificate check on agent to agent RPCs (#11998) 2022-02-04 20:35:20 -05:00
plan_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
plan_normalization_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
plan_queue.go cleanup: prevent leaks from time.After 2022-02-02 14:32:26 -06:00
plan_queue_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
raft_rpc.go
regions_endpoint.go
regions_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
rpc.go tests: deflake test that joins a server with non-voting servers to form qourum 2022-02-24 17:02:58 -06:00
rpc_test.go Merge branch 'main' into f-1.3-boogie-nights 2022-03-25 16:40:32 +01:00
scaling_endpoint.go chore: fixup inconsistent method receiver names. (#11704) 2021-12-20 11:44:21 +01:00
scaling_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
search_endpoint.go api: apply consistent behaviour of the reverse query parameter (#12244) 2022-03-11 19:44:52 -05:00
search_endpoint_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
search_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
serf.go core: remove all traces of unused protocol version 2022-02-18 16:12:36 -08:00
serf_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
server.go Merge branch 'main' into f-1.3-boogie-nights 2022-03-25 16:40:32 +01:00
server_setup_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
server_test.go core: store and check for Raft version changes (#12362) 2022-03-24 14:42:00 -04:00
service_registration_endpoint.go rpc: add service registration RPC endpoints. 2022-03-03 11:25:29 +01:00
service_registration_endpoint_test.go test: move remaining tests to use ci.Parallel. 2022-03-24 08:45:13 +01:00
stats_fetcher.go core: remove all traces of unused protocol version 2022-02-18 16:12:36 -08:00
stats_fetcher_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
status_endpoint.go core: remove all traces of unused protocol version 2022-02-18 16:12:36 -08:00
status_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
system_endpoint.go chore: fix incorrect docstring formatting. 2021-08-30 11:08:12 +02:00
system_endpoint_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
testing.go core: store and check for Raft version changes (#12362) 2022-03-24 14:42:00 -04:00
testing_oss.go gofmt all the files 2021-10-01 10:14:28 -04:00
timetable.go
timetable_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
util.go core: add deprecated mvn tag to serf (#12327) 2022-03-24 14:44:21 -04:00
util_test.go core: add deprecated mvn tag to serf (#12327) 2022-03-24 14:44:21 -04:00
vault.go cleanup: prevent leaks from time.After 2022-02-02 14:32:26 -06:00
vault_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00
vault_testing.go chore: fix incorrect docstring formatting. 2021-08-30 11:08:12 +02:00
worker.go scheduler: detect and log unexpected scheduling collisions (#11793) 2022-01-14 20:09:14 -05:00
worker_string_schedulerworkerstatus.go Make number of scheduler workers reloadable (#11593) 2022-01-06 11:56:13 -05:00
worker_string_workerstatus.go Make number of scheduler workers reloadable (#11593) 2022-01-06 11:56:13 -05:00
worker_test.go ci: swap ci parallelization for unconstrained gomaxprocs 2022-03-15 12:58:52 -05:00