luxolus
/
open-nomad
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity

variables: limit rekey eval to half the nack timeout (#15102) 

In order to limit how much the rekey job can monopolize a scheduler worker, we
limit how long it can run to 1min before stopping work and emitting a new
eval. But this exactly matches the default nack timeout, so it'll fail the eval
rather than getting a chance to emit a new one.

Set the timeout for the rekey eval to half the configured nack timeout.
This commit is contained in:
Tim Gross 2022-11-01 16:50:50 -04:00 committed by GitHub
parent 903b5baaa4
commit 38542f256e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.changelog/15102.txt Normal file
View File

@ -0,0 +1,3 @@
```release-note:bug
variables: Fixed a bug where a long-running rekey could hit the nack timeout
```

2
nomad/core_sched.go
View File

@ -1053,7 +1053,7 @@ func (c *CoreScheduler) rotateVariables(iter memdb.ResultIterator, eval *structs
// //
// Instead, we'll rate limit RPC requests and have a timeout. If we still // Instead, we'll rate limit RPC requests and have a timeout. If we still
// haven't finished the set by the timeout, emit a new eval. // haven't finished the set by the timeout, emit a new eval.
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Minute) ctx, cancel := context.WithTimeout(context.Background(), c.srv.GetConfig().EvalNackTimeout/2)
defer cancel() defer cancel()
limiter := rate.NewLimiter(rate.Limit(100), 100) limiter := rate.NewLimiter(rate.Limit(100), 100)