Commit Graph

4 Commits

Author SHA1 Message Date
swayne275 597c3d1296
Vault-1983: Use fairsharing to distribute workers between queues (#11789)
* prelim fairshare prototype, untested and prototype status

* add tests for new fairshare infra - this likely fails tests for being racy

* probably fix races for code and test

* one more lock to fix for races

* fairsharing queue work distribution, tests, fixes, etc

* comment, shorten wait time

* typos and comments

* fix inverted worker count logic

* Update helper/fairshare/jobmanager.go

typo

* Update helper/fairshare/jobmanager.go

clarify comment

* move back to round robin between queues

* improvements from self review

* add job manager stress test
2021-06-25 14:06:49 -06:00
Brian Kassouf a112161f60
expiration: Add a few metrics to measure revoke queue lengths (#10955)
* expiration: Add a few metrics to measure revoke queue lengths

* Update the metric names

* Add appropriate cluster labels

* Add metrics to docs

* Update jobmanager.go
2021-02-26 16:00:39 -08:00
swayne275 fe9e6e3d89
Make fairsharing stop function async (#11012)
* stop fairsharing layer async

* update function doc
2021-02-25 14:33:02 -07:00
swayne275 3994579603
VAULT-1401 and 1402 - preliminary fair sharing (#1701) (#10917)
* basic pool and start testing

* refactor a bit for testing

* workFunc, start/stop safety, testing

* cleanup function for worker quit, more tests

* redo public/private members

* improve tests, export types, switch uuid package

* fix loop capture bug, cleanup

* cleanup tests

* update worker pool file name, other improvements

* add job manager prototype

* remove remnants

* add functions to wait for job manager and worker pool to stop, other fixes

* test job manager functionality, fix bugs

* encapsulate how jobs are distributed to workers

* make worker job channel read only

* add job interface, more testing, fixes

* set name for dispatcher

* fix test races

* dispatcher and job manager constructors don't return errors

* logger now dependency injected

* make some members private, test fcn to get worker pool size

* make GetNumWorkers public

* Update helper/fairshare/jobmanager_test.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* make workerpool private

* remove custom worker names

* concurrency improvements

* remove worker pool cleanup function

* remove cleanup func from job manager, remove non blocking stop from fairshare

* stop fairshare when started in tests

* stop leaking job manager goroutine

* prototype channel for waking up to assign work

* fix typo/bug and add tests

* improve job manager wake up, fix test typo

* put channel drain back

* better start/pause test for job manager

* go mod vendor

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-02-12 14:51:52 -07:00