Commit Graph

24 Commits

Author SHA1 Message Date
Mahmood Ali acbfeb5815 Simplify Bootstrap logic in tests
This change updates tests to honor `BootstrapExpect` exclusively when
forming test clusters and removes test only knobs, e.g.
`config.DevDisableBootstrap`.

Background:

Test cluster creation is fragile.  Test servers don't follow the
BootstapExpected route like production clusters.  Instead they start as
single node clusters and then get rejoin and may risk causing brain
split or other test flakiness.

The test framework expose few knobs to control those (e.g.
`config.DevDisableBootstrap` and `config.Bootstrap`) that control
whether a server should bootstrap the cluster.  These flags are
confusing and it's unclear when to use: their usage in multi-node
cluster isn't properly documented.  Furthermore, they have some bad
side-effects as they don't control Raft library: If
`config.DevDisableBootstrap` is true, the test server may not
immediately attempt to bootstrap a cluster, but after an election
timeout (~50ms), Raft may force a leadership election and win it (with
only one vote) and cause a split brain.

The knobs are also confusing as Bootstrap is an overloaded term.  In
BootstrapExpect, we refer to bootstrapping the cluster only after N
servers are connected.  But in tests and the knobs above, it refers to
whether the server is a single node cluster and shouldn't wait for any
other server.

Changes:

This commit makes two changes:

First, it relies on `BootstrapExpected` instead of `Bootstrap` and/or
`DevMode` flags.  This change is relatively trivial.

Introduce a `Bootstrapped` flag to track if the cluster is bootstrapped.
This allows us to keep `BootstrapExpected` immutable.  Previously, the
flag was a config value but it gets set to 0 after cluster bootstrap
completes.
2020-03-02 13:47:43 -05:00
Seth Hoenig f0c3dca49c tests: swap lib/freeport for tweaked helper/freeport
Copy the updated version of freeport (sdk/freeport), and tweak it for use
in Nomad tests. This means staying below port 10000 to avoid conflicts with
the lib/freeport that is still transitively used by the old version of
consul that we vendor. Also provide implementations to find ephemeral ports
of macOS and Windows environments.

Ports acquired through freeport are supposed to be returned to freeport,
which this change now also introduces. Many tests are modified to include
calls to a cleanup function for Server objects.

This should help quite a bit with some flakey tests, but not all of them.
Our port problems will not go away completely until we upgrade our vendor
version of consul. With Go modules, we'll probably do a 'replace' to swap
out other copies of freeport with the one now in 'nomad/helper/freeport'.
2019-12-09 08:37:32 -06:00
Mahmood Ali 9bd56587cd Fix raft tests
Wait until leadership stabalizes and all non-voters get promoted before
killing leader
2019-09-03 14:53:29 -04:00
Alex Dadgar 4bdccab550 goimports 2019-01-22 15:44:31 -08:00
Alex Dadgar 17aac1c9de node heartbeat missed event 2018-05-22 14:05:46 -07:00
Alex Dadgar 93b0e42e6c Fix merge change 2018-03-12 16:25:49 -07:00
Alex Dadgar 4844317cc2
Merge pull request #3890 from hashicorp/b-heartbeat
Heartbeat improvements and handling failures during establishing leadership
2018-03-12 14:41:59 -07:00
Alex Dadgar 354faf2820 Fix flaky test 2018-02-20 16:05:43 -08:00
Alex Dadgar 601177c250 Add escape hatches when non-leader 2018-02-20 10:22:15 -08:00
Alex Dadgar a6dfffa4fa Add testing interfaces 2018-02-15 13:59:00 -08:00
Alex Dadgar 1d6cdfbdc3 lax timing 2017-10-23 16:48:20 -07:00
Alex Dadgar 06eddf243c parallel nomad tests 2017-07-25 17:39:36 -07:00
Alex Dadgar dea460281d Merge pull request #2282 from hashicorp/f-raft-v2-stage-one
Update to Raft V2 stage one
2017-02-08 15:26:16 -08:00
Alex Dadgar 04862ca10e Tests compile 2017-02-07 21:30:57 -08:00
Alex Dadgar 419bf2c497 Tests build 2017-02-02 17:50:06 -08:00
Alex Dadgar 78cfcd2724 Bump protocol version and update numOtherPeers 2017-02-02 13:52:31 -08:00
Alex Dadgar dd11824d48 Loosen timing a bit on travis 2017-01-26 15:06:14 -08:00
Armon Dadgar 2ff133c0e6 nomad: rename region1 to global. Fixes #41 2015-09-13 18:18:40 -07:00
Armon Dadgar 8bedd3769c nomad: unifying the state store API 2015-09-06 20:56:38 -07:00
Armon Dadgar 1a5579384a nomad: cleanup API descrepencies 2015-09-06 20:47:42 -07:00
Armon Dadgar 0702f17989 Rename client endpoint to node endpoint 2015-09-06 20:31:32 -07:00
Armon Dadgar d84f959dd8 nomad: fixing tests 2015-08-30 18:10:12 -07:00
Armon Dadgar 6c8791ccf8 nomad: client does periodic heartbeat 2015-08-22 18:16:05 -07:00
Armon Dadgar 678addc990 nomad: testing heartbeat resets 2015-08-22 17:37:50 -07:00