Commit graph

1000 commits

Author SHA1 Message Date
Sean Chittenden 973d924ab4 Reword comment after moving code into new packages 2016-03-25 12:34:46 -07:00
Sean Chittenden 78ec9f241d Change initialReblaanaceTimeout to a time.Duration
Pointed out by: @slackpad
2016-03-25 12:34:12 -07:00
Sean Chittenden 328728c88a Negative check: test an invalid condition 2016-03-25 12:22:33 -07:00
Sean Chittenden 22e546ff32 Test to make sure bootstrap is missing 2016-03-25 12:20:12 -07:00
Sean Chittenden 5f035da4f1 Be more Go idiomatic w/ variable names: s/valid/ok/g
Cargo culting is bad, m'kay?

Pointy Hat: sean-
2016-03-25 12:14:24 -07:00
Sean Chittenden e041c3905d Fix stale comment
Pointed out by: @slackpad
2016-03-25 12:00:40 -07:00
Sean Chittenden 45fc7c362e Add a comment for Client serverMgr 2016-03-25 11:59:27 -07:00
Sean Chittenden 5873b7e28e Correct a bogus goimport rewrite for tests 2016-03-23 22:35:49 -07:00
Sean Chittenden dcc64d91c6 Test ServerManager.refreshServerRebalanceTimer
Change the signature so it returns a value so that this can be tested externally with mock data.  See the sample table in TestServerManagerInternal_refreshServerRebalanceTimer() for the rate at which it will back off.  This function is mostly used to not cripple large clusters in the event of a partition.
2016-03-23 22:10:50 -07:00
Sean Chittenden 8e3b3d766d Add a handful more unit tests to the public interface 2016-03-23 22:10:50 -07:00
Sean Chittenden d5f72e8c07 Rename GetNumServers to NumServers()
Matches the style of the rest of the repo
2016-03-23 22:10:50 -07:00
Sean Chittenden 9de9cf90f1 Rename NewServerManger to just New
Follow go style recommendations now that this has been refactored out of the consul package and doesn't need the qualifier in the name.
2016-03-23 22:10:50 -07:00
Sean Chittenden 7faea986a0 Rename FindHealthyServer() to FindServer()
There is no guarantee the server coming back is healthy.  It's apt to be healthy by virtue of its place in the server list, but it's not guaranteed.
2016-03-23 22:10:50 -07:00
Sean Chittenden 18885e3214 cycleServer is a pure function, save the result 2016-03-23 22:10:50 -07:00
Sean Chittenden 4ec9ed4de2 Missed unit test cruft 2016-03-23 22:10:50 -07:00
Sean Chittenden b906e40811 Update comments to reflect reality 2016-03-23 22:10:50 -07:00
Sean Chittenden 1a09a5b2cf Remove additional cruft from ServerManager's channels
No longer needed code.
2016-03-23 22:10:50 -07:00
Sean Chittenden c980d492c6 Emulate a TryLock using atomic.CompareAndSwap
Prevent possible queueing behind serverConfigLock in the event that a server fails on a busy host.
2016-03-23 22:10:50 -07:00
Sean Chittenden 102dcafe76 Make use of interfaces
Use an interface instead of serf.Serf as arg to NewServerManager.  Bonus points for improved testability.

Pointed out by: @slackpad
2016-03-23 22:10:50 -07:00
Sean Chittenden 231768faea Simplify error handling
Rely on Serf for liveliness.  In the event of a failure, simply cycle the server to the end of the list.  If the server is unhealthy, Serf will reap the dead server.

Additional simplifications:

*) Only rebalance servers based on timers, not when a new server is readded to the cluster.
*) Back out the failure count in server_details.ServerDetails
2016-03-23 22:10:50 -07:00
Sean Chittenden 0c519aa90d Unbreak client tests by reverting to original test
Debugging code crept into the actual test and hung out for much longer than it should have.
2016-03-23 22:10:50 -07:00
Sean Chittenden 26e51376d9 Introduce asynchronous management of consul server lists
Instead of blocking the RPC call path and performing a potentially expensive calculation (including a call to `c.LANMembers()`), introduce a channel to request a rebalance.  Some events don't force a reshuffle, instead the extend the duration of the current rebalance window because the environment thrashed enough to redistribute a client's load.
2016-03-23 22:10:50 -07:00
Sean Chittenden 6ed37d1d8d Comment nits 2016-03-23 22:10:50 -07:00
Sean Chittenden c8ab3ae4cb Use saveServerConfig vs atomic.Value.Store(config) 2016-03-23 22:10:50 -07:00
Sean Chittenden 12377e80e6 Commit a handful of refactoring && copy/paste-o fixes 2016-03-23 22:10:50 -07:00
Sean Chittenden c1c17f158b Mutate copies of serverCfg.servers, not original
Removing any ambiguity re: ownership of the mutated server lists is a win for maintenance and debugging.
2016-03-23 22:10:50 -07:00
Sean Chittenden 753766cc5d rebalanceTimer may be nil during initialization
When first starting the server manager, it's possible that the rebalanceTimer in serverConfig will be nil, test accordingly.
2016-03-23 22:10:50 -07:00
Sean Chittenden d0e2792d5c Properly retain a pointer to the rebalanceTimer 2016-03-23 22:10:50 -07:00
Sean Chittenden 62785de865 Cosmetic and various other wordsmithing cleanups 2016-03-23 22:10:50 -07:00
Sean Chittenden 31de4290cf Document the various functions and their locking 2016-03-23 22:10:50 -07:00
Sean Chittenden ffcd939feb Use config convenience method to get config
'cause ELETTHECOMPILERSDOTHEWORK.  I don't need that cluttering up the subconscious with more complexity.
2016-03-23 22:10:50 -07:00
Sean Chittenden ed7fee7a3c Move consul.serverConfig out of the consul package
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
2016-03-23 22:10:50 -07:00
Sean Chittenden ab80393198 Rename serverConfigMtx to serverConfigLock
Pointed out by: @slackpad
2016-03-23 22:10:50 -07:00
Sean Chittenden 1866d94285 Refactor out the management of Consul servers
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
2016-03-23 22:10:50 -07:00
Sean Chittenden 73497f7915 Move consul.serverConfig out of the consul package
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
2016-03-23 22:10:50 -07:00
Sean Chittenden 2a52d3eb80 Rename serverConfigMtx to serverConfigLock
Pointed out by: @slackpad
2016-03-23 22:10:32 -07:00
Sean Chittenden 49425c5371 Refactor out the management of Consul servers
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
2016-03-23 22:09:46 -07:00
Sean Chittenden ebdccf0f35 Move consul.serverConfig out of the consul package
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
2016-03-23 22:05:29 -07:00
Sean Chittenden b7213d9daa Rename serverConfigMtx to serverConfigLock
Pointed out by: @slackpad
2016-03-23 22:05:05 -07:00
Sean Chittenden e29b8de0a6 Refactor out the management of Consul servers
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
2016-03-23 22:03:20 -07:00
Sean Chittenden 3730eaf6df Commit miss re: consuls variable rename 2016-03-23 16:24:29 -07:00
Sean Chittenden b33648ca5c Move consul.serverConfig out of the consul package
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
2016-03-23 16:16:22 -07:00
Sean Chittenden f3a69c939d Refactor consul.serverParts into server_details.ServerDetails
This may be short-lived, but it also seems like this is going to lead us down a path where ServerDetails is going to evolve into a more powerful package that will encapsulate more behavior behind a coherent API.
2016-03-23 16:15:47 -07:00
Sean Chittenden b3192ca410 Rename serverConfigMtx to serverConfigLock
Pointed out by: @slackpad
2016-03-23 16:15:47 -07:00
Sean Chittenden 82458fa9e8 Handle the case where there are no healthy servers
Pointed out by: @slackpad
2016-03-23 16:15:47 -07:00
Sean Chittenden 09d4c6439c Refactor out the management of Consul servers
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
2016-03-23 16:15:47 -07:00
Sean Chittenden 6bda2c007c Add a flag to denote that a server is disabled
A server is not normally disabled, but in the event of an RPC error, we want to mark a server as down to allow for fast failover to a different server.  This value must be an int in order to support atomic operations.

Additionally, this is the preliminary work required to bring up a server in a disabled state.  RPC health checks in the future could mark the server as alive, thereby creating an organic "slow start" feature for Consul.
2016-03-23 16:14:59 -07:00
Sean Chittenden 7de85906c1 Rename lastServer to preferredServer
Expanding the domain of lastServer beyond RPC() changes the meaning of this variable.  Rename accordingly to match the intent coming in a subsequent commit: a background thread will be in charge of rotating preferredServer.
2016-03-23 16:14:59 -07:00
Sean Chittenden 2949980a64 Warn if serf events have queued up past 80% of the limit
It is theoretically possible that the number of queued serf events can back up.  If this happens, emit a warning message if there are more than 200 events in queue.

Most notably, this can happen if `c.consulServerLock` is held for an "extended period of time".  The probability of anyone ever seeing this log message is hopefully low to nonexistent, but if it happens, the warning message indicating a large number of serf events fired while a lock was held is likely to be helpful (vs serf mysteriously blocking when attempting to add an event to a channel).
2016-03-23 16:14:11 -07:00
Sean Chittenden 2a0c12460d Commit miss re: consuls variable rename 2016-03-23 16:13:49 -07:00
Sean Chittenden 3ac1bcc799 Remove lastRPCTime
This mechanism isn't going to provide much value in the future.  Preemptively reduce the complexity of future work.
2016-03-23 16:13:49 -07:00
Sean Chittenden 72b7856045 Rename c.consuls to c.consulServers
Prep for breaking out maintenance of consuls into a new goroutine.
2016-03-23 16:10:27 -07:00
Sean Chittenden d1ef4ec7e2 Use rand.Int31n() to get power of two optimization
In cases where i+1 is a power of two, skip one modulo operation.
2016-03-23 16:00:39 -07:00
James Phillips 92e947dcc3 Gets rid of flaky sort check.
If we get a coordinate then this test will fail, so we only check the
first item in the list, which is deterministic.
2016-03-21 17:30:05 -07:00
James Phillips 265a8d4053 Increases timeouts for coordinate tests.
We take the interval and add the random stagger to it, so 2X is cutting it
too close and the unit tests are often flaky.
2016-03-21 16:44:35 -07:00
James Phillips 13b8ce0adc Merge pull request #1851 from hashicorp/f-ipv6-bind
Allow [::] as a bind address (binds to first public IPv6 address)
2016-03-19 16:16:19 -07:00
James Phillips 18e12aa886 Adds more specific checks for ipv6 addresses. 2016-03-19 16:14:45 -07:00
James Phillips e4ca18089f Removes leader from members and changes name since it's an address. 2016-03-18 17:07:11 -07:00
Sergey Romanov 11b73bb1a5 #735 add information about leader to consul members 2016-03-18 17:05:40 -07:00
Wim 508bc796a8 Allow [::] as a bind address (binds to first public IPv6 address) 2016-03-18 23:59:44 +01:00
James Phillips a9d640c024 Hardens the match interoplator against negative arguments. 2016-03-07 13:32:32 -08:00
James Phillips 63c826c2c0 Adds a comment about the embedded struct. 2016-03-07 10:45:39 -08:00
James Phillips 275c84a0cc Renames "debug" endpoint and structures to "explain". 2016-03-07 10:45:39 -08:00
James Phillips 8493640b09 Adds a prepared query debug endpoint. 2016-03-07 10:45:39 -08:00
James Phillips 918b1ace47 Applies prefix ACL to a catch-all template as a special case. 2016-03-07 10:45:39 -08:00
James Phillips 3c512fc089 Adds a test for the custom prepared query template indexer. 2016-03-07 10:45:39 -08:00
James Phillips 39d3094d50 Adds core query template tests to the state store. 2016-03-07 10:45:39 -08:00
James Phillips 06087633f0 Adds in basic query template lookups and vendors newly-updated memdb as well as improved iradix tree. 2016-03-07 10:45:39 -08:00
James Phillips 142e69befe Adds tests for the low-level template functions. 2016-03-07 10:45:39 -08:00
James Phillips b578fbbfc4 Adds tests for the string visitor. 2016-03-07 10:45:39 -08:00
James Phillips 2a9a5f823e Factors rendering down into the resolve function. 2016-03-07 10:45:39 -08:00
James Phillips 8e25451232 Splits walk functions out from the rest of the template code. 2016-03-07 10:45:39 -08:00
James Phillips fa60d575bf Integrates templates into state store and endpoint (sans tests). 2016-03-07 10:45:39 -08:00
James Phillips 62405110dc Wraps the prepared query to also store the compiled template. 2016-03-07 10:45:39 -08:00
James Phillips 98281be7df Adds basic query template compiler and renderer. 2016-03-07 10:45:39 -08:00
Mike Cowgill 25613895e3 one line schema change to not allow missing for sessions Table node index, Fixes #1774 2016-03-02 21:19:53 -08:00
James Phillips f0150ff5ce Adds missing token redact in the GET path. 2016-02-26 15:59:00 -08:00
James Phillips 48f2089d7f Merge pull request #1757 from hashicorp/f-revert-1667
Reverts server connection rebalancing changes from #1667
2016-02-24 18:07:13 -08:00
James Phillips c75256ac8b Adds a check for users re-submitting the redacted token. 2016-02-24 17:35:26 -08:00
James Phillips 2f7eac8b86 Renames "prepared_query" ACL policy to "query". 2016-02-24 17:02:06 -08:00
James Phillips 3b91618d7d Changes to more idiomatic "ok" pattern for prefix getter. 2016-02-24 16:26:43 -08:00
James Phillips 1c7ee582f9 Renames a unit test. 2016-02-24 16:17:20 -08:00
James Phillips d660311fbb Revert "Merge pull request #1667 from hashicorp/b-redistribute-clients"
This reverts commit 8f30dea4209491ebbe4ef9ab94dd8052d17bdbe9, reversing
changes made to eb27a02956e7e052c0bec6f96a0c0f7f6675f6a6.
2016-02-24 15:38:03 -08:00
James Phillips 54f0b7bbb6 Completes switch of prepared_query ACLs to govern query names. 2016-02-24 01:26:16 -08:00
James Phillips 633c231d67 Creates new "prepared-query" ACL type and new token capture behavior.
Prior to this change, prepared queries had the following behavior for
ACLs, which will need to change to support templates:

1. A management token, or a token with read access to the service being
   queried needed to be provided in order to create a prepared query.

2. The token used to create the prepared query was stored with the query
   in the state store and used to execute the query.

3. A management token, or the token used to create the query needed to be
   supplied to perform and CRUD operations on an existing prepared query.

This was pretty subtle and complicated behavior, and won't work for
templates since the service name is computed at execution time. To solve
this, we introduce a new "prepared-query" ACL type, where the prefix
applies to the query name for static prepared query types and to the
prefix for template prepared query types.

With this change, the new behavior is:

1. A management token, or a token with "prepared-query" write access to
   the query name or (soon) the given template prefix is required to do
   any CRUD operations on a prepared query, or to list prepared queries
   (the list is filtered by this ACL).

2. You will no longer need a management token to list prepared queries,
   but you will only be able to see prepared queries that you have access
   to (you get an empty list instead of permission denied).

3. When listing or getting a query, because it was easy to capture
   management tokens given the past behavior, this will always blank out
   the "Token" field (replacing the contents as <hidden>) for all tokens
   unless a management token is supplied. Going forward, we should
   discourage people from binding tokens for execution unless strictly
   necessary.

4. No token will be captured by default when a prepared query is created.
   If the user wishes to supply an execution token then can pass it in via
   the "Token" field in the prepared query definition. Otherwise, this
   field will default to empty.

5. At execution time, we will use the captured token if it exists with the
   prepared query definition, otherwise we will use the token that's passed
   in with the request, just like we do for other RPCs (or you can use the
   agent's configured token for DNS).

6. Prepared queries with no name (accessible only by ID) will not require
   ACLs to create or modify (execution time will depend on the service ACL
   configuration). Our argument here is that these are designed to be
   ephemeral and the IDs are as good as an ACL. Management tokens will be
   able to list all of these.

These changes enable templates, but also enable delegation of authority to
manage the prepared query namespace.
2016-02-23 17:12:43 -08:00
James Phillips f398e1880e Adds a test for node registration and tagged addresses. 2016-02-07 13:15:22 -08:00
James Phillips f163522f0c Moves tagged wan address to be managed by anti-entropy, not serf. 2016-02-07 13:12:42 -08:00
James Phillips 3f50d2ae7e Adds an FSM persist and restore test for tagged addresses. 2016-02-07 11:36:39 -08:00
James Phillips 79bd1fd4bb Sets up config for more address tags down the road, renames struct members. 2016-02-07 10:37:34 -08:00
Evan Gilman 71cf39b5f8 Use a map for additional node addresses 2016-02-06 23:01:45 -08:00
Evan Gilman e166d2a4c7 Use idiomatic name for wan_addr serf tag 2016-02-06 23:01:45 -08:00
James Phillips 1a828e3927 Store WanAddress during Service/Check sync 2016-02-06 23:01:45 -08:00
Evan Gilman fc61143b46 Store WanAddress during node registration 2016-02-06 23:01:45 -08:00
Evan Gilman 369e501dd5 Store WanAddress on Node 2016-02-06 23:01:45 -08:00
Sean Chittenden fc82b351b8 Use the server's address in debug logging, not the c.lastServer, which may be nil 2016-02-02 15:51:28 -08:00
Sean Chittenden 58225e0ee3 Remove unnecessary check, test was moved further up in scope 2016-02-02 11:13:58 -08:00
Sean Chittenden 6452b498e1 Use panic instead of returning a sentinel UUID values in unit tests 2016-02-01 23:15:19 -08:00
Sean Chittenden ef8bbca48f Continually rebalance client connections
Introduce a low-level background connection expiration mechanism wherein connections will be recycled periodically based on the size and health of the cluster.

For the vast majority of consul users, this will mean an average connection age of 150s.  For 10K node clusters it will take ~3min for clusters to rebalance their connections.  In the pathological case for a 100K cluster where 99K clients are in the minority talking to 1x server it will take ~26min to rebalance all connections.

It's possibe for clients recovering from a parititon to become fixated on a single server until the server or agent is restarted.  This is of particular interest to long-running environments with stable agents, where `allow_stale` is true, and partitions occur periodically.
2016-01-30 17:13:50 -08:00
Sean Chittenden 8a37e76cb0 Use rand.Int31n() vs unconditionally using modulus 2016-01-30 15:47:58 -08:00
Sean Chittenden 727bb42b1d Merge branch 'f-consul-lib' of ssh://github.com/hashicorp/consul into b-redistribute-clients 2016-01-30 15:40:54 -08:00