72f7a4061c
Misc comment improvements
2016-06-20 15:29:38 -07:00
9bf6e61655
Initialize a non-empty number of Consul Datacenters. No functional change.
2016-06-20 15:26:59 -07:00
b78c95d37e
Prefer rand.Int31n() over rand.Int31().
2016-06-20 15:26:27 -07:00
e81bf2a505
Fix deadlock in Consul RTT.
...
- consul/rtt.go:388: s.getDatacentersByDistance(). Acquires RLock()
- consul/rtt.go:341: sortDatacentersByDistance() RLock still held.
- consul/rtt.go:282: getDatacenterDistance() RLock still held.
- consul/rtt.go:268: getNodesForDatacenter(). Attempts to reacquire RLock(), hangs indefinitely.
2016-06-20 14:59:54 -07:00
89fe991ab7
consul: test raw PreferLocal functionality
2016-06-20 14:53:13 -07:00
1fef85cd2e
consul: support PreferLocal in PQ's
2016-06-20 14:24:40 -07:00
7482a9207d
Chase casting types.CheckID to a string into the state_store.
...
It turns out the indexer can only use strings as arguments when
creating a query. Cast `types.CheckID` to a `string` before calling
into `memdb`.
Ideally the indexer would be smart enough to do this at compile-time,
but I need to look into how to do this without reflection and the
runtime package. For the time being statically cast `types.CheckID`
to a `string` at the call sites.
2016-06-07 16:59:02 -04:00
ff45f8c8ff
Revert "Move `structs.CheckID` to a new top-level package, `types`."
...
This reverts commit 2bbd52e3b44ff1b60939a8400264d534662d6d51.
2016-06-07 16:59:02 -04:00
a4554b945c
Move `structs.CheckID` to a new top-level package, `types`.
...
Per discussion w/ @slackpad, move this type to its own top-level package
2016-06-07 16:59:02 -04:00
cd68cd3868
Move `structs.CheckID` to a new top-level package, `types`.
...
Per discussion w/ @slackpad, move this type to its own top-level package
2016-06-07 16:59:02 -04:00
0857e93d0b
Float a type balloon. Some strings are square pegs in round holes.
...
This experiment was brought about because of variable naming
confusion where name and checkIDs were interchanged. Gave CheckID
an Qualified Type Name and chased downstream changes.
2016-06-07 16:59:02 -04:00
ffcba3df58
Merge pull request #2028 from hashicorp/f-atomic-kv
...
Adds support for atomic transactions spanning multiple KV entries.
2016-05-15 13:46:05 -07:00
3756fb23a6
Remove unused peers variable from setupRaft().
2016-05-15 06:40:46 -07:00
a11f32a1da
Adds a get-tree verb to KV transaction operations.
2016-05-13 16:57:39 -07:00
0f94a7a326
Switches GETs to a filtering model for ACLs.
2016-05-13 15:58:55 -07:00
5fd99b13ef
Removes null results for deletes, and preps for more than one result from an operation.
2016-05-13 01:47:55 -07:00
2649a6336e
Adds a read-only optimized path for transactions.
2016-05-13 00:34:05 -07:00
0c34ed078c
Adds a comment for the txnKVS() function.
2016-05-12 16:11:26 -07:00
88b1c7d054
Makes get fail a transaction if the key doesn't exist.
2016-05-11 14:18:31 -07:00
3d35acaa90
De-nests the KV output structure (removes DirEnt member).
2016-05-11 13:48:03 -07:00
04a13ec3d7
Switches to "KV" instead of "KV" for the KV operations.
2016-05-11 10:58:27 -07:00
dc662f7e35
Refactors TxnRequest/TxnResponse into a form that will allow non-KV ops.
...
This isn't needed/used yet, but it's a good hook to get in there so we
can add more atomic operations in the future. The Go API hides this detail
so that feels like a KV-specific API. The implications on the REST API are
pretty minimal.
2016-05-11 01:39:10 -07:00
d980cbcd9d
Moves txn code into a new endpoint, not specific to KV.
2016-05-10 21:58:02 -07:00
907d8bab34
Fixes some go vet findings in a unit test.
2016-05-10 20:01:52 -07:00
94e2766423
Remove stray type definition
...
Noticed while working on Nomad Client's server selection code.
2016-05-10 18:56:28 -07:00
4eb89481df
Adds internal endpoint read ACL support and full unit tests.
2016-05-10 11:23:47 -07:00
6a96e052c4
Adds an empty get test case.
2016-05-09 22:18:26 -07:00
471160d8f0
Performs basic plumbing of KVS transactions through all the layers.
2016-05-09 22:15:49 -07:00
dca00c96f7
Adds state store support for atomic KVS ops.
2016-05-05 15:46:59 -07:00
a1a59bee73
Splits existing KVS operations into *Txn helpers for later reuse.
2016-05-04 14:20:11 -07:00
9185450fd5
Moves KVS-related state store code out into its own set of files.
2016-05-02 16:21:04 -07:00
c16b1ca178
Add the list of Raft peers to Consul's Stats
...
```
% consul info
[snip]
raft:
[snip]
raft_peers = 127.0.0.1:8300
[snip]
```
Poached from: Nomad Project
2016-04-28 15:08:48 -07:00
79153c3014
Merge pull request #1884 from mtchavez/1541-data-dir-perms
...
command: Data directory permission error message
2016-04-12 22:06:49 -07:00
6e177a9b44
Merge pull request #1895 from shoenig/fixtypo
...
doc: fix trivial typo s/NewFSMPath/NewFSM/
2016-04-12 21:53:24 -07:00
3f340716fd
Adds a clone method to HealthCheck and uses that in local.go.
2016-04-11 00:05:39 -07:00
c9602c561c
Add description to rpc test client pool member failure message
2016-04-01 19:17:38 -07:00
7f67c123b7
doc: fix trivial typo s/NewFSMPath/NewFSM/
2016-03-29 20:52:17 -05:00
5ae7835988
Rename server_details package to agent
2016-03-29 17:39:19 -07:00
7f06c71650
Add a quick package doc for the servers package
2016-03-29 16:22:53 -07:00
897282f77d
Rename serverConfig to serverList
...
serverList is a vastly more accurate name. Chase accordingly. No functional change other than types and APIs.
2016-03-29 16:17:16 -07:00
4984b6111d
Gratuitous rename 1/2
...
Reduce cognative load and perform an overdue rename. No functional change.
Rename the `server_manager` package to `servers`. Rename the `ServerManager` package to `Manager`. In `client`, rename `serverMgr` to `servers`.
2016-03-29 16:12:00 -07:00
4734e0113f
Remove two unused constants
2016-03-29 11:11:41 -07:00
cb9833b134
Remove useless comment residual from decomposing functions
2016-03-29 10:53:00 -07:00
1f049a3c38
EDYSLEXICMOMENT
2016-03-29 10:50:10 -07:00
177f64134e
Refactor out recocileServerList anon function
...
Add testing to reconcileServerList and test various server sizes.
Test that a percentage of nodes fail their Ping (50% in testing atm)
2016-03-29 02:45:38 -07:00
6609ee5d51
Teach fauxConnPool to fail a pct of the time
...
50% failure rate seems legit as a starting point w/ 100 servers.
2016-03-28 14:53:29 -07:00
7d26f7bfa7
Call NotifyFailedServers to rotate the server list
2016-03-28 14:12:41 -07:00
6a987062b9
Add log line re: server manager backing off and sleeping
...
This is useful in situations where the RPC rotate duration is greater than 1µs. WTB exponential backoff of logging so we don't spam forever.
2016-03-28 14:04:04 -07:00
689b79aef3
Remove old debugging lines of questionable future value
2016-03-28 14:02:53 -07:00
0b0a07a280
Shuffle in place
...
Don't create a copy and save the copy, not necessary any more.
2016-03-28 14:02:27 -07:00
e230b3a3b7
Nuke unnecessary comment
...
See above function comments for details
2016-03-28 13:57:36 -07:00
34a29a2107
Move FIXME comment to the right call site
2016-03-28 13:49:55 -07:00
b38d3d71c8
Rename the ConnPoolPinger interface to Pinger
2016-03-28 13:46:01 -07:00
d6b4345375
Return error from PingConsulServer
...
In order to report why a Ping failed, change the signature of PingConsulServers to include an error message.
2016-03-28 13:38:58 -07:00
6c9fb06511
Change the definition of the ServerDetails struct key
...
Use only the serf Name for now. Leaving the plumbing for now.
2016-03-28 12:53:19 -07:00
2bcff6bac4
Correct the comment to match reality
2016-03-28 12:32:30 -07:00
fc1edea1ef
Rename serverCfg to sc for consistency
2016-03-28 12:06:26 -07:00
988b05700d
Add a quick length check
...
Verify that AddServer behaved as expected
2016-03-28 11:38:12 -07:00
7181e42ba8
Switch the order of ServerDetails.String()
...
It's more natrual to have the network first. I think I flipped the order accidentally.
2016-03-28 11:37:25 -07:00
dca8fd2643
Move rebalance log statement from INFO to DEBUG
2016-03-27 01:32:04 -07:00
180edd8e7b
Chase the API bump re: refreshServerRebalanceTimer
...
If it works in prod, why shouldn't it work in the tests?
2016-03-27 00:04:52 -07:00
9b5dd7a785
Move initialization of the rebalanceTimer to New()
2016-03-27 00:03:48 -07:00
86d1bad541
Add a test for ConnPool.PingConsulServer
...
Spin up 5x servers, join and ping each server
2016-03-26 23:52:06 -07:00
f903005080
Expose ServerManager.ResetRebalanceTimer
...
Move the rebalance timer from ServerManager.Start's stack to struct ServerManager. This makes it possible to shuffle during tests without actually waiting >120s.
2016-03-26 23:41:01 -07:00
2ba281bc5a
Logging improvements
...
Comment out noisly loggers for the time being.
Improve the final logging statement to be useful and hint what the next active server for the client is going to be.
2016-03-26 22:41:08 -07:00
fab3981b1d
Standardize the log message based on the package
...
This log statement used to belong in the consul package but has since moved to the server manager package.
2016-03-26 22:29:00 -07:00
c6d9c42d9f
Reduce the error level from Fatal when unit testing
2016-03-26 22:07:09 -07:00
4747cf3cab
Start server rebalance task after init'ing Serf
...
Now that there is no longer an event loop driven directly by Serf, start the ServerManager task after Serf has been setup. When testing and adjusting timers and timeouts to unreasonably low values, it's possible to tickle a race condition where Serf's NumNodes() would fail because Serf had not been initialized.
2016-03-26 22:04:41 -07:00
2ddf82d9d8
Catch up to a few renames
2016-03-26 19:32:11 -07:00
640ced7c11
Use empty string for addr in ServerDetails.String()
2016-03-26 19:30:04 -07:00
e0f29c17cd
Guard against a nil ServerDetails.Addr
...
It's not clear how or why this would ever be nil, but some of the unit tests produce a nil addr. Be defensive.
2016-03-26 19:29:31 -07:00
2d9982eb27
Proactively ping server before rotation
...
Before shuffling the server list, proactively ping the next server in the list to establish the connection and verify the remote endpoint is healthy.
2016-03-26 19:28:13 -07:00
b3a8e2f115
Factor out the shuffle server
2016-03-26 19:19:04 -07:00
766ddae165
Revise comments re: cycleServer
...
Improve the comments to discuss what happens presently. Add a note to consider possibly calling to TestConsulServer proactively.
2016-03-26 18:53:13 -07:00
ac1d42e9d8
Comment why the interface is needed: cyclic import
2016-03-26 18:38:35 -07:00
a9b3dba05f
Add a struct key type for server_details
2016-03-26 17:58:12 -07:00
496f05b561
Add additional checks
2016-03-25 14:40:46 -07:00
c18158aac3
Delete the right tag
...
"role" != "consul"
2016-03-25 14:31:48 -07:00
b44554f882
Don't pass in sm, server manager is already in scope
...
Go closures are implicitly capturing lambdas.
2016-03-25 14:10:09 -07:00
2713899a5b
Trim residual complexity from server join notifications
...
Now that serf node join events are decoupled from rebalancing activities completely, remove the complixity of draining the channel and ensuring only one go routine was rebalancing the server list.
Now that we're no longer initializing a notification channel, we can remove the config load/save from `Start()`
2016-03-25 14:06:35 -07:00
b3298ce4c3
Only log in FindServers
...
In FindServer this is a useful warning hinting why its call failed. RPC returns error and leaves it to the higher level caller to do whatever it wants. As an operator, I'd have the detail necessary to know why the RPC call(s) failed.
2016-03-25 13:58:50 -07:00
f024272ab2
Initialize the rebalancce to clientRPCMinReuseDuration
...
In an earlier version there was a channel to notify when a new server was added, however this has long since been removed. Just default to the sane value of 2min before the first rebalance calc takes place.
Pointed out by: slackpad
2016-03-25 13:46:18 -07:00
89311a5859
Use range vs for
...
Returning a new array vs mutating an array in place so we can use range now.
2016-03-25 13:08:08 -07:00
643997623e
Comment updates
2016-03-25 13:06:59 -07:00
072f34cf02
Only rotate server list with more than one server
...
Fantastic observation by slackpad. This was left over from when there was a boolean for health in the server struct (vs current strategy where we use server position in the list and rely on serf to cleanup the stale members).
Pointed out by: slackpad
2016-03-25 12:54:36 -07:00
aadd274a13
Relocate saveServerConfig next to getServerConfig
...
Requested by: slackpad
2016-03-25 12:41:22 -07:00
cf271e7f65
Clarify that ConsulClusterInfo is an interface over serf
...
An interface was used to break a cyclic import dependency.
2016-03-25 12:38:40 -07:00
973d924ab4
Reword comment after moving code into new packages
2016-03-25 12:34:46 -07:00
78ec9f241d
Change initialReblaanaceTimeout to a time.Duration
...
Pointed out by: @slackpad
2016-03-25 12:34:12 -07:00
328728c88a
Negative check: test an invalid condition
2016-03-25 12:22:33 -07:00
22e546ff32
Test to make sure bootstrap is missing
2016-03-25 12:20:12 -07:00
5f035da4f1
Be more Go idiomatic w/ variable names: s/valid/ok/g
...
Cargo culting is bad, m'kay?
Pointy Hat: sean-
2016-03-25 12:14:24 -07:00
e041c3905d
Fix stale comment
...
Pointed out by: @slackpad
2016-03-25 12:00:40 -07:00
45fc7c362e
Add a comment for Client serverMgr
2016-03-25 11:59:27 -07:00
5873b7e28e
Correct a bogus goimport rewrite for tests
2016-03-23 22:35:49 -07:00
dcc64d91c6
Test ServerManager.refreshServerRebalanceTimer
...
Change the signature so it returns a value so that this can be tested externally with mock data. See the sample table in TestServerManagerInternal_refreshServerRebalanceTimer() for the rate at which it will back off. This function is mostly used to not cripple large clusters in the event of a partition.
2016-03-23 22:10:50 -07:00
8e3b3d766d
Add a handful more unit tests to the public interface
2016-03-23 22:10:50 -07:00
d5f72e8c07
Rename GetNumServers to NumServers()
...
Matches the style of the rest of the repo
2016-03-23 22:10:50 -07:00
9de9cf90f1
Rename NewServerManger to just New
...
Follow go style recommendations now that this has been refactored out of the consul package and doesn't need the qualifier in the name.
2016-03-23 22:10:50 -07:00
7faea986a0
Rename FindHealthyServer() to FindServer()
...
There is no guarantee the server coming back is healthy. It's apt to be healthy by virtue of its place in the server list, but it's not guaranteed.
2016-03-23 22:10:50 -07:00
18885e3214
cycleServer is a pure function, save the result
2016-03-23 22:10:50 -07:00
4ec9ed4de2
Missed unit test cruft
2016-03-23 22:10:50 -07:00
b906e40811
Update comments to reflect reality
2016-03-23 22:10:50 -07:00
1a09a5b2cf
Remove additional cruft from ServerManager's channels
...
No longer needed code.
2016-03-23 22:10:50 -07:00
c980d492c6
Emulate a TryLock using atomic.CompareAndSwap
...
Prevent possible queueing behind serverConfigLock in the event that a server fails on a busy host.
2016-03-23 22:10:50 -07:00
102dcafe76
Make use of interfaces
...
Use an interface instead of serf.Serf as arg to NewServerManager. Bonus points for improved testability.
Pointed out by: @slackpad
2016-03-23 22:10:50 -07:00
231768faea
Simplify error handling
...
Rely on Serf for liveliness. In the event of a failure, simply cycle the server to the end of the list. If the server is unhealthy, Serf will reap the dead server.
Additional simplifications:
*) Only rebalance servers based on timers, not when a new server is readded to the cluster.
*) Back out the failure count in server_details.ServerDetails
2016-03-23 22:10:50 -07:00
0c519aa90d
Unbreak client tests by reverting to original test
...
Debugging code crept into the actual test and hung out for much longer than it should have.
2016-03-23 22:10:50 -07:00
26e51376d9
Introduce asynchronous management of consul server lists
...
Instead of blocking the RPC call path and performing a potentially expensive calculation (including a call to `c.LANMembers()`), introduce a channel to request a rebalance. Some events don't force a reshuffle, instead the extend the duration of the current rebalance window because the environment thrashed enough to redistribute a client's load.
2016-03-23 22:10:50 -07:00
6ed37d1d8d
Comment nits
2016-03-23 22:10:50 -07:00
c8ab3ae4cb
Use saveServerConfig vs atomic.Value.Store(config)
2016-03-23 22:10:50 -07:00
12377e80e6
Commit a handful of refactoring && copy/paste-o fixes
2016-03-23 22:10:50 -07:00
c1c17f158b
Mutate copies of serverCfg.servers, not original
...
Removing any ambiguity re: ownership of the mutated server lists is a win for maintenance and debugging.
2016-03-23 22:10:50 -07:00
753766cc5d
rebalanceTimer may be nil during initialization
...
When first starting the server manager, it's possible that the rebalanceTimer in serverConfig will be nil, test accordingly.
2016-03-23 22:10:50 -07:00
d0e2792d5c
Properly retain a pointer to the rebalanceTimer
2016-03-23 22:10:50 -07:00
62785de865
Cosmetic and various other wordsmithing cleanups
2016-03-23 22:10:50 -07:00
31de4290cf
Document the various functions and their locking
2016-03-23 22:10:50 -07:00
ffcd939feb
Use config convenience method to get config
...
'cause ELETTHECOMPILERSDOTHEWORK. I don't need that cluttering up the subconscious with more complexity.
2016-03-23 22:10:50 -07:00
ed7fee7a3c
Move consul.serverConfig out of the consul package
...
Relocated to its own package, server_manager. This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary. More work is needed to be done here
2016-03-23 22:10:50 -07:00
ab80393198
Rename serverConfigMtx to serverConfigLock
...
Pointed out by: @slackpad
2016-03-23 22:10:50 -07:00
1866d94285
Refactor out the management of Consul servers
...
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.
This commit brings in a background task that proactively manages the server list and:
*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed
This is a WIP, more work in testing needs to be completed.
2016-03-23 22:10:50 -07:00
73497f7915
Move consul.serverConfig out of the consul package
...
Relocated to its own package, server_manager. This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary. More work is needed to be done here
2016-03-23 22:10:50 -07:00
2a52d3eb80
Rename serverConfigMtx to serverConfigLock
...
Pointed out by: @slackpad
2016-03-23 22:10:32 -07:00
49425c5371
Refactor out the management of Consul servers
...
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.
This commit brings in a background task that proactively manages the server list and:
*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed
This is a WIP, more work in testing needs to be completed.
2016-03-23 22:09:46 -07:00
ebdccf0f35
Move consul.serverConfig out of the consul package
...
Relocated to its own package, server_manager. This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary. More work is needed to be done here
2016-03-23 22:05:29 -07:00
b7213d9daa
Rename serverConfigMtx to serverConfigLock
...
Pointed out by: @slackpad
2016-03-23 22:05:05 -07:00
e29b8de0a6
Refactor out the management of Consul servers
...
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.
This commit brings in a background task that proactively manages the server list and:
*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed
This is a WIP, more work in testing needs to be completed.
2016-03-23 22:03:20 -07:00
3730eaf6df
Commit miss re: consuls variable rename
2016-03-23 16:24:29 -07:00
b33648ca5c
Move consul.serverConfig out of the consul package
...
Relocated to its own package, server_manager. This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary. More work is needed to be done here
2016-03-23 16:16:22 -07:00
f3a69c939d
Refactor consul.serverParts into server_details.ServerDetails
...
This may be short-lived, but it also seems like this is going to lead us down a path where ServerDetails is going to evolve into a more powerful package that will encapsulate more behavior behind a coherent API.
2016-03-23 16:15:47 -07:00
b3192ca410
Rename serverConfigMtx to serverConfigLock
...
Pointed out by: @slackpad
2016-03-23 16:15:47 -07:00
82458fa9e8
Handle the case where there are no healthy servers
...
Pointed out by: @slackpad
2016-03-23 16:15:47 -07:00
09d4c6439c
Refactor out the management of Consul servers
...
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.
This commit brings in a background task that proactively manages the server list and:
*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed
This is a WIP, more work in testing needs to be completed.
2016-03-23 16:15:47 -07:00
6bda2c007c
Add a flag to denote that a server is disabled
...
A server is not normally disabled, but in the event of an RPC error, we want to mark a server as down to allow for fast failover to a different server. This value must be an int in order to support atomic operations.
Additionally, this is the preliminary work required to bring up a server in a disabled state. RPC health checks in the future could mark the server as alive, thereby creating an organic "slow start" feature for Consul.
2016-03-23 16:14:59 -07:00
7de85906c1
Rename `lastServer` to `preferredServer`
...
Expanding the domain of lastServer beyond RPC() changes the meaning of this variable. Rename accordingly to match the intent coming in a subsequent commit: a background thread will be in charge of rotating preferredServer.
2016-03-23 16:14:59 -07:00
2949980a64
Warn if serf events have queued up past 80% of the limit
...
It is theoretically possible that the number of queued serf events can back up. If this happens, emit a warning message if there are more than 200 events in queue.
Most notably, this can happen if `c.consulServerLock` is held for an "extended period of time". The probability of anyone ever seeing this log message is hopefully low to nonexistent, but if it happens, the warning message indicating a large number of serf events fired while a lock was held is likely to be helpful (vs serf mysteriously blocking when attempting to add an event to a channel).
2016-03-23 16:14:11 -07:00
2a0c12460d
Commit miss re: consuls variable rename
2016-03-23 16:13:49 -07:00
3ac1bcc799
Remove lastRPCTime
...
This mechanism isn't going to provide much value in the future. Preemptively reduce the complexity of future work.
2016-03-23 16:13:49 -07:00
72b7856045
Rename c.consuls to c.consulServers
...
Prep for breaking out maintenance of consuls into a new goroutine.
2016-03-23 16:10:27 -07:00
d1ef4ec7e2
Use `rand.Int31n()` to get power of two optimization
...
In cases where i+1 is a power of two, skip one modulo operation.
2016-03-23 16:00:39 -07:00
92e947dcc3
Gets rid of flaky sort check.
...
If we get a coordinate then this test will fail, so we only check the
first item in the list, which is deterministic.
2016-03-21 17:30:05 -07:00
265a8d4053
Increases timeouts for coordinate tests.
...
We take the interval and add the random stagger to it, so 2X is cutting it
too close and the unit tests are often flaky.
2016-03-21 16:44:35 -07:00
13b8ce0adc
Merge pull request #1851 from hashicorp/f-ipv6-bind
...
Allow [::] as a bind address (binds to first public IPv6 address)
2016-03-19 16:16:19 -07:00
18e12aa886
Adds more specific checks for ipv6 addresses.
2016-03-19 16:14:45 -07:00
e4ca18089f
Removes leader from members and changes name since it's an address.
2016-03-18 17:07:11 -07:00
11b73bb1a5
#735 add information about leader to consul members
2016-03-18 17:05:40 -07:00
508bc796a8
Allow [::] as a bind address (binds to first public IPv6 address)
2016-03-18 23:59:44 +01:00
7215d9bdef
Obfuscate token for lookupACL error
2016-03-15 17:16:25 -04:00
a9d640c024
Hardens the match interoplator against negative arguments.
2016-03-07 13:32:32 -08:00
63c826c2c0
Adds a comment about the embedded struct.
2016-03-07 10:45:39 -08:00
275c84a0cc
Renames "debug" endpoint and structures to "explain".
2016-03-07 10:45:39 -08:00
8493640b09
Adds a prepared query debug endpoint.
2016-03-07 10:45:39 -08:00
918b1ace47
Applies prefix ACL to a catch-all template as a special case.
2016-03-07 10:45:39 -08:00
3c512fc089
Adds a test for the custom prepared query template indexer.
2016-03-07 10:45:39 -08:00
39d3094d50
Adds core query template tests to the state store.
2016-03-07 10:45:39 -08:00
06087633f0
Adds in basic query template lookups and vendors newly-updated memdb as well as improved iradix tree.
2016-03-07 10:45:39 -08:00
142e69befe
Adds tests for the low-level template functions.
2016-03-07 10:45:39 -08:00
b578fbbfc4
Adds tests for the string visitor.
2016-03-07 10:45:39 -08:00
2a9a5f823e
Factors rendering down into the resolve function.
2016-03-07 10:45:39 -08:00
8e25451232
Splits walk functions out from the rest of the template code.
2016-03-07 10:45:39 -08:00
fa60d575bf
Integrates templates into state store and endpoint (sans tests).
2016-03-07 10:45:39 -08:00
62405110dc
Wraps the prepared query to also store the compiled template.
2016-03-07 10:45:39 -08:00
98281be7df
Adds basic query template compiler and renderer.
2016-03-07 10:45:39 -08:00
25613895e3
one line schema change to not allow missing for sessions Table node index, Fixes #1774
2016-03-02 21:19:53 -08:00
f0150ff5ce
Adds missing token redact in the GET path.
2016-02-26 15:59:00 -08:00
48f2089d7f
Merge pull request #1757 from hashicorp/f-revert-1667
...
Reverts server connection rebalancing changes from #1667
2016-02-24 18:07:13 -08:00
c75256ac8b
Adds a check for users re-submitting the redacted token.
2016-02-24 17:35:26 -08:00
2f7eac8b86
Renames "prepared_query" ACL policy to "query".
2016-02-24 17:02:06 -08:00
3b91618d7d
Changes to more idiomatic "ok" pattern for prefix getter.
2016-02-24 16:26:43 -08:00
1c7ee582f9
Renames a unit test.
2016-02-24 16:17:20 -08:00
d660311fbb
Revert "Merge pull request #1667 from hashicorp/b-redistribute-clients"
...
This reverts commit 8f30dea4209491ebbe4ef9ab94dd8052d17bdbe9, reversing
changes made to eb27a02956e7e052c0bec6f96a0c0f7f6675f6a6.
2016-02-24 15:38:03 -08:00
54f0b7bbb6
Completes switch of prepared_query ACLs to govern query names.
2016-02-24 01:26:16 -08:00
633c231d67
Creates new "prepared-query" ACL type and new token capture behavior.
...
Prior to this change, prepared queries had the following behavior for
ACLs, which will need to change to support templates:
1. A management token, or a token with read access to the service being
queried needed to be provided in order to create a prepared query.
2. The token used to create the prepared query was stored with the query
in the state store and used to execute the query.
3. A management token, or the token used to create the query needed to be
supplied to perform and CRUD operations on an existing prepared query.
This was pretty subtle and complicated behavior, and won't work for
templates since the service name is computed at execution time. To solve
this, we introduce a new "prepared-query" ACL type, where the prefix
applies to the query name for static prepared query types and to the
prefix for template prepared query types.
With this change, the new behavior is:
1. A management token, or a token with "prepared-query" write access to
the query name or (soon) the given template prefix is required to do
any CRUD operations on a prepared query, or to list prepared queries
(the list is filtered by this ACL).
2. You will no longer need a management token to list prepared queries,
but you will only be able to see prepared queries that you have access
to (you get an empty list instead of permission denied).
3. When listing or getting a query, because it was easy to capture
management tokens given the past behavior, this will always blank out
the "Token" field (replacing the contents as <hidden>) for all tokens
unless a management token is supplied. Going forward, we should
discourage people from binding tokens for execution unless strictly
necessary.
4. No token will be captured by default when a prepared query is created.
If the user wishes to supply an execution token then can pass it in via
the "Token" field in the prepared query definition. Otherwise, this
field will default to empty.
5. At execution time, we will use the captured token if it exists with the
prepared query definition, otherwise we will use the token that's passed
in with the request, just like we do for other RPCs (or you can use the
agent's configured token for DNS).
6. Prepared queries with no name (accessible only by ID) will not require
ACLs to create or modify (execution time will depend on the service ACL
configuration). Our argument here is that these are designed to be
ephemeral and the IDs are as good as an ACL. Management tokens will be
able to list all of these.
These changes enable templates, but also enable delegation of authority to
manage the prepared query namespace.
2016-02-23 17:12:43 -08:00
f398e1880e
Adds a test for node registration and tagged addresses.
2016-02-07 13:15:22 -08:00
f163522f0c
Moves tagged wan address to be managed by anti-entropy, not serf.
2016-02-07 13:12:42 -08:00
3f50d2ae7e
Adds an FSM persist and restore test for tagged addresses.
2016-02-07 11:36:39 -08:00
79bd1fd4bb
Sets up config for more address tags down the road, renames struct members.
2016-02-07 10:37:34 -08:00
71cf39b5f8
Use a map for additional node addresses
2016-02-06 23:01:45 -08:00
e166d2a4c7
Use idiomatic name for wan_addr serf tag
2016-02-06 23:01:45 -08:00
1a828e3927
Store WanAddress during Service/Check sync
2016-02-06 23:01:45 -08:00
fc61143b46
Store WanAddress during node registration
2016-02-06 23:01:45 -08:00
369e501dd5
Store WanAddress on Node
2016-02-06 23:01:45 -08:00
fc82b351b8
Use the server's address in debug logging, not the c.lastServer, which may be nil
2016-02-02 15:51:28 -08:00
58225e0ee3
Remove unnecessary check, test was moved further up in scope
2016-02-02 11:13:58 -08:00
6452b498e1
Use panic instead of returning a sentinel UUID values in unit tests
2016-02-01 23:15:19 -08:00
ef8bbca48f
Continually rebalance client connections
...
Introduce a low-level background connection expiration mechanism wherein connections will be recycled periodically based on the size and health of the cluster.
For the vast majority of consul users, this will mean an average connection age of 150s. For 10K node clusters it will take ~3min for clusters to rebalance their connections. In the pathological case for a 100K cluster where 99K clients are in the minority talking to 1x server it will take ~26min to rebalance all connections.
It's possibe for clients recovering from a parititon to become fixated on a single server until the server or agent is restarted. This is of particular interest to long-running environments with stable agents, where `allow_stale` is true, and partitions occur periodically.
2016-01-30 17:13:50 -08:00
8a37e76cb0
Use rand.Int31n() vs unconditionally using modulus
2016-01-30 15:47:58 -08:00
727bb42b1d
Merge branch 'f-consul-lib' of ssh://github.com/hashicorp/consul into b-redistribute-clients
2016-01-30 15:40:54 -08:00
b216d4c11f
Rename clientRPCCache to clientRPCConnMaxIdle, change value
...
Increase the max idle time for agents talking to servers from 30s to 127s in order to allow for the reuse of connections that are being initiated by cron.
127s was chosen as the first prime above 120s (arbitrarily chose to use a prime) with the intent of reusing connections who are used by once-a-minute cron(8) jobs *and* who use a 60s jitter window (e.g. in vixie cron job execution can drift by up to 59s per job, or 119s for a once-a-minute cron job).
2016-01-30 15:27:46 -08:00
e83a5b7a70
Reuse the results from gettimeofday(2)...
...
Inside of a single RPC call, reuse time.Now().
2016-01-30 14:39:17 -08:00
c470553b6b
Factor out duplicate functions into a lib package
...
Consolidate code duplication and tests into a single lib package. Most of these functions were from various **/util.go functions that couldn't be imported due to cyclic imports. The consul/lib package is intended to be a terminal node in an import DAG and a place to stash various consul-only helper functions. Pulled in hashicorp/go-uuid instead of consolidating UUID access.
2016-01-29 16:57:45 -08:00
0010b788ed
Prevents watches from being orphaned when KVS blocking queries loop.
2016-01-20 07:18:47 -08:00
e0ae4878f6
Merge pull request #948 from hashicorp/iface-down-fix
...
Don't try to bind on address from inactive interface
2016-01-14 17:00:54 -08:00
e3ac96679e
consul: address comments
2016-01-05 09:45:36 -08:00
689698eefa
consul: disable serf snapshots in dev mode
2015-12-26 20:39:49 -05:00
ce4cf8a542
consul: dev mode works
2015-12-26 20:19:36 -05:00
533c79bd2b
Cleans up some small `go vet` findings.
2015-12-18 22:14:48 -08:00
92531812b6
Adds source address logging on RPC errors.
2015-12-16 14:03:03 -08:00
38bdcc9907
Adds a check to make sure query names can't be registered twice.
2015-12-02 09:04:51 -08:00
a033d7aed2
consul: shrink yamux recv buffer on idle streams
2015-11-27 17:20:57 -08:00
0e3f4af4ac
Makes all the query ops the correct type.
2015-11-17 09:27:10 -08:00
86bc3db7b3
Returns a zero index for a lookup error case.
2015-11-17 09:25:20 -08:00
bc60491890
Removes a useless empty import and fixes some stale comments.
2015-11-17 08:29:20 -08:00
f4943c1613
Makes UUID regex case-insensitive.
2015-11-16 22:57:47 -08:00
a1e02996e5
Moves conversion of nil slices up to HTTP layer for prepared queries.
2015-11-15 17:06:00 -08:00
c032dee945
Adds a paranoia set of the nodes slice to nil.
2015-11-15 17:06:00 -08:00
a0211db3b9
Adds a test to ensure we don't return a nil slice.
2015-11-15 17:06:00 -08:00
375312f07a
Gets rid of some unused constants.
2015-11-15 17:06:00 -08:00
e6ccf5b9ed
Returns a 404 from a get or execute of a nonexistent query.
2015-11-15 17:06:00 -08:00
c9ef552385
Plumbs the service name back and uses agent-specific TTL settings as a fallback.
2015-11-15 17:06:00 -08:00
cad6938474
Adds unit tests for prepared queries and DNS, using existing tests for equivalence.
2015-11-15 17:06:00 -08:00
7762a828d3
Adds query metadata to prepared query execute response.
2015-11-15 17:06:00 -08:00
0c02365bf2
Makes an empty prepared query list an empty slice, not a nil one.
2015-11-15 17:06:00 -08:00
3029906254
Adds a unit test for the new RTT getDatacentersByDistance fn.
2015-11-15 17:06:00 -08:00
697e2f4e71
Completes FSM support for prepared queries.
2015-11-15 17:06:00 -08:00
124410b66c
Adds an HTTP endpoint for prepared queries.
2015-11-15 17:06:00 -08:00
b209c2afe2
Adds an RPC endpoint injection method for testing.
2015-11-15 17:06:00 -08:00
4051db4e88
Changes Lookup to Get since we don't need it (only Execute does).
2015-11-15 17:06:00 -08:00
095806584d
Always increments the failovers counter, even for error-ed DCs.
2015-11-15 17:06:00 -08:00
07361d3e96
Adds test for remote datacenter selection and query logic.
2015-11-15 17:06:00 -08:00
6a1c571a38
Adds a test for the server wrapper.
2015-11-15 17:06:00 -08:00
9428f025f9
Adds tag filter tests.
2015-11-15 17:06:00 -08:00
da5cf9cdf2
Adds execute tests for prepared queries.
2015-11-15 17:06:00 -08:00
7c8404df4c
Adds status information about failovers to query results.
2015-11-15 17:06:00 -08:00
dc517aa288
Removes unused ACL filter.
2015-11-15 17:06:00 -08:00
00ea015770
Adds execute leader forward test for prepared queries.
2015-11-15 17:06:00 -08:00
264e86e533
Adds a leader forwarding case for prepared queries.
2015-11-15 17:06:00 -08:00
bb0f136412
Adds tests for query lookup and list endpoints.
2015-11-15 17:06:00 -08:00
495d00c0b8
Adds query parsing unit tests.
2015-11-15 17:06:00 -08:00
72f6c8a261
Adds ACL cases for apply.
2015-11-15 17:06:00 -08:00
76a55eed53
Completes non-ACL version of apply test.
2015-11-15 17:06:00 -08:00
bc0fba43d8
Adds lookup and list endpoints and basic end-to-end apply test.
2015-11-15 17:06:00 -08:00
3b3c7c02e4
Checks for valid UUIDs before calling in to index function.
2015-11-15 17:06:00 -08:00
ec5249357e
Clarifies comment about name vs. ID.
2015-11-15 17:06:00 -08:00
61e6cbc560
Skips unknown DCs during queries and chugs along in the face of errors.
2015-11-15 17:06:00 -08:00
0141438e6c
Moves sort to a query-time decision and adds back the limit.
2015-11-15 17:06:00 -08:00
25fac70924
Adds an explicit ACL check that will fail vs. trying other DCs.
2015-11-15 17:06:00 -08:00
347bb847c2
Changes "not" prefix from "~" to "!".
2015-11-15 17:06:00 -08:00
09034a84bd
Adds prefix "prepared" to everything prepared query-related.
2015-11-15 17:06:00 -08:00
2183565d83
Adds basic structure for prepared queries (needs tests).
2015-11-15 17:06:00 -08:00
6e7faa6239
Factors code for pulling the sorted list of DCs into a common place.
2015-11-15 17:06:00 -08:00
6a70cb9885
Adds a better shuffle test (similar to DNS).
2015-11-15 17:06:00 -08:00
781f9611e8
Changes structs and state store for prepared queries.
2015-11-15 17:06:00 -08:00
55f49d6e73
Merge pull request #1413 from hashicorp/b-coord-raft-errors
...
Adds a check for in-band error returns in the coordinate RaftApply.
2015-11-15 17:05:19 -08:00
26cf0a2974
Extends the session TTL max to 24 hours, and adds a warning to the docs.
2015-11-15 16:51:00 -08:00
04dffd7c46
Adds a check for in-band error returns in the coordinate RaftApply.
2015-11-15 16:50:46 -08:00
5b0697fc67
Changes batch update failure to a WARN since it's nbd.
2015-10-29 09:26:24 -07:00
eb4bfa3411
Prevents agents from considering Raft information when doing sync checks.
2015-10-28 14:32:00 -07:00
7c44a9b6c9
Fixes a bad error message.
2015-10-28 12:40:47 -07:00
5449096bc2
Adds a check for the minimum version as well.
2015-10-27 15:56:36 -07:00
1c678effde
Sets the ignore flag on coordinate update Raft log entries.
2015-10-27 14:44:34 -07:00
c1305a08ea
Makes the version upshift code look at the correct version field.
2015-10-27 14:44:34 -07:00
b91e2d3a97
Completes rebase of network coordinates to new memdb.
2015-10-23 15:23:01 -07:00
132e1d813b
Fixes configs now that Serf always caches coordinates.
2015-10-23 15:23:01 -07:00
aea2194ce3
Makes the default protocol 2 and lets 3 interoperate with 2.
2015-10-23 15:23:01 -07:00
7dd88f7de6
Zeroes out the height when testing exact distances.
2015-10-23 15:23:01 -07:00
7d5e29f6c6
Cleans up after latest rebase.
2015-10-23 15:23:01 -07:00
a74bdcba49
Moves sorting up into coordinate endpoint HTTP handlers.
2015-10-23 15:23:01 -07:00
a2bcef00a0
Adds endpoints for raw network coordinates.
2015-10-23 15:23:01 -07:00
e900fef95a
Fixes bad name for DC forwarding of Coordinate.Get.
2015-10-23 15:23:01 -07:00
ee1cf1e13b
Adds distance sorting to health endpoint. Cleans up unit tests.
2015-10-23 15:23:01 -07:00
019f656f39
Switches to the median over all DC nodes with known coordinates.
2015-10-23 15:23:01 -07:00
5609b2e889
Adds explicit check for empty node in source parameter.
2015-10-23 15:23:01 -07:00
5d75ce7852
Moves disable checks down into the sort routine.
2015-10-23 15:23:01 -07:00
59b710894d
Adds tests for HTTP interface. Removes a stray mark.
2015-10-23 15:23:01 -07:00
033e8e6625
Adds sort of DCs in catalog queries based on RTT. Cleans up.
...
* Makes the catalog endpoint respect disabling coordinates for all
RTT-sorting query types.
2015-10-23 15:23:01 -07:00
b63909cf67
Adds coordinate sorting support to catalog queries for nodes and service nodes.
2015-10-23 15:23:01 -07:00
9ba9a708f6
Scales coordinate sends to hit a fixed aggregate rate across the cluster.
2015-10-23 15:23:01 -07:00
d8b8a3719f
Simplifies the batching function and adds some comments.
2015-10-23 15:23:01 -07:00
a93a1a68b6
Adds snapshot save and restore of coordinates.
2015-10-23 15:23:01 -07:00
f71c79c53f
Does some small cleanups based on PR feedback.
...
* Holds coordinate updates in map and gets rid of the update channel.
* Cleans up config variables a bit.
2015-10-23 15:23:01 -07:00
1222772452
Hardens Consul from bad coordinates from other nodes.
2015-10-23 15:23:01 -07:00
e02ae7b6b4
Takes the node name out of the coordinate get call.
2015-10-23 15:23:01 -07:00
acb0dce829
Moves batching down into the state store and changes it to fail-fast.
...
* A batch of updates is done all in a single transaction.
* We no longer need to get an update to kick things, there's a periodic flush.
* If incoming updates overwhelm the configured flush rate they will be dumped with an error.
2015-10-23 15:23:01 -07:00
b6c31bdf2f
Flips the sense of the coordinate enable option.
2015-10-23 15:23:01 -07:00
9c069c5031
Removes one more WAN leftover.
2015-10-23 15:23:01 -07:00
edb9a119e2
Does a clean up pass on the Consul side.
2015-10-23 15:23:01 -07:00
ac4185b888
Merges config changes after rebase.
2015-10-23 15:23:01 -07:00
b805215237
Address comments
2015-10-23 15:23:01 -07:00
7d70d8b1d7
Add test for the GetWAN endpoint
2015-10-23 15:23:01 -07:00
e03a9d4c38
Add an endpoint for getting WAN coordinates
2015-10-23 15:23:01 -07:00
213f5a15e6
Fix tests
2015-10-23 15:23:01 -07:00
eb599a1745
Address comments
2015-10-23 15:23:01 -07:00
66d5a129bf
Add state store tests
2015-10-23 15:23:01 -07:00
88550714ca
Add more tests
2015-10-23 15:23:01 -07:00
139c9240ea
Address comments
2015-10-23 15:23:01 -07:00
23c08aeeb4
Use IndexedCoordinate instead
2015-10-23 15:23:01 -07:00
979c0c6c9e
Improve a test
2015-10-23 15:23:01 -07:00
b2cff43bb5
Complete logic for sending coordinates
2015-10-23 15:23:01 -07:00
019da1dae4
Fix tests
2015-10-23 15:23:01 -07:00
66b210afcb
Some fixes
2015-10-23 15:23:01 -07:00
def0a2f5ab
Change GET API a little bit
2015-10-23 15:23:01 -07:00
b5bbe2bcfa
Adding tests and stuff
2015-10-23 15:23:01 -07:00
e54c8f2ea0
Start adding stuff
2015-10-23 15:23:01 -07:00
aae298a179
Cleans up a little whitespace with go fmt.
2015-10-20 18:41:05 -07:00
54535d45d7
Gets rid of error prefixing in leader.go.
2015-10-20 13:37:11 -07:00
eb93d1d9ad
Puts all restore operations into a single transaction and optimizes watches.
2015-10-19 23:06:59 -07:00
96c5c9de2b
Gets rid of unique constraint on sessions index in session_checks.
2015-10-19 17:41:50 -07:00
3e93055cef
Switches sessions over to UUIDFieldIndex.
2015-10-19 17:09:02 -07:00
d905804514
Adds a special case for fill KVS listings to avoid a tombstone scan.
2015-10-19 16:22:27 -07:00
Sean Chittenden