luxolus/open-consul
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity
3748 commits 1 branch 1 tag 358 MiB
Commit graph

983 commits

Author SHA1 Message Date
James Phillips 06087633f0 Adds in basic query template lookups and vendors newly-updated memdb as well as improved iradix tree. 2016-03-07 10:45:39 -08:00
James Phillips 142e69befe Adds tests for the low-level template functions. 2016-03-07 10:45:39 -08:00
James Phillips b578fbbfc4 Adds tests for the string visitor. 2016-03-07 10:45:39 -08:00
James Phillips 2a9a5f823e Factors rendering down into the resolve function. 2016-03-07 10:45:39 -08:00
James Phillips 8e25451232 Splits walk functions out from the rest of the template code. 2016-03-07 10:45:39 -08:00
James Phillips fa60d575bf Integrates templates into state store and endpoint (sans tests). 2016-03-07 10:45:39 -08:00
James Phillips 62405110dc Wraps the prepared query to also store the compiled template. 2016-03-07 10:45:39 -08:00
James Phillips 98281be7df Adds basic query template compiler and renderer. 2016-03-07 10:45:39 -08:00
Mike Cowgill 25613895e3 one line schema change to not allow missing for sessions Table node index, Fixes #1774 2016-03-02 21:19:53 -08:00
James Phillips f0150ff5ce Adds missing token redact in the GET path. 2016-02-26 15:59:00 -08:00
James Phillips 48f2089d7f Merge pull request #1757 from hashicorp/f-revert-1667 
Reverts server connection rebalancing changes from #1667
 2016-02-24 18:07:13 -08:00
James Phillips c75256ac8b Adds a check for users re-submitting the redacted token. 2016-02-24 17:35:26 -08:00
James Phillips 2f7eac8b86 Renames "prepared_query" ACL policy to "query". 2016-02-24 17:02:06 -08:00
James Phillips 3b91618d7d Changes to more idiomatic "ok" pattern for prefix getter. 2016-02-24 16:26:43 -08:00
James Phillips 1c7ee582f9 Renames a unit test. 2016-02-24 16:17:20 -08:00
James Phillips d660311fbb Revert "Merge pull request #1667 from hashicorp/b-redistribute-clients" 
This reverts commit 8f30dea4209491ebbe4ef9ab94dd8052d17bdbe9, reversing
changes made to eb27a02956e7e052c0bec6f96a0c0f7f6675f6a6.
 2016-02-24 15:38:03 -08:00
James Phillips 54f0b7bbb6 Completes switch of prepared_query ACLs to govern query names. 2016-02-24 01:26:16 -08:00
James Phillips 633c231d67 Creates new "prepared-query" ACL type and new token capture behavior. 
Prior to this change, prepared queries had the following behavior for
ACLs, which will need to change to support templates:

1. A management token, or a token with read access to the service being
   queried needed to be provided in order to create a prepared query.

2. The token used to create the prepared query was stored with the query
   in the state store and used to execute the query.

3. A management token, or the token used to create the query needed to be
   supplied to perform and CRUD operations on an existing prepared query.

This was pretty subtle and complicated behavior, and won't work for
templates since the service name is computed at execution time. To solve
this, we introduce a new "prepared-query" ACL type, where the prefix
applies to the query name for static prepared query types and to the
prefix for template prepared query types.

With this change, the new behavior is:

1. A management token, or a token with "prepared-query" write access to
   the query name or (soon) the given template prefix is required to do
   any CRUD operations on a prepared query, or to list prepared queries
   (the list is filtered by this ACL).

2. You will no longer need a management token to list prepared queries,
   but you will only be able to see prepared queries that you have access
   to (you get an empty list instead of permission denied).

3. When listing or getting a query, because it was easy to capture
   management tokens given the past behavior, this will always blank out
   the "Token" field (replacing the contents as <hidden>) for all tokens
   unless a management token is supplied. Going forward, we should
   discourage people from binding tokens for execution unless strictly
   necessary.

4. No token will be captured by default when a prepared query is created.
   If the user wishes to supply an execution token then can pass it in via
   the "Token" field in the prepared query definition. Otherwise, this
   field will default to empty.

5. At execution time, we will use the captured token if it exists with the
   prepared query definition, otherwise we will use the token that's passed
   in with the request, just like we do for other RPCs (or you can use the
   agent's configured token for DNS).

6. Prepared queries with no name (accessible only by ID) will not require
   ACLs to create or modify (execution time will depend on the service ACL
   configuration). Our argument here is that these are designed to be
   ephemeral and the IDs are as good as an ACL. Management tokens will be
   able to list all of these.

These changes enable templates, but also enable delegation of authority to
manage the prepared query namespace.
 2016-02-23 17:12:43 -08:00
James Phillips f398e1880e Adds a test for node registration and tagged addresses. 2016-02-07 13:15:22 -08:00
James Phillips f163522f0c Moves tagged wan address to be managed by anti-entropy, not serf. 2016-02-07 13:12:42 -08:00
James Phillips 3f50d2ae7e Adds an FSM persist and restore test for tagged addresses. 2016-02-07 11:36:39 -08:00
James Phillips 79bd1fd4bb Sets up config for more address tags down the road, renames struct members. 2016-02-07 10:37:34 -08:00
Evan Gilman 71cf39b5f8 Use a map for additional node addresses 2016-02-06 23:01:45 -08:00
Evan Gilman e166d2a4c7 Use idiomatic name for wan_addr serf tag 2016-02-06 23:01:45 -08:00
James Phillips 1a828e3927 Store WanAddress during Service/Check sync 2016-02-06 23:01:45 -08:00
Evan Gilman fc61143b46 Store WanAddress during node registration 2016-02-06 23:01:45 -08:00
Evan Gilman 369e501dd5 Store WanAddress on Node 2016-02-06 23:01:45 -08:00
Sean Chittenden fc82b351b8 Use the server's address in debug logging, not the c.lastServer, which may be nil 2016-02-02 15:51:28 -08:00
Sean Chittenden 58225e0ee3 Remove unnecessary check, test was moved further up in scope 2016-02-02 11:13:58 -08:00
Sean Chittenden 6452b498e1 Use panic instead of returning a sentinel UUID values in unit tests 2016-02-01 23:15:19 -08:00
Sean Chittenden ef8bbca48f Continually rebalance client connections 
Introduce a low-level background connection expiration mechanism wherein connections will be recycled periodically based on the size and health of the cluster.

For the vast majority of consul users, this will mean an average connection age of 150s.  For 10K node clusters it will take ~3min for clusters to rebalance their connections.  In the pathological case for a 100K cluster where 99K clients are in the minority talking to 1x server it will take ~26min to rebalance all connections.

It's possibe for clients recovering from a parititon to become fixated on a single server until the server or agent is restarted.  This is of particular interest to long-running environments with stable agents, where `allow_stale` is true, and partitions occur periodically.
 2016-01-30 17:13:50 -08:00
Sean Chittenden 8a37e76cb0 Use rand.Int31n() vs unconditionally using modulus 2016-01-30 15:47:58 -08:00
Sean Chittenden 727bb42b1d Merge branch 'f-consul-lib' of ssh://github.com/hashicorp/consul into b-redistribute-clients 2016-01-30 15:40:54 -08:00
Sean Chittenden b216d4c11f Rename clientRPCCache to clientRPCConnMaxIdle, change value 
Increase the max idle time for agents talking to servers from 30s to 127s in order to allow for the reuse of connections that are being initiated by cron.

127s was chosen as the first prime above 120s (arbitrarily chose to use a prime) with the intent of reusing connections who are used by once-a-minute cron(8) jobs *and* who use a 60s jitter window (e.g. in vixie cron job execution can drift by up to 59s per job, or 119s for a once-a-minute cron job).
 2016-01-30 15:27:46 -08:00
Sean Chittenden e83a5b7a70 Reuse the results from gettimeofday(2)... 
Inside of a single RPC call, reuse time.Now().
 2016-01-30 14:39:17 -08:00
Sean Chittenden c470553b6b Factor out duplicate functions into a lib package 
Consolidate code duplication and tests into a single lib package.  Most of these functions were from various **/util.go functions that couldn't be imported due to cyclic imports.  The consul/lib package is intended to be a terminal node in an import DAG and a place to stash various consul-only helper functions.  Pulled in hashicorp/go-uuid instead of consolidating UUID access.
 2016-01-29 16:57:45 -08:00
James Phillips 0010b788ed Prevents watches from being orphaned when KVS blocking queries loop. 2016-01-20 07:18:47 -08:00
James Phillips e0ae4878f6 Merge pull request #948 from hashicorp/iface-down-fix 
Don't try to bind on address from inactive interface
 2016-01-14 17:00:54 -08:00
Ryan Uber e3ac96679e consul: address comments 2016-01-05 09:45:36 -08:00
Ryan Uber 689698eefa consul: disable serf snapshots in dev mode 2015-12-26 20:39:49 -05:00
Ryan Uber ce4cf8a542 consul: dev mode works 2015-12-26 20:19:36 -05:00
James Phillips 533c79bd2b Cleans up some small go vet findings. 2015-12-18 22:14:48 -08:00
James Phillips 92531812b6 Adds source address logging on RPC errors. 2015-12-16 14:03:03 -08:00
James Phillips 38bdcc9907 Adds a check to make sure query names can't be registered twice. 2015-12-02 09:04:51 -08:00
Armon Dadgar a033d7aed2 consul: shrink yamux recv buffer on idle streams 2015-11-27 17:20:57 -08:00
James Phillips 0e3f4af4ac Makes all the query ops the correct type. 2015-11-17 09:27:10 -08:00
James Phillips 86bc3db7b3 Returns a zero index for a lookup error case. 2015-11-17 09:25:20 -08:00
James Phillips bc60491890 Removes a useless empty import and fixes some stale comments. 2015-11-17 08:29:20 -08:00
James Phillips f4943c1613 Makes UUID regex case-insensitive. 2015-11-16 22:57:47 -08:00
James Phillips a1e02996e5 Moves conversion of nil slices up to HTTP layer for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips c032dee945 Adds a paranoia set of the nodes slice to nil. 2015-11-15 17:06:00 -08:00
James Phillips a0211db3b9 Adds a test to ensure we don't return a nil slice. 2015-11-15 17:06:00 -08:00
James Phillips 375312f07a Gets rid of some unused constants. 2015-11-15 17:06:00 -08:00
James Phillips e6ccf5b9ed Returns a 404 from a get or execute of a nonexistent query. 2015-11-15 17:06:00 -08:00
James Phillips c9ef552385 Plumbs the service name back and uses agent-specific TTL settings as a fallback. 2015-11-15 17:06:00 -08:00
James Phillips cad6938474 Adds unit tests for prepared queries and DNS, using existing tests for equivalence. 2015-11-15 17:06:00 -08:00
James Phillips 7762a828d3 Adds query metadata to prepared query execute response. 2015-11-15 17:06:00 -08:00
James Phillips 0c02365bf2 Makes an empty prepared query list an empty slice, not a nil one. 2015-11-15 17:06:00 -08:00
James Phillips 3029906254 Adds a unit test for the new RTT getDatacentersByDistance fn. 2015-11-15 17:06:00 -08:00
James Phillips 697e2f4e71 Completes FSM support for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 124410b66c Adds an HTTP endpoint for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips b209c2afe2 Adds an RPC endpoint injection method for testing. 2015-11-15 17:06:00 -08:00
James Phillips 4051db4e88 Changes Lookup to Get since we don't need it (only Execute does). 2015-11-15 17:06:00 -08:00
James Phillips 095806584d Always increments the failovers counter, even for error-ed DCs. 2015-11-15 17:06:00 -08:00
James Phillips 07361d3e96 Adds test for remote datacenter selection and query logic. 2015-11-15 17:06:00 -08:00
James Phillips 6a1c571a38 Adds a test for the server wrapper. 2015-11-15 17:06:00 -08:00
James Phillips 9428f025f9 Adds tag filter tests. 2015-11-15 17:06:00 -08:00
James Phillips da5cf9cdf2 Adds execute tests for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 7c8404df4c Adds status information about failovers to query results. 2015-11-15 17:06:00 -08:00
James Phillips dc517aa288 Removes unused ACL filter. 2015-11-15 17:06:00 -08:00
James Phillips 00ea015770 Adds execute leader forward test for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 264e86e533 Adds a leader forwarding case for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips bb0f136412 Adds tests for query lookup and list endpoints. 2015-11-15 17:06:00 -08:00
James Phillips 495d00c0b8 Adds query parsing unit tests. 2015-11-15 17:06:00 -08:00
James Phillips 72f6c8a261 Adds ACL cases for apply. 2015-11-15 17:06:00 -08:00
James Phillips 76a55eed53 Completes non-ACL version of apply test. 2015-11-15 17:06:00 -08:00
James Phillips bc0fba43d8 Adds lookup and list endpoints and basic end-to-end apply test. 2015-11-15 17:06:00 -08:00
James Phillips 3b3c7c02e4 Checks for valid UUIDs before calling in to index function. 2015-11-15 17:06:00 -08:00
James Phillips ec5249357e Clarifies comment about name vs. ID. 2015-11-15 17:06:00 -08:00
James Phillips 61e6cbc560 Skips unknown DCs during queries and chugs along in the face of errors. 2015-11-15 17:06:00 -08:00
James Phillips 0141438e6c Moves sort to a query-time decision and adds back the limit. 2015-11-15 17:06:00 -08:00
James Phillips 25fac70924 Adds an explicit ACL check that will fail vs. trying other DCs. 2015-11-15 17:06:00 -08:00
James Phillips 347bb847c2 Changes "not" prefix from "~" to "!". 2015-11-15 17:06:00 -08:00
James Phillips 09034a84bd Adds prefix "prepared" to everything prepared query-related. 2015-11-15 17:06:00 -08:00
James Phillips 2183565d83 Adds basic structure for prepared queries (needs tests). 2015-11-15 17:06:00 -08:00
James Phillips 6e7faa6239 Factors code for pulling the sorted list of DCs into a common place. 2015-11-15 17:06:00 -08:00
James Phillips 6a70cb9885 Adds a better shuffle test (similar to DNS). 2015-11-15 17:06:00 -08:00
James Phillips 781f9611e8 Changes structs and state store for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 55f49d6e73 Merge pull request #1413 from hashicorp/b-coord-raft-errors 
Adds a check for in-band error returns in the coordinate RaftApply.
 2015-11-15 17:05:19 -08:00
James Phillips 26cf0a2974 Extends the session TTL max to 24 hours, and adds a warning to the docs. 2015-11-15 16:51:00 -08:00
James Phillips 04dffd7c46 Adds a check for in-band error returns in the coordinate RaftApply. 2015-11-15 16:50:46 -08:00
James Phillips 5b0697fc67 Changes batch update failure to a WARN since it's nbd. 2015-10-29 09:26:24 -07:00
James Phillips eb4bfa3411 Prevents agents from considering Raft information when doing sync checks. 2015-10-28 14:32:00 -07:00
James Phillips 7c44a9b6c9 Fixes a bad error message. 2015-10-28 12:40:47 -07:00
James Phillips 5449096bc2 Adds a check for the minimum version as well. 2015-10-27 15:56:36 -07:00
James Phillips 1c678effde Sets the ignore flag on coordinate update Raft log entries. 2015-10-27 14:44:34 -07:00
James Phillips c1305a08ea Makes the version upshift code look at the correct version field. 2015-10-27 14:44:34 -07:00
James Phillips b91e2d3a97 Completes rebase of network coordinates to new memdb. 2015-10-23 15:23:01 -07:00
James Phillips 132e1d813b Fixes configs now that Serf always caches coordinates. 2015-10-23 15:23:01 -07:00
James Phillips aea2194ce3 Makes the default protocol 2 and lets 3 interoperate with 2. 2015-10-23 15:23:01 -07:00
James Phillips 7dd88f7de6 Zeroes out the height when testing exact distances. 2015-10-23 15:23:01 -07:00
James Phillips 7d5e29f6c6 Cleans up after latest rebase. 2015-10-23 15:23:01 -07:00
James Phillips a74bdcba49 Moves sorting up into coordinate endpoint HTTP handlers. 2015-10-23 15:23:01 -07:00
James Phillips a2bcef00a0 Adds endpoints for raw network coordinates. 2015-10-23 15:23:01 -07:00
James Phillips e900fef95a Fixes bad name for DC forwarding of Coordinate.Get. 2015-10-23 15:23:01 -07:00
James Phillips ee1cf1e13b Adds distance sorting to health endpoint. Cleans up unit tests. 2015-10-23 15:23:01 -07:00
James Phillips 019f656f39 Switches to the median over all DC nodes with known coordinates. 2015-10-23 15:23:01 -07:00
James Phillips 5609b2e889 Adds explicit check for empty node in source parameter. 2015-10-23 15:23:01 -07:00
James Phillips 5d75ce7852 Moves disable checks down into the sort routine. 2015-10-23 15:23:01 -07:00
James Phillips 59b710894d Adds tests for HTTP interface. Removes a stray mark. 2015-10-23 15:23:01 -07:00
James Phillips 033e8e6625 Adds sort of DCs in catalog queries based on RTT. Cleans up. 
* Makes the catalog endpoint respect disabling coordinates for all
  RTT-sorting query types.
 2015-10-23 15:23:01 -07:00
James Phillips b63909cf67 Adds coordinate sorting support to catalog queries for nodes and service nodes. 2015-10-23 15:23:01 -07:00
James Phillips 9ba9a708f6 Scales coordinate sends to hit a fixed aggregate rate across the cluster. 2015-10-23 15:23:01 -07:00
James Phillips d8b8a3719f Simplifies the batching function and adds some comments. 2015-10-23 15:23:01 -07:00
James Phillips a93a1a68b6 Adds snapshot save and restore of coordinates. 2015-10-23 15:23:01 -07:00
James Phillips f71c79c53f Does some small cleanups based on PR feedback. 
* Holds coordinate updates in map and gets rid of the update channel.
* Cleans up config variables a bit.
 2015-10-23 15:23:01 -07:00
James Phillips 1222772452 Hardens Consul from bad coordinates from other nodes. 2015-10-23 15:23:01 -07:00
James Phillips e02ae7b6b4 Takes the node name out of the coordinate get call. 2015-10-23 15:23:01 -07:00
James Phillips acb0dce829 Moves batching down into the state store and changes it to fail-fast. 
* A batch of updates is done all in a single transaction.
* We no longer need to get an update to kick things, there's a periodic flush.
* If incoming updates overwhelm the configured flush rate they will be dumped with an error.
 2015-10-23 15:23:01 -07:00
James Phillips b6c31bdf2f Flips the sense of the coordinate enable option. 2015-10-23 15:23:01 -07:00
James Phillips 9c069c5031 Removes one more WAN leftover. 2015-10-23 15:23:01 -07:00
James Phillips edb9a119e2 Does a clean up pass on the Consul side. 2015-10-23 15:23:01 -07:00
James Phillips ac4185b888 Merges config changes after rebase. 2015-10-23 15:23:01 -07:00
Derek Chiang b805215237 Address comments 2015-10-23 15:23:01 -07:00
Derek Chiang 7d70d8b1d7 Add test for the GetWAN endpoint 2015-10-23 15:23:01 -07:00
Derek Chiang e03a9d4c38 Add an endpoint for getting WAN coordinates 2015-10-23 15:23:01 -07:00
Derek Chiang 213f5a15e6 Fix tests 2015-10-23 15:23:01 -07:00
Derek Chiang eb599a1745 Address comments 2015-10-23 15:23:01 -07:00
Derek Chiang 66d5a129bf Add state store tests 2015-10-23 15:23:01 -07:00
Derek Chiang 88550714ca Add more tests 2015-10-23 15:23:01 -07:00
Derek Chiang 139c9240ea Address comments 2015-10-23 15:23:01 -07:00
Derek Chiang 23c08aeeb4 Use IndexedCoordinate instead 2015-10-23 15:23:01 -07:00
Derek Chiang 979c0c6c9e Improve a test 2015-10-23 15:23:01 -07:00
Derek Chiang b2cff43bb5 Complete logic for sending coordinates 2015-10-23 15:23:01 -07:00
Derek Chiang 019da1dae4 Fix tests 2015-10-23 15:23:01 -07:00
Derek Chiang 66b210afcb Some fixes 2015-10-23 15:23:01 -07:00
Derek Chiang def0a2f5ab Change GET API a little bit 2015-10-23 15:23:01 -07:00
Derek Chiang b5bbe2bcfa Adding tests and stuff 2015-10-23 15:23:01 -07:00
Derek Chiang e54c8f2ea0 Start adding stuff 2015-10-23 15:23:01 -07:00
James Phillips aae298a179 Cleans up a little whitespace with go fmt. 2015-10-20 18:41:05 -07:00
James Phillips 54535d45d7 Gets rid of error prefixing in leader.go. 2015-10-20 13:37:11 -07:00
James Phillips eb93d1d9ad Puts all restore operations into a single transaction and optimizes watches. 2015-10-19 23:06:59 -07:00
James Phillips 96c5c9de2b Gets rid of unique constraint on sessions index in session_checks. 2015-10-19 17:41:50 -07:00
James Phillips 3e93055cef Switches sessions over to UUIDFieldIndex. 2015-10-19 17:09:02 -07:00
James Phillips d905804514 Adds a special case for fill KVS listings to avoid a tombstone scan. 2015-10-19 16:22:27 -07:00
James Phillips 22a46f7bf5 Makes the iterator naming more consistent. 2015-10-19 15:51:11 -07:00
James Phillips 9a2fdff4c4 Converts sessions and ACLs over to iterators. 2015-10-19 14:56:22 -07:00
James Phillips d459d94b3f Converts KVS snapshot over to iterator. 2015-10-19 14:07:57 -07:00
James Phillips 498c4a989a Converts nodes, services, checks to iterators duing dumps; fixes tag drift bug. 
Realized that the conversions ServiceNode <-> NodeService were incomplete in a
few places so centralized those and added some tests.
 2015-10-19 13:55:35 -07:00
James Phillips 8ee52e9850 Gets rid of non-idomatic "state_store" alias in FSM. 2015-10-16 14:07:48 -07:00