Commit Graph

1238 Commits

Author SHA1 Message Date
Sean Chittenden 1c5d6eff76 Merge pull request #2702 from hashicorp/f-dns-nodeid
DNS lookup by Consul node ID
2017-02-01 16:23:18 -08:00
Sean Chittenden f8b64ec5f8
Treat a uuid prefix lookup error as a soft error, as if a node name
lookup returned nil.

Add a TODO to note where a future point of logging should occur once a
logger is present and a few additional comments to explain the program
flow.
2017-02-01 16:09:25 -08:00
Sean Chittenden 6844ebb43b
Treat a uuid prefix lookup error as a soft error, as if a node name lookup returned nil.
Add a TODO to note where a future point of logging should occur once a
logger is present.
2017-02-01 15:51:25 -08:00
Sean Chittenden 1d9c5a3efb
Run a test of `NodeServices()` with a NodeID as an argument. 2017-02-01 15:41:10 -08:00
Sean Chittenden b241cfc7fd
Whoops. Return an empty set in the event that there are multiple matches. 2017-02-01 15:18:00 -08:00
Sean Chittenden 324215c842
Rename `nodeName` to `nodeNameOrID`. 2017-02-01 14:59:24 -08:00
Sean Chittenden 19c2cd106a
Toggle `AllowMissing` to false to accommodate old clients without Node IDs. 2017-02-01 14:58:34 -08:00
Sean Chittenden c9eea45b1c
Enable looking up consul nodes by their node ID.
Assuming the following output from a consul agent:

```
==> Consul agent running!
           Version: 'v0.7.3-43-gc5e140c-dev (c5e140c+CHANGES)'
           Node ID: '40e4a748-2192-161a-0510-9bf59fe950b5'
         Node name: 'myhost'
```

it is now possible to lookup nodes by their Node Name or Node ID, or a
prefix match of the Node ID, with the following caveats re: the prefix
match:

1) first eight digits of the Node ID are a required minimum (eight was
   chosen as an arbitrary number)
2) the length of the Node ID must be an even number or no result will be
   returned.

```
% dig @127.0.0.1 -p 8600 myhost.node.dc1.consul.
myhost.node.dc1.consul.	0	IN	A	127.0.0.1
% dig @127.0.0.1 -p 8600 40e4a748-2192-161a-0510-9bf59fe950b5.node.dc1.consul.
40e4a748-2192-161a-0510-9bf59fe950b5.node.dc1.consul. 0	IN A 127.0.0.1
% dig @127.0.0.1 -p 8600 40e4a748.node.dc1.consul.
40e4a748.node.dc1.consul. 0	IN	A	127.0.0.1
% dig @127.0.0.1 -p 8600 40e4a74821.node.dc1.consul.
40e4a74821.node.dc1.consul. 0	IN	A	127.0.0.1
% dig @127.0.0.1 -p 8600 40e4a748-21.node.dc1.consul.
40e4a748-21.node.dc1.consul. 0	IN	A	127.0.0.1
```
2017-02-01 14:46:25 -08:00
Kyle Havlovitz 086975ac0a
Add TLSMinVersion to config options 2017-02-01 16:20:33 -05:00
Sean Chittenden e39dd09bfa
Small premature optimization in `isUUID()`.
If the length isn't `36`, return `false` immediately before firing up
the regexp engine.
2017-02-01 11:00:06 -08:00
James Phillips 84a487526b
Tweaks leader test now that we have new wait timing. 2017-01-25 22:12:22 -08:00
James Phillips d780c49bac
Keeps the old state store state if a restore fails. 2017-01-25 19:42:34 -08:00
James Phillips 418b627f4e
Bails out of blocking queries when a state restore occurs. 2017-01-25 19:00:32 -08:00
James Phillips f286716899
Tweaks a few comments. 2017-01-25 09:58:23 -08:00
James Phillips 8fcf695fa1
Pass state store pointer into the blocking query work function.
Previously the blocking functions all closed over the state store from
their first query, with would not have worked properly when a restore
occurred. This makes sure they get a frest state store pointer each time,
and that pointer is synchronized with the abandon watch.
2017-01-25 09:58:23 -08:00
James Phillips 20ae4b6139
Guts all the old blocking query code. 2017-01-25 09:58:23 -08:00
James Phillips 943ca1b2d9
Cuts KVS endpoints over to new fine-grained watch plumbing. 2017-01-25 09:58:22 -08:00
James Phillips c3a1014fbf
Adds a facility to notify when restores occur. 2017-01-25 09:58:22 -08:00
James Phillips e59f398d80
Adds fine-grained watches to session endpoints. 2017-01-25 09:58:22 -08:00
James Phillips 997934c94f
Adds fine-grained watches to prepared query endpoints. 2017-01-25 09:58:22 -08:00
James Phillips 557df50be6
Adds fine-grained watches to internal endpoints. 2017-01-25 09:58:22 -08:00
James Phillips e1dd427cf4
Adds fine-grained watches to coordinate endpoints. 2017-01-25 09:58:22 -08:00
James Phillips 2cbf45301e
Adds fine-grained watch support to ACL endpoints. 2017-01-25 09:58:22 -08:00
James Phillips 635afd1924
Adds fine-grained watches to health endpoints. 2017-01-25 09:58:22 -08:00
James Phillips a604c438a2
Fixes a race condition when updating the state store during a snapshot restore. 2017-01-25 09:58:22 -08:00
James Phillips 0891b6c74d
Adds fine-grained watches to catalog endpoints. 2017-01-25 09:58:22 -08:00
James Phillips 5afa233d28
Adds diff check for node and service parts of register requests.
We always did an update before which caused excessive watch churn, even
with our new fine-grained queries. This does a diff any only updates the
node and service records if something actually changed.
2017-01-25 09:58:22 -08:00
James Phillips 40ef50c738
Don't do any watch tracking for non-blocking queries. 2017-01-25 09:58:22 -08:00
James Phillips 034d1960b6
Removes some incorrect comments.
We can't actually return a fine-grained index from these tables unless
support is added for tombstones. Otherwise, the index could slip backwards
as things are deleted.
2017-01-25 09:58:22 -08:00
James Phillips 085672a126
Adds new variant of blocking query wrapper with WatchSet support. 2017-01-25 09:58:22 -08:00
Kyle Havlovitz a09b6ea8b4
Fix test import 2017-01-23 21:34:01 -05:00
Kyle Havlovitz cc271aa436
Merge branch 'master' into f-prepared-query-nodemeta 2017-01-23 20:17:48 -05:00
Kyle Havlovitz 3cd054e81f
Add tests for node meta in prepared queries and update docs 2017-01-23 19:17:30 -05:00
James Phillips 9e8952c126 Merge pull request #2661 from hashicorp/f-node-id
Adds basic support for node IDs.
2017-01-18 16:02:38 -08:00
James Phillips 60c135dd67
Uses clean replies each time so they are safe to receive map changes. 2017-01-18 15:40:19 -08:00
James Phillips abe7b33986
Fixes a startup ordering issue between Raft and Serf.
This fixes #2663 and fixes #1899. It's not super related to this PR,
but the startup time changes that this PR brings made this a lot worse
so I was able to track it down.
2017-01-18 15:06:15 -08:00
James Phillips 55cd1d975c
Adds catalog support for node IDs. 2017-01-18 14:26:42 -08:00
Kyle Havlovitz fdf2a10835
First pass at adding node meta filter to prepared queries 2017-01-18 16:23:33 -05:00
James Phillips 96bff003b7
Adds basic support for node IDs. 2017-01-17 22:47:59 -08:00
Kyle Havlovitz b3fdfb09d8
Minor formatting tweaks as a follow-up to #2654 2017-01-17 19:20:29 -05:00
Kyle Havlovitz 9080006a37
Add support for multiple metadata filters to remaining endpoints
Enabled multiple meta filters for /v1/catalog/nodes and /v1/catalog/services
2017-01-13 20:49:13 -05:00
Kyle Havlovitz e37f25dc02
Add node metadata filtering to remaining health/catalog endpoints 2017-01-13 20:08:43 -05:00
James Phillips c837d055cd
Breaks up the state store into several files. 2017-01-13 11:47:16 -08:00
Kyle Havlovitz 3620e5e199
Fix inconsistency in TestStateStore_ServicesByNodeMeta 2017-01-12 19:46:58 -05:00
Kyle Havlovitz 51a1834667
Add -node-meta to agent command line options 2017-01-11 16:09:04 -05:00
Kyle Havlovitz d77890a011
Validate metadata config earlier and handle multiple filters 2017-01-11 15:12:03 -05:00
Kyle Havlovitz 6b5cf20b1c
Fix formatting 2017-01-09 13:49:33 -08:00
Kyle Havlovitz aee766baba
Add meta key validations and more tests 2017-01-09 11:21:49 -08:00
Kyle Havlovitz b25f4c7d35
Add tests for node metadata functionality 2017-01-05 17:21:56 -08:00
Kyle Havlovitz 2a423c6e2c
Add support for setting node metadata fields 2017-01-05 14:10:26 -08:00
James Phillips 927aa3789a
Adds ACL management support to the agent. 2016-12-14 07:07:41 -08:00
nobody d069c0805e fix fmt.Errorf error
missing argument for Errorf("%q"): format reads arg 1, have only 0 args
2016-12-14 18:57:12 +08:00
James Phillips 71a775d986 Merge pull request #2592 from hashicorp/acl-complete-node-session
Adds complete ACL coverage for nodes and sessions.
2016-12-13 13:55:07 -08:00
James Phillips 535d6b21b4
Adds complete ACL coverage for /v1/session endpoints. 2016-12-12 21:59:22 -08:00
James Phillips 4aa070834b
Adds complete ACL coverage for /v1/internal/ui/node endpoints. 2016-12-12 18:22:10 -08:00
James Phillips ac967176ee
Adds complete ACL support for /v1/query/<query id or name>/execute.
This was already supported by previous changes to the ACL filter, so
we just added a test to show it working.
2016-12-12 17:28:06 -08:00
James Phillips 7e7c1ebb13
Fixes implementation of node ACLs for /v1/catalog/node/<node>.
This would return a "permission denied" error, but this changes it to
return the same response as a node that doesn't exist (as was originally
intended and written in the code comments).
2016-12-12 16:53:31 -08:00
James Phillips 7a21a591b8
Adds full ACL coverage for /v1/health endpoints. 2016-12-12 16:28:52 -08:00
Kyle Havlovitz 402f30d52e Merge pull request #2591 from hashicorp/snapshot-interval
Change raft snapshot interval to 5 seconds
2016-12-12 19:15:10 -05:00
James Phillips 8db53b4ac2
Adds complete ACL coverage for /v1/coordinate/nodes and Coordinate.Update RPC. 2016-12-12 14:52:27 -08:00
James Phillips 67f1fe8d0b
Adds support for a new "acl_agent_token" which is used for internal
catalog operations.
2016-12-12 14:52:27 -08:00
James Phillips 99a5ae2737
Bans anonymous queries that aren't tied to a session.
This gets us coverage of PQ creation under the existing service
policy or the soon-to-be-added session policy.
2016-12-12 14:52:27 -08:00
Kyle Havlovitz 3ae8bedd6d
Change raft snapshot interval to 5 seconds 2016-12-12 13:31:42 -05:00
James Phillips 1ec02e5af4
Adds complete ACL coverage for /v1/catalog/service/<service>. 2016-12-12 08:34:15 -08:00
James Phillips 4dc3b17078
Adds complete ACL coverage for /v1/catalog/nodes. 2016-12-10 16:49:19 -08:00
James Phillips 63a6abe03c
Adds complete ACL coverage for /v1/catalog/node/<node>. 2016-12-10 16:49:19 -08:00
James Phillips 8ed0b81adb
Adds complete ACL coverage for /v1/catalog/deregister.
This included some state store helpers to make this more efficient.
2016-12-09 21:04:44 -08:00
James Phillips 7fa693a338
Adds complete ACL coverage for /v1/catalog/register. 2016-12-09 21:04:37 -08:00
James Phillips 2b19a5340c
Removes the exception for the "consul" service in the catalog. 2016-12-07 17:58:23 -08:00
Kyle Havlovitz 26a26a489d Add retry with backoff to initial bootstrap checks (#2561) 2016-12-01 17:05:02 -05:00
Seth Vargo 1c55429a38
Add an API method for determining the best status
Given a list of HealthChecks, this determines the "best" status for the
collective group. This is useful for nodes and services, which may have
multiple checks associated with them.
2016-11-29 18:41:46 -05:00
Kyle Havlovitz 2d37a07476 Add keyring http endpoints 2016-11-22 20:10:43 -05:00
Kyle Havlovitz 9adc3854d1 Retry with backoff on session invalidation failure (#2475) 2016-11-04 21:53:22 -07:00
James Phillips be4056789f Moves the snapshot package up one level. (#2472) 2016-11-03 21:36:25 -07:00
Kyle Havlovitz 169cae2203 Disallow -bootstrap-expect flag in dev mode (#2464) 2016-11-03 01:54:43 -04:00
Kyle Havlovitz 440611f9f7 Add snapshot inspect subcommand (#2451) 2016-10-31 19:37:27 -04:00
Kyle Havlovitz c6f461aa25 Enable snapshots in dev mode (#2453) 2016-10-31 14:39:47 -04:00
James Phillips bc29610124 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
Kyle Havlovitz f0aa65754b Wait for agent joins to finish in TestClient_RPC 2016-10-25 17:48:11 -07:00
Kyle Havlovitz 2379b822ad Add wait logic to TestClient_RPC_Pool 2016-10-25 17:48:11 -07:00
James Phillips b423dee303
Fixes port numbers in peers.info. 2016-10-05 18:09:15 -07:00
James Phillips 8e76af4311 Merge pull request #2319 from hashicorp/f-bootstrap-abort
Adds check that aborts bootstrap mode if there's an existing cluster.
2016-09-01 09:49:03 -07:00
James Phillips 94c0e961eb
Fixes error message in test. 2016-09-01 09:48:08 -07:00
James Phillips ce93b82e1e
Makes port selection atomic in unit tests. 2016-09-01 01:01:28 -07:00
James Phillips d04a706a7c
Tweaks comment to be more correct. 2016-08-31 23:54:53 -07:00
James Phillips 4dd9b4b08a
Adds check that aborts bootstrap mode if there's an existing cluster. 2016-08-31 21:25:56 -07:00
James Phillips 750e1751ac
Copies the member data instead of referencing by pointer. 2016-08-30 16:54:21 -07:00
James Phillips 6be1e07fec
Makes the Raft configuration API easier to consume. 2016-08-30 11:30:56 -07:00
James Phillips 5df4b6bef2
Adds a log warning when operator peer changes occur. 2016-08-30 10:23:32 -07:00
James Phillips 1b7a16b7d3
Adds new consul operator endpoint, CLI, and ACL and some basic Raft commands. 2016-08-30 00:02:50 -07:00
James Phillips 29e52307cb
Makes empty checkServiceNode return a nil.
The change in #2308 had an inadvertent interface change, so we fix that with
a special case in this fix.
2016-08-29 19:12:07 -07:00
James Phillips 327fe725d9
Preallocates result struct, which was a profiling hot spot. 2016-08-26 16:34:28 -07:00
James Phillips c5b6ac3655
Removes leader_lease_timeout from stats. 2016-08-25 15:39:19 -07:00
James Phillips 2f4c237cff
Adds a max raft multiplier and tweaks documentation. 2016-08-25 15:36:05 -07:00
James Phillips 5df36fbd82
Stops scaling the commit timeout. 2016-08-25 15:05:40 -07:00
James Phillips f65ef936cb
Increases RPC hold timeout for new default timing.
Rather than scale this we just bump it up a bit. It'll be on the edge in
the lower-performance default mode, and will have plenty of margin in the
high-performance mode. This seems like a reasonable compromise to keep the
logic here simple vs. scaling, and seems inline with the expectations of
the different modes of operation.
2016-08-24 23:35:28 -07:00
James Phillips b339b0d2fc
Adds performance tuning capability for Raft, detuned defaults, and supplemental docs. 2016-08-24 21:58:37 -07:00
James Phillips 0bdbdf1ba8 Merge pull request #2226 from abhinavdahiya/rm-health-unknown
Fixes #1775; Removes 'unknown' state
2016-08-17 17:51:04 -07:00
James Phillips 1f539d9914
Makes the filled-in parts of ServiceNode more explicit. 2016-08-12 18:25:36 -07:00
David van Geest 360e196c93
Translate Address to tagged WAN address in HTTP API when appropriate. 2016-08-12 18:25:36 -07:00