Commit Graph

41 Commits

Author SHA1 Message Date
Kyle Havlovitz a09b6ea8b4
Fix test import 2017-01-23 21:34:01 -05:00
Kyle Havlovitz 3cd054e81f
Add tests for node meta in prepared queries and update docs 2017-01-23 19:17:30 -05:00
James Phillips 60c135dd67
Uses clean replies each time so they are safe to receive map changes. 2017-01-18 15:40:19 -08:00
Kyle Havlovitz fdf2a10835
First pass at adding node meta filter to prepared queries 2017-01-18 16:23:33 -05:00
James Phillips ac967176ee
Adds complete ACL support for /v1/query/<query id or name>/execute.
This was already supported by previous changes to the ACL filter, so
we just added a test to show it working.
2016-12-12 17:28:06 -08:00
James Phillips 99a5ae2737
Bans anonymous queries that aren't tied to a session.
This gets us coverage of PQ creation under the existing service
policy or the soon-to-be-added session policy.
2016-12-12 14:52:27 -08:00
James Phillips bc29610124 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
Ryan Uber 577523fc73 consul: sort source node first if at position <= 10 in PQ's 2016-07-01 14:28:58 -07:00
Ryan Uber ebacaa2d67 consul: send agent source data as separate query source 2016-06-30 16:51:18 -07:00
Ryan Uber 782a081925 consul: use source parameter for near prepared queries 2016-06-30 12:11:20 -07:00
Ryan Uber 270270a33a consul: send origin node + dc when executing prepared queries 2016-06-21 15:34:26 -07:00
Ryan Uber 925915c6ac consul: test baked-in distance sort 2016-06-21 12:54:18 -07:00
Ryan Uber 114e57fff1 consul: use the Near field instead of PreferLocal 2016-06-21 12:39:40 -07:00
Ryan Uber 89fe991ab7 consul: test raw PreferLocal functionality 2016-06-20 14:53:13 -07:00
James Phillips 265a8d4053 Increases timeouts for coordinate tests.
We take the interval and add the random stagger to it, so 2X is cutting it
too close and the unit tests are often flaky.
2016-03-21 16:44:35 -07:00
James Phillips 275c84a0cc Renames "debug" endpoint and structures to "explain". 2016-03-07 10:45:39 -08:00
James Phillips 8493640b09 Adds a prepared query debug endpoint. 2016-03-07 10:45:39 -08:00
James Phillips 918b1ace47 Applies prefix ACL to a catch-all template as a special case. 2016-03-07 10:45:39 -08:00
James Phillips 62405110dc Wraps the prepared query to also store the compiled template. 2016-03-07 10:45:39 -08:00
James Phillips f0150ff5ce Adds missing token redact in the GET path. 2016-02-26 15:59:00 -08:00
James Phillips c75256ac8b Adds a check for users re-submitting the redacted token. 2016-02-24 17:35:26 -08:00
James Phillips 2f7eac8b86 Renames "prepared_query" ACL policy to "query". 2016-02-24 17:02:06 -08:00
James Phillips 54f0b7bbb6 Completes switch of prepared_query ACLs to govern query names. 2016-02-24 01:26:16 -08:00
James Phillips 633c231d67 Creates new "prepared-query" ACL type and new token capture behavior.
Prior to this change, prepared queries had the following behavior for
ACLs, which will need to change to support templates:

1. A management token, or a token with read access to the service being
   queried needed to be provided in order to create a prepared query.

2. The token used to create the prepared query was stored with the query
   in the state store and used to execute the query.

3. A management token, or the token used to create the query needed to be
   supplied to perform and CRUD operations on an existing prepared query.

This was pretty subtle and complicated behavior, and won't work for
templates since the service name is computed at execution time. To solve
this, we introduce a new "prepared-query" ACL type, where the prefix
applies to the query name for static prepared query types and to the
prefix for template prepared query types.

With this change, the new behavior is:

1. A management token, or a token with "prepared-query" write access to
   the query name or (soon) the given template prefix is required to do
   any CRUD operations on a prepared query, or to list prepared queries
   (the list is filtered by this ACL).

2. You will no longer need a management token to list prepared queries,
   but you will only be able to see prepared queries that you have access
   to (you get an empty list instead of permission denied).

3. When listing or getting a query, because it was easy to capture
   management tokens given the past behavior, this will always blank out
   the "Token" field (replacing the contents as <hidden>) for all tokens
   unless a management token is supplied. Going forward, we should
   discourage people from binding tokens for execution unless strictly
   necessary.

4. No token will be captured by default when a prepared query is created.
   If the user wishes to supply an execution token then can pass it in via
   the "Token" field in the prepared query definition. Otherwise, this
   field will default to empty.

5. At execution time, we will use the captured token if it exists with the
   prepared query definition, otherwise we will use the token that's passed
   in with the request, just like we do for other RPCs (or you can use the
   agent's configured token for DNS).

6. Prepared queries with no name (accessible only by ID) will not require
   ACLs to create or modify (execution time will depend on the service ACL
   configuration). Our argument here is that these are designed to be
   ephemeral and the IDs are as good as an ACL. Management tokens will be
   able to list all of these.

These changes enable templates, but also enable delegation of authority to
manage the prepared query namespace.
2016-02-23 17:12:43 -08:00
James Phillips e6ccf5b9ed Returns a 404 from a get or execute of a nonexistent query. 2015-11-15 17:06:00 -08:00
James Phillips c9ef552385 Plumbs the service name back and uses agent-specific TTL settings as a fallback. 2015-11-15 17:06:00 -08:00
James Phillips cad6938474 Adds unit tests for prepared queries and DNS, using existing tests for equivalence. 2015-11-15 17:06:00 -08:00
James Phillips 7762a828d3 Adds query metadata to prepared query execute response. 2015-11-15 17:06:00 -08:00
James Phillips 4051db4e88 Changes Lookup to Get since we don't need it (only Execute does). 2015-11-15 17:06:00 -08:00
James Phillips 07361d3e96 Adds test for remote datacenter selection and query logic. 2015-11-15 17:06:00 -08:00
James Phillips 6a1c571a38 Adds a test for the server wrapper. 2015-11-15 17:06:00 -08:00
James Phillips 9428f025f9 Adds tag filter tests. 2015-11-15 17:06:00 -08:00
James Phillips da5cf9cdf2 Adds execute tests for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 00ea015770 Adds execute leader forward test for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 264e86e533 Adds a leader forwarding case for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips bb0f136412 Adds tests for query lookup and list endpoints. 2015-11-15 17:06:00 -08:00
James Phillips 495d00c0b8 Adds query parsing unit tests. 2015-11-15 17:06:00 -08:00
James Phillips 72f6c8a261 Adds ACL cases for apply. 2015-11-15 17:06:00 -08:00
James Phillips 76a55eed53 Completes non-ACL version of apply test. 2015-11-15 17:06:00 -08:00
James Phillips bc0fba43d8 Adds lookup and list endpoints and basic end-to-end apply test. 2015-11-15 17:06:00 -08:00
James Phillips 09034a84bd Adds prefix "prepared" to everything prepared query-related. 2015-11-15 17:06:00 -08:00
Renamed from consul/query_endpoint_test.go (Browse further)