Commit graph

101 commits

Author SHA1 Message Date
James Phillips bc29610124 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
Felix f9e001cb6e return an error if no logOutput is provided
Otherwise the code will panic at a later stage where it is more complicated to figure out what’s actually going on.
2016-09-17 17:46:13 +09:00
James Phillips 1b7a16b7d3
Adds new consul operator endpoint, CLI, and ACL and some basic Raft commands. 2016-08-30 00:02:50 -07:00
James Phillips bc333335be
Adds an X-Consul-Translate-Addresses to signal translation is enabled. 2016-08-16 11:31:41 -07:00
James Phillips f00e49112a
Installs a wrapper around HTTP endpoints that generate metrics. 2016-08-09 15:41:15 -07:00
James Phillips 4a931ae12e
Adds an ACL replication status endpoint. 2016-08-04 23:30:16 -07:00
James Phillips d980cbcd9d Moves txn code into a new endpoint, not specific to KV. 2016-05-10 21:58:02 -07:00
James Phillips 54b930103c Terminates pretty responses with a newline. 2016-05-10 20:02:36 -07:00
James Phillips 471160d8f0 Performs basic plumbing of KVS transactions through all the layers. 2016-05-09 22:15:49 -07:00
James Phillips bd4f2ee6b7 Adds a new PUT-based TTL check update endpoint. 2016-03-02 17:54:01 -08:00
James Phillips cbdff8296f Fixes redirect from / to /ui when internal UI is enabled. 2016-02-12 16:11:32 -08:00
Chris Cooper 0ce79f8198 add comment codes 2016-02-05 17:06:42 -05:00
Chris Cooper 4890220929 refactors http server error codes to use the http statuses defined in the http package instead of literals 2016-02-05 16:51:31 -05:00
Philippe M. Chiasson e308bc3733 remove address splitting 2016-01-05 11:49:40 -05:00
Philippe M. Chiasson 6f316138d3 Move logged client IP to the end of the log message 2015-12-30 15:39:34 -05:00
Philippe M. Chiasson 8e938d4d38 Merge branch 'master' of http://github.com/hashicorp/consul into issue/1447/http-access-logs 2015-12-30 15:38:30 -05:00
Ryan Uber 52ad989405 agent: static UI is configurable 2015-12-24 22:21:34 -05:00
Ryan Uber a262115679 agent: compile web assets into consul binary 2015-12-24 22:21:34 -05:00
Philippe M. Chiasson 396df9a0c1 Add client ip to http access logs
Fixes #1447
2015-11-24 15:06:33 -05:00
James Phillips 978fb443a7 Adds a note about obfuscating query name/ID from the logs. 2015-11-15 17:06:00 -08:00
James Phillips 124410b66c Adds an HTTP endpoint for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 0e7d5871c5 Installs a friendly handler for coordinate endpoints when coordinates are disabled. 2015-10-23 15:23:01 -07:00
James Phillips a2bcef00a0 Adds endpoints for raw network coordinates. 2015-10-23 15:23:01 -07:00
James Phillips fecd639b02 Changes ?near=self to a safer ?near=_agent, which is also clearer about what it does. 2015-10-23 15:23:01 -07:00
James Phillips 1724b9a6be Adds a magic "self" node name to distance queries. 2015-10-23 15:23:01 -07:00
James Phillips b63909cf67 Adds coordinate sorting support to catalog queries for nodes and service nodes. 2015-10-23 15:23:01 -07:00
David Adams 5f175add40 Add HTTP request header X-Consul-Token
Add support for an X-Consul-Token HTTP request header to specify the
token with which this request should be fulfilled. The header would have
precedence over the responding Agent's default token, but would have
lower precedence than a token specified in the query string.
2015-10-19 11:26:01 -05:00
Michael Puncel a94589ad67 Add http method to log output 2015-10-02 18:33:06 -07:00
Armon Dadgar 655666170a agent: Always enable the UI endpoints 2015-09-01 18:28:32 -07:00
Ryan Uber bc96c14a6f agent: document the scada http creation func 2015-08-25 17:19:11 -07:00
Ryan Uber 1378fd93b0 agent: scada client and HTTP server are tracked separately 2015-08-25 16:59:53 -07:00
Ryan Uber 47073c87c1 agent: don't replace empty tokens in the logs, fixes #1020 2015-06-12 00:11:37 -07:00
Ryan Uber 34fc9a37fd agent: parse raw query URL to avoid closing the request body early 2015-04-13 17:31:53 -07:00
Ryan Uber adbc5c92c6 agent: hide tokens from logs and monitor 2015-04-12 11:17:31 -07:00
Armon Dadgar a446fbd098 agent: Enable UI endpoints for SCADA 2015-02-18 15:15:02 -08:00
Armon Dadgar 6ee6f67695 agent: Use AtlasACLToken 2015-02-18 15:12:15 -08:00
Armon Dadgar 269153705b agent: Fixing setup of SCADA HTTP listener 2015-02-18 15:12:15 -08:00
Armon Dadgar d9f95e2983 agent: Fixing panic on shutdown 2015-02-18 15:12:15 -08:00
Armon Dadgar 948f9a0d00 agent: SCADA HTTP listener 2015-02-18 15:12:14 -08:00
Ryan Uber 068c4137cb agent/http: clean up socket setup 2015-01-23 17:57:04 -08:00
Armon Dadgar 14d55cb40a Merge pull request #622 from hashicorp/f-sockets
Unix domain sockets
2015-01-21 16:30:03 -08:00
Ryan Uber b71ffb1fa3 agent: change node maintenance endpoint 2015-01-21 10:57:44 -08:00
Ryan Uber c01277725a agent: beginning socket user/group/mode support as discussed in #612 2015-01-20 13:44:27 -08:00
Ryan Uber 8ea9eb87ea agent: node maintenance mode works 2015-01-16 15:38:13 -08:00
Ryan Uber 4f4eb204f6 agent: first pass at service maintenance mode 2015-01-16 15:37:51 -08:00
Ryan Uber 263614d596 agent: error if binding to existing socket file 2015-01-16 12:39:15 -08:00
Ryan Uber 61d2f71d14 agent: only ignore errors on IsNotExist() 2015-01-16 09:14:52 -08:00
Ryan Uber ad19c0afc2 agent: beginning refactor 2015-01-16 00:45:03 -08:00
Jeff Mitchell 400eadd671 RPC and HTTP interfaces fully generically-sockified so Unix is supported.
Client works for RPC; will honor CONSUL_RPC_ADDR. HTTP works via consul/api;
honors CONSUL_HTTP_ADDR.

The format of a Unix socket in configuration data is:
"unix://[/path/to/socket];[username or uid];[gid];[mode]"

Obviously, the user must have appropriate permissions to create the socket
file in the given path and assign the requested uid/gid. Also note that Go does
not support gid lookups from group name, so gid must be numeric. See
https://codereview.appspot.com/101310044

When connecting from the client, the format is just the first part of the
above line:
"unix://[/path/to/socket]"

This code is copyright 2014 Akamai Technologies, Inc. <opensource@akamai.com>
2015-01-14 19:31:21 +00:00
Armon Dadgar 1088a5c170 Merge pull request #558 from ceh/http-api-response-headers
add ability to specify response headers on the HTTP API
2015-01-05 11:36:08 -08:00