Commit Graph

67 Commits

Author SHA1 Message Date
James Phillips bc29610124 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
James Phillips d11a7a197c
Removes upper end of muxado handler. 2016-08-09 18:16:41 -07:00
Armon Dadgar 5d0a977bdf consul: Refactor forward to hold RPC when no leader is known 2016-07-10 13:24:06 -04:00
James Phillips 8358df599d Merge pull request #2127 from hashicorp/b-remote-consuls-locking
Ensure locking of `Server`'s `remoteConsuls`.
2016-06-21 10:00:04 -07:00
Sean Chittenden ebdb72ce0a
Ensure locking of `Server`'s `remoteConsuls`. 2016-06-20 22:59:49 -07:00
Sean Chittenden b78c95d37e
Prefer rand.Int31n() over rand.Int31(). 2016-06-20 15:26:27 -07:00
Sean Chittenden c470553b6b Factor out duplicate functions into a lib package
Consolidate code duplication and tests into a single lib package.  Most of these functions were from various **/util.go functions that couldn't be imported due to cyclic imports.  The consul/lib package is intended to be a terminal node in an import DAG and a place to stash various consul-only helper functions.  Pulled in hashicorp/go-uuid instead of consolidating UUID access.
2016-01-29 16:57:45 -08:00
James Phillips 92531812b6 Adds source address logging on RPC errors. 2015-12-16 14:03:03 -08:00
James Phillips 3fedffbffb Gets rid of the transitional "New" suffix on state store and RPC. 2015-10-15 14:59:09 -07:00
James Phillips 3f07f0685f Nukes old state store's connection to FSM and RPC. 2015-10-15 14:59:09 -07:00
James Phillips 04b365495d Completes state store for KV, sessions, tombstones, and nodes/services/checks (needs tests and integration). 2015-10-15 14:59:09 -07:00
James Phillips edae626f36 Integrates new state store for ACLs. 2015-10-15 14:59:09 -07:00
Ryan Uber 33741f9156 consul: initial pass at refactoring RPC using net-rpc-msgpackrpc 2015-10-13 22:15:23 -07:00
Anthony Scalisi 8d733b7fca remove various typos 2015-09-11 12:29:54 -07:00
Armon Dadgar 2c9592c5ee consul: lower default query time and add small stagger 2015-05-14 17:59:43 -07:00
Armon Dadgar 23a1df1548 consul: proactively clear timers 2015-05-14 17:42:47 -07:00
Armon Dadgar 2bc43844bc consul: ensure blocking query cleans any lingering state 2015-05-14 17:38:42 -07:00
Armon Dadgar 3bf337a6ac consul: thread the target DC through the RPC path 2015-05-11 13:09:19 -07:00
Armon Dadgar 2ca41b15df consul: updating for Raft API changes 2015-05-08 11:35:52 -07:00
Armon Dadgar 2d2fc71e0d consul: more telemetry on RPCs and queries 2015-05-07 17:25:12 -07:00
Armon Dadgar 0ea4f3d846 consul: Optimize KV watching 2015-01-05 16:41:57 -08:00
Ali Abbas 818fc22c9f * Fix race condition on read/write of shutdown bool variable of server and connection pool.
* In connection pool, there is no guarantee that .reap() cannot execute the same time as .Shutdown() is called. It also did not benefit to eval shutdown when a select is run on the shutdown channel.
* In server, same principle applies to handleConsulConn. Since we also have a shutdown channel, it makes more to use this than to loop on a bool variable.
2014-11-26 10:39:25 +01:00
Ryan Uber 4a8249db00 consul: fix obscure bug when launching goroutines from for loop 2014-11-19 16:37:40 -08:00
Ryan Uber fcacee723b consul: simplify keyring operations 2014-11-19 16:36:19 -08:00
Ryan Uber b3f251de9c command/keyring: clean up tests 2014-11-19 16:36:18 -08:00
Ryan Uber d02afd42fb agent: -encrypt appends to keyring if one exists 2014-11-19 16:36:01 -08:00
Ryan Uber 057c22db10 consul: generalize multi-DC RPC call broadcasts 2014-11-19 16:34:18 -08:00
Ryan Uber f6b5fc8c08 consul: cross-dc key rotation works 2014-11-19 16:34:17 -08:00
Ryan Uber f9b5b15a6b consul: use a function for ingesting responses 2014-11-19 16:34:17 -08:00
Ryan Uber a551a6e4a0 consul: refactor keyring, repeat RPC calls to all DC's 2014-11-19 16:34:17 -08:00
Armon Dadgar af90aa8026 Gofmt 2014-10-20 10:21:31 -07:00
Armon Dadgar 3f36515544 Switching to the pinned version of msgpack 2014-10-17 18:26:19 -07:00
Armon Dadgar c656bbfbcf Rename shared msgpack handle 2014-06-08 14:02:42 -07:00
Andrew M Bursavich d209517d50 reuse codec.MsgpackHandle 2014-06-07 01:13:38 -07:00
Armon Dadgar 319ab05b8c consul: Provide logger to yamux 2014-05-28 16:32:25 -07:00
Armon Dadgar 6b2fe4869b consul: Suppress messages about closed connections 2014-05-28 16:32:25 -07:00
Armon Dadgar 313f79913e consul: Pass protocol version for leader forwarding 2014-05-28 16:32:24 -07:00
Armon Dadgar 589105eee4 consul: Store the protocol version for a server 2014-05-28 16:32:24 -07:00
Armon Dadgar 8e636ef340 consul: Support for incoming Yamux connections 2014-05-28 16:32:24 -07:00
Armon Dadgar 5c90e9b43e consul: Suppress error if connection is closed 2014-05-23 16:28:55 -07:00
Armon Dadgar c54f53eaf7 consul: Remove RPC client tracking. Fixes #149. 2014-05-14 17:34:24 -07:00
Armon Dadgar 813ebf0b0c consul: Adding a Raft command enqueue limit 2014-05-09 19:18:06 -07:00
Armon Dadgar 144a531237 consul: Drop the Raft logs size warning from 8MB to 1MB 2014-05-06 14:10:08 -07:00
Armon Dadgar de507c0317 consul: Adding a warning for large raft entries 2014-04-28 22:25:09 -07:00
Armon Dadgar 7a3c3e718e consul: Adding consistent read enforcement 2014-04-21 11:49:21 -07:00
Armon Dadgar 710a823379 consul: Merging BlockingQuery into QueryOptions 2014-04-21 11:31:15 -07:00
Armon Dadgar 55b5fd4e6a consul: Use QueryMeta to simplify blockingRPC interface 2014-04-21 11:18:27 -07:00
Armon Dadgar 5b91b6f06e consul: Moving QueryMeta handling into blockingRPC 2014-04-21 11:04:52 -07:00
Armon Dadgar f7bcffe627 consul: Adding a method to enforce consistent read 2014-04-21 10:59:46 -07:00
Armon Dadgar 9e0e5259f9 consul: Adding support for QueryMeta on all queries 2014-04-21 10:59:46 -07:00