Commit Graph

45 Commits

Author SHA1 Message Date
James Phillips bc29610124 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
James Phillips 97a25e8564
Closes the conn on bad protocol version. 2016-08-09 18:13:53 -07:00
James Phillips 359587f70e
Removes support for muxado and protocol version 1. 2016-08-09 18:10:04 -07:00
Sean Chittenden 5ae7835988 Rename server_details package to agent 2016-03-29 17:39:19 -07:00
Sean Chittenden d6b4345375 Return error from PingConsulServer
In order to report why a Ping failed, change the signature of PingConsulServers to include an error message.
2016-03-28 13:38:58 -07:00
Sean Chittenden 2d9982eb27 Proactively ping server before rotation
Before shuffling the server list, proactively ping the next server in the list to establish the connection and verify the remote endpoint is healthy.
2016-03-26 19:28:13 -07:00
Armon Dadgar a033d7aed2 consul: shrink yamux recv buffer on idle streams 2015-11-27 17:20:57 -08:00
Ryan Uber 33741f9156 consul: initial pass at refactoring RPC using net-rpc-msgpackrpc 2015-10-13 22:15:23 -07:00
Dale Wijnand c5168e1263 Fix a bunch of typos. 2015-09-15 13:22:08 +01:00
James Phillips 8df55b6964 Changes to an unbuffered channel, since we just close it. 2015-08-13 11:38:02 -07:00
James Phillips 1c6414e806 Cleans up locking and factors markForUse into a Conn method. 2015-08-13 10:01:05 -07:00
James Phillips 18711b40b8 Gets rid of follow up attempts if the lead thread can't connect. 2015-08-12 20:14:48 -07:00
James Phillips 7d9db86b03 Adds missing ref count for the race condition case. 2015-08-12 19:26:01 -07:00
James Phillips 0efc49b510 Fixes #1165 by having threads wait for any outstanding connect to finish. 2015-08-12 18:48:15 -07:00
Armon Dadgar 9642384429 consul: support the new TLS wrapper 2015-05-11 15:15:36 -07:00
Armon Dadgar 3bf337a6ac consul: thread the target DC through the RPC path 2015-05-11 13:09:19 -07:00
Armon Dadgar a1de4b17c2 consul: use tlsutil.Wrapper instead of tls.Config directly 2015-05-11 13:09:19 -07:00
Armon Dadgar 32f3d17931 consul: do not clear connection cache on RPC errors 2015-04-07 14:17:20 -07:00
Armon Dadgar f3fd072418 Revert "* use defer to avoid tracking lock"
This reverts commit a030abdefc54470394a2a44008e02f3b3d0510ec.
This commit causes a connection to be leaked if there is a race with
another concurrent RPC.
2015-02-16 14:04:47 -08:00
Ali Abbas 40979b1159 * use defer to avoid tracking lock
* simplify control flow
2014-12-06 12:32:18 +01:00
Veres Lajos 850d5bdc32 typofixes - https://github.com/vlajos/misspell_fixer 2014-12-04 23:25:06 +00:00
Ali Abbas 818fc22c9f * Fix race condition on read/write of shutdown bool variable of server and connection pool.
* In connection pool, there is no guarantee that .reap() cannot execute the same time as .Shutdown() is called. It also did not benefit to eval shutdown when a select is run on the shutdown channel.
* In server, same principle applies to handleConsulConn. Since we also have a shutdown channel, it makes more to use this than to loop on a bool variable.
2014-11-26 10:39:25 +01:00
Atin Malaviya b4424a1a50 Moved TLS Config stuff to tlsutil package 2014-11-18 11:03:36 -05:00
Armon Dadgar af90aa8026 Gofmt 2014-10-20 10:21:31 -07:00
Armon Dadgar 3f36515544 Switching to the pinned version of msgpack 2014-10-17 18:26:19 -07:00
Armon Dadgar daa32dd6f8 consul: don't close a nil connection 2014-10-02 10:26:25 -07:00
Nelson Elhage 0a2476b20e Restore the 0.2 TLS verification behavior.
Namely, don't check the DNS names in TLS certificates when connecting to
other servers.

As of golang 1.3, crypto/tls no longer natively supports doing partial
verification (verifying the cert issuer but not the hostname), so we
have to disable verification entirely and then do the issuer
verification ourselves. Fortunately, crypto/x509 makes this relatively
straightforward.

If the "server_name" configuration option is passed, we preserve the
existing behavior of checking that server name everywhere.

No option is provided to retain the current behavior of checking the
remote certificate against the local node name, since that behavior
seems clearly buggy and unintentional, and I have difficulty imagining
it is actually being used anywhere. It would be relatively
straightforward to restore if desired, however.
2014-06-28 13:32:42 -07:00
Robert B Gordon 987c078957 Seems like we should actually check the reference count. 2014-06-13 11:25:01 -05:00
Armon Dadgar 06cd40d1e7 consul: fixing use of nil log output. Fixes #203 2014-06-09 11:16:53 -07:00
Armon Dadgar c656bbfbcf Rename shared msgpack handle 2014-06-08 14:02:42 -07:00
Andrew M Bursavich d209517d50 reuse codec.MsgpackHandle 2014-06-07 01:13:38 -07:00
Armon Dadgar 09a988e8d4 consul: Conn pool clean, spare existing streams 2014-05-28 16:55:39 -07:00
Armon Dadgar 319ab05b8c consul: Provide logger to yamux 2014-05-28 16:32:25 -07:00
Armon Dadgar d4a62e7c0d consul: ensure connections are properly closed 2014-05-28 16:32:24 -07:00
Armon Dadgar a79c3d2103 consul: Pool client connections, support for yamux connections 2014-05-28 16:32:24 -07:00
Armon Dadgar 5b18b000f2 consul: Connection pool supports TLS mode 2014-04-07 15:06:59 -07:00
Armon Dadgar ba765b193f consul: ConnPool multiplexes a single connection instead of using multiple 2014-02-05 16:19:05 -08:00
Armon Dadgar e9c0493c9e consul: Make some ConnPool methods private 2014-02-05 15:30:19 -08:00
Armon Dadgar 1b84705cb3 consul: ensure conn pool shutdown is fast 2014-02-05 14:20:18 -08:00
Armon Dadgar ac15ca1139 Better error logging 2013-12-24 12:22:42 -08:00
Armon Dadgar 32c822db1b Adding time based reaping to ConnPool 2013-12-19 15:42:17 -08:00
Armon Dadgar a8a9d2cd45 consul: helper to make RPC calls 2013-12-09 14:58:49 -08:00
Armon Dadgar f7f743dfa9 consul: ConnPool creates RPC client 2013-12-09 14:52:22 -08:00
Armon Dadgar c28ebbf60f consul: Write the byte to set the RPC mode 2013-12-09 14:29:20 -08:00
Armon Dadgar 205941ffcf consul: adding connection pool 2013-12-09 12:09:57 -08:00