12 lines
9.3 KiB
HTML
12 lines
9.3 KiB
HTML
<!DOCTYPE html> <html lang=en> <head> <meta charset=utf-8> <meta name=viewport content="width=device-width, initial-scale=1.0"> <meta name=description content=""> <meta name=author content=""> <title>Gossip Protocol - Serf</title> <link href="/stylesheets/bootstrap.min-82fe1490.css" media=screen rel=stylesheet /><link href="/stylesheets/main-e5014f86.css" media=screen rel=stylesheet /> <!--[if lt IE 9]><script src="/javascripts/html5shiv-310dd184.js"></script> <script src="/javascripts/respond.min-88c91176.js"></script><![endif]--> <script>
|
|
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
|
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
|
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
|
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
|
|
|
|
ga('create', 'UA-45101516-1', 'serfdom.io');
|
|
ga('send', 'pageview');
|
|
|
|
</script> </head> <body class="page-Gossip Protocol"> <div id=header> <div class=container> <a class="navbar-brand logo" href="/"> <span></span> </a> <a class="navbar-brand text rls-l" href="/">SERF</a> <ul class="buttons nav navbar-nav navbar-right rls-sb"> <li class=first><a href="/downloads.html">Download</a></li> <li><a href="https://github.com/hashicorp/serf">Github</a></li> </ul> <ul class="main-links nav navbar-nav navbar-right rls-sb"> <li><a href="/intro/index.html">Intro</a></li> <li><a href="/docs/index.html">Docs</a></li> <li><a href="/community.html">Community</a></li> </ul> </div> </div> <div class=container> <div class=col-md-4> <div class="docs-sidebar hidden-print affix-top" role=complementary> <ul class="nav docs-sidenav"> <li> <a href="/docs/index.html">Documentation Home</a> </li> <li> <a href="/docs/upgrading.html">Upgrading and Compatibility</a> <ul class=nav> <li> <a href="/docs/upgrading.html">Upgrading Serf</a> </li> <li> <a href="/docs/compatibility.html">Compatibility Promise</a> </li> </ul> </li> <li class=active> <a href="/docs/internals/index.html">Serf Internals</a> <ul class=nav> <li class=active> <a href="/docs/internals/gossip.html">Gossip Protocol</a> </li> <li> <a href="/docs/internals/security.html">Security Model</a> </li> <li> <a href="/docs/internals/simulator.html">Convergence Simulator</a> </li> </ul> </li> <li> <a href="/docs/commands/index.html">Serf Commands (CLI)</a> <ul class=nav> <li> <a href="/docs/commands/agent.html">agent</a> </li> <li> <a href="/docs/commands/event.html">event</a> </li> <li> <a href="/docs/commands/force-leave.html">force-leave</a> </li> <li> <a href="/docs/commands/join.html">join</a> </li> <li> <a href="/docs/commands/keygen.html">keygen</a> </li> <li> <a href="/docs/commands/leave.html">leave</a> </li> <li> <a href="/docs/commands/members.html">members</a> </li> <li> <a href="/docs/commands/monitor.html">monitor</a> </li> </ul> </li> <li> <a href="/docs/agent/basics.html">Serf Agent</a> <ul class=nav> <li> <a href="/docs/agent/basics.html">Running and Stopping</a> </li> <li> <a href="/docs/agent/options.html">Configuration</a> </li> <li> <a href="/docs/agent/event-handlers.html">Event Handlers</a> </li> <li> <a href="/docs/agent/encryption.html">Encryption</a> </li> <li> <a href="/docs/agent/rpc.html">RPC Protocol</a> </li> </ul> <li> <a href="/docs/roadmap.html">Roadmap</a> </li> </ul> </div> </div> <div class=col-md-8 role=main> <div class=bs-docs-section> <h1 id=toc_0>Gossip Protocol</h1> <p>Serf uses a <a href="#">gossip protocol</a> to broadcast messages to the cluster. This page documents the details of this internal protocol. The gossip protocol is based on <a href="//www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf">"SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol"</a>, with a few minor adaptations, mostly to increase propagation speed and convergence rate.</p> <div class="alert alert-block alert-warning"> <strong>Advanced Topic!</strong> This page covers the technical details of the internals of Serf. You don't need to know these details to effectively operate and use Serf. These details are documented here for those who wish to learn about them without having to go spelunking through the source code. </div> <h2 id=toc_1>SWIM Protocol Overview</h2> <p>Serf begins by joining an existing cluster or starting a new cluster. If starting a new cluster, additional nodes are expected to join it. New nodes in an existing cluster must be given the address of at least one existing member in order to join the cluster. The new member does a full state sync with the existing member over TCP and begins gossiping its existence to the cluster.</p> <p>Gossip is done over UDP with a configurable but fixed fanout and interval. This ensures that network usage is constant with regards to number of nodes. Complete state exchanges with a random node are done periodically over TCP, but much less often than gossip messages. This increases the likelihood that the membership list converges properly since the full state is exchanged and merged. The interval between full state exchanges is configurable or can be disabled entirely.</p> <p>Failure detection is done by periodic random probing using a configurable interval. If the node fails to ack within a reasonable time (typically some multiple of RTT), then an indirect probe is attempted. An indirect probe asks a configurable number of random nodes to probe the same node, in case there are network issues causing our own node to fail the probe. If both our probe and the indirect probes fail within a reasonable time, then the node is marked "suspicious" and this knowledge is gossiped to the cluster. A suspicious node is still considered a member of cluster. If the suspect member of the cluster does not disputes the suspicion within a configurable period of time, the node is finally considered dead, and this state is then gossiped to the cluster.</p> <p>This is a brief and incomplete description of the protocol. For a better idea, please read the <a href="//www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf">SWIM paper</a> in its entirety, along with the Serf source code.</p> <h2 id=toc_2>SWIM Modifications</h2> <p>As mentioned earlier, the gossip protocol is based on SWIM but includes minor changes, mostly to increase propogation speed and convergence rates.</p> <p>The changes from SWIM are noted here:</p> <ul> <li><p>Serf does a full state sync over TCP periodically. SWIM only propagates changes over gossip. While both are eventually consistent, Serf is able to more quickly reach convergence, as well as gracefully recover from network partitions.</p></li> <li><p>Serf has a dedicated gossip layer separate from the failure detection protocol. SWIM only piggybacks gossip messages on top of probe/ack messages. Serf uses piggybacking along with dedicated gossip messages. This feature lets you have a higher gossip rate (for example once per 200ms) and a slower failure detection rate (such as once per second), resulting in overall faster convergence rates and data propagation speeds.</p></li> <li><p>Serf keeps the state of dead nodes around for a set amount of time, so that when full syncs are requested, the requester also receives information about dead nodes. Because SWIM doesn't do full syncs, SWIM deletes dead node state immediately upon learning that the node is dead. This change again helps the cluster converge more quickly.</p></li> </ul> <h2 id=toc_3>Serf-Specific Messages</h2> <p>On top of the SWIM-based gossip layer, Serf sends some custom message types.</p> <p>Serf makes heavy use of <a href="//en.wikipedia.org/wiki/Lamport_timestamps">lamport clocks</a> to maintain some notion of message ordering despite being eventually consistent. Every message sent by Serf contains a lamport clock time.</p> <p>When a node gracefully leaves the cluster, Serf sends a <em>leave intent</em> through the gossip layer. Because the underlying gossip layer makes no differentiation between a node leaving the cluster and a node being detected as failed, this allows the higher level Serf layer to detect a failure versus a graceful leave.</p> <p>When a node joins the cluster, Serf sends a <em>join intent</em>. The purpose of this intent is solely to attach a lamport clock time to a join so that it can be ordered properly in case a leave comes out of order.</p> <p>For custom events, Serf sends a <em>user event</em> message. This message contains a lamport time, event name, and event payload. Because user events are sent along the gossip layer, which uses UDP, the payload and entire message framing must fit within a single UDP packet.</p> </div> </div> </div> <div id=footer> <div class=container> <div class=footer-links> <ul class="main-links nav navbar-nav rls-sb"> <li><a href="/intro/index.html">Intro</a></li> <li class=active><a href="/docs/index.html">Docs</a></li> <li><a href="/community.html">Community</a></li> </ul> <ul class="buttons nav navbar-nav rls-sb"> <li class=first><a href="/downloads.html">Download</a></li> <li><a href="https://github.com/hashicorp/serf">Github</a></li> </ul> </div> <div class=footer-logo> <span></span> </div> <div class="footer-hashi os"> <span>© 2013. A <a href="//www.hashicorp.com">HashiCorp</a> Project.</span> <a href="//www.hashicorp.com"><img src="/images/hashi-logo-s-3644fe63.png"></a> </div> </div> </div> <script src="javascripts/lib/d3.v3.min.js"></script> <script src="javascripts/app/deploy/site.js"></script> <script>
|
|
Serf.initialize();
|
|
</script> </body> </html> |