Kyle Havlovitz
9e1ba6fb4e
Make some final tweaks to autopilot package
2017-12-18 12:26:47 -08:00
Kyle Havlovitz
f347c8a531
More refactoring to make autopilot consul-agnostic
2017-12-12 17:46:28 -08:00
Kyle Havlovitz
8546a1d3c6
Move autopilot to a standalone package
2017-12-11 16:45:33 -08:00
James Phillips
3e7ea1931c
Moves the FSM into its own package.
...
This will help make it clearer what happens when we add some registration
plumbing for the different operations and snapshots.
2017-11-29 18:36:53 -08:00
James Phillips
36bb30e67a
Creates a registration mechanism for RPC endpoints.
2017-11-29 18:36:52 -08:00
James Phillips
d1ad538345
Makes RPC handling more robust when rolling servers. ( #3561 )
...
* Adds client-side retry for no leader errors.
This paves over the case where the client was connected to the leader
when it loses leadership.
* Adds a configurable server RPC drain time and a fail-fast path for RPCs.
When a server leaves it gets removed from the Raft configuration, so it will
never know who the new leader server ends up being. Without this we'd be
doomed to wait out the RPC hold timeout and then fail. This makes things fail
a little quicker while a sever is draining, and since we added a client retry
AND since the server doing this has already shut down and left the Serf LAN,
clients should retry against some other server.
* Makes the RPC hold timeout configurable.
* Reorders struct members.
* Sets the RPC hold timeout default for test servers.
* Bumps the leave drain time up to 5 seconds.
* Robustifies retries with a simpler client-side RPC hold.
* Reverts untended delete.
2017-10-10 15:19:50 -07:00
James Phillips
fcaa889116
Bumps default Raft protocol to version 3. ( #3477 )
...
* Changes default Raft protocol to 3.
* Changes numPeers() to report only voters.
This should have been there before, but it's more obvious that this
is incorrect now that we default the Raft protocol to 3, which puts
new servers in a read-only state while Autopilot waits for them to
become healthy.
* Fixes TestLeader_RollRaftServer.
* Fixes TestOperator_RaftRemovePeerByAddress.
* Fixes TestServer_*.
Relaxed the check for a given number of voter peers and instead do
a thorough check that all servers see each other in their Raft
configurations.
* Fixes TestACL_*.
These now just check for Raft replication to be set up, and don't
care about the number of voter peers.
* Fixes TestOperator_Raft_ListPeers.
* Fixes TestAutopilot_CleanupDeadServerPeriodic.
* Fixes TestCatalog_ListNodes_ConsistentRead_Fail.
* Fixes TestLeader_ChangeServerID and adjusts the conn pool to throw away
sockets when it sees io.EOF.
* Changes version to 1.0.0 in the options doc.
* Makes metrics test more deterministic with autopilot metrics possible.
2017-09-25 15:27:04 -07:00
Preetha Appan
8394ad08db
Introduce Code Policy validation via sentinel, with a noop implementation
2017-09-25 13:44:55 -05:00
Kyle Havlovitz
334e082848
Merge pull request #3431 from hashicorp/network-segments-oss
2017-09-01 10:24:58 -07:00
Kyle Havlovitz
ff994e9ade
Pass listeners into setupSegments
2017-08-31 17:56:43 -07:00
Kyle Havlovitz
5cc4b32a5d
Organize segments for a cleaner split between enterprise and OSS
2017-08-31 17:39:46 -07:00
Kyle Havlovitz
b77a0aa932
Fix some inconsistencies with segment logic and comments
2017-08-30 17:43:46 -07:00
Preetha Appan
0728a04dbb
Wire server provider for raft layer only on protocol version 3 and above, and update changelog
2017-08-30 14:36:47 -05:00
Kyle Havlovitz
6ded43131a
Add segment addr field to tags for LAN flood joiner
2017-08-30 11:58:29 -07:00
Kyle Havlovitz
107d7f6c5a
Add rpc_listener option to segment config
2017-08-30 11:58:29 -07:00
James Phillips
6a6eadd8c7
Adds open source side of network segments (feature is Enterprise-only).
2017-08-30 11:58:29 -07:00
Preetha Appan
e944370cde
More cleanup from code review
2017-08-30 12:31:36 -05:00
Preetha Appan
a215c764cd
Remove copy pasted duplicate line, update documentation.
2017-08-30 10:02:10 -05:00
Preetha Appan
5a29eb7486
Consolidate server lookup into one place and replace usages of localConsuls.
2017-08-30 09:30:33 -05:00
Preetha Appan
d8fe01db4c
Remove stray commented line
2017-08-30 09:30:33 -05:00
Preetha Appan
ca48e7e4c2
Remove server address tracking logic from manager/router and maintain it as part of lan event listener instead. Used sync.Map to track this, and added unit tests
2017-08-30 09:30:33 -05:00
Preetha Appan
b4a9d77d49
ServerAddressProvider interface also returns an error now
2017-08-30 09:30:33 -05:00
Preetha Appan
edb408bc22
Use config struct to create NetworkTransport layer when setting up raft
2017-08-30 09:30:33 -05:00
Preetha Appan
01f8e469aa
Implement AddressProvider and wire that up to raft transport layer to support server nodes changing their IP addresses in containerized environments
2017-08-30 09:30:33 -05:00
Frank Schroeder
d9e2a51887
agent: drop unused code
...
This code from http://github.com/hashicorp/consul/pull/3353 is no longer
required.
2017-08-22 00:02:46 +02:00
Frank Schroeder
c38dcf2d17
agent: move agent/consul/agent to agent/metadata
2017-08-09 14:36:52 +02:00
Frank Schroeder
85bdb77d90
agent: move agent/consul/servers to agent/router
2017-08-09 14:36:37 +02:00
Frank Schroeder
1d0bbfed9c
agent: move agent/consul/structs to agent/structs
2017-08-09 14:32:12 +02:00
Preetha Appan
6bac9355fd
Use sanitized version of node name of server in NS record, and start with "server" rather than "ns"
2017-08-07 11:11:55 +02:00
Preetha Appan
c38906daad
Add NS records and A records for each server. Constructs ns host names using the advertise address of the server.
2017-08-07 11:11:54 +02:00
James Phillips
803ed9a245
Adds secure introduction for the ACL replication token. ( #3357 )
...
Adds secure introduction for the ACL replication token, as well as a separate enable config for ACL replication.
2017-08-03 15:39:31 -07:00
Kyle Havlovitz
d985dbc36b
Add TLS setting to router areas
2017-07-14 17:38:08 -07:00
James Phillips
247f4a7e41
Simplifies Serf dynamic port selection code.
...
This isn't racy, it's just a little dirty. The listen will happen and a port
will be selected and injected into the config once the Serf instance is
created, so we don't need the retry loop here.
2017-07-07 09:22:34 +02:00
Frank Schroeder
b3189a566a
rpc: refactor sessionTimers and fix racy tests
...
The sessionTimers map was secured by a lock which wasn't used
properly in the tests. This lead to data races and failing tests
when accessing the length or the members of the map.
This patch adds a separate SessionTimers struct which is safe
for concurrent use and which ecapsulates the behavior of the
sessionTimers map.
2017-07-07 09:22:34 +02:00
Frank Schroeder
c33f7ecbe2
rpc: discover serf wan port before starting serf lan
...
When using dynamic ports for the serf clusters then
the actual bind port of the serf WAN cluster needs to
be discovered before the serf LAN cluster is started
since the serf LAN cluster announces the port of the WAN
cluster.
2017-07-07 09:22:34 +02:00
Frank Schroeder
84c90cbd07
rpc: bind rpc test server to port 0
2017-07-07 09:22:34 +02:00
Preetha Appan
ae656575ea
Rename to raftNotifyCh, fix typo
2017-07-06 09:10:36 -05:00
Preetha Appan
777504ff0e
Fixes deadlock between barrier write and leader notify channel read . Fixes #3230
2017-07-05 17:09:18 -05:00
Frank Schroeder
d3ab99244b
agent: make the RPC endpoint overwrite mechanism more transparent
...
This patch hides the RPC handler overwrite mechanism from the
rest of the code so that it works in all cases and that there
is no cooperation required from the tested code, i.e. we can
drop a.getEndpoint().
2017-06-21 05:42:39 +02:00
Frank Schroeder
2c47bc5d5b
agent: move conn pool for muxed connections into separate pkg
2017-06-21 05:42:39 +02:00
Frank Schroeder
e930b55f71
agent: move the SnapshotReplyFn out of the way
...
When splitting up the consul package into server and client
the SnapshotReplyFn needs to be in a separate package to avoid
a circular dependency.
2017-06-21 05:42:39 +02:00
Preetha Appan
bb559d8e6e
Minor fixes per code review
2017-06-20 19:43:07 -05:00
Preetha Appan
f616a8dd06
Code review feedback, fixed major logic bug
2017-06-16 10:49:54 -05:00
Preetha Appan
42d3a3f3db
Redo bug fix for stale reads on server startup, leveraging RPCHOldtimeout instead of maxQueryTime, plus tests
2017-06-15 22:41:30 -05:00
Frank Schroeder
cd837b0b18
pkg refactor
...
command/agent/* -> agent/*
command/consul/* -> agent/consul/*
command/agent/command{,_test}.go -> command/agent{,_test}.go
command/base/command.go -> command/base.go
command/base/* -> command/*
commands.go -> command/commands.go
The script which did the refactor is:
(
cd $GOPATH/src/github.com/hashicorp/consul
git mv command/agent/command.go command/agent.go
git mv command/agent/command_test.go command/agent_test.go
git mv command/agent/flag_slice_value{,_test}.go command/
git mv command/agent .
git mv command/base/command.go command/base.go
git mv command/base/config_util{,_test}.go command/
git mv commands.go command/
git mv consul agent
rmdir command/base/
gsed -i -e 's|package agent|package command|' command/agent{,_test}.go
gsed -i -e 's|package agent|package command|' command/flag_slice_value{,_test}.go
gsed -i -e 's|package base|package command|' command/base.go command/config_util{,_test}.go
gsed -i -e 's|package main|package command|' command/commands.go
gsed -i -e 's|base.Command|BaseCommand|' command/commands.go
gsed -i -e 's|agent.Command|AgentCommand|' command/commands.go
gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/commands.go
gsed -i -e 's|base\.||' command/commands.go
gsed -i -e 's|command\.||' command/commands.go
gsed -i -e 's|command|c|' main.go
gsed -i -e 's|range Commands|range command.Commands|' main.go
gsed -i -e 's|Commands: Commands|Commands: command.Commands|' main.go
gsed -i -e 's|base\.BoolValue|BoolValue|' command/operator_autopilot_set.go
gsed -i -e 's|base\.DurationValue|DurationValue|' command/operator_autopilot_set.go
gsed -i -e 's|base\.StringValue|StringValue|' command/operator_autopilot_set.go
gsed -i -e 's|base\.UintValue|UintValue|' command/operator_autopilot_set.go
gsed -i -e 's|\bCommand\b|BaseCommand|' command/base.go
gsed -i -e 's|BaseCommand Options|Command Options|' command/base.go
gsed -i -e 's|base.Command|BaseCommand|' command/*.go
gsed -i -e 's|c\.Command|c.BaseCommand|g' command/*.go
gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/*_test.go
gsed -i -e 's|base\.||' command/*_test.go
gsed -i -e 's|\bCommand\b|AgentCommand|' command/agent{,_test}.go
gsed -i -e 's|cmd.AgentCommand|cmd.BaseCommand|' command/agent.go
gsed -i -e 's|cli.AgentCommand = new(Command)|cli.Command = new(AgentCommand)|' command/agent_test.go
gsed -i -e 's|exec.AgentCommand|exec.Command|' command/agent_test.go
gsed -i -e 's|exec.BaseCommand|exec.Command|' command/agent_test.go
gsed -i -e 's|NewTestAgent|agent.NewTestAgent|' command/agent_test.go
gsed -i -e 's|= TestConfig|= agent.TestConfig|' command/agent_test.go
gsed -i -e 's|: RetryJoin|: agent.RetryJoin|' command/agent_test.go
gsed -i -e 's|\.\./\.\./|../|' command/config_util_test.go
gsed -i -e 's|\bverifyUniqueListeners|VerifyUniqueListeners|' agent/config{,_test}.go command/agent.go
gsed -i -e 's|\bserfLANKeyring\b|SerfLANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
gsed -i -e 's|\bserfWANKeyring\b|SerfWANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
gsed -i -e 's|\bNewAgent\b|agent.New|g' command/agent{,_test}.go
gsed -i -e 's|\bNewAgent|New|' agent/{acl_test,agent,testagent}.go
gsed -i -e 's|\bAgent\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bBool\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bConfig\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bDefaultConfig\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bDevConfig\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bMergeConfig\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bReadConfigPaths\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bParseMetaPair\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bSerfLANKeyring\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|\bSerfWANKeyring\b|agent.&|g' command/agent{,_test}.go
gsed -i -e 's|circonus\.agent|circonus|g' command/agent{,_test}.go
gsed -i -e 's|logger\.agent|logger|g' command/agent{,_test}.go
gsed -i -e 's|metrics\.agent|metrics|g' command/agent{,_test}.go
gsed -i -e 's|// agent.Agent|// agent|' command/agent{,_test}.go
gsed -i -e 's|a\.agent\.Config|a.Config|' command/agent{,_test}.go
gsed -i -e 's|agent\.AppendSliceValue|AppendSliceValue|' command/{configtest,validate}.go
gsed -i -e 's|consul/consul|agent/consul|' GNUmakefile
gsed -i -e 's|\.\./test|../../test|' agent/consul/server_test.go
# fix imports
f=$(grep -rl 'github.com/hashicorp/consul/command/agent' * | grep '\.go')
gsed -i -e 's|github.com/hashicorp/consul/command/agent|github.com/hashicorp/consul/agent|' $f
goimports -w $f
f=$(grep -rl 'github.com/hashicorp/consul/consul' * | grep '\.go')
gsed -i -e 's|github.com/hashicorp/consul/consul|github.com/hashicorp/consul/agent/consul|' $f
goimports -w $f
goimports -w command/*.go main.go
)
2017-06-10 18:52:45 +02:00