Commit graph

1167 commits

Author SHA1 Message Date
James Phillips fcaa889116 Bumps default Raft protocol to version 3. (#3477)
* Changes default Raft protocol to 3.

* Changes numPeers() to report only voters.

This should have been there before, but it's more obvious that this
is incorrect now that we default the Raft protocol to 3, which puts
new servers in a read-only state while Autopilot waits for them to
become healthy.

* Fixes TestLeader_RollRaftServer.

* Fixes TestOperator_RaftRemovePeerByAddress.

* Fixes TestServer_*.

Relaxed the check for a given number of voter peers and instead do
a thorough check that all servers see each other in their Raft
configurations.

* Fixes TestACL_*.

These now just check for Raft replication to be set up, and don't
care about the number of voter peers.

* Fixes TestOperator_Raft_ListPeers.

* Fixes TestAutopilot_CleanupDeadServerPeriodic.

* Fixes TestCatalog_ListNodes_ConsistentRead_Fail.

* Fixes TestLeader_ChangeServerID and adjusts the conn pool to throw away
sockets when it sees io.EOF.

* Changes version to 1.0.0 in the options doc.

* Makes metrics test more deterministic with autopilot metrics possible.
2017-09-25 15:27:04 -07:00
Frank Schröder 69a088ca85 New config parser, HCL support, multiple bind addrs (#3480)
* new config parser for agent

This patch implements a new config parser for the consul agent which
makes the following changes to the previous implementation:

 * add HCL support
 * all configuration fragments in tests and for default config are
   expressed as HCL fragments
 * HCL fragments can be provided on the command line so that they
   can eventually replace the command line flags.
 * HCL/JSON fragments are parsed into a temporary Config structure
   which can be merged using reflection (all values are pointers).
   The existing merge logic of overwrite for values and append
   for slices has been preserved.
 * A single builder process generates a typed runtime configuration
   for the agent.

The new implementation is more strict and fails in the builder process
if no valid runtime configuration can be generated. Therefore,
additional validations in other parts of the code should be removed.

The builder also pre-computes all required network addresses so that no
address/port magic should be required where the configuration is used
and should therefore be removed.

* Upgrade github.com/hashicorp/hcl to support int64

* improve error messages

* fix directory permission test

* Fix rtt test

* Fix ForceLeave test

* Skip performance test for now until we know what to do

* Update github.com/hashicorp/memberlist to update log prefix

* Make memberlist use the default logger

* improve config error handling

* do not fail on non-existing data-dir

* experiment with non-uniform timeouts to get a handle on stalled leader elections

* Run tests for packages separately to eliminate the spurious port conflicts

* refactor private address detection and unify approach for ipv4 and ipv6.

Fixes #2825

* do not allow unix sockets for DNS

* improve bind and advertise addr error handling

* go through builder using test coverage

* minimal update to the docs

* more coverage tests fixed

* more tests

* fix makefile

* cleanup

* fix port conflicts with external port server 'porter'

* stop test server on error

* do not run api test that change global ENV concurrently with the other tests

* Run remaining api tests concurrently

* no need for retry with the port number service

* monkey patch race condition in go-sockaddr until we understand why that fails

* monkey patch hcl decoder race condidtion until we understand why that fails

* monkey patch spurious errors in strings.EqualFold from here

* add test for hcl decoder race condition. Run with go test -parallel 128

* Increase timeout again

* cleanup

* don't log port allocations by default

* use base command arg parsing to format help output properly

* handle -dc deprecation case in Build

* switch autopilot.max_trailing_logs to int

* remove duplicate test case

* remove unused methods

* remove comments about flag/config value inconsistencies

* switch got and want around since the error message was misleading.

* Removes a stray debug log.

* Removes a stray newline in imports.

* Fixes TestACL_Version8.

* Runs go fmt.

* Adds a default case for unknown address types.

* Reoders and reformats some imports.

* Adds some comments and fixes typos.

* Reorders imports.

* add unix socket support for dns later

* drop all deprecated flags and arguments

* fix wrong field name

* remove stray node-id file

* drop unnecessary patch section in test

* drop duplicate test

* add test for LeaveOnTerm and SkipLeaveOnInt in client mode

* drop "bla" and add clarifying comment for the test

* split up tests to support enterprise/non-enterprise tests

* drop raft multiplier and derive values during build phase

* sanitize runtime config reflectively and add test

* detect invalid config fields

* fix tests with invalid config fields

* use different values for wan sanitiziation test

* drop recursor in favor of recursors

* allow dns_config.udp_answer_limit to be zero

* make sure tests run on machines with multiple ips

* Fix failing tests in a few more places by providing a bind address in the test

* Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder.

* Add porter to server_test.go to make tests there less flaky

* go fmt
2017-09-25 11:40:42 -07:00
Frank Schroeder 78ce956385
Update docs for addr_type option for AWS Cloud auto-join (#3471)
Fixes #3471
2017-09-25 09:54:58 +02:00
Frank Schroeder bcfe1190d4
Fix Azure cloud auto-join docs (#3466)
Fixes #3466
2017-09-25 02:44:04 +02:00
Michael Stewart 1040dbb2d4 Fix docs/guides/segements sidebar selection. 2017-09-19 16:45:39 -05:00
Preetha Appan df742843a4
Adds documentation for Sentinel integration in Consul Enterprise. 2017-09-19 09:02:53 -05:00
James Phillips 1906462eec Merge pull request #3456 from hashicorp/gossip-fix
Adds gossip keys to network segment memberlist configs.
2017-09-07 12:27:34 -07:00
James Phillips 02a3f3f27b
Shows the segment name in the keyring API and command output. 2017-09-07 12:17:39 -07:00
James Phillips 402ebe53ef Update outage.html.md 2017-09-06 21:19:46 -07:00
James Phillips 87eeec254f Update segments.html.markdown.erb 2017-09-06 16:42:13 -07:00
Kevin Fishner e70b8cc820 add newtork segments overview page 2017-09-04 15:44:13 -07:00
James Phillips f8a5476a3c Update options.html.md 2017-09-01 22:25:28 -07:00
James Phillips 76cd7963e5 Update maint.html.markdown.erb 2017-09-01 17:51:02 -07:00
phanidileep df4e77e179 Updating maint.html to remove node maint (#3354)
* Updating maint.html to remove node maint

as discussed in #3268 maint mode does not have any affect on the node health.

* Update maint.html.markdown.erb
2017-09-01 17:49:51 -07:00
James Phillips 1a64583c98 Fixes a bad link. 2017-09-01 15:07:18 -07:00
James Phillips bc9780baad Adds simple rate limiting for client agent RPC calls to Consul servers. (#3440)
* Added rate limiting for agent RPC calls.
* Initializes the rate limiter based on the config.
* Adds the rate limiter into the snapshot RPC path.
* Adds unit tests for the RPC rate limiter.
* Groups the RPC limit parameters under "limits" in the config.
* Adds some documentation about the RPC limiter.
* Sends a 429 response when the rate limiter kicks in.
* Adds docs for new telemetry.
* Makes snapshot telemetry look like RPC telemetry and cleans up comments.
2017-09-01 15:02:50 -07:00
James Phillips 20fcfe866e
Tweaks network segments guide. 2017-09-01 11:19:39 -07:00
Kyle Havlovitz 334e082848 Merge pull request #3431 from hashicorp/network-segments-oss 2017-09-01 10:24:58 -07:00
Kyle Havlovitz 5605d735a1
Update segment docs 2017-08-31 17:39:57 -07:00
Kyle Havlovitz 02c35fe0ba
Add doc sections for network segments 2017-08-31 11:19:08 -07:00
David Terry 5d891ba7e1 docs: add quotes around example invocation for google cloud retry join 2017-08-30 23:05:45 +02:00
Frank Schroeder 2190a4fb71
doc: document that retry-join values are URL encoded
Fixes #3417
2017-08-26 15:19:43 +02:00
Frank Schroeder b7c3c8756d
doc: fix parameter name for aws retry-join
Fixes #3417
2017-08-26 15:19:34 +02:00
Frank Schroeder 0e246054ef doc: document cloud auto-joining for retry-join-wan 2017-08-23 21:23:34 +02:00
Kevin Bidwell 8a53f556f9 Added configuration instructions for forwarding DNS queries from Unbound to consul. 2017-08-18 08:45:43 -06:00
Frank Schröder fd4bf4070e doc: retry_join is a string array (#3388) 2017-08-10 09:58:26 +02:00
James Phillips 48b1afeb26 Removes partial details in the retry_join config file section. (#3386) 2017-08-09 21:27:17 -07:00
Seth Vargo b6f4df4b03 Document the new auto-join in the config and CLI (#3381)
* Document the new auto-join in the config and CLI
* Mention and example DNS
2017-08-09 21:14:56 -07:00
Preetha Appan 00a5eb9071 Add note about configuring recursors to be able to resolve external services. 2017-08-09 11:13:30 -05:00
Kyle Havlovitz 160395d3c7
Add doc links for metrics endpoint 2017-08-08 13:05:38 -07:00
Kyle Havlovitz 975ded2714
Add support for labels/filters from go-metrics 2017-08-08 01:45:10 -07:00
Nicolae Vlădescu 8be48291da Add a more elegant alternative to the reap option
Add a more elegant alternative to the reap option using the native built-in docker init process
2017-08-07 11:49:40 +03:00
James Phillips 31676bba76
Adds a note about not replicating data to FAQ and federation-related spots. 2017-08-04 16:14:39 -07:00
James Phillips cdd2c75df6
Fixes a bunch of borken list nesting in the options doc. 2017-08-03 17:13:22 -07:00
James Phillips 803ed9a245 Adds secure introduction for the ACL replication token. (#3357)
Adds secure introduction for the ACL replication token, as well as a separate enable config for ACL replication.
2017-08-03 15:39:31 -07:00
Preetha Appan 4c0c912a52 Fix incorrect punctuation 2017-08-03 16:40:22 -05:00
Frank Schroeder d18141eb4b
doc: add retry-join docs for Softlayer 2017-08-03 21:35:30 +02:00
James Phillips c31b56a03e Adds a new /v1/acl/bootstrap API (#3349) 2017-08-02 17:05:18 -07:00
preetapan f6e0f5dc4b Merge pull request #3347 from hashicorp/containers_guide
Adds a containers guide to the documentation section.
2017-08-01 16:43:41 -05:00
Preetha Appan a708123164 Minor tweaks, fixed spacing issue with docker run examples 2017-08-01 16:38:34 -05:00
Preetha Appan ff4009bed2 Address more review comments 2017-08-01 15:25:13 -05:00
Preetha Appan bde197e161 Added links to new containers guide to navbar and index pages 2017-08-01 11:48:25 -05:00
Preetha Appan b5fc169e2c Added a section on configuration 2017-08-01 11:36:34 -05:00
Preetha Appan ec52d0036a More tweaks and grammar corrections 2017-08-01 11:13:24 -05:00
Preetha Appan d152b31364 More review feedback 2017-08-01 10:38:43 -05:00
Frank Schroeder 68e8f3d0f7 agent: use github.com/hashicorp/go-discover
Replace the provider specific node discovery code
with go-discover to support AWS, Azure and GCE.

Fixes #3282
2017-08-01 11:41:43 +02:00
Preetha Appan 7ab4255231 Addressed review comments 2017-07-31 19:56:42 -05:00
Preetha Appan a12cfe8918 Capitalize Consul and Docker throughout 2017-07-31 17:54:03 -05:00
Preetha Appan e4de25367d First pass at a consul containers guide 2017-07-31 17:19:15 -05:00
James Phillips 9b70009ab8 command/lock: Add -child-exitcode, return 2 on child error (#3329)
* Exit 2 if -child-exit-code and the child returned with an error.
* There is no platform independent way to check the exact return code of
* the child, so on error always return 2.
* Closes #947
* Closes #1503
2017-07-26 22:09:19 -07:00
James Phillips 005c9ad12a Update geo-failover.html.md 2017-07-26 16:42:35 -07:00
James Phillips cafe110669 Update geo-failover.html.md 2017-07-26 15:44:17 -07:00
James Phillips fe0f0285a9 Adds a geo failover guide using prepared queries. (#3328) 2017-07-26 15:40:01 -07:00
James Phillips 6e794ea1b3 Adds support for agent-side ACL token management via API instead of config files. (#3324)
* Adds token store and removes all runtime use of config for ACL tokens.
* Adds a new API for changing agent tokens on the fly.
2017-07-26 11:03:43 -07:00
James Phillips 923119a7d8 Update options.html.md 2017-07-24 22:37:12 -07:00
Jon Benson 623d4203c7 Fix spelling (#3311) 2017-07-20 14:48:45 -07:00
James Phillips 6e251d39b5 Updates consul exec docs to mention ACLs. (#3308)
* Update documentation for consul exec to mention all permissions it needs.

* Removed node write perm, not needed for consul exec

* Converts ACL note into a table like the API docs, adds ACL guide x-ref.
2017-07-20 09:30:08 -07:00
James Phillips 2d1111479e Clarifies endpoint blocking in the docs. 2017-07-19 17:14:19 -07:00
James Phillips 31ac5c45e1 Tweaks title. 2017-07-18 14:48:38 -07:00
Kyle Havlovitz 73ec6541f6
Add UpgradeVersionTag information to docs 2017-07-18 14:01:21 -07:00
James Phillips 30316a9a54 Update acl.html.md 2017-07-18 07:44:35 -07:00
James Phillips 8058f1b234 Improves structure of ACL guide. 2017-07-18 07:41:59 -07:00
James Phillips 52c69bbf17 Removes a merge conflict marker from the ACL guide. 2017-07-18 07:35:17 -07:00
James Phillips 3984530d63 Update upgrade-specific.html.md 2017-07-18 07:14:16 -07:00
James Phillips 0c57f90fd7 Update upgrade-specific.html.md 2017-07-18 07:11:59 -07:00
James Phillips fe7a24c846 Adds a missing newline to the snapshot agent page. 2017-07-17 22:22:50 -07:00
Charlie Voiselle 1d8c78645d Removed localhost from links (#3289)
Two links were still referring to localhost. Made them relative and tested.
2017-07-17 19:59:58 -07:00
James Phillips adc991fa5a Update upgrade-specific.html.md 2017-07-17 14:11:08 -07:00
James Phillips 788dd255a1 Adds new config to make script checks opt-in, updates documentation. (#3284) 2017-07-17 11:20:35 -07:00
James Phillips f7629a4f66 Fixes a broken link. 2017-07-17 00:07:34 -07:00
James Phillips 838591c916 Changes remote exec KV read to call GetTokenForAgent(). (#3283)
* Changes remote exec KV read to call GetTokenForAgent(), which can use
the acl_agent_token instead of the acl_token.

Fixes #3160.

* Fixes remote exec unit test with ACLs.

* Adds unhappy ACL path to unit tests for remote exec.
2017-07-16 21:12:16 -07:00
James Phillips 5876b81896 Adds node read privileges to the acl_agent_master_token. (#3277)
Fixes #3113.
2017-07-16 20:08:26 -07:00
Kyle Havlovitz 6ea05706a1
Add network area TLS setting to docs 2017-07-14 17:06:33 -07:00
Seth Vargo a2ba7071e9 Add catalog CLI functions (#3204) 2017-07-14 12:45:08 -07:00
James Phillips d63bf807a8 Update acl.html.md 2017-07-13 22:40:15 -07:00
James Phillips 8572931afe Cleans up version 8 ACLs in the agent and the docs. (#3248)
* Moves magic check and service constants into shared structs package.

* Removes the "consul" service from local state.

Since this service is added by the leader, it doesn't really make sense to
also keep it in local state (which requires special ACLs to configure), and
requires a bunch of special cases in the local state logic. This requires
fewer special cases and makes ACL bootstrapping cleaner.

* Makes coordinate update ACL log message a warning, similar to other AE warnings.

* Adds much more detailed examples for bootstrapping ACLs.

This can hopefully replace https://gist.github.com/slackpad/d89ce0e1cc0802c3c4f2d84932fa3234.
2017-07-13 22:33:47 -07:00
James Phillips a66d61622a Adds a version note about block_endpoints config. 2017-07-10 13:56:02 -07:00
James Phillips 68991da95f Adds the ability to blacklist specific HTTP endpoints. (#3252) 2017-07-10 13:51:25 -07:00
James Phillips 197394b156 Update autopilot.html.md 2017-07-07 14:38:35 -07:00
Seth Rutner 781b42e583 Update autopilot.html.md (#3244) 2017-07-07 14:34:59 -07:00
Brian Shumate bb18ffab27 DOCS: Add note in Ports Used section (#3241)
- Add note about port 8302 to address #3058
2017-07-07 09:38:56 -07:00
Chris Griggs 38f92aa4e8 Update # of ports (#3238)
I counted twice.
2017-07-06 13:50:12 -07:00
James Phillips 3388daf479 Adds links to Lifeguard info. 2017-07-06 12:19:39 -07:00
Frank Schroeder 913748bcc4
Revert "agent: add allowStale option for HTTP API (#3142)"
This reverts commit 1e0fd27a74f5b18775ce91a84310430de35a4a80.
2017-06-27 07:04:55 +02:00
James Phillips a8d90ccb2c Adds missing tls_skip_verify option to HTTP check example. 2017-06-24 15:37:52 -07:00
James Phillips 884a82b61a Update options.html.md 2017-06-24 15:28:23 -07:00
James Phillips 2184136284 Changes host-based node IDs from opt-out to opt-in. (#3187) 2017-06-24 09:36:53 -07:00
wojtkiewicz f320bb9083
agent: add allowStale option for HTTP API (#3142)
This patch adds an "allowStale" option to the HTTP API
configuration which allows stale reads to provide linear
read scalability.

Fixes #3142
2017-06-22 10:31:13 +02:00
wojtkiewicz 26c8697a40
agent: add "http_config"
This patch adds an "http_config" object to the config file
and moves the "http_api_response_headers" option there.

"http_api_response_headers" is now deprecated in favor of
"http_config.response_headers"
2017-06-22 10:31:11 +02:00
Preetha Appan 5bc9edb4ec Update documentation and json examples for snapshot agent 2017-06-21 21:15:02 -05:00
Frank Schröder 04b636d1f4 agent: notify systemd after JoinLAN (#2121)
This patch adds support for notifying systemd via the
NOTIFY_SOCKET by sending 'READY=1' to the socket after
a successful JoinLAN.

Fixes #2121
2017-06-21 06:43:55 +02:00
preetapan 9e527836be Merge pull request #3154 from hashicorp/issue_2644_redux
Fix stale reads on server startup. Consistent reads will now wait for up to config.RPCHoldTimeout for the server to get past its raft log, before returning an error. Servers that are starting up will eventually catch up. 
This fixes issue #2644
2017-06-20 19:47:12 -05:00
Kyle Havlovitz d52a0b2909 Update encryption.html.md 2017-06-16 12:54:25 -07:00
Preetha Appan 42d3a3f3db Redo bug fix for stale reads on server startup, leveraging RPCHOldtimeout instead of maxQueryTime, plus tests 2017-06-15 22:41:30 -05:00
Kyle Havlovitz 5e45aec642 Add an option to disable keyring file (#3145)
Also disables keyring file in dev mode.
2017-06-15 15:24:04 -07:00
Preetha Appan 59e2592872 Added documentation for new s3 server side encryption 2017-06-08 11:39:10 -07:00
James Phillips 536915fa9a Fixes version reference in encryption docs. 2017-06-07 21:31:48 -07:00
James Phillips a50a6289bb Merge pull request #3118 from katiebayes/master
Add enterprise badges and update middleman version
2017-06-07 10:41:57 -07:00
Frank Schröder e206ab5cb6 agent: support custom header and method for http checks (#3106)
This patch adds support for custom headers and
method for HTTP checks.

Fixes #2474
Fixes #2657
Fixes #3106
2017-06-07 01:11:56 +02:00
Katie Bayes 714f46f34b add enterprise badge style alert boxes, change guides/areas.html.md to erb file 2017-06-04 01:30:21 -04:00