Commit Graph

313 Commits

Author SHA1 Message Date
Paul Banks 251da1077f xDS Server Implementation (#4731)
* Vendor updates for gRPC and xDS server

* xDS server implementation for serving Envoy as a Connect proxy

* Address initial review comments

* consistent envoy package aliases; typos fixed; override TLS and authz for custom listeners

* Moar Typos

* Moar typos
2018-10-10 16:55:34 +01:00
Mitchell Hashimoto 9846999505
vendor: update mapstructure to v1.1.0
We require this change to support struct to struct decoding.
2018-09-30 19:15:40 -07:00
Matt Keeler ba4f912b25
Update Raft Vendoring (#4539)
Pulls in a fix for a potential memory leak regarding consistent reads that invoke VerifyLeader.
2018-09-06 15:07:42 -04:00
Mitchell Hashimoto f7a95e1a28
vendor k8s client lib 2018-09-05 14:59:02 -07:00
Mitchell Hashimoto 144b7efa51
Update go-discover vendor 2018-09-05 13:31:10 -07:00
Shubheksha 1afcabb0a2 replace old fork of text package (#4501) 2018-08-14 12:23:18 -07:00
Paul Banks 3adfe86f03 Update Serf and memberlist (#4511)
This includes fixes that improve gossip scalability on very large (> 10k node) clusters.

The Serf changes:
 - take snapshot disk IO out of the critical path for handling messages hashicorp/serf#524
 - make snapshot compaction much less aggressive - the old fixed threshold caused snapshots to be constantly compacted (synchronously with request handling) on clusters larger than about 2000 nodes! hashicorp/serf#525

Memberlist changes:
 - prioritize handling alive messages over suspect/dead to improve stability, and handle queue in LIFO order to avoid acting on info that 's already stale in the queue by the time we handle it. hashicorp/memberlist#159
 - limit the number of concurrent pushPull requests being handled at once to 128. In one test scenario with 10s of thousands of servers we saw channel and lock blocking cause over 3000 pushPulls at once which ballooned the memory of the server because each push pull contained a de-serialised list of all known 10k+ nodes and their tags for a total of about 60 million objects and 7GB of memory stuck. While the rest of the fixes here should prevent the same root cause from blocking in the same way, this prevents any other bug or source of contention from allowing pushPull messages to stack up and eat resources. hashicorp/memberlist#158
2018-08-09 13:16:13 -04:00
Siva Prasad a5ebab63e7 Vendoring update for go-discover. (#4412)
* New Providers added and updated vendoring for go-discover

* Vendor.json formatted using make vendorfmt

* Docs/Agent/auto-join: Added documentation for the new providers introduced in this PR

* Updated the golang.org/x/sys/unix in the vendor directory

* Agent: TestGoDiscoverRegistration updated to reflect the addition of new providers

* Deleted terraform.tfstate from vendor.

* Deleted terraform.tfstate.backup

Deleted terraform state file artifacts from unknown runs.

* Updated x/sys/windows vendor for Windows binary compilation
2018-07-25 16:21:04 -07:00
Matt Keeler 9757a6fb62 Vendor golang.org/x/sys/windows/svc 2018-07-12 11:29:57 -04:00
mkeeler 1da3c42867 Merge remote-tracking branch 'connect/f-connect' 2018-06-25 19:42:51 +00:00
Matt Keeler bc7e9b6fd4 Remove build tags from vendored vault file to allow for this to merge properly into enterprise 2018-06-25 12:26:10 -07:00
Matt Keeler 2f90768662 Vendor the vault api 2018-06-25 12:26:10 -07:00
Paul Banks 86a55892fd Remove go-diff vendor as assert.JSONEq output is way better for our case 2018-06-25 12:25:39 -07:00
Leo Zhang b498816e80
Fix invalid vendor.json syntax for go-discover 2018-06-15 02:02:12 -07:00
Kyle Havlovitz 80b6d0a6cf
Add missing vendor dep github.com/stretchr/objx 2018-06-14 09:42:13 -07:00
Matt Keeler 33148f482d Remove bogus second yamux vendoring 2018-06-04 16:28:33 -04:00
Matt Keeler 1e485ed727 Update yamux vendoring
Pulls in logging fixes.
2018-06-04 16:02:50 -04:00
Jack Pearkes c4112f2b9a
Merge pull request #4013 from sethvargo/sethvargo/user_agent
Add a helper for generating Consul's user-agent string
2018-06-01 09:13:38 -07:00
Matt Keeler 1c577b2012
Merge pull request #4131 from pierresouchay/enable_full_dns_compression
Enable full dns compression
2018-06-01 10:42:03 -04:00
Seth Vargo 5911fd5344
Update vendor for go-discover 2018-05-25 15:52:05 -04:00
Wim e8d0474a8e Add github.com/coredns/coredns/plugin/pkg/dnsutil files 2018-05-21 22:25:16 +02:00
Wim 9565b5415b Add github.com/coredns/coredns/plugin/pkg/dnsutil to vendor.json 2018-05-21 22:18:19 +02:00
Pierre Souchay 61e7d06174 Bump DNS lib to 1.0.7 with 14bits Len() fix 2018-05-16 10:52:51 +02:00
Matt Keeler 4d2a0308e8 Fix vendoring of two missed libs 2018-05-11 11:31:42 -04:00
Matt Keeler 586c91e8ea Update prometheus indirect deps 2018-05-11 11:18:15 -04:00
Matt Keeler ba376bcd2b Update the various deps of miekg/dns in our vendor.json 2018-05-11 10:52:05 -04:00
Matt Keeler 7928af61f2 Pull in miekg/dns deps on the golang crypto ed25519 packages 2018-05-11 10:31:27 -04:00
Kyle Havlovitz 7cd7f4acd7
vendor: pull in latest version of go-discover 2018-05-10 15:40:16 -07:00
Preetha Appan 98a04a0af9
Update serf to pick up clean leave fix 2018-05-04 15:51:55 -05:00
Paul Banks 06e1a62653
Merge pull request #4016 from pierresouchay/support_for_prometheus
Support for prometheus for metrics endpoint
2018-04-24 16:14:43 +01:00
Mitchell Hashimoto 17af5e6a24
vendor: add hashstructure and mock 2018-04-19 08:10:05 -07:00
Pierre Souchay e1c64f70df Added dependency github.com/prometheus/client_golang/prometheus/promhttp 2018-04-06 08:54:37 +02:00
Pierre Souchay 09a7546b12 Bump github.com/armon/go-metrics to allow having prometheus support 2018-04-05 18:21:32 +02:00
Yoann 23a6ad9356 Add support for compression in http api
The need has been spotted in issue https://github.com/hashicorp/consul/issues/3687.
Using "NYTimes/gziphandler", the http api responses can now be compressed if required.
The Go API requires compressed response if possible and handle the compressed response.
We here change only the http api (not the UI for instance).
2018-04-03 22:33:13 +02:00
Paul Banks e8372918a6
Actually add the `require` vendored files I intended to add in 0d5600ff60
Note that the vendor.json is already correct but the actual files were never checked in so report as missing:

```
$ govendor list | grep testify
 v  github.com/stretchr/testify/assert
  m github.com/stretchr/testify/require
```
2018-03-29 17:05:11 +01:00
Preetha Appan c115a2d7ba
vendorfmt 2018-03-28 10:25:49 -05:00
Pierre Souchay ab9dd18bec
Bump version of miekg/dns to 1.0.4
See https://github.com/hashicorp/consul/issues/3977

While trying to improve furthermore #3948 (This pull request is still valid since we are not using Compression to compute the result anyway).

I saw a strange behaviour of dns library.
Basically, msg.Len() and len(msg.Pack()) disagree on Message len.

Thus, calculation of DNS response is false consul relies on msg.Len() instead of the result of Pack()

This is linked to miekg/dns#453 and a fix has been provided with miekg/dns#454

Would it be possible to upgrade miekg/dns to a more recent function ?

Consul might for instance upgrade to a post 1.0 release such as https://github.com/miekg/dns/releases/tag/v1.0.4
2018-03-28 10:23:57 -05:00
Paul Banks eed1ef851b
Add vendored `testify/require` subpackage; upgrade `assert` to match. (#3986) 2018-03-27 15:19:15 +01:00
Preetha Appan 16962cccfa
Update yamux to pick up performance improvements 2018-03-26 08:56:40 -05:00
Mitchell Hashimoto fbac58280e
agent/consul/fsm: begin using testify/assert 2018-03-06 09:48:15 -08:00
Alvin Huang 6bc9f6844f remove old pkgs and put deps of missing packages in vendor.json 2018-02-23 17:08:24 -05:00
James Phillips 37cf6583db
Fixes a panic on TCP-based DNS lookups.
This came in via the monkey patch in #3861.

Fixes #3877
2018-02-08 17:57:41 -08:00
Preetha 1665908bed
Fix panic in azure go discover provider (#3876) 2018-02-08 16:46:33 -06:00
Preetha c460537b2a
Patch dns vendor code for picking up a TCP DOS attack bugfix (#3861) 2018-02-05 17:27:45 -06:00
James Phillips 0aa05cc5f0
Merge pull request #3855 from hashicorp/pr-3782-slackpad
Adds support for gRPC health checks.
2018-02-02 17:57:27 -08:00
James Phillips 2e660c7172
Updates hashicorp/go-discover to pull in support for Azure Virtual Machine Scale Sets. 2018-01-19 16:24:08 -08:00
James Phillips 503e01774b
Updates Serf to pickup fix for spammy zero RTT log messages.
Fixes #3789.
2018-01-19 14:47:12 -08:00
Dmytro Kostiuchenko a45f6ad740 Add gRPC health-check #3073 2018-01-04 16:42:30 -05:00
James Phillips ffb81aaba5
Updates go-discover to get monkey patch for golang.org/x/net/trace. 2018-01-03 13:22:42 -08:00
James Phillips 63274429ff
Updates hashicorp/go-cleanhttp to pick up new sanitizer. 2017-12-20 13:48:49 -08:00
James Phillips 7cc034dc13
Merge pull request #3646 from posener/update-complete
vendor: update complete version to v1.1
2017-12-14 17:51:33 -08:00
James Phillips 3ac47e3b1c
Updates Serf to pull in new queue depth controls. 2017-12-06 17:06:08 -08:00
James Phillips 68b698993f
Updated memberlist to fix negative RTT measurements.
Fixes #3704
2017-11-21 01:37:49 -08:00
James Phillips b166ce6969
Updates memberlist to pick up https://github.com/hashicorp/memberlist/pull/69.
Fixes #3671
2017-11-10 09:31:02 -08:00
Eyal Posener 2948fa9296 vendor: update complete version to v1.1
Update posener/complete to revision=v1.1.
Leave only once occurrence of posener/complete in vendor, there was an occurrence for
each package individualy.

The formatting of vendor/vendor.json has changed after using
the command "govendor fetch github.com/posener/complete@=v1.1"
2017-11-08 06:45:56 +02:00
James Phillips 8709f65afd
Adds HTTP/2 support to Consul's HTTPS server. (#3657)
* Refactors the HTTP listen path to create servers in the same spot.

* Adds HTTP/2 support to Consul's HTTPS server.

* Vendors Go HTTP/2 library and associated deps.
2017-11-07 15:06:59 -08:00
Frank Schröder bdb8c8f73e vendor: update github.com/hashicorp/go-sockaddr (#3633)
Pull in changes for

 * hashicorp/go-sockaddr#12
 * hashicorp/go-sockaddr#13
 * hashicorp/go-sockaddr#14
 * hashicorp/go-sockaddr#16
2017-10-31 17:05:57 -07:00
Frank Schröder 56561523cf vendor: update go-discover (#3634)
* vendor: update go-discover

Pull in providers:

 * Aliyun (Alibaba Cloud)
 * Digital Ocean
 * OpenStack (os)
 * Scaleway

* doc: use ... instead of xxx

* doc: strip trailing whitespace

* doc: add docs for aliyun, digitalocean, os and scaleway

* agent: fix test
2017-10-31 17:03:54 -05:00
Alex Dadgar 791738a3a7 format vendor 2017-10-23 16:31:44 -07:00
Alex Dadgar af09d86892 Make freeport testing friendly
This PR allows the caller to decide if they would like to have the
calling test fail, have the caller panic on error, or handle the errors
themselves.
2017-10-23 16:28:02 -07:00
Preetha Appan ad8b9171d6 Update serf library to pick up coordinate persistence fix 2017-10-21 21:19:43 -05:00
Preetha Appan d206639f83 REbase master serf 2017-10-20 10:33:59 -05:00
Preetha Appan 09c51e3119 Vendor update serf to pick up fix for out of range ping periods in coordinate subsystem 2017-10-20 10:14:15 -05:00
Matt McQuillan 63bd7cb687 Updating go-checkpoint lib to have a fixed timeout (#3564)
* Updating go-checkpoint lib to have a fixed timeout

* formatting vendor/vendor.json file per project spec
2017-10-17 17:01:23 -07:00
Frank Schroeder f8eb4006fa revert monkey patch for mitchellh/cli 2017-10-18 00:08:45 +02:00
Frank Schroeder ae76c975ae vendor: update github.com/mitchellh/mapstructure to replace monkey patch 2017-10-17 20:43:11 +02:00
Frank Schroeder 1a573e8fba vendor: update github.com/hashicorp/hcl to replace monkey patch 2017-10-17 20:43:11 +02:00
preetapan cb313a264c Merge pull request #3548 from hashicorp/go_retryablehttp_update
Update go-retryablehttp
2017-10-06 13:44:26 -05:00
Preetha Appan 10b7ab252f Update go-retryablehttp 2017-10-06 13:42:11 -05:00
preetapan 84b0af3150 Vendor update miekg/dns to pick up bugfixes and features (#3547) 2017-10-06 11:34:41 -07:00
Frank Schroeder 4e866ced39 command: backport patch for mitchellh/cli help flag handling (#3536)
This patch backports a fix which will show the correct usage screen for
command line flags.

This is considered a temporary fix until the code has been refactored.
Newer versions of the cli library require that the flag set is populated
when Help() is called or that it is populated within Help() itself.

Fixes #3536
2017-10-05 17:29:45 +02:00
Frank Schroeder 3d4fa7f65d Update github.com/hashicorp/go-discover to pull in new config parser
This patch updates the go-discover library to use the new config parser
which uses a different encoding scheme for the go-discover config DSL.
Values are no longer URL encoded but taken literally unless they contain
spaces, backslashes or double quotes. To support keys or values with
these special characters the string needs to be double quoted and usual
escaping rules apply.

Fixes #3417
2017-10-04 19:12:28 +02:00
Preetha Appan ae08f17c11 Updates vendor directory for raft to record raft v1.0.0. No functional changes 2017-10-03 17:19:10 -05:00
Frank Schröder cda0eacff1 Recursive sanitize (#3505)
* vendor: add github.com/sergi/go-diff/diffmatchpatch for diff'ing test output

* config: refactor Sanitize to recursively clean runtime config and format complex fields

* Removes an extra int cast.

* Adds a top-level check test case for sanitization.
2017-09-27 11:47:40 -07:00
James Phillips 123104b24d
Updates BoltDB to 1.3.1. 2017-09-26 14:53:01 -07:00
Frank Schroeder 4a98b26537
Revert monkey patch since it is not clear whether this is an issue at all. 2017-09-26 13:42:32 +02:00
Frank Schroeder 1530f12f56
serf: monkey patch https://github.com/hashicorp/serf/pull/486 2017-09-26 13:40:26 +02:00
Frank Schroeder 20a58ab272
Monkey patch mitchellh/mapstructure with code from https://github.com/mitchellh/mapstructure/pull/101 2017-09-26 12:42:49 +02:00
Frank Schroeder 3396f11cfd
Replace monkey patch with code from https://github.com/hashicorp/hcl/pull/213 2017-09-26 12:42:03 +02:00
Frank Schröder 69a088ca85 New config parser, HCL support, multiple bind addrs (#3480)
* new config parser for agent

This patch implements a new config parser for the consul agent which
makes the following changes to the previous implementation:

 * add HCL support
 * all configuration fragments in tests and for default config are
   expressed as HCL fragments
 * HCL fragments can be provided on the command line so that they
   can eventually replace the command line flags.
 * HCL/JSON fragments are parsed into a temporary Config structure
   which can be merged using reflection (all values are pointers).
   The existing merge logic of overwrite for values and append
   for slices has been preserved.
 * A single builder process generates a typed runtime configuration
   for the agent.

The new implementation is more strict and fails in the builder process
if no valid runtime configuration can be generated. Therefore,
additional validations in other parts of the code should be removed.

The builder also pre-computes all required network addresses so that no
address/port magic should be required where the configuration is used
and should therefore be removed.

* Upgrade github.com/hashicorp/hcl to support int64

* improve error messages

* fix directory permission test

* Fix rtt test

* Fix ForceLeave test

* Skip performance test for now until we know what to do

* Update github.com/hashicorp/memberlist to update log prefix

* Make memberlist use the default logger

* improve config error handling

* do not fail on non-existing data-dir

* experiment with non-uniform timeouts to get a handle on stalled leader elections

* Run tests for packages separately to eliminate the spurious port conflicts

* refactor private address detection and unify approach for ipv4 and ipv6.

Fixes #2825

* do not allow unix sockets for DNS

* improve bind and advertise addr error handling

* go through builder using test coverage

* minimal update to the docs

* more coverage tests fixed

* more tests

* fix makefile

* cleanup

* fix port conflicts with external port server 'porter'

* stop test server on error

* do not run api test that change global ENV concurrently with the other tests

* Run remaining api tests concurrently

* no need for retry with the port number service

* monkey patch race condition in go-sockaddr until we understand why that fails

* monkey patch hcl decoder race condidtion until we understand why that fails

* monkey patch spurious errors in strings.EqualFold from here

* add test for hcl decoder race condition. Run with go test -parallel 128

* Increase timeout again

* cleanup

* don't log port allocations by default

* use base command arg parsing to format help output properly

* handle -dc deprecation case in Build

* switch autopilot.max_trailing_logs to int

* remove duplicate test case

* remove unused methods

* remove comments about flag/config value inconsistencies

* switch got and want around since the error message was misleading.

* Removes a stray debug log.

* Removes a stray newline in imports.

* Fixes TestACL_Version8.

* Runs go fmt.

* Adds a default case for unknown address types.

* Reoders and reformats some imports.

* Adds some comments and fixes typos.

* Reorders imports.

* add unix socket support for dns later

* drop all deprecated flags and arguments

* fix wrong field name

* remove stray node-id file

* drop unnecessary patch section in test

* drop duplicate test

* add test for LeaveOnTerm and SkipLeaveOnInt in client mode

* drop "bla" and add clarifying comment for the test

* split up tests to support enterprise/non-enterprise tests

* drop raft multiplier and derive values during build phase

* sanitize runtime config reflectively and add test

* detect invalid config fields

* fix tests with invalid config fields

* use different values for wan sanitiziation test

* drop recursor in favor of recursors

* allow dns_config.udp_answer_limit to be zero

* make sure tests run on machines with multiple ips

* Fix failing tests in a few more places by providing a bind address in the test

* Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder.

* Add porter to server_test.go to make tests there less flaky

* go fmt
2017-09-25 11:40:42 -07:00
Frank Schroeder 85f0f71eb1
Add support to discover public v4 and v6 addresses on AWS (#3471)
Update github.com/hashicorp/go-discover/provider/aws to support the
'addr_type' option which allows detection of private_v4, public_v4 and
public_v6 addresses.

Fixes #3471
2017-09-25 03:16:27 +02:00
James Phillips d1055623a8
Updates vendor.json using latest vendorfmt. 2017-09-06 07:29:17 -07:00
Seth Vargo 97a4d36ec4
Update vendor to regroup by root commands 2017-09-06 10:20:25 -04:00
James Phillips 7ec036fea4
Updates vendor.json for latest vendorfmt changes. 2017-09-05 15:45:16 -07:00
James Phillips bc9780baad Adds simple rate limiting for client agent RPC calls to Consul servers. (#3440)
* Added rate limiting for agent RPC calls.
* Initializes the rate limiter based on the config.
* Adds the rate limiter into the snapshot RPC path.
* Adds unit tests for the RPC rate limiter.
* Groups the RPC limit parameters under "limits" in the config.
* Adds some documentation about the RPC limiter.
* Sends a 429 response when the rate limiter kicks in.
* Adds docs for new telemetry.
* Makes snapshot telemetry look like RPC telemetry and cleans up comments.
2017-09-01 15:02:50 -07:00
Preetha Appan aadaae1d2d Updating vendor directory for raft address provider interface changes 2017-08-30 09:57:48 -05:00
Preetha Appan 2514523920 Update raft library for windows snapshot fsync fixes. This fixes #3409 2017-08-24 16:44:05 -05:00
Frank Schroeder 509e667f94 vendor: upgrade github.com/hashicorp/go-discover
Pull in improved debug logging for AWS
2017-08-23 21:23:34 +02:00
Preetha Appan 040f8ae775 Update serf to pick up fixes for fsyncing snapshots and panic when coordinates are disabled 2017-08-17 16:35:06 -05:00
Preetha Appan 40d6e1fbc7 Update memberlist for a deadlock fix 2017-08-15 18:07:28 -05:00
Kyle Havlovitz 4daba5ddd5
vendor: update github.com/armon/go-metrics 2017-08-08 18:29:27 -07:00
Kyle Havlovitz 8c2e422074 Merge pull request #3369 from hashicorp/metrics-enhancements
Add support for labels/filters from go-metrics
2017-08-08 13:55:30 -07:00
Kyle Havlovitz 6fed5dc490
vendor: update github.com/armon/go-metrics 2017-08-08 12:33:47 -07:00
Kyle Havlovitz abcf13e3f4
vendor: update go-metrics 2017-08-08 01:19:30 -07:00
James Phillips 7fd45a6b67
Propagates a better error message from memberlist.
Fixes #3312.
2017-08-07 16:35:57 -07:00
Preetha Appan 5d15481d8a Pick up raft library change that fsyncs snapshot files correctly 2017-08-04 10:36:41 -05:00
Frank Schroeder 5f6b74d4ee
vendor: update hashicorp/go-discover to pull in hashicorp/go-discover#7 2017-08-03 21:00:37 +02:00
Frank Schroeder 68e8f3d0f7 agent: use github.com/hashicorp/go-discover
Replace the provider specific node discovery code
with go-discover to support AWS, Azure and GCE.

Fixes #3282
2017-08-01 11:41:43 +02:00
Frank Schroeder 5ddcdd41c2 vendor: add go-discover 2017-08-01 11:41:43 +02:00
Frank Schroeder d1c27705f3 vendor: drop unused deps for old discover code 2017-08-01 11:41:43 +02:00