Commit Graph

3257 Commits

Author SHA1 Message Date
hc-github-team-consul-core 93971cefd3 auto-updated agent/uiserver/bindata_assetfs.go from commit 5d89ba031 2021-04-29 09:46:32 +00:00
Freddy a5fc796d1f
Rename cluster config files to mesh as well (#10148) 2021-04-28 18:15:26 -06:00
Daniel Nephin 5fa077cf0d config: default UseStreamingBackend to true 2021-04-28 18:58:02 -04:00
Freddy 401f3010e0
Rename "cluster" config entry to "mesh" (#10127)
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.

Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
2021-04-28 16:13:29 -06:00
Daniel Nephin b4362552fb
Merge pull request #10112 from hashicorp/dnephin/remove-streaming-from-cache
streaming: replace agent/cache with submatview.Store
2021-04-28 17:31:42 -04:00
Daniel Nephin 65da18a986 submatview: fix godoc and comment typos
Co-authored-by: Paul Banks <banks@banksco.de>
2021-04-28 12:27:38 -04:00
hc-github-team-consul-core 16f1995d6d auto-updated agent/uiserver/bindata_assetfs.go from commit 345765f76 2021-04-28 11:21:09 +00:00
Daniel Nephin 872bb9db14 submatview: avoid sorting results unnecessarily
Previous getFromView would call view.Result when the result may not have been returned
(because the index is updated past the minIndex. This would allocate a slice and sort it
for no reason, because the values would never be returned.

Fix this by re-ordering the operations in getFromView.

The test changes in this commit were an attempt to cover the case where
an update is received but the index does not exceed the minIndex.
2021-04-27 19:03:17 -04:00
Daniel Nephin cf8520d85c submatview: only return materializer from getEntry
Also rename it to readEntry now that it doesn't return the entire entry. Based on feedback
in PR review, the full entry is not used by the caller, and accessing the fields wouldn't be
safe outside the lock, so it is safer to return only the Materializer
2021-04-27 19:03:17 -04:00
Daniel Nephin 768e0a7d03 submatview: godoc 2021-04-27 19:03:17 -04:00
Daniel Nephin 2dfacb2d49 rpcclient/health: convert tests to the new submatview.Store interface
Also fixes a minor data race in Materializer. Capture the error before
releasing the lock.
2021-04-27 19:03:17 -04:00
Daniel Nephin 1a6bff1109 agent: fix data race in tests caused by grpc log init
grpclog.SetLoggerV2 is meant to be called only once before any gRPC requests are received, but
each test that uses TestAgent will call NewBaseDeps again. Use a sync.Once to prevent the grpc
logging from being re-initialized by each test.

This will mean that a test can't use a fake logger to capture logs from the gRPC server.
2021-04-27 19:03:17 -04:00
Daniel Nephin 43b14bf57f submatview: fix two flaky tests
These tests can flake when we get a notification for an earlier event.
Retry the read from update channel a few times to make sure we get the
event we expect.
2021-04-27 19:03:17 -04:00
Daniel Nephin 8f47bbe89a rpcclient:health: fix a data race and flake in tests
Split the TestStreamingClient into the two logical components the real
client uses. This allows us to test multiple clients properly.

Previously writing of ctx from multiple Subscribe calls was showing a
data race.

Once this was fixed a test started to fail because the request had to be
made with a greater index, so that the store.Get call did not return
immediately.
2021-04-27 19:03:17 -04:00
Daniel Nephin 95c92343e4 rpcclient/health: fix data race in a test
The idleTTL was being written and read concurrently. Instead move the idleTTL to a struct
field so that when one test patches the TTL it does not impact others.

The background goroutines for the store can outlive a test because context cancellation
is async.
2021-04-27 19:03:17 -04:00
Daniel Nephin d257acee24 rpcclient: close the grpc.ClientConn on shutdown 2021-04-27 19:03:16 -04:00
Daniel Nephin 3cda0a7cc4 health: create health.Client in Agent.New 2021-04-27 19:03:16 -04:00
Daniel Nephin 0ea49c3e65 rpcclient/health: move all backend routing logic to client 2021-04-27 19:03:16 -04:00
Daniel Nephin 318bbd3e30 health: use blocking queries for near query parameter 2021-04-27 19:03:16 -04:00
Daniel Nephin 18c9e73832 connect: do not set QuerySource.Node
Setting this field to a value is equivalent to using the 'near' query paramter.
The intent is to sort the results by proximity to the node requesting
them. However with connect we send the results to envoy, which doesn't
care about the order, so setting this field is increasing the work
performed for no gain.

It is necessary to unset this field now because we would like connect
to use streaming, but streaming does not support sorting by proximity.
2021-04-27 19:03:16 -04:00
Daniel Nephin 440ab3e0ae submatview: move error return to NewMaterializer
So that we don't have to create views ahead of time, when we will never use that view.
2021-04-27 19:03:16 -04:00
Daniel Nephin aadb46b209 rpcclient/health: integrate submatview.Store into rpcclient/health 2021-04-27 19:03:16 -04:00
Daniel Nephin 26c44aacde rpcclient: move streaming cache tests 2021-04-27 19:03:16 -04:00
Daniel Nephin 31c9519712 submatview: test Store.Run 2021-04-27 19:03:16 -04:00
Daniel Nephin d23fd66aeb submatview: test store with Get and Notify calls together 2021-04-27 19:03:16 -04:00
Daniel Nephin 54a402d772 submatview: more test cases for Store.Get
And simplify the return value by using a Result type.
2021-04-27 19:03:16 -04:00
Daniel Nephin f7190b1c61 submatview: reduce the getFromView implementation
Remove View.Result error return value, it was always nil, and seems like it will likely always remain nill
since it is simply reading a stored value.

Also replace some cache types with local types.
2021-04-27 19:03:16 -04:00
Daniel Nephin f4573177ba submatview: track requests instead of notifiers
And only start expiration time when the last request ends. This makes tracking expiry simpler, and
ensures that no entry can be expired while there are active requests.
2021-04-27 19:03:16 -04:00
Daniel Nephin 468469bd27 submatview: set up expiry of materializers 2021-04-27 19:03:16 -04:00
Daniel Nephin f46a830e48 submatview: setup testing structure 2021-04-27 19:03:16 -04:00
Daniel Nephin e4c503c28e submatview: rough outline of the Get and Notify methods. 2021-04-27 19:03:16 -04:00
Daniel Nephin c23e98a5e6 submatview: setup the interface for the store 2021-04-27 19:03:15 -04:00
Matt Keeler 6c639be8ec
Add prometheus guage definitions for replication metrics. (#10109) 2021-04-23 17:05:33 -04:00
Matt Keeler 09bf05ec5d
Add replication metrics (#10073) 2021-04-22 11:20:53 -04:00
hashicorp-ci af54f11426 auto-updated agent/uiserver/bindata_assetfs.go from commit 7a095e7d2 2021-04-22 11:28:27 +00:00
hashicorp-ci 4dfa2ce1ab auto-updated agent/uiserver/bindata_assetfs.go from commit f6f282ac7 2021-04-22 11:21:21 +00:00
Daniel Nephin 281d7616fa
Merge pull request #10045 from hashicorp/dnephin/state-proxy-defaults
state: remove config-entries kind index
2021-04-20 17:12:50 -04:00
Daniel Nephin 6d1a5b3629 Handle ErrChunkingResubmit.Error properly
Previously canRetry was attempting to retrieve this error from args, however there was never
any callers that would pass an error to args.

With the change to raftApply to move this error to the error return value, it is now possible
to receive this error from the err argument.

This commit updates canRetry to check for ErrChunkingResubmit in err.
2021-04-20 13:29:31 -04:00
Daniel Nephin 8654adfc53 Handle FSM.Apply errors in raftApply
Previously we were inconsistently checking the response for errors. This
PR moves the response-is-error check into raftApply, so that all callers
can look at only the error response, instead of having to know that
errors could come from two places.

This should expose a few more errors that were previously hidden because
in some calls to raftApply we were ignoring the response return value.

Also handle errors more consistently. In some cases we would log the
error before returning it. This can be very confusing because it can
result in the same error being logged multiple times. Instead return
a wrapped error.
2021-04-20 13:29:29 -04:00
Freddy 55d67824d6
Disallow * as service-defaults name (#10069) 2021-04-19 14:23:01 -06:00
R.B. Boyer 35f2bf22b7
fix test when run in enterprise (#10048) 2021-04-15 17:25:25 -05:00
freddygv 25dfa0ce24 Don't panic on nil UpstreamConfiguration.Clone() 2021-04-15 15:46:21 -06:00
Daniel Nephin 95b361ecc8 state: remove unnecessary kind index
The query can be performed using a prefix query on the ID index.

Also backport some enterprise changes to prevent conflicts.
2021-04-15 17:37:28 -04:00
Daniel Nephin eb7f4b7ea4 state: use index constants for ConfigEntry indexes 2021-04-15 17:30:07 -04:00
hashicorp-ci 8293eedace auto-updated agent/uiserver/bindata_assetfs.go from commit daf897f1a 2021-04-15 20:19:13 +00:00
Freddy 5a9b75a443
Merge pull request #10016 from hashicorp/topology-update 2021-04-15 14:11:23 -06:00
Freddy 6d15569062
Split Upstream.Identifier() so non-empty namespace is always prepended in ent (#10031) 2021-04-15 13:54:40 -06:00
R.B. Boyer c88512fe14
connect: update centralized upstreams representation in service-defaults (#10015) 2021-04-15 14:21:44 -05:00
Kent 'picat' Gruber 2998db9cda
Merge pull request #10023 from hashicorp/fix-raw-kv-xss
Add content type headers to raw KV responses
2021-04-14 18:49:14 -04:00
Daniel Nephin 2a10f01bf5 snapshot: fix saving of auth methods
Previously only a single auth method would be saved to the snapshot. This commit fixes the typo
and adds to the test, to show that all auth methods are now saved.
2021-04-14 16:51:21 -04:00
Kent 'picat' Gruber 91aefe50ab Add content type headers to raw KV responses 2021-04-14 16:20:22 -04:00
freddygv 2ff8b9f2f5 Avoid returning a nil slice 2021-04-14 10:52:05 -06:00
Matt Keeler aa0eb60f57
Move static token resolution into the ACLResolver (#10013) 2021-04-14 12:39:35 -04:00
R.B. Boyer 0470d9ec25 fix broken golden tests 2021-04-14 11:36:47 -05:00
Freddy a85bfc0bd0
Merge pull request #10006 from hashicorp/api-ptrs 2021-04-14 10:21:08 -06:00
freddygv 7fd4c569ce Update viz endpoint to include topology from intentions 2021-04-14 10:20:15 -06:00
Freddy 57b998e027
Merge pull request #9987 from hashicorp/remove-kube-dns-hack 2021-04-14 10:00:53 -06:00
freddygv e175b309fb Fixup tests 2021-04-13 16:08:41 -06:00
freddygv 50c7915156 Convert new tproxy structs in api module into ptrs
This way we avoid serializing these when empty. Otherwise users of the
latest version of the api submodule cannot interact with older versions
of Consul, because a new api client would send keys that the older Consul
doesn't recognize yet.
2021-04-13 12:44:25 -06:00
Freddy 79257c1489
Merge pull request #10005 from hashicorp/tproxy-fixes 2021-04-13 11:45:40 -06:00
Freddy 754be9f6a4
Merge pull request #10000 from hashicorp/remove-upstream-cfg-validation
Remove zero-value validation of upstream cfg structs
2021-04-13 11:00:02 -06:00
freddygv 911d7dcaa8 Remove todo that was todone 2021-04-13 10:19:59 -06:00
freddygv 84a5f91a08 Avoid nil panic when cluster config doesn't exist 2021-04-13 10:17:11 -06:00
freddygv 83501d5415 Augment intention decision summary with DefaultAllow mode 2021-04-12 19:32:09 -06:00
freddygv 36e9326dab Fixup wildcard ent assertion 2021-04-12 17:04:33 -06:00
Freddy 2a3229992e
Merge pull request #9999 from hashicorp/update-enabling-tproxy 2021-04-12 16:37:04 -06:00
freddygv b83841ab29 Fixup bexpr filtering 2021-04-12 10:17:52 -06:00
freddygv e6d44ae03b Remove zero-value validation of upstream cfg structs
The zero value of these flags was already being excluded in the xDS
generation of circuit breaker/outlier detection config.

See: makeThresholdsIfNeeded and ToOutlierDetection.
2021-04-12 10:08:57 -06:00
freddygv eeccba945d Replace TransparentProxy bool with ProxyMode
This PR replaces the original boolean used to configure transparent
proxy mode. It was replaced with a string mode that can be set to:

- "": Empty string is the default for when the setting should be
defaulted from other configuration like config entries.
- "direct": Direct mode is how applications originally opted into the
mesh. Proxy listeners need to be dialed directly.
- "transparent": Transparent mode enables configuring Envoy as a
transparent proxy. Traffic must be captured and redirected to the
inbound and outbound listeners.

This PR also adds a struct for transparent proxy specific configuration.
Initially this is not stored as a pointer. Will revisit that decision
before GA.
2021-04-12 09:35:14 -06:00
freddygv fb4d07243d Avoid failing test due to undiscoverable node name 2021-04-12 09:26:55 -06:00
hashicorp-ci abc9daf1ea auto-updated agent/uiserver/bindata_assetfs.go from commit 84064f972 2021-04-12 13:08:41 +00:00
freddygv 9c219a5b58 Fixup mesh gateway docs 2021-04-11 15:48:04 -06:00
tarat44 9a2e630f86 fix formatting 2021-04-11 15:12:33 -04:00
tarat44 f2ba70d559 add WaitGroup to h2ping 2021-04-11 15:11:00 -04:00
tarat44 7ac3d84ad4 close h2ping client connections 2021-04-10 00:53:53 -04:00
Tara Tufano b8e7a90f77
add http2 ping health checks (#8431)
* add http2 ping checks

* fix test issue

* add h2ping check to config resources

* add new test and docs for h2ping

* fix grammatical inconsistency in H2PING documentation

* resolve rebase conflicts, add test for h2ping tls verification failure

* api documentation for h2ping

* update test config data with H2PING

* add H2PING to protocol buffers and update changelog

* fix typo in changelog entry
2021-04-09 15:12:10 -04:00
Iryna Shustava ff2e70f4ce
cli: Add new `consul connect redirect-traffic` command for applying traffic redirection rules when Transparent Proxy is enabled. (#9910)
* Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled.
* Add new iptables package for applying traffic redirection rules with iptables.
2021-04-09 11:48:10 -07:00
Freddy 920ba3db39
Merge pull request #9976 from hashicorp/centralized-upstream-fixups 2021-04-08 12:26:56 -06:00
Freddy 38819dac1f
Merge pull request #9042 from lawliet89/tg-rewrite 2021-04-08 11:49:23 -06:00
freddygv b97d3422a7 Stable sort cidr ranges to match on 2021-04-08 11:27:57 -06:00
freddygv 0d0205e0dc PR comments 2021-04-08 11:16:03 -06:00
Daniel Nephin 93818ebc5a
Merge pull request #9950 from hashicorp/dnephin/state-use-txn-everywhere
state: use Txn interface everywhere
2021-04-08 12:02:03 -04:00
Daniel Nephin 9db8ffb1c5
Merge pull request #9880 from hashicorp/dnephin/catalog-events-test-pattern
state: use runCase pattern for large test
2021-04-08 11:54:41 -04:00
Paul Banks ae5c0aad39
cache: Fix bug where connection errors can cause early cache expiry (#9979)
Fixes a cache bug where TTL is not updated while a value isn't changing or cache entry is returning fetch errors.
2021-04-08 11:11:15 +01:00
Paul Banks b61e00b772
cache: fix bug where TTLs were ignored leading to leaked memory in client agents (#9978)
* Fix bug in cache where TTLs are effectively ignored

This mostly affects streaming since streaming will immediately return from Fetch calls when the state is Closed on eviction which causes the race condition every time.

However this also affects all other cache types if the fetch call happens to return between the eviction and then next time around the Get loop by any client.

There is a separate bug that allows cache items to be evicted even when there are active clients which is the trigger here.

* Add changelog entry

* Update .changelog/9978.txt
2021-04-08 11:08:56 +01:00
Paul Banks 9115884c50
Merge pull request #9977 from hashicorp/grpc-tuning
streaming: Grpc tuning
2021-04-08 11:05:38 +01:00
freddygv 43091aae60 Avoid sending zero-value upstream defaults from api 2021-04-07 15:03:42 -06:00
freddygv cc4933ea0e Revert "Avoid accumulating synthetic upstreams"
This reverts commit 86672df4fad094cd7e044bf4db168162594517c2.
2021-04-07 14:30:30 -06:00
freddygv 69822fa5ae Remove kube-dns resolution since clusterip will be a tagged addr 2021-04-07 14:15:21 -06:00
hashicorp-ci 014547159c auto-updated agent/uiserver/bindata_assetfs.go from commit a0d12ff16 2021-04-07 16:48:59 +00:00
freddygv 7698be3636 Avoid accumulating synthetic upstreams
Synthetic upstreams from service-defaults config are stored locally in
the Upstreams list. Since these come from service-defaults they should
be cleaned up locally when no longer present in the service config
response.
2021-04-07 09:32:48 -06:00
freddygv ddc6c9b7ca Ensure mesh gateway mode override is set for upstreams for intentions 2021-04-07 09:32:48 -06:00
freddygv 619dc5ede4 Finish resolving upstream defaults in proxycfg 2021-04-07 09:32:48 -06:00
freddygv a1fd3b0271 Pass down upstream defaults to client proxies
This is needed in case the client proxy is in TransparentProxy mode.
Typically they won't have explicit configuration for every upstream, so
this ensures the settings can be applied to all of them when generating
xDS config.
2021-04-07 09:32:47 -06:00
freddygv 042753fc26 Prevent wildcard destinations for proxies and upstreams 2021-04-07 09:32:47 -06:00
freddygv c2e74e21bc Prevent requests without UpstreamIDs from being flagged as legacy.
New clients in transparent proxy mode can send requests for service
config resolution without any upstream args because they do not have
explicitly defined upstreams.

Old clients on the other hand will never send requests without the
Upstreams args unless they don't have upstreams, in which case we do not
send back upstream config.
2021-04-07 09:32:47 -06:00
freddygv d19151f162 Fixup doc phrasing 2021-04-07 09:32:47 -06:00
freddygv 8566495f4f Prevent synthetic upstreams without addresses from failing duplicate ip/port validation 2021-04-07 09:32:47 -06:00
Paul Banks 571324ad08 Tune streaming backoff on errors to retry a bit faster when TCP connections drop 2021-04-07 14:13:30 +01:00
Paul Banks f4257f91f6 Set gRPC keepalives to mirror Yamux keepalive behaviour 2021-04-07 14:09:22 +01:00