Commit Graph

3162 Commits

Author SHA1 Message Date
Christopher Broglie 94b02c3954 Add support for configuring TLS ServerName for health checks
Some TLS servers require SNI, but the Golang HTTP client doesn't
include it in the ClientHello when connecting to an IP address. This
change adds a new TLSServerName field to health check definitions to
optionally set it. This fixes #9473.
2021-03-16 18:16:44 -04:00
freddygv cba952a6a5 Add cache-type for Internal.IntentionUpstreams 2021-03-16 11:06:47 -06:00
Daniel Nephin 0b3930272d state: convert services.node and checks.node indexes
Using NodeIdentity to share the indexes with both.
2021-03-16 13:00:31 -04:00
freddygv b79039c21c Prefix match type vars to match use 2021-03-16 09:49:24 -06:00
freddygv fed983fe9a Pass txn into service list queries 2021-03-16 09:33:08 -06:00
freddygv 26ba0c0fc8 Pass txn into intention match queries 2021-03-16 08:03:52 -06:00
freddygv d7f3bcc8bb Replace CertURI.Authorize() calls.
AuthorizeIntentionTarget is a generalized version of the old function,
and can be evaluated against sources or destinations.
2021-03-15 18:06:04 -06:00
freddygv eb6c0cbea0 Fixup typo, comments, and regression 2021-03-15 17:50:47 -06:00
freddygv 9bfb0969f9 Fixup upstream test 2021-03-15 17:20:30 -06:00
freddygv 940b7a98d1 Finish cleanup from ServiceConfigRequest changes 2021-03-15 16:38:01 -06:00
freddygv a67c92b961 Update service manager to pass MeshGateway with config req 2021-03-15 16:08:03 -06:00
freddygv 871e1d3e31 PR comments 2021-03-15 16:02:03 -06:00
Daniel Nephin 0b5dfee00a state: use runCase pattern for large test
The TestServiceHealthEventsFromChanges function was over 1400 lines.
Attempting to debug test failures in test functions this large is
difficult. It requires scrolling to the line which defines the testcase
because the failure message only includes the line number of the
assertion, not the line number of the test case.

This is an excellent example of where test tables stop working well, and
start being a problem. To mitigate this problem, the runCase pattern can
be used. When one of these tests fails, a failure message will print the
line number of both the test case and the assertion. This allows a
developer to quickly jump to both of the relevant lines, signficanting
reducing the time it takes to debug test failures.

For example, one such failure could look like this:

    catalog_events_test.go:1610: case: service reg, new node
    catalog_events_test.go:1605: assertion failed: values are not equal
2021-03-15 17:53:16 -04:00
freddygv 04fbc104cd Pass MeshGateway config in service config request
ResolveServiceConfig is called by service manager before the proxy
registration is in the catalog. Therefore we should pass proxy
registration flags in the request rather than trying to fetch
them from the state store (where they may not exist yet).
2021-03-15 14:32:13 -06:00
freddygv d90240d367 Restore old Envoy prefix on escape hatches
This is done because after removing ID and NodeName from
ServiceConfigRequest we will no longer know whether a request coming in
is for a Consul client earlier than v1.10.
2021-03-15 14:12:57 -06:00
freddygv 13cce3419a Only lowercase the protocol when normalizing 2021-03-15 14:12:15 -06:00
freddygv f584c2d7c5 Add omitempty across the board for UpstreamConfig 2021-03-15 13:23:18 -06:00
freddygv 3b2169b36d Add RPC endpoint for intention upstreams 2021-03-15 08:50:35 -06:00
freddygv e4e14639b2 Add state store function for intention upstreams 2021-03-15 08:50:35 -06:00
freddygv 4976c000b7 Refactor IntentionDecision
This enables it to be called for many upstreams or downstreams of a
service while only querying intentions once.

Additionally, decisions are now optionally denied due to L7 permissions
being present. This enables the function to be used to filter for
potential upstreams/downstreams of a service.
2021-03-15 08:50:35 -06:00
Daniel Nephin 2a53b8293a proxycfg: use rpcclient/health.Client instead of passing around cache name
This should allow us to swap out the implementation with something other
than `agent/cache` without making further code changes.
2021-03-12 11:46:04 -05:00
Daniel Nephin c33570be34 catalog_events: set the right key for connect snapshots 2021-03-12 11:35:43 -05:00
Daniel Nephin 410b1261c2 proxycfg: Use streaming in connect state 2021-03-12 11:35:42 -05:00
Daniel Nephin e2215d9f0f rpcclient: use streaming for connect health 2021-03-12 11:35:42 -05:00
freddygv 3d85c29445 Update content hash due to new field 2021-03-11 19:59:19 -07:00
freddygv d27208ce7c Fixup more tests 2021-03-11 16:26:55 -07:00
freddygv be78d3c39a Fixup protobufs and tests 2021-03-11 14:58:59 -07:00
Kyle Havlovitz 237b41ac8f
Merge pull request #9672 from hashicorp/ca-force-skip-xc
connect/ca: Allow ForceWithoutCrossSigning for all providers
2021-03-11 11:49:15 -08:00
freddygv 3de0b32cc5 Update service manager to store centrally configured upstreams 2021-03-11 11:37:21 -07:00
freddygv 7a3625f58b Add TransparentProxy opt to proxy definition 2021-03-11 11:37:21 -07:00
freddygv ac70808792 Restore old escape hatch alias 2021-03-11 11:36:35 -07:00
freddygv c30157d2f2 Turn Limits and PassiveHealthChecks into pointers 2021-03-11 11:04:40 -07:00
hashicorp-ci 5dbeeee673 auto-updated agent/uiserver/bindata_assetfs.go from commit fa6687b7f 2021-03-11 09:34:21 +00:00
freddygv b98abb6f09 Update server-side config resolution and client-side merging 2021-03-10 21:05:11 -07:00
freddygv 68148a1dae finish moving UpstreamConfig and related fields to structs pkg 2021-03-10 21:04:13 -07:00
Daniel Nephin 4877183bc6
Merge pull request #9797 from hashicorp/dnephin/state-index-node-id
state: convert nodes.ID to the new pattern of functional indexers
2021-03-10 17:34:23 -05:00
Daniel Nephin 51ad94360b state: move ConfigEntryKindName
Previously this type was defined in structs, but unlike the other types in structs this type
is not used by RPC requests. By moving it to state we can better indicate that this is not
an API type, but part of the state implementation.
2021-03-10 12:27:22 -05:00
Daniel Nephin 5c5ba9564d
Merge pull request #9796 from hashicorp/dnephin/state-cleanup-catalog-index-oss
state: remove duplicate tableCheck indexes
2021-03-10 12:20:09 -05:00
Daniel Nephin 97d4e872d3
Merge pull request #9851 from panascais-forks/fix-wan-ipv6-key
Fix advertise_addr_wan_ipv6 configuration key
2021-03-10 11:56:07 -05:00
Daniel Nephin 94820e67a8 structs: remove EnterpriseMeta.GetNamespace
I added this recently without realizing that the method already existed and was named
NamespaceOrEmpty. Replace all calls to GetNamespace with NamespaceOrEmpty or NamespaceOrDefault
as appropriate.
2021-03-09 15:17:26 -05:00
Daniel Nephin 848314782e
Merge pull request #9671 from hashicorp/streaming/terminating-gateway-events
state: Add terminating gateway events for streaming
2021-03-09 14:20:21 -05:00
Daniel Nephin 97bc073bd9 state: adjust compare for catalog events
Document that this comparison should roughly match MatchesKey

Only sort by overrideKey or service name, but not both
Add namespace to the sort.

The client side also builds a map of these based on the namespace/node/service key, so the only order
that really matters is the ordering of register/dereigster events.
2021-03-09 14:00:36 -05:00
Daniel Nephin 0d3bb68255 state: handle terminating gateway events properly in snapshot
Refactored out a function that can be used for both the snapshot and stream of events to translate
an event into an appropriate connect event.

Previously terminating gateway events would have used the wrong key in the snapshot, which would have
caused them to be filtered out later on.

Also removed an unused function, and some commented out code.
2021-03-09 14:00:35 -05:00
Kyle Havlovitz de3fba8ef3 Add remaining terminating gateway tests for namespaces
Co-Authored-By: Daniel Nephin <dnephin@hashicorp.com>
2021-03-09 14:00:35 -05:00
Daniel Nephin 38aeb88908 Start to setup enterprise tests for terminating gateway streaming events.
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:35 -05:00
Daniel Nephin d0b37f18f0 state: Add support for override of namespace
in MatchesKey
also tests for MatchesKey

Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:35 -05:00
Daniel Nephin ba59727337 state: update calls to ensureConfigEntryTxn
The EnterpriseMeta paramter was removed after this code was written, but before it merged.

Also the table name constant has changed.
2021-03-09 14:00:35 -05:00
Daniel Nephin 730cc575e6 state: add 2 more test cases for terminate gateway streaming events
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:34 -05:00
Kyle Havlovitz eadc8546a9 Added 6 new test cases for terminating gateway events
Co-Authored-By: Daniel Nephin <dnephin@hashicorp.com>
2021-03-09 14:00:34 -05:00
Daniel Nephin 15b0d5f62b state: Add two more tests for connect events with terminating gateways
And expand one test case to cover more.

Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:34 -05:00
Daniel Nephin abab373b89 state: Include the override key in the sorting of events
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:34 -05:00
Kyle Havlovitz f31582624d state: Add terminating gateway events on updating a config entry
Co-Authored-By: Daniel Nephin <dnephin@hashicorp.com>
2021-03-09 14:00:34 -05:00
Daniel Nephin f42a2ca8a3 state: add first terminating catalog catalog event
Health of a terminating gateway instance changes
- Generate an event for creating/destroying this instance of the terminating gateway,
  duplicate it for each affected service

Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:33 -05:00
Silas Rech f71bba20e7
Fix advertise_addr_wan_ipv6 configuration key 2021-03-09 14:56:44 +01:00
hashicorp-ci da1663bed9 auto-updated agent/uiserver/bindata_assetfs.go from commit 33d038377 2021-03-09 09:35:32 +00:00
freddygv 4bbd495b54 Create new types for service-defaults upstream cfg 2021-03-08 22:10:27 -07:00
hashicorp-ci f0e34dfadb auto-updated agent/uiserver/bindata_assetfs.go from commit 308e5a480 2021-03-08 12:28:15 +00:00
Daniel Nephin 1184ceff9e state: convert nodes.ID to new functional pattern
In preparation for adding other identifiers to the index.
2021-03-05 12:30:40 -05:00
R.B. Boyer 503041f216
xds: default to speaking xDS v3, but allow for v2 to be spoken upon request (#9658)
- Also add support for envoy 1.17.0
2021-02-26 16:23:15 -06:00
Daniel Nephin 4a44cfd676
Merge pull request #9188 from hashicorp/dnephin/more-streaming-tests
Add more streaming tests
2021-02-26 12:36:55 -05:00
Daniel Nephin d7ffd6c27d
Merge pull request #9759 from hashicorp/dnephin/streaming-default-rpc-enabled
streaming: default rpc.enable_streaming to true
2021-02-26 12:08:00 -05:00
Daniel Nephin 4ef9578a07
Merge pull request #9703 from pierresouchay/streaming_tags_and_case_insensitive
Streaming filter tags + case insensitive lookups for Service Names
2021-02-26 12:06:26 -05:00
Daniel Nephin 2cc3282d5d catalog_events: set the right key for connect snapshots
Add a test for catalog_event snapshot on connect topic
2021-02-25 14:30:39 -05:00
Daniel Nephin 85da1af04c consul: Add integration tests of streaming.
Restored from streaming-rpc-final branch.

Co-authored-by: Paul Banks <banks@banksco.de>
2021-02-25 14:30:39 -05:00
Daniel Nephin e8beda4685 state: Add a test for ServiceHealthSnapshot 2021-02-25 14:08:10 -05:00
Daniel Nephin 88bbde56da agent: add a test for streaming in the service health endpoint
Co-authored-by: Paul Banks <banks@banksco.de>
2021-02-25 14:08:10 -05:00
Daniel Nephin af2431793b streaming: default rpc.enable_streaming to true
So that all servers will start the grpc server used by streaming
2021-02-25 14:06:04 -05:00
hashicorp-ci 4ebdbf57d7 auto-updated agent/uiserver/bindata_assetfs.go from commit 779f7f7b6 2021-02-25 09:41:02 +00:00
John Cowen 2d500d24b8
ui: Remove any trailing fullstop/period DNS characters from Gateways UI API (#9752)
Previous to this commit, the API response would include Gateway
Addresses in the form `domain.name.:8080`, which due to the addition of
the port is probably not the expected response.

This commit rightTrims any `.` characters from the end of the domain
before formatting the address to include the port resulting in
`domain.name:8080`
2021-02-25 09:34:47 +00:00
R.B. Boyer 4336d522c1
test: omit envoy golden test files that differ from the latest version (#9807)
Since we currently do no version switching this removes 75% of the PR
noise.

To generate all *.golden files were removed and then I ran:

    go test ./agent/xds -update
2021-02-24 14:04:31 -06:00
hashicorp-ci 8ebffea6a6 auto-updated agent/uiserver/bindata_assetfs.go from commit f9b0e50ca 2021-02-24 09:08:42 +00:00
Yong Wen Chua 677f798443
Update test fixtures 2021-02-24 16:24:32 +08:00
Yong Wen Chua e1820517aa
Auto Rewrite Host Headers for Terminating Gateways
Tries to partially address
https://github.com/hashicorp/consul/issues/8707
2021-02-24 16:23:39 +08:00
hashicorp-ci 26d41f076b auto-updated agent/uiserver/bindata_assetfs.go from commit 8b12d0d09 2021-02-23 09:02:41 +00:00
R.B. Boyer cdc5e99184
xds: remove deprecated usages of xDS (#9602)
Note that this does NOT upgrade to xDS v3. That will come in a future PR.

Additionally:

- Ignored staticcheck warnings about how github.com/golang/protobuf is deprecated.
- Shuffled some agent/xds imports in advance of a later xDS v3 upgrade.
- Remove support for envoy 1.13.x but don't add in 1.17.x yet. We have to wait until the xDS v3 support is added in a follow-up PR.

Fixes #8425
2021-02-22 15:00:15 -06:00
R.B. Boyer b8f228781a
connect: if the token given to the vault provider returns no data avoid a panic (#9806)
Improves #9800
2021-02-22 14:08:49 -06:00
hashicorp-ci 962c5ae214 auto-updated agent/uiserver/bindata_assetfs.go from commit 2c2e9d1ce 2021-02-22 14:58:06 +00:00
Daniel Nephin 8ec1b06f3a
Merge pull request #9683 from hashicorp/dnephin/fix-zombie-service-dereg
local: use agent token to deregister services
2021-02-19 18:56:40 -05:00
Daniel Nephin 76a365d410 local: default to the agent token instead of the user token
When de-registering in anti-entropy sync, when there is no service or
check token.

The agent token will fall back to the default (aka user) token if no agent
token is set, so the existing behaviour still works, but it will prefer
the agent token over the user token if both are set.

ref: https://www.consul.io/docs/agent/options#acl_tokens

The agent token seems more approrpiate in this case, since this is an
"internal operation", not something initiated by the user.
2021-02-19 18:35:08 -05:00
Daniel Nephin 5d478df9b5
Merge pull request #9763 from hashicorp/dnephin/cache-warn-on-error-in-notify
cache: log a warning when Cache.Notify handles an error
2021-02-19 18:30:36 -05:00
Daniel Nephin dd45c4cfe4 state: add a test case for memdb indexers 2021-02-19 17:14:46 -05:00
Daniel Nephin 7e4d693aaa state: support for functional indexers
These new functional indexers provide a few advantages:

1. enterprise differences can be isolated to a single function (the
   indexer function), making code easier to change
2. as a consequence of (1) we no longer need to wrap all the calls to
   Txn operations, making code easier to read.
3. by removing reflection we should increase the performance of all
   operations.

One important change is in making all the function signatures the same.

https://blog.golang.org/errors-are-values

An extra boolean return value for SingleIndexer.FromObject is superfluous.
The error value can indicate when the index value could not be created.
By removing this extra return value we can use the same signature for both
indexer functions.

This has the nice properly of a function being usable for both indexing operations.
2021-02-19 17:14:46 -05:00
Daniel Nephin 88a9bd6d3c state: remove duplicate index on the checks table
By using a new pattern for more specific indexes. This allows us to use
the same index for both service checks and node checks. It removes the
abstraction around memdb.Txn operations, and isolates all of the
enterprise differences in a single place (the indexer).
2021-02-19 17:14:46 -05:00
Daniel Nephin b781fec664 state: remove duplicate function
catalogChecksForNodeService was a duplicate of catalogListServiceChecks
2021-02-19 17:14:46 -05:00
R.B. Boyer e87d2bb24f
xds: only try to create an ipv6 expose checks listener if ipv6 is supported by the kernel (#9765)
Fixes #9311

This only fails if the kernel has ipv6 hard-disabled. It is not sufficient to merely not provide an ipv6 address for a network interface.
2021-02-19 14:38:43 -06:00
hashicorp-ci 0cd20ba653 auto-updated agent/uiserver/bindata_assetfs.go from commit 8263879e6 2021-02-19 16:48:03 +00:00
hashicorp-ci 5622d8b74d auto-updated agent/uiserver/bindata_assetfs.go from commit 7be21a902 2021-02-18 18:37:38 +00:00
hashicorp-ci 7770196315 auto-updated agent/uiserver/bindata_assetfs.go from commit 1507dd8ab 2021-02-17 19:01:18 +00:00
hashicorp-ci 4a8b1c2f0d auto-updated agent/uiserver/bindata_assetfs.go from commit 1b5087f6e 2021-02-17 18:56:34 +00:00
Mark Anderson 1cf6a435dc
Add fields to the /acl/auth-methods endpoint. (#9741)
* A GET of the /acl/auth-method/:name endpoint returns the fields
MaxTokenTTL and TokenLocality, while a LIST (/acl/auth-methods) does
not.

The list command returns a filtered subset of the full set. This is
somewhat deliberate, so that secrets aren't shown, but the TTL and
Locality fields aren't (IMO) security critical, and it is useful for
the front end to be able to show them.

For consistency these changes mirror the 'omit empty' and string
representation choices made for the GET call.

This includes changes to the gRPC and API code in the client.

The new output looks similar to this
curl 'http://localhost:8500/v1/acl/auth-methods' | jq '.'

  {
    "MaxTokenTTL": "8m20s",
    "Name": "minikube-ttl-local2",
    "Type": "kubernetes",
    "Description": "minikube auth method",
    "TokenLocality": "local",
    "CreateIndex": 530,
    "ModifyIndex": 530,
    "Namespace": "default"
  }
]

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Add changelog

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-02-17 08:16:57 -08:00
Daniel Nephin 577b3a1370
Merge pull request #9452 from hashicorp/dnephin/config-tests-flags-and-edgecases
config: make TestBuilder_BuildAndValidate_ConfigFlagsAndEdgecases easier to work with
2021-02-16 16:43:49 -05:00
Daniel Nephin d33bc493af
Merge pull request #9720 from hashicorp/dnephin/ent-meta-ergo-1
structs: rename EnterpriseMeta constructor
2021-02-16 15:31:58 -05:00
Daniel Nephin 53c82cee86
Merge pull request #9772 from hashicorp/streamin-fix-bad-cached-snapshot
streaming: fix snapshot cache bug
2021-02-16 15:28:00 -05:00
Daniel Nephin b17967827d
Merge pull request #9728 from hashicorp/dnephin/state-index-table
state: document how index table is used
2021-02-16 15:27:27 -05:00
Daniel Nephin c40d063a0e structs: rename EnterpriseMeta constructor
To match the Go convention.
2021-02-16 14:45:43 -05:00
Daniel Nephin 005702e098 config: rename testCase fields
So that their name better reflects their purpose
2021-02-16 14:40:16 -05:00
Daniel Nephin e6badb3129 config: make TestLoad_IntegrationWithFlags easier to work with
Replace the large table of tests with individual calls to run(). By using
runCase, failure messages will include the line number for the test case, as
well as a line number from the test functions.

Example:
=== FAIL: agent/config TestLoad_IntegrationWithFlags/failing_case (0.01s)
    runtime_test.go:4721: case: failing case
    runtime_test.go:4864: error "data_dir cannot be empty" does not contain "I expected this error"

Previous:
    runtime_test.go:4864: error "data_dir cannot be empty" does not contain "I expected this error"

Without the line number to the testCase data, debugging these tests is
difficult. It is impossible to jump directly to the test case, and
difficult to find the location because of many similarly named cases.
2021-02-16 14:40:16 -05:00
Daniel Nephin db53954a3f move test case iteration to caller
To make the test case logic more obvious
2021-02-16 14:40:16 -05:00
Daniel Nephin 7499e38f2b config: remove test cases for impossible scenarios
AEInterval is overridden by NonUserSource, so there is no way for a user
to set this value. These two cases represented impossible real world
scenarios.

Instead the test is replaced with one that shows that the AEInterval can
not be set by config.

This change allows us to remove the hcltail and jsontail fields from
testCase
2021-02-16 14:40:15 -05:00
Daniel Nephin 5f2c358255 Fix error checking
There were two identical if lines, remove one and use a switch/case so the compiler
can catch this problem.
2021-02-16 14:40:15 -05:00
Daniel Nephin 1ff2525cee config: move testing shims to LoadOpts
This will allow us to eventually port these tests to use config.Load instead of
unexported functions.
2021-02-16 14:40:15 -05:00
Daniel Nephin a877ec79b6 Remove patchActual, does not appear to be necessary
also small rename and fix comment
2021-02-16 14:40:15 -05:00
Daniel Nephin a29b848e3b stream: fix a snapshot cache bug
Previously a snapshot created as part of a resumse-stream request could have incorrectly
cached the newSnapshotToFollow event. This would cause clients to error because they
received an unexpected framing event.
2021-02-16 12:52:23 -05:00
Daniel Nephin 2726c65fbe stream: test the snapshot cache is saved correctly
when the cache entry is created from resuming a stream.
2021-02-16 12:08:43 -05:00
Daniel Nephin e47131bfe6 cache: log a warning when Cache.Notify handles an error
Without these warnings, errors are silently ignored, which can make
debugging problems more challenging.
2021-02-12 13:02:23 -05:00
Daniel Nephin a5a1fb2098
Merge pull request #9758 from hashicorp/dnephin/fix-streaming-bugs
http: error if near is used with streaming
2021-02-12 10:37:29 -05:00
Daniel Nephin 0683964519 streaming: move ServiceTag and NodeMetaFiltering to the cache-entry
So that all the client side filtering is in the same place. Previously
only the bexpr filter was in the cache-entry.

Also makes a small change to the filtering so that instead of rebuilding
slices of items, the filtering can return a bool to determine if the
event payload is saved or not.
2021-02-11 20:20:09 -05:00
Daniel Nephin d8f10deda3
Merge pull request #9727 from mbanikazemi/typos
Correcting the changed function name in comment
2021-02-11 14:46:10 -05:00
Daniel Nephin da62f22aff http: error if near is used with streaming 2021-02-11 14:10:38 -05:00
hashicorp-ci 3b442c77f4 auto-updated agent/uiserver/bindata_assetfs.go from commit 96204a21d 2021-02-11 11:42:07 +00:00
hashicorp-ci 7c29524b95 auto-updated agent/uiserver/bindata_assetfs.go from commit 369637aff 2021-02-11 10:10:21 +00:00
hashicorp-ci cd87ee21f8 auto-updated agent/uiserver/bindata_assetfs.go from commit eda38a0ba 2021-02-11 09:54:29 +00:00
R.B. Boyer 194fb0d144
connect: update supported envoy point releases to 1.16.2, 1.15.3, 1.14.6, 1.13.7 (#9737) 2021-02-10 13:11:15 -06:00
Pierre Souchay 54dbcd0bb9 Do not filter tags unless req.TagFilter is set
Send empty array [] instead of [""] in DNS requests when TagFilter is not set

Do not change case sensitivity of services anymore in `getServiceNodes()` since
cache keys are now case insensitive
2021-02-10 10:36:11 +01:00
Daniel Nephin bd122bb9f5 streaming: double the cache TTL
10 minutes is the default blocking query timeout. Using the same value results in us hitting
the expired cache entry bug frequently. By extending this TTL we at least mitigate the problem.

The underlying bug still needs to be fixed.
2021-02-09 14:36:26 -05:00
Daniel Nephin 3ab546623f submatview: do not reset retry waiter when materializer is reset
The materializer is often reset when an error is received. By resetting
the retryWaiter we effectively never wait. The retryWaiter should only
be reset when we get an event without error. This is done in
Materializer.updateView().
2021-02-09 13:56:50 -05:00
Daniel Nephin 041f96639d api: Use blocking query for health when near is set
Streaming can not be used for these queries because the near query
paramter indicates a specific sort of the results, and that sort
requires data that is not available to the client from the streaming
API.
2021-02-09 13:55:33 -05:00
Pierre Souchay 2c44a731c8 Use lower case for serviceName computation of cache keys 2021-02-09 19:19:40 +01:00
Matt Keeler 19c99dc104
Stop background refresh of cached data for requests that result in ACL not found errors (#9738) 2021-02-09 10:15:53 -05:00
Freddy 5a50b26767
Avoid potential proxycfg/xDS deadlock using non-blocking send 2021-02-08 16:14:06 -07:00
R.B. Boyer 91d9544803
connect: connect CA Roots in the primary datacenter should use a SigningKeyID derived from their local intermediate (#9428)
This fixes an issue where leaf certificates issued in primary
datacenters using Vault as a Connect CA would be reissued very
frequently (every ~20 seconds) because the logic meant to detect root
rotation was errantly triggering.

The hash of the rootCA was being compared against a hash of the
intermediateCA and always failing. This doesn't apply to the Consul
built-in CA provider because there is no intermediate in use in the
primary DC.

This is reminiscent of #6513
2021-02-08 13:18:51 -06:00
Pierre Souchay cec640a1f2 [Streaming] Properly filters node-meta queries on health
This wil fix https://github.com/hashicorp/consul/issues/9730
2021-02-08 17:53:18 +01:00
freddygv a417f88e44 Update comments on avoiding proxycfg deadlock 2021-02-08 09:45:45 -07:00
R.B. Boyer 77424e179a
xds: prevent LDS flaps in mesh gateways due to unstable datacenter lists (#9651)
Also fix a similar issue in Terminating Gateways that was masked by an overzealous test.
2021-02-08 10:19:57 -06:00
Mohammad Banikazemi 45b11c713a Correcting the changed function name in comment
Signed-off-by: Mohammad Banikazemi <mbanikazemi@gmail.com>
2021-02-06 20:23:40 -05:00
freddygv 0a8f2f2105 Retry send after timer fires, in case no updates occur 2021-02-05 18:00:59 -07:00
Daniel Nephin cdda3b9321 state: Use the tableIndex constant 2021-02-05 18:37:45 -05:00
Daniel Nephin de841bd459 state: Document index table
And move the IndexEntry (which is stored in the table) next to the table
schema definition.
2021-02-05 18:37:45 -05:00
R.B. Boyer 05d767b8d6
xds: deduplicate mesh gateway listeners in a stable way (#9650)
In a situation where the mesh gateway is configured to bind to multiple
network interfaces, we use a feature called 'tagged addresses'.
Sometimes an address is duplicated across multiple tags such as 'lan'
and 'lan_ipv4'.

There is code to deduplicate these things when creating envoy listeners,
but that code doesn't ensure that the same tag wins every time. If the
winning tag flaps between xDS discovery requests it will cause the
listener to be drained and replaced.
2021-02-05 16:28:07 -06:00
freddygv 8de6b2590c Make xDS labeling consistent with proxycfg 2021-02-05 15:15:52 -07:00
freddygv 57c29aba5d Update proxycfg logging, labels were already attached 2021-02-05 15:14:49 -07:00
Daniel Nephin 23cfbc8f8d
Merge pull request #9719 from hashicorp/oss/state-store-4
state: remove registerSchema
2021-02-05 14:02:38 -05:00
Daniel Nephin dc70f583d4
Merge pull request #9718 from hashicorp/oss/dnephin/ent-meta-in-state-store-3
state: convert all table name constants to the new prefix pattern
2021-02-05 14:02:07 -05:00
Daniel Nephin eb5d71fd19
Merge pull request #9665 from hashicorp/dnephin/state-store-indexes-2
state: move config-entries table definition to config_entries_schema.go
2021-02-05 14:01:08 -05:00
Daniel Nephin 9beadc578b
Merge pull request #9664 from hashicorp/dnephin/state-store-indexes
state: move ACL schema and index definitions to acl_schema.go
2021-02-05 13:38:31 -05:00
Daniel Nephin b747b27afd state: remove the need for registerSchema
registerSchema creates some indirection which is not necessary in this
case. newDBSchema can call each of the tables.

Enterprise tables can be added from the existing withEnterpriseSchema
shim.
2021-02-05 12:19:56 -05:00
Daniel Nephin 33621706ac state: rename table name constants to use pattern
the 'table' prefix is shorter, and also reads better in queries.
2021-02-05 12:12:19 -05:00
Daniel Nephin 8569295116 state: rename connect constants 2021-02-05 12:12:19 -05:00
Daniel Nephin afdbf2a8ef state: rename table name constants to new pattern
Using Apps Hungarian Notation for these constants makes the memdb queries more readable.
2021-02-05 12:12:18 -05:00
Pierre Souchay c466b08481 Streaming filter tags + case insensitive lookups for Service Names
Will fix:
 * https://github.com/hashicorp/consul/issues/9695
 * https://github.com/hashicorp/consul/issues/9702
2021-02-04 11:00:51 +01:00
Daniel Nephin f929a7117e state: Remove unnecessary entMeta arg to EnsureConfigEntry 2021-02-03 18:10:38 -05:00
freddygv a0be7dcc1d Add trace logs to proxycfg state runner and xds srv 2021-02-02 12:26:38 -07:00
freddygv 0fb96afe31 Avoid potential deadlock using non-blocking send
Deadlock scenario:
    1. Due to scheduling, the state runner sends one snapshot into
    snapCh and then attempts to send a second. The first send succeeds
    because the channel is buffered, but the second blocks.
    2. Separately, Manager.Watch is called by the xDS server after
    getting a discovery request from Envoy. This function acquires the
    manager lock and then blocks on receiving the CurrentSnapshot from
    the state runner.
    3. Separately, there is a Manager goroutine that reads the snapshots
    from the channel in step 1. These reads are done to notify proxy
    watchers, but they require holding the manager lock. This goroutine
    goes to acquire that lock, but can't because it is held by step 2.

Now, the goroutine from step 3 is waiting on the one from step 2 to
release the lock. The goroutine from step 2 won't release the lock until
the goroutine in step 1 advances. But the goroutine in step 1 is waiting
for the one in step 3. Deadlock.

By making this send non-blocking step 1 above can proceed. The coalesce
timer will be reset and a new valid snapshot will be delivered after it
elapses or when one is requested by xDS.
2021-02-02 11:31:14 -07:00
hashicorp-ci 7e7b56e13e auto-updated agent/uiserver/bindata_assetfs.go from commit e0ff7080a 2021-02-02 10:08:48 +00:00
hashicorp-ci d979b67d63 auto-updated agent/uiserver/bindata_assetfs.go from commit 0b7d676dc 2021-02-01 17:55:03 +00:00
hashicorp-ci 083fa1693b auto-updated agent/uiserver/bindata_assetfs.go from commit 3aef5cde2 2021-02-01 17:35:20 +00:00
Kyle Havlovitz 1dee4173c1 connect/ca: Allow ForceWithoutCrossSigning for all providers
This allows setting ForceWithoutCrossSigning when reconfiguring the CA
for any provider, in order to forcibly move to a new root in cases where
the old provider isn't reachable or able to cross-sign for whatever
reason.
2021-01-29 13:38:11 -08:00
hashicorp-ci d156596776 auto-updated agent/uiserver/bindata_assetfs.go from commit 3477b1de7 2021-01-29 16:03:41 +00:00
Daniel Nephin 09425b22a1 state: rename config-entries table const to match new pattern 2021-01-28 20:34:34 -05:00
Daniel Nephin 7d17e20270 state: move config-entries table to new pattern 2021-01-28 20:34:15 -05:00
Daniel Nephin 825b8ade39 state: use indexID
this change was already made to enterprise, so backporting it.
2021-01-28 20:30:08 -05:00
Daniel Nephin 2a262f07fc state: Move ACL schema indexes to match Ent
and use constants for table and index names.
2021-01-28 20:05:09 -05:00
Daniel Nephin 2eea58bcc4
Merge pull request #9302 from hashicorp/dnephin/add-service-3
agent: remove ServiceManager.Start goroutine
2021-01-28 16:59:41 -05:00
Daniel Nephin fc88c28bb9 config: make config.TestLoad_FullConfig use config.Load
This commit makes a number of changes that should make
TestLoad_FullConfig easier to work with, and make the test more like
real world scenarios.

* use separate files in testdata/ dir to store the config source.
  Separate files are much easier to edit because editors can syntax
  highlight json/hcl, and it makes strings easier to find. Previously
  trying to find strings would match strings used in other tests.
* use the exported config.Load interface instead of internal NewBuilder
  and BuildAndValidate.
* remove the tail config overrides, which are only necessary with
  nonZero works.
2021-01-27 17:51:53 -05:00
Daniel Nephin aa42ff5d81 config: Unexport Builder and NewBuilder
This type and constructor are implementation details of config loading.
All callers should use config.Load.
2021-01-27 17:41:53 -05:00
Daniel Nephin 08b2039f5a config: replace calls to config.NewBuilder with config.Load
This is another incremental change to reduce config loading to a single
small interface. All calls to NewBuilder can be replaced with Load.
2021-01-27 17:34:43 -05:00
Daniel Nephin bd545963cc config: improve the interface of Load
This commit reduces the interface to Load() a bit, in preparation for
unexporting NewBuilder and having everything call Load.

The three arguments are reduced to a single argument by moving the other
two into the options struct.

The three return values are reduced to two by moving the RuntimeConfig
and Warnings into a LoadResult struct.
2021-01-27 17:34:43 -05:00
Daniel Nephin 63ba836257
Merge pull request #9252 from hashicorp/dnephin/config-unmethod
config: remove Builder receiver from funcs that dont use it
2021-01-27 17:31:17 -05:00
Matt Keeler 1379b5f7d6
Upgrade raft-autopilot and wait for autopilot it to stop when revoking leadership (#9644)
Fixes: 9626
2021-01-27 11:14:52 -05:00
hashicorp-ci 614c57a9c6 auto-updated agent/uiserver/bindata_assetfs.go from commit 25f989753 2021-01-27 10:47:58 +00:00
Hans Hasselberg 623aab5880
Add flags to support CA generation for Connect (#9585) 2021-01-27 08:52:15 +01:00
hashicorp-ci 773f1c6b21 auto-updated agent/uiserver/bindata_assetfs.go from commit 92f0eb3bd 2021-01-26 18:00:09 +00:00
hashicorp-ci 68f65fd4a9 auto-updated agent/uiserver/bindata_assetfs.go from commit 82a62cd2e 2021-01-26 17:47:18 +00:00
Daniel Nephin 18fcce575a
Merge pull request #9301 from hashicorp/dnephin/add-service-2
agent: reduce AddService 2
2021-01-26 12:01:34 -05:00
R.B. Boyer 5777fa1f59
server: initialize mgw-wanfed to use local gateways more on startup (#9528)
Fixes #9342
2021-01-25 17:30:38 -06:00
Daniel Nephin e1e94ce69c agent: use the new lib/mutex for stateLock
Previously the ServiceManager had to run a separate goroutine so that it could block on a channel
send/receive instead of a lock. Using this mutex with TryLock allows us to cancel the lock when
the serviceConfigWatch is stopped.

Without this change removing the ServiceManager.Start goroutine would not be possible because
when AddService is called it acquires the stateLock. While that lock is held, if there are
existing watches for the service, the old watch will be stopped, and the goroutine holding the
lock will attempt to wait for that watcher goroutine to exit.

If the goroutine is handling an update (serviceConfigWatch.handleUpdate) then it can block on
acquiring the stateLock and deadlock the agent.  With this change the context is cancelled as
and the goroutine will exit instead of waiting on the stateLock.
2021-01-25 18:01:47 -05:00
Daniel Nephin 08e8ed0a7e agent: remove ServiceManager goroutine
The ServiceManager.Start goroutine was used to serialize calls to
agent.addServiceInternal.

All the goroutines which sent events to the channel would block waiting
for a response from that same goroutine, which is effectively the same
as a synchronous call without any channels.

This commit removes the goroutine and channels, and instead calls
addServiceInternal directly. Since all of these goroutines will need to
take the agent.stateLock, the mutex handles the serializing of calls.
2021-01-25 18:01:47 -05:00
Daniel Nephin 081173652c agent: Minor cosmetic changes in ServiceManager
Also use the non-deprecated func in a test
2021-01-25 18:01:47 -05:00
Daniel Nephin 627371bc37 agent: update godoc for AddServiceRequest 2021-01-25 18:01:03 -05:00
Daniel Nephin 121fd18f9a agent: move checkStateSnapshot
Move the field into the struct for addServiceLocked. Also don't require
setting a default value, so that the callers can leave it as nil if they
don't already have a snapshot.
2021-01-25 18:01:03 -05:00
Daniel Nephin 4d0dd9f5ff agent: move two fields off of AddServiceRequest 2021-01-25 18:01:03 -05:00
Daniel Nephin 76ba6c2198 agent: Replace two fields on AddServiceRequest with a func field
The two previous fields were mutually exclusive. They can be represented
with a single function which provides the value.
2021-01-25 18:01:03 -05:00
Daniel Nephin 5e31bdf51a agent: remove serviceRegiration type
Replace with the existing AddServiceRequest struct. These structs are
almost identical. Additionally, the only reason the serviceRegistration
struct existed was to recreate an AddServiceRequest.

By storing and re-using the AddServiceRequest we remove the need to
translate into one type and back to the original type.

We also remove the extra parameters to a function, because those values
are already available from the AddServiceRequest field.

Also a minor optimization to only call tokens.AgentToken() when
necessary. Previous it was being called every time, but the value was
being ignored if the AddServiceRequest had a token.
2021-01-25 18:01:03 -05:00
Daniel Nephin 75b2c55291 agent: remove an a branch in the AddService flow
Handle the decision to use ServiceManager in a single place. Instead of
calling ServiceManager.AddService, then calling back into
addServiceInternal, only call ServiceManager.AddService if we are going
to use it.

This change removes some small duplication and removes a branch from the
AddService flow.
2021-01-25 18:01:03 -05:00
Daniel Nephin 1ce7cdd892 agent: use fields directly, not temp variables
The temprorary variables make it much harder to trace where and how struct
fields are used. If a field is only used a small number of times than
refer to the field directly.
2021-01-25 17:25:04 -05:00
Daniel Nephin 493e987a88 agent: addServiceIternalRequest
Move fields that are only relevant for addServiceInternal onto a new
struct and embed AddServiceRequest.
2021-01-25 17:25:04 -05:00
Daniel Nephin a0b11b4c20 agent: move deprecated AddServiceFromSource to a test file
The method is only used in tests, and only exists for legacy calls.

There was one other package which used this method in tests. Export
the AddServiceRequest and a couple of its fields so the new function can
be used in those tests.
2021-01-25 17:25:03 -05:00
Daniel Nephin de1a80b368 agent: use a single method for Agent.AddService 2021-01-25 17:25:03 -05:00
Daniel Nephin 1c8eca2bfd agent: rename AddService->AddServiceFromSource
In preparation for extracting a single AddService func that accepts a request struct.
2021-01-25 17:25:01 -05:00
Daniel Nephin d7d081f402
Merge pull request #9420 from hashicorp/dnephin/reduce-duplicate-in-catalog-schema
state: reduce interface for Enterprise schema
2021-01-25 17:04:25 -05:00
Chris Boulton 448212060a
connect: add local_request_timeout_ms to configure local_app http timeouts (#9554) 2021-01-25 13:50:00 -06:00
R.B. Boyer 6622185d64
server: use the presense of stored federation state data as a sign that we already activated the federation state feature flag (#9519)
This way we only have to wait for the serf barrier to pass once before
we can make use of federation state APIs Without this patch every
restart needs to re-compute the change.
2021-01-25 13:24:32 -06:00
hashicorp-ci 7c8cfb7a2a auto-updated agent/uiserver/bindata_assetfs.go from commit 09d917618 2021-01-25 19:09:57 +00:00
hashicorp-ci b9d7f04760 auto-updated agent/uiserver/bindata_assetfs.go from commit 88d5e00f9 2021-01-25 18:52:54 +00:00
hashicorp-ci eefa22b79b auto-updated agent/uiserver/bindata_assetfs.go from commit bb9573832 2021-01-25 18:19:10 +00:00
R.B. Boyer e3f5a77ade
server: add OSS stubs supporting validation of source namespaces in service-intentions config entries (#9527) 2021-01-25 11:27:38 -06:00
R.B. Boyer 0247f409a0
server: when wan federating via mesh gateways only do heuristic primary DC bypass on the leader (#9366)
Fixes #9341
2021-01-22 10:03:24 -06:00
hashicorp-ci cae9b96b00 auto-updated agent/uiserver/bindata_assetfs.go from commit 02772e46a 2021-01-20 18:46:55 +00:00
John Cowen d3ecb6d7a0
Fix -ui-content-path without regex (#9569)
* Add templating to inject JSON into an application/json script tag

Plus an external script in order to pick it out and inject the values we
need injecting into ember's environment meta tag.

The UI still uses env style naming (CONSUL_*) but we uses the new style
JSON/golang props behind the scenes.

Co-authored-by: Paul Banks <banks@banksco.de>
2021-01-20 18:40:46 +00:00
hashicorp-ci 30d26fa90e auto-updated agent/uiserver/bindata_assetfs.go from commit ffb6680e7 2021-01-20 17:08:19 +00:00
John Cowen 82a5e6d6fc
api: Ensure the internal/ui/gateway-service-nodes endpoint responds with an array (#9593)
In some circumstances this endpoint will have no results in it (dues to
ACLs, Namespaces, filtering or missing configuration).

This ensures that the response is at least an empty array (`[]`) rather
than `null`
2021-01-20 16:59:02 +00:00
Matt Keeler d871a6f260
Fix flaky test by marking mock expectations as optional (#9596)
These expectations are optional because in a slow CI environment the deadline to cancell the context might occur before the go routine reaches issuing the RPC. Either way we are successfully ensuring context cancellation is working.
2021-01-20 10:58:27 -05:00
hashicorp-ci 884a2d61dc auto-updated agent/uiserver/bindata_assetfs.go from commit 30014ff8f 2021-01-20 15:43:19 +00:00
Freddy 5519051c84
Update topology mapping Refs on all proxy instance deletions (#9589)
* Insert new upstream/downstream mapping to persist new Refs

* Avoid upserting mapping copy if it's a no-op

* Add test with panic repro

* Avoid deleting up/downstreams from inside memdb iterator

* Avoid deleting gateway mappings from inside memdb iterator

* Add CHANGELOG entry

* Tweak changelog entry

Co-authored-by: Paul Banks <banks@banksco.de>
2021-01-20 15:17:26 +00:00
Daniel Nephin 979749d86e state: do not delete from inside an iteration
Deleting from memdb inside an interation can cause a panic from Iterator.Next. This
case is technically safe (for now) because the iterator is using the root radix tree
not a modified one.

However this could break at any time if someone adds an insert or delete to the coordinates table
before this place in the function.

It also sets a bad example, because generally deletes in an interator are not safe. So this
commit uses the pattern we have in other places to move the deletes out of the iteration.
2021-01-19 17:00:07 -05:00
Matt Keeler 2d7a4073e3
Merge pull request #9570 from hashicorp/bugfix/9498 2021-01-19 16:30:04 -05:00
Matt Keeler 2d2ce1fb0c
Ensure that CA initialization does not block leader election.
After fixing that bug I uncovered a couple more:

Fix an issue where we might try to cross sign a cert when we never had a valid root.
Fix a potential issue where reconfiguring the CA could cause either the Vault or AWS PCA CA providers to delete resources that are still required by the new incarnation of the CA.
2021-01-19 15:27:48 -05:00
hashicorp-ci 4b3361fb03 auto-updated agent/uiserver/bindata_assetfs.go from commit be694366a 2021-01-19 15:47:02 +00:00
hashicorp-ci 88ab4cd159 auto-updated agent/uiserver/bindata_assetfs.go from commit 41a4a9f4f 2021-01-19 15:29:55 +00:00
Daniel Nephin 52a1d78e39 state: add a regression test for state store schema
To allow the index to be refactored without accidental changes.

To update the expected value run: 'go test ./agent/consul/state -update'
2021-01-15 18:49:55 -05:00