Commit Graph

4469 Commits

Author SHA1 Message Date
Freddy 137bfbb02b Update public listener with SPIFFE Validator
Envoy's SPIFFE certificate validation extension allows for us to
validate against different root certificates depending on the trust
domain of the dialing proxy.

If there are any trust bundles from peers in the config snapshot then we
use the SPIFFE validator as the validation context, rather than the
usual TrustedCA.

The injected validation config includes the local root certificates as
well.
2022-06-01 17:06:33 -06:00
freddygv ad6dbe081a Add agent cache-type for TrustBundleListByService
There are a handful of changes in this commit:
* When querying trust bundles for a service we need to be able to
  specify the namespace of the service.
* The endpoint needs to track the index because the cache watches use
  it.
* Extracted bulk of the endpoint's logic to a state store function
  so that index tracking could be tested more easily.
* Removed check for service existence, deferring that sort of work to ACL authz
* Added the cache type
2022-06-01 17:05:10 -06:00
freddygv 073c9e3a91 Update assumptions around exported-service config
Given that the exported-services config entry can use wildcards, the
precedence for wildcards is handled as with intentions. The most exact
match is the match that applies for any given service. We do not take
the union of all that apply.

Another update that was made was to reflect that only one
exported-services config entry applies to any given service in a
partition. This is a pre-existing constraint that gets enforced by
the Normalize() method on that config entry type.
2022-06-01 17:03:51 -06:00
Freddy 6ef38eaea7
Configure upstream TLS context with peer root certs (#13321)
For mTLS to work between two proxies in peered clusters with different root CAs,
proxies need to configure their outbound listener to use different root certificates
for validation.

Up until peering was introduced proxies would only ever use one set of root certificates
to validate all mesh traffic, both inbound and outbound. Now an upstream proxy
may have a leaf certificate signed by a CA that's different from the dialing proxy's.

This PR makes changes to proxycfg and xds so that the upstream TLS validation
uses different root certificates depending on which cluster is being dialed.
2022-06-01 15:53:52 -06:00
R.B. Boyer addfef5065
test: regenerate golden files (#13336)
make envoy-regen
    go test ./agent/config -update
2022-06-01 15:17:03 -05:00
Chris S. Kim 58ffa0488d
Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
Dan Upton e6dc26e087
proxycfg: replace direct agent cache usage with interfaces (#13320)
This is the OSS portion of enterprise PRs 1904, 1905, 1906, 1907, 1949,
and 1971.

It replaces the proxycfg manager's direct dependency on the agent cache
with interfaces that will be implemented differently when serving xDS
sessions from a Consul server.
2022-06-01 16:18:06 +01:00
Chris S. Kim 44a318ef73
Reimplement fs.FileInfo interface (#13315)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-06-01 11:09:51 -04:00
Dhia Ayachi d4a04457e1
update gateway-services table with endpoints (#13217)
* update gateway-services table with endpoints

* fix failing test

* remove unneeded config in test

* rename "endpoint" to "destination"

* more endpoint renaming to destination in tests

* update isDestination based on service-defaults config entry creation

* use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist)

* set unknown state to empty to avoid modifying alot of tests

* fix logic to set the kind correctly on CRUD

* fix failing tests

* add missing tests and fix service delete

* fix failing test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* fix a bug with kind and add relevant test

* fix compile error

* fix failing tests

* add kind to clone

* fix failing tests

* fix failing tests in catalog endpoint

* fix service dump test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* remove duplicate tests

* rename consts and fix kind when no destination is defined in the service-defaults.

* rename Kind to ServiceKind and change switch to use .(type)

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
2022-05-31 16:20:12 -04:00
Chris S. Kim ea1e4aa52d
Update repo to use go:embed (#10996)
Replace bindata packages with stdlib go:embed.
Modernize some uiserver code with newer interfaces introduced in go 1.16 (mainly working with fs.File instead of http.File.
Remove steps that are no longer used from our build files.
Add Github Action to detect differences in agent/uiserver/dist and verify that the files are correct (by compiling UI assets and comparing contents).
2022-05-31 15:33:56 -04:00
Riddhi Shah d558914a0f
[OSS] Fix merge central config tests (#13309)
Setting the right enterprise meta to fix the merge central config tests.
Re-added the tests that were failing on the OSS to ENT merge.
2022-05-31 12:04:19 -07:00
freddygv 14bff4fba6 Use embedded SpiffeID for peered upstreams 2022-05-31 09:55:37 -06:00
freddygv 4d3e09e8f8 Remove intermediate representation of SPIFFE IDs
xDS only ever uses the string representation, so we can avoid passing
around connect.SpiffeIDService objects around.
2022-05-31 09:55:37 -06:00
freddygv 5cd5108075 Return SPIFFE ID for connect proxies in PeerMeta
Proxies dialing exporting services need to know the SPIFFE ID of
services dialed so that the upstream's SANs can be validated.

This commit attaches the SPIFFE ID to all connect proxies exported over
the peering stream so that they are available to importing clusters.

The data in the SPIFFE ID cannot be re-constructed in peer clusters
because the partition of exported services is overwritten on imports.
2022-05-31 09:55:37 -06:00
Freddy a75af9d94a
[OSS] Add grpc endpoint to fetch a specific trust bundle (#13292)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-05-31 09:54:40 -06:00
Matt Keeler b9e8b5c692
Fix a flaky test (#13282)
At the end of this test we were trying to ensure that updating a service in the local state causes it to re-register the service with the config manager.

The config manager in the same method will also call RegisteredProxies to determine if any need to be removed. This portion of the test is not attempting to verify that behavior.

Because the test is only blocked waiting for the Register event before it can end and assert all the mock expectations were met, we may not see the call to RegisteredProxies. This is especially apparent when tests are run with the race detector.

As we don’t actually care if that method is executed before the end of the test we can simply transition from expecting it to be called exactly once to a 0 or 1 times assertion.
2022-05-27 13:25:08 -04:00
Dan Upton a6a6d5a8ee
Enable servers to configure arbitrary proxies from the catalog (#13244)
OSS port of enterprise PR 1822

Includes the necessary changes to the `proxycfg` and `xds` packages to enable
Consul servers to configure arbitrary proxies using catalog data.

Broadly, `proxycfg.Manager` now has public methods for registering,
deregistering, and listing registered proxies — the existing local agent
state-sync behavior has been moved into a separate component that makes use of
these methods.

When an xDS session is started for a proxy service in the catalog, a goroutine
will be spawned to watch the service in the server's state store and
re-register it with the `proxycfg.Manager` whenever it is updated (and clean
it up when the client goes away).
2022-05-27 12:38:52 +01:00
alex 2d8664d384
monitor leadership in peering service (#13257)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2022-05-26 17:55:16 -07:00
Riddhi Shah 8714ade534
Termporarily disable validation of merge central config response (#13266)
Temporarily disabling the validation of merge central config response since
it is breaking OSS to ENT merging.
A follow up PR will patch the fixes.
2022-05-26 13:49:40 -07:00
Chris S. Kim d73a9522cb
Add support for streaming CA roots to peers (#13260)
Sender watches for changes to CA roots and sends
them through the replication stream. Receiver saves
CA roots to tablePeeringTrustBundle
2022-05-26 15:24:09 -04:00
Riddhi Shah 6f57acc1bf
Remove tests failing on ent (#13255)
Will follow up with the fixed version of these tests that passes in ent.
2022-05-26 10:17:59 -07:00
John Cowen bf5f1482fd
Export top-level HCP Enabled go-template variable for UI (#13165)
* Update ui template data to export HCPEnabled at the top level
2022-05-26 17:23:56 +01:00
DanStough 65ca7e0bfb fix: multiple grpc/http2 services for ingress listeners 2022-05-26 10:43:58 -04:00
Riddhi Shah e5f1d8dce4
Add support for merge-central-config query param (#13001)
Adds a new query param merge-central-config for use with the below endpoints:

/catalog/service/:service
/catalog/connect/:service
/health/service/:service
/health/connect/:service

If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).
2022-05-25 13:20:17 -07:00
R.B. Boyer 4f9a9bb851
remove a source of test panics (#13227) 2022-05-25 14:33:00 -05:00
R.B. Boyer dae47101fa
api: ensure peering API endpoints do not use protobufs (#13204)
I noticed that the JSON api endpoints for peerings json encodes protobufs directly, rather than converting them into their `api` package equivalents before marshal/unmarshaling them.

I updated this and used `mog` to do the annoying part in the middle. 

Other changes:
- the status enum was converted into the friendlier string form of the enum for readability with tools like `curl`
- some of the `api` library functions were slightly modified to match other similar endpoints in UX (cc: @ndhanushkodi )
- peeringRead returns `nil` if not found
- partitions are NOT inferred from the agent's partition (matching 1.11-style logic)
2022-05-25 13:43:35 -05:00
R.B. Boyer bc10055edc
peering: replicate expected SNI, SPIFFE, and service protocol to peers (#13218)
The importing peer will need to know what SNI and SPIFFE name
corresponds to each exported service. Additionally it will need to know
at a high level the protocol in use (L4/L7) to generate the appropriate
connection pool and local metrics.

For replicated connect synthetic entities we edit the `Connect{}` part
of a `NodeService` to have a new section:

    {
      "PeerMeta": {
        "SNI": [
          "web.default.default.owt.external.183150d5-1033-3672-c426-c29205a576b8.consul"
        ],
        "SpiffeID": [
          "spiffe://183150d5-1033-3672-c426-c29205a576b8.consul/ns/default/dc/dc1/svc/web"
        ],
        "Protocol": "tcp"
      }
    }

This data is then replicated and saved as-is at the importing side. Both
SNI and SpiffeID are slices for now until I can be sure we don't need
them for how mesh gateways will ultimately work.
2022-05-25 12:37:44 -05:00
R.B. Boyer 69191fc0da
peering: disable requirement for mesh gateways initially (#13213) 2022-05-25 10:13:23 -05:00
Kyle Havlovitz cebf7b23f6
Merge pull request #13143 from hashicorp/envoy-connection-limit
Add connection limit setting to service defaults
2022-05-25 07:48:50 -07:00
Kyle Havlovitz f5f949d486 Fix proto lint errors after version bump 2022-05-24 18:44:54 -07:00
Kyle Havlovitz 749591ec98 Specify go_package explicitly 2022-05-24 10:22:53 -07:00
cskh b7eec4c05b
fix: non-leader agents return 404 on Get Intention exact api (#13179)
* fix: non-leader agents return 404 on Get Intention exact api

- rpc call method appends extra error message, so change == to
  "Strings.Contains"

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-05-24 13:21:15 -04:00
Kyle Havlovitz 03dea180ad Add connection limit setting to service defaults 2022-05-24 10:13:38 -07:00
DanStough 2c8ca25d8a chore(test): Update bats version 2022-05-24 11:56:08 -04:00
DanStough df59d8ab0d feat: add endpoint struct to ServiceConfigEntry 2022-05-24 11:56:08 -04:00
alex 451dc50f4f
peering: expose IsLeader, hung up on dialer if follower (#13164)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-05-23 11:30:58 -07:00
Matt Keeler 1fd02a13c2
Migrate from `protoc` to `buf` (#12841)
* Install `buf` instead of `protoc`
* Created `buf.yaml` and `buf.gen.yaml` files in the two proto directories to control how `buf` generates/lints proto code.
* Invoke `buf` instead of `protoc`
* Added a `proto-format` make target.
* Committed the reformatted proto files.
* Added a `proto-lint` make target.
* Integrated proto linting with CI
* Fixed tons of proto linter warnings.
* Got rid of deprecated builtin protoc-gen-go grpc plugin usage. Moved to direct usage of protoc-gen-go-grpc.
* Unified all proto directories / go packages around using pb prefixes but ensuring all proto packages do not have the prefix.
2022-05-23 10:37:52 -04:00
cskh 39cb731988
Upgrade golangci-lint for go v1.18 (#13176) 2022-05-23 10:26:45 -04:00
R.B. Boyer 3b12a5179f
test: fix flaky test TestEventBufferFuzz (#13175) 2022-05-23 09:22:30 -05:00
Matt Keeler c629e89289
Fix tests broken in #13173 (#13178)
I changed the error type returned in a situation but didn’t update the tests to expect that error.
2022-05-23 10:00:06 -04:00
Matt Keeler 8a968299dd
Fix flaky tests in the agent/grpc/public/services/serverdiscovery package (#13173)
Occasionally we had seen the TestWatchServers_ACLToken_PermissionDenied be flagged as flaky in circleci. This change should fix that.

Why it fixes it is complicated. The test was failing with a panic when a mocked ACL Resolver was being called more times than expected. I struggled for a while to determine how that could be. This test should call authorize once and only once and the error returned should cause the stream to be terminated and the error returned to the gRPC client. Another oddity was no amount of running this test locally seemed to be able to reproduce the issue. I ran the test hundreds of thousands of time and it always passed.

It turns out that there is nothing wrong with the test. It just so happens that the panic from unexpected invocation of a mocked call happened during the test but was caused by a previous test (specifically the TestWatchServers_StreamLifecycle test)

The stream from the previous test remained open after all the test Cleanup functions were run and it just so happened that when the EventPublisher eventually picked up that the context was cancelled during cleanup, it force closes all subscriptions which causes some loops to be re-entered and the streams to be reauthorized. Its that looping in response to forced subscription closures that causes the mock to eventually panic. All the components, publisher, server, client all operate based on contexts. We cancel all those contexts but there is no syncrhonous way to know when they are stopped.

We could have implemented a syncrhonous stop but in the context of an actual running Consul, context cancellation + async stopping is perfectly fine. What we (Dan and I) eventually thought was that the behavior of grpc streams such as this when a server was shutting down wasn’t super helpful. What we would want is for a client to be able to distinguish between subscription closed because something may have changed requiring re-authentication and subscription closed because the server is shutting down. That way we can send back appropriate error messages to detail that the server is shutting down and not confuse users with potentially needing to resubscribe.

So thats what this PR does. We have introduced a shutting down state to our event subscriptions and the various streaming gRPC services that rely on the event publisher will all just behave correctly and actually stop the stream (not attempt transparent reauthorization) if this particular error is the one we get from the stream. Additionally the error that gets transmitted back through gRPC when this does occur indicates to the consumer that the server is going away. That is more helpful so that a client can then attempt to reconnect to another server.
2022-05-23 08:59:13 -04:00
R.B. Boyer 69d3e729a4
agent: allow for service discovery queries involving peer name to use streaming (#13168) 2022-05-20 15:27:01 -05:00
Dan Upton 30775ed54d
proxycfg: remove dependency on `cache.UpdateEvent` (#13144)
OSS portion of enterprise PR 1857.

This removes (most) references to the `cache.UpdateEvent` type in the
`proxycfg` package.

As we're going to be direct usage of the agent cache with interfaces that
can be satisfied by alternative server-local datasources, it doesn't make
sense to depend on this type everywhere anymore (particularly on the
`state.ch` channel).

We also plan to extract `proxycfg` out of Consul into a shared library in
the future, which would require removing this dependency.

Aside from a fairly rote find-and-replace, the main change is that the
`cache.Cache` and `health.Client` types now accept a callback function
parameter, rather than a `chan<- cache.UpdateEvents`. This allows us to
do the type conversion without running another goroutine.
2022-05-20 15:47:40 +01:00
R.B. Boyer 63a9175bd6
peering: accept replication stream of discovery chain information at the importing side (#13151) 2022-05-19 16:37:52 -05:00
R.B. Boyer 68789effeb
test: TestServer_RPC_MetricsIntercept should use a concurrency-safe metrics store (#13157) 2022-05-19 15:39:28 -05:00
cskh df27fa0c84
Retry on bad dogstatsd connection (#13091)
- Introduce a new telemetry configurable parameter retry_failed_connection. User can set the value to true to let consul agent continue its start process on failed connection to datadog server. When set to false, agent will stop on failed start. The default behavior is true.

Co-authored-by: Dan Upton <daniel@floppy.co>
Co-authored-by: Evan Culver <eculver@users.noreply.github.com>
2022-05-19 16:03:46 -04:00
R.B. Boyer 91691eca87 peering: replicate discovery chains information to importing peers
Treat each exported service as a "discovery chain" and replicate one
synthetic CheckServiceNode for each chain and remote mesh gateway.

The health will be a flattened generated check of the checks for that
mesh gateway node.
2022-05-19 14:21:44 -05:00
R.B. Boyer bf05e8c1f1 prefactor some functions out of the monolithic file 2022-05-19 14:21:29 -05:00
R.B. Boyer 09861a2792
test: fix incorrect use of t instead of r in retry test (#13146) 2022-05-19 14:00:07 -05:00
Dan Upton 7492357b43
config: prevent top-level `verify_incoming` enabling mTLS on gRPC port (#13118)
Fixes #13088

This is a backwards-compatibility bug introduced in 1.12.
2022-05-18 16:15:57 +01:00
Freddy 6c868b6c0e
Patches to peering initiation for POC demo (#13076)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-05-13 13:01:00 -06:00
Dhia Ayachi 70b93ea693
When a host header is defined override `req.Host` in the metrics ui (#13071)
* When a host header is defined override the req.Host in the metrics ui endpoint.

* add changelog
2022-05-13 14:05:22 -04:00
Freddy 160acdf876
Actually block when syncing subscriptions (#13066)
By changing to use WatchCtx we will actually block for changes to the peering list. WatchCh creates a goroutine to collect errors from WatchCtx and returns immediately.

The existing behavior wouldn't result in a tight loop because of the rate limiting in the surrounding function, but it would still lead to more work than is necessary.
2022-05-12 17:36:14 -06:00
Evan Culver 535e811020
peering: add TrustBundleListByService endpoint (#13048) 2022-05-12 15:58:22 -07:00
Freddy 8894365c5a
[OSS] Add upsert handling for receiving CheckServiceNode (#13061) 2022-05-12 15:04:44 -06:00
Matt Keeler 42aec5caf4
Watch the singular service resolver instead of the list + filtering to 1 (#13012)
* Watch the singular service resolver instead of the list + filtering to 1

* Rename the ConfigEntries cache type to ConfigEntryList
2022-05-12 16:34:17 -04:00
R.B. Boyer 26cfdc1967
structs: add convenience methods to sort slices of ServiceName values (#13038) 2022-05-12 10:08:50 -05:00
R.B. Boyer b932d0dabc
test: ensure this package uses freeport for port allocation (#13036) 2022-05-11 14:20:50 -05:00
R.B. Boyer c855df87ec
remove remaining shim runStep functions (#13015)
Wraps up the refactor from #13013
2022-05-10 16:24:45 -05:00
R.B. Boyer 9ad10318cd
add general runstep test helper instead of copying it all over the place (#13013) 2022-05-10 15:25:51 -05:00
Jared Kirschner a18d7092cb
Merge pull request #12463 from hashicorp/docs/consistency-mode-improvements
Improve consistency mode docs
2022-05-09 23:04:00 -04:00
Jared Kirschner bc69d70a55 docs: clarify consistency mode operation
Changes include:
- Add diagrams of the operation of different consistency modes
- Note that only stale reads benefit from horizontal scaling
- Increase scannability with headings
- Document consistency mode defaults and how to override for
  DNS and HTTP API interfaces
- Document X-Consul-Effective-Consistency response header
2022-05-09 16:39:48 -07:00
FFMMM ea083062d2
add err msg on PeeringRead not found (#12986)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2022-05-09 15:22:42 -07:00
FFMMM 76a6647700
expose meta tags for peering (#12964) 2022-05-09 13:47:37 -07:00
Mark Anderson 785a6d9c8e Add oss test
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-09 10:07:19 -07:00
Mark Anderson be059184dd Fix up enterprise version tag.
Changes to how the version string was handled created small regression with the release of consul 1.12.0 enterprise.

Many tools use the Config:Version field reported by the agent/self resource to determine whether Consul is an enterprise or OSS instance, expect something like 1.12.0+ent for enterprise and simply 1.12.0 for OSS. This was accidentally broken during the runup to 1.12.x

This work fixes the value returned by both the self endpoint in ["Config"]["Version"] and the metrics consul.version field.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-09 10:07:19 -07:00
Evan Culver d64726c8e9
peering: add store.PeeringsForService implementation (#12957) 2022-05-06 12:35:31 -07:00
Eric Haberkorn fa6e17bdfd
Merge pull request #12956 from hashicorp/suport-lambda-connect-proxy
Support Invoking Lambdas from Sidecar Proxies
2022-05-06 08:17:38 -04:00
Eric 999fc2dbd5 Support making requests to lambda from connect proxies. 2022-05-05 17:42:30 -04:00
FFMMM 5a6d1cad4e
api: add PeeeringList, polish (#12934) 2022-05-05 14:15:42 -07:00
Riddhi Shah a9c96b6975
Validate port on mesh service registration (#12881)
Add validation to ensure connect native services have a port or socketpath specified on catalog registration.
This was the only missing piece to ensure all mesh services are validated for a port (or socketpath) specification on catalog registration.
2022-05-05 09:13:30 -07:00
Mark Anderson 18193f2916
Support vault namespaces in connect CA (#12904)
* Support vault namespaces in connect CA

Follow on to some missed items from #12655

From an internal ticket "Support standard "Vault namespace in the
path" semantics for Connect Vault CA Provider"

Vault allows the namespace to be specified as a prefix in the path of
a PKI definition, but our usage of the Vault API includes calls that
don't support a namespaced key. In particular the sys.* family of
calls simply appends the key, instead of prefixing the namespace in
front of the path.

Unfortunately it is difficult to reliably parse a path with a
namespace; only vault knows what namespaces are present, and the '/'
separator can be inside a key name, as well as separating path
elements. This is in use in the wild; for example
'dc1/intermediate-key' is a relatively common naming schema.

Instead we add two new fields: RootPKINamespace and
IntermediatePKINamespace, which are the absolute namespace paths
'prefixed' in front of the respective PKI Paths.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 19:41:55 -07:00
Chris S. Kim e55aac9d30
Default discovery chain when upstream targets a DestinationPeer (#12942) 2022-05-04 16:25:25 -04:00
Mark Anderson 1497421b65
Merge pull request #12878 from hashicorp/ma/x-forwarded-client-cert
Support x-forwarded-client-cert
2022-05-04 11:05:44 -07:00
Dan Upton 6bfdb48560
acl: gRPC login and logout endpoints (#12935)
Introduces two new public gRPC endpoints (`Login` and `Logout`) and
includes refactoring of the equivalent net/rpc endpoints to enable the
majority of logic to be reused (i.e. by extracting the `Binder` and
`TokenWriter` types).

This contains the OSS portions of the following enterprise commits:

- 75fcdbfcfa6af21d7128cb2544829ead0b1df603
- bce14b714151af74a7f0110843d640204082630a
- cc508b70fbf58eda144d9af3d71bd0f483985893
2022-05-04 17:38:45 +01:00
Mark Anderson 13f5a1f6a8 Fix tests for APPEND_FORWARD change
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 08:50:59 -07:00
Mark Anderson 69c129c73f Change to use APPEND_FORWARD for terminating gateway
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 08:50:59 -07:00
Mark Anderson db0c61303f Update mesh config tests
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 08:50:59 -07:00
Mark Anderson e6282c7c64 Docs and changelog edits
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 08:50:59 -07:00
Mark Anderson c6dbc34172 Fixup missed config entry
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 08:50:59 -07:00
Mark Anderson d8f4cc5537 Add x-forwarded-client-cert headers
Description
Add x-fowarded-client-cert information on trusted incoming connections.

Envoy provides support forwarding and annotating the
x-forwarded-client-cert header via the forward_client_cert_details
set_current_client_cert_details filter fields. It would be helpful for
consul to support this directly in its config. The escape hatches are
a bit cumbersome for this purpose.

This has been implemented on incoming connections to envoy. Outgoing
(from the local service through the sidecar) will not have a
certificate, and so are left alone.

A service on an incoming connection will now get headers something like this:

```
X-Forwarded-Client-Cert:[By=spiffe://efad7282-d9b2-3298-f6d8-38b37fb58df3.consul/ns/default/dc/dc1/svc/counting;Hash=61ad5cbdfcb50f5a3ec0ca60923d61613c149a9d4495010a64175c05a0268ab2;Cert="-----BEGIN%20CERTIFICATE-----%0AMIICHDCCAcOgAwIBAgIBCDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktMTli%0AYXdyb2YuY29uc3VsLmNhLmVmYWQ3MjgyLmNvbnN1bDAeFw0yMjA0MjkwMzE0NTBa%0AFw0yMjA1MDIwMzE0NTBaMAAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARVIZ7Y%0AZEXfbOGBfxGa7Vuok1MIng%2FuzLQK2xLVlSTIPDbO5hstTGP%2B%2FGx182PYFP3jYqk5%0Aq6rYWe1wiPNMA30Io4H8MIH5MA4GA1UdDwEB%2FwQEAwIDuDAdBgNVHSUEFjAUBggr%0ABgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH%2FBAIwADApBgNVHQ4EIgQgrp4q50oX%0AHHghMbxz5Bk8OJFWMdfgH0Upr350WlhyxvkwKwYDVR0jBCQwIoAgUe6uERAIj%2FLM%0AyuFzDc3Wbp9TGAKBJYAwyhF14ToOQCMwYgYDVR0RAQH%2FBFgwVoZUc3BpZmZlOi8v%0AZWZhZDcyODItZDliMi0zMjk4LWY2ZDgtMzhiMzdmYjU4ZGYzLmNvbnN1bC9ucy9k%0AZWZhdWx0L2RjL2RjMS9zdmMvZGFzaGJvYXJkMAoGCCqGSM49BAMCA0cAMEQCIDwb%0AFlchufggNTijnQ5SUcvTZrWlZyq%2FrdVC20nbbmWLAiAVshNNv1xBqJI1NmY2HI9n%0AgRMfb8aEPVSuxEHhqy57eQ%3D%3D%0A-----END%20CERTIFICATE-----%0A";Chain="-----BEGIN%20CERTIFICATE-----%0AMIICHDCCAcOgAwIBAgIBCDAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktMTli%0AYXdyb2YuY29uc3VsLmNhLmVmYWQ3MjgyLmNvbnN1bDAeFw0yMjA0MjkwMzE0NTBa%0AFw0yMjA1MDIwMzE0NTBaMAAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARVIZ7Y%0AZEXfbOGBfxGa7Vuok1MIng%2FuzLQK2xLVlSTIPDbO5hstTGP%2B%2FGx182PYFP3jYqk5%0Aq6rYWe1wiPNMA30Io4H8MIH5MA4GA1UdDwEB%2FwQEAwIDuDAdBgNVHSUEFjAUBggr%0ABgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH%2FBAIwADApBgNVHQ4EIgQgrp4q50oX%0AHHghMbxz5Bk8OJFWMdfgH0Upr350WlhyxvkwKwYDVR0jBCQwIoAgUe6uERAIj%2FLM%0AyuFzDc3Wbp9TGAKBJYAwyhF14ToOQCMwYgYDVR0RAQH%2FBFgwVoZUc3BpZmZlOi8v%0AZWZhZDcyODItZDliMi0zMjk4LWY2ZDgtMzhiMzdmYjU4ZGYzLmNvbnN1bC9ucy9k%0AZWZhdWx0L2RjL2RjMS9zdmMvZGFzaGJvYXJkMAoGCCqGSM49BAMCA0cAMEQCIDwb%0AFlchufggNTijnQ5SUcvTZrWlZyq%2FrdVC20nbbmWLAiAVshNNv1xBqJI1NmY2HI9n%0AgRMfb8aEPVSuxEHhqy57eQ%3D%3D%0A-----END%20CERTIFICATE-----%0A";Subject="";URI=spiffe://efad7282-d9b2-3298-f6d8-38b37fb58df3.consul/ns/default/dc/dc1/svc/dashboard]
```

Closes #12852
2022-05-04 08:50:58 -07:00
Kyle Havlovitz 369f4848e3
Merge pull request #12885 from hashicorp/acl-err-cache
Store and return RPC error in ACL cache entries
2022-05-03 10:44:22 -07:00
Kyle Havlovitz 3bd001fb29 Return ACLRemoteError from cache and test it correctly 2022-05-03 10:05:26 -07:00
FFMMM 4cd68b4534
[sync oss] api: add peering api module (#12911) 2022-05-02 11:49:05 -07:00
Chris S. Kim 829554c706
peering: Make Upstream peer-aware (#12900)
Adds DestinationPeer field to Upstream.
Adds Peer field to UpstreamID and its string conversion functions.
2022-04-29 18:12:51 -04:00
Chris S. Kim 33bfaf5671
Cleanup peering files that used error types that were removed (#12892) 2022-04-29 14:02:26 -04:00
Mathew Estafanous 893b740dff
Unify various status errors into one HTTP error type. (#12594)
Replaces specific error types for HTTP Status codes with 
a generic HTTPError type.

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-04-29 13:42:49 -04:00
Kyle Havlovitz f84ed5f70b Store and return rpc error in acl cache entries 2022-04-28 09:08:55 -07:00
R.B. Boyer 642b75b60b
health: ensure /v1/health/service/:service endpoint returns the most recent results when a filter is used with streaming (#12640)
The primary bug here is in the streaming subsystem that makes the overall v1/health/service/:service request behave incorrectly when servicing a blocking request with a filter provided.

There is a secondary non-streaming bug being fixed here that is much less obvious related to when to update the `reply` variable in a `blockingQuery` evaluation. It is unlikely that it is triggerable in practical environments and I could not actually get the bug to manifest, but I fixed it anyway while investigating the original issue.

Simple reproduction (streaming):

1. Register a service with a tag.

        curl -sL --request PUT 'http://localhost:8500/v1/agent/service/register' \
            --header 'Content-Type: application/json' \
            --data-raw '{ "ID": "ID1", "Name": "test", "Tags":[ "a" ], "EnableTagOverride": true }'

2. Do an initial filter query that matches on the tag.

        curl -sLi --get 'http://localhost:8500/v1/health/service/test' --data-urlencode 'filter=a in Service.Tags'

3. Note you get one result. Use the `X-Consul-Index` header to establish
   a blocking query in another terminal, this should not return yet.

        curl -sLi --get 'http://localhost:8500/v1/health/service/test?index=$INDEX' --data-urlencode 'filter=a in Service.Tags'

4. Re-register that service with a different tag.

        curl -sL --request PUT 'http://localhost:8500/v1/agent/service/register' \
            --header 'Content-Type: application/json' \
            --data-raw '{ "ID": "ID1", "Name": "test", "Tags":[ "b" ], "EnableTagOverride": true }'

5. Your blocking query from (3) should return with a header
   `X-Consul-Query-Backend: streaming` and empty results if it works
   correctly `[]`.

Attempts to reproduce with non-streaming failed (where you add `&near=_agent` to the read queries and ensure `X-Consul-Query-Backend: blocking-query` shows up in the results).
2022-04-27 10:39:45 -05:00
R.B. Boyer c949de9f50
structs: ensure exported-services PeerName field can be addressed as peer_name (#12862) 2022-04-27 10:27:21 -05:00
Dhia Ayachi 9dc5200155
update raft to v1.3.8 (#12844)
* update raft to v1.3.7

* add changelog

* fix compilation error

* fix HeartbeatTimeout

* fix ElectionTimeout to reload only if value is valid

* fix default values for `ElectionTimeout` and `HeartbeatTimeout`

* fix test defaults

* bump raft to v1.3.8
2022-04-25 10:19:26 -04:00
R.B. Boyer 809344a6f5
peering: initial sync (#12842)
- Add endpoints related to peering: read, list, generate token, initiate peering
- Update node/service/check table indexing to account for peers
- Foundational changes for pushing service updates to a peer
- Plumb peer name through Health.ServiceNodes path

see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679,
     ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663,
     ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634,
     ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555

Co-authored-by: R.B. Boyer <rb@hashicorp.com>
Co-authored-by: freddygv <freddy@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Evan Culver <eculver@hashicorp.com>
Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>
2022-04-21 17:34:40 -05:00
Will Jordan 45ffdc360e
Add timeout to Client RPC calls (#11500)
Adds a timeout (deadline) to client RPC calls, so that streams will no longer hang indefinitely in unstable network conditions.

Co-authored-by: kisunji <ckim@hashicorp.com>
2022-04-21 16:21:35 -04:00
Matt Keeler f49adfaaf0
Implement the ServerDiscovery.WatchServers gRPC endpoint (#12819)
* Implement the ServerDiscovery.WatchServers gRPC endpoint
* Fix the ConnectCA.Sign gRPC endpoints metadata forwarding.
* Unify public gRPC endpoints around the public.TraceID function for request_id logging
2022-04-21 12:56:18 -04:00
Blake Covarrubias 2beea7eb7c
acl: Clarify node/service identities must be lowercase (#12807)
Modify ACL error message for invalid node/service identities names to
clearly state only lowercase alphanumeric characters are supported.
2022-04-21 09:29:16 -07:00
R.B. Boyer bbd38e95ce
chore: upgrade mockery to v2 and regenerate (#12836) 2022-04-21 09:48:21 -05:00
R.B. Boyer cf0c5110be
ca: fix a bug that caused a non blocking leaf cert query after a blocking leaf cert query to block (#12820)
Fixes #12048

Fixes #12319

Regression introduced in #11693

Local reproduction steps:
1. `consul agent -dev`
2. `curl -sLiv 'localhost:8500/v1/agent/connect/ca/leaf/web'`
3. make note of the `X-Consul-Index` header returned 
4. `curl -sLi 'localhost:8500/v1/agent/connect/ca/leaf/web?index=<VALUE_FROM_STEP_3>'`
5. Kill the above curl when it hangs with Ctrl-C
6. Repeat (2) and it should not hang.
2022-04-20 12:21:47 -05:00
Riddhi Shah 1d49f5c84e
[OSS] gRPC call to get envoy bootstrap params (#12825)
Adds a new gRPC endpoint to get envoy bootstrap params. The new consul-dataplane service will use this
endpoint to generate an envoy bootstrap configuration.
2022-04-19 17:24:21 -07:00
Matt Keeler 3badd4c35c
Add event generation for autopilot state updates (#12626)
Whenever autopilot updates its state it notifies Consul. That notification will then trigger Consul to extract out the ready server information. If the ready servers have changed, then an event will be published to notify any subscribers of the full set of ready servers.

All these ready server event things are contained within an autopilotevents package instead of the consul package to make importing them into the grpc related packages possible
2022-04-19 13:03:03 -04:00
Evan Culver 9d0b5bf8e9
connect: Add Envoy 1.22 to integration tests, remove Envoy 1.18 (#12805)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-04-18 09:36:07 -07:00
DanStough a050aa39b9 Update go version to 1.18.1 2022-04-18 11:41:10 -04:00
Kyle Havlovitz 449a0f855c
Add an internal env var for managed cluster config in the ui (#12796) 2022-04-15 09:55:52 -07:00
John Murret 1e118a5410 set vault namespaces on vault client prior to logging in with the vault auth method 2022-04-14 12:18:06 -06:00
Evan Culver e62745c82c
connect: Add Envoy 1.21.1 to support matrix, remove 1.17.4 (#12777) 2022-04-14 10:44:42 -07:00
Dan Upton 769d1d6e8e
ConnectCA.Sign gRPC Endpoint (#12787)
Introduces a gRPC endpoint for signing Connect leaf certificates. It's also
the first of the public gRPC endpoints to perform leader-forwarding, so
establishes the pattern of forwarding over the multiplexed internal RPC port.
2022-04-14 14:26:14 +01:00
Kyle Havlovitz 199f1c7200
Fix namespace default field names in expanded token output 2022-04-13 16:46:39 -07:00
Paul Glass 5eea62b47a
acl: Adjust region handling in AWS IAM auth method (#12774)
* acl: Adjust region handling in AWS IAM auth method
2022-04-13 14:31:37 -05:00
Eric Haberkorn 7961901153
Merge pull request #12773 from hashicorp/fix-lambda-intentions-and-routing
Implement Routing and Intentions for Lambdas
2022-04-13 13:01:15 -04:00
Eric b062f8c2aa Implement routing and intentions for AWS Lambdas 2022-04-13 11:45:25 -04:00
Karl Cardenas b0b197964c
Merge pull request #12562 from hashicorp/docs/blake-agent-config
docs: Agent configuration hierarchy reorganization
2022-04-12 12:33:42 -07:00
FFMMM cf7e6484aa
add more labels to RequestRecorder (#12727)
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2022-04-12 10:50:25 -07:00
Matt Keeler 2a4ca71d3f
Move to using a shared EventPublisher (#12673)
Previously we had 1 EventPublisher per state.Store. When a state store was closed/abandoned such as during a consul snapshot restore, this had the behavior of force closing subscriptions for that topic and evicting event snapshots from the cache.

The intention of this commit is to keep all that behavior. To that end, the shared EventPublisher now supports the ability to refresh a topic. That will perform the force close + eviction. The FSM upon abandoning the previous state.Store will call RefreshTopic for all the topics with events generated by the state.Store.
2022-04-12 09:47:42 -04:00
Blake Covarrubias 3175bf6b1b Remove .html extensions from docs URLs 2022-04-11 17:38:49 -07:00
Natalie Smith cd17e98800 docs: fix yet more references to agent/options 2022-04-11 17:38:49 -07:00
R.B. Boyer f5a882f66c
fix broken test (#12741) 2022-04-11 10:56:57 -05:00
Jared Kirschner 9a81a77264
Merge pull request #12725 from hashicorp/clarify-service-deregister-after-critical-message
improve error msg for deregister critical service
2022-04-07 18:01:54 -04:00
R.B. Boyer f4eac06b21
xds: ensure that all connect timeout configs can apply equally to tproxy direct dial connections (#12711)
Just like standard upstreams the order of applicability in descending precedence:

1. caller's `service-defaults` upstream override for destination
2. caller's `service-defaults` upstream defaults
3. destination's `service-resolver` ConnectTimeout
4. system default of 5s

Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
2022-04-07 16:58:21 -05:00
Jared Kirschner 1835e761fd improve error msg for deregister critical service
If a service is automatically registered because it has a critical health check
for longer than deregister_critical_service_after, the error message will now
include:
- mention of the deregister_critical_service_after option
- the value of deregister_critical_service_after for that check
2022-04-07 14:50:02 -07:00
Kyle Havlovitz a2c28526a7
Merge pull request #12685 from hashicorp/http-check-redirect-option
Add a field to disable following redirects on http checks
2022-04-07 11:29:27 -07:00
Matt Keeler 3447880091
Enable running autopilot state updates on all servers (#12617)
* Fixes a lint warning about t.Errorf not supporting %w

* Enable running autopilot on all servers

On the non-leader servers all they do is update the state and do not attempt any modifications.

* Fix the RPC conn limiting tests

Technically they were relying on racey behavior before. Now they should be reliable.
2022-04-07 10:48:48 -04:00
FFMMM 0f68bf879a
[rpc/middleware][consul] plumb intercept off, add server level happy test (#12692) 2022-04-06 14:33:05 -07:00
FFMMM 6bdde40d5e
lower log to trace (#12708) 2022-04-06 11:37:08 -07:00
Kyle Havlovitz b3db499c74 Add a field to disable following redirects on http checks 2022-04-05 16:12:18 -07:00
Mark Anderson ed3e42296d Fixup acl.EnterpriseMeta
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-04-05 15:11:49 -07:00
Mark Anderson 0905c1d83d Manual Structs fixup
Change things by hand that I couldn't figure out how to automate

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-04-05 14:51:10 -07:00
Mark Anderson 497b300c76 add new entmeta stuff.
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-04-05 14:49:31 -07:00
R.B. Boyer 12523197f9
syncing changes back from enterprise (#12701) 2022-04-05 15:46:56 -05:00
Riddhi Shah 0e5d46e9c4
Merge pull request #12695 from hashicorp/feature-negotiation-grpc-api-oss
[OSS] Supported dataplane features gRPC endpoint
2022-04-05 11:26:33 -07:00
Dan Upton e3d2b91e34
ca: move ConnectCA.Sign authorization logic to CAManager (#12609)
OSS sync of enterprise changes at 8d6fd125
2022-04-05 13:16:20 -05:00
Kyle Havlovitz 9380343689
Merge pull request #12672 from hashicorp/tgate-san-validation
Respect SNI with terminating gateways and log a warning if it isn't set alongside TLS
2022-04-05 11:15:59 -07:00
Riddhi Shah f053279c4e [OSS] Supported dataplane features gRPC endpoint
Adds a new gRPC service and endpoint to return the list of supported
consul dataplane features. The Consul Dataplane will use this API to
customize its interaction with that particular server.
2022-04-05 07:38:58 -07:00
Dan Upton e48c1611ee
WatchRoots gRPC endpoint (#12678)
Adds a new gRPC streaming endpoint (WatchRoots) that dataplane clients will
use to fetch the current list of active Connect CA roots and receive new
lists whenever the roots are rotated.
2022-04-05 15:26:14 +01:00
Dhia Ayachi cdcb249449
add a rate limiter to config auto-reload (#12490)
* add config watcher to the config package

* add logging to watcher

* add test and refactor to add WatcherEvent.

* add all API calls and fix a bug with recreated files

* add tests for watcher

* remove the unnecessary use of context

* Add debug log and a test for file rename

* use inode to detect if the file is recreated/replaced and only listen to create events.

* tidy ups (#1535)

* tidy ups

* Add tests for inode reconcile

* fix linux vs windows syscall

* fix linux vs windows syscall

* fix windows compile error

* increase timeout

* use ctime ID

* remove remove/creation test as it's a use case that fail in linux

* fix linux/windows to use Ino/CreationTime

* fix the watcher to only overwrite current file id

* fix linter error

* fix remove/create test

* set reconcile loop to 200 Milliseconds

* fix watcher to not trigger event on remove, add more tests

* on a remove event try to add the file back to the watcher and trigger the handler if success

* fix race condition

* fix flaky test

* fix race conditions

* set level to info

* fix when file is removed and get an event for it after

* fix to trigger handler when we get a remove but re-add fail

* fix error message

* add tests for directory watch and fixes

* detect if a file is a symlink and return an error on Add

* rename Watcher to FileWatcher and remove symlink deref

* add fsnotify@v1.5.1

* fix go mod

* do not reset timer on errors, rename OS specific files

* rename New func

* events trigger on write and rename

* add missing test

* fix flaking tests

* fix flaky test

* check reconcile when removed

* delete invalid file

* fix test to create files with different mod time.

* back date file instead of sleeping

* add watching file in agent command.

* fix watcher call to use new API

* add configuration and stop watcher when server stop

* add certs as watched files

* move FileWatcher to the agent start instead of the command code

* stop watcher before replacing it

* save watched files in agent

* add add and remove interfaces to the file watcher

* fix remove to not return an error

* use `Add` and `Remove` to update certs files

* fix tests

* close events channel on the file watcher even when the context is done

* extract `NotAutoReloadableRuntimeConfig` is a separate struct

* fix linter errors

* add Ca configs and outgoing verify to the not auto reloadable config

* add some logs and fix to use background context

* add tests to auto-config reload

* remove stale test

* add tests to changes to config files

* add check to see if old cert files still trigger updates

* rename `NotAutoReloadableRuntimeConfig` to `StaticRuntimeConfig`

* fix to re add both key and cert file. Add test to cover this case.

* review suggestion

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add check to static runtime config changes

* fix test

* add changelog file

* fix review comments

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* update flag description

Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>

* fix compilation error

* add static runtime config support

* fix test

* fix review comments

* fix log test

* Update .changelog/12329.txt

Co-authored-by: Dan Upton <daniel@floppy.co>

* transfer tests to runtime_test.go

* fix filewatcher Replace to not deadlock.

* avoid having lingering locks

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* split ReloadConfig func

* fix warning message

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* convert `FileWatcher` into an interface

* fix compilation errors

* fix tests

* extract func for adding and removing files

* add a coalesceTimer with a very small timer

* extract coaelsce Timer and add a shim for testing

* add tests to coalesceTimer fix to send remaining events

* set `coalesceTimer` to 1 Second

* support symlink, fix a nil deref.

* fix compile error

* fix compile error

* refactor file watcher rate limiting to be a Watcher implementation

* fix linter issue

* fix runtime config

* fix runtime test

* fix flaky tests

* fix compile error

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* fix agent New to return an error if File watcher New return an error

* quit timer loop if ctx is canceled

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>
Co-authored-by: Daniel Upton <daniel@floppy.co>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-04-04 11:31:39 -04:00
Eric Haberkorn 66c40e073a
Merge pull request #12681 from hashicorp/lambda-patching-tweaks
Tweak the Lambda Envoy configuration generated by the serverless patcher
2022-04-01 19:59:30 -04:00
FFMMM b0cba2ec03
mark disable_compat_1.9 to deprecate in 1.13, change default to true (#12675)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-04-01 10:35:56 -07:00
R.B. Boyer 3060b5cb8f
xds: errors from the xds serverless plugin are fatal (#12682) 2022-04-01 10:30:26 -05:00
Eric 66391186ce Tweak the Lambda Envoy configuration generated by the serverless patcher
- Move from `strip_matching_host_port` to `strip_any_host_port`
- Remove `auto_host_rewrite` since it conflicts with `strip_any_host_port`
2022-04-01 11:13:44 -04:00
Eric Haberkorn 9b291d362a
Merge pull request #12676 from hashicorp/implement-lambda-patching
Implement Lambda Patching in the Serverless Plugin
2022-04-01 09:58:56 -04:00
Mark Anderson aa29324a24
Avoid using sys/mounts to enable namespaces (#12655)
* Avoid doing list of /sys/mounts

From an internal ticket "Support standard "Vault namespace in the path" semantics for Connect Vault CA Provider"

Vault allows the namespace to be specified as a prefix in the path of
a PKI definition, but this doesn't currently work for
```IntermediatePKIPath``` specifications, because we attempt to list
all of the paths to check if ours is already defined. This doesn't
really work in a namespaced world.

This changes the IntermediatePKIPath code to follow the same pattern
as the root key, where we directly get the key rather than listing.

This code is difficult to write automated tests for because it relies
on features of Vault Enterprise, which isn't currently part of our
test framework, so it was tested manually.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* add changelog

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-03-31 23:35:38 -07:00
Kyle Havlovitz 116b6c57cb Use the GatewayService SNI field for upstream SAN validation 2022-03-31 13:54:25 -07:00
Eric c7204528c5 Implement Lambda Patching in the Serverless Plugin 2022-03-31 16:45:32 -04:00
Kyle Havlovitz c2a48990cf
Merge pull request #12670 from hashicorp/token-read-expanded
oss: Add expanded token read flag and endpoint option
2022-03-31 12:24:11 -07:00
Kyle Havlovitz 4974d8471b Log a warning when a terminating gateway service has TLS but not SNI configured 2022-03-31 12:18:40 -07:00
Dhia Ayachi 8552efa955
auto-reload configuration when config files change (#12329)
* add config watcher to the config package

* add logging to watcher

* add test and refactor to add WatcherEvent.

* add all API calls and fix a bug with recreated files

* add tests for watcher

* remove the unnecessary use of context

* Add debug log and a test for file rename

* use inode to detect if the file is recreated/replaced and only listen to create events.

* tidy ups (#1535)

* tidy ups

* Add tests for inode reconcile

* fix linux vs windows syscall

* fix linux vs windows syscall

* fix windows compile error

* increase timeout

* use ctime ID

* remove remove/creation test as it's a use case that fail in linux

* fix linux/windows to use Ino/CreationTime

* fix the watcher to only overwrite current file id

* fix linter error

* fix remove/create test

* set reconcile loop to 200 Milliseconds

* fix watcher to not trigger event on remove, add more tests

* on a remove event try to add the file back to the watcher and trigger the handler if success

* fix race condition

* fix flaky test

* fix race conditions

* set level to info

* fix when file is removed and get an event for it after

* fix to trigger handler when we get a remove but re-add fail

* fix error message

* add tests for directory watch and fixes

* detect if a file is a symlink and return an error on Add

* rename Watcher to FileWatcher and remove symlink deref

* add fsnotify@v1.5.1

* fix go mod

* do not reset timer on errors, rename OS specific files

* rename New func

* events trigger on write and rename

* add missing test

* fix flaking tests

* fix flaky test

* check reconcile when removed

* delete invalid file

* fix test to create files with different mod time.

* back date file instead of sleeping

* add watching file in agent command.

* fix watcher call to use new API

* add configuration and stop watcher when server stop

* add certs as watched files

* move FileWatcher to the agent start instead of the command code

* stop watcher before replacing it

* save watched files in agent

* add add and remove interfaces to the file watcher

* fix remove to not return an error

* use `Add` and `Remove` to update certs files

* fix tests

* close events channel on the file watcher even when the context is done

* extract `NotAutoReloadableRuntimeConfig` is a separate struct

* fix linter errors

* add Ca configs and outgoing verify to the not auto reloadable config

* add some logs and fix to use background context

* add tests to auto-config reload

* remove stale test

* add tests to changes to config files

* add check to see if old cert files still trigger updates

* rename `NotAutoReloadableRuntimeConfig` to `StaticRuntimeConfig`

* fix to re add both key and cert file. Add test to cover this case.

* review suggestion

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add check to static runtime config changes

* fix test

* add changelog file

* fix review comments

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* update flag description

Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>

* fix compilation error

* add static runtime config support

* fix test

* fix review comments

* fix log test

* Update .changelog/12329.txt

Co-authored-by: Dan Upton <daniel@floppy.co>

* transfer tests to runtime_test.go

* fix filewatcher Replace to not deadlock.

* avoid having lingering locks

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* split ReloadConfig func

* fix warning message

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* convert `FileWatcher` into an interface

* fix compilation errors

* fix tests

* extract func for adding and removing files

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>
Co-authored-by: Daniel Upton <daniel@floppy.co>
2022-03-31 15:11:49 -04:00
Kyle Havlovitz 9a2474381a Add expanded token read flag and endpoint option 2022-03-31 10:49:49 -07:00
FFMMM 8b184197b3
polish rpc.service.call metric behavior (#12624) 2022-03-31 10:49:37 -07:00
Paul Glass aae6d8080d
Add IAM Auth Method (#12583)
This adds an aws-iam auth method type which supports authenticating to Consul using AWS IAM identities.

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-03-31 10:18:48 -05:00
Eric Haberkorn 9751626828
Merge pull request #12659 from hashicorp/bump-go-control-plane
Bump Go Control Plane
2022-03-30 15:07:47 -04:00
R.B. Boyer e9230e93d8
xds: adding control of the mesh-wide min/max TLS versions and cipher suites from the mesh config entry (#12601)
- `tls.incoming`: applies to the inbound mTLS targeting the public
  listener on `connect-proxy` and `terminating-gateway` envoy instances

- `tls.outgoing`: applies to the outbound mTLS dialing upstreams from
  `connect-proxy` and `ingress-gateway` envoy instances

Fixes #11966
2022-03-30 13:43:59 -05:00
R.B. Boyer ee11dff5a5 similar bump 2022-03-30 13:28:00 -05:00
R.B. Boyer 47693e3ebf fail on error and use ptypes.MarshalAny for now instead of anypb.New 2022-03-30 13:27:49 -05:00
Eric 91a493efe9 Bump go-control-plane
* `go get cloud.google.com/go@v0.59.0`
* `go get github.com/envoyproxy/go-control-plane@v0.9.9`
* `make envoy-library`
* Bumpprotoc to 3.15.8
2022-03-30 13:11:27 -04:00
R.B. Boyer d4e80b8800
server: ensure that service-defaults meta is incorporated into the discovery chain response (#12511)
Also add a new "Default" field to the discovery chain response to clients
2022-03-30 10:04:18 -05:00
FFMMM 0fd6cdc900
introduce EmptyReadRequest for status_endpoint (#12653)
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2022-03-29 18:05:45 -07:00
Fulvio d1938482e7
remove DualStack field from check TCP #12629 (#12630) 2022-03-29 14:56:01 -04:00
Eric ab5b5e85f5 remove the rest of gogo 2022-03-28 17:34:41 -04:00
Eric 8fd73ede3e remove gogo from acl protobufs 2022-03-28 16:20:56 -04:00
Connor f8fc317731
Fix leaked Vault LifetimeRenewers (#12607)
* Fix leaked Vault LifetimeRenewers

When the Vault CA Provider is reconfigured we do not stop the
LifetimeRenewers which can cause them to leak until the Consul processes
recycles. On Configure execute stopWatcher if it exists and is not nil
before starting a new renewal

* Add jitter before restarting the LifetimeWatcher

If we fail to login to Vault or our token is no longer valid we can
overwhelm a Vault instance with many requests very quickly by restarting
the LifetimeWatcher. Before restarting the LifetimeWatcher provide a
backoff time of 1 second or less.

* Use a retry.Waiter instead of RandomStagger

* changelog

* gofmt'd

* Swap out bool for atomic.Unit32 in test

* Provide some extra clarification in comment and changelog
2022-03-28 09:58:16 -05:00
Eric 523e054c81 assorted changes required to remove gogo 2022-03-25 09:55:36 -04:00
FFMMM 560f8cbc89
fix bad oss sync, use gauges not counters (#12611) 2022-03-24 14:41:30 -07:00
Kyle Havlovitz d9f31345e0
Merge pull request #12596 from hashicorp/overview-endpoint
oss: Add overview UI internal endpoint
2022-03-24 14:27:54 -07:00
Mike Morris 8020fb2098
agent: convert listener config to TLS types (#12522)
* tlsutil: initial implementation of types/TLSVersion

tlsutil: add test for parsing deprecated agent TLS version strings

tlsutil: return TLSVersionInvalid with error

tlsutil: start moving tlsutil cipher suite lookups over to types/tls

tlsutil: rename tlsLookup to ParseTLSVersion, add cipherSuiteLookup

agent: attempt to use types in runtime config

agent: implement b.tlsVersion validation in config builder

agent: fix tlsVersion nil check in builder

tlsutil: update to renamed ParseTLSVersion and goTLSVersions

tlsutil: fixup TestConfigurator_CommonTLSConfigTLSMinVersion

tlsutil: disable invalid config parsing tests

tlsutil: update tests

auto_config: lookup old config strings from base.TLSMinVersion

auto_config: update endpoint tests to use TLS types

agent: update runtime_test to use TLS types

agent: update TestRuntimeCinfig_Sanitize.golden

agent: update config runtime tests to expect TLS types

* website: update Consul agent tls_min_version values

* agent: fixup TLS parsing and compilation errors

* test: fixup lint issues in agent/config_runtime_test and tlsutil/config_test

* tlsutil: add CHACHA20_POLY1305 cipher suites to goTLSCipherSuites

* test: revert autoconfig tls min version fixtures to old format

* types: add TLSVersions public function

* agent: add warning for deprecated TLS version strings

* agent: move agent config specific logic from tlsutil.ParseTLSVersion into agent config builder

* tlsutil(BREAKING): change default TLS min version to TLS 1.2

* agent: move ParseCiphers logic from tlsutil into agent config builder

* tlsutil: remove unused CipherString function

* agent: fixup import for types package

* Revert "tlsutil: remove unused CipherString function"

This reverts commit 6ca7f6f58d268e617501b7db9500113c13bae70c.

* agent: fixup config builder and runtime tests

* tlsutil: fixup one remaining ListenerConfig -> ProtocolConfig

* test: move TLS cipher suites parsing test from tlsutil into agent config builder tests

* agent: remove parseCiphers helper from auto_config_endpoint_test

* test: remove unused imports from tlsutil

* agent: remove resolved FIXME comment

* tlsutil: remove TODO and FIXME in cipher suite validation

* agent: prevent setting inherited cipher suite config when TLS 1.3 is specified

* changelog: add entry for converting agent config to TLS types

* agent: remove FIXME in runtime test, this is covered in builder tests with invalid tls9 value now

* tlsutil: remove config tests for values checked at agent config builder boundary

* tlsutil: remove tls version check from loadProtocolConfig

* tlsutil: remove tests and TODOs for logic checked in TestBuilder_tlsVersion and TestBuilder_tlsCipherSuites

* website: update search link for supported Consul agent cipher suites

* website: apply review suggestions for tls_min_version description

* website: attempt to clean up markdown list formatting for tls_min_version

* website: moar linebreaks to fix tls_min_version formatting

* Revert "website: moar linebreaks to fix tls_min_version formatting"

This reverts commit 38585927422f73ebf838a7663e566ac245f2a75c.

* autoconfig: translate old values for TLSMinVersion

* agent: rename var for translated value of deprecated TLS version value

* Update agent/config/deprecated.go

Co-authored-by: Dan Upton <daniel@floppy.co>

* agent: fix lint issue

* agent: fixup deprecated config test assertions for updated warning

Co-authored-by: Dan Upton <daniel@floppy.co>
2022-03-24 15:32:25 -04:00
Kyle Havlovitz 0d5cbf6f30 Sort by partition/ns/servicename instead of the reverse 2022-03-24 12:16:05 -07:00
FFMMM 8f98bbda75
[metrics][rpc]: add basic prefix filter test for new rpc metric (#12598)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2022-03-23 13:29:12 -07:00
Kyle Havlovitz 1b654c9807 Clean up ent meta id usage in overview summary 2022-03-23 12:47:12 -07:00
Eric 98b733e41a remove gogo from pbservice 2022-03-23 12:18:01 -04:00
Mark Anderson 28c925f6d0 Fixup dropped SecretID usage
Looks like something got munged at some point. Not sure how it slipped in, but my best guess is that because TestTxn_Apply_ACLDeny is marked flaky we didn't block merge because it failed.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-03-22 21:20:03 -07:00
Kyle Havlovitz 04f1d9bcc9 oss: Add overview UI internal endpoint 2022-03-22 17:05:09 -07:00
Dhia Ayachi 5cb24b9bf8
split `pbcommon` to `pbcommon` and `pbcommongogo` (#12587)
* mogify needed pbcommon structs

* mogify needed pbconnect structs

* fix compilation errors and make config_translate_test pass

* add missing file

* remove redundant oss func declaration

* fix EnterpriseMeta to copy the right data for enterprise

* rename pbcommon package to pbcommongogo

* regenerate proto and mog files

* add missing mog files

* add pbcommon package

* pbcommon no mog

* fix enterprise meta code generation

* fix enterprise meta code generation (pbcommongogo)

* fix mog generation for gogo

* use `protoc-go-inject-tag` to inject tags

* rename proto package

* pbcommon no mog

* use `protoc-go-inject-tag` to inject tags

* add non gogo proto to make file

* fix proto get
2022-03-22 16:30:00 -04:00
Dan Upton 2fe06f663b
streaming: emit events when Connect CA Roots change (#12590)
OSS sync of enterprise changes at 614f786d
2022-03-22 19:13:59 +00:00
FFMMM 76d8798590
factor out recording func, add unit tests (#12585)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2022-03-22 09:31:54 -07:00
Dan Upton fb441e323a
Restructure gRPC server setup (#12586)
OSS sync of enterprise changes at 0b44395e
2022-03-22 12:40:24 +00:00
FFMMM 08f2838b78
pre register new rpc metric, rename metric (#12582) 2022-03-21 17:26:32 -07:00
Mark Anderson 2b367626f0
Add source of authority annotations to the PermissionDeniedError output. (#12567)
This extends the acl.AllowAuthorizer with source of authority information.

The next step is to unify the AllowAuthorizer and ACLResolveResult structures; that will be done in a separate PR.

Part of #12481

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-03-18 10:32:25 -07:00
Dan Upton 57f0f42733
Support per-listener TLS configuration ⚙️ (#12504)
Introduces the capability to configure TLS differently for Consul's
listeners/ports (i.e. HTTPS, gRPC, and the internal multiplexed RPC
port) which is useful in scenarios where you may want the HTTPS or
gRPC interfaces to present a certificate signed by a well-known/public
CA, rather than the certificate used for internal communication which
must have a SAN in the form `server.<dc>.consul`.
2022-03-18 10:46:58 +00:00
Evan Culver 27711fe5c7
lib: add validation package + DNS label validation (#12535)
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-03-17 18:31:28 -07:00
FFMMM 3c08843847
[sync oss] add net/rpc interceptor implementation (#12573)
* sync ent changes from 866dcb0667

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

* update oss go.mod

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2022-03-17 16:02:26 -07:00
Jared Kirschner 038fd90925
Merge pull request #11821 from hashicorp/error-if-get-request-has-body
http: error if GET request has non-empty body
2022-03-16 18:34:27 -04:00
Jared Kirschner 13712de2e7 http: WARN if GET request has non-empty body
Give the user a hint that they might be doing something wrong if their GET
request has a non-empty body, which can easily happen using curl's
--data-urlencode if specifying request type via "--request GET" rather than
"--get". See https://github.com/hashicorp/consul/issues/11471.
2022-03-16 14:19:50 -07:00
Eric ae1cdc85b1 Remove the stdduration gogo extension 2022-03-16 12:12:29 -04:00
mrspanishviking 1ae820ea0a
Revert "[Docs] Agent configuration hierarchy " 2022-03-15 16:13:58 -07:00
trujillo-adam 667976c94f fixing merge conflicts part 3 2022-03-15 15:25:03 -07:00
Eric Haberkorn 90ecd7b5fe
Merge pull request #12556 from hashicorp/wire-up-serverless-patcher
Create and wire up the serverless patcher
2022-03-15 14:05:40 -04:00
Eric Haberkorn d58f230bb8
Merge pull request #12557 from hashicorp/remove-healthcheck-gogo-stdduration
Remove Gogo Stdduration From the Healthcheck Protobufs
2022-03-15 13:20:49 -04:00
Eric fa13ce3e45 Remove gogo stdduration from the healthcheck protobufs 2022-03-15 10:51:40 -04:00
Eric 07bf0c5ce8 Create and wire up the serverless patcher 2022-03-15 10:12:57 -04:00
trujillo-adam 60a88bb40f merging new hierarchy for agent configuration 2022-03-14 15:44:41 -07:00
Mark Anderson ab099e5fcb
Refactor config checks oss (#12550)
Currently the config_entry.go subsystem delegates authorization decisions via the ConfigEntry interface CanRead and CanWrite code. Unfortunately this returns a true/false value and loses the details of the source.

This is not helpful, especially since it the config subsystem can be more complex to understand, since it covers so many domains.

This refactors CanRead/CanWrite to return a structured error message (PermissionDenied or the like) with more details about the reason for denial.

Part of #12241

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-03-11 13:45:51 -08:00
Eric Haberkorn 3297bc6988
Merge pull request #12536 from hashicorp/add-serverless-config
Add the `connect.enable_serverless_plugin` configuration option
2022-03-11 09:39:36 -05:00
Eric Haberkorn 35cb0c1312
Merge pull request #12539 from hashicorp/make-xds-lib
Make the xdscommon package
2022-03-11 09:21:10 -05:00
Eric 2d27f9dc47 Add the `connect.enable_serverless_plugin` configuration option. 2022-03-11 09:16:00 -05:00
Mark Anderson 5591cb1e11
Bulk acl message fixup oss (#12470)
* First pass for helper for bulk changes

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Convert ACLRead and ACLWrite to new form

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* AgentRead and AgentWRite

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Fix EventWrite

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* KeyRead, KeyWrite, KeyList

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* KeyRing

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* NodeRead NodeWrite

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* OperatorRead and OperatorWrite

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* PreparedQuery

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Intention partial

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Fix ServiceRead, Write ,etc

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Error check ServiceRead?

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Fix Sessionread/Write

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Fixup snapshot ACL

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Error fixups for txn

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Add changelog

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Fixup review comments

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-03-10 18:48:27 -08:00
Eric 3f9b84e6a4 Make an xdscommon package that will be shared between Consul and Envoy plugins 2022-03-08 14:57:23 -05:00
Eric Haberkorn b4276e971e
Merge pull request #12529 from hashicorp/add-meta-to-service-config-response
Add `Meta` to `ServiceConfigResponse`
2022-03-07 16:35:21 -05:00
Eric Haberkorn 45312886fe Code review changes 2022-03-07 14:39:33 -05:00
R.B. Boyer cc4733e60d
proxycfg: change how various proxycfg test helpers for making ConfigSnapshot copies works to be more correct and less error prone (#12531)
Prior to this PR for the envoy xDS golden tests in the agent/xds package we
were hand-creating a proxycfg.ConfigSnapshot structure in the proper format for
input to the xDS generator. Over time this intermediate structure has gotten
trickier to build correctly for the various tests.

This PR proposes to switch to using the existing mechanism for turning a
structs.NodeService and a sequence of cache.UpdateEvent copies into a
proxycfg.ConfigSnapshot, as that is less error prone to construct and aligns
more with how the data arrives.

NOTE: almost all of this is in test-related code. I tried super hard to craft
correct event inputs to get the golden files to be the same, or similar enough
after construction to feel ok that i recreated the spirit of the original test
cases.
2022-03-07 11:47:14 -06:00
Eric 3d46f9ef7c Add `Meta` to `ServiceConfigResponse` 2022-03-07 10:05:18 -05:00
R.B. Boyer b63a0f3909
reduce flakiness/raciness of errNotFound and errNotChanged blocking query tests (#12518)
Improves tests from #12362

These tests try to setup the following concurrent scenario:

1. (goroutine 1) execute read RPC with index=0
2. (goroutine 1) get response from (1) @ index=10
3. (goroutine 1) execute read RPC with index=10 and block
4. (goroutine 2) WHILE (3) is blocking, start slamming the system with stray writes that will cause the WatchSet to wakeup
5. (goroutine 2) after doing all writes, shut down the reader above
6. (goroutine 1) stops reading, double checks that it only ever woke up once (from 1)
2022-03-04 11:20:01 -06:00
R.B. Boyer 07b92a2855
server: fix spurious blocking query suppression for discovery chains (#12512)
Minor fix for behavior in #12362

IsDefault sometimes returns true even if there was a proxy-defaults or service-defaults config entry that was consulted. This PR fixes that.
2022-03-03 16:54:41 -06:00
Daniel Nephin 8f4b6af68a
Merge pull request #12298 from jorgemarey/b-persistnewrootandconfig
Avoid raft change when no config is provided on persistNewRootAndConfig
2022-03-03 11:03:50 -05:00