Commit Graph

19532 Commits

Author SHA1 Message Date
Chris S. Kim c262065276
Warn if ACL is enabled but no token is provided to Envoy (#15967) 2023-01-16 12:31:56 -05:00
Dhia Ayachi 9e99715ee8
avoid logging RPC errors when it's specific rate limiter errors (#15968)
* avoid logging RPC errors when it's specific rate limiter errors

* simplify if statements
2023-01-16 12:08:09 -05:00
Derek Menteer f926c7643f
Enforce lowercase peer names. (#15697)
Enforce lowercase peer names.

Prior to this change peer names could be mixed case.
This can cause issues, as peer names are used as DNS labels
in various locations. It also caused issues with envoy
configuration.
2023-01-13 14:20:28 -06:00
Ranjandas 8e9fe563fa
Update TG Docs with SAN match option when using SNI (#15971)
When using SNI in Terminating Gateway, Consul configures envoy to
have strict SAN matching. This requires all external services to
have SANs in their certificates and not having it will throw
CERTIFICATE_VERIFY_FAILED error.
2023-01-12 19:55:36 -08:00
Frank DiRocco 9420fd229b
Update go-discover to support ECS discovery (#13782)
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2023-01-12 12:06:29 -06:00
Michael Wilkerson d224744f6a
removed Consul requirements from documentation as it may be confusing (#15958)
* removed Consul requirements from documentation as it may be confusing
2023-01-11 21:01:30 -08:00
R.B. Boyer 04673cb6e4
test: general cleanup and fixes for the container integration test suite (#15959)
- remove dep on consul main module
- use 'consul tls' subcommands instead of tlsutil
- use direct json config construction instead of agent/config structs
- merge libcluster and libagent packages together
- more widely use BuildContext
- get the OSS/ENT runner stuff working properly
- reduce some flakiness
- fix some correctness related to http/https API
2023-01-11 15:34:27 -06:00
Dan Stough 27d74de701
feat: add access logs to dataplane bootstrap rpc (#15951) 2023-01-11 13:40:09 -05:00
Chris Hut ad123d9564
ui/fix linting error and failing ui test (#15953)
* fix linting error

* Fix the failing version test as well
2023-01-11 10:48:04 -07:00
Dan Stough 797bfb1677
docs(access logs): new docs for access logging (#15948)
Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
2023-01-11 16:41:02 +00:00
Matt Keeler 554f1e6fee
Protobuf Modernization (#15949)
* Protobuf Modernization

Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf

Marshallers (protobuf and json) needed some changes to account for different APIs.

Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package.

This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces.

* Fix go-mod-tidy make target to work on all modules
2023-01-11 09:39:10 -05:00
Paul Glass 1bf1686ebc
Add new config_file_service_registration token (#15828) 2023-01-10 10:24:02 -06:00
Dan Stough e0b7694a90
[OSS] feat: add access log config to consul envoy connect (#15946) 2023-01-09 15:16:00 -05:00
Chris S. Kim 82d6d12a13
Output user-friendly name for anonymous token (#15884) 2023-01-09 12:28:53 -06:00
Dan Upton f24f8c681f
Rate limit improvements and fixes (#15917)
- Fixes a panic when Operation.SourceAddr is nil (internal net/rpc calls)
- Adds proper HTTP response codes (429 and 503) for rate limit errors
- Makes the error messages clearer
- Enables automatic retries for rate-limit errors in the net/rpc stack
2023-01-09 10:20:05 +00:00
Semir Patel 2a90faa4b1
emit metrics for global rate limiting (#15891) 2023-01-06 17:49:33 -06:00
Jared Kirschner c088753e7b
docs: cluster peering upgrade clarification (#15935) 2023-01-06 18:30:10 -05:00
Thomas Eckert 8e3d0da54b
Update the README for the Consul API (#15936) 2023-01-06 21:10:56 +00:00
Derek Menteer a5915bd99f
Fix issue where TLS configuration was ignored for unix sockets in consul connect envoy. (#15913)
Fix issue where TLS configuration was ignored for unix sockets in consul connect envoy.
Disable xds check on bootstrap mode and change check to warn only.
2023-01-06 12:34:49 -06:00
Dhia Ayachi f17bc5ed73
inject logger and create logdrop sink (#15822)
* inject logger and create logdrop sink

* init sink with an empty struct instead of nil

* wrap a logger instead of a sink and add a discard logger to avoid double logging

* fix compile errors

* fix linter errors

* Fix bug where log arguments aren't properly formatted

* Move log sink construction outside of handler

* Add prometheus definition and docs for log drop counter

Co-authored-by: Daniel Upton <daniel@floppy.co>
2023-01-06 11:33:53 -07:00
Eric Haberkorn 01a0142d1f
Add the Lua Envoy extension (#15906) 2023-01-06 12:13:40 -05:00
Jared Kirschner a8840e710e
docs: fix markdown heading levels (#15912) 2023-01-06 11:37:19 -05:00
Matt Keeler 482c9e1122
Fix usage of protoc-gen-go-binary (#15920)
* Fix usage of protoc-gen-go-binary
* Pin protoc-gen-go-binary version
2023-01-06 11:30:54 -05:00
Tu Nguyen 9d1f030980
Update example link for Consul K8s NGINX ingress controller (#15916) 2023-01-06 08:11:13 -08:00
lbausch 5199137a6b
docs: Fix link (#14890)
Co-authored-by: David Yu <dyu@hashicorp.com>
2023-01-06 05:21:56 +00:00
claire labry 70f06e71f5
Onboard consul to use new version/VERSION file and reproducible actions (#15631)
* Onboard consul to use new .release/VERSION file and reproducible actions-go-build

* Onboard consul to use new .release/VERSION file and reproducible actions

* Onboard consul to use new .release/VERSION file and reproducible actions

* fix to consul

* Onboard consul to use new .release/VERSION file and reproducible actions

* Onboard consul to use new .release/VERSION file and reproducible actions

* Onboard consul to use new .release/VERSION file and reproducible actions

* test out ent changes

* just or testing

* Added setup go for build ui

* try removing VERSION file out of .release dir

* add checkout action for build ui and update checkout version

* try no -dev marker

* try removing extra ldflags

* test version

* add back in setup-go step?

* Update utils.js

read from static VERSION file

* remove actions-setup go

* add 1.15.0-dev

* Using prepare workflow for pre-stable channel workflow

* Test prepare workflow

* Remove set-product-version branch from release pipeline

* Use METADATA in environment

* Correct env vars

* Remove current branch from build trigger list

Co-authored-by: emilymianeil <emilymianeil@gmail.com>
Co-authored-by: Sarah <sthompson@hashicorp.com>
Co-authored-by: hc-github-team-nomad-core <github-team-nomad-core@hashicorp.com>
Co-authored-by: emily neil <63985869+emilymianeil@users.noreply.github.com>
2023-01-05 12:16:47 -08:00
Paul Glass a36839d9c3
Fix TLS_BadVerify test assertions on macOS (#15903) 2023-01-05 11:47:45 -06:00
Dan Upton 76fea384a3
grpc/acl: fix bug where ACL token was required even if disabled (#15904)
Fixes a bug introduced by #15346 where we'd always require an ACL
token even if ACLs were disabled because we were erroneously
treating `nil` identity as anonymous.
2023-01-05 16:31:18 +00:00
Dan Upton a36173e2b5
Fix missing comment in generated enterprise rate-limit file (#15895)
Fixes bug in #15564 where gofmt would strip out the generated code
warning comment because it was on the same line as the build tag.
2023-01-05 10:31:14 +00:00
Dan Upton 15c7c03fa5
grpc: switch servers and retry on error (#15892)
This is the OSS portion of enterprise PR 3822.

Adds a custom gRPC balancer that replicates the router's server cycling
behavior. Also enables automatic retries for RESOURCE_EXHAUSTED errors,
which we now get for free.
2023-01-05 10:21:27 +00:00
Nick Irvine 2c37b0afd1
fix: return error when config file with unknown extension is passed (#15107) 2023-01-04 16:57:00 -08:00
Tu Nguyen 39e6ca3bf6
Update api gateway version to latest (#15197)
* Update api gateway version to latest

* Update website/content/docs/api-gateway/install.mdx

* update to latest apigw version 0.5.1

* update consul and helm version
2023-01-04 12:44:02 -08:00
cskh bb797ff36c
Refactoring the peering integ test to accommodate coming changes of o… (#15885)
* Refactoring the peering integ test to accommodate coming changes of other upgrade scenarios.

- Add a utils package under test that contains methods to set up various test scenarios.
- Deduplication: have a single CreatingPeeringClusterAndSetup replace
  CreatingAcceptingClusterAndSetup and CreateDialingClusterAndSetup.
- Separate peering cluster creation and server registration.

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
2023-01-04 15:28:15 -05:00
Florian Apolloner cb5389cc89
Allow Operator Generated bootstrap token (#14437)
Add support to provide an initial token via the bootstrap HTTP API, similar to hashicorp/nomad#12520
2023-01-04 20:19:33 +00:00
Semir Patel 8242459c66
Wire up the rate limiter to net/rpc calls (#15879) 2023-01-04 13:38:44 -06:00
Tu Nguyen e3997b9533
Update links to new get started tutorials (#15154) 2023-01-04 09:58:08 -08:00
Dan Upton 1d95609fb7
grpc: `protoc` plugin for generating gRPC rate limit specifications (#15564)
Adds automation for generating the map of `gRPC Method Name → Rate Limit Type`
used by the middleware introduced in #15550, and will ensure we don't forget
to add new endpoints.

Engineers must annotate their RPCs in the proto file like so:

```
rpc Foo(FooRequest) returns (FooResponse) {
  option (consul.internal.ratelimit.spec) = {
    operation_type: READ,
  };
}
```

When they run `make proto` a protoc plugin `protoc-gen-consul-rate-limit` will
be installed that writes rate-limit specs as a JSON array to a file called
`.ratelimit.tmp` (one per protobuf package/directory).

After running Buf, `make proto` will execute a post-process script that will
ingest all of the `.ratelimit.tmp` files and generate a Go file containing the
mappings in the `agent/grpc-middleware` package. In the enterprise repository,
it will write an additional file with the enterprise-only endpoints.

If an engineer forgets to add the annotation to a new RPC, the plugin will
return an error like so:

```
RPC Foo is missing rate-limit specification, fix it with:

	import "proto-public/annotations/ratelimit/ratelimit.proto";

	service Bar {
	  rpc Foo(...) returns (...) {
	    option (hashicorp.consul.internal.ratelimit.spec) = {
	      operation_type: OPERATION_READ | OPERATION_WRITE | OPERATION_EXEMPT,
	    };
	  }
	}
```

In the future, this annotation can be extended to support rate-limit
category (e.g. KV vs Catalog) and to determine the retry policy.
2023-01-04 16:07:02 +00:00
Dan Upton 4719b717ea
grpc/acl: relax permissions required for "core" endpoints (#15346)
Previously, these endpoints required `service:write` permission on _any_
service as a sort of proxy for "is the caller allowed to participate in
the mesh?".

Now, they're called as part of the process of establishing a server
connection by any consumer of the consul-server-connection-manager
library, which will include non-mesh workloads (e.g. Consul KV as a
storage backend for Vault) as well as ancillary components such as
consul-k8s' acl-init process, which likely won't have `service:write`
permission.

So this commit relaxes those requirements to accept *any* valid ACL token
on the following gRPC endpoints:

- `hashicorp.consul.dataplane.DataplaneService/GetSupportedDataplaneFeatures`
- `hashicorp.consul.serverdiscovery.ServerDiscoveryService/WatchServers`
- `hashicorp.consul.connectca.ConnectCAService/WatchRoots`
2023-01-04 12:40:34 +00:00
Hans Hasselberg af2464126a
fix cli string for id flag (#15695) 2023-01-03 20:36:26 +00:00
Derek Menteer 2af14c0084
Fix issue with incorrect proxycfg watch on upstream peer-targets. (#15865)
This fixes an issue where the incorrect partition was given to the
upstream target watch, which meant that failover logic would not
work correctly.
2023-01-03 10:44:08 -06:00
Derek Menteer 76cc876ac3
Fix agent cache incorrectly notifying unchanged protobufs. (#15866)
Fix agent cache incorrectly notifying unchanged protobufs.

This change fixes a situation where the protobuf private fields
would be read by reflect.DeepEqual() and indicate data was modified.
This resulted in change notifications being fired every time, which
could cause performance problems in proxycfg.
2023-01-03 10:11:56 -06:00
Dan Upton 006138beb4
Wire in rate limiter to handle internal and external gRPC calls (#15857) 2022-12-23 13:42:16 -06:00
Dan Stough 1586c7ba10
chore: fix remote docker make target (#15870) 2022-12-22 15:44:18 -05:00
Dan Stough 38d65efb72
[OSS] feat: access logs for listeners and listener filters (#15864)
* feat: access logs for listeners and listener filters

* changelog

* fix integration test
2022-12-22 15:18:15 -05:00
Gerard Nguyen 8a543e11be
docs: update intentions config (#15868) 2022-12-22 20:07:23 +00:00
Freddy 0cc8f45f28
Expand service mesh dev docs (#15867) 2022-12-22 12:18:38 -07:00
Jeff Boruszak ddba394070
Removed technical preview callout (#15872) 2022-12-22 12:52:34 -06:00
Nitya Dhanushkodi e0e4505f44
add extensions for local service to GetExtensionConfigurations (#15871)
This gets the extensions information for the local service into the snapshot and ExtensionConfigurations for a proxy. It grabs the extensions from config entries and puts them in structs.NodeService.Proxy field, which already is copied into the config snapshot.

Also:
* add EnvoyExtensions to api.AgentService so that it matches structs.NodeService
2022-12-22 10:03:33 -08:00
Nitya Dhanushkodi 2800774f68
[OSS] extensions: refactor PluginConfiguration into a more generic type ExtensionConfiguration (#15846)
* extensions: refactor PluginConfiguration into a more generic type
ExtensionConfiguration

Also:
* adds endpoints configuration to lambda golden tests
* uses string constant for builtin/aws/lambda
Co-authored-by: Eric <eric@haberkorn.co>
2022-12-20 22:26:20 -08:00
John Murret 2a0aeb2349
Rate Limit Handler - ensure rate limiting is not in the code path when not configured (#15819)
* Rate limiting handler - ensure configuration has changed before modifying limiters

* Updating test to validate arguments to UpdateConfig

* Removing duplicate test.  Updating mock.

* Renaming NullRateLimiter to NullRequestLimitsHandler

* Rate Limit Handler - ensure rate limiting is not in the code path when not configured

* Update agent/consul/rate/handler.go

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>

* formatting handler.go

* Rate limiting handler - ensure configuration has changed before modifying limiters

* Updating test to validate arguments to UpdateConfig

* Removing duplicate test.  Updating mock.

* adding logging for when UpdateConfig is called but the config has not changed.

* Update agent/consul/rate/handler.go

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>

* Update agent/consul/rate/handler_test.go

Co-authored-by: Dan Upton <daniel@floppy.co>

* modifying existing variable name based on pr feedback

* updating a broken merge conflict;

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
Co-authored-by: Dan Upton <daniel@floppy.co>
2022-12-20 15:00:22 -07:00