Commit Graph

73 Commits

Author SHA1 Message Date
Semir Patel 3fb8dda960
[BACKPORT] 1.16.x manual backport of OSS->CE branch (#18549) 2023-08-23 11:53:44 -05:00
hc-github-team-consul-core 7b03d95d6f
backport of commit f2e26c36eced387f8b5396e5969f37c8d4b6493c (#17870)
Co-authored-by: Daniel Upton <daniel@floppy.co>
2023-06-26 12:40:50 +00:00
Matt Keeler 08f9cd1678
Implement the service endpoints controller (#17216)
* Add a ReplaceType dep mapper and move them into their own file

* Implement the service endpoints controller

* Implement a Catalog Controllers Integration Test
2023-06-06 17:09:48 -04:00
Matt Keeler e909289454
Various bits of cleanup detected when using Go Workspaces (#17462)
TLDR with many modules the versions included in each diverged quite a bit. Attempting to use Go Workspaces produces a bunch of errors.

This commit:

1. Fixes envoy-library-references.sh to work again
2. Ensures we are pulling in go-control-plane@v0.11.0 everywhere (previously it was at that version in some modules and others were much older)
3. Remove one usage of golang/protobuf that caused us to have a direct dependency on it.
4. Remove deprecated usage of the Endpoint field in the grpc resolver.Target struct. The current version of grpc (v1.55.0) has removed that field and recommended replacement with URL.Opaque and calls to the Endpoint() func when needing to consume the previous field.
4. `go work init <all the paths to go.mod files>` && `go work sync`. This syncrhonized versions of dependencies from the main workspace/root module to all submodules
5. Updated .gitignore to ignore the go.work and go.work.sum files. This seems to be standard practice at the moment.
6. Update doc comments in protoc-gen-consul-rate-limit to be go fmt compatible
7. Upgraded makefile infra to perform linting, testing and go mod tidy on all modules in a flexible manner.
8. Updated linter rules to prevent usage of golang/protobuf
9. Updated a leader peering test to account for an extra colon in a grpc error message.
2023-06-05 16:08:39 -04:00
Dan Stough e502be8c6e
[OSS] gRPC Blocking Queries (#17426)
* feat: initial grpc blocking queries

* changelog and docs update
2023-05-23 17:29:10 -04:00
Matt Keeler 6216a96f93
Add the workload health controller (#17215) 2023-05-19 13:53:29 -04:00
R.B. Boyer ce6bf1d82e
fix two typos (#17389) 2023-05-17 08:50:26 -07:00
Matt Keeler e23577b8aa
Add a Node health controller (#17214)
This will aggregate all HealthStatus objects owned by the Node and update the status of the Node with an overall health.
2023-05-15 09:55:03 -04:00
Dan Upton 35674ae6b9
docs: initial documentation for the new State Store (#17315) 2023-05-15 12:34:36 +01:00
Matt Keeler d4e176ba09
Add type validations for the catalog resources (#17211)
Also adding some common resource validation error types to the internal/resource package.
2023-05-12 09:24:55 -04:00
Dan Upton 0d54d9a678
resource: optionally compare timestamps in `EqualStatus` (#17275) 2023-05-10 10:37:54 +01:00
Semir Patel f8b900d555
Reaper controller for cascading deletes of owner resources (#17256) 2023-05-09 13:57:40 -05:00
Dan Upton 270df96301
resource: add helpers for more efficiently comparing IDs etc (#17224) 2023-05-09 19:02:24 +01:00
Dan Upton 91f76b6fb2
controller: deduplicate items in queue (#17168) 2023-05-09 18:14:20 +01:00
Dan Upton 979ef66885
Controller Runtime 2023-05-09 15:25:55 +01:00
Matt Keeler 6919dabb50
Register new catalog & mesh protobuf types with the resource registry (#17225) 2023-05-08 15:36:35 -04:00
Semir Patel 9615837c60
resource: List resources by owner (#17190) 2023-05-08 12:26:19 -05:00
Semir Patel 896c39d98c
Create tombstone on resource `Delete` (#17108) 2023-04-28 10:49:08 -05:00
Dan Upton 15cc86ba91
Cleanup from unblocking the pipeline 🧹 (#17121) 2023-04-26 13:59:58 +01:00
Semir Patel cf50def90b
Fix or disable pipeline breaking changes that made it into main in last day or so (#17130)
* Fix straggler from renaming Register->RegisterTypes

* somehow a lint failure got through previously

* Fix lint-consul-retry errors

* adding in fix for success jobs getting skipped. (#17132)

* Temporarily disable inmem backend conformance test to get green pipeline

* Another test needs disabling

---------

Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-04-25 15:17:48 -05:00
Dan Upton f7c4f04060
Controller Supervision (#17016) 2023-04-25 12:52:35 +01:00
Dan Upton 5979752994
storage: fix bug where WatchList would (rarely) return duplicate events (#17067) 2023-04-25 11:48:13 +01:00
hashicorp-copywrite[bot] 87aee8308b
[COMPLIANCE] Add Copyright and License Headers (#16854)
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
2023-04-20 12:40:22 +00:00
Semir Patel 0674f30fc1
Tenancy wildcard validaton for `Write`, `Read`, and `Delete` endpoints (#17004) 2023-04-17 16:33:20 -05:00
Semir Patel f9311318e1
Add mutate hook to `Write` endpoint (#16958) 2023-04-12 16:50:07 -05:00
Semir Patel 53a0755f03
Enforce ACLs on resource `Write` and `Delete` endpoints (#16956) 2023-04-12 16:22:44 -05:00
Dan Upton d46543631c
resource: `WriteStatus` endpoint (#16886) 2023-04-11 19:23:14 +01:00
Semir Patel 8d0d600ea3
Resource validation hook for `Write` endpoint (#16950) 2023-04-11 06:55:32 -05:00
Semir Patel ca19954c08
Check acls on resource `Read`, `List`, and `WatchList` (#16842) 2023-04-11 06:10:14 -05:00
Dan Upton 4e8ab7a390
Resource `Write` endpoint (#16786) 2023-04-06 10:40:04 +01:00
Dan Upton 52ce151221
Raft storage backend (#16619) 2023-04-04 17:30:06 +01:00
Dan Upton 37207b4e4c
storage: fix resource leak in Watch (#16817) 2023-03-31 13:24:19 +01:00
Ronald 71fb0a723e
Copyright headers for missing files/folders (#16708)
* copyright headers for agent folder
2023-03-28 18:48:58 -04:00
Dan Upton 71f67d7979
Add storage backend interface and in-memory implementation (#16538)
Introduces `storage.Backend`, which will serve as the interface between the
Resource Service and the underlying storage system (Raft today, but in the
future, who knows!).

The primary design goal of this interface is to keep its surface area small,
and push as much functionality as possible into the layers above, so that new
implementations can be added with little effort, and easily proven to be
correct. To that end, we also provide a suite of "conformance" tests that can
be run against a backend implementation to check it behaves correctly.

In this commit, we introduce an initial in-memory storage backend, which is
suitable for tests and when running Consul in development mode. This backend is
a thin wrapper around the `Store` type, which implements a resource database
using go-memdb and our internal pub/sub system. `Store` will also be used to
handle reads in our Raft backend, and in the future, used as a local cache for
external storage systems.
2023-03-27 10:30:53 +01:00
Poonam Jadhav 3be683fcc1
feat: add category annotation to RPC and gRPC methods (#16646) 2023-03-20 11:24:29 -04:00
Semir Patel f0c36029b8
Basic resource type registry (#16622) 2023-03-14 13:30:25 -05:00
Matt Keeler 9ef8ef9f3e
Remove private prefix from proto-gen-rpc-glue e2e test (#16433) 2023-03-03 14:05:14 -05:00
Matt Keeler f3c80c4eef
Protobuf Refactoring for Multi-Module Cleanliness (#16302)
Protobuf Refactoring for Multi-Module Cleanliness

This commit includes the following:

Moves all packages that were within proto/ to proto/private
Rewrites imports to account for the packages being moved
Adds in buf.work.yaml to enable buf workspaces
Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml
Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes)

Why:

In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage.
There were some recent changes to have our own ratelimiting annotations.
The two combined were not working when I was trying to use them together (attempting to rebase another branch)
Buf workspaces should be the solution to the problem
Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root.
This resulted in proto file name conflicts in the Go global protobuf type registry.
The solution to that was to add in a private/ directory into the path within the proto/ directory.
That then required rewriting all the imports.

Is this safe?

AFAICT yes
The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc)
Other than imports, there were no changes to any generated code as a result of this.
2023-02-17 16:14:46 -05:00
Matt Keeler 554f1e6fee
Protobuf Modernization (#15949)
* Protobuf Modernization

Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf

Marshallers (protobuf and json) needed some changes to account for different APIs.

Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package.

This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces.

* Fix go-mod-tidy make target to work on all modules
2023-01-11 09:39:10 -05:00
Dan Upton a36173e2b5
Fix missing comment in generated enterprise rate-limit file (#15895)
Fixes bug in #15564 where gofmt would strip out the generated code
warning comment because it was on the same line as the build tag.
2023-01-05 10:31:14 +00:00
Dan Upton 1d95609fb7
grpc: `protoc` plugin for generating gRPC rate limit specifications (#15564)
Adds automation for generating the map of `gRPC Method Name → Rate Limit Type`
used by the middleware introduced in #15550, and will ensure we don't forget
to add new endpoints.

Engineers must annotate their RPCs in the proto file like so:

```
rpc Foo(FooRequest) returns (FooResponse) {
  option (consul.internal.ratelimit.spec) = {
    operation_type: READ,
  };
}
```

When they run `make proto` a protoc plugin `protoc-gen-consul-rate-limit` will
be installed that writes rate-limit specs as a JSON array to a file called
`.ratelimit.tmp` (one per protobuf package/directory).

After running Buf, `make proto` will execute a post-process script that will
ingest all of the `.ratelimit.tmp` files and generate a Go file containing the
mappings in the `agent/grpc-middleware` package. In the enterprise repository,
it will write an additional file with the enterprise-only endpoints.

If an engineer forgets to add the annotation to a new RPC, the plugin will
return an error like so:

```
RPC Foo is missing rate-limit specification, fix it with:

	import "proto-public/annotations/ratelimit/ratelimit.proto";

	service Bar {
	  rpc Foo(...) returns (...) {
	    option (hashicorp.consul.internal.ratelimit.spec) = {
	      operation_type: OPERATION_READ | OPERATION_WRITE | OPERATION_EXEMPT,
	    };
	  }
	}
```

In the future, this annotation can be extended to support rate-limit
category (e.g. KV vs Catalog) and to determine the retry policy.
2023-01-04 16:07:02 +00:00
Chris S. Kim 450c8b9eff
[OSS] Add boilerplate for proto files implementing BlockableQuery (#15554) 2022-11-25 15:46:56 -05:00
Kyle Schochenmaier 2b1e5f69e2
removes ioutil usage everywhere which was deprecated in go1.16 (#15297)
* update go version to 1.18 for api and sdk, go mod tidy
* removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward.
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-11-10 10:26:01 -06:00
Chris S. Kim ae1646706f Regenerate files according to 1.19.2 formatter 2022-10-24 16:12:08 -04:00
Chris S. Kim f7d248312e Remove unused methods from template 2022-10-18 16:35:16 -04:00
Chris S. Kim e4c20ec190
Refactor client RPC timeouts (#14965)
Fix an issue where rpc_hold_timeout was being used as the timeout for non-blocking queries. Users should be able to tune read timeouts without fiddling with rpc_hold_timeout. A new configuration `rpc_read_timeout` is created.

Refactor some implementation from the original PR 11500 to remove the misleading linkage between RPCInfo's timeout (used to retry in case of certain modes of failures) and the client RPC timeouts.
2022-10-18 15:05:09 -04:00
Paul Glass a9f17c0f99
Extract AWS auth implementation out of Consul (#13760) 2022-07-19 16:26:44 -05:00
alex 84b458149f
fix leader annotation (#13786)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-18 10:34:59 -07:00
alex 5eaab0efcb
peering: add warning about AllowStaleRead (#13768) 2022-07-15 09:56:33 -07:00
Evan Culver bea369ec73
internal: port RPC glue changes from Enterprise (#13034)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-05-11 15:03:07 -07:00