Commit Graph

375 Commits

Author SHA1 Message Date
hc-github-team-consul-core adefe8d16a
Backport of [NET-5944] security: Update Go version to 1.20.10 and `x/net` to 0.17.0 into release/1.16.x (#19234)
* backport of commit d7d9de95642a625307fecc8cff29bda1c22fa983

* backport of commit 0794b1ce74d37623d50f5e52cb289a48e8558d18

---------

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
2023-10-17 00:12:24 +00:00
Nathan Coleman f77e477ebe
Revert "Backport of NET-5530 Support response header modifiers on http-route config entry into release/1.16.x" (#18770)
Revert "Backport of NET-5530 Support response header modifiers on http-route config entry into release/1.16.x (#18725)"

This reverts commit b8a0e849832378b63732b40f6ea849b9c6c35803.
2023-09-12 19:58:24 +00:00
hc-github-team-consul-core 07c75c2b27
Backport of NET-5530 Support response header modifiers on http-route config entry into release/1.16.x (#18725)
* NET-5530 Support response header modifiers on http-route config entry (#18646)

* Add response header filters to http-route config entry definitions

* Map response header filters from config entry when constructing route destination

* Support response header modifiers at the service level as well

* Update protobuf definitions

* Update existing unit tests

* Add response filters to route consolidation logic

* Make existing unit tests more robust

* Add missing docstring

* Add changelog entry

* Add response filter modifiers to existing integration test

* Add more robust testing for response header modifiers in the discovery chain

* Add more robust testing for request header modifiers in the discovery chain

* Modify test to verify that service filter modifiers take precedence over rule filter modifiers

* Generate deep-copy code

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-09-08 20:17:35 +00:00
Semir Patel 3fb8dda960
[BACKPORT] 1.16.x manual backport of OSS->CE branch (#18549) 2023-08-23 11:53:44 -05:00
hc-github-team-consul-core 8936fe951a
Backport of [NET-5163] Support locality testing in consul-container into release/1.16.x (#18503)
backport of commit 61b7c0d76ff33f883dfad875d1a39bfd115b332b

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
2023-08-17 15:09:30 +00:00
hc-github-team-consul-core e08f826ce6
Backport of NET-5187: Upgrade test timeout due to log producer errors into release/1.16.x (#18472)
backport of commit 59bfc81d49a905480329a38c0d8d0109ea100d3e

Co-authored-by: NiniOak <anita.akaeze@hashicorp.com>
2023-08-15 17:20:44 +00:00
cskh 25f45d1266
Backport of bump testcontainers-go to 0.22.0 into release/1.16.x (#18417) 2023-08-09 13:04:32 -04:00
cskh 97d44d170b
Backport 1.16.x Upgrade test: remove outdated test and disable log due to verbosity (… (#18413)
Upgrade test: remove outdated test and disable log due to verbosity (#18403)

* remove outdated test

* disable log since we have too many parallel tests
2023-08-08 18:23:07 -04:00
Poonam Jadhav b485fa2104
fix: go sum revert (#18389) 2023-08-04 18:10:54 -04:00
Poonam Jadhav 37e606d6a3 release: bump submodules versions in test containers module 2023-08-04 17:37:34 -04:00
hc-github-team-consul-core d1a52f31a2
Backport of [NET-5146] security: Update Go version to 1.20.7 and `x/net` to 0.13.0 into release/1.16.x (#18363)
backport of commit 905e371607112dc00c55cae53c907b989a651f61

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
2023-08-02 18:36:08 +00:00
hc-github-team-consul-core 74a4aac071
Backport of Update submodules to latest following 1.16.0 into release/1.16.x (#18198)
Update submodules to latest following 1.16.0

Align all our internal use of submodules on the latest versions.

Manual backport of 235d9c5ca5bccdad2e853f0ac052e25efdef1ca5.
2023-07-20 01:35:24 +00:00
hc-github-team-consul-core 7aef7ebc42
Backport of [NET-4865] Bump golang.org/x/net to 0.12.0 into release/1.16.x (#18189)
Bump golang.org/x/net to 0.12.0

While not necessary to directly address CVE-2023-29406 (which should be
handled by using a patched version of Go when building), an
accompanying change to HTTP/2 error handling does impact agent code.

See https://go-review.googlesource.com/c/net/+/506995 for the HTTP/2
change.

Bump this dependency across our submodules as well for the sake of
potential indirect consumers of `x/net/http`.

Manual backport of 84cbf09185ebfc59f9fcf486d1c4983ef129bf95.
2023-07-19 12:22:18 -04:00
hc-github-team-consul-core e1ef25d91a
Backport of [NET-4792] Add integrations tests for jwt-auth into release/1.16.x (#18173)
backport of commit e588acf2ee4752334f640be389ae3843aac501de

Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
2023-07-18 19:14:05 +00:00
hc-github-team-consul-core 761fdacda6
Backport of chore: bump upgrade integrations tests to 1.15, 116 [NET-4743] into release/1.16.x (#18148)
Co-authored-by: Nick Irvine <115657443+nfi-hashicorp@users.noreply.github.com>
2023-07-18 10:04:11 -07:00
hc-github-team-consul-core bb0c534363
Backport of Re-order expected/actual for assertContainerState in consul container tests into release/1.16.x (#18158)
backport of commit d5bed27d980058151d884980ad70071c8f235f37

Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
2023-07-17 18:29:36 +00:00
hc-github-team-consul-core db4b2cb577
Backport of Use JWT-auth filter in metadata mode & Delegate validation to RBAC filter into release/1.16.x (#18153)
## Backport

This PR is auto-generated from #18062 to be assessed for backporting due
to the inclusion of the label backport/1.16.



The below text is copied from the body of the original PR.

---

### Description

<!-- Please describe why you're making this change, in plain English.
-->

- Currently the jwt-auth filter doesn't take into account the service
identity when validating jwt-auth, it only takes into account the path
and jwt provider during validation. This causes issues when multiple
source intentions restrict access to an endpoint with different JWT
providers.
- To fix these issues, rather than use the JWT auth filter for
validation, we use it in metadata mode and allow it to forward the
successful validated JWT token payload to the RBAC filter which will
make the decisions.

This PR ensures requests with and without JWT tokens successfully go
through the jwt-authn filter. The filter however only forwards the data
for successful/valid tokens. On the RBAC filter level, we check the
payload for claims and token issuer + existing rbac rules.

### Testing & Reproduction steps

<!--

* In the case of bugs, describe how to replicate
* If any manual tests were done, document the steps and the conditions
to replicate
* Call out any important/ relevant unit tests, e2e tests or integration
tests you have added or are adding

-->

- This test covers a multi level jwt requirements (requirements at top
level and permissions level). It also assumes you have envoy running,
you have a redis and a sidecar proxy service registered, and have a way
to generate jwks with jwt. I mostly use:
https://www.scottbrady91.com/tools/jwt for this.

- first write your proxy defaults
```
Kind = "proxy-defaults"
name = "global"
config {
  protocol = "http"
}
```
- Create two providers 
```
Kind = "jwt-provider"
Name = "auth0"
Issuer = "https://ronald.local"

JSONWebKeySet = {
    Local = {
     JWKS = "eyJrZXlzIjog....."
    }
}
```

```
Kind = "jwt-provider"
Name = "okta"
Issuer = "https://ronald.local"

JSONWebKeySet = {
   Local = {
     JWKS = "eyJrZXlzIjogW3...."
    }
}
```

- add a service intention
```
Kind = "service-intentions"
Name = "redis"

JWT = {
  Providers = [
    {
      Name = "okta"
    },
  ]
}

Sources = [
  {
    Name = "*"
    Permissions = [{
      Action = "allow"
      HTTP = {
        PathPrefix = "/workspace"
      }
      JWT = {
        Providers = [
          {
            Name = "okta"
            VerifyClaims = [
              {
                  Path = ["aud"]
                  Value = "my_client_app"
              },
              {
                Path = ["sub"]
                Value = "5be86359073c434bad2da3932222dabe"
              }
            ]
          },
        ]
      }

    },
    {
      Action = "allow"
      HTTP = {
        PathPrefix = "/"
      }
      JWT = {
        Providers = [
          {
            Name = "auth0"
          },
        ]
      }

    }]
  }
]
```
- generate 3 jwt tokens: 1 from auth0 jwks, 1 from okta jwks with
different claims than `/workspace` expects and 1 with correct claims
- connect to your envoy (change service and address as needed) to view
logs and potential errors. You can add: `-- --log-level debug` to see
what data is being forwarded
```
consul connect envoy -sidecar-for redis1 -grpc-addr 127.0.0.1:8502
```
- Make the following requests: 
```
curl -s -H "Authorization: Bearer $Auth0_TOKEN" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v

RBAC filter denied

curl -s -H "Authorization: Bearer $Okta_TOKEN_with_wrong_claims" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v

RBAC filter denied

curl -s -H "Authorization: Bearer $Okta_TOKEN_with_correct_claims" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v

Successful request
```


### TODO

* [x] Update test coverage
* [ ] update integration tests (follow-up PR)
* [x] appropriate backport labels added


---

<details>
<summary> Overview of commits </summary>

  - 70536f5a38507d7468f62d00dd93a6968a3d9cf3 

</details>

Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
2023-07-17 15:50:21 +00:00
hc-github-team-consul-core 42a6d1e70f
Backport of [OSS] Fix initial_fetch_timeout to wait for all xDS resources into release/1.16.x (#18065)
* backport of commit 8a2f60ddae1a6ac561544e9cae80e9a037ad06d5

* backport of commit e17e53c93373fadedd61e904949e87c0c7d5ed26

* backport of commit d919d55c2eb4f206840f8d880edda8d5ad8c5fb4

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-07-10 21:27:56 +00:00
hc-github-team-consul-core 9d8ba91bc6
Add first integration test for jwt auth with intention (#18005) (#18029)
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
2023-07-06 07:54:30 -04:00
hc-github-team-consul-core a83bd1c1dc
Backport of [OSS] Improve Gateway Test Coverage of Catalog Health into release/1.16.x (#18014)
* backport of commit 954bd6ab1f1a2a00f549b10ad435cdead8d2cae2

* backport of commit 85c32d8f2e7e2c3a2855fe7a8fc4d10e3865b81f

* backport of commit 7ea3d622d75b4a69b8fc51d181b79c6b170ea47a

* backport of commit 127ae69c6dc967d575929e920813e7fe0d3fdef1

* backport of commit e04099b6cdd5dc20a36a19897816069669b2ef92

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-07-05 15:36:32 -04:00
hc-github-team-consul-core 810870d1c8
Backport of Integration test for ext-authz Envoy extension into release/1.16.x (#18003)
* backport of commit 6699b173136276c3b9d6bc7ed126d8f5dbd7c0a1

* resolve failed cherrypick

* remove extraneous changes to agent.go

---------

Co-authored-by: Chris Thain <chris.m.thain@gmail.com>
2023-07-04 12:45:08 -07:00
hc-github-team-consul-core 08547ba585
backport of commit c0afba3a0c2ae093fee756a9019d49db25367d69 (#17975)
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2023-06-30 14:20:50 +00:00
hc-github-team-consul-core bd85965844
backport of commit 21263c8a004dffe7e8fcefc234d80abfa3ec66d5 (#17811)
Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
2023-06-20 14:17:28 +00:00
Michael Zalimeni e1458b94f5
Update submodules for 1.16.0-rc1 release (#17626)
Update to new RC submodule versions. These will be updated again for the
1.16.0 release.
2023-06-08 18:45:42 -04:00
Michael Zalimeni 378a15af32
Fix Property Override Services parsing (#17584)
Ensure that the embedded api struct is properly parsed when
deserializing config containing a set ResourceFilter.Services field.

Also enhance existing integration test to guard against bugs and
exercise this field.
2023-06-06 15:40:37 -04:00
Michael Zalimeni 1147603f97
Add Prop Override Envoy extension integration test (#17569) 2023-06-06 10:04:31 -04:00
Matt Keeler e909289454
Various bits of cleanup detected when using Go Workspaces (#17462)
TLDR with many modules the versions included in each diverged quite a bit. Attempting to use Go Workspaces produces a bunch of errors.

This commit:

1. Fixes envoy-library-references.sh to work again
2. Ensures we are pulling in go-control-plane@v0.11.0 everywhere (previously it was at that version in some modules and others were much older)
3. Remove one usage of golang/protobuf that caused us to have a direct dependency on it.
4. Remove deprecated usage of the Endpoint field in the grpc resolver.Target struct. The current version of grpc (v1.55.0) has removed that field and recommended replacement with URL.Opaque and calls to the Endpoint() func when needing to consume the previous field.
4. `go work init <all the paths to go.mod files>` && `go work sync`. This syncrhonized versions of dependencies from the main workspace/root module to all submodules
5. Updated .gitignore to ignore the go.work and go.work.sum files. This seems to be standard practice at the moment.
6. Update doc comments in protoc-gen-consul-rate-limit to be go fmt compatible
7. Upgraded makefile infra to perform linting, testing and go mod tidy on all modules in a flexible manner.
8. Updated linter rules to prevent usage of golang/protobuf
9. Updated a leader peering test to account for an extra colon in a grpc error message.
2023-06-05 16:08:39 -04:00
Poonam Jadhav 67be774707
feat: expose logs method on container interface (#17526) 2023-05-31 13:18:00 -04:00
Paul Glass 3759844033
Integration test for permissive mTLS (#17205)
* Integration test for permissive mTLS
2023-05-23 11:11:27 -05:00
Paul Glass 71992b9c3b
Only synthesize anonymous token in primary DC (#17231)
* Only synthesize anonymous token in primary DC
* Add integration test for wan fed issue
2023-05-23 09:38:04 -05:00
sarahalsmiller 97532900a5
xds: generate endpoints directly from API gateway snapshot (#17390)
* endpoints xds cluster configuration

* resources test fix

* fix reversion in resources_test

* Update agent/proxycfg/api_gateway.go

Co-authored-by: John Maguire <john.maguire@hashicorp.com>

* gofmt

* Modify getReadyUpstreams to filter upstreams by listener (#17410)

Each listener would previously have all upstreams from any route that bound to the listener. This is problematic when a route bound to one listener also binds to other listeners and so includes upstreams for multiple listeners. The list for a given listener would then wind up including upstreams for other listeners.

* Update agent/proxycfg/api_gateway.go

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>

* Restore import blocking

* Skip to next route if route has no upstreams

* cleanup

* change set from bool to empty struct

---------

Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-05-19 18:50:59 +00:00
cskh e8ccc911af
consul-container test: no splitting and on single runner (#17394) 2023-05-17 14:57:12 -04:00
R.B. Boyer 00a9e03f44
test: slight refactoring ahead of peering testing improvements (#17387) 2023-05-16 14:57:24 -05:00
R.B. Boyer a20102560e
test: fix oss/ent drift in gateway container tests (#17365) 2023-05-16 11:49:27 -05:00
cskh 2f18616aa7
upgrade test: fix on-the-fly-image build and downsize runner (#17331) 2023-05-15 09:33:05 -04:00
cskh b5b0a34ca5
consul-container: mitigate the drift from ent repo (#17323) 2023-05-12 13:03:30 -04:00
cskh 09de8cedca
Container test: fix container test slow image build (#17316)
Container integ test: fix container test slow image build
2023-05-11 22:49:49 +00:00
cskh b5d2fafd7a
Upgrade test target image (#17226)
* upgrade test: add targetimage name as parameter to upgrade function

- the image name of latest version and target version could be
  different. Add the parameter of targetImage to the upgrade
  function

* fix a bug of expected error
2023-05-08 12:02:31 -04:00
Semir Patel 406c1afc04
Support Envoy's MaxEjectionPercent and BaseEjectionTime config entries for passive health checks (#15979)
* Add MaxEjectionPercent to config entry

* Add BaseEjectionTime to config entry

* Add MaxEjectionPercent and BaseEjectionTime to protobufs

* Add MaxEjectionPercent and BaseEjectionTime to api

* Fix integration test breakage

* Verify MaxEjectionPercent and BaseEjectionTime in integration test upstream confings

* Website docs for MaxEjectionPercent and BaseEjection time

* Add `make docs` to browse docs at http://localhost:3000

* Changelog entry

* so that is the difference between consul-docker and dev-docker

* blah

* update proto funcs

* update proto

---------

Co-authored-by: Maliz <maliheh.monshizadeh@hashicorp.com>
2023-04-26 15:59:48 -07:00
Paul Glass 69e9e21bf4
TProxy integration test (#17103)
* TProxy integration test
* Fix GHA compatibility integration test command

Previously, when test splitting allocated multiple test directories to a
runner, the workflow ran `go tests "./test/dir1 ./test/dir2"` which
results in a directory not found error. This fixes that.
2023-04-26 11:49:38 -05:00
Semir Patel cf50def90b
Fix or disable pipeline breaking changes that made it into main in last day or so (#17130)
* Fix straggler from renaming Register->RegisterTypes

* somehow a lint failure got through previously

* Fix lint-consul-retry errors

* adding in fix for success jobs getting skipped. (#17132)

* Temporarily disable inmem backend conformance test to get green pipeline

* Another test needs disabling

---------

Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-04-25 15:17:48 -05:00
Paul Banks ce96b2c69d
De-flake snapshot test (#17120) 2023-04-25 15:25:26 +01:00
John Maguire c5b7164b16
APIGW Normalize Status Conditions (#16994)
* normalize status conditions for gateways and routes

* Added tests for checking condition status and panic conditions for
validating combinations, added dummy code for fsm store

* get rid of unneeded gateway condition generator struct

* Remove unused file

* run go mod tidy

* Update tests, add conflicted gateway status

* put back removed status for test

* Fix linting violation, remove custom conflicted status

* Update fsm commands oss

* Fix incorrect combination of type/condition/status

* cleaning up from PR review

* Change "invalidCertificate" to be of accepted status

* Move status condition enums into api package

* Update gateways controller and generated code

* Update conditions in fsm oss tests

* run go mod tidy on consul-container module to fix linting

* Fix type for gateway endpoint test

* go mod tidy from changes to api

* go mod tidy on troubleshoot

* Fix route conflicted reason

* fix route conflict reason rename

* Fix text for gateway conflicted status

* Add valid certificate ref condition setting

* Revert change to resolved refs to be handled in future PR
2023-04-24 16:22:55 -04:00
R.B. Boyer e76795dc08
fix the linter (#17077) 2023-04-20 17:49:08 -04:00
Anita Akaeze af6e061d05
NET-3648: Add script to get consul and envoy version (#17060) 2023-04-20 13:11:11 -04:00
hashicorp-copywrite[bot] 87aee8308b
[COMPLIANCE] Add Copyright and License Headers (#16854)
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
2023-04-20 12:40:22 +00:00
John Murret d7c488762e
ci: remove test-integrations CircleCI workflow (#16928)
* remove all CircleCI files

* remove references to CircleCI

* remove more references to CircleCI

* pin golangci-lint to v1.51.1 instead of v1.51
2023-04-19 16:19:29 +00:00
Dhia Ayachi 41064eb20b
add ability to start container tests in debug mode and attach a debugger (#16887)
* add ability to start container tests in debug mode and attach a debugger to consul while running it.

* add a debug message with the debug port

* use pod to get the right port

* fix image used in basic test

* add more data to identify which container to debug.

* fix comment

Co-authored-by: Evan Culver <eculver@users.noreply.github.com>

* rename debugUri to debugURI

---------

Co-authored-by: Evan Culver <eculver@users.noreply.github.com>
2023-04-18 09:49:53 -04:00
Michael Wilkerson 4edb1b553d
* added Sameness Group to proto files (#16998)
- added Sameness Group to config entries
- added Sameness Group to subscriptions

* generated proto files

* added Sameness Group events to the state store
- added test cases

* Refactored health RPC Client
- moved code that is common to rpcclient under rpcclient common.go. This will help set us up to support future RPC clients

* Refactored proxycfg glue views
- Moved views to rpcclient config entry. This will allow us to reuse this code for a config entry client

* added config entry RPC Client
- Copied most of the testing code from rpcclient/health

* hooked up new rpcclient in agent

* fixed documentation and comments for clarity
2023-04-14 09:24:46 -07:00
cskh ebf0910d54
upgrade test: config nodeName, nodeid, and inherited persistent data for consul container (#16931) 2023-04-12 18:00:56 -04:00