From 3046ad707b23d40dcb097db959b4373867ed452b Mon Sep 17 00:00:00 2001 From: Mark Anderson Date: Tue, 31 May 2022 16:57:50 -0700 Subject: [PATCH 001/247] yUpdate website/content/docs/connect/ca/vault.mdx Port some changes that were made to the backport branch but not in the original PR. Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Signed-off-by: Mark Anderson --- website/content/docs/connect/ca/vault.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/content/docs/connect/ca/vault.mdx b/website/content/docs/connect/ca/vault.mdx index 02294c2c2..7ab597ffc 100644 --- a/website/content/docs/connect/ca/vault.mdx +++ b/website/content/docs/connect/ca/vault.mdx @@ -137,7 +137,7 @@ The configuration options are listed below. that authorized it. - `RootPKINamespace` / `root_pki_namespace` (`string: `) - The absolute namespace - that the `RootPKIPath` is in. Setting this overrides the `Namespace` option for the `RootPKIPath`. Introduced in 1.12.1 + that the `RootPKIPath` is in. Setting this parameter overrides the `Namespace` option for the `RootPKIPath`. Introduced in 1.12.1. - `IntermediatePKIPath` / `intermediate_pki_path` (`string: `) - The path to a PKI secrets engine for the generated intermediate certificate. @@ -149,7 +149,7 @@ The configuration options are listed below. datacenter must specify a unique `intermediate_pki_path`. - `IntermediatePKINamespace` / `intermediate_pki_namespace` (`string: `) - The absolute namespace - that the `IntermediatePKIPath` is in. Setting this overrides the `Namespace` option for the `IntermediatePKIPath`. Introduced in 1.12.1 + that the `IntermediatePKIPath` is in. Setting this parameter overrides the `Namespace` option for the `IntermediatePKIPath`. Introduced in 1.12.1. - `CAFile` / `ca_file` (`string: ""`) - Specifies an optional path to the CA certificate used for Vault communication. If unspecified, this will fallback From ac747888fd7008827afc1dbcbba9768eceafebd4 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Wed, 1 Jun 2022 13:56:22 -0400 Subject: [PATCH 002/247] docs(consul-api-gateway): add GatewayClassConfig deployment.defaultInstances --- website/content/docs/api-gateway/consul-api-gateway-install.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 087c0e70e..2f8b1ac33 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -168,6 +168,7 @@ The following table describes the allowed parameters for the `spec` array: | `consul.ports.http` | Specifies the port for Consul's HTTP server. | Integer | `8500` | | `consul.scheme` | Specifies the scheme to use for connecting to Consul. The supported values are `"http"` and `"https"`. | String | `"http"` | | `copyAnnotations.service` | List of annotations to copy to the gateway service. | Array | `["external-dns.alpha.kubernetes.io/hostname"]` | +| `deployment.defaultInstances` | Specifies the number of gateway instances that should be deployed by default. | Integer | 1 | | `image.consulAPIGateway` | The image to use for consul-api-gateway. View available image tags on [DockerHub](https://hub.docker.com/r/hashicorp/consul-api-gateway/tags). | String | `"hashicorp/consul-api-gateway:RELEASE_VERSION"` | | `image.envoy` | Specifies the container image to use for Envoy. View available image tags on [DockerHub](https://hub.docker.com/r/envoyproxy/envoy/tags). | String | `"envoyproxy/envoy:RELEASE_VERSION"` | | `logLevel` | Specifies the error reporting level for logs. You can specify the following values: `error`, `warning`, `info`, `debug`, `trace`. | String | `"info"` | From df1666765a466fde750f11561432e42414102ad1 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Wed, 1 Jun 2022 13:59:50 -0400 Subject: [PATCH 003/247] docs(consul-api-gateway): add GatewayClassConfig deployment.maxInstances and deployment.minInstances --- website/content/docs/api-gateway/consul-api-gateway-install.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 2f8b1ac33..405c917ee 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -169,6 +169,8 @@ The following table describes the allowed parameters for the `spec` array: | `consul.scheme` | Specifies the scheme to use for connecting to Consul. The supported values are `"http"` and `"https"`. | String | `"http"` | | `copyAnnotations.service` | List of annotations to copy to the gateway service. | Array | `["external-dns.alpha.kubernetes.io/hostname"]` | | `deployment.defaultInstances` | Specifies the number of gateway instances that should be deployed by default. | Integer | 1 | +| `deployment.maxInstances` | Specifies the maximum allowed number of gateway instances. | Integer | 8 | +| `deployment.minInstances` | Specifies the minimum allowed number of gateway instances. | Integer | 1 | | `image.consulAPIGateway` | The image to use for consul-api-gateway. View available image tags on [DockerHub](https://hub.docker.com/r/hashicorp/consul-api-gateway/tags). | String | `"hashicorp/consul-api-gateway:RELEASE_VERSION"` | | `image.envoy` | Specifies the container image to use for Envoy. View available image tags on [DockerHub](https://hub.docker.com/r/envoyproxy/envoy/tags). | String | `"envoyproxy/envoy:RELEASE_VERSION"` | | `logLevel` | Specifies the error reporting level for logs. You can specify the following values: `error`, `warning`, `info`, `debug`, `trace`. | String | `"info"` | From 601c13eefec8ca37fefab4a1c33fe8abf31f4425 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Wed, 1 Jun 2022 14:06:18 -0400 Subject: [PATCH 004/247] docs(consul-api-gateway): gateway instances -> instances per gateway --- .../content/docs/api-gateway/consul-api-gateway-install.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 405c917ee..c231083ab 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -168,9 +168,9 @@ The following table describes the allowed parameters for the `spec` array: | `consul.ports.http` | Specifies the port for Consul's HTTP server. | Integer | `8500` | | `consul.scheme` | Specifies the scheme to use for connecting to Consul. The supported values are `"http"` and `"https"`. | String | `"http"` | | `copyAnnotations.service` | List of annotations to copy to the gateway service. | Array | `["external-dns.alpha.kubernetes.io/hostname"]` | -| `deployment.defaultInstances` | Specifies the number of gateway instances that should be deployed by default. | Integer | 1 | -| `deployment.maxInstances` | Specifies the maximum allowed number of gateway instances. | Integer | 8 | -| `deployment.minInstances` | Specifies the minimum allowed number of gateway instances. | Integer | 1 | +| `deployment.defaultInstances` | Specifies the number of instances per gateway that should be deployed by default. | Integer | 1 | +| `deployment.maxInstances` | Specifies the maximum allowed number of instances per gateway. | Integer | 8 | +| `deployment.minInstances` | Specifies the minimum allowed number of instances per gateway. | Integer | 1 | | `image.consulAPIGateway` | The image to use for consul-api-gateway. View available image tags on [DockerHub](https://hub.docker.com/r/hashicorp/consul-api-gateway/tags). | String | `"hashicorp/consul-api-gateway:RELEASE_VERSION"` | | `image.envoy` | Specifies the container image to use for Envoy. View available image tags on [DockerHub](https://hub.docker.com/r/envoyproxy/envoy/tags). | String | `"envoyproxy/envoy:RELEASE_VERSION"` | | `logLevel` | Specifies the error reporting level for logs. You can specify the following values: `error`, `warning`, `info`, `debug`, `trace`. | String | `"info"` | From 2601a7d5633eaf8033bd74e1bef872f5f1cb1535 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Wed, 1 Jun 2022 14:42:08 -0400 Subject: [PATCH 005/247] docs(consul-api-gateway): add Gateway scaling section --- .../consul-api-gateway-install.mdx | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index c231083ab..3e2e6be97 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -266,6 +266,25 @@ Add the `listener` object to the `gateway` configuration and specify the followi Refer to the [Kubernetes Gateway API documentation](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.Listener) for details about configuring listeners. +#### Scaling + +A logical gateway object can be scaled to multiple instances within the bounds set in its associated GatewayClassConfig by using the [`kubectl scale`](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment) command. + +``` +> kubectl get deployment --selector api-gateway.consul.hashicorp.com/name=example-gateway +NAME READY UP-TO-DATE AVAILABLE +example-gateway 1/1 1 1 +``` +``` +> kubectl scale deployment/example-gateway --replicas=3 +deployment.apps/api-gateway scaled +``` +``` +> k get deployment -n consul --selector api-gateway.consul.hashicorp.com/name=api-gateway +NAME READY UP-TO-DATE AVAILABLE +api-gateway 3/3 3 3 +``` + ### Route Routes are independent configuration objects that are associated with specific listeners. From 75166b6fbd88502ffb7bcf1b3c4f121b7b529b30 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Wed, 1 Jun 2022 15:00:06 -0400 Subject: [PATCH 006/247] docs(consul-api-gateway): fixup CLI prompt to match convention --- .../content/docs/api-gateway/consul-api-gateway-install.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 3e2e6be97..99b1ea676 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -271,16 +271,16 @@ Refer to the [Kubernetes Gateway API documentation](https://gateway-api.sigs.k8s A logical gateway object can be scaled to multiple instances within the bounds set in its associated GatewayClassConfig by using the [`kubectl scale`](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment) command. ``` -> kubectl get deployment --selector api-gateway.consul.hashicorp.com/name=example-gateway +$ kubectl get deployment --selector api-gateway.consul.hashicorp.com/name=example-gateway NAME READY UP-TO-DATE AVAILABLE example-gateway 1/1 1 1 ``` ``` -> kubectl scale deployment/example-gateway --replicas=3 +$ kubectl scale deployment/example-gateway --replicas=3 deployment.apps/api-gateway scaled ``` ``` -> k get deployment -n consul --selector api-gateway.consul.hashicorp.com/name=api-gateway +$ k get deployment -n consul --selector api-gateway.consul.hashicorp.com/name=api-gateway NAME READY UP-TO-DATE AVAILABLE api-gateway 3/3 3 3 ``` From 39a450bb75f2fbdc5f083c658afa234872db44bf Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Wed, 1 Jun 2022 15:01:17 -0400 Subject: [PATCH 007/247] docs(consul-api-gateway): fixup code snippets in gateway scaling section --- .../docs/api-gateway/consul-api-gateway-install.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 99b1ea676..6aed5da1d 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -277,12 +277,12 @@ example-gateway 1/1 1 1 ``` ``` $ kubectl scale deployment/example-gateway --replicas=3 -deployment.apps/api-gateway scaled +deployment.apps/example-gateway scaled ``` ``` -$ k get deployment -n consul --selector api-gateway.consul.hashicorp.com/name=api-gateway -NAME READY UP-TO-DATE AVAILABLE -api-gateway 3/3 3 3 +$ kubectl get deployment --selector api-gateway.consul.hashicorp.com/name=example-gateway +NAME READY UP-TO-DATE AVAILABLE +example-gateway 3/3 3 3 ``` ### Route From a6079aa190aa759e09eaf055b5c8b64cbf3cd2ff Mon Sep 17 00:00:00 2001 From: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> Date: Thu, 2 Jun 2022 18:04:17 -0400 Subject: [PATCH 008/247] docs: show HCP Consul supports CTS enterprise --- website/content/docs/enterprise/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/enterprise/index.mdx b/website/content/docs/enterprise/index.mdx index 4dd288651..8375a3237 100644 --- a/website/content/docs/enterprise/index.mdx +++ b/website/content/docs/enterprise/index.mdx @@ -79,7 +79,7 @@ Available enterprise features per Consul form and license include: | [Redundancy Zones](/docs/enterprise/redundancy) | Not applicable | Yes | With Global Visibility, Routing, and Scale module | | [Read Replicas](/docs/enterprise/read-scale) | No | Yes | With Global Visibility, Routing, and Scale module | | [Automated Upgrades](/docs/enterprise/upgrades) | All tiers | Yes | Yes | -| [Consul-Terraform-Sync Enterprise](/docs/nia/enterprise) | No | Yes | Yes | +| [Consul-Terraform-Sync Enterprise](/docs/nia/enterprise) | All tiers | Yes | Yes | | [Network Areas](/docs/enterprise/federation) | No | Yes | With Global Visibility, Routing, and Scale module | | [Network Segments](/docs/enterprise/network-segments) | No | Yes | With Global Visibility, Routing, and Scale module | | [OIDC Auth Method](/docs/acl/auth-methods/oidc) | No | Yes | Yes | From 7602b6ebf2928830462235ed440b56f5cf51e2fb Mon Sep 17 00:00:00 2001 From: Dhia Ayachi Date: Tue, 7 Jun 2022 15:03:59 -0400 Subject: [PATCH 009/247] Egress gtw/connect destination intentions (#13341) * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough * remove duplicate tests * first draft of destinations intention in connect proxy * remove ServiceDestinationList * fix failing tests * fix agent/consul failing tests * change to filter intentions in the state store instead of adding a field. * fix failing tests * fix comment * fix comments * store service kind destination and add relevant tests * changes based on review * filter on destinations when querying source match * Apply suggestions from code review Co-authored-by: alex <8968914+acpana@users.noreply.github.com> * fix style * Apply suggestions from code review Co-authored-by: Dan Stough * rename destinationType to targetType. Co-authored-by: Dan Stough Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: github-team-consul-core --- agent/consul/intention_endpoint.go | 2 +- agent/consul/internal_endpoint.go | 2 +- agent/consul/state/catalog.go | 18 +------ agent/consul/state/config_entry.go | 12 ++++- agent/consul/state/config_entry_intention.go | 49 +++++++++++++++----- agent/consul/state/config_entry_test.go | 39 ++++++++++++++++ agent/consul/state/intention.go | 39 ++++++++++++---- agent/consul/state/intention_test.go | 4 +- agent/structs/intention.go | 9 ++++ agent/structs/structs.go | 5 ++ 10 files changed, 137 insertions(+), 42 deletions(-) diff --git a/agent/consul/intention_endpoint.go b/agent/consul/intention_endpoint.go index fc6db87db..b2c2d49c0 100644 --- a/agent/consul/intention_endpoint.go +++ b/agent/consul/intention_endpoint.go @@ -764,7 +764,7 @@ func (s *Intention) Check(args *structs.IntentionQueryRequest, reply *structs.In Partition: query.SourcePartition, Name: query.SourceName, } - _, intentions, err := store.IntentionMatchOne(nil, entry, structs.IntentionMatchSource) + _, intentions, err := store.IntentionMatchOne(nil, entry, structs.IntentionMatchSource, structs.IntentionTargetService) if err != nil { return fmt.Errorf("failed to query intentions for %s/%s", query.SourceNS, query.SourceName) } diff --git a/agent/consul/internal_endpoint.go b/agent/consul/internal_endpoint.go index 20169562d..113abd2dd 100644 --- a/agent/consul/internal_endpoint.go +++ b/agent/consul/internal_endpoint.go @@ -405,7 +405,7 @@ func (m *Internal) GatewayIntentions(args *structs.IntentionQueryRequest, reply Partition: gs.Service.PartitionOrDefault(), Name: gs.Service.Name, } - idx, intentions, err := state.IntentionMatchOne(ws, entry, structs.IntentionMatchDestination) + idx, intentions, err := state.IntentionMatchOne(ws, entry, structs.IntentionMatchDestination, structs.IntentionTargetService) if err != nil { return err } diff --git a/agent/consul/state/catalog.go b/agent/consul/state/catalog.go index ab4797fda..ee1623956 100644 --- a/agent/consul/state/catalog.go +++ b/agent/consul/state/catalog.go @@ -4009,14 +4009,7 @@ func (s *Store) ServiceTopology( Partition: entMeta.PartitionOrDefault(), Name: service, } - _, srcIntentions, err := compatIntentionMatchOneTxn( - tx, - ws, - matchEntry, - - // The given service is a source relative to its upstreams - structs.IntentionMatchSource, - ) + _, srcIntentions, err := compatIntentionMatchOneTxn(tx, ws, matchEntry, structs.IntentionMatchSource, structs.IntentionTargetService) if err != nil { return 0, nil, fmt.Errorf("failed to query intentions for %s", sn.String()) } @@ -4128,14 +4121,7 @@ func (s *Store) ServiceTopology( downstreamSources[svc.Name.String()] = source } - _, dstIntentions, err := compatIntentionMatchOneTxn( - tx, - ws, - matchEntry, - - // The given service is a destination relative to its downstreams - structs.IntentionMatchDestination, - ) + _, dstIntentions, err := compatIntentionMatchOneTxn(tx, ws, matchEntry, structs.IntentionMatchDestination, structs.IntentionTargetService) if err != nil { return 0, nil, fmt.Errorf("failed to query intentions for %s", sn.String()) } diff --git a/agent/consul/state/config_entry.go b/agent/consul/state/config_entry.go index e2cd8600f..f0e4c069e 100644 --- a/agent/consul/state/config_entry.go +++ b/agent/consul/state/config_entry.go @@ -356,12 +356,16 @@ func deleteConfigEntryTxn(tx WriteTxn, idx uint64, kind, name string, entMeta *a if gsKind == structs.GatewayServiceKindDestination { gsKind = structs.GatewayServiceKindUnknown } - if err := checkGatewayWildcardsAndUpdate(tx, idx, &structs.ServiceName{Name: c.GetName(), EnterpriseMeta: *c.GetEnterpriseMeta()}, gsKind); err != nil { + serviceName := structs.NewServiceName(c.GetName(), c.GetEnterpriseMeta()) + if err := checkGatewayWildcardsAndUpdate(tx, idx, &serviceName, gsKind); err != nil { return fmt.Errorf("failed updating gateway mapping: %s", err) } - if err := checkGatewayAndUpdate(tx, idx, &structs.ServiceName{Name: c.GetName(), EnterpriseMeta: *c.GetEnterpriseMeta()}, gsKind); err != nil { + if err := checkGatewayAndUpdate(tx, idx, &serviceName, gsKind); err != nil { return fmt.Errorf("failed updating gateway mapping: %s", err) } + if err := cleanupKindServiceName(tx, idx, serviceName, structs.ServiceKindDestination); err != nil { + return fmt.Errorf("failed to cleanup service name: \"%s\"; err: %v", serviceName, err) + } } } @@ -422,6 +426,10 @@ func insertConfigEntryWithTxn(tx WriteTxn, idx uint64, conf structs.ConfigEntry) if err := checkGatewayAndUpdate(tx, idx, &sn, gsKind); err != nil { return fmt.Errorf("failed updating gateway mapping: %s", err) } + + if err := upsertKindServiceName(tx, idx, structs.ServiceKindDestination, sn); err != nil { + return fmt.Errorf("failed to persist service name: %v", err) + } } } diff --git a/agent/consul/state/config_entry_intention.go b/agent/consul/state/config_entry_intention.go index 27c4912e6..bd539489a 100644 --- a/agent/consul/state/config_entry_intention.go +++ b/agent/consul/state/config_entry_intention.go @@ -208,7 +208,7 @@ func (s *Store) configIntentionMatchTxn(tx ReadTxn, ws memdb.WatchSet, args *str // improving that in the future, the test cases shouldn't have to // change for that. - index, ixns, err := configIntentionMatchOneTxn(tx, ws, entry, args.Type) + index, ixns, err := configIntentionMatchOneTxn(tx, ws, entry, args.Type, structs.IntentionTargetService) if err != nil { return 0, nil, err } @@ -224,14 +224,15 @@ func (s *Store) configIntentionMatchTxn(tx ReadTxn, ws memdb.WatchSet, args *str } func configIntentionMatchOneTxn( - tx ReadTxn, - ws memdb.WatchSet, + tx ReadTxn, ws memdb.WatchSet, matchEntry structs.IntentionMatchEntry, matchType structs.IntentionMatchType, + targetType structs.IntentionTargetType, ) (uint64, structs.Intentions, error) { switch matchType { + // targetType is only relevant for Source matches as egress Destinations can only be Intention Destinations in the mesh case structs.IntentionMatchSource: - return readSourceIntentionsFromConfigEntriesTxn(tx, ws, matchEntry.Name, matchEntry.GetEnterpriseMeta()) + return readSourceIntentionsFromConfigEntriesTxn(tx, ws, matchEntry.Name, matchEntry.GetEnterpriseMeta(), targetType) case structs.IntentionMatchDestination: return readDestinationIntentionsFromConfigEntriesTxn(tx, ws, matchEntry.Name, matchEntry.GetEnterpriseMeta()) default: @@ -239,7 +240,13 @@ func configIntentionMatchOneTxn( } } -func readSourceIntentionsFromConfigEntriesTxn(tx ReadTxn, ws memdb.WatchSet, serviceName string, entMeta *acl.EnterpriseMeta) (uint64, structs.Intentions, error) { +func readSourceIntentionsFromConfigEntriesTxn( + tx ReadTxn, + ws memdb.WatchSet, + serviceName string, + entMeta *acl.EnterpriseMeta, + targetType structs.IntentionTargetType, +) (uint64, structs.Intentions, error) { idx := maxIndexTxn(tx, tableConfigEntries) var ( @@ -249,9 +256,7 @@ func readSourceIntentionsFromConfigEntriesTxn(tx ReadTxn, ws memdb.WatchSet, ser names := getIntentionPrecedenceMatchServiceNames(serviceName, entMeta) for _, sn := range names { - results, err = readSourceIntentionsFromConfigEntriesForServiceTxn( - tx, ws, sn.Name, &sn.EnterpriseMeta, results, - ) + results, err = readSourceIntentionsFromConfigEntriesForServiceTxn(tx, ws, sn.Name, &sn.EnterpriseMeta, results, targetType) if err != nil { return 0, nil, err } @@ -263,7 +268,14 @@ func readSourceIntentionsFromConfigEntriesTxn(tx ReadTxn, ws memdb.WatchSet, ser return idx, results, nil } -func readSourceIntentionsFromConfigEntriesForServiceTxn(tx ReadTxn, ws memdb.WatchSet, serviceName string, entMeta *acl.EnterpriseMeta, results structs.Intentions) (structs.Intentions, error) { +func readSourceIntentionsFromConfigEntriesForServiceTxn( + tx ReadTxn, + ws memdb.WatchSet, + serviceName string, + entMeta *acl.EnterpriseMeta, + results structs.Intentions, + targetType structs.IntentionTargetType, +) (structs.Intentions, error) { sn := structs.NewServiceName(serviceName, entMeta) iter, err := tx.Get(tableConfigEntries, indexSource, sn) @@ -276,7 +288,23 @@ func readSourceIntentionsFromConfigEntriesForServiceTxn(tx ReadTxn, ws memdb.Wat entry := v.(*structs.ServiceIntentionsConfigEntry) for _, src := range entry.Sources { if src.SourceServiceName() == sn { - results = append(results, entry.ToIntention(src)) + entMeta := entry.DestinationServiceName().EnterpriseMeta + kind, err := GatewayServiceKind(tx, entry.DestinationServiceName().Name, &entMeta) + if err != nil { + return nil, err + } + switch targetType { + case structs.IntentionTargetService: + if kind == structs.GatewayServiceKindService || kind == structs.GatewayServiceKindUnknown { + results = append(results, entry.ToIntention(src)) + } + case structs.IntentionTargetDestination: + if kind == structs.GatewayServiceKindDestination { + results = append(results, entry.ToIntention(src)) + } + default: + return nil, fmt.Errorf("invalid target type") + } } } } @@ -298,7 +326,6 @@ func readDestinationIntentionsFromConfigEntriesTxn(tx ReadTxn, ws memdb.WatchSet results = append(results, entry.ToIntentions()...) } } - // Sort the results by precedence sort.Sort(structs.IntentionPrecedenceSorter(results)) diff --git a/agent/consul/state/config_entry_test.go b/agent/consul/state/config_entry_test.go index 5d2cdf9fd..b874f41af 100644 --- a/agent/consul/state/config_entry_test.go +++ b/agent/consul/state/config_entry_test.go @@ -355,6 +355,11 @@ func TestStore_ServiceDefaults_Kind_Destination(t *testing.T) { require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindDestination) + _, kindServices, err := s.ServiceNamesOfKind(ws, structs.ServiceKindDestination) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindDestination) + ws = memdb.NewWatchSet() _, _, err = s.GatewayServices(ws, "Gtwy1", nil) require.NoError(t, err) @@ -369,6 +374,10 @@ func TestStore_ServiceDefaults_Kind_Destination(t *testing.T) { require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindUnknown) + _, kindServices, err = s.ServiceNamesOfKind(ws, structs.ServiceKindDestination) + require.NoError(t, err) + require.Len(t, kindServices, 0) + } func TestStore_ServiceDefaults_Kind_NotDestination(t *testing.T) { @@ -475,6 +484,11 @@ func TestStore_Service_TerminatingGateway_Kind_Service_Destination(t *testing.T) require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindService) + _, kindServices, err := s.ServiceNamesOfKind(ws, structs.ServiceKindTypical) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindTypical) + require.NoError(t, s.EnsureConfigEntry(0, destination)) _, gatewayServices, err = s.GatewayServices(nil, "Gtwy1", nil) @@ -482,6 +496,11 @@ func TestStore_Service_TerminatingGateway_Kind_Service_Destination(t *testing.T) require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindService) + _, kindServices, err = s.ServiceNamesOfKind(ws, structs.ServiceKindTypical) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindTypical) + ws = memdb.NewWatchSet() _, _, err = s.GatewayServices(ws, "Gtwy1", nil) require.NoError(t, err) @@ -496,6 +515,11 @@ func TestStore_Service_TerminatingGateway_Kind_Service_Destination(t *testing.T) require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindDestination) + _, kindServices, err = s.ServiceNamesOfKind(ws, structs.ServiceKindDestination) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindDestination) + } func TestStore_Service_TerminatingGateway_Kind_Destination_Service(t *testing.T) { @@ -541,6 +565,11 @@ func TestStore_Service_TerminatingGateway_Kind_Destination_Service(t *testing.T) require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindDestination) + _, kindServices, err := s.ServiceNamesOfKind(ws, structs.ServiceKindDestination) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindDestination) + require.NoError(t, s.EnsureNode(0, &structs.Node{Node: "node1"})) require.NoError(t, s.EnsureService(0, "node1", service)) @@ -552,6 +581,11 @@ func TestStore_Service_TerminatingGateway_Kind_Destination_Service(t *testing.T) require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindService) + _, kindServices, err = s.ServiceNamesOfKind(ws, structs.ServiceKindTypical) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindTypical) + ws = memdb.NewWatchSet() _, _, err = s.GatewayServices(ws, "Gtwy1", nil) require.NoError(t, err) @@ -566,6 +600,11 @@ func TestStore_Service_TerminatingGateway_Kind_Destination_Service(t *testing.T) require.Len(t, gatewayServices, 1) require.Equal(t, gatewayServices[0].ServiceKind, structs.GatewayServiceKindDestination) + _, kindServices, err = s.ServiceNamesOfKind(ws, structs.ServiceKindDestination) + require.NoError(t, err) + require.Len(t, kindServices, 1) + require.Equal(t, kindServices[0].Kind, structs.ServiceKindDestination) + } func TestStore_Service_TerminatingGateway_Kind_Service(t *testing.T) { diff --git a/agent/consul/state/intention.go b/agent/consul/state/intention.go index 821288f3b..89f7304f6 100644 --- a/agent/consul/state/intention.go +++ b/agent/consul/state/intention.go @@ -600,6 +600,7 @@ func legacyIntentionGetTxn(tx ReadTxn, ws memdb.WatchSet, id string) (uint64, *s // Convert the interface{} if it is non-nil var result *structs.Intention + if intention != nil { result = intention.(*structs.Intention) } @@ -842,11 +843,12 @@ func (s *Store) IntentionMatchOne( ws memdb.WatchSet, entry structs.IntentionMatchEntry, matchType structs.IntentionMatchType, + destinationType structs.IntentionTargetType, ) (uint64, structs.Intentions, error) { tx := s.db.Txn(false) defer tx.Abort() - return compatIntentionMatchOneTxn(tx, ws, entry, matchType) + return compatIntentionMatchOneTxn(tx, ws, entry, matchType, destinationType) } func compatIntentionMatchOneTxn( @@ -854,6 +856,7 @@ func compatIntentionMatchOneTxn( ws memdb.WatchSet, entry structs.IntentionMatchEntry, matchType structs.IntentionMatchType, + destinationType structs.IntentionTargetType, ) (uint64, structs.Intentions, error) { usingConfigEntries, err := areIntentionsInConfigEntries(tx, ws) @@ -863,7 +866,7 @@ func compatIntentionMatchOneTxn( if !usingConfigEntries { return legacyIntentionMatchOneTxn(tx, ws, entry, matchType) } - return configIntentionMatchOneTxn(tx, ws, entry, matchType) + return configIntentionMatchOneTxn(tx, ws, entry, matchType, destinationType) } func legacyIntentionMatchOneTxn( @@ -949,8 +952,12 @@ type ServiceWithDecision struct { // IntentionTopology returns the upstreams or downstreams of a service. Upstreams and downstreams are inferred from // intentions. If intentions allow a connection from the target to some candidate service, the candidate service is considered // an upstream of the target. -func (s *Store) IntentionTopology(ws memdb.WatchSet, - target structs.ServiceName, downstreams bool, defaultDecision acl.EnforcementDecision) (uint64, structs.ServiceList, error) { +func (s *Store) IntentionTopology( + ws memdb.WatchSet, + target structs.ServiceName, + downstreams bool, + defaultDecision acl.EnforcementDecision, +) (uint64, structs.ServiceList, error) { tx := s.db.ReadTxn() defer tx.Abort() @@ -965,13 +972,17 @@ func (s *Store) IntentionTopology(ws memdb.WatchSet, resp := make(structs.ServiceList, 0) for _, svc := range services { - resp = append(resp, svc.Name) + resp = append(resp, structs.ServiceName{Name: svc.Name.Name, EnterpriseMeta: svc.Name.EnterpriseMeta}) } return idx, resp, nil } -func (s *Store) intentionTopologyTxn(tx ReadTxn, ws memdb.WatchSet, - target structs.ServiceName, downstreams bool, defaultDecision acl.EnforcementDecision) (uint64, []ServiceWithDecision, error) { +func (s *Store) intentionTopologyTxn( + tx ReadTxn, ws memdb.WatchSet, + target structs.ServiceName, + downstreams bool, + defaultDecision acl.EnforcementDecision, +) (uint64, []ServiceWithDecision, error) { var maxIdx uint64 @@ -987,7 +998,7 @@ func (s *Store) intentionTopologyTxn(tx ReadTxn, ws memdb.WatchSet, Partition: target.PartitionOrDefault(), Name: target.Name, } - index, intentions, err := compatIntentionMatchOneTxn(tx, ws, entry, intentionMatchType) + index, intentions, err := compatIntentionMatchOneTxn(tx, ws, entry, intentionMatchType, structs.IntentionTargetService) if err != nil { return 0, nil, fmt.Errorf("failed to query intentions for %s", target.String()) } @@ -1017,6 +1028,16 @@ func (s *Store) intentionTopologyTxn(tx ReadTxn, ws memdb.WatchSet, maxIdx = index } services = append(services, ingress...) + } else { + // destinations can only ever be upstream, since they are only allowed as intention destination. + index, destinations, err := serviceNamesOfKindTxn(tx, ws, structs.ServiceKindDestination, *wildcardMeta) + if err != nil { + return index, nil, fmt.Errorf("failed to list destination names: %v", err) + } + if index > maxIdx { + maxIdx = index + } + services = append(services, destinations...) } // When checking authorization to upstreams, the match type for the decision is `destination` because we are deciding @@ -1056,7 +1077,7 @@ func (s *Store) intentionTopologyTxn(tx ReadTxn, ws memdb.WatchSet, } result = append(result, ServiceWithDecision{ - Name: candidate, + Name: structs.ServiceName{Name: candidate.Name, EnterpriseMeta: candidate.EnterpriseMeta}, Decision: decision, }) } diff --git a/agent/consul/state/intention_test.go b/agent/consul/state/intention_test.go index fde26d1d9..4a369d166 100644 --- a/agent/consul/state/intention_test.go +++ b/agent/consul/state/intention_test.go @@ -1533,7 +1533,7 @@ func TestStore_IntentionMatchOne_table(t *testing.T) { Namespace: "default", Name: query, } - _, matches, err := s.IntentionMatchOne(nil, entry, typ) + _, matches, err := s.IntentionMatchOne(nil, entry, typ, structs.IntentionTargetService) require.NoError(t, err) // Verify matches @@ -1873,7 +1873,7 @@ func TestStore_IntentionDecision(t *testing.T) { Partition: acl.DefaultPartitionName, Name: tc.src, } - _, intentions, err := s.IntentionMatchOne(nil, entry, structs.IntentionMatchSource) + _, intentions, err := s.IntentionMatchOne(nil, entry, structs.IntentionMatchSource, structs.IntentionTargetService) if err != nil { require.NoError(t, err) } diff --git a/agent/structs/intention.go b/agent/structs/intention.go index feefc5672..ee1f89c0a 100644 --- a/agent/structs/intention.go +++ b/agent/structs/intention.go @@ -494,6 +494,15 @@ const ( IntentionSourceConsul IntentionSourceType = "consul" ) +type IntentionTargetType string + +const ( + // IntentionTargetService is a service within the Consul catalog. + IntentionTargetService IntentionTargetType = "service" + // IntentionTargetDestination is a destination defined through a service-default config entry. + IntentionTargetDestination IntentionTargetType = "destination" +) + // Intentions is a list of intentions. type Intentions []*Intention diff --git a/agent/structs/structs.go b/agent/structs/structs.go index 94c425610..a7b4dcd73 100644 --- a/agent/structs/structs.go +++ b/agent/structs/structs.go @@ -1178,6 +1178,11 @@ const ( // This service allows external traffic to enter the mesh based on // centralized configuration. ServiceKindIngressGateway ServiceKind = "ingress-gateway" + + // ServiceKindDestination is a Destination for the Connect feature. + // This service allows external traffic to exit the mesh through a terminating gateway + //based on centralized configuration. + ServiceKindDestination ServiceKind = "destination" ) // Type to hold a address and port of a service From 3deaf767f2141bcbb9472fd7c210353d7fcb93bb Mon Sep 17 00:00:00 2001 From: Dhia Ayachi Date: Tue, 7 Jun 2022 15:55:02 -0400 Subject: [PATCH 010/247] Egress gtw/intention rpc endpoint (#13354) * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough * remove duplicate tests * first draft of destinations intention in connect proxy * remove ServiceDestinationList * fix failing tests * fix agent/consul failing tests * change to filter intentions in the state store instead of adding a field. * fix failing tests * fix comment * fix comments * store service kind destination and add relevant tests * changes based on review * filter on destinations when querying source match * change state store API to get an IntentionTarget parameter * add intentions tests * add destination upstream endpoint * fix failing test * fix failing test and a bug with wildcard intentions * fix failing test * Apply suggestions from code review Co-authored-by: alex <8968914+acpana@users.noreply.github.com> * add missing test and clarify doc * fix style * gofmt intention.go * fix merge introduced issue Co-authored-by: Dan Stough Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: github-team-consul-core --- .../intention_upstreams_destination.go | 52 +++++ .../intention_upstreams_destination_test.go | 52 +++++ agent/consul/helper_test.go | 48 ++++- agent/consul/internal_endpoint.go | 27 ++- agent/consul/internal_endpoint_test.go | 44 ++++ agent/consul/state/catalog.go | 4 +- agent/consul/state/config_entry_intention.go | 3 +- agent/consul/state/intention.go | 24 +-- agent/consul/state/intention_test.go | 200 +++++++++++++++++- 9 files changed, 429 insertions(+), 25 deletions(-) create mode 100644 agent/cache-types/intention_upstreams_destination.go create mode 100644 agent/cache-types/intention_upstreams_destination_test.go diff --git a/agent/cache-types/intention_upstreams_destination.go b/agent/cache-types/intention_upstreams_destination.go new file mode 100644 index 000000000..ae1012c35 --- /dev/null +++ b/agent/cache-types/intention_upstreams_destination.go @@ -0,0 +1,52 @@ +package cachetype + +import ( + "fmt" + + "github.com/hashicorp/consul/agent/cache" + "github.com/hashicorp/consul/agent/structs" +) + +// IntentionUpstreamsDestinationName Recommended name for registration. +const IntentionUpstreamsDestinationName = "intention-upstreams-destination" + +// IntentionUpstreamsDestination supports fetching upstreams for a given gateway name. +type IntentionUpstreamsDestination struct { + RegisterOptionsBlockingRefresh + RPC RPC +} + +func (i *IntentionUpstreamsDestination) Fetch(opts cache.FetchOptions, req cache.Request) (cache.FetchResult, error) { + var result cache.FetchResult + + // The request should be a ServiceSpecificRequest. + reqReal, ok := req.(*structs.ServiceSpecificRequest) + if !ok { + return result, fmt.Errorf( + "Internal cache failure: request wrong type: %T", req) + } + + // Lightweight copy this object so that manipulating QueryOptions doesn't race. + dup := *reqReal + reqReal = &dup + + // Set the minimum query index to our current index so we block + reqReal.QueryOptions.MinQueryIndex = opts.MinIndex + reqReal.QueryOptions.MaxQueryTime = opts.Timeout + + // Always allow stale - there's no point in hitting leader if the request is + // going to be served from cache and end up arbitrarily stale anyway. This + // allows cached service-discover to automatically read scale across all + // servers too. + reqReal.AllowStale = true + + // Fetch + var reply structs.IndexedServiceList + if err := i.RPC.RPC("Internal.IntentionUpstreamsDestination", reqReal, &reply); err != nil { + return result, err + } + + result.Value = &reply + result.Index = reply.QueryMeta.Index + return result, nil +} diff --git a/agent/cache-types/intention_upstreams_destination_test.go b/agent/cache-types/intention_upstreams_destination_test.go new file mode 100644 index 000000000..7aa2d02ef --- /dev/null +++ b/agent/cache-types/intention_upstreams_destination_test.go @@ -0,0 +1,52 @@ +package cachetype + +import ( + "testing" + "time" + + "github.com/hashicorp/consul/agent/cache" + "github.com/hashicorp/consul/agent/structs" + "github.com/stretchr/testify/mock" + "github.com/stretchr/testify/require" +) + +func TestIntentionUpstreamsDestination(t *testing.T) { + rpc := TestRPC(t) + typ := &IntentionUpstreamsDestination{RPC: rpc} + + // Expect the proper RPC call. This also sets the expected value + // since that is return-by-pointer in the arguments. + var resp *structs.IndexedServiceList + rpc.On("RPC", "Internal.IntentionUpstreamsDestination", mock.Anything, mock.Anything).Return(nil). + Run(func(args mock.Arguments) { + req := args.Get(1).(*structs.ServiceSpecificRequest) + require.Equal(t, uint64(24), req.QueryOptions.MinQueryIndex) + require.Equal(t, 1*time.Second, req.QueryOptions.MaxQueryTime) + require.True(t, req.AllowStale) + require.Equal(t, "foo", req.ServiceName) + + services := structs.ServiceList{ + {Name: "foo"}, + } + reply := args.Get(2).(*structs.IndexedServiceList) + reply.Services = services + reply.QueryMeta.Index = 48 + resp = reply + }) + + // Fetch + resultA, err := typ.Fetch(cache.FetchOptions{ + MinIndex: 24, + Timeout: 1 * time.Second, + }, &structs.ServiceSpecificRequest{ + Datacenter: "dc1", + ServiceName: "foo", + }) + require.NoError(t, err) + require.Equal(t, cache.FetchResult{ + Value: resp, + Index: 48, + }, resultA) + + rpc.AssertExpectations(t) +} diff --git a/agent/consul/helper_test.go b/agent/consul/helper_test.go index 807bb8be2..75048b913 100644 --- a/agent/consul/helper_test.go +++ b/agent/consul/helper_test.go @@ -1307,7 +1307,7 @@ func registerIntentionUpstreamEntries(t *testing.T, codec rpc.ClientCodec, token } registerTestCatalogEntriesMap(t, codec, registrations) - // Add intentions: deny all and web -> api + // Add intentions: deny all and web -> api and web -> api.example.com entries := []structs.ConfigEntryRequest{ { Datacenter: "dc1", @@ -1323,6 +1323,20 @@ func registerIntentionUpstreamEntries(t *testing.T, codec rpc.ClientCodec, token }, WriteRequest: structs.WriteRequest{Token: token}, }, + { + Datacenter: "dc1", + Entry: &structs.ServiceIntentionsConfigEntry{ + Kind: structs.ServiceIntentions, + Name: "api.example.com", + Sources: []*structs.SourceIntention{ + { + Name: "web", + Action: structs.IntentionActionAllow, + }, + }, + }, + WriteRequest: structs.WriteRequest{Token: token}, + }, { Datacenter: "dc1", Entry: &structs.ServiceIntentionsConfigEntry{ @@ -1342,4 +1356,36 @@ func registerIntentionUpstreamEntries(t *testing.T, codec rpc.ClientCodec, token var out bool require.NoError(t, msgpackrpc.CallWithCodec(codec, "ConfigEntry.Apply", &req, &out)) } + + // Add destinations + dests := []structs.ConfigEntryRequest{ + { + Datacenter: "dc1", + Entry: &structs.ServiceConfigEntry{ + Kind: structs.ServiceDefaults, + Name: "api.example.com", + Destination: &structs.DestinationConfig{ + Address: "api.example.com", + Port: 443, + }, + }, + WriteRequest: structs.WriteRequest{Token: token}, + }, + { + Datacenter: "dc1", + Entry: &structs.ServiceConfigEntry{ + Kind: structs.ServiceDefaults, + Name: "kafka.store.com", + Destination: &structs.DestinationConfig{ + Address: "172.168.2.1", + Port: 9003, + }, + }, + WriteRequest: structs.WriteRequest{Token: token}, + }, + } + for _, req := range dests { + var out bool + require.NoError(t, msgpackrpc.CallWithCodec(codec, "ConfigEntry.Apply", &req, &out)) + } } diff --git a/agent/consul/internal_endpoint.go b/agent/consul/internal_endpoint.go index 113abd2dd..f1fda470f 100644 --- a/agent/consul/internal_endpoint.go +++ b/agent/consul/internal_endpoint.go @@ -224,6 +224,27 @@ func (m *Internal) IntentionUpstreams(args *structs.ServiceSpecificRequest, repl if done, err := m.srv.ForwardRPC("Internal.IntentionUpstreams", args, reply); done { return err } + return m.internalUpstreams(args, reply, structs.IntentionTargetService) +} + +// IntentionUpstreamsDestination returns the upstreams of a service. Upstreams are inferred from intentions. +// If intentions allow a connection from the target to some candidate destination, the candidate destination is considered +// an upstream of the target.this is performs the same logic as IntentionUpstreams endpoint but for destination upstreams only. +func (m *Internal) IntentionUpstreamsDestination(args *structs.ServiceSpecificRequest, reply *structs.IndexedServiceList) error { + // Exit early if Connect hasn't been enabled. + if !m.srv.config.ConnectEnabled { + return ErrConnectNotEnabled + } + if args.ServiceName == "" { + return fmt.Errorf("Must provide a service name") + } + if done, err := m.srv.ForwardRPC("Internal.IntentionUpstreamsDestination", args, reply); done { + return err + } + return m.internalUpstreams(args, reply, structs.IntentionTargetDestination) +} + +func (m *Internal) internalUpstreams(args *structs.ServiceSpecificRequest, reply *structs.IndexedServiceList, intentionTarget structs.IntentionTargetType) error { authz, err := m.srv.ResolveTokenAndDefaultMeta(args.Token, &args.EnterpriseMeta, nil) if err != nil { @@ -244,7 +265,7 @@ func (m *Internal) IntentionUpstreams(args *structs.ServiceSpecificRequest, repl defaultDecision := authz.IntentionDefaultAllow(nil) sn := structs.NewServiceName(args.ServiceName, &args.EnterpriseMeta) - index, services, err := state.IntentionTopology(ws, sn, false, defaultDecision) + index, services, err := state.IntentionTopology(ws, sn, false, defaultDecision, intentionTarget) if err != nil { return err } @@ -272,7 +293,7 @@ func (m *Internal) IntentionUpstreams(args *structs.ServiceSpecificRequest, repl }) } -// GatewayServiceNodes returns all the nodes for services associated with a gateway along with their gateway config +// GatewayServiceDump returns all the nodes for services associated with a gateway along with their gateway config func (m *Internal) GatewayServiceDump(args *structs.ServiceSpecificRequest, reply *structs.IndexedServiceDump) error { if done, err := m.srv.ForwardRPC("Internal.GatewayServiceDump", args, reply); done { return err @@ -350,7 +371,7 @@ func (m *Internal) GatewayServiceDump(args *structs.ServiceSpecificRequest, repl return err } -// Match returns the set of intentions that match the given source/destination. +// GatewayIntentions Match returns the set of intentions that match the given source/destination. func (m *Internal) GatewayIntentions(args *structs.IntentionQueryRequest, reply *structs.IndexedIntentions) error { // Forward if necessary if done, err := m.srv.ForwardRPC("Internal.GatewayIntentions", args, reply); done { diff --git a/agent/consul/internal_endpoint_test.go b/agent/consul/internal_endpoint_test.go index 9354b4f66..3737f3a08 100644 --- a/agent/consul/internal_endpoint_test.go +++ b/agent/consul/internal_endpoint_test.go @@ -2323,6 +2323,50 @@ func TestInternal_IntentionUpstreams(t *testing.T) { }) } +func TestInternal_IntentionUpstreamsDestination(t *testing.T) { + if testing.Short() { + t.Skip("too slow for testing.Short") + } + + t.Parallel() + dir1, s1 := testServer(t) + defer os.RemoveAll(dir1) + defer s1.Shutdown() + + testrpc.WaitForLeader(t, s1.RPC, "dc1") + + codec := rpcClient(t, s1) + defer codec.Close() + + // Services: + // api and api-proxy on node foo + // web and web-proxy on node foo + // + // Intentions + // * -> * (deny) intention + // web -> api (allow) + registerIntentionUpstreamEntries(t, codec, "") + + t.Run("api.example.com", func(t *testing.T) { + retry.Run(t, func(r *retry.R) { + args := structs.ServiceSpecificRequest{ + Datacenter: "dc1", + ServiceName: "web", + } + var out structs.IndexedServiceList + require.NoError(r, msgpackrpc.CallWithCodec(codec, "Internal.IntentionUpstreamsDestination", &args, &out)) + + // foo/api + require.Len(r, out.Services, 1) + + expectUp := structs.ServiceList{ + structs.NewServiceName("api.example.com", structs.DefaultEnterpriseMetaInDefaultPartition()), + } + require.Equal(r, expectUp, out.Services) + }) + }) +} + func TestInternal_IntentionUpstreams_BlockOnNoChange(t *testing.T) { if testing.Short() { t.Skip("too slow for testing.Short") diff --git a/agent/consul/state/catalog.go b/agent/consul/state/catalog.go index ee1623956..d76f87bf6 100644 --- a/agent/consul/state/catalog.go +++ b/agent/consul/state/catalog.go @@ -3975,7 +3975,7 @@ func (s *Store) ServiceTopology( // Only transparent proxies / connect native services have upstreams from intentions if hasTransparent || connectNative { - idx, intentionUpstreams, err := s.intentionTopologyTxn(tx, ws, sn, false, defaultAllow) + idx, intentionUpstreams, err := s.intentionTopologyTxn(tx, ws, sn, false, defaultAllow, structs.IntentionTargetService) if err != nil { return 0, nil, err } @@ -4092,7 +4092,7 @@ func (s *Store) ServiceTopology( downstreamSources[dn.String()] = structs.TopologySourceRegistration } - idx, intentionDownstreams, err := s.intentionTopologyTxn(tx, ws, sn, true, defaultAllow) + idx, intentionDownstreams, err := s.intentionTopologyTxn(tx, ws, sn, true, defaultAllow, structs.IntentionTargetService) if err != nil { return 0, nil, err } diff --git a/agent/consul/state/config_entry_intention.go b/agent/consul/state/config_entry_intention.go index bd539489a..ebbc9e7b7 100644 --- a/agent/consul/state/config_entry_intention.go +++ b/agent/consul/state/config_entry_intention.go @@ -299,7 +299,8 @@ func readSourceIntentionsFromConfigEntriesForServiceTxn( results = append(results, entry.ToIntention(src)) } case structs.IntentionTargetDestination: - if kind == structs.GatewayServiceKindDestination { + // wildcard is needed here to be able to consider destinations in the wildcard intentions + if kind == structs.GatewayServiceKindDestination || entry.HasWildcardDestination() { results = append(results, entry.ToIntention(src)) } default: diff --git a/agent/consul/state/intention.go b/agent/consul/state/intention.go index 89f7304f6..cff5ed353 100644 --- a/agent/consul/state/intention.go +++ b/agent/consul/state/intention.go @@ -957,11 +957,12 @@ func (s *Store) IntentionTopology( target structs.ServiceName, downstreams bool, defaultDecision acl.EnforcementDecision, + intentionTarget structs.IntentionTargetType, ) (uint64, structs.ServiceList, error) { tx := s.db.ReadTxn() defer tx.Abort() - idx, services, err := s.intentionTopologyTxn(tx, ws, target, downstreams, defaultDecision) + idx, services, err := s.intentionTopologyTxn(tx, ws, target, downstreams, defaultDecision, intentionTarget) if err != nil { requested := "upstreams" if downstreams { @@ -982,6 +983,7 @@ func (s *Store) intentionTopologyTxn( target structs.ServiceName, downstreams bool, defaultDecision acl.EnforcementDecision, + intentionTarget structs.IntentionTargetType, ) (uint64, []ServiceWithDecision, error) { var maxIdx uint64 @@ -998,7 +1000,7 @@ func (s *Store) intentionTopologyTxn( Partition: target.PartitionOrDefault(), Name: target.Name, } - index, intentions, err := compatIntentionMatchOneTxn(tx, ws, entry, intentionMatchType, structs.IntentionTargetService) + index, intentions, err := compatIntentionMatchOneTxn(tx, ws, entry, intentionMatchType, intentionTarget) if err != nil { return 0, nil, fmt.Errorf("failed to query intentions for %s", target.String()) } @@ -1010,7 +1012,13 @@ func (s *Store) intentionTopologyTxn( // Ideally those should be excluded as well, since they can't be upstreams/downstreams without a proxy. // Maybe narrow serviceNamesOfKindTxn to services represented by proxies? (ingress, sidecar- wildcardMeta := structs.WildcardEnterpriseMetaInPartition(structs.WildcardSpecifier) - index, services, err := serviceNamesOfKindTxn(tx, ws, structs.ServiceKindTypical, *wildcardMeta) + var services []*KindServiceName + if intentionTarget == structs.IntentionTargetService { + index, services, err = serviceNamesOfKindTxn(tx, ws, structs.ServiceKindTypical, *wildcardMeta) + } else { + // destinations can only ever be upstream, since they are only allowed as intention destination. + index, services, err = serviceNamesOfKindTxn(tx, ws, structs.ServiceKindDestination, *wildcardMeta) + } if err != nil { return index, nil, fmt.Errorf("failed to list ingress service names: %v", err) } @@ -1028,16 +1036,6 @@ func (s *Store) intentionTopologyTxn( maxIdx = index } services = append(services, ingress...) - } else { - // destinations can only ever be upstream, since they are only allowed as intention destination. - index, destinations, err := serviceNamesOfKindTxn(tx, ws, structs.ServiceKindDestination, *wildcardMeta) - if err != nil { - return index, nil, fmt.Errorf("failed to list destination names: %v", err) - } - if index > maxIdx { - maxIdx = index - } - services = append(services, destinations...) } // When checking authorization to upstreams, the match type for the decision is `destination` because we are deciding diff --git a/agent/consul/state/intention_test.go b/agent/consul/state/intention_test.go index 4a369d166..18ba63914 100644 --- a/agent/consul/state/intention_test.go +++ b/agent/consul/state/intention_test.go @@ -2185,7 +2185,197 @@ func TestStore_IntentionTopology(t *testing.T) { idx++ } - idx, got, err := s.IntentionTopology(nil, tt.target, tt.downstreams, tt.defaultDecision) + idx, got, err := s.IntentionTopology(nil, tt.target, tt.downstreams, tt.defaultDecision, structs.IntentionTargetService) + require.NoError(t, err) + require.Equal(t, tt.expect.idx, idx) + + // ServiceList is from a map, so it is not deterministically sorted + sort.Slice(got, func(i, j int) bool { + return got[i].String() < got[j].String() + }) + require.Equal(t, tt.expect.services, got) + }) + } +} + +func TestStore_IntentionTopology_Destination(t *testing.T) { + node := structs.Node{ + Node: "foo", + Address: "127.0.0.1", + } + + services := []structs.NodeService{ + { + ID: structs.ConsulServiceID, + Service: structs.ConsulServiceName, + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + { + ID: "web-1", + Service: "web", + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + { + ID: "mysql-1", + Service: "mysql", + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + } + destinations := []structs.ServiceConfigEntry{ + { + Name: "api.test.com", + Destination: &structs.DestinationConfig{}, + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + { + Name: "kafka.store.org", + Destination: &structs.DestinationConfig{}, + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + } + + type expect struct { + idx uint64 + services structs.ServiceList + } + tests := []struct { + name string + defaultDecision acl.EnforcementDecision + intentions []structs.ServiceIntentionsConfigEntry + target structs.ServiceName + downstreams bool + expect expect + }{ + { + name: "(upstream) acl allow all but intentions deny one, destination target", + defaultDecision: acl.Allow, + intentions: []structs.ServiceIntentionsConfigEntry{ + { + Kind: structs.ServiceIntentions, + Name: "api.test.com", + Sources: []*structs.SourceIntention{ + { + Name: "web", + Action: structs.IntentionActionDeny, + }, + }, + }, + }, + target: structs.NewServiceName("web", nil), + downstreams: false, + expect: expect{ + idx: 7, + services: structs.ServiceList{ + { + Name: "kafka.store.org", + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + }, + }, + }, + { + name: "(upstream) acl deny all intentions allow one, destination target", + defaultDecision: acl.Deny, + intentions: []structs.ServiceIntentionsConfigEntry{ + { + Kind: structs.ServiceIntentions, + Name: "kafka.store.org", + Sources: []*structs.SourceIntention{ + { + Name: "web", + Action: structs.IntentionActionAllow, + }, + }, + }, + }, + target: structs.NewServiceName("web", nil), + downstreams: false, + expect: expect{ + idx: 7, + services: structs.ServiceList{ + { + Name: "kafka.store.org", + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + }, + }, + }, + { + name: "(upstream) acl deny all check only destinations show, service target", + defaultDecision: acl.Deny, + intentions: []structs.ServiceIntentionsConfigEntry{ + { + Kind: structs.ServiceIntentions, + Name: "api", + Sources: []*structs.SourceIntention{ + { + Name: "web", + Action: structs.IntentionActionAllow, + }, + }, + }, + }, + target: structs.NewServiceName("web", nil), + downstreams: false, + expect: expect{ + idx: 7, + services: structs.ServiceList{}, + }, + }, + { + name: "(upstream) acl allow all check only destinations show, service target", + defaultDecision: acl.Allow, + intentions: []structs.ServiceIntentionsConfigEntry{ + { + Kind: structs.ServiceIntentions, + Name: "api", + Sources: []*structs.SourceIntention{ + { + Name: "web", + Action: structs.IntentionActionAllow, + }, + }, + }, + }, + target: structs.NewServiceName("web", nil), + downstreams: false, + expect: expect{ + idx: 7, + services: structs.ServiceList{ + { + Name: "api.test.com", + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + { + Name: "kafka.store.org", + EnterpriseMeta: *structs.DefaultEnterpriseMetaInDefaultPartition(), + }, + }, + }, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s := testConfigStateStore(t) + + var idx uint64 = 1 + require.NoError(t, s.EnsureNode(idx, &node)) + idx++ + + for _, svc := range services { + require.NoError(t, s.EnsureService(idx, "foo", &svc)) + idx++ + } + for _, d := range destinations { + require.NoError(t, s.EnsureConfigEntry(idx, &d)) + idx++ + } + for _, ixn := range tt.intentions { + require.NoError(t, s.EnsureConfigEntry(idx, &ixn)) + idx++ + } + + idx, got, err := s.IntentionTopology(nil, tt.target, tt.downstreams, tt.defaultDecision, structs.IntentionTargetDestination) require.NoError(t, err) require.Equal(t, tt.expect.idx, idx) @@ -2211,7 +2401,7 @@ func TestStore_IntentionTopology_Watches(t *testing.T) { target := structs.NewServiceName("web", structs.DefaultEnterpriseMetaInDefaultPartition()) ws := memdb.NewWatchSet() - index, got, err := s.IntentionTopology(ws, target, false, acl.Deny) + index, got, err := s.IntentionTopology(ws, target, false, acl.Deny, structs.IntentionTargetService) require.NoError(t, err) require.Equal(t, uint64(0), index) require.Empty(t, got) @@ -2233,7 +2423,7 @@ func TestStore_IntentionTopology_Watches(t *testing.T) { // Reset the WatchSet ws = memdb.NewWatchSet() - index, got, err = s.IntentionTopology(ws, target, false, acl.Deny) + index, got, err = s.IntentionTopology(ws, target, false, acl.Deny, structs.IntentionTargetService) require.NoError(t, err) require.Equal(t, uint64(2), index) require.Empty(t, got) @@ -2255,7 +2445,7 @@ func TestStore_IntentionTopology_Watches(t *testing.T) { // require.False(t, watchFired(ws)) // Result should not have changed - index, got, err = s.IntentionTopology(ws, target, false, acl.Deny) + index, got, err = s.IntentionTopology(ws, target, false, acl.Deny, structs.IntentionTargetService) require.NoError(t, err) require.Equal(t, uint64(3), index) require.Empty(t, got) @@ -2270,7 +2460,7 @@ func TestStore_IntentionTopology_Watches(t *testing.T) { require.True(t, watchFired(ws)) // Reset the WatchSet - index, got, err = s.IntentionTopology(nil, target, false, acl.Deny) + index, got, err = s.IntentionTopology(nil, target, false, acl.Deny, structs.IntentionTargetService) require.NoError(t, err) require.Equal(t, uint64(4), index) From 00cdae18d513ec16b1f1cd0d47d7884faf5aa5d6 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Tue, 7 Jun 2022 16:17:45 -0400 Subject: [PATCH 011/247] Update website/content/docs/api-gateway/consul-api-gateway-install.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/api-gateway/consul-api-gateway-install.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 6aed5da1d..00938dff2 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -268,7 +268,7 @@ Refer to the [Kubernetes Gateway API documentation](https://gateway-api.sigs.k8s #### Scaling -A logical gateway object can be scaled to multiple instances within the bounds set in its associated GatewayClassConfig by using the [`kubectl scale`](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment) command. +You can scale a logical gateway object to multiple instances with the [`kubectl scale`](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment) command. The object scales according to the bounds set in GatewayClassConfig. ``` $ kubectl get deployment --selector api-gateway.consul.hashicorp.com/name=example-gateway From 24ca62ef1c0bdc7795904aefb98e7772b7a559d9 Mon Sep 17 00:00:00 2001 From: Mike Morris Date: Tue, 7 Jun 2022 16:24:35 -0400 Subject: [PATCH 012/247] Update website/content/docs/api-gateway/consul-api-gateway-install.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/api-gateway/consul-api-gateway-install.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/api-gateway/consul-api-gateway-install.mdx b/website/content/docs/api-gateway/consul-api-gateway-install.mdx index 00938dff2..74fd062b0 100644 --- a/website/content/docs/api-gateway/consul-api-gateway-install.mdx +++ b/website/content/docs/api-gateway/consul-api-gateway-install.mdx @@ -168,7 +168,7 @@ The following table describes the allowed parameters for the `spec` array: | `consul.ports.http` | Specifies the port for Consul's HTTP server. | Integer | `8500` | | `consul.scheme` | Specifies the scheme to use for connecting to Consul. The supported values are `"http"` and `"https"`. | String | `"http"` | | `copyAnnotations.service` | List of annotations to copy to the gateway service. | Array | `["external-dns.alpha.kubernetes.io/hostname"]` | -| `deployment.defaultInstances` | Specifies the number of instances per gateway that should be deployed by default. | Integer | 1 | +| `deployment.defaultInstances` | Specifies the number of instances to deploy by default for each gateway. | Integer | 1 | | `deployment.maxInstances` | Specifies the maximum allowed number of instances per gateway. | Integer | 8 | | `deployment.minInstances` | Specifies the minimum allowed number of instances per gateway. | Integer | 1 | | `image.consulAPIGateway` | The image to use for consul-api-gateway. View available image tags on [DockerHub](https://hub.docker.com/r/hashicorp/consul-api-gateway/tags). | String | `"hashicorp/consul-api-gateway:RELEASE_VERSION"` | From bf647bc9d2ff03ac2f182efd50d95d7c1621cde2 Mon Sep 17 00:00:00 2001 From: "R.B. Boyer" <4903+rboyer@users.noreply.github.com> Date: Tue, 7 Jun 2022 16:29:09 -0500 Subject: [PATCH 013/247] peering: avoid a race between peering establishment and termination (#13389) --- agent/consul/client_test.go | 2 +- agent/consul/leader_peering.go | 37 ++++++++++++++++++++++++++++++++-- proto/pbpeering/peering.go | 4 ++++ 3 files changed, 40 insertions(+), 3 deletions(-) diff --git a/agent/consul/client_test.go b/agent/consul/client_test.go index d8f0fbd4d..5c35e3f33 100644 --- a/agent/consul/client_test.go +++ b/agent/consul/client_test.go @@ -510,7 +510,7 @@ func newDefaultDeps(t *testing.T, c *Config) Deps { logger := hclog.NewInterceptLogger(&hclog.LoggerOptions{ Name: c.NodeName, - Level: hclog.Trace, + Level: testutil.TestLogLevel, Output: testutil.NewLogBuffer(t), }) diff --git a/agent/consul/leader_peering.go b/agent/consul/leader_peering.go index c0f69b168..41736e3d0 100644 --- a/agent/consul/leader_peering.go +++ b/agent/consul/leader_peering.go @@ -8,7 +8,6 @@ import ( "fmt" "net" - "github.com/hashicorp/consul/agent/rpc/peering" "github.com/hashicorp/go-hclog" "github.com/hashicorp/go-memdb" "github.com/hashicorp/go-multierror" @@ -17,6 +16,7 @@ import ( "google.golang.org/grpc/credentials" "github.com/hashicorp/consul/agent/pool" + "github.com/hashicorp/consul/agent/rpc/peering" "github.com/hashicorp/consul/agent/structs" "github.com/hashicorp/consul/proto/pbpeering" ) @@ -50,6 +50,39 @@ func (s *Server) stopPeeringStreamSync() { // syncPeeringsAndBlock is a long-running goroutine that is responsible for watching // changes to peerings in the state store and managing streams to those peers. func (s *Server) syncPeeringsAndBlock(ctx context.Context, logger hclog.Logger, cancelFns map[string]context.CancelFunc) error { + // We have to be careful not to introduce a data race here. We want to + // compare the current known peerings in the state store with known + // connected streams to know when we should TERMINATE stray peerings. + // + // If you read the current peerings from the state store, then read the + // current established streams you could lose the data race and have the + // sequence of events be: + // + // 1. list peerings [A,B,C] + // 2. persist new peering [D] + // 3. accept new stream for [D] + // 4. list streams [A,B,C,D] + // 5. terminate [D] + // + // Which is wrong. If we instead ensure that (4) happens before (1), given + // that you can't get an established stream without first passing a "does + // this peering exist in the state store?" inquiry then this happens: + // + // 1. list streams [A,B,C] + // 2. list peerings [A,B,C] + // 3. persist new peering [D] + // 4. accept new stream for [D] + // 5. terminate [] + // + // Or even this is fine: + // + // 1. list streams [A,B,C] + // 2. persist new peering [D] + // 3. accept new stream for [D] + // 4. list peerings [A,B,C,D] + // 5. terminate [] + connectedStreams := s.peeringService.ConnectedStreams() + state := s.fsm.State() // Pull the state store contents and set up to block for changes. @@ -121,7 +154,7 @@ func (s *Server) syncPeeringsAndBlock(ctx context.Context, logger hclog.Logger, // Clean up active streams of peerings that were deleted from the state store. // TODO(peering): This is going to trigger shutting down peerings we generated a token for. Is that OK? - for stream, doneCh := range s.peeringService.ConnectedStreams() { + for stream, doneCh := range connectedStreams { if _, ok := stored[stream]; ok { // Active stream is in the state store, nothing to do. continue diff --git a/proto/pbpeering/peering.go b/proto/pbpeering/peering.go index 4c4265021..e52affe4a 100644 --- a/proto/pbpeering/peering.go +++ b/proto/pbpeering/peering.go @@ -93,6 +93,10 @@ func (x ReplicationMessage_Response_Operation) GoString() string { return x.String() } +func (x PeeringState) GoString() string { + return x.String() +} + func (r *TrustBundleReadRequest) CacheInfo() cache.RequestInfo { info := cache.RequestInfo{ // TODO(peering): Revisit whether this is the token to use once request types accept a token. From 744265f028e7b3c593fabf4503a87f6a0046bea9 Mon Sep 17 00:00:00 2001 From: Daniel Kimsey Date: Tue, 10 May 2022 09:59:03 -0500 Subject: [PATCH 014/247] Update go-grpc/grpc to resolve conection memory leak Reported in #12288 The initial test reported was ported and accurately reproduced the issue. However, since it is a test of an upstream library's internal behavior it won't be codified in our test suite. Refer to the ticket/PR for details on how to demonstrate the behavior. --- .changelog/13051.txt | 3 +++ go.mod | 2 +- go.sum | 4 +++- 3 files changed, 7 insertions(+), 2 deletions(-) create mode 100644 .changelog/13051.txt diff --git a/.changelog/13051.txt b/.changelog/13051.txt new file mode 100644 index 000000000..61a7db951 --- /dev/null +++ b/.changelog/13051.txt @@ -0,0 +1,3 @@ +```release-note:bug +deps: Update go-grpc/grpc, resolving connection memory leak +``` diff --git a/go.mod b/go.mod index 96f2f8478..0f71dff66 100644 --- a/go.mod +++ b/go.mod @@ -80,7 +80,7 @@ require ( golang.org/x/sys v0.0.0-20220412211240-33da011f77ad golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e google.golang.org/genproto v0.0.0-20200623002339-fbb79eadd5eb - google.golang.org/grpc v1.36.0 + google.golang.org/grpc v1.37.1 google.golang.org/protobuf v1.27.1 gopkg.in/square/go-jose.v2 v2.5.1 gotest.tools/v3 v3.0.3 diff --git a/go.sum b/go.sum index 7467ff229..722d90784 100644 --- a/go.sum +++ b/go.sum @@ -165,6 +165,7 @@ github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymF github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= +github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= github.com/envoyproxy/go-control-plane v0.10.1 h1:cgDRLG7bs59Zd+apAWuzLQL95obVYAymNJek76W3mgw= github.com/envoyproxy/go-control-plane v0.10.1/go.mod h1:AY7fTTXNdv/aJ2O5jwpxAPOWUZ7hQAEvzN5Pf27BkQQ= github.com/envoyproxy/protoc-gen-validate v0.1.0 h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A= @@ -924,8 +925,9 @@ google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8 google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk= google.golang.org/grpc v1.33.1/go.mod h1:fr5YgcSWrqhRRxogOsw7RzIpsmvOZ6IcH4kBYTpR3n0= -google.golang.org/grpc v1.36.0 h1:o1bcQ6imQMIOpdrO3SWf2z5RV72WbDwdXuK0MDlc8As= google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= +google.golang.org/grpc v1.37.1 h1:ARnQJNWxGyYJpdf/JXscNlQr/uv607ZPU9Z7ogHi+iI= +google.golang.org/grpc v1.37.1/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= From 80556c9ffc6909105974b3278faa05bc1c73b6c3 Mon Sep 17 00:00:00 2001 From: Dhia Ayachi Date: Wed, 8 Jun 2022 10:38:55 -0400 Subject: [PATCH 015/247] Fix intentions wildcard dest (#13397) * when enterprise meta are wildcard assume it's a service intention * fix partition and namespace * move kind outside the loops * get the kind check outside the loop and add a comment Co-authored-by: github-team-consul-core --- agent/consul/state/config_entry_intention.go | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/agent/consul/state/config_entry_intention.go b/agent/consul/state/config_entry_intention.go index ebbc9e7b7..4f1a883c0 100644 --- a/agent/consul/state/config_entry_intention.go +++ b/agent/consul/state/config_entry_intention.go @@ -286,13 +286,18 @@ func readSourceIntentionsFromConfigEntriesForServiceTxn( for v := iter.Next(); v != nil; v = iter.Next() { entry := v.(*structs.ServiceIntentionsConfigEntry) + entMeta := entry.DestinationServiceName().EnterpriseMeta + // if we have a wildcard namespace or partition assume we are querying a service intention + // as destination intentions will never be queried as wildcard + kind := structs.GatewayServiceKindService + if entMeta.NamespaceOrDefault() != acl.WildcardName && entMeta.PartitionOrDefault() != acl.WildcardName { + kind, err = GatewayServiceKind(tx, entry.DestinationServiceName().Name, &entMeta) + if err != nil { + return nil, err + } + } for _, src := range entry.Sources { if src.SourceServiceName() == sn { - entMeta := entry.DestinationServiceName().EnterpriseMeta - kind, err := GatewayServiceKind(tx, entry.DestinationServiceName().Name, &entMeta) - if err != nil { - return nil, err - } switch targetType { case structs.IntentionTargetService: if kind == structs.GatewayServiceKindService || kind == structs.GatewayServiceKindUnknown { From f58fca2048d58aeb233f064e617eabf1b9b84b21 Mon Sep 17 00:00:00 2001 From: Michael Klein Date: Wed, 8 Jun 2022 17:07:51 +0200 Subject: [PATCH 016/247] ui: upgrade ember composable helper (#13394) Upgrade ember-composable-helpers to version 5.x. This version contains the pick-helper which makes composition in the template layer easier with Octane. {{!-- this is usually hard to do with Octane --}} Version 5.x also fixes a regression with sort-by that according to @johncowen was the reason why the version was pinned to 4.0.0 at the moment. Version 5 of ember-composable-helpers removes the contains-helper in favor of includes which I changed all occurences for. --- .changelog/13394.txt | 3 +++ .../consul/lock-session/notifications/README.mdx | 2 +- .../components/consul/nspace/search-bar/index.hbs | 2 +- .../consul/partition/search-bar/index.hbs | 2 +- .../consul/auth-method/search-bar/index.hbs | 10 +++++----- .../consul/health-check/search-bar/index.hbs | 8 ++++---- .../consul/intention/permission/form/index.hbs | 2 +- .../consul/intention/search-bar/index.hbs | 4 ++-- .../app/components/consul/kv/search-bar/index.hbs | 4 ++-- .../components/consul/node/search-bar/index.hbs | 4 ++-- .../components/consul/policy/search-bar/index.hbs | 6 +++--- .../components/consul/role/search-bar/index.hbs | 2 +- .../app/components/consul/server/card/index.hbs | 4 ++-- .../consul/service-instance/search-bar/index.hbs | 6 +++--- .../components/consul/service/search-bar/index.hbs | 14 +++++++------- .../components/consul/token/search-bar/index.hbs | 4 ++-- .../consul/upstream-instance/search-bar/index.hbs | 2 +- .../consul/upstream/search-bar/index.hbs | 4 ++-- .../app/components/disclosure-menu/menu/index.hbs | 2 +- .../app/components/form-group/element/index.hbs | 2 +- .../consul-ui/app/components/policy-form/index.hbs | 2 +- .../consul-ui/app/styles/base/icons/README.mdx | 2 +- .../consul-ui/app/templates/application.hbs | 2 +- .../app/templates/dc/services/show/topology.hbs | 2 +- ui/packages/consul-ui/package.json | 2 +- ui/yarn.lock | 14 +++++++------- 26 files changed, 57 insertions(+), 54 deletions(-) create mode 100644 .changelog/13394.txt diff --git a/.changelog/13394.txt b/.changelog/13394.txt new file mode 100644 index 000000000..c00759f44 --- /dev/null +++ b/.changelog/13394.txt @@ -0,0 +1,3 @@ +```release-note:improvement +ui: upgrade ember-composable-helpers to v5.x +``` diff --git a/ui/packages/consul-lock-sessions/app/components/consul/lock-session/notifications/README.mdx b/ui/packages/consul-lock-sessions/app/components/consul/lock-session/notifications/README.mdx index 6d90f0fff..07ee84b1c 100644 --- a/ui/packages/consul-lock-sessions/app/components/consul/lock-session/notifications/README.mdx +++ b/ui/packages/consul-lock-sessions/app/components/consul/lock-session/notifications/README.mdx @@ -30,7 +30,7 @@ A Notification component specifically for LockSessions. {{#let components.Optgroup components.Option as |Optgroup Option|}} {{#each @filter.searchproperty.default as |prop|}} - {{/each}} diff --git a/ui/packages/consul-partitions/app/components/consul/partition/search-bar/index.hbs b/ui/packages/consul-partitions/app/components/consul/partition/search-bar/index.hbs index 1d2999c9b..b0f15ec30 100644 --- a/ui/packages/consul-partitions/app/components/consul/partition/search-bar/index.hbs +++ b/ui/packages/consul-partitions/app/components/consul/partition/search-bar/index.hbs @@ -55,7 +55,7 @@ as |key value|}} {{#let components.Optgroup components.Option as |Optgroup Option|}} {{#each @filter.searchproperty.default as |prop|}} - {{/each}} diff --git a/ui/packages/consul-ui/app/components/consul/auth-method/search-bar/index.hbs b/ui/packages/consul-ui/app/components/consul/auth-method/search-bar/index.hbs index ec8b98336..be122a1d8 100644 --- a/ui/packages/consul-ui/app/components/consul/auth-method/search-bar/index.hbs +++ b/ui/packages/consul-ui/app/components/consul/auth-method/search-bar/index.hbs @@ -55,7 +55,7 @@ as |key value|}} {{#let components.Optgroup components.Option as |Optgroup Option|}} {{#each @filter.searchproperty.default as |prop|}} - {{/each}} @@ -78,10 +78,10 @@ as |key value|}} {{#let components.Optgroup components.Option as |Optgroup Option|}} - - + + {{#if (env 'CONSUL_SSO_ENABLED')}} - + {{/if}} {{/let}} @@ -100,7 +100,7 @@ as |key value|}} {{#let components.Optgroup components.Option as |Optgroup Option|}} {{#each (array "local" "global") as |option|}} - {{/each}} diff --git a/ui/packages/consul-ui/app/components/consul/health-check/search-bar/index.hbs b/ui/packages/consul-ui/app/components/consul/health-check/search-bar/index.hbs index 78392f665..9c784527a 100644 --- a/ui/packages/consul-ui/app/components/consul/health-check/search-bar/index.hbs +++ b/ui/packages/consul-ui/app/components/consul/health-check/search-bar/index.hbs @@ -55,7 +55,7 @@ as |key value|}} {{#let components.Optgroup components.Option as |Optgroup Option|}} {{#each @filter.searchproperty.default as |prop|}} - {{/each}} @@ -80,7 +80,7 @@ as |key value|}} {{#let components.Optgroup components.Option as |Optgroup Option|}} {{#each (array "passing" "warning" "critical" "empty") as |state|}} -