Commit Graph

4291 Commits

Author SHA1 Message Date
R.B. Boyer 70b143ddc5
auto-config: ensure the feature works properly with partitions (#11699) 2021-12-01 13:32:34 -06:00
Daniel Nephin 963a9819d0 ca: add some godoc and func for finding leaf signing cert
This will be used in a follow up commit.
2021-11-30 18:36:41 -05:00
Daniel Nephin 056a52ba64 sdk/freeport: rename Port to GetOne
For better consistency with GetN
2021-11-30 17:32:41 -05:00
Chris S. Kim e9c661db7f
Refactor test helper (#11689)
Allow custom ACL root tokens to be passed
2021-11-30 13:22:07 -05:00
Chris S. Kim 0ec67cc2d1
acl: Fill authzContext from token in Coordinate endpoints (#11688) 2021-11-30 13:17:41 -05:00
freddygv 76146dfc5b Move ent config test to ent file 2021-11-29 12:15:17 -07:00
freddygv 6d51282adf Prevent partition-exports entry from OSS usage
Validation was added on the config entry kind since that is called when
validating config entries to bootstrap via agent configuration and when
applying entries via the config RPC endpoint.
2021-11-29 11:24:16 -07:00
Daniel Nephin 4f0d092c95 testing: remove unnecessary calls to freeport
Previously we believe it was necessary for all code that required ports
to use freeport to prevent conflicts.

https://github.com/dnephin/freeport-test shows that it is actually save
to use port 0 (`127.0.0.1:0`) as long as it is passed directly to
`net.Listen`, and the listener holds the port for as long as it is
needed.

This works because freeport explicitly avoids the ephemeral port range,
and port 0 always uses that range. As you can see from the test output
of https://github.com/dnephin/freeport-test, the two systems never use
overlapping ports.

This commit converts all uses of freeport that were being passed
directly to a net.Listen to use port 0 instead. This allows us to remove
a bit of wrapping we had around httptest, in a couple places.
2021-11-29 12:19:43 -05:00
Daniel Nephin 20a8e11bf2 testing: use the new freeport interfaces 2021-11-27 15:39:46 -05:00
Daniel Nephin 2cf41e4dc8 go-sso: remove returnFunc now that freeport handles return 2021-11-27 15:29:38 -05:00
Daniel Nephin 8219e8571e sdk: add freeport functions that use t.Cleanup 2021-11-27 15:04:43 -05:00
Daniel Nephin 772d8f7381 ca: clean up unnecessary raft.Apply response checking
In d2ab767fef21244e9fe3b9887ea70fc177912381 raftApply was changed to handle this check in
a single place, instad of having every caller check it. It looks like these few places
were missed when I did that clean up.

This commit removes the remaining resp.(error) checks, since they are all no-ops now.
2021-11-26 17:57:55 -05:00
Daniel Nephin 48954adfdc
Merge pull request #11339 from hashicorp/dnephin/ca-manager-isolate-secondary-2
ca: reduce use of state in the secondary
2021-11-26 14:41:45 -05:00
Daniel Nephin 8240286956 ca: remove state check in secondarySetPrimaryRoots
This function is only ever called from operations that have already acquired the state lock, so checking
the value of state can never fail.

This change is being made in preparation for splitting out a separate type for the secondary logic. The
state can't easily be shared, so really only the expored top-level functions should acquire the 'state lock'.
2021-11-26 14:14:47 -05:00
Daniel Nephin 877094e2fa ca: remove actingSecondaryCA
This commit removes the actingSecondaryCA field, and removes the stateLock around it. This field
was acting as a proxy for providerRoot != nil, so replace it with that check instead.

The two methods which called secondarySetCAConfigured already set the state, so checking the
state again at this point will not catch runtime errors (only programming errors, which we can catch with tests).
In general, handling state transitions should be done on the "entrypoint" methods where execution starts, not
in every internal method.

This is being done to remove some unnecessary references to c.state, in preparations for extracting
types for primary/secondary.
2021-11-26 14:14:47 -05:00
Daniel Nephin cd5f6b2dfb ca: reduce consul provider backend interface a bit
This makes it easier to fake, which will allow me to use the ConsulProvider as
an 'external PKI' to test a customer setup where the actual root CA is not
the root we use for the Consul CA.

Replaces a call to the state store to fetch the clusterID with the
clusterID field already available on the built-in provider.
2021-11-25 11:46:06 -05:00
Dhia Ayachi f605689154
Partition/kv indexid sessions (#11639)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* partition `tableSessions` table

* fix sessions to use UUID and fix prefix index

* fix oss build

* clean up unused functions

* fix oss compilation

* add a partition indexer for sessions

* Fix oss to not have partition index

* fix oss tests

* remove unused operations_ent.go and operations_oss.go func

* convert `indexNodeCheck` of `session_checks` table

* partition `indexID` and `indexSession` of `tableSessionChecks`

* remove partition for Checks as it's always use the session partition

* partition sessions index id table

* fix rebase issues

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-11-24 11:34:36 -05:00
Dhia Ayachi b1c4be3da0
Partition session checks store (#11638)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add `singleValueID` implementation assertions

* partition `tableSessions` table

* fix sessions to use UUID and fix prefix index

* fix oss build

* clean up unused functions

* fix oss compilation

* add a partition indexer for sessions

* Fix oss to not have partition index

* fix oss tests

* remove unused operations_ent.go and operations_oss.go func

* remove unused const

* convert `IndexID` of `session_checks` table

* convert `indexSession` of `session_checks` table

* convert `indexNodeCheck` of `session_checks` table

* partition `indexID` and `indexSession` of `tableSessionChecks`

* fix oss linter

* fix review comments

* remove partition for Checks as it's always use the session partition

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-11-24 09:10:38 -05:00
Chris S. Kim c22adc8dc7
cleanup: Clarify deprecated legacy intention endpoints (#11635) 2021-11-23 19:32:18 -05:00
Chris S. Kim d2b86e7f48
Merge from ent (#11506) 2021-11-19 11:50:44 -05:00
R.B. Boyer fa7a66cd30
agent: purge service/check registration files for incorrect partitions on reload (#11607) 2021-11-18 14:44:20 -06:00
Iryna Shustava bd3fb0d0e9
connect: Support auth methods for the vault connect CA provider (#11573)
* Support vault auth methods for the Vault connect CA provider
* Rotate the token (re-authenticate to vault using auth method) when the token can no longer be renewed
2021-11-18 13:15:28 -07:00
Daniel Nephin fee9696d4f ca: use the cluster ID passed to the primary
instead of fetching it from the state store.
2021-11-16 16:57:22 -05:00
Daniel Nephin 07a33a1526 ca: accept only the cluster ID to SpiffeIDSigningForCluster
To make it more obivous where ClusterID is used, and remove the need to create a struct
when only one field is used.
2021-11-16 16:57:21 -05:00
Will Jordan 2e66b7a5e6
Update node info sync comment (#11465) 2021-11-16 11:16:11 -08:00
R.B. Boyer 83bf7ab3ff
re-run gofmt on 1.17 (#11579)
This should let freshly recompiled golangci-lint binaries using Go 1.17
pass 'make lint'
2021-11-16 12:04:01 -06:00
R.B. Boyer 086ff42b56
partitions: various refactors to support partitioning the serf LAN pool (#11568) 2021-11-15 09:51:14 -06:00
freddygv f33eae6fe1 Update proxycfg for ingress service partitions 2021-11-12 14:33:31 -07:00
freddygv dc7ea2ef1e Accept partition for ingress services 2021-11-12 14:33:14 -07:00
freddygv 5ac1ab359b Move assertion to after config fetch 2021-11-10 10:50:08 -07:00
freddygv 2261d51515 Use ClusterID to check for readiness
The TrustDomain is populated from the Host() method which includes the
hard-coded "consul" domain. This means that despite having an empty
cluster ID, the TrustDomain won't be empty.
2021-11-10 10:45:22 -07:00
freddygv 482d3bc610 Prevent replicating partition-exports 2021-11-09 16:42:42 -07:00
freddygv 739490df12 handle error scenario of empty local DC 2021-11-09 16:42:42 -07:00
freddygv b9b41625b9 Restrict DC for partition-exports writes
There are two restrictions:
- Writes from the primary DC which explicitly target a secondary DC.
- Writes to a secondary DC that do not explicitly target the primary DC.

The first restriction is because the config entry is not supported in
secondary datacenters.

The second restriction is to prevent the scenario where a user writes
the config entry to a secondary DC, the write gets forwarded to the
primary, but then the config entry does not apply in the secondary.
This makes the scope more explicit.
2021-11-09 16:42:42 -07:00
Freddy eb2b40b22d
Update filter chain creation for sidecar/ingress listeners (#11245)
The duo of `makeUpstreamFilterChainForDiscoveryChain` and `makeListenerForDiscoveryChain` were really hard to reason about, and led to concealing a bug in their branching logic. There were several issues here:

- They tried to accomplish too much: determining filter name, cluster name, and whether RDS should be used. 
- They embedded logic to handle significantly different kinds of upstream listeners (passthrough, prepared query, typical services, and catch-all)
- They needed to coalesce different data sources (Upstream and CompiledDiscoveryChain)

Rather than handling all of those tasks inside of these functions, this PR pulls out the RDS/clusterName/filterName logic.

This refactor also fixed a bug with the handling of [UpstreamDefaults](https://www.consul.io/docs/connect/config-entries/service-defaults#defaults). These defaults get stored as UpstreamConfig in the proxy snapshot with a DestinationName of "*", since they apply to all upstreams. However, this wildcard destination name must not be used when creating the name of the associated upstream cluster. The coalescing logic in the original functions here was in some situations creating clusters with a `*.` prefix, which is not a valid destination.
2021-11-09 14:43:51 -07:00
Kyle Havlovitz 14591de8d2
Merge pull request #11461 from deblasis/feature/empty_client_addr_warning
config: warn the user if client_addr is empty
2021-11-09 09:37:38 -08:00
Daniel Upton caa5b5a5a6
xds: prefer fed state gateway definitions if they're fresher (#11522)
Fixes an issue described in #10132, where if two DCs are WAN federated
over mesh gateways, and the gateway in the non-primary DC is terminated
and receives a new IP address (as is commonly the case when running them
on ephemeral compute instances) the primary DC is unable to re-establish
its connection until the agent running on its own gateway is restarted.

This was happening because we always preferred gateways discovered by
the `Internal.ServiceDump` RPC (which would fail because there's no way
to dial the remote DC) over those discovered in the federation state,
which is replicated as long as the primary DC's gateway is reachable.
2021-11-09 16:45:36 +00:00
Freddy 0ad360fadf
Merge pull request #11514 from hashicorp/dnephin/ca-fix-secondary-init
ca: properly handle the case where the secondary initializes after the primary
2021-11-08 17:16:16 -07:00
freddygv e6622ab0ab Avoid returning empty roots with uninitialized CA
Currently getCARoots could return an empty object with an empty trust
domain before the CA is initialized. This commit returns an error while
there is no CA config or no trust domain.

There could be a CA config and no trust domain because the CA config can
be created in InitializeCA before initialization succeeds.
2021-11-08 16:51:49 -07:00
Dhia Ayachi f61892393f
refactor session state store tables to use the new index pattern (#11525)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add `singleValueID` implementation assertions

* partition `tableSessions` table

* fix sessions to use UUID and fix prefix index

* fix oss build

* clean up unused functions

* fix oss compilation

* add a partition indexer for sessions

* Fix oss to not have partition index

* fix oss tests

* remove unused func `prefixIndexFromServiceNameAsString`

* fix test error check

* remove unused operations_ent.go and operations_oss.go func

* remove unused const

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-11-08 16:20:50 -05:00
Dhia Ayachi dfafd4e38c
KV refactoring, part 2 (#11512)
* add partition to the kv get pretty print

* fix failing test

* add test for kvs RPC endpoint
2021-11-08 11:43:21 -05:00
Dhia Ayachi 17190c0076
KV state store refactoring and partitioning (#11510)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* partition kvs indexID table

* add `partitionedIndexEntryName` in oss for test purpose

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add `singleValueID` implementation assertions

* remove entmeta reference from oss

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-11-08 09:35:56 -05:00
Giulio Micheloni 10cdc0a5c8
Merge branch 'main' into serve-panic-recovery 2021-11-06 16:12:06 +01:00
Daniel Nephin 69ad7c0544 ca: Only initialize clusterID in the primary
The secondary must get the clusterID from the primary
2021-11-05 18:08:44 -04:00
Daniel Nephin 3173582b75 ca: return an error when secondary fails to initialize
Previously secondaryInitialize would return nil in this case, which prevented the
deferred initialize from happening, and left the CA in an uninitialized state until a config
update or root rotation.

To fix this I extracted the common parts into the delegate implementation. However looking at this
again, it seems like the handling in secondaryUpdateRoots is impossible, because that function
should never be called before the secondary is initialzied. I beleive we can remove some of that
logic in a follow up.
2021-11-05 18:02:51 -04:00
Daniel Nephin db29ad346b acl: remove id and revision from Policy constructors
The fields were removed in a previous commit.

Also remove an unused constructor for PolicyMerger
2021-11-05 15:45:08 -04:00
Daniel Nephin 617b11302f acl: remove Policy.ID and Policy.Revision
These two fields do not appear to be used anywhere. We use the structs.ACLPolicy ID in the
ACLResolver cache, but the acl.Policy ID and revision are not used.
2021-11-05 15:43:52 -04:00
R.B. Boyer 1d8e7bb565
rename helper method to reflect the non-deprecated terminology (#11509) 2021-11-05 13:51:50 -05:00
Connor b3af482e09
Support Vault Namespaces explicitly in CA config (#11477)
* Support Vault Namespaces explicitly in CA config

If there is a Namespace entry included in the Vault CA configuration,
set it as the Vault Namespace on the Vault client

Currently the only way to support Vault namespaces in the Consul CA
config is by doing one of the following:
1) Set the VAULT_NAMESPACE environment variable which will be picked up
by the Vault API client
2) Prefix all Vault paths with the namespace

Neither of these are super pleasant. The first requires direct access
and modification to the Consul runtime environment. It's possible and
expected, not super pleasant.

The second requires more indepth knowledge of Vault and how it uses
Namespaces and could be confusing for anyone without that context. It
also infers that it is not supported

* Add changelog

* Remove fmt.Fprint calls

* Make comment clearer

* Add next consul version to website docs

* Add new test for default configuration

* go mod tidy

* Add skip if vault not present

* Tweak changelog text
2021-11-05 11:42:28 -05:00
R.B. Boyer 7fbf749bc4
segments: ensure that the serf_lan_allowed_cidrs applies to network segments (#11495) 2021-11-04 17:17:19 -05:00
Mark Anderson e9a0fa7d36
Remove some usage of md5 from the system (#11491)
* Remove some usage of md5 from the system

OSS side of https://github.com/hashicorp/consul-enterprise/pull/1253

This is a potential security issue because an attacker could conceivably manipulate inputs to cause persistence files to collide, effectively deleting the persistence file for one of the colliding elements.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-11-04 13:07:54 -07:00
FFMMM 9afecfa10c
plumb thru root cert tll to the aws ca provider (#11449)
* plumb thru root cert ttl to the aws ca provider

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

* Update .changelog/11449.txt

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
2021-11-04 12:19:08 -07:00
FFMMM e7ffef54ee
fix aws pca certs (#11470)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2021-11-03 12:21:24 -07:00
Mathew Estafanous 508664440d
Convert (some) test endpoints to use ServeHTTP instead of direct calls to handlers. (#11445) 2021-11-03 11:12:36 -04:00
FFMMM 27227c0fd2
add root_cert_ttl option for consul connect, vault ca providers (#11428)
* add root_cert_ttl option for consul connect, vault ca providers

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>

* add changelog, pr feedback

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

* Update .changelog/11428.txt, more docs

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* Update website/content/docs/agent/options.mdx

Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>
2021-11-02 11:02:10 -07:00
Daniel Nephin 0ec2a804df
Merge pull request #10690 from tarat44/h2c-support-in-ping-checks
add support for h2c in h2 ping health checks
2021-11-02 13:53:06 -04:00
Alessandro De Blasis 2b3f4efbab config: warn the user if client_addr is empty
if the provided value is empty string then the client services
(DNS, HTTP, HTTPS, GRPC) are not listening and the user is not notified
in any way about what's happening.
Also, since a not provided client_addr defaults to 127.0.0.1, we make sure
we are not getting unwanted warnings

Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2021-11-01 22:47:20 +00:00
Daniel Nephin 00ed2b243f
Merge pull request #10771 from hashicorp/dnephin/emit-telemetry-metrics-immediately
telemetry: improve cert expiry metrics
2021-11-01 18:31:03 -04:00
freddygv ecccf22fd7 Exclude default partition from GatewayKey string
This will behave the way we handle SNI and SPIFFE IDs, where the default
partition is excluded.

Excluding the default ensures that don't attempt to compare default.dc2
to dc2 in OSS.
2021-11-01 14:45:52 -06:00
freddygv d944e6ae3a Update GatewayKeys deduplication
Federation states data is only keyed on datacenter, so it cannot be
directly compared against keys for gateway groups.
2021-11-01 13:58:53 -06:00
freddygv ce43e8cf99 Store GatewayKey in proxycfg snapshot for re-use 2021-11-01 13:58:53 -06:00
freddygv 51c888a41a Update locality check in xds 2021-11-01 13:58:53 -06:00
freddygv 6657c88296 Update locality check in proxycfg 2021-11-01 13:58:53 -06:00
Daniel Nephin c706bf135c
Merge pull request #11340 from hashicorp/dnephin/ca-manager-provider
ca: split the Provider interface into Primary/Secondary
2021-11-01 14:11:15 -04:00
Daniel Nephin eaaceedf31
Merge pull request #11338 from hashicorp/dnephin/ca-manager-isolate-secondary
ca: clearly identify methods that are primary-only or secondary-only
2021-11-01 14:10:31 -04:00
Daniel Upton a620b6be2e
Support Check-And-Set deletion of config entries (#11419)
Implements #11372
2021-11-01 16:42:01 +00:00
Dhia Ayachi 4d763ef9e6
regenerate expired certs (#11462)
* regenerate expired certs

* add documentation to generate tests certificates
2021-11-01 11:40:16 -04:00
Jared Kirschner 6dfcbeceec
Merge pull request #11348 from kbabuadze/fix-answers-alt-domain
Fix answers for alt domain
2021-10-29 17:09:20 -04:00
R.B. Boyer d40d098321
agent: for various /v1/agent endpoints parse the partition parameter on the request (#11444)
Also update the corresponding CLI commands to send the parameter
appropriately.

NOTE: Behavioral changes are not happening in this PR.
2021-10-28 16:44:38 -05:00
R.B. Boyer 017e9d5ae4
agent: add a clone function for duplicating the serf lan configuration (#11443) 2021-10-28 16:11:26 -05:00
Daniel Nephin a8d6392ab5 Add tests for cert expiry metrics 2021-10-28 14:38:57 -04:00
Daniel Nephin 503dee2d80
Merge pull request #10671 from hashicorp/dnephin/fix-subscribe-test-flake
subscribe: improve TestSubscribeBackend_IntegrationWithServer_DeliversAllMessages
2021-10-28 12:57:09 -04:00
Evan Culver b3c92f22b1
connect: Remove support for Envoy 1.16 (#11354) 2021-10-27 18:51:35 -07:00
Evan Culver 98acbfa79c
connect: Add support for Envoy 1.20 (#11277) 2021-10-27 18:38:10 -07:00
freddygv 3dd21023bc Ensure partition-exports kind gets marshalled
The api module has decoding functions that rely on 'kind' being present
of payloads. This is so that we can decode into the appropriate api type
for the config entry.

This commit ensures that a static kind is marshalled in responses from
Consul's api endpoints so that the api module can decode them.
2021-10-27 15:01:26 -06:00
Daniel Nephin 0a19d7fd76 agent: move agent tls metric monitor to a more appropriate place
And add a test for it
2021-10-27 16:26:09 -04:00
Daniel Nephin 1b2144c982 telemetry: set cert expiry metrics to NaN on start
So that followers do not report 0, which would make alerting difficult.
2021-10-27 15:19:25 -04:00
Daniel Nephin a7fcf14c5c telemetry: fix cert expiry metrics by removing labels
These labels should be set by whatever process scrapes Consul (for
prometheus), or by the agent that receives them (for datadog/statsd).

We need to remove them here because the labels are part of the "metric
key", so we'd have to pre-declare the metrics with the labels. We could
do that, but that is extra work for labels that should be added from
elsewhere.

Also renames the closure to be more descriptive.
2021-10-27 15:19:25 -04:00
Daniel Nephin 4300daa2e6 telemetry: only emit leader cert expiry metrics on the servers 2021-10-27 15:19:25 -04:00
Daniel Nephin 9de725c17d telemetry: prevent stale values from cert monitors
Prometheus scrapes metrics from each process, so when leadership transfers to a different node
the previous leader would still be reporting the old cached value.

By setting NaN, I believe we should zero-out the value, so that prometheus should only consider the
value from the new leader.
2021-10-27 15:19:25 -04:00
Daniel Nephin 616cc9b6f8 telemetry: improve cert expiry metrics
Emit the metric immediately so that after restarting an agent, the new expiry time will be
emitted. This is particularly important when this metric is being monitored, because we want
the alert to resovle itself immediately.

Also fixed a bug that was exposed in one of these metrics. The CARoot can be nil, so we have
to handle that case.
2021-10-27 15:19:25 -04:00
Daniel Nephin 24951f0c7e subscribe: attempt to fix a flaky test
TestSubscribeBackend_IntegrationWithServer_DeliversAllMessages has been
flaking a few times. This commit cleans up the test a bit, and improves
the failure output.

I don't believe this actually fixes the flake, but I'm not able to
reproduce it reliably.

The failure appears to be that the event with Port=0 is being sent in
both the snapshot and as the first event after the EndOfSnapshot event.

Hopefully the improved logging will show us if these are really
duplicate events, or actually different events with different indexes.
2021-10-27 15:09:09 -04:00
Freddy ae76144f55
Merge pull request #11435 from hashicorp/ent-authorizer-refactor
[OSS] Export ACLs refactor
2021-10-27 13:04:40 -06:00
Freddy 520bda999b
Merge pull request #11432 from hashicorp/ap/exports-mgw
[OSS] Update mesh gateways to handle partitions
2021-10-27 12:54:53 -06:00
freddygv 592965d61e Rework acl exports interface 2021-10-27 12:50:39 -06:00
Freddy 9bbeea0432
Merge pull request #11433 from hashicorp/exported-service-acls
[OSS] acl: Expand ServiceRead and NodeRead to account for partition exports
2021-10-27 12:48:08 -06:00
freddygv 05f91bd2b8 Update comments 2021-10-27 12:36:44 -06:00
Freddy d8ae915160
Merge pull request #11431 from hashicorp/ap/exports-proxycfg
[OSS] Update partitioned mesh gw handling for connect proxies
2021-10-27 11:27:43 -06:00
Freddy 8e23a6a0cc
Merge pull request #11416 from hashicorp/ap/exports-update
Rename service-exports to partition-exports
2021-10-27 11:27:31 -06:00
freddygv 40271beb38 Fixup partitions assertion 2021-10-27 11:15:25 -06:00
freddygv 67412ac5e7 Fixup imports 2021-10-27 11:15:25 -06:00
freddygv 4de3537391 Split up locality check from hostname check 2021-10-27 11:15:25 -06:00
freddygv 9769b31641 Move the exportingpartitions constant to enterprise 2021-10-27 11:15:25 -06:00
freddygv 0391a65772 Replace default partition check 2021-10-27 11:15:25 -06:00
freddygv ee45ac9dc5 PR comments 2021-10-27 11:15:25 -06:00
freddygv f99946553a Leave todo about default name 2021-10-27 11:15:25 -06:00
freddygv 9d375ad6d2 Add oss impl of registerEntCache 2021-10-27 11:15:25 -06:00
freddygv 183849416b Register the ExportingPartitions cache type 2021-10-27 11:15:25 -06:00
freddygv 8b5a9369eb Account for partitions in xds gen for mesh gw
This commit avoids skipping gateways in remote partitions of the local
DC when generating listeners/clusters/endpoints.
2021-10-27 11:15:25 -06:00
freddygv d1d513b1b3 Account for partition in SNI for gateways 2021-10-27 11:15:25 -06:00
freddygv 4f0432be5e Update xds pkg to account for GatewayKey 2021-10-27 09:03:56 -06:00
freddygv f3f15640a9 Update mesh gateway proxy watches for partitions
This commit updates mesh gateway watches for cross-partitions
communication.

* Mesh gateways are keyed by partition and datacenter.

* Mesh gateways will now watch gateways in partitions that export
services to their partition.

* Mesh gateways in non-default partitions will not have cross-datacenter
watches. They are not involved in traditional WAN federation.
2021-10-27 09:03:56 -06:00
freddygv af662c8c1c Avoid mixing named and unnamed params 2021-10-26 23:42:25 -06:00
freddygv 1de62bb0a2 Avoid passing nil config pointer 2021-10-26 23:42:25 -06:00
freddygv 4a2e40aa3c Avoid panic on nil partitionAuthorizer config
partitionAuthorizer.config can be nil if it wasn't provided on calls to
newPartitionAuthorizer outside of the ACLResolver. This usage happens
often in tests.

This commit: adds a nil check when the config is going to be used,
updates non-test usage of NewPolicyAuthorizerWithDefaults to pass a
non-nil config, and dettaches setEnterpriseConf from the ACLResolver.
2021-10-26 23:42:25 -06:00
freddygv 015d85cd74 Update NodeRead for partition-exports
When issuing cross-partition service discovery requests, ACL filtering
often checks for NodeRead privileges. This is because the common return
type is a CheckServiceNode, which contains node data.
2021-10-26 23:42:11 -06:00
Kyle Havlovitz afb0976eac acl: pass PartitionInfo through ent ACLConfig 2021-10-26 23:41:52 -06:00
Kyle Havlovitz 56d1858c4a acl: Expand ServiceRead logic to look at service-exports for cross-partition 2021-10-26 23:41:32 -06:00
freddygv 4737ad118d Swap in structs.EqualPartitions for cmp 2021-10-26 23:36:01 -06:00
freddygv 1bade08f91 Replace Split with SplitN 2021-10-26 23:36:01 -06:00
freddygv 3966677aaf Finish removing useInDatacenter 2021-10-26 23:36:01 -06:00
freddygv 69476221c1 Update XDS for sidecars dialing through gateways 2021-10-26 23:35:48 -06:00
freddygv ea311d2e47 Configure sidecars to watch gateways in partitions
Previously the datacenter of the gateway was the key identifier, now it
is the datacenter and partition.

When dialing services in other partitions or datacenters we now watch
the appropriate partition.
2021-10-26 23:35:37 -06:00
freddygv feaebde1f1 Remove useInDatacenter from disco chain requests
useInDatacenter was used to determine whether the mesh gateway mode of
the upstream should be returned in the discovery chain target. This
commit makes it so that the mesh gateway mode is returned every time,
and it is up to the caller to decide whether mesh gateways should be
watched or used.
2021-10-26 23:35:21 -06:00
R.B. Boyer e27e58c6cc
agent: refactor the agent delegate interface to be partition friendly (#11429) 2021-10-26 15:08:55 -05:00
Chris S. Kim 27f8a85664
agent: Ensure partition is considered in agent endpoints (#11427) 2021-10-26 15:20:57 -04:00
Konstantine 2f9ee8e558 remove spaces 2021-10-26 12:38:13 -04:00
Konstantine be14f6da90 fix altDomain responses for services where address is IP, added tests 2021-10-26 12:38:13 -04:00
Konstantine eec9d66e22 fix encodeIPAsFqdn to return alt-domain when requested, added test case 2021-10-26 12:38:12 -04:00
Konstantine 9d6797a463 fixed altDomain response for NS type queries, and added test 2021-10-26 12:38:12 -04:00
Konstantine 0735e12412 edited TestDNS_AltDomains_Service to test responses for altDomains, and added TXT additional section check 2021-10-26 12:38:12 -04:00
Konstantine 8972e093d9 fixed alt-domain answer for SRV records, and TXT records in additional section 2021-10-26 12:38:12 -04:00
Chris S. Kim 3f736467e6
ui: Pass primary dc through to uiserver (#11317)
Co-authored-by: John Cowen <johncowen@users.noreply.github.com>
2021-10-26 10:30:17 -04:00
freddygv 83d4d0e108 Remove outdated partition label from test 2021-10-25 18:47:02 -06:00
freddygv c3e381b4c1 Rename service-exports to partition-exports
Existing config entries prefixed by service- are specific to individual
services. Since this config entry applies to partitions it is being
renamed.

Additionally, the Partition label was changed to Name because using
Partition at the top-level and in the enterprise meta was leading to the
enterprise meta partition being dropped by msgpack.
2021-10-25 17:58:48 -06:00
Daniel Nephin f24bad2a52
Merge pull request #11232 from hashicorp/dnephin/acl-legacy-remove-docs
acl: add docs and changelog for the removal of the legacy ACL system
2021-10-25 18:38:00 -04:00
Daniel Nephin f7cdd210fe Update agent/consul/acl_client.go
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2021-10-25 17:25:14 -04:00
Daniel Nephin 732b841dd7 state: remove support for updating legacy ACL tokens 2021-10-25 17:25:14 -04:00
Daniel Nephin 76b007dacd acl: remove init check for legacy anon token
This token should always already be migrated from a previous version.
2021-10-25 17:25:14 -04:00
Daniel Nephin 8ae6ee4e36 acl: remove legacy parameter to ACLDatacenter
It is no longer used now that legacy ACLs have been removed.
2021-10-25 17:25:14 -04:00
Daniel Nephin d778113773 acl: remove ACLTokenTypeManagement 2021-10-25 17:25:14 -04:00
Daniel Nephin 2f0eba1980 acl: remove ACLTokenTypeClient,
along with the last test referencing it.
2021-10-25 17:25:14 -04:00
Daniel Nephin 88c6aeea34 acl: remove legacy arg to store.ACLTokenSet
And remove the tests for legacy=true
2021-10-25 17:25:14 -04:00
Daniel Nephin b31a7fc498 acl: remove EmbeddedPolicy
This method is no longer. It only existed for legacy tokens, which are no longer supported.
2021-10-25 17:25:14 -04:00
Daniel Nephin ceaa36f983 acl: remove tests for resolving legacy tokens
The code for this was already removed, which suggests this is not actually testing what it claims.

I'm guessing these are still resolving because the tokens are converted to non-legacy tokens?
2021-10-25 17:25:14 -04:00
Daniel Nephin a46e3bd2fc acl: stop replication on leadership lost
It seems like this was missing. Previously this was only called by init of ACLs during an upgrade.
Now that legacy ACLs are  removed, nothing was calling stop.

Also remove an unused method from client.
2021-10-25 17:24:12 -04:00
Daniel Nephin 15cd8c7ab8 Remove incorrect TODO 2021-10-25 17:20:06 -04:00
Daniel Nephin 589b238374 acl: move the legacy ACL struct to the one package where it is used
It is now only used for restoring snapshots. We can remove it in phase 2.
2021-10-25 17:20:06 -04:00
Daniel Nephin 0ba5d0afcd acl: remove most of the rest of structs/acl_legacy.go 2021-10-25 17:20:06 -04:00
Paul Banks ab5cdce760
Merge pull request #11163 from hashicorp/feature/ingress-tls-mixed
Add support for enabling connect-based ingress TLS per listener.
2021-10-25 21:36:01 +01:00
FFMMM 6433a57d3c
fix autopilot_failure_tolerance, add autopilot metrics test case (#11399)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2021-10-25 10:55:59 -07:00
FFMMM 67a624a49f
use *telemetry.MetricsPrefix as prometheus.PrometheusOpts.Name (#11290)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2021-10-21 13:33:01 -07:00
Dhia Ayachi 75f69a98a2
fix leadership transfer on leave suggestions (#11387)
* add suggestions

* set isLeader to false when leadership transfer succeed
2021-10-21 14:02:26 -04:00
Dhia Ayachi 2d1ac1f7d0
try to perform a leadership transfer when leaving (#11376)
* try to perform a leadership transfer when leaving

* add a changelog
2021-10-21 12:44:31 -04:00
Kyle Havlovitz 752a285552 Add new service-exports config entry 2021-10-20 12:24:18 -07:00
Jared Kirschner 716b05f934
Merge pull request #11293 from bisakhmondal/service_filter
expression validation of service-resolver subset filter
2021-10-20 08:57:37 -04:00
Paul Banks 4808b97d9c Rebase and rebuild golden files for Envoy version bump 2021-10-19 21:37:58 +01:00
Paul Banks ff405d35c7 Refactor `resolveListenerSDSConfig` to pass in whole config 2021-10-19 20:58:29 +01:00
Paul Banks 5c8702b182 Add support for enabling connect-based ingress TLS per listener. 2021-10-19 20:58:28 +01:00
Giulio Micheloni b549de831d Restored comment. 2021-10-16 18:05:32 +01:00
Giulio Micheloni a5a4eb9cae Separete test file and no stack trace in ret error 2021-10-16 18:02:03 +01:00
Giulio Micheloni 10814d934e Merge branch 'main' of https://github.com/hashicorp/consul into hashicorp-main 2021-10-16 16:59:32 +01:00
R.B. Boyer 55dd52cb17
acl: small OSS refactors to help ensure that auth methods with namespace rules work with partitions (#11323) 2021-10-14 15:38:05 -05:00
freddygv f76fddb28e Use stored entmeta to fill authzContext 2021-10-14 08:57:40 -06:00
freddygv bdf3e951f8 Ensure partition is handled by auto-encrypt 2021-10-14 08:32:45 -06:00
FFMMM bb228ab165
fix: only add prom autopilot gauges to servers (#11241)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2021-10-13 09:25:30 -07:00
Chris S. Kim 0a6d683c84
Update Intentions.List with partitions (#11299) 2021-10-13 10:47:12 -04:00
R.B. Boyer 3e8ece97a8
acl: fix bug in 'consul members' filtering with partitions (#11263) 2021-10-13 09:18:16 -05:00
Bisakh Mondal 929ad1e80f
add service resolver subset filter validation 2021-10-13 02:56:04 +05:30
Connor 2cd80e5f66
Merge pull request #11222 from hashicorp/clly/service-mesh-metrics
Start tracking connect service mesh usage metrics
2021-10-11 14:35:03 -05:00
Connor Kelly 2119351f77
Replace fmt.Sprintf with function 2021-10-11 12:43:38 -05:00
tarat44 baec141df3 preload json values in structs to determine defaults 2021-10-10 17:52:26 -04:00
Daniel Nephin e37b5846fd ca: split Primary/Secondary Provider
To make it more clear which methods are necessary for each scenario. This can
also prevent problems which force all DCs to use the same Vault instance, which
is currently a problem.
2021-10-10 15:48:02 -04:00
Daniel Nephin 571acb872e ca: extract primaryUpdateRootCA
This function is only run when the CAManager is a primary. Extracting this function
makes it clear which parts of UpdateConfiguration are run only in the primary and
also makes the cleanup logic simpler. Instead of both a defer and a local var we
can call the cleanup function in two places.
2021-10-10 15:26:55 -04:00
Daniel Nephin a65594d8ec ca: rename functions to use a primary or secondary prefix
This commit renames functions to use a consistent pattern for identifying the functions that
can only be called when the Manager is run as the primary or secondary.

This is a step toward eventually creating separate types and moving these methods off of CAManager.
2021-10-10 15:26:55 -04:00
Daniel Nephin 20f0efd8c1 ca: make receiver variable name consistent
Every other method uses c not ca
2021-10-10 15:26:55 -04:00
tarat44 e3a18e5203 add test cases for h2ping_use_tls default behavior 2021-10-09 17:12:52 -04:00
FFMMM 7f28301212
fix consul_autopilot_healthy metric emission (#11231)
https://github.com/hashicorp/consul/issues/10730
2021-10-08 10:31:50 -07:00
Connor Kelly 38986d6371
Rename ConfigUsageEnterprise to EnterpriseConfigEntryUsage 2021-10-08 10:53:34 -05:00
Connor Kelly 76b3c4ed3c
Rename and prefix ConfigEntry in Usage table
Rename ConfigUsage functions to ConfigEntry

prefix ConfigEntry kinds with the ConfigEntry table name to prevent
potential conflicts
2021-10-07 16:19:55 -05:00
Connor Kelly 0e39a7a333
Add connect specific prefix to Usage table
Ensure that connect Kind's are separate from ConfigEntry Kind's to
prevent miscounting
2021-10-07 16:16:23 -05:00
tarat44 bda1998175 only set default on H2PingUseTLS if H2PING is set 2021-10-06 22:13:01 -04:00
Daniel Nephin 51e498717f docs: add notice that legacy ACLs have been removed.
Add changelog

Also remove a metric that is no longer emitted that was missed in a
previous step.
2021-10-05 18:30:22 -04:00
Daniel Nephin 577f2649bf acl: remove unused translate rules endpoint
The CLI command does not use this endpoint, so we can remove it. It was missed in an
earlier pass.
2021-10-05 18:26:05 -04:00
Connor Kelly f9ba7c39b5
Add changelog, website and metric docs
Add changelog to document what changed.
Add entry to telemetry section of the website to document what changed
Add docs to the usagemetric endpoint to help document the metrics in code
2021-10-05 13:34:24 -05:00
Joshua Montgomery 5446009299
Fixing SOA record to use alt domain when alt domain in use (#10431) 2021-10-05 10:47:27 -04:00
tarat44 35faff55f8 fix test 2021-10-05 00:48:09 -04:00
tarat44 1c1405552a fix formatting 2021-10-05 00:15:04 -04:00
tarat44 e46b41d04d fix formatting 2021-10-05 00:12:23 -04:00
tarat44 f8b47cdfcd change config option to H2PingUseTLS 2021-10-05 00:12:21 -04:00
tarat44 ed4ca3db49 add support for h2c in h2 ping health checks 2021-10-04 22:51:08 -04:00
Daniel Nephin e03b7e4c68
Merge pull request #11182 from hashicorp/dnephin/acl-legacy-remove-upgrade
acl: remove upgrade from legacy, start in non-legacy mode
2021-10-04 17:25:39 -04:00
Evan Culver e47c5c5ceb
Merge pull request #11118 from hashicorp/eculver/remove-envoy-1.15
Remove support for Envoy 1.15
2021-10-04 23:14:24 +02:00
Evan Culver d279c60010
Merge pull request #11115 from hashicorp/eculver/envoy-1.19.1
Add support for Envoy 1.19.1
2021-10-04 23:13:26 +02:00
Daniel Nephin b9f0014d70 acl: remove updateEnterpriseSerfTags
The only remaining caller is a test helper, and the tests don't use the enterprise gossip
pools.
2021-10-04 17:01:51 -04:00
Daniel Nephin 5ac360b22d
Merge pull request #11126 from hashicorp/dnephin/acl-legacy-remove-resolve-and-get-policy
acl: remove ACL.GetPolicy RPC endpoint and ACLResolver.resolveTokenLegacy
2021-10-04 16:29:51 -04:00
Connor Kelly ed5693b537
Add metrics to count the number of service-mesh config entries 2021-10-04 14:50:17 -05:00
Connor Kelly 9c487389cf
Add metrics to count connect native service mesh instances
This will add the counts of the service mesh instances tagged by
whether or not it is connect native
2021-10-04 14:37:05 -05:00
Connor Kelly 8000ea45ca
Add metrics to count service mesh Kind instance counts
This will add the counts of service mesh instances tagged by the
different ServiceKind's.
2021-10-04 14:36:59 -05:00
Daniel Nephin b6435259c3 acl: fix test failures caused by remocving legacy ACLs
This commit two test failures:

1. Remove check for "in legacy ACL mode", the actual upgrade will be removed in a following commit.
2. Remove the early WaitForLeader in dc2, because with it the test was
   failing with ACL not found.
2021-10-01 18:03:10 -04:00
Evan Culver e74ce0fb2e
Add 1.15 versions to too old list 2021-10-01 11:28:26 -07:00
Chris S. Kim 3c8ca0dbd2
agent: Reject partitions in legacy intention endpoints (#11181) 2021-10-01 13:18:57 -04:00
Chris S. Kim bf94949d48
Support partitions in parseIntentionStringComponent (#11202) 2021-10-01 12:36:12 -04:00
Dhia Ayachi 8bd52995d1
fix token list by auth method (#11196)
* add tests to OIDC authmethod and fix entMeta when retrieving auth-methods

* fix oss compilation error
2021-10-01 12:00:43 -04:00
Evan Culver 4cdcaf3658
Merge branch 'eculver/envoy-1.19.1' into eculver/remove-envoy-1.15 2021-09-30 11:32:28 -07:00
Evan Culver 7b157bba4e
regenerate more envoy golden files 2021-09-30 10:57:47 -07:00
Daniel Nephin ec935a2486 acl: call stop for the upgrade goroutine when done
TestAgentLeaks_Server was reporting a goroutine leak without this. Not sure if it would actually
be a leak in production or if this is due to the test setup, but seems easy enough to call it
this way until we remove legacyACLTokenUpgrade.
2021-09-29 17:36:43 -04:00
Daniel Nephin 0c077d0527 acl: only run startACLUpgrade once
Since legacy ACL tokens can no longer be created we only need to run this upgrade a single
time when leadership is estalbished.
2021-09-29 16:22:01 -04:00
Daniel Nephin f21097beda acl: remove reading of serf acl tags
We no long need to read the acl serf tag, because servers are always either ACL enabled or
ACL disabled.

We continue to write the tag so that during an upgarde older servers will see the tag.
2021-09-29 15:45:11 -04:00
Daniel Nephin b866e3c4f4 acl: fix test failure
For some reason removing legacy ACL upgrade requires using an ACL token now
for this WaitForLeader.
2021-09-29 15:21:30 -04:00
Daniel Nephin ebb2388605 acl: remove legacy ACL upgrades from Server
As part of removing the legacy ACL system
2021-09-29 15:19:23 -04:00
Daniel Nephin 41a97360ca acl: fix test failures caused by remocving legacy ACLs
This commit two test failures:

1. Remove check for "in legacy ACL mode", the actual upgrade will be removed in a following commit.
2. Use the root token in WaitForLeader, because without it the test was
   failing with ACL not found.
2021-09-29 15:15:50 -04:00
Daniel Nephin b73b68d696 acl: remove ACL.GetPolicy endpoint and resolve legacy acls
And all code that was no longer used once those two were removed.
2021-09-29 14:33:19 -04:00
Daniel Nephin b8da06a34d acl: remove ACL upgrading from Clients
As part of removing the legacy ACL system ACL upgrading and the flag for
legacy ACLs is removed from Clients.

This commit also removes the 'acls' serf tag from client nodes. The tag is only ever read
from server nodes.

This commit also introduces a constant for the acl serf tag, to make it easier to track where
it is used.
2021-09-29 14:02:38 -04:00
Daniel Nephin 33a5448604
Merge pull request #11136 from hashicorp/dnephin/acl-resolver-fix-default-authz
acl: fix default Authorizer for down_policy extend-cache/async-cache
2021-09-29 13:45:12 -04:00
Daniel Nephin afb1dd5827
Merge pull request #11110 from hashicorp/dnephin/acl-legacy-remove-initialize
acl: remove initializeLegacyACL and the rest of the legacy FSM commands
2021-09-29 13:44:30 -04:00
Daniel Nephin a9ac148c92
Merge pull request #10999 from hashicorp/dnephin/revert-config-xds-port
Revert config xds_port
2021-09-29 13:39:15 -04:00
Daniel Nephin bd28d23b55 command/envoy: stop using the DebugConfig from Self endpoint
The DebugConfig in the self endpoint can change at any time. It's not a stable API.

This commit adds the XDSPort to a stable part of the XDS api, and changes the envoy command to read
this new field.

It includes support for the old API as well, in case a newer CLI is used with an older API, and
adds a test for both cases.
2021-09-29 13:21:28 -04:00
Daniel Nephin 2995ac61f2 acl: remove the last of the legacy FSM
Replace it with an implementation that returns an error, and rename some symbols
to use a Deprecated suffix to make it clear.

Also remove the ACLRequest struct, which is no longer referenced.
2021-09-29 12:42:23 -04:00
Daniel Nephin a8358f7575 acl: remove bootstrap-init FSM operation 2021-09-29 12:42:23 -04:00
Daniel Nephin ea2e0ad2ec acl: remove initializeLegacyACL from leader init 2021-09-29 12:42:23 -04:00
Daniel Nephin 4e36442583 acl: remove ACLDelete FSM command, and state store function
These are no longer used now that ACL.Apply has been removed.
2021-09-29 12:42:23 -04:00
Daniel Nephin 7e37c9a765 acl: remove legacy field to ACLBoostrap 2021-09-29 12:42:23 -04:00
Daniel Nephin 402d3792b6 Revert "Merge pull request #10588 from hashicorp/dnephin/config-fix-ports-grpc"
This reverts commit 74fb650b6b966588f8faeec26935a858af2b8bb5, reversing
changes made to 58bd8173364effb98b9fd9f9b98d31dd887a9bac.
2021-09-29 12:28:41 -04:00
Daniel Nephin d4c48a3f23
Merge pull request #11101 from hashicorp/dnephin/acl-legacy-remove-rpc-2
acl: remove legacy ACL.Apply RPC
2021-09-29 12:23:55 -04:00
Daniel Nephin 69a83aefcf
Merge pull request #11177 from hashicorp/dnephin/remove-entmeta-methods
structs: remove EnterpriseMeta helper methods
2021-09-29 12:08:07 -04:00
Daniel Nephin acb62aa896
Merge pull request #10986 from hashicorp/dnephin/acl-legacy-remove-rpc
acl: remove legacy ACL RPC - part 1
2021-09-29 12:04:09 -04:00
Daniel Nephin 1bc07c5166 structs: rename the last helper method.
This one gets used a bunch, but we can rename it to make the behaviour more obvious.
2021-09-29 11:48:38 -04:00
Daniel Nephin 93b3e110b6 structs: remove another helper
We already have a helper funtion.
2021-09-29 11:48:03 -04:00
Daniel Nephin 17652227f6 structs: remove two methods that were only used once each.
These methods only called a single function. Wrappers like this end up making code harder to read
because it adds extra ways of doing things.

We already have many helper functions for constructing these types, we don't need additional methods.
2021-09-29 11:47:03 -04:00
Daniel Nephin a0e08086f7
Merge pull request #10988 from hashicorp/dnephin/acl-legacy-remove-config
acl: isolate deprecated config and warn when they are used
2021-09-29 11:40:14 -04:00
Daniel Nephin 3f4f7d2f3f
Merge pull request #9456 from hashicorp/dnephin/config-deprecation
config: Use DeprecatedConfig struct for deprecated config fields
2021-09-29 11:37:40 -04:00
Evan Culver cb5ef13fde
Merge remote-tracking branch 'origin/eculver/remove-envoy-1.15' into eculver/remove-envoy-1.15 2021-09-28 16:06:36 -07:00
Evan Culver eaa9394cb2
Fix typo
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2021-09-29 01:05:45 +02:00
Evan Culver 64f94b10ce
Merge branch 'eculver/envoy-1.19.1' into eculver/remove-envoy-1.15 2021-09-28 15:59:43 -07:00
Evan Culver 807871224a
Merge branch 'main' into eculver/envoy-1.19.1 2021-09-28 15:58:20 -07:00
Chris S. Kim 3f79aaf509
Cleanup unnecessary normalizing method (#11169) 2021-09-28 15:31:12 -04:00
Daniel Nephin 4ed9476a61
Merge pull request #11084 from krastin/krastin-autopilot-loggingtypo
Fix a tiny typo in logging in autopilot.go
2021-09-28 15:11:11 -04:00
Evan Culver e2363c13ff
Merge branch 'main' into eculver/envoy-1.19.1 2021-09-28 11:54:33 -07:00
Chris S. Kim 90fe20c3a2
agent: Clean up unused built-in proxy config (#11165) 2021-09-28 11:29:10 -04:00
Daniel Nephin 30fe14eed3 acl: fix default authorizer for down_policy
This was causing a nil panic because a nil authorizer is no longer valid after the cleanup done
in https://github.com/hashicorp/consul/pull/10632.
2021-09-23 18:12:22 -04:00
Daniel Nephin a6a7069ecf Remove t.Parallel from TestACLResolver_DownPolicy
These tests run in under 10ms, t.Parallel does nothing but slow them down and
make failures harder to debug when one panics.
2021-09-23 18:12:22 -04:00
Dhia Ayachi 4505cb2920
Refactor table index acl phase 2 (#11133)
* extract common methods from oss and ent

* remove unreachable code

* add missing normalize for binding rules

* fix oss to use Query
2021-09-23 15:26:09 -04:00
Daniel Nephin cc46fcc53e config: Move ACLEnableKeyListPolicy to DeprecatedConfig 2021-09-23 15:15:00 -04:00
Daniel Nephin 107c24a68a config: move acl_ttl to DeprecatedConfig 2021-09-23 15:14:59 -04:00
Daniel Nephin 5eb2bebdf8 config: move acl_{default,down}_policy to DeprecatedConfig 2021-09-23 15:14:59 -04:00
Daniel Nephin 408eb0e08e config: Deprecate EnableACLReplication
replaced by ACL.TokenReplication
2021-09-23 15:14:59 -04:00
Daniel Nephin d54db5917f config: move ACL master token and replication to DeprecatedConfig 2021-09-23 15:14:59 -04:00
Paul Banks f8412cf5fa
Merge pull request #10903 from hashicorp/feature/ingress-sds
Add Support to for providing TLS certificates for Ingress listeners from an SDS source
2021-09-23 16:19:05 +01:00
Dhia Ayachi ebe333b947
Refactor table index (#11131)
* convert tableIndex to use the new pattern

* make `indexFromString` available for oss as well

* refactor `indexUpdateMaxTxn`
2021-09-23 11:06:23 -04:00
Paul Banks d57931124f Final readability tweaks from review 2021-09-23 10:17:12 +01:00
Paul Banks 66c625a64d Fix subtle loop bug and add test 2021-09-23 10:13:41 +01:00
Paul Banks 7198d0bd80 Refactor SDS validation to make it more contained and readable 2021-09-23 10:13:19 +01:00
Paul Banks fe4f69613c Refactor Ingress-specific lister code to separate file 2021-09-23 10:13:19 +01:00
Paul Banks f4f0793a10 Minor PR typo and cleanup fixes 2021-09-23 10:13:19 +01:00
Paul Banks 4cc1ccf892 Revert abandonned changes to proxycfg for Ent test consistency 2021-09-23 10:13:19 +01:00
Paul Banks d812a0edc7 Fix merge conflict in xds tests 2021-09-23 10:12:37 +01:00
Paul Banks a24efd20fc Fix some more Enterprise Normalization issues affecting tests 2021-09-23 10:12:37 +01:00
Paul Banks 15969327c0 Remove unused argument to fix lint error 2021-09-23 10:09:11 +01:00
Paul Banks 9422e4ebc7 Handle namespaces in route names correctly; add tests for enterprise 2021-09-23 10:09:11 +01:00
Paul Banks 9d576a08dc Update xDS routes to support ingress services with different TLS config 2021-09-23 10:08:02 +01:00
Paul Banks 8a4254a894 Update xDS Listeners with SDS support 2021-09-23 10:08:02 +01:00
Paul Banks 8548e15f1b Update proxycfg to hold more ingress config state 2021-09-23 10:08:02 +01:00
Paul Banks 0e410a1b1f Add ingress-gateway config for SDS 2021-09-23 10:08:02 +01:00
Daniel Nephin 3e6dc2a843 acl: remove ACL.Apply
As part of removing the legacy ACL system.
2021-09-22 18:28:08 -04:00
Daniel Nephin 2ce64e2837 acl: made acl rules in tests slightly more specific
When converting these tests from the legacy ACL system to the new RPC endpoints I
initially changed most things to use _prefix rules, because that was equivalent to
the old legacy rules.

This commit modifies a few of those rules to be a bit more specific by replacing the _prefix
rule with a non-prefix one where possible.
2021-09-22 18:24:56 -04:00
Mark Anderson c87d57bfeb
partitions/authmethod-index work from enterprise (#11056)
* partitions/authmethod-index work from enterprise

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-09-22 13:19:20 -07:00
Chris S. Kim d222f170a7
connect: Allow upstream listener escape hatch for prepared queries (#11109) 2021-09-22 15:27:10 -04:00
Evan Culver 88a899d06a
connect: remove support for Envoy 1.15 2021-09-22 11:48:50 -07:00
R.B. Boyer ba13416b57
grpc: strip local ACL tokens from RPCs during forwarding if crossing datacenters (#11099)
Fixes #11086
2021-09-22 13:14:26 -05:00
Daniel Nephin 66453d2de9 config: Move two more fields to DeprecatedConfig
And add a test for deprecated config fields.
2021-09-22 13:23:03 -04:00
Daniel Nephin 23f070e0a1 config: Introduce DeprecatedConfig
This struct allows us to move all the deprecated config options off of
the main config struct, and keeps all the deprecation logic in a single
place, instead of spread across 3+ places.
2021-09-22 13:22:16 -04:00
Evan Culver 4d222cfcd0
add 1.19.x versions to test config 2021-09-22 09:30:45 -07:00
Connor bc04a155fb
Merge pull request #11090 from hashicorp/clly/kv-usage-metrics
Add KVUsage to consul state usage metrics
2021-09-22 11:26:56 -05:00
Connor Kelly bfe6b64ca7
Strip out go 1.17 bits 2021-09-22 11:04:48 -05:00
Matt Keeler 7c1ef8f515 Add a mock Agent delegate to ease/improve some types of testing 2021-09-22 10:23:01 -04:00
hc-github-team-consul-core 320b20c708 auto-updated agent/uiserver/bindata_assetfs.go from commit 9c0233cf5 2021-09-22 13:05:38 +00:00
hc-github-team-consul-core 949416c071 auto-updated agent/uiserver/bindata_assetfs.go from commit cfbd1bb84 2021-09-22 09:26:14 +00:00
Daniel Nephin b40bdc9e98 acl: remove remaining tests that use ACL.Apply
In preparation for removing ACL.Apply.

Tests for ACL.Apply, ACL.GetPolicy, and ACL upgrades were removed
because all 3 of those will be removed shortly.

The forth test appears to be for the ACLResolver cache, so the test was moved to the correct
test file, and the name was updated to make it obvious what is being tested.
2021-09-21 19:35:26 -04:00
Evan Culver 69f4cc7532
regenerate envoy golden files 2021-09-21 16:21:00 -07:00
Evan Culver b104b7719c
add envoy 1.19.1 2021-09-21 15:39:36 -07:00
Daniel Nephin ab91d254a3 fsm: restore the legacy commands
and emit a helpful error message.
2021-09-21 18:35:12 -04:00
Daniel Nephin 0180dd67ff Convert tests to the new ACL system
In preparation for removing ACL.Apply
2021-09-21 18:35:12 -04:00
Daniel Nephin b639f47e3c config: use the new ACL system in tests
In preparation for removing ACL.Apply
2021-09-21 17:57:29 -04:00
Daniel Nephin 2702aecc27 catalog: use the new ACL system in tests
In preparation for removing ACL.Apply
2021-09-21 17:57:29 -04:00
Daniel Nephin b6218b75d9 Update 4 non-acl tests that used the legacy ACL.Apply
These tests don't really care about the endpoint, they just need some way to create an ACL token.
2021-09-21 17:57:29 -04:00
Daniel Nephin ad9748adc3 acl: remove two commented out tests for legacy ACL replication
They were commented out in 2018.
2021-09-21 17:57:29 -04:00
Daniel Nephin 5a31a2e167 acl: replace legacy Get and List RPCs with an error impl
These endpoints are being removed as part of the legacy ACL system.
2021-09-21 17:57:29 -04:00
Daniel Nephin 26f3380688 acl: remove a couple legacy ACL operation constants
structs.ACLForceSet was deprecated 4 years ago, it should be safe to remove now.
ACLBootstrapNow was removed in a recent commit. While it is technically possible that a cluster with mixed version
could still attempt a legacy boostrap, we documented that the legacy system was deprecated in 1.4, so no
clusters that are being upgraded should be attempting a legacy boostrap.
2021-09-21 17:57:29 -04:00
Daniel Nephin af8c10afc4 acl: Remove unused ACLPolicyIDType 2021-09-21 17:57:29 -04:00
Daniel Nephin 5493ff06cc
Merge pull request #10985 from hashicorp/dnephin/acl-legacy-remove-replication
acl: remove legacy ACL replication
2021-09-21 17:56:54 -04:00
Connor 64852cd3e5
Apply suggestions from code review
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2021-09-21 10:52:46 -05:00
R.B. Boyer 2773bd94d7
xds: fix representation of incremental xDS subscriptions (#10987)
Fixes #10563

The `resourceVersion` map was doing two jobs prior to this PR. The first job was
to track what version of every resource we know envoy currently has. The
second was to track subscriptions to those resources (by way of the empty
string for a version). This mostly works out fine, but occasionally leads to
consul removing a resource and accidentally (effectively) unsubscribing at the
same time.

The fix separates these two jobs. When all of the resources for a subscription
are removed we continue to track the subscription until envoy explicitly
unsubscribes
2021-09-21 09:58:56 -05:00
Connor Kelly 973b7b5c78
Fix test 2021-09-20 13:44:43 -05:00
Connor Kelly 698fc291a9
Add KVUsage to consul state usage metrics
This change will add the number of entries in the consul KV store to the
already existing usage metrics.
2021-09-20 12:41:54 -05:00
R.B. Boyer 55b36dd056
xds: ensure the active streams counters are 64 bit aligned on 32 bit systems (#11085) 2021-09-20 11:07:11 -05:00
Krastin Krastev ba13dbf24c
Update autopilot.go
Fixing a minuscule typo in logging
2021-09-20 14:40:58 +02:00
Freddy f1b2ef30d1
Merge pull request #11071 from hashicorp/partitions/ixn-decisions 2021-09-16 15:18:23 -06:00
freddygv 661f520841 Fixup proxycfg tproxy case 2021-09-16 15:05:28 -06:00
freddygv 12eec88dff Remove ent checks from oss test 2021-09-16 14:53:28 -06:00
R.B. Boyer 7fa8f19077
acl: ensure the global management policy grants all necessary partition privileges (#11072) 2021-09-16 15:53:10 -05:00
freddygv cf56be7d8d Ensure partition is defaulted in authz 2021-09-16 14:39:01 -06:00
freddygv b5a8935bb8 Default the partition in ixn check 2021-09-16 14:39:01 -06:00
freddygv caafc1905e Fixup test 2021-09-16 14:39:01 -06:00
freddygv 8a9bf3748c Account for partitions in ixn match/decision 2021-09-16 14:39:01 -06:00
Jeff Widman a8f396c55f
Bump `go-discover` to fix broken dep tree (#10898) 2021-09-16 15:31:22 -04:00
hc-github-team-consul-core 5a6f9e38b1 auto-updated agent/uiserver/bindata_assetfs.go from commit 1d9d3349c 2021-09-16 17:31:08 +00:00
R.B. Boyer 4e7b6888e3
acl: fix intention:*:write checks (#11061)
This is a partial revert of #10793
2021-09-16 11:08:45 -05:00
Freddy 88627700d0
Merge pull request #11051 from hashicorp/partitions/fixes 2021-09-16 09:29:00 -06:00
Freddy 494764ee2d
acl: small resolver changes to account for partitions (#11052)
Also refactoring the enterprise side of a test to make it easier to reason about.
2021-09-16 09:17:02 -05:00
freddygv 7927a97c2f Fixup manager tests 2021-09-15 17:24:05 -06:00
freddygv dc549eca30 Default partition in match endpoint 2021-09-15 17:23:52 -06:00
freddygv 0cdcbbb4c9 Pass partition to intention match query 2021-09-15 17:23:52 -06:00
freddygv a57c52ca32 Ensure partition is used for SAN validation 2021-09-15 17:23:48 -06:00
Mark Anderson 08b222cfc3
ACL Binding Rules table partitioning (#11044)
* ACL Binding Rules table partitioning

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-09-15 13:26:08 -07:00
hc-github-team-consul-core 23e3f865b0 auto-updated agent/uiserver/bindata_assetfs.go from commit fc14a412f 2021-09-15 18:55:29 +00:00
hc-github-team-consul-core abe0195257 auto-updated agent/uiserver/bindata_assetfs.go from commit b16a6fa03 2021-09-15 17:14:42 +00:00
Dhia Ayachi 25ea1a9276
use const instead of literals for `tableIndex` (#11039) 2021-09-15 10:24:04 -04:00
Mark Anderson ffe3806aaf
Refactor `indexAuthMethod` in `tableACLBindingRules` (#11029)
* Port consul-enterprise #1123 to OSS

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Fixup missing query field

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* change to re-trigger ci system

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-09-15 09:34:19 -04:00
Freddy 8804577de1
Merge pull request #11024 from hashicorp/partitions/rbac 2021-09-14 11:18:19 -06:00
Freddy 27f40ccf51
Update error texts (#11022)
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-09-14 11:08:06 -06:00
freddygv f209408918 Update spiffe ID patterns used for RBAC 2021-09-14 11:00:03 -06:00
freddygv 0e30151eaa Expand testing of simplifyNotSourceSlice for partitions 2021-09-14 10:55:15 -06:00
freddygv a65da57a3d Expand testing of removeSameSourceIntentions for partitions 2021-09-14 10:55:09 -06:00
freddygv e9d78a20c7 Account for partition when matching src intentions 2021-09-14 10:55:02 -06:00
Daniel Nephin 44d91ea56f
Add failures_before_warning to checks (#10969)
Signed-off-by: Jakub Sokołowski <jakub@status.im>

* agent: add failures_before_warning setting

The new setting allows users to specify the number of check failures
that have to happen before a service status us updated to be `warning`.
This allows for more visibility for detected issues without creating
alerts and pinging administrators. Unlike the previous behavior, which
caused the service status to not update until it reached the configured
`failures_before_critical` setting, now Consul updates the Web UI view
with the `warning` state and the output of the service check when
`failures_before_warning` is breached.

The default value of `FailuresBeforeWarning` is the same as the value of
`FailuresBeforeCritical`, which allows for retaining the previous default
behavior of not triggering a warning.

When `FailuresBeforeWarning` is set to a value higher than that of
`FailuresBeforeCritical it has no effect as `FailuresBeforeCritical`
takes precedence.

Resolves: https://github.com/hashicorp/consul/issues/10680

Signed-off-by: Jakub Sokołowski <jakub@status.im>

Co-authored-by: Jakub Sokołowski <jakub@status.im>
2021-09-14 12:47:52 -04:00
Dhia Ayachi 4992218676
convert expiration indexed in ACLToken table to use `indexerSingle` (#11018)
* move intFromBool to be available for oss

* add expiry indexes

* remove dead code: `TokenExpirationIndex`

* fix remove indexer `TokenExpirationIndex`

* fix rebase issue
2021-09-13 14:37:16 -04:00
Dhia Ayachi 1f23bdf388
add locality indexer partitioning (#11016)
* convert `Roles` index to use `indexerSingle`

* split authmethod write indexer to oss and ent

* add index locality

* add locality unit tests

* move intFromBool to be available for oss

* use Bool func

* refactor `aclTokenList` to merge func
2021-09-13 11:53:00 -04:00
Dhia Ayachi 3638825db8
convert `indexAuthMethod` index to use `indexerSingle` (#11014)
* convert `Roles` index to use `indexerSingle`

* fix oss build

* split authmethod write indexer to oss and ent

* add auth method unit tests
2021-09-10 16:56:56 -04:00
Paul Banks ecbe8f0656 Include namespace and partition in error messages when validating ingress header manip 2021-09-10 21:11:00 +01:00
Paul Banks e6642c6dae Refactor HTTPHeaderModifiers.MergeDefaults based on feedback 2021-09-10 21:11:00 +01:00
Paul Banks a1acb7ec3b Fix enterprise test failures caused by differences in normalizing EnterpriseMeta 2021-09-10 21:11:00 +01:00
Paul Banks 3484d77b18 Fix enterprise discovery chain tests; Fix multi-level split merging 2021-09-10 21:11:00 +01:00
Paul Banks e0ad412f1d Remove unnecessary check 2021-09-10 21:09:24 +01:00
Paul Banks 5c6d27555b Fix discovery chain test fixtures 2021-09-10 21:09:24 +01:00
Paul Banks bc1c86df96 Integration tests for all new header manip features 2021-09-10 21:09:24 +01:00
Paul Banks 1dd1683ed9 Header manip for split legs plumbing 2021-09-10 21:09:24 +01:00
Paul Banks f70f7b2389 Header manip for service-router plumbed through 2021-09-10 21:09:24 +01:00
Paul Banks fc2ed4cdf4 Ingress gateway header manip plumbing 2021-09-10 21:09:24 +01:00
Paul Banks 2db02cdba2 Add HTTP header manip for router and splitter entries 2021-09-10 21:09:24 +01:00
Paul Banks 7ac9b46f08 Header manip and validation added for ingress-gateway entries 2021-09-10 21:09:24 +01:00
Dhia Ayachi 82b30f8020
convert `Roles` index to use `indexerMulti` (#11013)
* convert `Roles` index to use `indexerMulti`

* add role test in oss

* fix oss to use the right index func

* preallocate slice
2021-09-10 16:04:33 -04:00
Dhia Ayachi 569e18d002
convert indexPolicies in ACLTokens table to the new index (#11011) 2021-09-10 14:57:37 -04:00
Dhia Ayachi 0d0edeec27
convert indexSecret to the new index (#11007) 2021-09-10 09:10:11 -04:00
Dhia Ayachi f0cbe25ca6
convert indexAccessor to the new index (#11002) 2021-09-09 16:28:04 -04:00
Hans Hasselberg 24c6ce0be0
tls: consider presented intermediates during server connection tls handshake. (#10964)
* use intermediates when verifying

* extract connection state

* remove useless import

* add changelog entry

* golint

* better error

* wording

* collect errors

* use SAN.DNSName instead of CommonName

* Add test for unknown intermediate

* improve changelog entry
2021-09-09 21:48:54 +02:00
Chris S. Kim 3fb797382b
Sync enterprise changes to oss (#10994)
This commit updates OSS with files for enterprise-specific admin partitions feature work
2021-09-08 11:59:30 -04:00
Kyle Havlovitz a7b5a5d1b4
Merge pull request #10984 from hashicorp/mesh-resource
acl: adding a new mesh resource
2021-09-07 15:06:20 -07:00
Dhia Ayachi 96d7842118
partition dicovery chains (#10983)
* partition dicovery chains

* fix default partition for OSS
2021-09-07 16:29:32 -04:00
Daniel Nephin 4d5a39e622 acl: remove ACL.IsSame
The only caller of this method was removed in a recent commit along with replication.
2021-09-03 12:59:12 -04:00
Daniel Nephin 4dd5bb8e3b acl: remove legacy ACL replication 2021-09-03 12:42:06 -04:00
R.B. Boyer 4206f585f0 acl: adding a new mesh resource 2021-09-03 09:12:03 -04:00
Dhia Ayachi 72391dc99c
try to infer command partition from node partition (#10981) 2021-09-03 08:37:23 -04:00
Dhia Ayachi eb19271fd7
add partition to SNI when partition is non default (#10917) 2021-09-01 10:35:39 -04:00
Freddy 11672defaf
connect: update envoy supported versions to latest patch release
(#10961)

Relevant advisory: 
https://github.com/envoyproxy/envoy/security/advisories/GHSA-6g4j-5vrw-2m8h
2021-08-31 10:39:18 -06:00
Evan Culver 93f94ac24f
rpc: authorize raft requests (#10925) 2021-08-26 15:04:32 -07:00
hc-github-team-consul-core a758581ab6 auto-updated agent/uiserver/bindata_assetfs.go from commit eeeb91bea 2021-08-26 18:13:08 +00:00
Chris S. Kim 86de20c975
ent->oss test fix (#10926) 2021-08-26 14:06:49 -04:00
hc-github-team-consul-core 5c67517647 auto-updated agent/uiserver/bindata_assetfs.go from commit a907e1d87 2021-08-26 18:02:18 +00:00
hc-github-team-consul-core d9022ce788 auto-updated agent/uiserver/bindata_assetfs.go from commit a0b0ed2bc 2021-08-26 16:06:09 +00:00
Chris S. Kim efbdf7e117
api: expose upstream routing configurations in topology view (#10811)
Some users are defining routing configurations that do not have associated services. This commit surfaces these configs in the topology visualization. Also fixes a minor internal bug with non-transparent proxy upstream/downstream references.
2021-08-25 15:20:32 -04:00
R.B. Boyer 6b5a58de50
acl: some acl authz refactors for nodes (#10909) 2021-08-25 13:43:11 -05:00
hc-github-team-consul-core c95ec5007d auto-updated agent/uiserver/bindata_assetfs.go from commit a777b0a9b 2021-08-25 13:46:51 +00:00
hc-github-team-consul-core 9b2dd8b155 auto-updated agent/uiserver/bindata_assetfs.go from commit 8192dde48 2021-08-25 11:39:14 +00:00
R.B. Boyer a84f5fa25d
grpc: ensure that streaming gRPC requests work over mesh gateway based wan federation (#10838)
Fixes #10796
2021-08-24 16:28:44 -05:00
hc-github-team-consul-core 6b574abc89 auto-updated agent/uiserver/bindata_assetfs.go from commit 05a28c311 2021-08-24 16:04:24 +00:00
Giulio Micheloni 387f6f717b Fix merge conflicts 2021-08-22 19:35:08 +01:00
Giulio Micheloni 10b03c3f4e
Merge branch 'main' into serve-panic-recovery 2021-08-22 20:31:11 +02:00
Giulio Micheloni 465e9fecda grpc, xds: recovery middleware to return and log error in case of panic
1) xds and grpc servers:
   1.1) to use recovery middleware with callback that prints stack trace to log
   1.2) callback turn the panic into a core.Internal error
2) added unit test for grpc server
2021-08-22 19:06:26 +01:00
freddygv 79e181be73 Avoid passing zero value into variadic 2021-08-20 17:40:33 -06:00
freddygv ed79e38a36 Update comment for test function 2021-08-20 17:40:33 -06:00
freddygv b1050e4229 Update prepared query cluster SAN validation
Previously SAN validation for prepared queries was broken because we
validated against the name, namespace, and datacenter for prepared
queries.

However, prepared queries can target:

- Services with a name that isn't their own
- Services in multiple datacenters

This means that the SpiffeID to validate needs to be based on the
prepared query endpoints, and not the prepared query's upstream
definition.

This commit updates prepared query clusters to account for that.
2021-08-20 17:40:33 -06:00
freddygv 1f192eb7d9 Fixup proxy config test fixtures
- The TestNodeService helper created services with the fixed name "web",
and now that name is overridable.

- The discovery chain snapshot didn't have prepared query endpoints so
the endpoints tests were missing data for prepared queries
2021-08-20 17:38:57 -06:00
R.B. Boyer 60591d55f7
agent: add partition labels to catalog API metrics where appropriate (#10890) 2021-08-20 15:09:39 -05:00
R.B. Boyer b6be94e7fa
fixing various bits of enterprise meta plumbing to be more correct (#10889) 2021-08-20 14:34:23 -05:00
Dhia Ayachi f766b6dff7
oss portion of ent #1069 (#10883) 2021-08-20 12:57:45 -04:00
R.B. Boyer d730298f59
state: partition the nodes.uuid and nodes.meta indexes as well (#10882) 2021-08-19 16:17:59 -05:00
R.B. Boyer 61f1c01b83
agent: ensure that most agent behavior correctly respects partition configuration (#10880) 2021-08-19 15:09:42 -05:00
Daniel Nephin 4a0ae4048d
Merge pull request #10849 from hashicorp/dnephin/contrib-doc-xds-auth
xds: document how authorization works
2021-08-18 13:25:16 -04:00
R.B. Boyer e565409c6a
state: partition the usage metrics subsystem (#10867) 2021-08-18 09:27:15 -05:00
Daniel Nephin 9df2464c7c xds: document how authorization works 2021-08-17 19:26:34 -04:00
R.B. Boyer 1cef3c99c2
state: adjust streaming event generation to account for partitioned nodes (#10860)
Also re-enabled some tests that had to be disabled in the prior PR.
2021-08-17 16:49:26 -05:00
R.B. Boyer e50e13d2ab
state: partition nodes and coordinates in the state store (#10859)
Additionally:

- partitioned the catalog indexes appropriately for partitioning
- removed a stray reference to a non-existent index named "node.checks"
2021-08-17 13:29:39 -05:00
Daniel Nephin 5a82859ee1 acl: small improvements to ACLResolver disable due to RPC error
Remove the error return, so that not handling is not reported as an
error by errcheck. It was returning the error passed as an arg
unmodified so there is no reason to return the same value that was
passed in.

Remove the term upstreams to remove any confusion with the term used in
service mesh.

Remove the AutoDisable field, and replace it with the TTL value, using 0
to indicate the setting is turned off.

Replace "not Before" with "After".

Add some test coverage to show the behaviour is still correct.
2021-08-17 13:34:18 -04:00
Daniel Nephin 09ae0ab94a acl: make ACLDisabledTTL a constant
This field was never user-configurable. We always overwrote the value with 120s from
NonUserSource. However, we also never copied the value from RuntimeConfig to consul.Config,
So the value in NonUserSource was always ignored, and we used the default value of 30s
set by consul.DefaultConfig.

All of this code is an unnecessary distraction because a user can not actually configure
this value.

This commit removes the fields and uses a constant value instad. Someone attempting to set
acl.disabled_ttl in their config will now get an error about an unknown field, but previously
the value was completely ignored, so the new behaviour seems more correct.

We have to keep this field in the AutoConfig response for backwards compatibility, but the value
will be ignored by the client, so it doesn't really matter what value we set.
2021-08-17 13:34:18 -04:00
Daniel Nephin a8bc964241 Fix test failures
Tests only specified one of the fields, but in production we copy the
value from a single place, so we can do the same in tests.

The AutoConfig test broke because of the problem noticed in a previous
commit. The DisabledTTL is not wired up properly so it reports 0s here.
Changed the test to use an explicit value.
2021-08-17 13:32:52 -04:00
Daniel Nephin 0d69b49f41 config: remove ACLResolver settings from RuntimeConfig 2021-08-17 13:32:52 -04:00
Daniel Nephin 75baa22e64 acl: remove ACLResolver config fields from consul.Config 2021-08-17 13:32:52 -04:00
Daniel Nephin 454f62eacc acl: replace ACLResolver.Config with its own struct
This is step toward decoupling ACLResolver from the agent/consul
package.
2021-08-17 13:32:52 -04:00
Daniel Nephin 5e5ad62679 acl: remove ACLRulesTranslateLegacyToken API endpoint 2021-08-17 13:10:02 -04:00
Daniel Nephin be0358df02 acl: remove legacy bootstrap
Return an explicit error from the RPC, and remove the flag from the HTTP API.
2021-08-17 13:10:00 -04:00
Daniel Nephin d877673268 agent: update some tests that were using legacy ACL endpoints
The tests were updated to use the new ACL endpoints now that the legacy ones have been removed.
2021-08-17 13:09:30 -04:00
Daniel Nephin 10791b007d http: update legacy ACL endpoints to return an error
Also move a test for the ACLReplicationStatus endpoint into the correct file.
2021-08-17 13:09:29 -04:00
Daniel Nephin 4f54d9708c acl: add some notes about removing legacy ACL system 2021-08-17 13:08:29 -04:00
Daniel Nephin e4c6bee7e6
Merge pull request #10792 from hashicorp/dnephin/rename-authz-vars
acl: use authz consistently as the variable name for an acl.Authorizer
2021-08-17 13:07:17 -04:00
Daniel Nephin 7f71a672f3
Merge pull request #10807 from hashicorp/dnephin/remove-acl-datacenter
config: remove ACLDatacenter
2021-08-17 13:07:09 -04:00
Daniel Nephin 608b291565 acl: use authz consistently as the variable name for an acl.Authorizer
Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r682147950

Renames all variables for acl.Authorizer to use `authz`. Previously some
places used `rule` which I believe was an old name carried over from the
legacy ACL system.

A couple places also used authorizer.

This commit also removes another couple of authorizer nil checks that
are no longer necessary.
2021-08-17 12:14:10 -04:00
hc-github-team-consul-core e1da3da0e2 auto-updated agent/uiserver/bindata_assetfs.go from commit ae9c31338 2021-08-16 16:10:17 +00:00
Kyle Havlovitz 470558708e
Merge pull request #10843 from hashicorp/partitions/rename-default
oss: Rename default partition
2021-08-12 14:45:53 -07:00
Kyle Havlovitz 98969c018a oss: Rename default partition 2021-08-12 14:31:37 -07:00
Daniel Nephin 7c865d03ac proxycfg: Lookup the agent token as a default
When no ACL token is provided with the service registration.
2021-08-12 15:51:34 -04:00
Daniel Nephin d189524e71 proxycfg: Add a test to show the bug
When a token is not provided at registration, the agent token is not being used.
2021-08-12 15:47:59 -04:00
Mike Morris 86d76cb099
deps: upgrade gogo-protobuf to v1.3.2 (#10813)
* deps: upgrade gogo-protobuf to v1.3.2

* go mod tidy using go 1.16

* proto: regen protobufs after upgrading gogo/protobuf

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-12 14:05:46 -04:00
Mark Anderson 03a3ec2b55
Fixup to support unix domain socket via command line (#10758)
Missed the need to add support for unix domain socket config via
api/command line. This is a variant of the problems described in
it is easy to drop one.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-08-12 10:05:22 -07:00
hc-github-team-consul-core f02ea91a8b auto-updated agent/uiserver/bindata_assetfs.go from commit ab6a67520 2021-08-11 17:05:51 +00:00
Giulio Micheloni 0bf124502e grpc Server: turn panic into error through middleware 2021-08-07 13:21:12 +01:00
Daniel Nephin 364ef3d052 server: remove defaulting of PrimaryDatacenter
The constructor for Server is not at all the appropriate place to be setting default
values for a config struct that was passed in.

In production this value is always set from agent/config. In tests we should set the
default in a test helper.
2021-08-06 18:45:24 -04:00
Daniel Nephin 87fb26fd65
Merge pull request #10612 from bigmikes/acl-replication-fix
acl: acl replication routine to report the last error message
2021-08-06 18:29:51 -04:00
Daniel Nephin 047abdd73c acl: remove ACLDatacenter
This field has been unnecessary for a while now. It was always set to the same value
as PrimaryDatacenter. So we can remove the duplicate field and use PrimaryDatacenter
directly.

This change was made by GoLand refactor, which did most of the work for me.
2021-08-06 18:27:00 -04:00
Giulio Micheloni 5c34a48d45 String type instead of error type and changelog. 2021-08-06 22:35:27 +01:00
Daniel Nephin 9435118179 acl: remove Server.ResolveTokenIdentityAndDefaultMeta
This method suffered from similar naming to a couple other methods on Server, and had not great
re-use (2 callers). By copying a few of the lines into one of the callers we can move the
implementation into the second caller.

Once moved, we can see that ResolveTokenAndDefaultMeta is identical in both Client and Server, and
likely should be further refactored, possibly into ACLResolver.

This change is being made to make ACL resolution easier to trace.
2021-08-05 15:20:13 -04:00
Daniel Nephin 25f40de163 acl: remove Server.ResolveTokenToIdentityAndAuthorizer
This method was an alias for ACLResolver.ResolveTokenToIdentityAndAuthorizer. By removing the
method that does nothing the code becomes easier to trace.
2021-08-05 15:20:13 -04:00
Daniel Nephin 695963acb7 acl: recouple acl filtering from ACLResolver
ACL filtering only needs an authorizer and a logger. We can decouple filtering from
the ACLResolver by passing in the necessary logger.

This change is being made in preparation for moving the ACLResolver into an acl package
2021-08-05 15:20:13 -04:00
Daniel Nephin ba2f9a65d1 acl: remove unused error return
filterACLWithAuthorizer could never return an error. This change moves us a little bit
closer to being able to enable errcheck and catch problems caused by unhandled error
return values.
2021-08-05 15:20:13 -04:00
Daniel Nephin c80b9565e2 acl: rename acl.Authorizer vars to authz
For consistency
2021-08-05 15:19:47 -04:00
Daniel Nephin 37c67cb280 acl: move vet functions
These functions are moved to the one place they are called to improve code locality.

They are being moved out of agent/consul/acl.go in preparation for moving
ACLResolver to an acl package.
2021-08-05 15:19:24 -04:00
Daniel Nephin c8eedabc7c acl: move vetRegisterWithACL and vetDeregisterWithACL
These functions are used in only one place. Move the functions next to their one caller
to improve code locality.

This change is being made in preparation for moving the ACLResolver into an
acl package. The moved functions were previously in the same file as the ACLResolver.
By moving them out of that file we may be able to move the entire file
with fewer modifications.
2021-08-05 15:17:54 -04:00
Daniel Nephin b223c2bc25
Merge pull request #10770 from hashicorp/dnephin/log-cert-expiration
telemetry: add log message when certs are about to expire
2021-08-05 15:17:20 -04:00
Daniel Nephin c866f1041a
Merge pull request #10793 from hashicorp/dnephin/acl-intentions
acl: small cleanup of a couple Authorization flows
2021-08-05 15:16:49 -04:00
Dhia Ayachi 40baf98159
defer setting the state before returning to avoid stuck in `INITIALIZING` state (#10630)
* defer setting the state before returning to avoid being stuck in `INITIALIZING` state

* add changelog

* move comment with the right if statement

* ca: report state transition error from setSTate

* update comment to reflect state transition

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-05 14:51:19 -04:00
Daniel Nephin 79ab48ef81
Merge pull request #10768 from hashicorp/dnephin/agent-tls-cert-expiration-metric
telemetry: add Agent TLS Certificate expiration metric
2021-08-04 18:42:02 -04:00
Daniel Nephin 0ca9e875e2 acl: remove special handling of services in txn_endpoint
Follow up to: https://github.com/hashicorp/consul/pull/10738#discussion_r680190210

Previously we were passing an Authorizer that would always allow the
operation, then later checking the authorization using vetServiceTxnOp.

On the surface this seemed strange, but I think it was actually masking
a bug as well. Over time `servicePreApply` was changed to add additional
authorization for `service.Proxy.DestinationServiceName`, but because
we were passing a nil Authorizer, that authorization was not handled on
the txn_endpoint.

`TxnServiceOp.FillAuthzContext` has some special handling in enterprise,
so we need to make sure to continue to use that from the Txn endpoint.

This commit removes the `vetServiceTxnOp` function, and passes in the
`FillAuthzContext` function so that `servicePreApply` can be used by
both the catalog and txn endpoints. This should be much less error prone
and prevent bugs like this in the future.
2021-08-04 18:32:20 -04:00
hc-github-team-consul-core ef162f8390 auto-updated agent/uiserver/bindata_assetfs.go from commit bcd53e73a 2021-08-04 22:27:44 +00:00
Daniel Nephin f6d5a85561 acl: move check for Intention.DestinationName into Authorizer
Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r680134445

Move the check for the Intention.DestinationName into the Authorizer to remove the
need to check what kind of Authorizer is being used.

It sounds like this check is only for legacy ACLs, so is probably just a safeguard
.
2021-08-04 18:06:44 -04:00
Daniel Nephin 3dc113ada6
Merge pull request #10738 from hashicorp/dnephin/remove-authorizer-nil-checks-2
acl: remove the last of the authz == nil checks
2021-08-04 17:41:40 -04:00
Daniel Nephin 2e9aa91256
Merge pull request #10737 from hashicorp/dnephin/remove-authorizer-nil-checks
acl: remove authz == nil checks
2021-08-04 17:39:34 -04:00
Daniel Nephin 210a850353 telemetry: add log message when certs are about to expire 2021-08-04 14:18:59 -04:00
Daniel Nephin 13aa7b70d5 telemetry: fix a couple bugs in cert expiry metrics
1. do not emit the metric if Query fails
2. properly check for PrimaryUsersIntermediate, the logic was inverted

Also improve the logging by including the metric name in the log message
2021-08-04 13:51:44 -04:00
Daniel Nephin 1673b3a68c telemetry: add a metric for agent TLS cert expiry 2021-08-04 13:51:44 -04:00
Dhia Ayachi 6ed6966a1f
fix state index for `CAOpSetRootsAndConfig` op (#10675)
* fix state index for `CAOpSetRootsAndConfig` op

* add changelog

* Update changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* remove the change log as it's not needed

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-04 13:07:49 -04:00
hc-github-team-consul-core 4b2ada0dcc auto-updated agent/uiserver/bindata_assetfs.go from commit 8ad1ab9c0 2021-08-04 16:47:13 +00:00
Evan Culver 57aabe3455
checks: Add Interval and Timeout to API response (#10717) 2021-08-03 15:26:49 -07:00
Daniel Nephin 953c9bee4f acl: Remove the remaining authz == nil checks
These checks were a bit more involved. They were previously skipping some code paths
when the authorizer was nil. After looking through these it seems correct to remove the
authz == nil check, since it will never evaluate to true.
2021-07-30 14:55:35 -04:00
Daniel Nephin e4821a58ee acl: remove acl == nil checks 2021-07-30 14:28:19 -04:00
Daniel Nephin fbaeac9ecf acl: remove authz == nil checks
These case are already impossible conditions, because most of these functions already start
with a check for ACLs being disabled. So the code path being removed could never be reached.

The one other case (ConnectAuthorized) was already changed in a previous commit. This commit
removes an impossible branch because authz == nil can never be true.
2021-07-30 13:58:35 -04:00
Daniel Nephin b6d9d0d9f7 acl: remove many instances of authz == nil 2021-07-30 13:58:35 -04:00
Daniel Nephin bbc05ae869 agent: remove unused agent methods
These methods are no longer used. Remove the methods, and update the
tests to use actual method used by production code.

Also removes the 'authz == nil' check is no longer a possible code path
now that we are returning a non-nil acl.Authorizer when ACLs are disabled.
2021-07-30 13:58:35 -04:00
Daniel Nephin 2503f27a36 acl: remove rule == nil checks 2021-07-30 13:58:35 -04:00
hc-github-team-consul-core 701d4ffef0 auto-updated agent/uiserver/bindata_assetfs.go from commit 2ee501be8 2021-07-30 17:58:27 +00:00
Daniel Nephin 475fec5670
Merge pull request #10632 from hashicorp/pairing/acl-authorizer-when-acl-disabled
acls: Update ACL authorizer to return meaningful permission when ACLs are disabled
2021-07-30 13:22:55 -04:00
Evan Culver 241b6429c3 Fix intention endpoint test 2021-07-30 12:58:45 -04:00
Daniel Nephin 9b41e7287f acl: use acl.ManangeAll when ACLs are disabled
Instead of returning nil and checking for nilness

Removes a bunch of nil checks, and fixes one test failures.
2021-07-30 12:58:24 -04:00
Blake Covarrubias f97e843c61 Add OSS changes for specifying audit log permission mode 2021-07-30 09:58:11 -07:00
Daniel Nephin f2f5aba1bf
Merge pull request #10707 from hashicorp/dnephin/streaming-setup-default-timeout
streaming: set default query timeout
2021-07-28 18:29:28 -04:00
Daniel Nephin 057e8320f9 streaming: set a default timeout
The blocking query backend sets the default value on the server side.
The streaming backend does not using blocking queries, so we must set the timeout on
the client.
2021-07-28 17:50:00 -04:00
hc-github-team-consul-core f39d36d346 auto-updated agent/uiserver/bindata_assetfs.go from commit eb5512fb7 2021-07-27 21:39:22 +00:00
Chris S. Kim 33d7d48767
sync enterprise files with oss (#10705) 2021-07-27 17:09:59 -04:00
Daniel Nephin cfc829275c http: don't log an error if the request is cancelled
Now that we have at least one endpoint that uses context for cancellation we can
encounter this scenario where the returned error is a context.Cancelled or
context.DeadlineExceeded.

If the request.Context().Err() is not nil, then we know the request itself was cancelled, so
we can log a different message at Info level, instad of the error.
2021-07-27 17:06:59 -04:00
Daniel Nephin bad2c4ef67
Merge pull request #10399 from hashicorp/dnephin/debug-stream-metrics
debug: use the new metrics stream in debug command
2021-07-27 13:23:15 -04:00
Daniel Nephin 7d24564ff0 http: add tests for AgentMetricsStream 2021-07-26 17:53:33 -04:00
Daniel Nephin cf2e25c6bb http: emit indented JSON in the metrics stream endpoint
To remove the need to decode and re-encode in the CLI
2021-07-26 17:53:33 -04:00
Daniel Nephin d716f709fd debug: use the new metrics stream in debug command 2021-07-26 17:53:32 -04:00
Freddy b136b1795a
Reset root prune interval after TestLeader_CARootPruning completes
#10645

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-07-26 15:43:40 -06:00
Chris S. Kim 6341183a84
agent: update proxy upstreams to inherit namespace from service (#10688) 2021-07-26 17:12:29 -04:00
Freddy 57ca0ed480
Log the correlation ID when blocking queries fire (#10689)
Knowing that blocking queries are firing does not provide much
information on its own. If we know the correlation IDs we can
piece together which parts of the snapshot have been populated.

Some of these responses might be empty from the blocking
query timing out. But if they're returning quickly I think we
can reasonably assume they contain data.
2021-07-23 16:36:17 -06:00
R.B. Boyer c271976445
state: refactor some node/coordinate state store functions to take an EnterpriseMeta (#10687)
Note the field is not used yet.
2021-07-23 13:42:23 -05:00
R.B. Boyer b2facb35a9
replumbing a bunch of api and agent structs for partitions (#10681) 2021-07-22 14:33:22 -05:00
R.B. Boyer 254557a1f6
sync changes to oss files made in enterprise (#10670) 2021-07-22 13:58:08 -05:00
R.B. Boyer 62ac98b564
agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669) 2021-07-22 13:20:45 -05:00
Dhia Ayachi b725605fe4
config raft apply silent error (#10657)
* return an error when the index is not valid

* check response as bool when applying `CAOpSetConfig`

* remove check for bool response

* fix error message and add check to test

* fix comment

* add changelog
2021-07-22 10:32:27 -04:00