Commit Graph

3705 Commits

Author SHA1 Message Date
Daniel Nephin 7f71a672f3
Merge pull request #10807 from hashicorp/dnephin/remove-acl-datacenter
config: remove ACLDatacenter
2021-08-17 13:07:09 -04:00
Daniel Nephin 608b291565 acl: use authz consistently as the variable name for an acl.Authorizer
Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r682147950

Renames all variables for acl.Authorizer to use `authz`. Previously some
places used `rule` which I believe was an old name carried over from the
legacy ACL system.

A couple places also used authorizer.

This commit also removes another couple of authorizer nil checks that
are no longer necessary.
2021-08-17 12:14:10 -04:00
hc-github-team-consul-core e1da3da0e2 auto-updated agent/uiserver/bindata_assetfs.go from commit ae9c31338 2021-08-16 16:10:17 +00:00
Kyle Havlovitz 470558708e
Merge pull request #10843 from hashicorp/partitions/rename-default
oss: Rename default partition
2021-08-12 14:45:53 -07:00
Kyle Havlovitz 98969c018a oss: Rename default partition 2021-08-12 14:31:37 -07:00
Daniel Nephin 7c865d03ac proxycfg: Lookup the agent token as a default
When no ACL token is provided with the service registration.
2021-08-12 15:51:34 -04:00
Daniel Nephin d189524e71 proxycfg: Add a test to show the bug
When a token is not provided at registration, the agent token is not being used.
2021-08-12 15:47:59 -04:00
Mike Morris 86d76cb099
deps: upgrade gogo-protobuf to v1.3.2 (#10813)
* deps: upgrade gogo-protobuf to v1.3.2

* go mod tidy using go 1.16

* proto: regen protobufs after upgrading gogo/protobuf

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-12 14:05:46 -04:00
Mark Anderson 03a3ec2b55
Fixup to support unix domain socket via command line (#10758)
Missed the need to add support for unix domain socket config via
api/command line. This is a variant of the problems described in
it is easy to drop one.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-08-12 10:05:22 -07:00
hc-github-team-consul-core f02ea91a8b auto-updated agent/uiserver/bindata_assetfs.go from commit ab6a67520 2021-08-11 17:05:51 +00:00
Daniel Nephin 364ef3d052 server: remove defaulting of PrimaryDatacenter
The constructor for Server is not at all the appropriate place to be setting default
values for a config struct that was passed in.

In production this value is always set from agent/config. In tests we should set the
default in a test helper.
2021-08-06 18:45:24 -04:00
Daniel Nephin 87fb26fd65
Merge pull request #10612 from bigmikes/acl-replication-fix
acl: acl replication routine to report the last error message
2021-08-06 18:29:51 -04:00
Daniel Nephin 047abdd73c acl: remove ACLDatacenter
This field has been unnecessary for a while now. It was always set to the same value
as PrimaryDatacenter. So we can remove the duplicate field and use PrimaryDatacenter
directly.

This change was made by GoLand refactor, which did most of the work for me.
2021-08-06 18:27:00 -04:00
Giulio Micheloni 5c34a48d45 String type instead of error type and changelog. 2021-08-06 22:35:27 +01:00
Daniel Nephin 9435118179 acl: remove Server.ResolveTokenIdentityAndDefaultMeta
This method suffered from similar naming to a couple other methods on Server, and had not great
re-use (2 callers). By copying a few of the lines into one of the callers we can move the
implementation into the second caller.

Once moved, we can see that ResolveTokenAndDefaultMeta is identical in both Client and Server, and
likely should be further refactored, possibly into ACLResolver.

This change is being made to make ACL resolution easier to trace.
2021-08-05 15:20:13 -04:00
Daniel Nephin 25f40de163 acl: remove Server.ResolveTokenToIdentityAndAuthorizer
This method was an alias for ACLResolver.ResolveTokenToIdentityAndAuthorizer. By removing the
method that does nothing the code becomes easier to trace.
2021-08-05 15:20:13 -04:00
Daniel Nephin 695963acb7 acl: recouple acl filtering from ACLResolver
ACL filtering only needs an authorizer and a logger. We can decouple filtering from
the ACLResolver by passing in the necessary logger.

This change is being made in preparation for moving the ACLResolver into an acl package
2021-08-05 15:20:13 -04:00
Daniel Nephin ba2f9a65d1 acl: remove unused error return
filterACLWithAuthorizer could never return an error. This change moves us a little bit
closer to being able to enable errcheck and catch problems caused by unhandled error
return values.
2021-08-05 15:20:13 -04:00
Daniel Nephin c80b9565e2 acl: rename acl.Authorizer vars to authz
For consistency
2021-08-05 15:19:47 -04:00
Daniel Nephin 37c67cb280 acl: move vet functions
These functions are moved to the one place they are called to improve code locality.

They are being moved out of agent/consul/acl.go in preparation for moving
ACLResolver to an acl package.
2021-08-05 15:19:24 -04:00
Daniel Nephin c8eedabc7c acl: move vetRegisterWithACL and vetDeregisterWithACL
These functions are used in only one place. Move the functions next to their one caller
to improve code locality.

This change is being made in preparation for moving the ACLResolver into an
acl package. The moved functions were previously in the same file as the ACLResolver.
By moving them out of that file we may be able to move the entire file
with fewer modifications.
2021-08-05 15:17:54 -04:00
Daniel Nephin b223c2bc25
Merge pull request #10770 from hashicorp/dnephin/log-cert-expiration
telemetry: add log message when certs are about to expire
2021-08-05 15:17:20 -04:00
Daniel Nephin c866f1041a
Merge pull request #10793 from hashicorp/dnephin/acl-intentions
acl: small cleanup of a couple Authorization flows
2021-08-05 15:16:49 -04:00
Dhia Ayachi 40baf98159
defer setting the state before returning to avoid stuck in `INITIALIZING` state (#10630)
* defer setting the state before returning to avoid being stuck in `INITIALIZING` state

* add changelog

* move comment with the right if statement

* ca: report state transition error from setSTate

* update comment to reflect state transition

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-05 14:51:19 -04:00
Daniel Nephin 79ab48ef81
Merge pull request #10768 from hashicorp/dnephin/agent-tls-cert-expiration-metric
telemetry: add Agent TLS Certificate expiration metric
2021-08-04 18:42:02 -04:00
Daniel Nephin 0ca9e875e2 acl: remove special handling of services in txn_endpoint
Follow up to: https://github.com/hashicorp/consul/pull/10738#discussion_r680190210

Previously we were passing an Authorizer that would always allow the
operation, then later checking the authorization using vetServiceTxnOp.

On the surface this seemed strange, but I think it was actually masking
a bug as well. Over time `servicePreApply` was changed to add additional
authorization for `service.Proxy.DestinationServiceName`, but because
we were passing a nil Authorizer, that authorization was not handled on
the txn_endpoint.

`TxnServiceOp.FillAuthzContext` has some special handling in enterprise,
so we need to make sure to continue to use that from the Txn endpoint.

This commit removes the `vetServiceTxnOp` function, and passes in the
`FillAuthzContext` function so that `servicePreApply` can be used by
both the catalog and txn endpoints. This should be much less error prone
and prevent bugs like this in the future.
2021-08-04 18:32:20 -04:00
hc-github-team-consul-core ef162f8390 auto-updated agent/uiserver/bindata_assetfs.go from commit bcd53e73a 2021-08-04 22:27:44 +00:00
Daniel Nephin f6d5a85561 acl: move check for Intention.DestinationName into Authorizer
Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r680134445

Move the check for the Intention.DestinationName into the Authorizer to remove the
need to check what kind of Authorizer is being used.

It sounds like this check is only for legacy ACLs, so is probably just a safeguard
.
2021-08-04 18:06:44 -04:00
Daniel Nephin 3dc113ada6
Merge pull request #10738 from hashicorp/dnephin/remove-authorizer-nil-checks-2
acl: remove the last of the authz == nil checks
2021-08-04 17:41:40 -04:00
Daniel Nephin 2e9aa91256
Merge pull request #10737 from hashicorp/dnephin/remove-authorizer-nil-checks
acl: remove authz == nil checks
2021-08-04 17:39:34 -04:00
Daniel Nephin 210a850353 telemetry: add log message when certs are about to expire 2021-08-04 14:18:59 -04:00
Daniel Nephin 13aa7b70d5 telemetry: fix a couple bugs in cert expiry metrics
1. do not emit the metric if Query fails
2. properly check for PrimaryUsersIntermediate, the logic was inverted

Also improve the logging by including the metric name in the log message
2021-08-04 13:51:44 -04:00
Daniel Nephin 1673b3a68c telemetry: add a metric for agent TLS cert expiry 2021-08-04 13:51:44 -04:00
Dhia Ayachi 6ed6966a1f
fix state index for `CAOpSetRootsAndConfig` op (#10675)
* fix state index for `CAOpSetRootsAndConfig` op

* add changelog

* Update changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* remove the change log as it's not needed

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-04 13:07:49 -04:00
hc-github-team-consul-core 4b2ada0dcc auto-updated agent/uiserver/bindata_assetfs.go from commit 8ad1ab9c0 2021-08-04 16:47:13 +00:00
Evan Culver 57aabe3455
checks: Add Interval and Timeout to API response (#10717) 2021-08-03 15:26:49 -07:00
Daniel Nephin 953c9bee4f acl: Remove the remaining authz == nil checks
These checks were a bit more involved. They were previously skipping some code paths
when the authorizer was nil. After looking through these it seems correct to remove the
authz == nil check, since it will never evaluate to true.
2021-07-30 14:55:35 -04:00
Daniel Nephin e4821a58ee acl: remove acl == nil checks 2021-07-30 14:28:19 -04:00
Daniel Nephin fbaeac9ecf acl: remove authz == nil checks
These case are already impossible conditions, because most of these functions already start
with a check for ACLs being disabled. So the code path being removed could never be reached.

The one other case (ConnectAuthorized) was already changed in a previous commit. This commit
removes an impossible branch because authz == nil can never be true.
2021-07-30 13:58:35 -04:00
Daniel Nephin b6d9d0d9f7 acl: remove many instances of authz == nil 2021-07-30 13:58:35 -04:00
Daniel Nephin bbc05ae869 agent: remove unused agent methods
These methods are no longer used. Remove the methods, and update the
tests to use actual method used by production code.

Also removes the 'authz == nil' check is no longer a possible code path
now that we are returning a non-nil acl.Authorizer when ACLs are disabled.
2021-07-30 13:58:35 -04:00
Daniel Nephin 2503f27a36 acl: remove rule == nil checks 2021-07-30 13:58:35 -04:00
hc-github-team-consul-core 701d4ffef0 auto-updated agent/uiserver/bindata_assetfs.go from commit 2ee501be8 2021-07-30 17:58:27 +00:00
Daniel Nephin 475fec5670
Merge pull request #10632 from hashicorp/pairing/acl-authorizer-when-acl-disabled
acls: Update ACL authorizer to return meaningful permission when ACLs are disabled
2021-07-30 13:22:55 -04:00
Evan Culver 241b6429c3 Fix intention endpoint test 2021-07-30 12:58:45 -04:00
Daniel Nephin 9b41e7287f acl: use acl.ManangeAll when ACLs are disabled
Instead of returning nil and checking for nilness

Removes a bunch of nil checks, and fixes one test failures.
2021-07-30 12:58:24 -04:00
Blake Covarrubias f97e843c61 Add OSS changes for specifying audit log permission mode 2021-07-30 09:58:11 -07:00
Daniel Nephin f2f5aba1bf
Merge pull request #10707 from hashicorp/dnephin/streaming-setup-default-timeout
streaming: set default query timeout
2021-07-28 18:29:28 -04:00
Daniel Nephin 057e8320f9 streaming: set a default timeout
The blocking query backend sets the default value on the server side.
The streaming backend does not using blocking queries, so we must set the timeout on
the client.
2021-07-28 17:50:00 -04:00
hc-github-team-consul-core f39d36d346 auto-updated agent/uiserver/bindata_assetfs.go from commit eb5512fb7 2021-07-27 21:39:22 +00:00
Chris S. Kim 33d7d48767
sync enterprise files with oss (#10705) 2021-07-27 17:09:59 -04:00
Daniel Nephin cfc829275c http: don't log an error if the request is cancelled
Now that we have at least one endpoint that uses context for cancellation we can
encounter this scenario where the returned error is a context.Cancelled or
context.DeadlineExceeded.

If the request.Context().Err() is not nil, then we know the request itself was cancelled, so
we can log a different message at Info level, instad of the error.
2021-07-27 17:06:59 -04:00
Daniel Nephin bad2c4ef67
Merge pull request #10399 from hashicorp/dnephin/debug-stream-metrics
debug: use the new metrics stream in debug command
2021-07-27 13:23:15 -04:00
Daniel Nephin 7d24564ff0 http: add tests for AgentMetricsStream 2021-07-26 17:53:33 -04:00
Daniel Nephin cf2e25c6bb http: emit indented JSON in the metrics stream endpoint
To remove the need to decode and re-encode in the CLI
2021-07-26 17:53:33 -04:00
Daniel Nephin d716f709fd debug: use the new metrics stream in debug command 2021-07-26 17:53:32 -04:00
Freddy b136b1795a
Reset root prune interval after TestLeader_CARootPruning completes
#10645

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-07-26 15:43:40 -06:00
Chris S. Kim 6341183a84
agent: update proxy upstreams to inherit namespace from service (#10688) 2021-07-26 17:12:29 -04:00
Freddy 57ca0ed480
Log the correlation ID when blocking queries fire (#10689)
Knowing that blocking queries are firing does not provide much
information on its own. If we know the correlation IDs we can
piece together which parts of the snapshot have been populated.

Some of these responses might be empty from the blocking
query timing out. But if they're returning quickly I think we
can reasonably assume they contain data.
2021-07-23 16:36:17 -06:00
R.B. Boyer c271976445
state: refactor some node/coordinate state store functions to take an EnterpriseMeta (#10687)
Note the field is not used yet.
2021-07-23 13:42:23 -05:00
R.B. Boyer b2facb35a9
replumbing a bunch of api and agent structs for partitions (#10681) 2021-07-22 14:33:22 -05:00
R.B. Boyer 254557a1f6
sync changes to oss files made in enterprise (#10670) 2021-07-22 13:58:08 -05:00
R.B. Boyer 62ac98b564
agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669) 2021-07-22 13:20:45 -05:00
Dhia Ayachi b725605fe4
config raft apply silent error (#10657)
* return an error when the index is not valid

* check response as bool when applying `CAOpSetConfig`

* remove check for bool response

* fix error message and add check to test

* fix comment

* add changelog
2021-07-22 10:32:27 -04:00
Freddy 7d48383041
Avoid panic on concurrent writes to cached service config map (#10647)
If multiple instances of a service are co-located on the same node then
their proxies will all share a cache entry for their resolved service
configuration. This is because the cache key contains the name of the
watched service but does not take into account the ID of the watching
proxies.

This means that there will be multiple agent service manager watches
that can wake up on the same cache update. These watchers then
concurrently modify the value in the cache when merging the resolved
config into the local proxy definitions.

To avoid this concurrent map write we will only delete the key from
opaque config in the local proxy definition after the merge, rather
than from the cached value before the merge.
2021-07-20 10:09:29 -06:00
hc-github-team-consul-core aa97ed5ac6 auto-updated agent/uiserver/bindata_assetfs.go from commit 1eb7a83ee 2021-07-20 15:15:10 +00:00
Blake Covarrubias 441a6c9969
Add DNS recursor strategy option (#10611)
This change adds a new `dns_config.recursor_strategy` option which
controls how Consul queries DNS resolvers listed in the `recursors`
config option. The supported options are `sequential` (default), and
`random`.

Closes #8807

Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Priyanka Sengupta <psengupta@flatiron.com>
2021-07-19 15:22:51 -07:00
Daniel Nephin 901a5cdd8c
Merge pull request #10396 from hashicorp/dnephin/fix-more-data-races
Fix some data races
2021-07-16 18:21:58 -04:00
Daniel Nephin 23dfb8e9ad
Merge pull request #10009 from hashicorp/dnephin/trim-dns-response-with-edns
dns: properly trim response when EDNS is used
2021-07-16 18:09:25 -04:00
Daniel Nephin db29c51cd2 acl: use SetHash consistently in testPolicyForID
A previous commit used SetHash on two of the cases to fix a data race. This commit applies
that change to all cases. Using SetHash in this test helper should ensure that the
test helper behaves closer to production.
2021-07-16 17:59:56 -04:00
Daniel Nephin 63772f7ac4 dns: improve naming of error to match DNS terminology
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
2021-07-16 12:40:24 -04:00
Dhia Ayachi 079decdabd fix truncate when NS is set
Also: fix test to catch the issue
2021-07-16 12:40:11 -04:00
Evan Culver 521c423075
acls: Show `AuthMethodNamespace` when reading/listing ACL token meta (#10598) 2021-07-15 10:38:52 -07:00
Daniel Nephin b4ab87111c
Merge pull request #10567 from hashicorp/dnephin/config-unexport-build
config: unexport the remaining builder methods
2021-07-15 12:05:19 -04:00
Freddy a942a2e025
Merge pull request #10621 from hashicorp/vuln/validate-sans 2021-07-15 09:43:55 -06:00
Daniel Nephin f286ea0922
Fix godoc comment
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2021-07-15 11:22:46 -04:00
R.B. Boyer e018d8a10b
xds: ensure single L7 deny intention with default deny policy does not result in allow action (CVE-2021-36213) (#10619) 2021-07-15 10:09:00 -05:00
hc-github-team-consul-core 6bf7c98227 auto-updated agent/uiserver/bindata_assetfs.go from commit 0762da3a6 2021-07-15 11:23:49 +00:00
Giulio Micheloni 3a1afd8f57 acl: fix error type into a string type for serialization issue
acl_endpoint_test.go:507:
        	Error Trace:	acl_endpoint_test.go:507
        	            				retry.go:148
        	            				retry.go:149
        	            				retry.go:103
        	            				acl_endpoint_test.go:504
        	Error:      	Received unexpected error:
        	            	codec.decoder: decodeValue: Cannot decode non-nil codec value into nil error (1 methods)
        	Test:       	TestACLEndpoint_ReplicationStatus
2021-07-15 11:31:44 +02:00
freddygv b6b42c34dc Add TODOs about partition handling 2021-07-14 22:21:55 -06:00
freddygv 3d4fa44c22 Update golden files 2021-07-14 22:21:55 -06:00
freddygv a7de87e95b Validate SANs for passthrough clusters and failovers 2021-07-14 22:21:55 -06:00
freddygv a6f7d806f6 Update golden files to account for SAN validation 2021-07-14 22:21:55 -06:00
freddygv 3f11449363 Validate Subject Alternative Name for upstreams
These changes ensure that the identity of services dialed is
cryptographically verified.

For all upstreams we validate against SPIFFE IDs in the format used by
Consul's service mesh:

spiffe://<trust-domain>/ns/<namespace>/dc/<datacenter>/svc/<service>
2021-07-14 22:20:27 -06:00
Daniel Nephin 27871498f0 Fix a data race in TestACLResolver_Client
By setting the hash when we create the policy.

```
WARNING: DATA RACE
Read at 0x00c0028b4b10 by goroutine 1182:
  github.com/hashicorp/consul/agent/structs.(*ACLPolicy).SetHash()
      /home/daniel/pers/code/consul/agent/structs/acl.go:701 +0x40d
  github.com/hashicorp/consul/agent/structs.ACLPolicies.resolveWithCache()
      /home/daniel/pers/code/consul/agent/structs/acl.go:779 +0xfe
  github.com/hashicorp/consul/agent/structs.ACLPolicies.Compile()
      /home/daniel/pers/code/consul/agent/structs/acl.go:809 +0xf1
  github.com/hashicorp/consul/agent/consul.(*ACLResolver).ResolveTokenToIdentityAndAuthorizer()
      /home/daniel/pers/code/consul/agent/consul/acl.go:1226 +0x6ef
  github.com/hashicorp/consul/agent/consul.resolveTokenAsync()
      /home/daniel/pers/code/consul/agent/consul/acl_test.go:66 +0x5c

Previous write at 0x00c0028b4b10 by goroutine 1509:
  github.com/hashicorp/consul/agent/structs.(*ACLPolicy).SetHash()
      /home/daniel/pers/code/consul/agent/structs/acl.go:730 +0x3a8
  github.com/hashicorp/consul/agent/structs.ACLPolicies.resolveWithCache()
      /home/daniel/pers/code/consul/agent/structs/acl.go:779 +0xfe
  github.com/hashicorp/consul/agent/structs.ACLPolicies.Compile()
      /home/daniel/pers/code/consul/agent/structs/acl.go:809 +0xf1
  github.com/hashicorp/consul/agent/consul.(*ACLResolver).ResolveTokenToIdentityAndAuthorizer()
      /home/daniel/pers/code/consul/agent/consul/acl.go:1226 +0x6ef
  github.com/hashicorp/consul/agent/consul.resolveTokenAsync()
      /home/daniel/pers/code/consul/agent/consul/acl_test.go:66 +0x5c

Goroutine 1182 (running) created at:
  github.com/hashicorp/consul/agent/consul.TestACLResolver_Client.func4()
      /home/daniel/pers/code/consul/agent/consul/acl_test.go:1669 +0x459
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Goroutine 1509 (running) created at:
  github.com/hashicorp/consul/agent/consul.TestACLResolver_Client.func4()
      /home/daniel/pers/code/consul/agent/consul/acl_test.go:1668 +0x415
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
```
2021-07-14 18:58:16 -04:00
Daniel Nephin c3c8058fd7 agent: remove deprecated call in a test 2021-07-14 18:58:16 -04:00
Daniel Nephin 9d471269d8 agent: fix a data race in a test
The test was modifying a pointer to a struct that had been passed to
another goroutine. Instead create a new struct to modify.

```
WARNING: DATA RACE
Write at 0x00c01407c3c0 by goroutine 832:
  github.com/hashicorp/consul/agent.TestServiceManager_PersistService_API()
      /home/daniel/pers/code/consul/agent/service_manager_test.go:446 +0x1d86
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Previous read at 0x00c01407c3c0 by goroutine 938:
  reflect.typedmemmove()
      /usr/lib/go/src/runtime/mbarrier.go:177 +0x0
  reflect.Value.Set()
      /usr/lib/go/src/reflect/value.go:1569 +0x13b
  github.com/mitchellh/copystructure.(*walker).Primitive()
      /home/daniel/go/pkg/mod/github.com/mitchellh/copystructure@v1.0.0/copystructure.go:289 +0x190
  github.com/mitchellh/reflectwalk.walkPrimitive()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:252 +0x31b
  github.com/mitchellh/reflectwalk.walk()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:179 +0x24d
  github.com/mitchellh/reflectwalk.walkStruct()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x4ec
  github.com/mitchellh/reflectwalk.walk()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x656
  github.com/mitchellh/reflectwalk.walkStruct()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x4ec
  github.com/mitchellh/reflectwalk.walk()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x656
  github.com/mitchellh/reflectwalk.Walk()
      /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:92 +0x164
  github.com/mitchellh/copystructure.Config.Copy()
      /home/daniel/go/pkg/mod/github.com/mitchellh/copystructure@v1.0.0/copystructure.go:69 +0xe7
  github.com/mitchellh/copystructure.Copy()
      /home/daniel/go/pkg/mod/github.com/mitchellh/copystructure@v1.0.0/copystructure.go:13 +0x84
  github.com/hashicorp/consul/agent.mergeServiceConfig()
      /home/daniel/pers/code/consul/agent/service_manager.go:362 +0x56
  github.com/hashicorp/consul/agent.(*serviceConfigWatch).handleUpdate()
      /home/daniel/pers/code/consul/agent/service_manager.go:279 +0x250
  github.com/hashicorp/consul/agent.(*serviceConfigWatch).runWatch()
      /home/daniel/pers/code/consul/agent/service_manager.go:246 +0x2d4

Goroutine 832 (running) created at:
  testing.(*T).Run()
      /usr/lib/go/src/testing/testing.go:1238 +0x5d7
  testing.runTests.func1()
      /usr/lib/go/src/testing/testing.go:1511 +0xa6
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
  testing.runTests()
      /usr/lib/go/src/testing/testing.go:1509 +0x612
  testing.(*M).Run()
      /usr/lib/go/src/testing/testing.go:1417 +0x3b3
  main.main()
      _testmain.go:1181 +0x236

Goroutine 938 (running) created at:
  github.com/hashicorp/consul/agent.(*serviceConfigWatch).start()
      /home/daniel/pers/code/consul/agent/service_manager.go:223 +0x4e4
  github.com/hashicorp/consul/agent.(*ServiceManager).AddService()
      /home/daniel/pers/code/consul/agent/service_manager.go:98 +0x344
  github.com/hashicorp/consul/agent.(*Agent).addServiceLocked()
      /home/daniel/pers/code/consul/agent/agent.go:1942 +0x2e4
  github.com/hashicorp/consul/agent.(*Agent).AddService()
      /home/daniel/pers/code/consul/agent/agent.go:1929 +0x337
  github.com/hashicorp/consul/agent.TestServiceManager_PersistService_API()
      /home/daniel/pers/code/consul/agent/service_manager_test.go:400 +0x17c4
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

```
2021-07-14 18:58:16 -04:00
Daniel Nephin 6703787740 agent: fix a data race in DNS tests
The dnsConfig pulled from the atomic.Value is a pointer, so modifying it in place
creates a data race. Use the exported ReloadConfig interface instead.
2021-07-14 18:58:16 -04:00
Daniel Nephin 2946e42a9e agent: fix two data race in agent tests
The LogOutput io.Writer used by TestAgent must allow concurrent reads and writes, and a
bytes.Buffer does not allow this. The bytes.Buffer must be wrapped with a lock to make this safe.
2021-07-14 18:58:16 -04:00
Daniel Nephin ff26294d63 consul: fix data race in leader CA tests
Some global variables are patched to shorter values in these tests. But the goroutines that read
them can outlive the test because nothing waited for them to exit.

This commit adds a Wait() method to the routine manager, so that tests can wait for the goroutines
to exit. This prevents the data race because the 'reset to original value' can happen
after all other goroutines have stopped.
2021-07-14 18:58:15 -04:00
Daniel Nephin edd755b7ab dns: correct rcode for qtype not supported
A previous commit started using QueryRefuced, but that is not correct. QueryRefuced refers to
the OpCode, not the query type.

Instead use errNoAnswer because we have no records for that query type.
2021-07-14 17:48:50 -04:00
Dhia Ayachi 48171c43f4 Check response len do not exceed max Buffer size 2021-07-14 17:15:34 -04:00
Dhia Ayachi 8fcac3cef6 add missing test for truncate 2021-07-14 17:15:34 -04:00
Daniel Nephin b4abf8b0ec dns: remove network parameter from two funcs
Now that trimDNSResponse is handled by the caller we don't need to pass this value
around. We can remove it from both the serviceLookup struct, and two functions.
2021-07-14 17:15:34 -04:00
Daniel Nephin 4712e24749 dns: trim response immediately before the write
Previously the response was being trimmed before adding the EDNS values, which could cause it to exceed
the max size.
2021-07-14 17:15:34 -04:00
Daniel Nephin a9e9c6c23e dns: handle errors from dispatch 2021-07-14 17:15:34 -04:00
Daniel Nephin 6cf9ecc1c9 dns: error response from dispatch
So that dispatch can communicate status back to the caller.
2021-07-14 17:15:34 -04:00
Daniel Nephin 9298cfe0f6 dns: refactor dispatch to use an explicit return in each case
In preparation for changing the return value, so that SOA, eDNS trimming and 'not found'
errors can be handled in a single place.
2021-07-14 17:15:34 -04:00
Daniel Nephin b09aa1e3c6 dns: small refactor to setEDNS to return early
Using a guard clause instead of a long nested if.

The diff is best viewed with whitespace turned off.
2021-07-14 17:15:34 -04:00
Daniel Nephin f1bc7bd49a dns: remove unused method
It was added in 5934f803bfb54c1ceeeb6518398f1b82a726459f but it was never used.
2021-07-14 17:15:34 -04:00
Daniel Nephin e3d781d99c dns: remove unnecessary function wrapping
The dispatch function was called from a single place and did nothing but add a default value.
Removing it makes code easier to trace by removing an unnecessary hop.
2021-07-14 17:15:33 -04:00
Kyle Havlovitz e97bc2bda7 http: add partition query param parsing 2021-07-14 12:07:38 -07:00
hc-github-team-consul-core 8c5723ec98 auto-updated agent/uiserver/bindata_assetfs.go from commit 3e80e637b 2021-07-14 18:00:42 +00:00
Giulio Micheloni 96fe1f4078 acl: acl replication routine to report the last error message 2021-07-14 11:50:23 +02:00
Daniel Nephin 57c5a40869
Merge pull request #10588 from hashicorp/dnephin/config-fix-ports-grpc
config: rename `ports.grpc` to `ports.xds`
2021-07-13 13:11:38 -04:00
Daniel Nephin 15300b873a fix backwards compat for envoy command
The compatv2 integration tests were failing because they use an older CLI version with a newer
HTTP API. This commit restores the GRPCPort field to the DebugConfig output to allow older
CIs to continue to fetch the port.
2021-07-13 12:31:49 -04:00
Daniel Nephin 25dc14f036 Apply suggestions from code review
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-07-13 12:31:49 -04:00
Daniel Nephin a5f93e5596 command/envoy: stop using the DebugConfig from Self endpoint
The DebugConfig in the self endpoint can change at any time. It's not a stable API.

With the previous change to rename GRPCPort to XDSPort this command would have broken.
This commit adds the XDSPort to a stable part of the XDS api, and changes the envoy command to read
this new field.

It includes support for the old API as well, in case a newer CLI is used with an older API, and
adds a test for both cases.
2021-07-13 12:31:49 -04:00
Daniel Nephin ef6bc739a1 config: update config settings and flags for ports.xds 2021-07-13 12:31:48 -04:00
Dhia Ayachi 53b45a8441
check expiry date of the root/intermediate before using it to sign a leaf (#10500)
* ca: move provider creation into CAManager

This further decouples the CAManager from Server. It reduces the interface between them and
removes the need for the SetLogger method on providers.

* ca: move SignCertificate to CAManager

To reduce the scope of Server, and keep all the CA logic together

* ca: move SignCertificate to the file where it is used

* auto-config: move autoConfigBackend impl off of Server

Most of these methods are used exclusively for the AutoConfig RPC
endpoint. This PR uses a pattern that we've used in other places as an
incremental step to reducing the scope of Server.

* fix linter issues

* check error when `raftApplyMsgpack`

* ca: move SignCertificate to CAManager

To reduce the scope of Server, and keep all the CA logic together

* check expiry date of the intermediate before using it to sign a leaf

* fix typo in comment

Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>

* Fix test name

* do not check cert start date

* wrap error to mention it is the intermediate expired

* Fix failing test

* update comment

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* use shim to avoid sleep in test

* add root cert validation

* remove duplicate code

* Revert "fix linter issues"

This reverts commit 6356302b54f06c8f2dee8e59740409d49e84ef24.

* fix import issue

* gofmt leader_connect_ca

* add changelog entry

* update error message

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* fix error message in test

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2021-07-13 12:15:06 -04:00
R.B. Boyer ae8b526be8
connect/ca: ensure edits to the key type/bits for the connect builtin CA will regenerate the roots (#10330)
progress on #9572
2021-07-13 11:12:07 -05:00
R.B. Boyer 0537922c6c
connect/ca: require new vault mount points when updating the key type/bits for the vault connect CA provider (#10331)
progress on #9572
2021-07-13 11:11:46 -05:00
Daniel Nephin 58cf5767a8
Merge pull request #10479 from hashicorp/dnephin/ca-provider-explore-2
ca: move Server.SignIntermediate to CAManager
2021-07-12 19:03:43 -04:00
Daniel Nephin a22bdb2ac9
Merge pull request #10445 from hashicorp/dnephin/ca-provider-explore
ca: isolate more of the CA logic in CAManager
2021-07-12 15:26:23 -04:00
Daniel Nephin fdb0ba8041 ca: use provider constructors to be more consistent
Adds a contructor for the one provider that did not have one.
2021-07-12 14:04:34 -04:00
Dhia Ayachi 3eac4ffda4 check error when `raftApplyMsgpack` 2021-07-12 13:42:51 -04:00
Daniel Nephin 34c8585b29 auto-config: move autoConfigBackend impl off of Server
Most of these methods are used exclusively for the AutoConfig RPC
endpoint. This PR uses a pattern that we've used in other places as an
incremental step to reducing the scope of Server.
2021-07-12 13:42:40 -04:00
Daniel Nephin 605275b4dc ca: move SignCertificate to the file where it is used 2021-07-12 13:42:39 -04:00
Daniel Nephin c2e85f25d4 ca: move SignCertificate to CAManager
To reduce the scope of Server, and keep all the CA logic together
2021-07-12 13:42:39 -04:00
Daniel Nephin 6fced99ea9
Merge pull request #10590 from hashicorp/dnephin/tls-config-less-copy
config: remove duplicate tlsutil.Config fields from agent/consul.Config
2021-07-12 13:00:52 -04:00
hc-github-team-consul-core dfff26a758 auto-updated agent/uiserver/bindata_assetfs.go from commit a96e87aec 2021-07-12 13:33:26 +00:00
Dhia Ayachi a0320169fe add missing state reset when stopping ca manager 2021-07-12 09:32:36 -04:00
Daniel Nephin 68d5f7769a ca: fix mockCAServerDelegate to work with the new interface
raftApply was removed so ApplyCARequest needs to handle all the possible operations

Also set the providerShim to use the mock provider.

other changes are small test improvements that were necessary to debug the failures.
2021-07-12 09:32:36 -04:00
Daniel Nephin 6d4b0ce194 ca: remove unused method
and small refactor to getCAProvider so that GoLand is less confused about what it is doing.
Previously it was reporting that the for condition was always true, which was not the case.
2021-07-12 09:32:35 -04:00
Daniel Nephin 4330122d9a ca: remove raftApply from delegate interface
After moving ca.ConsulProviderStateDelegate into the interface we now
have the ApplyCARequest method which does the same thing. Use this more
specific method instead of raftApply.
2021-07-12 09:32:35 -04:00
Daniel Nephin fae0a8f851 ca: move generateCASignRequest to the delegate
This method on Server was only used by the caDelegateWithState, so move it there
until we can move it entirely into CAManager.
2021-07-12 09:32:35 -04:00
Daniel Nephin d4bb9fd97a ca: move provider creation into CAManager
This further decouples the CAManager from Server. It reduces the interface between them and
removes the need for the SetLogger method on providers.
2021-07-12 09:32:33 -04:00
Daniel Nephin fc629d9eaa ca-manager: move provider shutdown into CAManager
Reducing the coupling between Server and CAManager
2021-07-12 09:27:28 -04:00
Daniel Nephin 1e23d181b5 config: remove misleading UseTLS field
This field was documented as enabling TLS for outgoing RPC, but that was not the case.
All this field did was set the use_tls serf tag.

Instead of setting this field in a place far from where it is used, move the logic to where
the serf tag is set, so that the code is much more obvious.
2021-07-09 19:01:45 -04:00
Daniel Nephin 3c60a46376 config: remove duplicate TLSConfig fields from agent/consul.Config
tlsutil.Config already presents an excellent structure for this
configuration. Copying the runtime config fields to agent/consul.Config
makes code harder to trace, and provides no advantage.

Instead of copying the fields around, use the tlsutil.Config struct
directly instead.

This is one small step in removing the many layers of duplicate
configuration.
2021-07-09 18:49:42 -04:00
Daniel Nephin 2ab6be6a88 config: update GRPCPort and addr in runtime config 2021-07-09 12:31:53 -04:00
Daniel Nephin 9c6458c6c2 rename GRPC->XDS where appropriate 2021-07-09 12:17:45 -04:00
Evan Culver 5ff191ad99
Add support for returning ACL secret IDs for accessors with acl:write (#10546) 2021-07-08 15:13:08 -07:00
Daniel Nephin dcb90fb832
Merge pull request #10570 from hashicorp/copy-of-master
Changes that were accidentally merged into the old master branch
2021-07-08 16:28:56 -04:00
R.B. Boyer 0e6a482b76
config: add agent config flag for enterprise clients to indicate they wish to join a particular partition (#10572) 2021-07-08 10:03:38 -05:00
Dhia Ayachi e5dbf5e55b
Add ca certificate metrics (#10504)
* add intermediate ca metric routine

* add Gauge config for intermediate cert

* Stop metrics routine when stopping leader

* add changelog entry

* updage changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* use variables instead of a map

* go imports sort

* Add metrics for primary and secondary ca

* start metrics routine in the right DC

* add telemetry documentation

* update docs

* extract expiry fetching in a func

* merge metrics for primary and secondary into signing ca metric

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-07-07 09:41:01 -04:00
hc-github-team-consul-core 83c543cd6b auto-updated agent/uiserver/bindata_assetfs.go from commit 6fbeea5de 2021-07-07 10:51:32 +00:00
Jared Kirschner 37f25aed1d
Merge pull request #10559 from jkirschner-hashicorp/fix-autopilot-config-post-default-values
Fix defaults for autopilot config update
2021-07-06 19:19:52 -04:00
hc-github-team-consul-core 93607fa2ee auto-updated agent/uiserver/bindata_assetfs.go from commit 2c4f22a9f 2021-07-06 22:54:28 +00:00
Daniel Nephin 14527dd005
Merge pull request #10552 from hashicorp/dnephin/ca-remove-rotation-period
ca: remove unused RotationPeriod field
2021-07-06 18:49:33 -04:00
Daniel Nephin e8e5defc71 config: unexport the remaining builder methods
And remove BuildAndValidate. This commit completes some earlier work to reduce the config
interface a single Load function.

The last remaining test was converted to use Load instad of BuildAndValidate.
2021-07-06 18:42:09 -04:00
Jared Kirschner 1449806c3d Fix defaults for autopilot config update
Previously, for a POST request to the /v1/operator/autopilot/configuration
endpoint, any fields not included in the payload were set to a zero-initialized
value rather than the documented default value.

Now, if an optional field is not included in the payload, it will be set to its
documented default value:
- CleanupDeadServers:      true
- LastContactThreshold:    "200ms"
- MaxTrailingLogs:         250
- MinQuorum:               0
- ServerStabilizationTime: "10s"
- RedundancyZoneTag:       ""
- DisableUpgradeMigration: false
- UpgradeVersionTag:       ""
2021-07-06 18:39:40 -04:00
hc-github-team-consul-core 164db92b15 auto-updated agent/uiserver/bindata_assetfs.go from commit 74070c095 2021-07-06 16:06:51 +00:00
hc-github-team-consul-core ff2360d430 auto-updated agent/uiserver/bindata_assetfs.go from commit 5f73de6fb 2021-07-06 15:50:57 +00:00
jkirschner-hashicorp 31bbab8ae7
Merge pull request #10560 from jkirschner-hashicorp/change-sane-to-reasonable
Replace use of 'sane' where appropriate
2021-07-06 11:46:04 -04:00
Daniel Nephin b4a10443d1 ca: remove unused RotationPeriod field
This field was never used. Since it is persisted as part of a map[string]interface{} it
is pretty easy to remove it.
2021-07-05 19:15:44 -04:00
Jared Kirschner 4c3b1b8b7b Replace use of 'sane' where appropriate
HashiCorp voice, style, and language guidelines recommend avoiding ableist
language unless its reference to ability is accurate in a particular use.
2021-07-02 12:18:46 -04:00
Dhia Ayachi b57cf27e8f
Format certificates properly (rfc7468) with a trailing new line (#10411)
* trim carriage return from certificates when inserting rootCA in the inMemDB

* format rootCA properly when returning the CA on the connect CA endpoint

* Fix linter warnings

* Fix providers to trim certs before returning it

* trim newlines on write when possible

* add changelog

* make sure all provider return a trailing newline after the root and intermediate certs

* Fix endpoint to return trailing new line

* Fix failing test with vault provider

* make test more robust

* make sure all provider return a trailing newline after the leaf certs

* Check for suffix before removing newline and use function

* Add comment to consul provider

* Update change log

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* fix typo

* simplify code callflow

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* extract requireNewLine as shared func

* remove dependency to testify in testing file

* remove extra newline in vault provider

* Add cert newline fix to envoy xds

* remove new line from mock provider

* Remove adding a new line from provider and fix it when the cert is read

* Add a comment to explain the fix

* Add missing for leaf certs

* fix missing new line

* fix missing new line in leaf certs

* remove extra new line in test

* updage changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* fix in vault provider and when reading cache (RPC call)

* fix AWS provider

* fix failing test in the provider

* remove comments and empty lines

* add check for empty cert in test

* fix linter warnings

* add new line for leaf and private key

* use string concat instead of Sprintf

* fix new lines for leaf signing

* preallocate slice and remove append

* Add new line to `SignIntermediate` and `CrossSignCA`

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-06-30 20:48:29 -04:00
Daniel Nephin 72ea979c39
Merge pull request #10515 from hashicorp/dnephin/fix-arm32-atomic-aligment
Fix panic on 32-bit platforms
2021-06-30 16:40:20 -04:00
Daniel Nephin 843e08bb23 testing: fix a test for 32-bit
The hcl decoding apparently uses strconv.ParseInt, which fails to parse a 64bit int.
Since hcl v1 is basically EOl, it seems unlikely we'll fix this in hcl.

Since this test is only about loading values from config files, the extra large number
doesn't seem important. Trim a few zeros from the numbers so that they parse
properly on 32bit platforms.

Also skip a slow test when -short is used.
2021-06-29 16:10:21 -04:00
Daniel Nephin e226733b26 fix 64-bit aligment for 32-bit platforms
sync/atomic must be used with 64-bit aligned fields, and that alignment is difficult to
ensure unless the field is the first one in the struct.

https://golang.org/pkg/sync/atomic/#pkg-note-BUG.
2021-06-29 16:10:21 -04:00
Daniel Nephin ffefcdc025 streaming: support X-Cache-Hit header
If a value was already available in the local view the request is considered a cache hit.
If the materialized had to wait for a value, it is considered a cache miss.
2021-06-28 17:29:23 -04:00
Daniel Nephin a4a390d7c5 streaming: fix enable of streaming in the client
And add checks to all the tests that explicitly use streaming.
2021-06-28 17:23:14 -04:00
Daniel Nephin 62beaa80f3 Remove a racy and failing test
This test is super racy (it's not just a single line).

This test also starts failing once streaming is enabled, because the
cache rate limit no longer applies to the requests in the test. The
queries use streaming instead of the cache.

This test is no longer valid, and the functionality is already well
tested by TestCacheThrottle.  Instead of spending time rewriting this
test, let's remove it.

```
WARNING: DATA RACE
Read at 0x00c01de410fc by goroutine 735:
  github.com/hashicorp/consul/agent.TestCacheRateLimit.func1()
      /home/daniel/pers/code/consul/agent/agent_test.go:1024 +0x9af
  github.com/hashicorp/consul/testrpc.WaitForTestAgent()
      /home/daniel/pers/code/consul/testrpc/wait.go:99 +0x209
  github.com/hashicorp/consul/agent.TestCacheRateLimit.func1()
      /home/daniel/pers/code/consul/agent/agent_test.go:966 +0x1ad
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Previous write at 0x00c01de410fc by goroutine 605:
  github.com/hashicorp/consul/agent.TestCacheRateLimit.func1.2()
      /home/daniel/pers/code/consul/agent/agent_test.go:998 +0xe9

Goroutine 735 (running) created at:
  testing.(*T).Run()
      /usr/lib/go/src/testing/testing.go:1238 +0x5d7
  github.com/hashicorp/consul/agent.TestCacheRateLimit()
      /home/daniel/pers/code/consul/agent/agent_test.go:961 +0x375
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202

Goroutine 605 (finished) created at:
  github.com/hashicorp/consul/agent.TestCacheRateLimit.func1()
      /home/daniel/pers/code/consul/agent/agent_test.go:1022 +0x91e
  github.com/hashicorp/consul/testrpc.WaitForTestAgent()
      /home/daniel/pers/code/consul/testrpc/wait.go:99 +0x209
  github.com/hashicorp/consul/agent.TestCacheRateLimit.func1()
      /home/daniel/pers/code/consul/agent/agent_test.go:966 +0x1ad
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1193 +0x202
```
2021-06-28 17:23:13 -04:00
Daniel Nephin d0e32cc3ba http: add an X-Consul-Query-Backend header to responses
So that it is easier to detect and test when streaming is being used.
2021-06-28 16:44:58 -04:00
Daniel Nephin 902bd80989
Merge pull request #10506 from hashicorp/dnephin/docs-rpc-query-metrics
docs: correct some misleading telemetry docs
2021-06-28 12:33:57 -04:00
Daniel Nephin 86244967c5 docs: correct some misleading telemetry docs
The query metrics are actually reported for all read queries, not only
ones that use a MinIndex to block for updates.

Also clarify the raft.apply metric is only on the leader.
2021-06-28 12:20:53 -04:00
R.B. Boyer 30ccd5c2d9
connect: include optional partition prefixes in SPIFFE identifiers (#10507)
NOTE: this does not include any intentions enforcement changes yet
2021-06-25 16:47:47 -05:00
R.B. Boyer c3d5a2a5ab
connect/ca: cease including the common name field in generated certs (#10424)
As part of this change, we ensure that the SAN extensions are marked as
critical when the subject is empty so that AWS PCA tolerates the loss of
common names well and continues to function as a Connect CA provider.

Parts of this currently hack around a bug in crypto/x509 and can be
removed after https://go-review.googlesource.com/c/go/+/329129 lands in
a Go release.

Note: the AWS PCA tests do not run automatically, but the following
passed locally for me:

    ENABLE_AWS_PCA_TESTS=1 go test ./agent/connect/ca -run TestAWS
2021-06-25 13:00:00 -05:00
hc-github-team-consul-core f0f5d9bfc4 auto-updated agent/uiserver/bindata_assetfs.go from commit ace794d21 2021-06-25 09:47:01 +00:00
Dhia Ayachi 8b967b3bb6
return an empty record when asked for an addr dns with type other then A, AAAA and ANY (#10401)
* return an invalid record when asked for an addr dns with type other then A and AAAA

* add changelog

* fix ANY use case and add a test for it

* update changelog type

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* return empty response if the question record type do not match for addr

* set comment in the right place

* return A\AAAA record in extra section if record type is not A\AAAA for addr

* Fix failing test

* remove commented code

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* use require for test validation

* use variable to init struct

* fix failing test

* Update agent/dns.go

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* Update .changelog/10401.txt

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* Update agent/dns.go

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* Update agent/dns.go

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* Update agent/dns.go

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* fix compilation error

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-06-24 20:44:44 -04:00
Daniel Nephin cefbb5bf3f
Merge pull request #10476 from hashicorp/dnephin/ca-primary-uses-intermediate
ca: replace ca.PrimaryIntermediateProviders
2021-06-24 14:05:19 -04:00
R.B. Boyer 9778bee35a
structs: prohibit config entries from referencing more than one partition at a time (#10478)
affected kinds: service-defaults, ingress-gateway, terminating-gateway, service-intentions
2021-06-23 16:44:10 -05:00
R.B. Boyer 952df8b491
structs: prevent service-defaults upstream configs from using wildcard names or namespaces (#10475) 2021-06-23 15:48:54 -05:00
Daniel Nephin 72b30174fa ca: replace ca.PrimaryIntermediateProviders
With an optional interface that providers can use to indicate if they
use an intermediate cert in the primary DC.

This removes the need to look up the provider config when renewing the
intermediate.
2021-06-23 15:47:30 -04:00
R.B. Boyer b412ca0f89
structs: add some missing config entry validation and clean up tests (#10465)
Affects kinds: service-defaults, ingress-gateway, terminating-gateway
2021-06-23 14:11:23 -05:00
hc-github-team-consul-core a6a8e421d2 auto-updated agent/uiserver/bindata_assetfs.go from commit c78f7ecb2 2021-06-23 08:24:11 +00:00
Daniel Nephin 60614a3f68
Merge pull request #10444 from hashicorp/dnephin/tls-cert-exploration-2
tlsutil: reduce interface provided to auto-config
2021-06-22 15:32:51 -04:00
Daniel Nephin 86c9cb037f tlsutil: reduce interface provided to auto-config
Replace two methods with a single one that returns the cert. This moves more
of the logic into the single caller (auto-config).

tlsutil.Configurator is widely used. By keeping it smaller and focused only on storing and
returning TLS config, we make the code easier to follow.

These two methods were more related to auto-config than to tlsutil, so reducing the interface
moves the logic closer to the feature that requires it.
2021-06-22 14:11:28 -04:00
hc-github-team-consul-core 5add535d78 auto-updated agent/uiserver/bindata_assetfs.go from commit 043f631b7 2021-06-22 18:01:05 +00:00
hc-github-team-consul-core 605267d3e7 auto-updated agent/uiserver/bindata_assetfs.go from commit 4bddd5210 2021-06-22 13:24:58 +00:00
Daniel Nephin 6a61c5d772 proxycfg: remove unused method
This method was accidentally re-introduced in an earlier rebase. It was
removed in ed1082510dc80523b1f2a3a740fa5a13c77594f9 as part of the tproxy work.
2021-06-21 15:54:40 -04:00
Daniel Nephin 41bf0670a8 proxycfg: move each handler into a seprate file
There is no interaction between these handlers, so splitting them into separate files
makes it easier to discover the full implementation of each kindHandler.
2021-06-21 15:48:40 -04:00
hc-github-team-consul-core 27b6c61384 auto-updated agent/uiserver/bindata_assetfs.go from commit 5f17062b0 2021-06-21 11:11:47 +00:00
hc-github-team-consul-core 7b8af4dba4 auto-updated agent/uiserver/bindata_assetfs.go from commit 9eab71514 2021-06-21 10:59:56 +00:00
hc-github-team-consul-core 365ab6df11 auto-updated agent/uiserver/bindata_assetfs.go from commit ac424187f 2021-06-21 10:45:46 +00:00
Daniel Nephin f4c1f982d1
Merge pull request #9924 from hashicorp/dnephin/cert-expiration-metric
connect: emit a metric for the seconds until root CA expiry
2021-06-18 14:18:55 -04:00
Daniel Nephin 96896409d6
Merge pull request #9489 from hashicorp/dnephin/proxycfg-state-2
proxycfg: split state into a handler for each kind
2021-06-18 13:57:28 -04:00
Daniel Nephin 1da58902aa
Merge pull request #10425 from hashicorp/dnephin/tls-cert-exploration
tlsutil: fix a possible panic, and make the package safer
2021-06-18 13:54:07 -04:00
Daniel Nephin b0a2252fa0 inline assignment 2021-06-17 15:43:04 -04:00
Nitya Dhanushkodi ffbbe9e73f proxycfg: reference to entry in map should not panic 2021-06-17 11:49:04 -07:00
Daniel Nephin b7293242f1 Replace type conversion with embedded structs 2021-06-17 13:23:35 -04:00
Daniel Nephin 40ff895927 proxycfg: split state into kind-specific types
This commit extracts all the kind-specific logic into handler types, and
keeps the generic parts on the state struct. This change should make it
easier to add new kinds, and see the implementation of each kind more
clearly.
2021-06-16 14:04:01 -04:00
Daniel Nephin b57f03feff proxycfg: unmethod hostnameEndpoints
the method receiver can be replaced by the first argument.

This will allow us to extract more from the state struct in the future.
2021-06-16 14:03:30 -04:00
Daniel Nephin f2ae6cb47c Remove duplicate import
because two PRs crossed paths.
2021-06-16 13:19:54 -04:00
Daniel Nephin b40174ccf2
Merge pull request #9466 from hashicorp/dnephin/proxycfg-state
proxycfg: prepare state for split by kind
2021-06-16 13:14:26 -04:00
R.B. Boyer 38d4b75ab3
xds: fix flaky protocol tests (#10410) 2021-06-16 11:57:43 -05:00
Freddy 3127dac0fb
Merge pull request #10404 from hashicorp/ingress-stats 2021-06-15 14:28:07 -06:00
R.B. Boyer 1645b6aafe
xds: adding more delta protocol tests (#10398)
Fixes #10125
2021-06-15 15:21:07 -05:00
freddygv bdb3af918c Regen golden files 2021-06-15 14:18:25 -06:00
Freddy 0d05cbb105
Update agent/xds/listeners.go
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-06-15 14:09:26 -06:00
Freddy 0e417e006e
Omit empty tproxy config in JSON responses (#10402) 2021-06-15 13:53:35 -06:00
Nitya Dhanushkodi 08ed3edf71
proxycfg: Ensure that endpoints for explicit upstreams in other datacenters are watched in transparent mode (#10391)
Co-authored-by: Freddy Vallenilla <freddy@hashicorp.com>
2021-06-15 11:00:26 -07:00
freddygv e07542211a Remove unused param 2021-06-15 11:19:45 -06:00
Dhia Ayachi 4b75f15fb7
improve monitor performance (#10368)
* remove flush for each write to http response in the agent monitor endpoint

* fix race condition when we stop and start monitor multiple times, the doneCh is closed and never recover.

* start log reading goroutine before adding the sink to avoid filling the log channel before getting a chance of reading from it

* flush every 500ms to optimize log writing in the http server side.

* add changelog file

* add issue url to changelog

* fix changelog url

* Update changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* use ticker to flush and avoid race condition when flushing in a different goroutine

* stop the ticker when done

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* Revert "fix race condition when we stop and start monitor multiple times, the doneCh is closed and never recover."

This reverts commit 1eeddf7a

* wait for log consumer loop to start before registering the sink

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-06-15 12:05:52 -04:00
freddygv 450ac44126 Update ingress gateway stats labeling
In the absence of stats_tags to handle this pattern, when we pass
"ingress_upstream.$port" as the stat_prefix, Envoy splits up that prefix
and makes the port a part of the metric name.

For example:
- stat_prefix: ingress_upstream.8080

This leads to metric names like envoy_http_8080_no_route. Changing the
stat_prefix to ingress_upstream_80880 yields the expected metric names
such as envoy_http_no_route.

Note that we don't encode the destination's name/ns/dc in this
stat_prefix because for HTTP services ingress gateways use a single
filter chain. Only cluster metrics are available on a per-upstream
basis.
2021-06-15 08:52:18 -06:00
freddygv 5d7188e290 Update terminating gateway stats labeling
This change makes it so that the stat prefix for terminating gateways
matches that of connect proxies. By using the structure of
"upstream.svc.ns.dc" we can extract labels for the destination service,
namespace, and datacenter.
2021-06-15 08:52:18 -06:00
R.B. Boyer 8d5f81b460
xds: ensure that dependent xDS resources are reconfigured during primary type warming (#10381)
Updates to a cluster will clear the associated endpoints, and updates to
a listener will clear the associated routes. Update the incremental xDS
logic to account for this implicit cleanup so that we can finish warming
the clusters and listeners.

Fixes #10379
2021-06-14 17:20:27 -05:00
Daniel Nephin e36800cefa Update metric name
and handle the case where there is no active root CA.
2021-06-14 17:01:16 -04:00
Daniel Nephin 548796ae13 connect: emit a metric for the number of seconds until root CA expiration 2021-06-14 16:57:01 -04:00
Freddy f399fd2add
Rename CatalogDestinationsOnly (#10397)
CatalogDestinationsOnly is a passthrough that would enable dialing
addresses outside of Consul's catalog. However, when this flag is set to
true only _connect_ endpoints for services can be dialed.

This flag is being renamed to signal that non-Connect endpoints can't be
dialed by transparent proxies when the value is set to true.
2021-06-14 14:15:09 -06:00
Freddy f19b1f0058
Relax validation for expose.paths config (#10394)
Previously we would return an error if duplicate paths were specified.
This could lead to problems in cases where a user has the same path,
say /healthz, on two different ports.

This validation was added to signal a potential misconfiguration.
Instead we will only check for duplicate listener ports, since that is
what would lead to ambiguity issues when generating xDS config.

In the future we could look into using a single listener and creating
distinct filter chains for each path/port.
2021-06-14 14:04:11 -06:00
Daniel Nephin cbcc1a3a86 proxycfg: extract two types from state struct
These two new struct types will allow us to make polymorphic handler for each kind, instad of
having all the logic for each proxy kind on the state struct.
2021-06-10 17:42:17 -04:00
Daniel Nephin b99da95e70 proxycfg: pass context around where it is needed
context.Context should never be stored on a struct (as it says in the godoc) because it is easy to
to end up with the wrong context when it is stored.

Also see https://blog.golang.org/context-and-structs

This change is also in preparation for splitting state into kind-specific handlers so that the
implementation of each kind is grouped together.
2021-06-10 17:34:50 -04:00
Daniel Nephin 95315e9e06 http: add PrimaryDatacenter to the /v1/agent/self response
This field is available in DebugConfig, but that field is not stable and could change at any time.

The consul-k8s needs to be able to detect the primary DC for tests, so adding this field to the
stable part of the API response.
2021-06-10 17:19:16 -04:00
Freddy 61ae2995b7
Add flag for transparent proxies to dial individual instances (#10329) 2021-06-09 14:34:17 -06:00
Daniel Nephin b5503223ae submatview: add test cases for store.Get with timeout and no index
Also set a more unique name for the serviceRequest.Type to prevent potential name conflicts
in the future.
2021-06-08 18:04:38 -04:00
Daniel Nephin 450cce60a1
Merge pull request #10364 from hashicorp/dnephin/streaming-e2e-test
submatview: and Store integration test with stream backend
2021-06-08 16:13:45 -04:00
Freddy 62facc1a04
Revert "Avoid adding original_dst filter when not needed" (#10365) 2021-06-08 13:18:41 -06:00
Daniel Nephin b8717966f1 submatview: and Store integration test with stream backend 2021-06-08 12:15:35 -04:00
Daniel Nephin 20f7a72792 stream: remove bufferItem.NextLink
Both NextLink and NextNoBlock had the same logic, with slightly
different return values. By adding a bool return value (similar to map
lookups) we can remove the duplicate method.
2021-06-07 17:04:46 -04:00
Daniel Nephin 48f388f590 stream: fix a bug with creating a snapshot
The head of the topic buffer was being ignored when creating a snapshot. This commit fixes
the bug by ensuring that the head of the topic buffer is included in the snapshot
before handing it off to the subscription.
2021-06-04 18:33:04 -04:00
Daniel Nephin 4fb6c5a137 submatview: fix a bug with Store.Get
When info.Timeout is 0, it should have no timeout. Previously it was using a 0 duration timeout
which caused it to return without waiting.

This bug was masked by using a timeout in the tests. Removing the timeout caused the tests to fail.
2021-06-03 17:48:44 -04:00
Paul Ewing e454a9aae0
usagemetrics: add cluster members to metrics API (#10340)
This PR adds cluster members to the metrics API. The number of members per
segment are reported as well as the total number of members.

Tested by running a multi-node cluster locally and ensuring the numbers were
correct. Also added unit test coverage to add the new expected gauges to
existing test cases.
2021-06-03 08:25:53 -07:00
Daniel Nephin 0dfb7da610 grpc: fix a data race by using a static resolver
We have seen test flakes caused by 'concurrent map read and map write', and the race detector
reports the problem as well (prevent us from running some tests with -race).

The root of the problem is the grpc expects resolvers to be registered at init time
before any requests are made, but we were using a separate resolver for each test.

This commit introduces a resolver registry. The registry is registered as the single
resolver for the consul scheme. Each test uses the Authority section of the target
(instead of the scheme) to identify the resolver that should be used for the test.
The scheme is used for lookup, which is why it can no longer be used as the unique
key.

This allows us to use a lock around the map of resolvers, preventing the data race.
2021-06-02 11:35:38 -04:00
Daniel Nephin 2dcfe4a0d5 submatview: improve a couple comments 2021-06-01 17:49:31 -04:00
Dhia Ayachi 9f2f9ac3a5
make tests use a dummy node_name to avoid environment related failures (#10262)
* fix tests to use a dummy nodeName and not fail when hostname is not a valid nodeName

* remove conditional testing

* add test when node name is invalid
2021-06-01 11:58:03 -04:00
Daniel Nephin dcf80907a9 structs: fix cache keys
So that requests are cached properly, and the cache does not return the wrong data for a
request.
2021-05-31 17:22:16 -04:00
Daniel Nephin 857799cd56 structs: add two cache completeness tests types that implement cache.Request 2021-05-31 16:54:41 -04:00
Daniel Nephin 01790fbcb7 structs: improve the interface of assertCacheInfoKeyIsComplete 2021-05-31 16:54:41 -04:00
Daniel Nephin 9de439f66a structs: Add more cache key tests 2021-05-31 16:54:40 -04:00
Dhia Ayachi 0c13f80d5a
RPC Timeout/Retries account for blocking requests (#8978) 2021-05-27 17:29:43 -04:00
hc-github-team-consul-core aad8acb6ad auto-updated agent/uiserver/bindata_assetfs.go from commit 18190fb07 2021-05-27 15:00:34 +00:00
Dhia Ayachi 00f7e0772a
debug: remove the CLI check for debug_enabled (#10273)
* debug: remove the CLI check for debug_enabled

The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.

Also:
- fix the API client to return errors on non-200 status codes for debug
  endpoints
- improve the failure messages when pprof data can not be collected

Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>

* remove parallel test runs

parallel runs create a race condition that fail the debug tests

* Add changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-05-27 09:41:53 -04:00
hc-github-team-consul-core c2dc8cf56b auto-updated agent/uiserver/bindata_assetfs.go from commit ddee7afbb 2021-05-27 12:29:12 +00:00
Freddy c61e2bbda7
Ensure passthrough clusters can be created (#10301) 2021-05-26 15:05:14 -06:00
Freddy 7cfd7e9ec1
Avoid adding original_dst filter when not needed (#10302) 2021-05-26 15:04:45 -06:00
Matt Keeler 7e4ea16149 Move some things around to allow for license updating via config reload
The bulk of this commit is moving the LeaderRoutineManager from the agent/consul package into its own package: lib/gort. It also got a renaming and its Start method now requires a context. Requiring that context required updating a whole bunch of other places in the code.
2021-05-25 09:57:50 -04:00
Dhia Ayachi 6aa915db8d
upgrade golangci-lint to v1.40.1 (#10276)
Also: fix linter issue detected with newer version
2021-05-24 22:22:37 -04:00
Matt Keeler 58b934133d hcs-1936: Prepare for adding license auto-retrieval to auto-config in enterprise 2021-05-24 13:20:30 -04:00
Matt Keeler 82f5cb3f08 Preparation for changing where license management is done. 2021-05-24 10:19:31 -04:00
hc-github-team-consul-core 9f765b2582 auto-updated agent/uiserver/bindata_assetfs.go from commit 600f85753 2021-05-24 11:37:54 +00:00
hc-github-team-consul-core 5e3aa6c7ae auto-updated agent/uiserver/bindata_assetfs.go from commit dd4a66808 2021-05-24 10:56:33 +00:00
Daniel Nephin 21f35ab863
Merge pull request #10272 from hashicorp/dnephin/backport-namespace-license-fix
Backport some ent changes for serf tags
2021-05-21 12:31:34 -04:00
Matt Keeler 84c6c56578 Add OSS bits for supporting specifying the enterprise license via config 2021-05-20 16:11:33 -04:00
Daniel Nephin f2cf586414 Refactor of serf feature flag tags.
This refactor is to make it easier to see how serf feature flags are
encoded as serf tags, and where those feature flags are read.

- use constants for both the prefix and feature flag name. A constant
  makes it much easier for an IDE to locate the read and write location.
- isolate the feature-flag encoding logic in the metadata package, so
  that the feature flag prefix can be unexported. Only expose a function
  for encoding the flags into tags. This logic is now next to the logic
  which reads the tags.
- remove the duplicate `addEnterpriseSerfTags` functions. Both Client
  and Server structs had the same implementation. And neither
  implementation needed the method receiver.
2021-05-20 12:57:06 -04:00
Daniel Nephin d9959ba811
Merge pull request #10200 from hashicorp/dnephin/backport-audit-log-config-changes
config: backport audit log config changes from enterprise
2021-05-19 10:58:28 -04:00
hc-github-team-consul-core dd040e13d6 auto-updated agent/uiserver/bindata_assetfs.go from commit 39302041e 2021-05-19 10:11:29 +00:00
Joshua Shanks 9e4051ec65 GH-8728 add raft default values 2021-05-18 14:51:14 -04:00
hc-github-team-consul-core 2730185ac6 auto-updated agent/uiserver/bindata_assetfs.go from commit 8301e79c5 2021-05-18 15:35:50 +00:00
hc-github-team-consul-core 2350928f7a auto-updated agent/uiserver/bindata_assetfs.go from commit d1bbe0895 2021-05-17 12:32:31 +00:00
R.B. Boyer 7c9763d027
xds: emit a labeled gauge of connected xDS streams by version (#10243)
Fixes #10099
2021-05-14 13:59:13 -05:00
R.B. Boyer b90877b440
server: ensure that central service config flattening properly resets the state each time (#10239)
The prior solution to call reply.Reset() aged poorly since newer fields
were added to the reply, but not added to Reset() leading serial
blocking query loops on the server to blend replies.

This could manifest as a service-defaults protocol change from
default=>http not reverting back to default after the config entry
reponsible was deleted.
2021-05-14 10:21:44 -05:00
R.B. Boyer c42899eafa
agent: ensure we hash the non-deprecated upstream fields on ServiceConfigRequest (#10240) 2021-05-14 10:15:48 -05:00
hc-github-team-consul-core 2416a6ddde auto-updated agent/uiserver/bindata_assetfs.go from commit 04bd57617 2021-05-13 10:42:23 +00:00
Iryna Shustava 7a41dbd9b6
Save exposed ports in agent's store and expose them via API (#10173)
* Save exposed HTTP or GRPC ports to the agent's store
* Add those the health checks API so we can retrieve them from the API
* Change redirect-traffic command to also exclude those ports from inbound traffic redirection when expose.checks is set to true.
2021-05-12 13:51:39 -07:00
Daniel Nephin 1a22454704
Merge pull request #10217 from hashicorp/dnephin/test-flakes
testing: attempt to fix some test flakes
2021-05-12 15:38:29 -04:00
R.B. Boyer 05b52a3d63
connect: update supported envoy versions to 1.18.3, 1.17.3, 1.16.4, and 1.15.5 (#10231) 2021-05-12 14:06:06 -05:00
Kyle Havlovitz c2382203b7
Merge pull request #9995 from tarat44/fix-close-h2ping-client-connections
Add graceful shutdown of client connection and WaitGroup to H2PING
2021-05-11 10:45:26 -07:00
hc-github-team-consul-core 4699313f19 auto-updated agent/uiserver/bindata_assetfs.go from commit ecbeaa87c 2021-05-11 15:39:38 +00:00