Commit Graph

695 Commits

Author SHA1 Message Date
Daniel Nephin 064c43ee69 debug: restore cancel on SigInt
Some previous changes broke interrupting the debug on SigInterupt. This change restores
the original behaviour by passing a context to requests.

Since a new API client function was required to pass the context, I had
it also return an io.ReadCloser, so that output can be streamed to files
instead of fully buffering in process memory.
2021-08-18 12:29:34 -04:00
R.B. Boyer e50e13d2ab
state: partition nodes and coordinates in the state store (#10859)
Additionally:

- partitioned the catalog indexes appropriately for partitioning
- removed a stray reference to a non-existent index named "node.checks"
2021-08-17 13:29:39 -05:00
Daniel Nephin baa27e2b4c api: remove client library tests for old API endpoints
These methods are being kept around in the api module for now to allow the api module
to work with older versions of the HTTP API, but we are no longer able to test them.

Since the endpoints are removed there is no way for them to change, so there does not
appear to be a need to keep running the tests.
2021-08-17 13:09:30 -04:00
Kyle Havlovitz 98969c018a oss: Rename default partition 2021-08-12 14:31:37 -07:00
Mark Anderson 03a3ec2b55
Fixup to support unix domain socket via command line (#10758)
Missed the need to add support for unix domain socket config via
api/command line. This is a variant of the problems described in
it is easy to drop one.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-08-12 10:05:22 -07:00
Daniel Nephin 87fb26fd65
Merge pull request #10612 from bigmikes/acl-replication-fix
acl: acl replication routine to report the last error message
2021-08-06 18:29:51 -04:00
Giulio Micheloni 5c34a48d45 String type instead of error type and changelog. 2021-08-06 22:35:27 +01:00
Daniel Nephin 475fec5670
Merge pull request #10632 from hashicorp/pairing/acl-authorizer-when-acl-disabled
acls: Update ACL authorizer to return meaningful permission when ACLs are disabled
2021-07-30 13:22:55 -04:00
Blake Covarrubias 766b64eac2
api: Support QueryOptions on additional agent endpoints (#10691)
Add support for setting QueryOptions on the following agent API endpoints:

- /agent/health/service/name/:name
- /agent/health/service/id/:id
- /agent/service/maintenance/:id

This follows the same pattern used in #9903 to support query options
for other agent API endpoints.

Resolves #9710
2021-07-30 10:07:13 -07:00
Evan Culver da96c645b2 Fix api/agent test 2021-07-30 12:58:46 -04:00
Evan Culver b628f6235d Fix api/agent test 2021-07-30 12:58:46 -04:00
Daniel Nephin d716f709fd debug: use the new metrics stream in debug command 2021-07-26 17:53:32 -04:00
R.B. Boyer b2facb35a9
replumbing a bunch of api and agent structs for partitions (#10681) 2021-07-22 14:33:22 -05:00
R.B. Boyer 254557a1f6
sync changes to oss files made in enterprise (#10670) 2021-07-22 13:58:08 -05:00
Evan Culver 521c423075
acls: Show `AuthMethodNamespace` when reading/listing ACL token meta (#10598) 2021-07-15 10:38:52 -07:00
Kyle Havlovitz e97bc2bda7 http: add partition query param parsing 2021-07-14 12:07:38 -07:00
Evan Culver db250e2114
tests: skip cases that depend on test server when -short flag set (#10576) 2021-07-09 16:42:30 -07:00
Evan Culver 5ff191ad99
Add support for returning ACL secret IDs for accessors with acl:write (#10546) 2021-07-08 15:13:08 -07:00
Daniel Nephin dcb90fb832
Merge pull request #10570 from hashicorp/copy-of-master
Changes that were accidentally merged into the old master branch
2021-07-08 16:28:56 -04:00
Jared Kirschner 37f25aed1d
Merge pull request #10559 from jkirschner-hashicorp/fix-autopilot-config-post-default-values
Fix defaults for autopilot config update
2021-07-06 19:19:52 -04:00
Jared Kirschner 1449806c3d Fix defaults for autopilot config update
Previously, for a POST request to the /v1/operator/autopilot/configuration
endpoint, any fields not included in the payload were set to a zero-initialized
value rather than the documented default value.

Now, if an optional field is not included in the payload, it will be set to its
documented default value:
- CleanupDeadServers:      true
- LastContactThreshold:    "200ms"
- MaxTrailingLogs:         250
- MinQuorum:               0
- ServerStabilizationTime: "10s"
- RedundancyZoneTag:       ""
- DisableUpgradeMigration: false
- UpgradeVersionTag:       ""
2021-07-06 18:39:40 -04:00
Daniel Nephin b4a10443d1 ca: remove unused RotationPeriod field
This field was never used. Since it is persisted as part of a map[string]interface{} it
is pretty easy to remove it.
2021-07-05 19:15:44 -04:00
Kyle Nusbaum 89ec847f7b api: change io.Discard to ioutil.Discard 2021-06-23 11:45:20 -05:00
Daniel Nephin de1f766d03 api: properly close the response body
reads resp.Body until EOF, so that the http client is able to re-use the TCP connection.
2021-06-14 18:52:59 -04:00
Freddy f399fd2add
Rename CatalogDestinationsOnly (#10397)
CatalogDestinationsOnly is a passthrough that would enable dialing
addresses outside of Consul's catalog. However, when this flag is set to
true only _connect_ endpoints for services can be dialed.

This flag is being renamed to signal that non-Connect endpoints can't be
dialed by transparent proxies when the value is set to true.
2021-06-14 14:15:09 -06:00
Freddy 61ae2995b7
Add flag for transparent proxies to dial individual instances (#10329) 2021-06-09 14:34:17 -06:00
Dhia Ayachi 00f7e0772a
debug: remove the CLI check for debug_enabled (#10273)
* debug: remove the CLI check for debug_enabled

The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.

Also:
- fix the API client to return errors on non-200 status codes for debug
  endpoints
- improve the failure messages when pprof data can not be collected

Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>

* remove parallel test runs

parallel runs create a race condition that fail the debug tests

* Add changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-05-27 09:41:53 -04:00
Rémi Lapeyre 5e33428b22
Always set the Content-Type header when a body is present (#10204)
* Always set the Content-Type header when a body is present

Closes https://github.com/hashicorp/consul/issues/10011

* Add Changelog entry

* Add more Content-Type exceptions

* Fix tests
2021-05-25 16:03:48 +01:00
Matt Keeler 64359617b5 Deprecate API driven licensing.
The two methods in the API client to Put or Reset a license will now always return an error.
2021-05-21 11:08:50 -04:00
Iryna Shustava 7a41dbd9b6
Save exposed ports in agent's store and expose them via API (#10173)
* Save exposed HTTP or GRPC ports to the agent's store
* Add those the health checks API so we can retrieve them from the API
* Change redirect-traffic command to also exclude those ports from inbound traffic redirection when expose.checks is set to true.
2021-05-12 13:51:39 -07:00
Daniel Nephin 5b1c14a95e api: fix mesh config entry tests for default namespace 2021-05-06 13:25:32 -04:00
Mark Anderson d911041082 Fixup checksums for tests
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 10:13:41 -07:00
Mark Anderson a78a891d56 Fixup filtering tests
Missed annotations for SocketPath, LocalSocketPath, LocalServiceSocketPath

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 09:28:17 -07:00
Daniel Nephin 55f620d636
Merge pull request #10155 from hashicorp/dnephin/config-entry-remove-fields
config-entry: remove Kind and Name field from Mesh config entry
2021-05-04 17:27:56 -04:00
Daniel Nephin 0e5e1270b6 config-entries: add a test for the API client
Also fixes a bug with listing kind=mesh config entries. ValidateConfigEntryKind was only being used by
the List endpoint, and was yet another place where we have to enumerate all the kinds.

This commit removes ValidateConfigEntryKind and uses MakeConfigEntry instead. This change removes
the need to maintain two separate functions at the cost of creating an instance of the config entry which will be thrown away immediately.
2021-05-04 17:14:21 -04:00
Mark Anderson 10963d0cbd Add support for downstreams
Enhance config by adding SocketPath and LocalSocketPath config values

Supports syntax of the form:
```
services {
  name = "sock_forwarder"
  id = "sock_forwarder.1"
  socket_path = "/tmp/downstream_3.sock"
  connect {
    sidecar_service {
      proxy {
	local_service_socket_path = "/tmp/downstream.sock"
      }
    }
  }
}
```

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 5f04b6abe2 Convert mode to string representation
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 1d9a7df5d4 First changes for unix domain sockets upstreams
Start making structure changes to support unix domain socket address for upstreams

	upstreams = [
	  {
	    destination_name = "echo-service"
	    local_bind_socket_path = "/tmp/upstream.sock"
	    config {
	      passive_health_check {
		interval = "10s"
		max_failures = 42
	      }
	    }
	  }

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Daniel Nephin fcfd244269 config-entry: use custom MarshalJSON for mesh type
So that the Kind field is added to the JSON object.
2021-04-29 17:49:43 -04:00
Daniel Nephin bf4c289804 config-entry: remove Kind and Name field from Mesh config entry
No config entry needs a Kind field. It is only used to determine the Go type to
target. As we introduce new config entries (like this one) we can remove the kind field
and have the GetKind method return the single supported value.

In this case (similar to proxy-defaults) the Name field is also unnecessary. We always
use the same value. So we can omit the name field entirely.
2021-04-29 17:11:21 -04:00
Freddy 401f3010e0
Rename "cluster" config entry to "mesh" (#10127)
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.

Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
2021-04-28 16:13:29 -06:00
R.B. Boyer c88512fe14
connect: update centralized upstreams representation in service-defaults (#10015) 2021-04-15 14:21:44 -05:00
freddygv 50c7915156 Convert new tproxy structs in api module into ptrs
This way we avoid serializing these when empty. Otherwise users of the
latest version of the api submodule cannot interact with older versions
of Consul, because a new api client would send keys that the older Consul
doesn't recognize yet.
2021-04-13 12:44:25 -06:00
freddygv eeccba945d Replace TransparentProxy bool with ProxyMode
This PR replaces the original boolean used to configure transparent
proxy mode. It was replaced with a string mode that can be set to:

- "": Empty string is the default for when the setting should be
defaulted from other configuration like config entries.
- "direct": Direct mode is how applications originally opted into the
mesh. Proxy listeners need to be dialed directly.
- "transparent": Transparent mode enables configuring Envoy as a
transparent proxy. Traffic must be captured and redirected to the
inbound and outbound listeners.

This PR also adds a struct for transparent proxy specific configuration.
Initially this is not stored as a pointer. Will revisit that decision
before GA.
2021-04-12 09:35:14 -06:00
freddygv 9c219a5b58 Fixup mesh gateway docs 2021-04-11 15:48:04 -06:00
Freddy 920ba3db39
Merge pull request #9976 from hashicorp/centralized-upstream-fixups 2021-04-08 12:26:56 -06:00
freddygv 4ddf754b6d Fixup test 2021-04-08 11:53:07 -06:00
Daniel Nephin db40e937b7
Merge pull request #9925 from hashicorp/dnephin/update-memberlist
Update memberlist to v0.2.3
2021-04-08 12:17:29 -04:00
freddygv 43091aae60 Avoid sending zero-value upstream defaults from api 2021-04-07 15:03:42 -06:00
R.B. Boyer 82245585c6
connect: add toggle to globally disable wildcard outbound network access when transparent proxy is enabled (#9973)
This adds a new config entry kind "cluster" with a single special name "cluster" where this can be controlled.
2021-04-06 13:19:59 -05:00
Daniel Nephin 78cb867c8e Update memberlist to v0.2.3
To pickup data race fixes
2021-03-24 18:20:19 -04:00
Seth Hoenig dcbe30ef08 api: enable query options on agent endpoints
This PR adds support for setting QueryOptions on a few agent API
endpoints. Nomad needs to be able to set the Namespace field on
these endpoints to:
 - query for services / checks in a namespace
 - deregister services / checks in a namespace
 - update TTL status on checks in a namespace
2021-03-19 13:08:26 -05:00
Freddy 63dcb7fa76
Add TransparentProxy option to proxy definitions 2021-03-17 17:01:45 -06:00
Freddy fb252e87a4
Add per-upstream configuration to service-defaults 2021-03-17 16:59:51 -06:00
Christopher Broglie 94b02c3954 Add support for configuring TLS ServerName for health checks
Some TLS servers require SNI, but the Golang HTTP client doesn't
include it in the ClientHello when connecting to an IP address. This
change adds a new TLSServerName field to health check definitions to
optionally set it. This fixes #9473.
2021-03-16 18:16:44 -04:00
freddygv 871e1d3e31 PR comments 2021-03-15 16:02:03 -06:00
freddygv d90240d367 Restore old Envoy prefix on escape hatches
This is done because after removing ID and NodeName from
ServiceConfigRequest we will no longer know whether a request coming in
is for a Consul client earlier than v1.10.
2021-03-15 14:12:57 -06:00
freddygv f584c2d7c5 Add omitempty across the board for UpstreamConfig 2021-03-15 13:23:18 -06:00
Matt Keeler 8d09d610dd
AutopilotServerHealth now handles the 429 status code (#8599)
AutopilotServerHealthy now handles the 429 status code

Previously we would error out and not parse the response. Now either a 200 or 429 status code are considered expected statuses and will result in the method returning the reply allowing API consumers to not only see if the system is healthy or not but which server is unhealthy.
2021-03-12 09:40:49 -05:00
freddygv eeed9011b4 And another test fix 2021-03-11 18:39:53 -07:00
Kyle Havlovitz 237b41ac8f
Merge pull request #9672 from hashicorp/ca-force-skip-xc
connect/ca: Allow ForceWithoutCrossSigning for all providers
2021-03-11 11:49:15 -08:00
freddygv 7a3625f58b Add TransparentProxy opt to proxy definition 2021-03-11 11:37:21 -07:00
freddygv c30157d2f2 Turn Limits and PassiveHealthChecks into pointers 2021-03-11 11:04:40 -07:00
freddygv 4bbd495b54 Create new types for service-defaults upstream cfg 2021-03-08 22:10:27 -07:00
Mark Anderson 1cf6a435dc
Add fields to the /acl/auth-methods endpoint. (#9741)
* A GET of the /acl/auth-method/:name endpoint returns the fields
MaxTokenTTL and TokenLocality, while a LIST (/acl/auth-methods) does
not.

The list command returns a filtered subset of the full set. This is
somewhat deliberate, so that secrets aren't shown, but the TTL and
Locality fields aren't (IMO) security critical, and it is useful for
the front end to be able to show them.

For consistency these changes mirror the 'omit empty' and string
representation choices made for the GET call.

This includes changes to the gRPC and API code in the client.

The new output looks similar to this
curl 'http://localhost:8500/v1/acl/auth-methods' | jq '.'

  {
    "MaxTokenTTL": "8m20s",
    "Name": "minikube-ttl-local2",
    "Type": "kubernetes",
    "Description": "minikube auth method",
    "TokenLocality": "local",
    "CreateIndex": 530,
    "ModifyIndex": 530,
    "Namespace": "default"
  }
]

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Add changelog

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-02-17 08:16:57 -08:00
Kyle Havlovitz 1dee4173c1 connect/ca: Allow ForceWithoutCrossSigning for all providers
This allows setting ForceWithoutCrossSigning when reconfiguring the CA
for any provider, in order to forcibly move to a new root in cases where
the old provider isn't reachable or able to cross-sign for whatever
reason.
2021-01-29 13:38:11 -08:00
Conor Mongey d648e6511e
Move header methods from config to client 2021-01-20 01:30:54 +00:00
Conor Mongey e2a2ebeb9e
Only override headers if they're set 2021-01-20 01:12:19 +00:00
Conor Mongey e4b4c31d28
Prefer http.Header over map[string]string to allow for multi-valued headers 2021-01-20 01:12:19 +00:00
Conor Mongey 9064930f08
Allow setting arbitrary headers in API client 2021-01-20 01:12:19 +00:00
Daniel Nephin 489cedbac5
Merge pull request #8609 from hashicorp/dnephin/add-query-options-to-ServiceRegister
api: Add a context to ServiceRegisterOpts
2021-01-06 18:52:49 -05:00
Mike Morris 67a11e4d16 Merge pull request #9270 from hashicorp/release/1.9.0
merge: release/1.9.0 back into 1.9.x
2020-11-24 17:36:47 -05:00
R.B. Boyer 7bcbc59dea
command: when generating envoy bootstrap configs use the datacenter returned from the agent services endpoint (#9229)
Fixes #9215
2020-11-19 15:27:31 -06:00
Matt Keeler 4bca029be9
Refactor to call non-voting servers read replicas (#9191)
Co-authored-by: Kit Patella <kit@jepsen.io>
2020-11-17 10:53:57 -05:00
Freddy 2763833d32
Add DC and NS support for Envoy metrics (#9207)
This PR updates the tags that we generate for Envoy stats.

Several of these come with breaking changes, since we can't keep two stats prefixes for a filter.
2020-11-16 16:37:19 -07:00
Matt Keeler 372f21767f
Remove this constant as it is soon to be changing and we want to prevent backwards compat issues (#9193) 2020-11-13 17:09:51 -05:00
R.B. Boyer a5bd1ba323
agent: return the default ACL policy to callers as a header (#9101)
Header is: X-Consul-Default-ACL-Policy=<allow|deny>

This is of particular utility when fetching matching intentions, as the
fallthrough for a request that doesn't match any intentions is to
enforce using the default acl policy.
2020-11-12 10:38:32 -06:00
Matt Keeler 58f98db227
Add a CLI command for retrieving the autopilot configuration. (#9142) 2020-11-11 13:19:02 -05:00
Matt Keeler 755fb72994
Switch to using the external autopilot module 2020-11-09 09:22:11 -05:00
R.B. Boyer 9b37ea7dcb
Revert "Add namespace support for metrics (OSS) (#9117)" (#9124)
This reverts commit 06b3b017d326853dbb53bc0ec08ce371265c5ce9.
2020-11-06 10:24:32 -06:00
Freddy 874efe705f
Add namespace support for metrics (OSS) (#9117) 2020-11-05 18:24:29 -07:00
R.B. Boyer d2f09ca306
upstream some differences from enterprise (#8902) 2020-10-09 09:42:53 -05:00
Matt Keeler 141eb60f06
Add per-agent reconnect timeouts (#8781)
This allows for client agent to be run in a more stateless manner where they may be abruptly terminated and not expected to come back. If advertising a per-agent reconnect timeout using the advertise_reconnect_timeout configuration when that agent leaves, other agents will wait only that amount of time for the agent to come back before reaping it.

This has the advantageous side effect of causing servers to deregister the node/services/checks for that agent sooner than if the global reconnect_timeout was used.
2020-10-08 15:02:19 -04:00
R.B. Boyer 140c88a9f4
api: unflake some intention-related api tests (#8857) 2020-10-07 13:32:53 -05:00
R.B. Boyer 35c4efd220
connect: support defining intentions using layer 7 criteria (#8839)
Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).
2020-10-06 17:09:13 -05:00
R.B. Boyer d6dce2332a
connect: intentions are now managed as a new config entry kind "service-intentions" (#8834)
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
R.B. Boyer e84c032ea7
api: support GetMeta() and GetNamespace() on all config entry kinds (#8764)
Fixes #8755

Since I was updating the interface, i also added the missing `GetNamespace()`.

Depending upon how you look at it, this is a breaking change since it adds methods to the exported interface `api.ConfigEntry`. Given that you cannot define your own config entry kinds, and all of the machinery of the `api.Client` acts like a factory to construct the canned ones from the rest of the module, this feels like it's not a problematic change as it would only break someone who had reimplemented the `ConfigEntry` interface themselves for no apparent utility?
2020-09-29 09:11:57 -05:00
freddygv 33af8dab9a Resolve conflicts against master 2020-09-11 18:41:58 -06:00
freddygv 60cb306524 Add session flag to cookie config 2020-09-11 18:34:03 -06:00
freddygv 5871b667a5 Revert EnvoyConfig nesting 2020-09-11 09:21:43 -06:00
Seth Hoenig 99b822c4b3
api: create fresh http client for unix sockets (#8602)
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2020-09-06 12:27:39 -04:00
Freddy cfc4283c60
Make LockDelay configurable in api locks (#8621) 2020-09-04 13:38:26 -06:00
Daniel Nephin b7bfa5e6df api: Add a context to ServiceRegisterOpts
So that operations can timeout instead of hanging forever
2020-09-03 12:25:06 -04:00
freddygv 02d6acd8fc Ensure resolver node with LB isn't considered default 2020-09-03 08:55:57 -06:00
freddygv daad3b9210 Remove LB infix and move injection to xds 2020-09-02 15:13:50 -06:00
R.B. Boyer b0bde51e70
connect: all config entries pick up a meta field (#8596)
Fixes #8595
2020-09-02 14:10:25 -05:00
freddygv d7bda050e0 Restructure structs and other PR comments 2020-09-02 09:10:50 -06:00
freddygv 58a018c20b Add documentation for resolver LB cfg 2020-08-28 14:46:13 -06:00
freddygv 391d569a45 Add LB policy to service-resolver 2020-08-27 19:44:02 -06:00
Jack 145bcdc2bb
Add http2 and grpc support to ingress gateways (#8458) 2020-08-27 15:34:08 -06:00