Commit Graph

134 Commits

Author SHA1 Message Date
Freddy cb55fa3742
Enable CLI to register terminating gateways (#7500)
* Enable CLI to register terminating gateways

* Centralize gateway proxy configuration
2020-03-26 10:20:56 -06:00
Akshay Ganeshen fd32016ce9
feat: support sending body in HTTP checks (#6602) 2020-02-10 09:27:12 -07:00
Fouad Zaryouh b33cf06cf7
api: add replace-existing-checks param to the api package (#7136) 2020-02-03 10:11:40 +01:00
Chris Piraino 3dd0b59793
Allow users to configure either unstructured or JSON logging (#7130)
* hclog Allow users to choose between unstructured and JSON logging
2020-01-28 17:50:41 -06:00
neillyt 5f5085aa75 Fix Namespace Typo (#7030) 2020-01-16 20:34:04 -07:00
Matt Keeler 442924c35a
Sync of OSS changes to support namespaces (#6909) 2019-12-09 21:26:41 -05:00
Freddy caf658d0d3
Store check type in catalog (#6561) 2019-10-17 20:33:11 +02:00
Sarah Christoff 9b93dd93c9
Prune Unhealthy Agents (#6571)
* Add -prune flag to ForceLeave
2019-10-04 16:10:02 -05:00
Freddy 5eace88ce2
Expose HTTP-based paths through Connect proxy (#6446)
Fixes: #5396

This PR adds a proxy configuration stanza called expose. These flags register
listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only
listening on the loopback interface, while still accepting traffic from non
Connect-enabled services.

Under expose there is a boolean checks flag that would automatically expose all
registered HTTP and gRPC check paths.

This stanza also accepts a paths list to expose individual paths. The primary
use case for this functionality would be to expose paths for third parties like
Prometheus or the kubelet.

Listeners for requests to exposed paths are be configured dynamically at run
time. Any time a proxy, or check can be registered, a listener can also be
created.

In this initial implementation requests to these paths are not
authenticated/encrypted.
2019-09-25 20:55:52 -06:00
Mike Morris 88df658243
connect: remove managed proxies (#6220)
* connect: remove managed proxies implementation and all supporting config options and structs

* connect: remove deprecated ProxyDestination

* command: remove CONNECT_PROXY_TOKEN env var

* agent: remove entire proxyprocess proxy manager

* test: remove all managed proxy tests

* test: remove irrelevant managed proxy note from TestService_ServerTLSConfig

* test: update ContentHash to reflect managed proxy removal

* test: remove deprecated ProxyDestination test

* telemetry: remove managed proxy note

* http: remove /v1/agent/connect/proxy endpoint

* ci: remove deprecated test exclusion

* website: update managed proxies deprecation page to note removal

* website: remove managed proxy configuration API docs

* website: remove managed proxy note from built-in proxy config

* website: add note on removing proxy subdirectory of data_dir
2019-08-09 15:19:30 -04:00
Matt Keeler bcb3439c4c
Fix some tests that I broke when refactoring the ConfigSnapshot (#6051)
* Fix some tests that I broke when refactoring the ConfigSnapshot

* Make sure the MeshGateway config is added to all the right api structs

* Fix some more tests
2019-07-01 19:47:58 -04:00
Matt Keeler 39bb0e3e77 Implement Mesh Gateways
This includes both ingress and egress functionality.
2019-07-01 16:28:30 -04:00
Matt Keeler b6688a6b5b
Add tagged addresses for services (#5965)
This allows addresses to be tagged at the service level similar to what we allow for nodes already. The address translation that can be enabled with the `translate_wan_addrs` config was updated to take these new addresses into account as well.
2019-06-17 10:51:50 -04:00
Matt Keeler ac78c23021
Implement data filtering of some endpoints (#5579)
Fixes: #4222 

# Data Filtering

This PR will implement filtering for the following endpoints:

## Supported HTTP Endpoints

- `/agent/checks`
- `/agent/services`
- `/catalog/nodes`
- `/catalog/service/:service`
- `/catalog/connect/:service`
- `/catalog/node/:node`
- `/health/node/:node`
- `/health/checks/:service`
- `/health/service/:service`
- `/health/connect/:service`
- `/health/state/:state`
- `/internal/ui/nodes`
- `/internal/ui/services`

More can be added going forward and any endpoint which is used to list some data is a good candidate.

## Usage

When using the HTTP API a `filter` query parameter can be used to pass a filter expression to Consul. Filter Expressions take the general form of:

```
<selector> == <value>
<selector> != <value>
<value> in <selector>
<value> not in <selector>
<selector> contains <value>
<selector> not contains <value>
<selector> is empty
<selector> is not empty
not <other expression>
<expression 1> and <expression 2>
<expression 1> or <expression 2>
```

Normal boolean logic and precedence is supported. All of the actual filtering and evaluation logic is coming from the [go-bexpr](https://github.com/hashicorp/go-bexpr) library

## Other changes

Adding the `Internal.ServiceDump` RPC endpoint. This will allow the UI to filter services better.
2019-04-16 12:00:15 -04:00
R.B. Boyer caadf208bd
api: fix panic in 'consul acl set-agent-token' (#5533)
api: fix panic in 'consul acl set-agent-token'

Fixes #5531
2019-03-21 15:04:40 -05:00
Matt Keeler 0c76a4389f
ACL Token Persistence and Reloading (#5328)
This PR adds two features which will be useful for operators when ACLs are in use.

1. Tokens set in configuration files are now reloadable.
2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload)

Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change.

Some other secondary changes:

* Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly.
* Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. 
* Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents.
* Docs updated to reflect the API additions and to show using the new endpoints.
* Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.
2019-02-27 14:28:31 -05:00
Grégoire Seux 6a57c7fec5 Implement /v1/agent/health/service/<service name> endpoint (#3551)
This endpoint aggregates all checks related to <service id> on the agent
and return an appropriate http code + the string describing the worst
check.

This allows to cleanly expose service status to other component, hiding
complexity of multiple checks.
This is especially useful to use consul to feed a load balancer which
would delegate health checking to consul agent.

Exposing this endpoint on the agent is necessary to avoid a hit on
consul servers and avoid decreasing resiliency (this endpoint will work
even if there is no consul leader in the cluster).
2019-01-07 09:39:23 -05:00
Jack Pearkes 197d62c6ca New command: consul debug (#4754)
* agent/debug: add package for debugging, host info

* api: add v1/agent/host endpoint

* agent: add v1/agent/host endpoint

* command/debug: implementation of static capture

* command/debug: tests and only configured targets

* agent/debug: add basic test for host metrics

* command/debug: add methods for dynamic data capture

* api: add debug/pprof endpoints

* command/debug: add pprof

* command/debug: timing, wg, logs to disk

* vendor: add gopsutil/disk

* command/debug: add a usage section

* website: add docs for consul debug

* agent/host: require operator:read

* api/host: improve docs and no retry timing

* command/debug: fail on extra arguments

* command/debug: fixup file permissions to 0644

* command/debug: remove server flags

* command/debug: improve clarity of usage section

* api/debug: add Trace for profiling, fix profile

* command/debug: capture profile and trace at the same time

* command/debug: add index document

* command/debug: use "clusters" in place of members

* command/debug: remove address in output

* command/debug: improve comment on metrics sleep

* command/debug: clarify usage

* agent: always register pprof handlers and protect

This will allow us to avoid a restart of a target agent
for profiling by always registering the pprof handlers.

Given this is a potentially sensitive path, it is protected
with an operator:read ACL and enable debug being
set to true on the target agent. enable_debug still requires
a restart.

If ACLs are disabled, enable_debug is sufficient.

* command/debug: use trace.out instead of .prof

More in line with golang docs.

* agent: fix comment wording

* agent: wrap table driven tests in t.run()
2018-10-19 08:41:03 -07:00
R.B. Boyer 4427417140 cli: avoid passing envoy bootstrap configuration as arguments (#4747)
Play a trick with CLOEXEC to pass the envoy bootstrap configuration as
an open file descriptor to the exec'd envoy process. The file only
briefly touches disk before being unlinked.

We convince envoy to read from this open file descriptor by using the
/dev/fd/$FDNUMBER mechanism to read the open file descriptor as a file.

Because the filename no longer has an extension envoy's sniffing logic
falls back on JSON instead of YAML, so the bootstrap configuration must
be generated as JSON instead.
2018-10-10 16:55:34 +01:00
Paul Banks 979e1c9c94 Add -sidecar-for and new /agent/service/:service_id endpoint (#4691)
- A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it:
   - **supports hash-based blocking** and so;
   - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config.
   - It's not proxy specific and so works for any service.
   - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested).
 - The built-in proxy now uses the new endpoint exclusively for it's config
 - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered.
 - Several fixes for edge cases for SidecarService
 - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.
2018-10-10 16:55:34 +01:00
Paul Banks 7038fe6b71 Add SidecarService Syntax sugar to Service Definition (#4686)
* Added new Config for SidecarService in ServiceDefinitions.

* WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too.

* Test coverage for the new sidecarServiceFromNodeService method.

* Test API registratrion with SidecarService

* Recursive Key Translation 🤦

* Add tests for nested sidecar defintion arrays to ensure they are translated correctly

* Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration.

Add tests for deregistration.

* API struct for agent register. No other endpoint should be affected yet.

* Additional test cases to cover updates to API registrations
2018-10-10 16:55:34 +01:00
Paul Banks 92fe8c8e89 Add Proxy Upstreams to Service Definition (#4639)
* Refactor Service Definition ProxyDestination.

This includes:
 - Refactoring all internal structs used
 - Updated tests for both deprecated and new input for:
   - Agent Services endpoint response
   - Agent Service endpoint response
   - Agent Register endpoint
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Register
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Services endpoint response
   - Catalog Node endpoint response
   - Catalog Service endpoint response
 - Updated API tests for all of the above too (both deprecated and new forms of register)

TODO:
 - config package changes for on-disk service definitions
 - proxy config endpoint
 - built-in proxy support for new fields

* Agent proxy config endpoint updated with upstreams

* Config file changes for upstreams.

* Add upstream opaque config and update all tests to ensure it works everywhere.

* Built in proxy working with new Upstreams config

* Command fixes and deprecations

* Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts...

TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct.

* Fix translated keys in API registration.
≈

* Fixes from docs
 - omit some empty undocumented fields in API
 - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally.

* Documentation updates for Upstreams in service definition

* Fixes for tests broken by many refactors.

* Enable travis on f-connect branch in this branch too.

* Add consistent Deprecation comments to ProxyDestination uses

* Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs
2018-10-10 16:55:34 +01:00
Pierre Souchay 473e589d86 Implementation of Weights Data structures (#4468)
* Implementation of Weights Data structures

Adding this datastructure will allow us to resolve the
issues #1088 and #4198

This new structure defaults to values:
```
   { Passing: 1, Warning: 0 }
```

Which means, use weight of 0 for a Service in Warning State
while use Weight 1 for a Healthy Service.
Thus it remains compatible with previous Consul versions.

* Implemented weights for DNS SRV Records

* DNS properly support agents with weight support while server does not (backwards compatibility)

* Use Warning value of Weights of 1 by default

When using DNS interface with only_passing = false, all nodes
with non-Critical healthcheck used to have a weight value of 1.
While having weight.Warning = 0 as default value, this is probably
a bad idea as it breaks ascending compatibility.

Thus, we put a default value of 1 to be consistent with existing behaviour.

* Added documentation for new weight field in service description

* Better documentation about weights as suggested by @banks

* Return weight = 1 for unknown Check states as suggested by @banks

* Fixed typo (of -> or) in error message as requested by @mkeeler

* Fixed unstable unit test TestRetryJoin

* Fixed unstable tests

* Fixed wrong Fatalf format in `testrpc/wait.go`

* Added notes regarding DNS SRV lookup limitations regarding number of instances

* Documentation fixes and clarification regarding SRV records with weights as requested by @banks

* Rephrase docs
2018-09-07 15:30:47 +01:00
Mitchell Hashimoto 62512adb84 api: support native connect 2018-06-25 12:24:10 -07:00
Paul Banks cac32ba071
More test cleanup 2018-06-14 09:42:16 -07:00
Mitchell Hashimoto 536f31571b
agent: change connect command paths to be slices, not strings
This matches other executable configuration and allows us to cleanly
separate executable from arguments without trying to emulate shell
parsing.
2018-06-14 09:42:08 -07:00
Paul Banks 02ab461dae
TLS watching integrated into Service with some basic tests.
There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.
2018-06-14 09:42:07 -07:00
Paul Banks a90f69faa4
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list. 2018-06-14 09:41:58 -07:00
Paul Banks 280382c25f
Add tests all the way up through the endpoints to ensure duplicate src/destination is supported and so ultimately deny/allow nesting works.
Also adds a sanity check test for `api.Agent().ConnectAuthorize()` and a fix for a trivial bug in it.
2018-06-14 09:41:57 -07:00
Mitchell Hashimoto 26f254fac0
api: rename Authorize field to ClientCertURI 2018-06-14 09:41:56 -07:00
Mitchell Hashimoto 9de861d722
api: fix up some comments and rename IssuedCert to LeafCert 2018-06-14 09:41:56 -07:00
Mitchell Hashimoto b5b301aa2a
api: endpoints for working with CA roots, agent authorize, etc. 2018-06-14 09:41:55 -07:00
Paul Banks 894ee3c5b0
Add Connect agent, catalog and health endpoints to api Client 2018-06-14 09:41:54 -07:00
Kyle Havlovitz a480434517
Remove the script field from checks in favor of args 2018-05-08 15:31:53 -07:00
Pierre Souchay ee47eb7d7d Added Missing Service Meta synchronization and field 2018-04-21 17:34:29 +02:00
Preetha Appan d9d9944179
Renames agent API layer for service metadata to "meta" for consistency 2018-03-28 09:04:50 -05:00
Pierre Souchay 824b72cf90 Merge remote-tracking branch 'origin/master' into service_metadata 2018-02-11 13:20:49 +01:00
Kyle Havlovitz 7100161f64
Add gRPC fields to client api and agent check endpoint docs 2018-02-09 16:02:27 -08:00
Pierre Souchay 3acc5b58d4 Added support for Service Metadata 2018-02-07 01:54:42 +01:00
Veselkov Konstantin 05666113a4 remove golint warnings 2018-01-28 22:40:13 +04:00
James Hartig 1f3b110b65 Added CheckID and Name to AgentServiceCheck struct 2018-01-08 15:59:55 -05:00
Kyle Havlovitz 49a010555f
Update AgentCheck struct in api 2017-11-01 15:30:29 -07:00
Kyle Havlovitz 16908be034
Add deregister critical service field and refactor duration parsing 2017-10-25 19:17:41 -07:00
Kyle Havlovitz 7d82ece118
Added remaining HTTP health check fields to structs 2017-10-25 19:37:30 +02:00
Kyle Havlovitz 84a07ea113
Expose SkipNodeUpdate field and some health check info in the http api 2017-10-25 19:37:30 +02:00
James Phillips 3d52f42715 Fixes API client for ScriptArgs and updates documentation. (#3589)
* Updates the API client to support the current `ScriptArgs` parameter
for checks.

* Updates docs for checks to explain the `ScriptArgs` parameter issue.

* Adds mappings for "args" and "script-args" to give th API parity
with config.

* Adds checks on return codes.

* Removes debug logging that shows empty when args are used.
2017-10-18 11:28:39 -07:00
James Phillips c629773b40
Makes the all segments query explict, and the default for `consul members`. 2017-09-05 12:22:20 -07:00
Kyle Havlovitz 5cc4b32a5d
Organize segments for a cleaner split between enterprise and OSS 2017-08-31 17:39:46 -07:00
Kyle Havlovitz d9fc2b3d75
Update coord display in ui to account for segments 2017-08-30 11:58:29 -07:00
James Phillips 6a6eadd8c7
Adds open source side of network segments (feature is Enterprise-only). 2017-08-30 11:58:29 -07:00
James Phillips a2bb506cea
Adds a guard to make sure that empty log lines don't close consul monitor.
Fixes #3253.
2017-08-08 16:05:29 -07:00
Kyle Havlovitz 879651a4ea
Update api structs for metrics endpoint 2017-08-08 12:33:59 -07:00
Kyle Havlovitz 975ded2714
Add support for labels/filters from go-metrics 2017-08-08 01:45:10 -07:00
James Phillips 803ed9a245 Adds secure introduction for the ACL replication token. (#3357)
Adds secure introduction for the ACL replication token, as well as a separate enable config for ACL replication.
2017-08-03 15:39:31 -07:00
James Phillips 6e794ea1b3 Adds support for agent-side ACL token management via API instead of config files. (#3324)
* Adds token store and removes all runtime use of config for ACL tokens.
* Adds a new API for changing agent tokens on the fly.
2017-07-26 11:03:43 -07:00
James Bardin 96ae8c1231 accept recv-only channels for cancellations (#3271)
Cancellation channels are often derived from a Context, which
returns a directional `<-chan struct{}` from Done(). In order to use
this with parts of of the consul API, one is required to create a new
channel and dispatch a separate goroutine to watch for context
cancellation and close the new channel.

Changing the signature for the methods that take cancellation channels
will allow easier integration with existing uses of Context. Since the
cancellation pattern only reads from these channels, there should be no
backwards incompatibility with existing codebases, and most of the
methods already accept only the correct type.
2017-07-14 16:31:44 -07:00
Frank Schröder e206ab5cb6 agent: support custom header and method for http checks (#3106)
This patch adds support for custom headers and
method for HTTP checks.

Fixes #2474
Fixes #2657
Fixes #3106
2017-06-07 01:11:56 +02:00
Brian Shumate 9d87883c85
API: Add struct fields, fixes #2882
- Add CreateIndex and ModifyIndex to AgentService
- Add CreateIndex and ModifyIndex to Node
2017-04-13 11:36:19 -04:00
Kyle Havlovitz dd05afb32e Add reload/leave http endpoints (#2516) 2016-11-30 13:29:42 -05:00
James Phillips d97eb758c6 Merge pull request #2530 from coffeehc/master
adapt to server's checker field TLSSkipVerify type
2016-11-29 16:00:21 -08:00
Kyle Havlovitz 475408633a Add QueryOptions to api package's monitor 2016-11-28 18:36:26 -05:00
Kyle Havlovitz d91854f3b0 Add monitor http endpoint 2016-11-28 18:36:26 -05:00
coffee 1187daa15b adapt to server's checker field TLSSkipVerify type 2016-11-21 18:02:38 +08:00
James Phillips fde94fb537
Adds notes field to API.
Closes #2336.
2016-11-17 16:33:50 -08:00
Kyle McCullough 697ea0f8eb Add setting to skip ssl certificate verification for HTTP checks (#1984)
* http check: add setting to skip ssl certificate verification

* update http check documentation

* fix typo in documentation

* Add TLSSkipVerify to agent api
2016-11-03 13:17:30 -07:00
James Phillips f517f9ed0d
Fixes a typo and adds an admonition about only being in Consul 0.7+. 2016-08-16 09:27:20 -07:00
James Phillips b4f981c837
Adds ability to deregister a service based on critical check state longer than a timeout. 2016-08-16 01:00:26 -07:00
Sean Chittenden 0f45d1b76d Correct a small typo 2016-04-23 20:18:19 -07:00
Sean Chittenden 5bc4a2c2ab consul/ uses structs.Health*, the api uses api.Health* 2016-04-23 16:06:58 -07:00
Sean Chittenden 65f9017c63 Update Check API to use constants
Use constants where appropriate to advocate their use.  Also add a deprecation notice re: `updateTTL`.
2016-04-23 16:01:59 -07:00
Diptanu Choudhury ad085de7b7 Added some constants in the api for check health statuses 2016-03-24 11:26:07 -07:00
James Phillips a126240a12 Adds support for new PUT API for checks. 2016-03-04 15:18:25 -08:00
James Phillips 551a4fc031 Adds support for EnableTagOverride to the API client. 2016-02-16 11:45:29 -08:00
James Phillips 23536f2b97 Adds Docker checks support to client API.
Also changed `DockerContainerId` to `DockerContainerID`, and updated the agent
API docs to reflect their support for Docker checks.
2015-11-18 07:40:02 -08:00
Peter Fern 86edd7169c Add TCP check type
Adds the ability to simply check whether a TCP socket accepts
connections to determine if it is healthy.  This is a light-weight -
though less comprehensive than scripting - method of checking network
service health.

The check parameter `tcp` should be set to the `address:port`
combination for the service to be tested.  Supports both IPv6 and IPv4,
in the case of a hostname that resolves to both, connections will be
attempted via both protocol versions, with the first successful
connection returning a successful check result.

Example check:

```json
{
  "check": {
    "id": "ssh",
    "name": "SSH (TCP)",
    "tcp": "example.com:22",
    "interval": "10s"
  }
}
```
2015-07-24 14:06:05 +10:00
Ryan Mills 370853d7ff Allow specifying a status field in the agent/service/register and agent/check/register endpoints.
This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'.
If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 02:00:31 +00:00
Ryan Uber c248d5badd api: allow timeout field to be specified for checks 2015-02-05 23:34:30 -08:00
Nicholas Capo 25d2faf43b api/agent: Support HTTP service checks in client api 2015-02-03 20:13:55 +00:00
Armon Dadgar e5e519df4b Merge pull request #625 from hashicorp/f-maintcmd
New "maint" command
2015-01-22 11:56:49 -08:00
Dr. Stefan Schimanski 1ef12ae24b Add agent service Address field to the api
The Address field was introduced in #570. This patch extends this to the api.
2015-01-22 10:26:27 +01:00
Ryan Uber d10529c233 api: support reason flag for maintenance mode 2015-01-21 13:02:47 -08:00
Ryan Uber 98d47d2b9a api: add node/service maintenance mode functions 2015-01-21 10:51:43 -08:00
Ryan Uber a9069b5993 api: support multiple checks during service/check registration 2015-01-20 21:48:45 -08:00
Ryan Uber 5172b21ee8 api: initial import from armon/consul-api 2015-01-06 10:40:00 -08:00