Commit Graph

118 Commits

Author SHA1 Message Date
Jack Pearkes 197d62c6ca New command: consul debug (#4754)
* agent/debug: add package for debugging, host info

* api: add v1/agent/host endpoint

* agent: add v1/agent/host endpoint

* command/debug: implementation of static capture

* command/debug: tests and only configured targets

* agent/debug: add basic test for host metrics

* command/debug: add methods for dynamic data capture

* api: add debug/pprof endpoints

* command/debug: add pprof

* command/debug: timing, wg, logs to disk

* vendor: add gopsutil/disk

* command/debug: add a usage section

* website: add docs for consul debug

* agent/host: require operator:read

* api/host: improve docs and no retry timing

* command/debug: fail on extra arguments

* command/debug: fixup file permissions to 0644

* command/debug: remove server flags

* command/debug: improve clarity of usage section

* api/debug: add Trace for profiling, fix profile

* command/debug: capture profile and trace at the same time

* command/debug: add index document

* command/debug: use "clusters" in place of members

* command/debug: remove address in output

* command/debug: improve comment on metrics sleep

* command/debug: clarify usage

* agent: always register pprof handlers and protect

This will allow us to avoid a restart of a target agent
for profiling by always registering the pprof handlers.

Given this is a potentially sensitive path, it is protected
with an operator:read ACL and enable debug being
set to true on the target agent. enable_debug still requires
a restart.

If ACLs are disabled, enable_debug is sufficient.

* command/debug: use trace.out instead of .prof

More in line with golang docs.

* agent: fix comment wording

* agent: wrap table driven tests in t.run()
2018-10-19 08:41:03 -07:00
Kyle Havlovitz 96a35f8abc re-add Connect multi-dc config changes
This reverts commit 8bcfbaffb6588b024cd1a3cf0952e6bfa7d9e900.
2018-10-19 08:41:03 -07:00
Jack Pearkes 847a0a5266 Revert "Connect multi-dc config" (#4784) 2018-10-11 17:32:45 +01:00
Aestek 260a9880ae [Security] Add finer control over script checks (#4715)
* Add -enable-local-script-checks options

These options allow for a finer control over when script checks are enabled by
giving the option to only allow them when they are declared from the local
file system.

* Add documentation for the new option

* Nitpick doc wording
2018-10-11 13:22:11 +01:00
Kyle Havlovitz 5b98a602af agent: add primary_datacenter and connect replication config options 2018-10-10 12:17:59 -07:00
Paul Banks 0523efa2fe merge feedback: fix typos; actually use deliverLatest added previously but not plumbed in 2018-10-10 16:55:34 +01:00
Paul Banks 7a8023a57f Fix up tests broken by master merge; add proxy tests to services command (and fix it!); actually run the proxycfg.Manager 2018-10-10 16:55:34 +01:00
Paul Banks 1e4c5a1811 Connect Envoy Command (#4735)
* Plumb xDS server and proxyxfg into the agent startup

* Add `consul connect envoy` command to allow running Envoy as a connect sidecar.

* Add test for help tabs; typos and style fixups from review
2018-10-10 16:55:34 +01:00
Paul Banks 979e1c9c94 Add -sidecar-for and new /agent/service/:service_id endpoint (#4691)
- A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it:
   - **supports hash-based blocking** and so;
   - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config.
   - It's not proxy specific and so works for any service.
   - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested).
 - The built-in proxy now uses the new endpoint exclusively for it's config
 - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered.
 - Several fixes for edge cases for SidecarService
 - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.
2018-10-10 16:55:34 +01:00
Paul Banks 7038fe6b71 Add SidecarService Syntax sugar to Service Definition (#4686)
* Added new Config for SidecarService in ServiceDefinitions.

* WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too.

* Test coverage for the new sidecarServiceFromNodeService method.

* Test API registratrion with SidecarService

* Recursive Key Translation 🤦

* Add tests for nested sidecar defintion arrays to ensure they are translated correctly

* Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration.

Add tests for deregistration.

* API struct for agent register. No other endpoint should be affected yet.

* Additional test cases to cover updates to API registrations
2018-10-10 16:55:34 +01:00
Paul Banks 92fe8c8e89 Add Proxy Upstreams to Service Definition (#4639)
* Refactor Service Definition ProxyDestination.

This includes:
 - Refactoring all internal structs used
 - Updated tests for both deprecated and new input for:
   - Agent Services endpoint response
   - Agent Service endpoint response
   - Agent Register endpoint
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Register
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Services endpoint response
   - Catalog Node endpoint response
   - Catalog Service endpoint response
 - Updated API tests for all of the above too (both deprecated and new forms of register)

TODO:
 - config package changes for on-disk service definitions
 - proxy config endpoint
 - built-in proxy support for new fields

* Agent proxy config endpoint updated with upstreams

* Config file changes for upstreams.

* Add upstream opaque config and update all tests to ensure it works everywhere.

* Built in proxy working with new Upstreams config

* Command fixes and deprecations

* Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts...

TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct.

* Fix translated keys in API registration.
≈

* Fixes from docs
 - omit some empty undocumented fields in API
 - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally.

* Documentation updates for Upstreams in service definition

* Fixes for tests broken by many refactors.

* Enable travis on f-connect branch in this branch too.

* Add consistent Deprecation comments to ProxyDestination uses

* Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs
2018-10-10 16:55:34 +01:00
Paul O'Connor 964f1c3bb6 Fix prometheus error message (#4745) 2018-10-03 14:47:56 -07:00
Pierre Souchay 508b67c32a Ensure that Proxies ARE always cleaned up, event with DeregisterCriticalServiceAfter (#4649)
This fixes https://github.com/hashicorp/consul/issues/4648
2018-09-11 17:34:09 +01:00
Pierre Souchay 473e589d86 Implementation of Weights Data structures (#4468)
* Implementation of Weights Data structures

Adding this datastructure will allow us to resolve the
issues #1088 and #4198

This new structure defaults to values:
```
   { Passing: 1, Warning: 0 }
```

Which means, use weight of 0 for a Service in Warning State
while use Weight 1 for a Healthy Service.
Thus it remains compatible with previous Consul versions.

* Implemented weights for DNS SRV Records

* DNS properly support agents with weight support while server does not (backwards compatibility)

* Use Warning value of Weights of 1 by default

When using DNS interface with only_passing = false, all nodes
with non-Critical healthcheck used to have a weight value of 1.
While having weight.Warning = 0 as default value, this is probably
a bad idea as it breaks ascending compatibility.

Thus, we put a default value of 1 to be consistent with existing behaviour.

* Added documentation for new weight field in service description

* Better documentation about weights as suggested by @banks

* Return weight = 1 for unknown Check states as suggested by @banks

* Fixed typo (of -> or) in error message as requested by @mkeeler

* Fixed unstable unit test TestRetryJoin

* Fixed unstable tests

* Fixed wrong Fatalf format in `testrpc/wait.go`

* Added notes regarding DNS SRV lookup limitations regarding number of instances

* Documentation fixes and clarification regarding SRV records with weights as requested by @banks

* Rephrase docs
2018-09-07 15:30:47 +01:00
Matt Keeler cbd0afc87c
Handle resolving proxy tokens when parsing HTTP requests (#4453)
Fixes: #4441

This fixes the issue with Connect Managed Proxies + ACLs being broken.

The underlying problem was that the token parsed for most http endpoints was sent untouched to the servers via the RPC request. These changes make it so that at the HTTP endpoint when parsing the token we additionally attempt to convert potential proxy tokens into regular tokens before sending to the RPC endpoint. Proxy tokens are only valid on the agent with the managed proxy so the resolution has to happen before it gets forwarded anywhere.
2018-07-30 09:11:51 -04:00
Matt Keeler 953b72318f Persist proxies from config files
Also change how loadProxies works. Now it will load all persisted proxies into a map, then when loading config file proxies will look up the previous proxy token in that map.
2018-07-18 17:04:35 -04:00
Paul Banks 1d6e1ace11 register TCP check for managed proxies 2018-06-25 12:25:40 -07:00
Paul Banks 42e28fa4d1 Limit proxy telemetry config to only be visible with authenticated with a proxy token 2018-06-25 12:25:39 -07:00
Paul Banks ca68136ac7 Refactor to use embedded struct. 2018-06-25 12:25:39 -07:00
Paul Banks fd3681f35b Allow user override of proxy telemetry config 2018-06-25 12:25:38 -07:00
Paul Banks ff162ffdde Basic proxy telemetry working; not sure if it's too ugly; need to instrument things we care about 2018-06-25 12:25:38 -07:00
Paul Banks ced9b2bee4 Expose telemetry config from RuntimeConfig to proxy config endpoint 2018-06-25 12:25:38 -07:00
Paul Banks 3d51c2aeac Get agent cache tests passing without global hit count (which is racy).
Few other fixes in here just to get a clean run locally - they are all also fixed in other PRs but shouldn't conflict.

This should be robust to timing between goroutines now.
2018-06-25 12:25:37 -07:00
Mitchell Hashimoto 8cb57b9316 agent: disallow deregistering a managed proxy directly 2018-06-25 12:25:12 -07:00
Mitchell Hashimoto 47c0e0dde6 agent: deregister service deregisters the proxy along with it 2018-06-25 12:25:12 -07:00
Mitchell Hashimoto f551413714 agent: disallow API registration with managed proxy if not enabled 2018-06-25 12:25:11 -07:00
Paul Banks 3a00574a13 Persist proxy state through agent restart 2018-06-25 12:24:08 -07:00
Mitchell Hashimoto 771842255a
address comment feedback 2018-06-14 09:42:22 -07:00
Mitchell Hashimoto 9249662c6c
agent: leaf endpoint accepts name, not service ID
This change is important so that requests can made representing a
service that may not be registered with the same local agent.
2018-06-14 09:42:20 -07:00
Mitchell Hashimoto 787ce3b269
agent: address feedback 2018-06-14 09:42:20 -07:00
Mitchell Hashimoto b02502be73
agent: comments to point to differing logic 2018-06-14 09:42:17 -07:00
Paul Banks 73f2a49ef1
Fix broken api test for service Meta (logical conflict rom OSS). Add test that would make this much easier to catch in future. 2018-06-14 09:42:17 -07:00
Paul Banks 834ed1d25f
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes. 2018-06-14 09:42:16 -07:00
Paul Banks 5abf47472d
Verify trust domain on /authorize calls 2018-06-14 09:42:16 -07:00
Mitchell Hashimoto b4f990bc6c
agent: verify local proxy tokens for CA leaf + tests 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto 8f7b5f93cd
agent: verify proxy token for ProxyConfig endpoint + tests 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto 52665f7d23
agent: clean up defaulting of proxy configuration
This cleans up and unifies how proxy settings defaults are applied.
2018-06-14 09:42:10 -07:00
Mitchell Hashimoto ed14e9edf8
agent: resolve some conflicts and fix tests 2018-06-14 09:42:10 -07:00
Paul Banks 02ab461dae
TLS watching integrated into Service with some basic tests.
There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.
2018-06-14 09:42:07 -07:00
Paul Banks dcd277de8a
Wire up agent leaf endpoint to cache framework to support blocking. 2018-06-14 09:42:07 -07:00
Paul Banks 6f566f750e
Basic `watch` support for connect proxy config and certificate endpoints.
- Includes some bug fixes for previous `api` work and `agent` that weren't tested
 - Needed somewhat pervasive changes to support hash based blocking - some TODOs left in our watch toolchain that will explicitly fail on hash-based watches.
 - Integration into `connect` is partially done here but still WIP
2018-06-14 09:42:05 -07:00
Mitchell Hashimoto a1f8cb9570
agent: augment /v1/connect/authorize to cache intentions 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto 8bb4fd95a6
agent: initialize the cache and cache the CA roots 2018-06-14 09:42:00 -07:00
Paul Banks a90f69faa4
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list. 2018-06-14 09:41:58 -07:00
Paul Banks 9d11cd9bf4
Fix various test failures and vet warnings.
Intention de-duplication in previously merged PR actualy failed some tests that were not caught be me or CI. I ran the test files for state changes but they happened not to trigger this case so I made sure they did first and then fixed. That fixed some upstream intention endpoint tests that I'd not run as part of testing the previous fix.
2018-06-14 09:41:58 -07:00
Paul Banks 8a4410b549
Refactor localBlockingQuery to use memdb.WatchSet. Much simpler and correct as a bonus! 2018-06-14 09:41:58 -07:00
Paul Banks cbd8606651
Add X-Consul-ContentHash header; implement removing all proxies; add load/unload test. 2018-06-14 09:41:57 -07:00
Paul Banks 44afb5c699
Agent Connect Proxy config endpoint with hash-based blocking 2018-06-14 09:41:57 -07:00
Paul Banks c2266b134a
HTTP agent registration allows proxy to be defined. 2018-06-14 09:41:57 -07:00
Mitchell Hashimoto 62b746c380
agent: rename authorize param ClientID to ClientCertURI 2018-06-14 09:41:56 -07:00