Commit Graph

1790 Commits

Author SHA1 Message Date
Hans Hasselberg 1a520d65b4
Builtin tls helper (#5078)
* command: add tls subcommand
* website: update docs and guide
2018-12-19 09:22:49 +01:00
Jack Pearkes 5faa61a906 Doc changes for 1.4 Final (#4870)
* website: add multi-dc enterprise landing page

* website: switch all 1.4.0 alerts/RC warnings

* website: connect product wording

Co-Authored-By: pearkes <jackpearkes@gmail.com>

* website: remove RC notification

* commmand/acl: fix usage docs for ACL tokens

* agent: remove comment, OperatorRead

* website: improve multi-dc docs

Still not happy with this but tried to make it slightly more informative.

* website: put back acl guide warning for 1.4.0

* website: simplify multi-dc page and respond to feedback

* Fix Multi-DC typos on connect index page.

* Improve Multi-DC overview.

A full guide is a WIP and will be added post-release.

* Fixes typo avaiable > available
2018-11-13 13:43:53 +00:00
Paul Banks 952ee6f546
Allow ACL legacy migration via CLI (#4882)
* Adds a flag to `consul acl token update` that allows legacy ACLs to be upgraded via the CLI.

Also fixes a bug where descriptions are deleted if not specified.

* Remove debug
2018-11-05 14:32:09 +00:00
R.B. Boyer 917488abc2 command/debug: make better use of atomic operations to write out the debug snapshots to disk 2018-11-02 13:13:49 -05:00
R.B. Boyer a5d57f5326
fix comment typos (#4890) 2018-11-02 12:00:39 -05:00
Paul Banks d24a65eb8c
Doc and whitespace fixes for translate-rules command (#4877) 2018-10-31 17:28:04 +00:00
Martin Halder 62dae2fd9f website: fix minor typo in documentation (#4864) 2018-10-29 01:33:42 -07:00
Matt Keeler 8fa3d61d25
Implement CLI token cloning & special ID handling (#4827)
* Implement CLI token cloning & special ID handling

* Update a couple CLI commands to take some alternative options.

* Document the CLI.

* Update the policy list and set-agent-token synopsis
2018-10-24 10:24:29 -04:00
Dhi Aurrahman 342e2696dc connect: Fix comment DYNAMIC_DNS to LOGICAL_DNS (#4799)
LOGICAL_DNS is one of the supported service discovery types [1].

[1] https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/service_discovery#arch-overview-service-discovery-types
2018-10-24 07:02:01 -07:00
Matt Keeler 99e0a124cb
New ACLs (#4791)
This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week.
Description

At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers.

On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though.

    Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though.
    All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management.
    Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are:
        A server running the new system must still support other clients using the legacy system.
        A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system.
        The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode.

So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 12:04:07 -04:00
Jack Pearkes 197d62c6ca New command: consul debug (#4754)
* agent/debug: add package for debugging, host info

* api: add v1/agent/host endpoint

* agent: add v1/agent/host endpoint

* command/debug: implementation of static capture

* command/debug: tests and only configured targets

* agent/debug: add basic test for host metrics

* command/debug: add methods for dynamic data capture

* api: add debug/pprof endpoints

* command/debug: add pprof

* command/debug: timing, wg, logs to disk

* vendor: add gopsutil/disk

* command/debug: add a usage section

* website: add docs for consul debug

* agent/host: require operator:read

* api/host: improve docs and no retry timing

* command/debug: fail on extra arguments

* command/debug: fixup file permissions to 0644

* command/debug: remove server flags

* command/debug: improve clarity of usage section

* api/debug: add Trace for profiling, fix profile

* command/debug: capture profile and trace at the same time

* command/debug: add index document

* command/debug: use "clusters" in place of members

* command/debug: remove address in output

* command/debug: improve comment on metrics sleep

* command/debug: clarify usage

* agent: always register pprof handlers and protect

This will allow us to avoid a restart of a target agent
for profiling by always registering the pprof handlers.

Given this is a potentially sensitive path, it is protected
with an operator:read ACL and enable debug being
set to true on the target agent. enable_debug still requires
a restart.

If ACLs are disabled, enable_debug is sufficient.

* command/debug: use trace.out instead of .prof

More in line with golang docs.

* agent: fix comment wording

* agent: wrap table driven tests in t.run()
2018-10-19 08:41:03 -07:00
Aestek 260a9880ae [Security] Add finer control over script checks (#4715)
* Add -enable-local-script-checks options

These options allow for a finer control over when script checks are enabled by
giving the option to only allow them when they are declared from the local
file system.

* Add documentation for the new option

* Nitpick doc wording
2018-10-11 13:22:11 +01:00
Paul Banks 0523efa2fe merge feedback: fix typos; actually use deliverLatest added previously but not plumbed in 2018-10-10 16:55:34 +01:00
Paul Banks bcf2cc2de3 cli: envoy command default gRPC port (#4768)
* Default gRPC port; Start on some basic tests for argument and ENV handling; Make Exec test less platform-dependent.

* Allow hot-restarts

* Remove debug
2018-10-10 16:55:34 +01:00
R.B. Boyer 4427417140 cli: avoid passing envoy bootstrap configuration as arguments (#4747)
Play a trick with CLOEXEC to pass the envoy bootstrap configuration as
an open file descriptor to the exec'd envoy process. The file only
briefly touches disk before being unlinked.

We convince envoy to read from this open file descriptor by using the
/dev/fd/$FDNUMBER mechanism to read the open file descriptor as a file.

Because the filename no longer has an extension envoy's sniffing logic
falls back on JSON instead of YAML, so the bootstrap configuration must
be generated as JSON instead.
2018-10-10 16:55:34 +01:00
Paul Banks 7a8023a57f Fix up tests broken by master merge; add proxy tests to services command (and fix it!); actually run the proxycfg.Manager 2018-10-10 16:55:34 +01:00
Paul Banks 1e4c5a1811 Connect Envoy Command (#4735)
* Plumb xDS server and proxyxfg into the agent startup

* Add `consul connect envoy` command to allow running Envoy as a connect sidecar.

* Add test for help tabs; typos and style fixups from review
2018-10-10 16:55:34 +01:00
Paul Banks cba42d6790 XDS Server Config (#4730)
* Config for the coming XDS server

* Default gRPC to 8502 for -dev mode; Re-merge the command Info output that shows gRPC.
2018-10-10 16:55:34 +01:00
Paul Banks 979e1c9c94 Add -sidecar-for and new /agent/service/:service_id endpoint (#4691)
- A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it:
   - **supports hash-based blocking** and so;
   - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config.
   - It's not proxy specific and so works for any service.
   - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested).
 - The built-in proxy now uses the new endpoint exclusively for it's config
 - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered.
 - Several fixes for edge cases for SidecarService
 - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.
2018-10-10 16:55:34 +01:00
Paul Banks 92fe8c8e89 Add Proxy Upstreams to Service Definition (#4639)
* Refactor Service Definition ProxyDestination.

This includes:
 - Refactoring all internal structs used
 - Updated tests for both deprecated and new input for:
   - Agent Services endpoint response
   - Agent Service endpoint response
   - Agent Register endpoint
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Register
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Services endpoint response
   - Catalog Node endpoint response
   - Catalog Service endpoint response
 - Updated API tests for all of the above too (both deprecated and new forms of register)

TODO:
 - config package changes for on-disk service definitions
 - proxy config endpoint
 - built-in proxy support for new fields

* Agent proxy config endpoint updated with upstreams

* Config file changes for upstreams.

* Add upstream opaque config and update all tests to ensure it works everywhere.

* Built in proxy working with new Upstreams config

* Command fixes and deprecations

* Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts...

TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct.

* Fix translated keys in API registration.
≈

* Fixes from docs
 - omit some empty undocumented fields in API
 - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally.

* Documentation updates for Upstreams in service definition

* Fixes for tests broken by many refactors.

* Enable travis on f-connect branch in this branch too.

* Add consistent Deprecation comments to ProxyDestination uses

* Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs
2018-10-10 16:55:34 +01:00
Paul Banks bed72f6078 Rename proxy package (re-run of #4550) (#4638)
* Rename agent/proxy package to reflect that it is limited to managed proxy processes

Rationale: we have several other components of the agent that relate to Connect proxies for example the ProxyConfigManager component needed for Envoy work. Those things are pretty separate from the focus of this package so far which is only concerned with managing external proxy processes so it's nota good fit to put code for that in here, yet there is a naming clash if we have other packages related to proxy functionality that are not in the `agent/proxy` package.

Happy to bikeshed the name. I started by calling it `managedproxy` but `managedproxy.Manager` is especially unpleasant. `proxyprocess` seems good in that it's more specific about purpose but less clearly connected with the concept of "managed proxies". The names in use are cleaner though e.g. `proxyprocess.Manager`.

This rename was completed automatically using golang.org/x/tools/cmd/gomvpkg.

Depends on #4541

* Fix missed windows tagged files
2018-10-10 16:55:34 +01:00
Mitchell Hashimoto 95d5089bd8
command/services: just add additional output feedback on success 2018-10-02 12:48:46 -07:00
Mitchell Hashimoto 5fb6bf481e
command/services: add test to ensure that dev mode introduces no
services
2018-10-02 12:45:00 -07:00
Mitchell Hashimoto 1b7f836398
command: register new commands 2018-10-01 09:17:36 -07:00
Mitchell Hashimoto 74b297faef
command/services/register: flag-based registration 2018-10-01 09:16:14 -07:00
Mitchell Hashimoto 6459387cd8
command/services/deregister: tests for flag validation 2018-10-01 08:55:32 -07:00
Mitchell Hashimoto 0f82c9570b
command/services/deregister: -id flag for deletion 2018-10-01 08:53:30 -07:00
Mitchell Hashimoto c69d845edf
command/services/deregister: basics working from file 2018-10-01 08:39:27 -07:00
Mitchell Hashimoto 85f6ea4007
command/services: move the config helpers to parent package 2018-10-01 08:27:59 -07:00
Mitchell Hashimoto 664d4badd5
command/services/register: registration from files work 2018-10-01 08:05:57 -07:00
Mitchell Hashimoto a125cb02b1
command/services/register: config mapping tests 2018-09-30 19:17:45 -07:00
Mitchell Hashimoto 0997792cea
command/services 2018-09-27 23:52:17 -07:00
Benjamin Sago 9aa00d45b6 Exit with error code 1 when failing to list DCs (#4583)
Fixes #4582.
2018-09-12 09:55:02 -07:00
Pierre Souchay 5ecf9823d2 Fix more unstable tests in agent and command 2018-09-12 14:49:27 +01:00
Pierre Souchay 7a42c31330 Fix unstable tests in agent, api, and command/watch 2018-09-10 16:58:53 +01:00
Pierre Souchay 473e589d86 Implementation of Weights Data structures (#4468)
* Implementation of Weights Data structures

Adding this datastructure will allow us to resolve the
issues #1088 and #4198

This new structure defaults to values:
```
   { Passing: 1, Warning: 0 }
```

Which means, use weight of 0 for a Service in Warning State
while use Weight 1 for a Healthy Service.
Thus it remains compatible with previous Consul versions.

* Implemented weights for DNS SRV Records

* DNS properly support agents with weight support while server does not (backwards compatibility)

* Use Warning value of Weights of 1 by default

When using DNS interface with only_passing = false, all nodes
with non-Critical healthcheck used to have a weight value of 1.
While having weight.Warning = 0 as default value, this is probably
a bad idea as it breaks ascending compatibility.

Thus, we put a default value of 1 to be consistent with existing behaviour.

* Added documentation for new weight field in service description

* Better documentation about weights as suggested by @banks

* Return weight = 1 for unknown Check states as suggested by @banks

* Fixed typo (of -> or) in error message as requested by @mkeeler

* Fixed unstable unit test TestRetryJoin

* Fixed unstable tests

* Fixed wrong Fatalf format in `testrpc/wait.go`

* Added notes regarding DNS SRV lookup limitations regarding number of instances

* Documentation fixes and clarification regarding SRV records with weights as requested by @banks

* Rephrase docs
2018-09-07 15:30:47 +01:00
Pierre Souchay e974ebd62e Fixed flaky tests (#4626) 2018-09-04 12:31:51 +01:00
Siva Prasad 59dea9a31f
Adds a new command line flag -log-file for file based logging. (#4581)
* Added log-file flag to capture Consul logs in a user specified file

* Refactored code.

* Refactored code. Added flags to rotate logs based on bytes and duration

* Added the flags for log file and log rotation on the webpage

* Fixed TestSantize from failing due to the addition of 3 flags

* Introduced changes : mutex, data-dir log writes, rotation logic

* Added test for logfile and updated the default log destination for docs

* Log name now uses UnixNano

* TestLogFile is now uses t.Parallel()

* Removed unnecessary int64Val function

* Updated docs to reflect default log name for log-file

* No longer writes to data-dir and adds .log if the filename has no extension
2018-08-29 16:56:58 -04:00
Pierre Souchay 09ff5f7224 Fixed unit test TestCatalogListServicesCommand (#4592) 2018-08-27 13:53:46 -04:00
Pierre Souchay 73824ab65f Fixed unstable test TestRTTCommand_LAN in command/rtt (#4585) 2018-08-27 11:37:13 -04:00
Pierre Souchay 5f603d86ad Fix unstable test TestRegisterMonitor_heartbeat (#4568) 2018-08-24 13:33:58 -04:00
Shubheksha 1afcabb0a2 replace old fork of text package (#4501) 2018-08-14 12:23:18 -07:00
Freddy cbe61dfcec
Improve reliability of tests with TestAgent (#4525)
- Add WaitForTestAgent to tests flaky due to missing serfHealth registration

- Fix bug in retries calling Fatalf with *testing.T

- Convert TestLockCommand_ChildExitCode to table driven test
2018-08-14 12:08:33 -04:00
Freddy e06cdb5278
Address flakiness in command/exec tests (#4517)
* Add fn to wait for TestAgent node and check registration

* Add waits for TestAgent and retries before timeouts in exec_test
2018-08-10 15:04:07 -04:00
Pierre Souchay fd927ea110 BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472)
- Improve resilience of testrpc.WaitForLeader()

- Add additionall retry to CI

- Increase "go test" timeout to 8m

- Add wait for cluster leader to several tests in the agent package

- Add retry to some tests in the api and command packages
2018-08-06 19:46:09 -04:00
Mitchell Hashimoto d0d40a4c34
Merge pull request #4314 from hashicorp/b-ignore-check
command/connect/proxy: ignore check doesn't exist on -register
2018-07-25 11:26:40 -05:00
Mitchell Hashimoto 9091f2b108
command/connect/proxy: ignore check doesn't exist on -register 2018-06-29 10:58:06 -07:00
Siva 2182e289a3 Merge branch 'master' of github.com:hashicorp/consul into WinService 2018-06-26 16:49:50 -04:00
Paul Banks ca68136ac7 Refactor to use embedded struct. 2018-06-25 12:25:39 -07:00
Paul Banks 23be6ad1c8 StartupTelemetry => InitTelemetry 2018-06-25 12:25:39 -07:00