Commit Graph

1912 Commits

Author SHA1 Message Date
Matt Keeler e873dbe111
Add optional JSON format to the ACL CLI commands output (#7198)
* Add ACL CLI commands output format option.

Add command level formatter, that incapsulates command output printing
logiс that depends on the command `-format` option.
Move Print* functions from acl_helpers to prettyFormatter. Add jsonFormatter.

* Return error code in case of formatting failure.

* Add acl commands -format option to doc.
2020-03-26 13:16:21 -04:00
Freddy cb55fa3742
Enable CLI to register terminating gateways (#7500)
* Enable CLI to register terminating gateways

* Centralize gateway proxy configuration
2020-03-26 10:20:56 -06:00
Artur Mullakhmetov 29407260ab Return error code in case of formatting failure. 2020-03-26 19:05:10 +03:00
Artur Mullakhmetov e5b3b329ea Add ACL CLI commands output format option.
Add command level formatter, that incapsulates command output printing
logiс that depends on the command `-format` option.
Move Print* functions from acl_helpers to prettyFormatter. Add jsonFormatter.
2020-03-26 19:05:10 +03:00
Daniel Nephin 2569b2c6dd
command/envoy: Refactor flag parsing/validation (#7504) 2020-03-26 08:19:21 -06:00
Daniel Nephin 0377b87690 Remove unnecessary methods
They call only a single method and add no additional functionality
2020-03-24 18:35:07 -04:00
Daniel Nephin 1021a06181 cmd: use env vars as defaults
Insted of setting them afterward in Run.

This change required a small re-ordering of the test to patch the
environment before calling New()
2020-03-24 18:34:46 -04:00
Daniel Nephin 1eab9e06f0 Fix tests failing on master
The default version was changed in https://github.com/hashicorp/consul/pull/7452
which caused these tests to fail.
2020-03-23 16:38:14 -04:00
Hans Hasselberg 92a9bf1e13
envoy: default to 1.13.1 (#7452) 2020-03-17 22:23:42 +01:00
Hans Hasselberg 672db9bef6
docs: fix filenames (#7453) 2020-03-17 21:00:45 +01:00
Chris Piraino f13d6ca812
Log "vew version available" message at info level (#7462) 2020-03-17 09:53:15 -05:00
R.B. Boyer e142934a9c
fix flaky TestCatalogListNodesCommand_verticalBar test (#7422) 2020-03-10 11:01:13 -05:00
R.B. Boyer a7fb26f50f
wan federation via mesh gateways (#6884)
This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch:

There are several distinct chunks of code that are affected:

* new flags and config options for the server

* retry join WAN is slightly different

* retry join code is shared to discover primary mesh gateways from secondary datacenters

* because retry join logic runs in the *agent* and the results of that
  operation for primary mesh gateways are needed in the *server* there are
  some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur
  at multiple layers of abstraction just to pass the data down to the right
  layer.

* new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers

* the function signature for RPC dialing picked up a new required field (the
  node name of the destination)

* several new RPCs for manipulating a FederationState object:
  `FederationState:{Apply,Get,List,ListMeshGateways}`

* 3 read-only internal APIs for debugging use to invoke those RPCs from curl

* raft and fsm changes to persist these FederationStates

* replication for FederationStates as they are canonically stored in the
  Primary and replicated to the Secondaries.

* a special derivative of anti-entropy that runs in secondaries to snapshot
  their local mesh gateway `CheckServiceNodes` and sync them into their upstream
  FederationState in the primary (this works in conjunction with the
  replication to distribute addresses for all mesh gateways in all DCs to all
  other DCs)

* a "gateway locator" convenience object to make use of this data to choose
  the addresses of gateways to use for any given RPC or gossip operation to a
  remote DC. This gets data from the "retry join" logic in the agent and also
  directly calls into the FSM.

* RPC (`:8300`) on the server sniffs the first byte of a new connection to
  determine if it's actually doing native TLS. If so it checks the ALPN header
  for protocol determination (just like how the existing system uses the
  type-byte marker).

* 2 new kinds of protocols are exclusively decoded via this native TLS
  mechanism: one for ferrying "packet" operations (udp-like) from the gossip
  layer and one for "stream" operations (tcp-like). The packet operations
  re-use sockets (using length-prefixing) to cut down on TLS re-negotiation
  overhead.

* the server instances specially wrap the `memberlist.NetTransport` when running
  with gateway federation enabled (in a `wanfed.Transport`). The general gist is
  that if it tries to dial a node in the SAME datacenter (deduced by looking
  at the suffix of the node name) there is no change. If dialing a DIFFERENT
  datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh
  gateways to eventually end up in a server's :8300 port.

* a new flag when launching a mesh gateway via `consul connect envoy` to
  indicate that the servers are to be exposed. This sets a special service
  meta when registering the gateway into the catalog.

* `proxycfg/xds` notice this metadata blob to activate additional watches for
  the FederationState objects as well as the location of all of the consul
  servers in that datacenter.

* `xds:` if the extra metadata is in place additional clusters are defined in a
  DC to bulk sink all traffic to another DC's gateways. For the current
  datacenter we listen on a wildcard name (`server.<dc>.consul`) that load
  balances all servers as well as one mini-cluster per node
  (`<node>.server.<dc>.consul`)

* the `consul tls cert create` command got a new flag (`-node`) to help create
  an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 15:59:02 -05:00
Alex Dzyoba 827c5d9010
command: change delim in columnize to funny node names (#6652)
When node name contains vertical bar symbol some commands output is
garbled because `|` is used as a delimiter in `columnize.SimpleFormat`.

This commit changes format string to use `\x1f` - ASCII unit
separator[1] as a delimiter and also adds test to cover this case.

Affected commands:

* `consul catalog nodes`
* `consul members`
* `consul operator raft list-peers`
* `consul intention get`

Fixes #3951.

[1]: https://en.wikipedia.org/wiki/Delimiter#Solutions
2020-03-09 11:24:56 +01:00
Johannes Scheuermann bc87976708
agent: log error when agent crashes in an early stage (#7411) 2020-03-09 10:45:21 +01:00
Chris Piraino 5dd410a8c6
Fix -mesh-gateway flag help text (#7265) 2020-02-11 14:48:58 -06:00
Lars Lehtonen 5d6a821ae7
monitor: fix dropped error (#7206) 2020-02-11 10:57:50 +01:00
Hans Hasselberg 71ce832990
connect: add validations around intermediate cert ttl (#7213) 2020-02-11 00:05:49 +01:00
Hans Hasselberg 4ae725cab2
add envoy version 1.12.2 and 1.13.0 to the matrix (#7240)
* add 1.12.2

* add envoy 1.13.0

* Introduce -envoy-version to get 1.10.0 passing.

* update old version and fix consul-exec case

* add envoy_version and fix check

* Update Envoy CLI tests to account for the 1.13 compatibility changes.

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2020-02-10 14:53:04 -05:00
R.B. Boyer 62d0f76fc0
cli: fix typo in -namespace help text (#7225) 2020-02-05 14:43:25 -06:00
Paschalis Tsilias 1b81cccbf9
Expose Envoy's /stats for statsd agents (#7173)
* Expose Envoy /stats for statsd agents; Add testcases

* Remove merge conflict leftover

* Add support for prefix instead of path; Fix docstring to mirror these changes

* Add new config field to docs; Add testcases to check that /stats/prometheus is exposed as well

* Parametrize matchType (prefix or path) and value

* Update website/source/docs/connect/proxies/envoy.md

Co-Authored-By: Paul Banks <banks@banksco.de>

Co-authored-by: Paul Banks <banks@banksco.de>
2020-02-03 17:19:34 +00:00
Michael Hofer ee3b157eda
docs: add missing Autopilot -min-quorum documentation (#7192) 2020-02-03 10:59:53 +01:00
Lars Lehtonen da9086cd03
cli: check previously ignored errors when updating a policy (#6565) 2020-02-03 10:14:30 +01:00
R.B. Boyer 1d7e4f7de5
cli: improve the file safety of 'consul tls' subcommands (#7186)
- also fixing the signature of file.WriteAtomicWithPerms
2020-01-31 10:12:36 -06:00
Matt Keeler 25568626f7
Fix a couple bugs regarding intentions with namespaces (#7169) 2020-01-29 17:30:38 -05:00
Chris Piraino 3dd0b59793
Allow users to configure either unstructured or JSON logging (#7130)
* hclog Allow users to choose between unstructured and JSON logging
2020-01-28 17:50:41 -06:00
Anthony Scalisi 4b92c2deee fix spelling errors (#7135) 2020-01-27 07:00:33 -06:00
Matt Keeler 485a0a65ea
Updates to Config Entries and Connect for Namespaces (#7116) 2020-01-24 10:04:58 -05:00
Paul Lesiak d8e6c68ccc bug: Consul lock does not receive signals if lock not acquired (#5909) 2020-01-22 12:44:48 -07:00
Hans Hasselberg 315ba7d6ad
connect: check if intermediate cert needs to be renewed. (#6835)
Currently when using the built-in CA provider for Connect, root certificates are valid for 10 years, however secondary DCs get intermediates that are valid for only 1 year. There is no mechanism currently short of rotating the root in the primary that will cause the secondary DCs to renew their intermediates.
This PR adds a check that renews the cert if it is half way through its validity period.

In order to be able to test these changes, a new configuration option was added: IntermediateCertTTL which is set extremely low in the tests.
2020-01-17 23:27:13 +01:00
Aestek 9329cbac0a Add support for dual stack IPv4/IPv6 network (#6640)
* Use consts for well known tagged adress keys

* Add ipv4 and ipv6 tagged addresses for node lan and wan

* Add ipv4 and ipv6 tagged addresses for service lan and wan

* Use IPv4 and IPv6 address in DNS
2020-01-17 09:54:17 -05:00
Hans Hasselberg 0e2605335f
connect: use inline_string instead for envoy ca(#7024) 2020-01-10 15:57:54 +01:00
Matt Keeler 9ea83a749b
Revert "Remove docs refs to NS inference from ACL token" (#6976)
This reverts commit 3a8426de9c76e7d8dd2728e4ae78bc4e5e18626a.

# Conflicts:
#	command/flags/http.go
#	website/source/api/acl/binding-rules.html.md
#	website/source/api/acl/policies.html.md
#	website/source/api/acl/roles.html.md
#	website/source/api/acl/tokens.html.md
#	website/source/api/kv.html.md
#	website/source/api/session.html.md
#	website/source/docs/commands/_http_api_namespace_options.html.md
2019-12-20 11:52:50 -05:00
Matt Keeler 5c56aab3be
Change how namespaces are specified for the CLI (#6960) 2019-12-18 11:06:39 -05:00
Hans Hasselberg c8aeb07499
envoy: replace ca filename with inline_bytes. (#6822) 2019-12-13 17:44:48 +01:00
freddygv 775ea7af6e Remove docs refs to NS inference from ACL token 2019-12-10 13:50:28 -07:00
Matt Keeler 442924c35a
Sync of OSS changes to support namespaces (#6909) 2019-12-09 21:26:41 -05:00
Matt Keeler b9996e6bbe
Add Namespace support to the API module and the CLI commands (#6874)
Also update the Docs and fixup the HTTP API to return proper errors when someone attempts to use Namespaces with an OSS agent.

Add Namespace HTTP API docs

Make all API endpoints disallow unknown fields
2019-12-06 11:14:56 -05:00
Aestek 08aa4364a3 Fix consul connect token env variable doc (#5942)
The cli documentation for consul connect commands incorrectly indicated
to use CONSUL_TOKEN instead of CONSUL_HTTP_TOKEN env var.
2019-12-04 14:01:03 -06:00
Sarah Adams 1f5b333290
give feedback to CLI user on forceleave command if node does not exist (#6841) 2019-12-02 11:06:15 -08:00
Matt Keeler 90ae4a1f1e
OSS KV Modifications to Support Namespaces 2019-11-25 12:57:35 -05:00
Chris Piraino a30fac8a89 Allow services register command to register an unnamed check
The logic in parsing data files and converting them to data structures
accidentally removed healthchecks with no Name field, even though we
explicitly state in API documentation that is allowed.

We remove the check for "len(results.Checks) == 1" because if the length
of the array is more than 0, we know that it is not a zero value array.
This allows us to register a singular, unnamed check via the CLI.

Fixes #6796
2019-11-15 14:28:36 -06:00
Sarah Christoff 86b30bbfbe
Set MinQuorum variable in Autopilot (#6654)
* Add MinQuorum to Autopilot
2019-10-29 09:04:41 -05:00
rerorero e210b0c854 fix: incorrect struct tag and WaitGroup usage (#6649)
* remove duplicated json tag

* fix: incorrect wait group usage
2019-10-18 13:59:29 -04:00
Sarah Christoff 9b93dd93c9
Prune Unhealthy Agents (#6571)
* Add -prune flag to ForceLeave
2019-10-04 16:10:02 -05:00
Freddy 5eace88ce2
Expose HTTP-based paths through Connect proxy (#6446)
Fixes: #5396

This PR adds a proxy configuration stanza called expose. These flags register
listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only
listening on the loopback interface, while still accepting traffic from non
Connect-enabled services.

Under expose there is a boolean checks flag that would automatically expose all
registered HTTP and gRPC check paths.

This stanza also accepts a paths list to expose individual paths. The primary
use case for this functionality would be to expose paths for third parties like
Prometheus or the kubelet.

Listeners for requests to exposed paths are be configured dynamically at run
time. Any time a proxy, or check can be registered, a listener can also be
created.

In this initial implementation requests to these paths are not
authenticated/encrypted.
2019-09-25 20:55:52 -06:00
Sarah Adams 8e673371df
test: ensure all TestAgent constructions use a constructor (#6443)
ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place

Fixes #6435
2019-09-05 10:24:36 -07:00
Sarah Adams f8fa10fecb
refactor & add better retry logic to NewTestAgent (#6363)
Fixes #6361
2019-09-03 15:05:51 -07:00
R.B. Boyer d6456fddeb
connect: introduce ExternalSNI field on service-defaults (#6324)
Compiling this will set an optional SNI field on each DiscoveryTarget.
When set this value should be used for TLS connections to the instances
of the target. If not set the default should be used.

Setting ExternalSNI will disable mesh gateway use for that target. It also 
disables several service-resolver features that do not make sense for an 
external service.
2019-08-19 12:19:44 -05:00
R.B. Boyer 91df06098c
xds: improve how envoy metrics are emitted (#6312)
Since generated envoy clusters all are named using (mostly) SNI syntax
we can have envoy read the various fields out of that structure and emit
it as stats labels to the various telemetry backends.

I changed the delimiter for the 'customization hash' from ':' to '~'
because ':' is always reencoded by envoy as '_' when generating metrics
keys.
2019-08-16 09:30:17 -05:00