Commit Graph

1948 Commits

Author SHA1 Message Date
Kyle Havlovitz 5aefdea1a8
Standardize support for Tagged and BindAddresses in Ingress Gateways (#7924)
* Standardize support for Tagged and BindAddresses in Ingress Gateways

This updates the TaggedAddresses and BindAddresses behavior for Ingress
to match Mesh/Terminating gateways. The `consul connect envoy` command
now also allows passing an address without a port for tagged/bind
addresses.

* Update command/connect/envoy/envoy.go

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* PR comments

* Check to see if address is an actual IP address

* Update agent/xds/listeners.go

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* fix whitespace

Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2020-05-21 09:08:12 -05:00
Daniel Nephin 545bd766e7 Fix a number of problems found by staticcheck
Some of these problems are minor (unused vars), but others are real bugs (ignored errors).

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2020-05-19 16:50:14 -04:00
Freddy 7e71b4d70d
Use proxy-id in gateway auto-registration (#7845) 2020-05-13 11:56:53 -06:00
Daniel Nephin 2e0f750f1a Add unconvert linter
To find unnecessary type convertions
2020-05-12 13:47:25 -04:00
R.B. Boyer 940e5ad160
acl: add auth method for JWTs (#7846) 2020-05-11 20:59:29 -05:00
R.B. Boyer c54211ad52
cli: ensure 'acl auth-method update' doesn't deep merge the Config field (#7839) 2020-05-11 14:21:17 -05:00
Kyle Havlovitz 28b4819882
Merge pull request #7759 from hashicorp/ingress/tls-hosts
Add TLS option for Ingress Gateway listeners
2020-05-11 09:18:43 -07:00
R.B. Boyer bf70ad1802
cli: oss refactors to support making the auth method CLI aware of namespace rules in enterprise (#7812) 2020-05-07 17:08:42 -05:00
R.B. Boyer 095f0503e8
test: make auth method cli crud test work in both oss and ent (#7800) 2020-05-06 15:16:50 -05:00
Chris Piraino bebf1d5df9 Add TLS field to ingress API structs
- Adds test in api and command/config/write packages
2020-05-06 15:12:02 -05:00
R.B. Boyer ea21280636
test: make auth method cli crud test helper ignore the default namespace (#7799) 2020-05-06 15:09:47 -05:00
Chris Piraino 210dda5682 Allow Hosts field to be set on an ingress config entry
- Validate that this cannot be set on a 'tcp' listener nor on a wildcard
service.
- Add Hosts field to api and test in consul config write CLI
- xds: Configure envoy with user-provided hosts from ingress gateways
2020-05-06 15:06:13 -05:00
Chris Piraino 837d2aa7d2 Remove service_subset field from ingress config entry
We decided that this was not a useful MVP feature, and just added
unnecessary complexity
2020-05-06 15:06:13 -05:00
R.B. Boyer c9c557477b
acl: add MaxTokenTTL field to auth methods (#7779)
When set to a non zero value it will limit the ExpirationTime of all
tokens created via the auth method.
2020-05-04 17:02:57 -05:00
s-christoff 2535cb85eb
cli: Add -config flag to "acl authmethod update/create" (#7776) 2020-05-04 16:21:28 -05:00
R.B. Boyer 265d2ea9e1
acl: add DisplayName field to auth methods (#7769)
Also add a few missing acl fields in the api.
2020-05-04 15:18:25 -05:00
Freddy f5c1e5268b
TLS Origination for Terminating Gateways (#7671) 2020-04-27 16:25:37 -06:00
R.B. Boyer f1d8ea7018
cli: ensure that 'snapshot save' is fsync safe and also only writes to the requested file on success (#7698) 2020-04-24 17:34:47 -05:00
R.B. Boyer 032e0ae901
cli: fix usage of gzip.Reader to better detect corrupt snapshots during save/restore (#7697) 2020-04-24 17:18:56 -05:00
Chris Piraino 0ab9aa9489
Add support for ingress-gateway in CLI command (#7618)
* Add support for ingress-gateway in CLI command

- Supports -register command
- Creates a static Envoy listener that exposes only the /ready API so
that we can register a TCP healthcheck against the ingress gateway
itself
- Updates ServiceAddressValue.String() to be more in line with Value()
2020-04-14 09:48:02 -05:00
Daniel Nephin bdbb704c5c Fix golden file for envoy tests
The envoy version was updated after the PR which added this test was opened, and
merged before the test was merged, so it ended up with the wrong version.
2020-04-13 12:58:02 -04:00
Daniel Nephin a2135d012b
Merge pull request #7608 from hashicorp/dnephin/grpc-default-scheme
command/envoy: enable TLS when CONSUL_HTTP_ADDR=https://...
2020-04-13 12:30:26 -04:00
Hans Hasselberg b78220981c
connect: support envoy 1.14.1 (#7624) 2020-04-09 20:58:22 +02:00
Freddy f5b9688336
Add decode rules for Expose cfg in service-defaults (#7611) 2020-04-07 19:37:47 -06:00
Daniel Nephin 575ad5c39f Fix CONSUL_HTTP_ADDR=https not enabling TLS
Use the config instead of attempting to reparse the env var.
2020-04-07 18:16:53 -04:00
Daniel Nephin 97c9f73261 Step 3: fix a bug in api.NewClient and fix the tests
The api client should never rever to HTTP if the user explicitly
requested TLS. This change broke some tests because the tests always use
an non-TLS http server, but some tests explicitly enable TLS.
2020-04-07 18:02:56 -04:00
Daniel Nephin ae42dea2d5 Step 2: extract the grpc address logic and a new type
The new grpcAddress function contains all of the logic to translate the
command line options into the values used in the template.

The new type has two advantages.

1. It introduces a logical grouping of values in the BootstrapTplArgs
   struct which is exceptionally large. This grouping makes the struct
   easier to understand because each set of nested values can be seen
   as a single entity.
2. It gives us a reasonable return value for this new function.
2020-04-07 16:36:51 -04:00
Daniel Nephin 5092aaf9b8 Step 1: move all the grpcAddr logic into the same spot
There is no reason a reader should have to jump around to find this value. It is only
used in 1 place
2020-04-07 15:53:12 -04:00
Freddy f5eb6ab539
Fix regression with gateway registration and update docs (#7582) 2020-04-02 12:52:11 -06:00
Daniel Nephin bc02b8fbe6
Merge pull request #7562 from hashicorp/dnephin/remove-tname-from-name
testing: Remove old default value from NewTestAgent() calls
2020-04-01 11:48:45 -04:00
Daniel Nephin 09c6ac8b92 Rename NewTestAgentWithFields to StartTestAgent
This function now only starts the agent.

Using:

git grep -l 'StartTestAgent(t, true,' | \
        xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 17:14:55 -04:00
Daniel Nephin d623dcbd01 Convert the remaining calls to NewTestAgentWithFields
After removing the t.Name() parameter with sed, convert the last few tests which
use a custom name to call NewTestAgentWithFields instead.
2020-03-31 17:14:55 -04:00
Daniel Nephin 8b6877febd Remove name from NewTestAgent
Using:

git grep -l 'NewTestAgent(t, t.Name(),' | \
    xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 16:13:44 -04:00
Freddy 8a1e53754e
Add config entry for terminating gateways (#7545)
This config entry will be used to configure terminating gateways.

It accepts the name of the gateway and a list of services the gateway will represent.

For each service users will be able to specify: its name, namespace, and additional options for TLS origination.

Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>
Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
2020-03-31 13:27:32 -06:00
Kyle Havlovitz 01a23b8eb4
Add config entry/state for Ingress Gateways (#7483)
* Add Ingress gateway config entry and other relevant structs

* Add api package tests for ingress gateways

* Embed EnterpriseMeta into ingress service struct

* Add namespace fields to api module and test consul config write decoding

* Don't require a port for ingress gateways

* Add snakeJSON and camelJSON cases in command test

* Run Normalize on service's ent metadata

Sadly cannot think of a way to test this in OSS.

* Every protocol requires at least 1 service

* Validate ingress protocols

* Update agent/structs/config_entry_gateways.go

Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2020-03-31 11:59:10 -05:00
Daniel Nephin 7064dfbfba command: remove unused logOutput field 2020-03-30 14:11:27 -04:00
Matt Keeler e873dbe111
Add optional JSON format to the ACL CLI commands output (#7198)
* Add ACL CLI commands output format option.

Add command level formatter, that incapsulates command output printing
logiс that depends on the command `-format` option.
Move Print* functions from acl_helpers to prettyFormatter. Add jsonFormatter.

* Return error code in case of formatting failure.

* Add acl commands -format option to doc.
2020-03-26 13:16:21 -04:00
Freddy cb55fa3742
Enable CLI to register terminating gateways (#7500)
* Enable CLI to register terminating gateways

* Centralize gateway proxy configuration
2020-03-26 10:20:56 -06:00
Artur Mullakhmetov 29407260ab Return error code in case of formatting failure. 2020-03-26 19:05:10 +03:00
Artur Mullakhmetov e5b3b329ea Add ACL CLI commands output format option.
Add command level formatter, that incapsulates command output printing
logiс that depends on the command `-format` option.
Move Print* functions from acl_helpers to prettyFormatter. Add jsonFormatter.
2020-03-26 19:05:10 +03:00
Daniel Nephin 2569b2c6dd
command/envoy: Refactor flag parsing/validation (#7504) 2020-03-26 08:19:21 -06:00
Daniel Nephin 0377b87690 Remove unnecessary methods
They call only a single method and add no additional functionality
2020-03-24 18:35:07 -04:00
Daniel Nephin 1021a06181 cmd: use env vars as defaults
Insted of setting them afterward in Run.

This change required a small re-ordering of the test to patch the
environment before calling New()
2020-03-24 18:34:46 -04:00
Daniel Nephin 1eab9e06f0 Fix tests failing on master
The default version was changed in https://github.com/hashicorp/consul/pull/7452
which caused these tests to fail.
2020-03-23 16:38:14 -04:00
Hans Hasselberg 92a9bf1e13
envoy: default to 1.13.1 (#7452) 2020-03-17 22:23:42 +01:00
Hans Hasselberg 672db9bef6
docs: fix filenames (#7453) 2020-03-17 21:00:45 +01:00
Chris Piraino f13d6ca812
Log "vew version available" message at info level (#7462) 2020-03-17 09:53:15 -05:00
R.B. Boyer e142934a9c
fix flaky TestCatalogListNodesCommand_verticalBar test (#7422) 2020-03-10 11:01:13 -05:00
R.B. Boyer a7fb26f50f
wan federation via mesh gateways (#6884)
This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch:

There are several distinct chunks of code that are affected:

* new flags and config options for the server

* retry join WAN is slightly different

* retry join code is shared to discover primary mesh gateways from secondary datacenters

* because retry join logic runs in the *agent* and the results of that
  operation for primary mesh gateways are needed in the *server* there are
  some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur
  at multiple layers of abstraction just to pass the data down to the right
  layer.

* new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers

* the function signature for RPC dialing picked up a new required field (the
  node name of the destination)

* several new RPCs for manipulating a FederationState object:
  `FederationState:{Apply,Get,List,ListMeshGateways}`

* 3 read-only internal APIs for debugging use to invoke those RPCs from curl

* raft and fsm changes to persist these FederationStates

* replication for FederationStates as they are canonically stored in the
  Primary and replicated to the Secondaries.

* a special derivative of anti-entropy that runs in secondaries to snapshot
  their local mesh gateway `CheckServiceNodes` and sync them into their upstream
  FederationState in the primary (this works in conjunction with the
  replication to distribute addresses for all mesh gateways in all DCs to all
  other DCs)

* a "gateway locator" convenience object to make use of this data to choose
  the addresses of gateways to use for any given RPC or gossip operation to a
  remote DC. This gets data from the "retry join" logic in the agent and also
  directly calls into the FSM.

* RPC (`:8300`) on the server sniffs the first byte of a new connection to
  determine if it's actually doing native TLS. If so it checks the ALPN header
  for protocol determination (just like how the existing system uses the
  type-byte marker).

* 2 new kinds of protocols are exclusively decoded via this native TLS
  mechanism: one for ferrying "packet" operations (udp-like) from the gossip
  layer and one for "stream" operations (tcp-like). The packet operations
  re-use sockets (using length-prefixing) to cut down on TLS re-negotiation
  overhead.

* the server instances specially wrap the `memberlist.NetTransport` when running
  with gateway federation enabled (in a `wanfed.Transport`). The general gist is
  that if it tries to dial a node in the SAME datacenter (deduced by looking
  at the suffix of the node name) there is no change. If dialing a DIFFERENT
  datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh
  gateways to eventually end up in a server's :8300 port.

* a new flag when launching a mesh gateway via `consul connect envoy` to
  indicate that the servers are to be exposed. This sets a special service
  meta when registering the gateway into the catalog.

* `proxycfg/xds` notice this metadata blob to activate additional watches for
  the FederationState objects as well as the location of all of the consul
  servers in that datacenter.

* `xds:` if the extra metadata is in place additional clusters are defined in a
  DC to bulk sink all traffic to another DC's gateways. For the current
  datacenter we listen on a wildcard name (`server.<dc>.consul`) that load
  balances all servers as well as one mini-cluster per node
  (`<node>.server.<dc>.consul`)

* the `consul tls cert create` command got a new flag (`-node`) to help create
  an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 15:59:02 -05:00
Alex Dzyoba 827c5d9010
command: change delim in columnize to funny node names (#6652)
When node name contains vertical bar symbol some commands output is
garbled because `|` is used as a delimiter in `columnize.SimpleFormat`.

This commit changes format string to use `\x1f` - ASCII unit
separator[1] as a delimiter and also adds test to cover this case.

Affected commands:

* `consul catalog nodes`
* `consul members`
* `consul operator raft list-peers`
* `consul intention get`

Fixes #3951.

[1]: https://en.wikipedia.org/wiki/Delimiter#Solutions
2020-03-09 11:24:56 +01:00