diff --git a/website/source/docs/connect/proxies/envoy.md b/website/source/docs/connect/proxies/envoy.md index b97311145..c248a88ec 100644 --- a/website/source/docs/connect/proxies/envoy.md +++ b/website/source/docs/connect/proxies/envoy.md @@ -14,257 +14,330 @@ optionally exposing a gRPC service on the local agent that serves [Envoy's xDS configuration API](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md). -Currently Consul only supports TCP proxying between services, however HTTP and -gRPC features are planned for the near future along with first class ways to -configure them in Consul. +Consul can configure Envoy sidecars to proxy http/1.1, http2 or gRPC traffic at +L7 or any other tcp-based protocol at L4. Prior to Consul 1.5.0 Envoy proxies +could only proxy tcp at L4. -As an interim solution, [custom Envoy configuration](#custom-configuration) can -be specified in [proxy service definition](/docs/connect/proxies.html) allowing -more powerful features of Envoy to be used. +Currently configuration of additional L7 features is limited, however we have +plans to support a wider range of features in the next major release +cycle. + +As an interim solution, you can add [custom Envoy configuration](#custom-configuration) +in the [proxy service definition](/docs/connect/proxies.html) allowing +you to use the more powerful features of Envoy. ## Supported Versions -Consul's Envoy support was added in version 1.3.0. It has been tested against -Envoy 1.7.1 and 1.8.0. +Consul's Envoy support was added in version 1.3.0. The following table shows +compatible Envoy versions. +| Consul Version | Compatible Envoy Versions | +|---|---| +| 1.5.x and higher | 1.9.1, 1.8.0† | +| 1.3.x, 1.4.x | 1.9.1, 1.8.0†, 1.7.0† | + +!> **Security Note:** Envoy versions lower than 1.9.1 are vulnerable to + [CVE-2019-9900](https://github.com/envoyproxy/envoy/issues/6434) and + [CVE-2019-9901](https://github.com/envoyproxy/envoy/issues/6435). Both are + related to HTTP request parsing and so only affect Consul Connect users if they + have configured HTTP routing rules via the ["escape + hatch"](#custom-configuration). Still, we recommend that you use Envoy 1.9.1 where + possible. ## Getting Started To get started with Envoy and see a working example you can follow the [Using Envoy with Connect](/docs/guides/connect-envoy.html) guide. -## Limitations +## Configuration -The following list limitations of the Envoy integration as released in 1.3.0. -All of these are planned to be lifted in the near future. +Envoy proxies require two types of configuration: an initial _bootstrap +configuration_ and dynamic configuration that is discovered from a "management +server", in this case Consul. - * Default Envoy configuration only supports Layer 4 (TCP) proxying. More - [advanced listener configuration](#advanced-listener-configuration) is - possible but experimental and requires deep Envoy knowledge. First class - workflows for configuring Layer 7 features across the cluster are planned for - the near future. - * There is currently no way to override the configuration of upstream clusters - which makes it impossible to configure Envoy features like circuit breakers, - load balancing policy, custom protocol settings etc. This will be fixed in a - near-future release first with an "escape hatch" similar to the one for - listeners below, then later with first-class support. - * The configuration delivered to Envoy is suitable for a sidecar proxy - currently. Later we plan to support more flexibility to be able to configure - Envoy as an edge router or gateway and similar. - * There is currently no way to disable the public listener and have a "client - only" sidecar for services that don't expose Connect-enabled service but want - to consume others. This will be fixed in a near-future release. - * Once authorized, a persistent TCP connection will not be closed if the - intentions change to deny access. This is currently a limitation of how TCP - proxy and network authz filter work in Envoy. All new connections will be - denied though and destination services can limit exposure by closing inbound - connections periodically or by a rolling restart of the destination service - as an emergency measure. +The bootstrap configuration at a minimum needs to configure the proxy with an +identity (node id) and the location of it's local Consul agent from which it +discovers all of it's dynamic configuration. See [Bootstrap +Configuration](#bootstrap-configuration) for more details. + +The dynamic configuration Consul Connect provides to each Envoy instance includes: + + - TLS certificates and keys to enable mutual authentication and keep certificates + rotating. + - Service-discovery results for upstreams to enable each sidecar proxy to load-balance + outgoing connections. + - L7 configuration including timeouts and protocol-specific options. + +For more information on the parts of the Envoy proxy runtime configuration +that are currently controllable via Consul Connect see [Dynamic +Configuration](#dynamic-configuration). + +We plan to enable more and more of Envoy's features through +Connect's first-class configuration over time, however some advanced users will +need additional control to configure Envoy in specific ways. To enable this, we +provide several ["escape hatch"](#advanced-configuration) options that allow +users to provide low-level raw Envoy config syntax for some sub-components in each +Envoy instance. This allows operators to have full control over and +responsibility for correctly configuring Envoy and ensuring version support etc. ## Bootstrap Configuration -Envoy requires an initial bootstrap configuration that directs it to the local -agent for further configuration discovery. - -To assist in generating this, Consul 1.3.0 adds a [`consul connect envoy` +Envoy requires an initial bootstrap configuration file. The easiest way to +create this is using the [`consul connect envoy` command](/docs/commands/connect/envoy.html). The command can either output the -bootstrap configuration directly or can generate it and then `exec` the Envoy -binary as a convenience wrapper. +bootstrap configuration directly to stdout or can generate it and then `exec` +the Envoy binary as a convenience wrapper. -Some Envoy configuration options like metrics and tracing sinks can only be -specified via the bootstrap config currently and so a custom bootstrap must be -used. In order to work with Connect it's necessary to start with the following -basic template and add additional configuration as needed. +Because some Envoy configuration options like metrics and tracing sinks can only be +specified via the bootstrap configuration, Connect as of Consul 1.5.0 adds +the ability to control some parts of the bootstrap config via proxy +configuration options. -```yaml -admin: - # access_log_path and address are required by Envoy, Consul doesn't care what - # they are set to though and never accesses the admin API. -node: - # cluter is required by Envoy but Consul doesn't use it - cluster: "" -static_resources: - clusters: - # local_agent is the "cluster" used to make further discovery requests for - # config and should point to the gRPC port of the local Consul agent instance. - - name: local_agent - connect_timeout: 1s - type: STATIC - # tls_context is needed if and only if Consul agent TLS is configured - tls_context: - common_tls_context: - validation_context: - trusted_ca: - filename: "" - http2_protocol_options: {} - hosts: - - socket_address: - address: "" - port_value: "" -dynamic_resources: - lds_config: - ads: {} - cds_config: - ads: {} - ads_config: - api_type: GRPC - grpc_services: - initial_metadata: - - key: "x-consul-token" - token: "" - envoy_grpc: - cluster_name: local_agent -``` +Users can add the following configuration items to the [global `proxy-defaults` +configuration entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) or override them directly in the `proxy.config` field +of a [proxy service +definition](/docs/connect/proxies.html#proxy-service-definitions) or +[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block. -This configures a "cluster" pointing to the local Consul agent and sets that as -the target for discovering all types of dynamic resources. +- `envoy_statsd_url` - A URL in the form `udp://ip:port` identifying a UDP + StatsD listener that Envoy should deliver metrics to. For example, this may be + `udp://127.0.0.1:8125` if every host has a local StatsD listener. In this case + users can configure this property once in the [global `proxy-defaults` +configuration entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) for convenience. Currently, TCP is not supported. -~> **Security Note**: The bootstrap configuration must contain the Consul ACL -token authorizing the proxy to identify as the target service. As such it should -be treated as a secret value and handled with care - an attacker with access to -one is able to obtain Connect TLS certificates for the target service and so -access anything that service is authorized to connect to. + ~> **Note:** currently the url **must use an ip address** not a dns name due + to the way Envoy is setup for StatsD. -## Advanced Listener Configuration + Users can also specify the whole parameter in the form `$ENV_VAR_NAME`, which + will cause the `consul connect envoy` command to resolve the actual URL from + the named environment variable when it runs. This, for example, allows each + pod in a Kubernetes cluster to learn of a pod-specific IP address for StatsD + when the Envoy instance is bootstrapped while still allowing global + configuration of all proxies to use StatsD in the [global `proxy-defaults` +configuration entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults). The env variable must contain a full valid URL + value as specified above and nothing else. It is not currently possible to use + environment variables as only part of the URL. -Consul 1.3.0 includes initial Envoy support which includes automatic Layer 4 -(TCP) proxying over mTLS, and authorization. Near future versions of Consul will -bring Layer 7 features like HTTP-path-based routing, retries, tracing and more. +- `envoy_dogstatsd_url` - The same as `envoy_statsd_url` with the following + differences in behavior: + - Envoy will use dogstatsd tags instead of statsd dot-separated metric names. + - As well as `udp://`, a `unix://` URL may be specified if your agent can + listen on a unix socket (e.g. the dogstatsd agent). --> **Advanced Topic!** This section covers an optional way of taking almost -complete control of Envoy's listener configuration which is provided as a way to -experiment with advanced integrations ahead of full layer 7 feature support. -While we don't plan to remove the ability to do this in the future, it should be -considered experimental and requires in-depth knowledge of Envoy's configuration -format. +- `envoy_prometheus_bind_addr` - Specifies that the proxy should expose a Prometheus + metrics endpoint to the _public_ network. It must be supplied in the form + `ip:port` and port and the ip/port combination must be free within the network + namespace the proxy runs. Typically the IP would be `0.0.0.0` to bind to all + available interfaces or a pod IP address. -For advanced users there is an "escape hatch" available in 1.3.0. The -`proxy.config` map in the [proxy service -definition](/docs/connect/proxies.html#proxy-service-definitions) may contain a -special key called `envoy_public_listener_json`. If this is set, it's value must -be a string containing the serialized proto3 JSON encoding of a complete [envoy -listener -config](https://www.envoyproxy.io/docs/envoy/v1.8.0/api-v2/api/v2/lds.proto). -Each upstream listener may also be customized in the same way by adding a -`envoy_listener_json` key to the `config` map of [the upstream -definition](/docs/connect/proxies.html#upstream-configuration-reference). + -> **Note:** Envoy versions prior to 1.10 do not export timing histograms + using the internal Prometheus endpoint. Consul 1.5.0 [doesn't yet support + Envoy 1.10](#supported-versions) although support will soon be added. -The JSON supplied may describe a protobuf `types.Any` message with `@type` set -to `type.googleapis.com/envoy.api.v2.Listener`, or it may be the direct encoding -of the listener with no `@type` field. +- `envoy_stats_tags` - Specifies one or more static tags that will be added to + all metrics produced by the proxy. -Once parsed, it is passed to Envoy in place of the listener config that Consul -would typically configure. The only modifications Consul will make to the config -provided are noted below. +- `envoy_stats_flush_interval` - Configures Envoy's + [`stats_flush_interval`](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-stats-flush-interval). -#### Public Listener Configuration +There are more possibilities available in the [Advanced +Configuration](#advanced-configuration) section that allow incremental or +complete control over the bootstrap configuration generated. -For the `proxy.config.envoy_public_listener_json`, every `FilterChain` added to -the listener will have it's `TlsContext` overwritten with the Connect TLS -certificates. This means there is no way to override Connect TLS settings or the -requirement for all inbound clients to present valid Connect certificates. +## Dynamic Configuration -Also, every `FilterChain` will have the `envoy.ext_authz` filter prepended to -the filters array to ensure that all incoming connections must be authorized -explicitly by the Consul agent based on their presented client certificate. +Consul automatically generates Envoy's dynamic configuration based on its +knowledge of the cluster. Users may specify default configuration options for +each service such as which protocol they speak. Consul will use this information +to configure appropriate proxy settings for that service's proxies and also for +the upstream listeners of any downstream service. -To work properly with Consul Connect, the public listener should bind to the -same address in the service definition so it is discoverable. It may also use -the special cluster name `local_app` to forward requests to a single local -instance if the proxy was configured [as a -sidecar](/docs/connect/proxies.html#sidecar-proxy-fields). +Users can define a service's protocol in its [`service-defaults` configuration +entry](/docs/agent/config_entries.html#service-defaults-service-defaults). Agents with +[`enable_central_service_config`](/docs/agent/options.html#enable_central_service_config) +set to true will automatically discover the protocol when configuring a proxy +for a service. The proxy will discover the main protocol of the service it +represents and use this to configure its main public listener. It will also +discover the protocols defined for any of its upstream services and +automatically configure its upstream listeners appropriately too as below. -#### Example +This automated discovery results in Consul auto-populating the `proxy.config` +and `proxy.upstreams[*].config` fields of the [proxy service +definition](/docs/connect/proxies.html#proxy-service-definitions) that is +actually registered. -The following example shows a public listener being configured with an http -connection manager. As specified this behaves exactly like the default TCP proxy -filter however it provides metrics on HTTP request volume and response codes. +### Proxy Config Options -If additional config outside of the listener is needed (for example the -top-level `tracing` configuration to send traces to a collecting service), those -currently need to be added to a custom bootstrap. You may generate the default -connect bootstrap with the [`consul connect envoy -bootstrap` -command](/docs/commands/connect/envoy.html) and then add the required additional -resources. +These fields may also be overridden explicitly in the [proxy service +definition](/docs/connect/proxies.html#proxy-service-definitions), or defined in +the [global `proxy-defaults` configuration +entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) to act as +defaults that are inherited by all services. -```text -service { - kind = "connect-proxy" - name = "web-http-aware-proxy" - port = 8080 - proxy { - destination_service_name = "web" - destination_service_id = "web" - config { - envoy_public_listener_json = < **Advanced Topic!** This section covers options that allow users to take almost +complete control of Envoy's configuration. We provide these options so users can +experiment or take advantage of features not yet fully supported in Consul Connect. We +plan to retain this ability in the future, but it should still be considered +experimental because it requires in-depth knowledge of Envoy's configuration format. +Users should consider Envoy version compatibility when using these features because they can configure Envoy in ways that +are outside of Consul's control. Incorrect configuration could prevent all +proxies in your mesh from functioning correctly, or bypass the security +guarantees Connect is designed to enforce. + +### Configuration Formatting + +All configurations are specified as strings containing the serialized proto3 JSON encoding +of the specified Envoy configuration type. They are full JSON types except where +noted. + +The JSON supplied may describe a protobuf `types.Any` message with an `@type` +field set to the appropriate type (for example +`type.googleapis.com/envoy.api.v2.Listener`), or it may be the direct encoding +with no `@type` field. + +### Advanced Bootstrap Options + +Users may add the following configuration items to the [global `proxy-defaults` +configuration +entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) or +override them directly in the `proxy.config` field of a [proxy service +definition](/docs/connect/proxies.html#proxy-service-definitions) or +[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block. + +- `envoy_extra_static_clusters_json` - Specifies one or more [Envoy + clusters](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/cds.proto#cluster) + that will be appended to the array of [static + clusters](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-staticresources-clusters) + in the bootstrap config. This allows adding custom clusters for tracing sinks + for example. For a single cluster just encode a single object, for multiple, + they should be comma separated with no trailing comma suitable for + interpolating directly into a JSON array inside the braces. +- `envoy_extra_static_listeners_json` - Similar to + `envoy_extra_static_clusters_json` but appends [static + listener](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-staticresources-listeners) definitions. + Can be used to setup limited access that bypasses Connect mTLS or + authorization for health checks or metrics. +- `envoy_extra_stats_sinks_json` - Similar to `envoy_extra_static_clusters_json` + but for [stats sinks](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-stats-sinks). These are appended to any sinks defined by use of the + higher-level [`envoy_statsd_url`](#envoy_statsd_url) or + [`envoy_dogstatsd_url`](#envoy_dogstatsd_url) config options. +- `envoy_stats_config_json` - The entire [stats + config](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-stats-config). + If provided this will override the higher-level + [`envoy_stats_tags`](#envoy_stats_tags). It allows full control over dynamic + tag replacements etc. +- `envoy_tracing_json` - The entire [tracing + config](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-tracing). + Most tracing providers will also require adding static clusters to define the + endpoints to send tracing data to. + +### Escape-Hatch Overrides + +Users may add the following configuration items to the [global `proxy-defaults` +configuration +entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) or +override them directly in the `proxy.config` field of a [proxy service +definition](/docs/connect/proxies.html#proxy-service-definitions) or +[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block. + +- `envoy_bootstrap_json_tpl` - Specifies a template in Go template syntax that + is used in place of [the default + template](https://github.com/hashicorp/consul/blob/b64bda880843afaaf44591c3200f921626716849/command/connect/envoy/bootstrap_tpl.go#L87) + when generating bootstrap via [`consul connect envoy` + command](/docs/commands/connect/envoy.html). The variables that are available + to be interpolated are [documented + here](https://github.com/hashicorp/consul/blob/b64bda880843afaaf44591c3200f921626716849/command/connect/envoy/bootstrap_tpl.go#L5). + This offers complete control of the proxy's bootstrap although major + deviations from the default template may break Consul's ability to correctly + manage the proxy or enforce it's security model. +- `envoy_public_listener_json` - Specifies a complete + [Listener](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/lds.proto) + to be delivered in place of the main public listener that the proxy used to + accept inbound connections. This will be used verbatim with the following + exceptions: + - Every `FilterChain` added to the listener will have its `TlsContext` + overridden by the Connect TLS certificates and validation context. This + means there is no way to override Connect's mutual TLS for the public + listener. + - Every `FilterChain` will have the `envoy.ext_authz` filter prepended to the + filters array to ensure that all inbound connections are authorized by + Connect. +- `envoy_local_cluster_json` - Specifies a complete [Envoy + cluster](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/cds.proto#cluster) + to be delivered in place of the local application cluster. This allows + customization of timeouts, rate limits, load balancing strategy etc. + + +The following configuration items may be overridden directly in the +`proxy.upstreams[].config` field of a [proxy service +definition](/docs/connect/proxies.html#proxy-service-definitions) or +[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block. + +- `envoy_listener_json` - Specifies a complete + [Listener](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/lds.proto) + to be delivered in place of the upstream listener that the proxy exposes to + the application for outbound connections. This will be used verbatim with the + following exceptions: + - Every `FilterChain` added to the listener will have its `TlsContext` + overridden by the Connect TLS certificates and validation context. This + means there is no way to override Connect's mutual TLS for the public + listener. +- `envoy_cluster_json` - Specifies a complete [Envoy + cluster](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/cds.proto#cluster) + to be delivered in place of the discovered upstream cluster. This allows + customization of timeouts, circuit breaking, rate limits, load balancing + strategy etc.