983 lines
40 KiB
Plaintext
983 lines
40 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Connect - Envoy Integration
|
|
description: Consul Connect has first-class support for configuring Envoy proxy.
|
|
---
|
|
|
|
# Envoy Integration
|
|
|
|
Consul Connect has first class support for using
|
|
[Envoy](https://www.envoyproxy.io) as a proxy. Consul configures Envoy by
|
|
optionally exposing a gRPC service on the local agent that serves [Envoy's xDS
|
|
configuration
|
|
API](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-docs/xds_protocol).
|
|
|
|
Consul can configure Envoy sidecars to proxy traffic over the following protocols:
|
|
|
|
| Protocol | Service network support |
|
|
| ----------------------- | ----------------------- |
|
|
| HTTP/1.1 | L7 |
|
|
| HTTP2 | L7 |
|
|
| gRPC | L7 |
|
|
| All TCP-based protocols | L4 |
|
|
|
|
On Consul 1.5.0 and older, Envoy proxies can only proxy TCP traffic at L4.
|
|
|
|
Some [L7 features](/docs/connect/l7-traffic) can be configured using [configuration entries](/docs/agent/config-entries). You can add [custom Envoy configurations](#advanced-configuration) to the [proxy service definition](/docs/connect/registration/service-registration) to use Envoy features that are not currently exposed through configuration entries. Adding custom Envoy configurations to the service definition is an interim solution that enables you to use the more powerful features of Envoy.
|
|
|
|
~> **Note:** When using Envoy with Consul and not using the [`consul connect envoy` command](/commands/connect/envoy)
|
|
Envoy must be run with the `--max-obj-name-len` option set to `256` or greater for Envoy versions prior to 1.11.0.
|
|
|
|
## Supported Versions
|
|
|
|
The following matrix describes Envoy compatibility for the currently supported **n-2 major Consul releases**. For previous Consul version compatibility please view the respective versioned docs for this page.
|
|
|
|
Consul supports **four major Envoy releases** at the beginning of each major Consul release. Consul maintains compatibility with Envoy patch releases for each major version so that users can benefit from bug and security fixes in Envoy. As a policy, Consul will add support for a new major versions of Envoy in a Consul major release. Support for newer versions of Envoy will not be added to existing releases.
|
|
|
|
| Consul Version | Compatible Envoy Versions |
|
|
| ------------------- | -----------------------------------------------------------------------------------|
|
|
| 1.13.x | 1.23.0, 1.22.2, 1.21.4, 1.20.6 |
|
|
| 1.12.x | 1.22.2, 1.21.4, 1.20.6, 1.19.5 |
|
|
| 1.11.x | 1.20.6, 1.19.5, 1.18.6, 1.17.4<sup>1</sup> |
|
|
|
|
1. Envoy 1.20.1 and earlier are vulnerable to [CVE-2022-21654](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21654) and [CVE-2022-21655](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-21655). Both CVEs were patched in Envoy versions 1.18.6, 1.19.3, and 1.20.2.
|
|
Envoy 1.16.x and older releases are no longer supported (see [HCSEC-2022-07](https://discuss.hashicorp.com/t/hcsec-2022-07-consul-s-connect-service-mesh-affected-by-recent-envoy-security-releases/36332)). Consul 1.9.x clusters should be upgraded to 1.10.x and Envoy upgraded to the latest supported Envoy version for that release, 1.18.6.
|
|
|
|
## Getting Started
|
|
|
|
To get started with Envoy and see a working example you can follow the [Using
|
|
Envoy with Connect](https://learn.hashicorp.com/tutorials/consul/service-mesh-with-envoy-proxy) tutorial.
|
|
|
|
## Configuration
|
|
|
|
Envoy proxies require two types of configuration: an initial _bootstrap
|
|
configuration_ and a _dynamic configuration_ that is discovered from a "management
|
|
server", in this case Consul.
|
|
|
|
The bootstrap configuration at a minimum needs to configure the proxy with an
|
|
identity (node id) and the location of its local Consul agent from which it
|
|
discovers all of its dynamic configuration. See [Bootstrap
|
|
Configuration](#bootstrap-configuration) for more details.
|
|
|
|
The dynamic configuration Consul Connect provides to each Envoy instance includes:
|
|
|
|
- TLS certificates and keys to enable mutual authentication and keep certificates
|
|
rotating.
|
|
- [Intentions] to enforce service-to-service authorization rules.
|
|
- Service-discovery results for upstreams to enable each sidecar proxy to load-balance
|
|
outgoing connections.
|
|
- L7 configuration including timeouts and protocol-specific options.
|
|
- Configuration to [expose specific HTTP paths](/docs/connect/registration/service-registration#expose-paths-configuration-reference).
|
|
|
|
For more information on the parts of the Envoy proxy runtime configuration
|
|
that are currently controllable via Consul Connect see [Dynamic
|
|
Configuration](#dynamic-configuration).
|
|
|
|
We plan to enable more and more of Envoy's features through
|
|
Connect's first-class configuration over time, however some advanced users will
|
|
need additional control to configure Envoy in specific ways. To enable this, we
|
|
provide several ["escape hatch"](#advanced-configuration) options that allow
|
|
users to provide low-level raw Envoy config syntax for some sub-components in each
|
|
Envoy instance. This allows operators to have full control over and
|
|
responsibility for correctly configuring Envoy and ensuring version support etc.
|
|
|
|
## Intention Enforcement
|
|
|
|
[Intentions] are enforced using Envoy's RBAC filters. Depending on the
|
|
configured [protocol] of the proxied service, intentions are either enforced
|
|
per-connection (L4) using a network filter, or per-request (L7) using an HTTP
|
|
filter.
|
|
|
|
-> **Note:** Prior to Consul 1.9.0 intentions were exclusively enforced
|
|
per-connection (L4) using an `ext_authz` network filter.
|
|
|
|
## Fetching Certificates
|
|
|
|
Envoy will use the [`CONSUL_HTTP_TOKEN`](/commands#consul_http_token) and [`CONSUL_HTTP_ADDR`](/commands#consul_http_addr) environment variables to contact Consul to fetch certificates if the following conditions are met:
|
|
|
|
- The `CONSUL_HTTP_TOKEN` environment variable contains a Consul ACL token.
|
|
- The Consul ACL token has the necessary permissions to read configuration for that service.
|
|
|
|
If TLS is enabled on Consul, you will also need to add the following environment variables _prior_ to starting Envoy:
|
|
|
|
- [`CONSUL_CACERT`](/commands#consul_cacert)
|
|
- [`CONSUL_CLIENT_CERT`](/commands#consul_client_cert)
|
|
- [`CONSUL_CLIENT_KEY`](/commands#consul_client_key)
|
|
- [`CONSUL_HTTP_SSL`](/commands#consul_http_ssl)
|
|
|
|
## Bootstrap Configuration
|
|
|
|
Envoy requires an initial bootstrap configuration file. The easiest way to
|
|
create this is using the [`consul connect envoy`
|
|
command](/commands/connect/envoy). The command can either output the
|
|
bootstrap configuration directly to stdout, or generate the configuration and issue an `exec` command
|
|
to the Envoy binary as a convenience wrapper.
|
|
|
|
Because some Envoy configuration options, such as metrics and tracing sinks, can only be
|
|
specified via the bootstrap configuration, Connect as of Consul 1.5.0 adds
|
|
the ability to control some parts of the bootstrap config via proxy configuration options.
|
|
|
|
Add the following configuration items to the [global `proxy-defaults`
|
|
configuration entry](/docs/connect/config-entries/proxy-defaults) or override them directly in the `proxy.config` field
|
|
of a [proxy service
|
|
definition](/docs/connect/registration/service-registration) or
|
|
[`sidecar_service`](/docs/connect/registration/sidecar-service) block.
|
|
|
|
- `envoy_statsd_url` - A URL in the form `udp://ip:port` identifying a UDP
|
|
StatsD listener that Envoy should deliver metrics to. For example, this may be
|
|
`udp://127.0.0.1:8125` if every host has a local StatsD listener. In this case
|
|
users can configure this property once in the [global `proxy-defaults`
|
|
configuration entry](/docs/connect/config-entries/proxy-defaults) for convenience. Currently, TCP is not supported.
|
|
|
|
~> **Note:** currently the url **must use an ip address** not a dns name due
|
|
to the way Envoy is setup for StatsD.
|
|
|
|
Expansion of the environment variable `HOST_IP` is supported, e.g.
|
|
`udp://${HOST_IP}:8125`.
|
|
|
|
Users can also specify the whole parameter in the form `$ENV_VAR_NAME`, which
|
|
will cause the `consul connect envoy` command to resolve the actual URL from
|
|
the named environment variable when it runs. This, for example, allows each
|
|
pod in a Kubernetes cluster to learn of a pod-specific IP address for StatsD
|
|
when the Envoy instance is bootstrapped while still allowing global
|
|
configuration of all proxies to use StatsD in the [global `proxy-defaults`
|
|
configuration entry](/docs/connect/config-entries/proxy-defaults). The env
|
|
variable must contain a full valid URL value as specified above and nothing else.
|
|
|
|
- `envoy_dogstatsd_url` - The same as `envoy_statsd_url` with the following
|
|
differences in behavior:
|
|
|
|
- Envoy will use dogstatsd tags instead of statsd dot-separated metric names.
|
|
- As well as `udp://`, a `unix://` URL may be specified if your agent can
|
|
listen on a unix socket (e.g. the dogstatsd agent).
|
|
|
|
- `envoy_prometheus_bind_addr` - Specifies that the proxy should expose a Prometheus
|
|
metrics endpoint to the _public_ network. It must be supplied in the form
|
|
`ip:port` and port and the ip/port combination must be free within the network
|
|
namespace the proxy runs. Typically the IP would be `0.0.0.0` to bind to all
|
|
available interfaces or a pod IP address.
|
|
|
|
-> **Note:** Envoy versions prior to 1.10 do not export timing histograms
|
|
using the internal Prometheus endpoint.
|
|
|
|
- `envoy_stats_bind_addr` - Specifies that the proxy should expose the /stats prefix
|
|
to the _public_ network. It must be supplied in the form `ip:port` and
|
|
the ip/port combination must be free within the network namespace the proxy runs.
|
|
Typically the IP would be `0.0.0.0` to bind to all available interfaces or a pod IP address.
|
|
|
|
- `envoy_stats_tags` - Specifies one or more static tags that will be added to
|
|
all metrics produced by the proxy.
|
|
|
|
- `envoy_stats_flush_interval` - Configures Envoy's
|
|
[`stats_flush_interval`](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-flush-interval).
|
|
|
|
The [Advanced Configuration](#advanced-configuration) section describes additional configurations that allow incremental or complete control over the bootstrap configuration generated.
|
|
|
|
## Dynamic Configuration
|
|
|
|
Consul automatically generates Envoy's dynamic configuration based on its
|
|
knowledge of the cluster. Users may specify default configuration options for
|
|
each service such as which protocol they speak. Consul will use this information
|
|
to configure appropriate proxy settings for that service's proxies and also for
|
|
the upstream listeners of any downstream service.
|
|
|
|
One example is how users can define a service's protocol in a [`service-defaults` configuration
|
|
entry](/docs/connect/config-entries/service-defaults). Agents with
|
|
[`enable_central_service_config`](/docs/agent/config/config-files#enable_central_service_config)
|
|
set to true will automatically discover the protocol when configuring a proxy
|
|
for a service. The proxy will discover the main protocol of the service it
|
|
represents and use this to configure its main public listener. It will also
|
|
discover the protocols defined for any of its upstream services and
|
|
automatically configure its upstream listeners appropriately too as below.
|
|
|
|
This automated discovery results in Consul auto-populating the `proxy.config`
|
|
and `proxy.upstreams[*].config` fields of the [proxy service
|
|
definition](/docs/connect/registration/service-registration) that is
|
|
actually registered.
|
|
|
|
To learn about other options that can be configured centrally see the
|
|
[Configuration Entries](/docs/agent/config-entries) docs.
|
|
|
|
### Proxy Config Options
|
|
|
|
These fields may also be overridden explicitly in the [proxy service
|
|
definition](/docs/connect/registration/service-registration), or defined in
|
|
the [global `proxy-defaults` configuration
|
|
entry](/docs/connect/config-entries/proxy-defaults) to act as
|
|
defaults that are inherited by all services.
|
|
|
|
- `protocol` - The protocol the service speaks. Connect's Envoy integration
|
|
currently supports the following `protocol` values:
|
|
|
|
- `tcp` - Unless otherwise specified this is the default, which causes Envoy
|
|
to proxy at L4. This provides all the security benefits of Connect's mTLS
|
|
and works for any TCP-based protocol. Load-balancing and metrics are
|
|
available at the connection level.
|
|
- `http` - This specifies that the service speaks HTTP/1.x. Envoy will setup an
|
|
`http_connection_manager` and will be able to load-balance requests
|
|
individually to available upstream services. Envoy will also emit L7 metrics
|
|
such as request rates broken down by HTTP response code family (2xx, 4xx, 5xx,
|
|
etc).
|
|
- `http2` - This specifies that the service speaks http2 (specifically h2c since
|
|
Envoy will still only connect to the local service instance via plain TCP not
|
|
TLS). This behaves much like `http` with L7 load-balancing and metrics but has
|
|
additional settings that correctly enable end-to-end http2.
|
|
- `grpc` - gRPC is a common RPC protocol based on http2. In addition to the
|
|
http2 support above, Envoy listeners will be configured with a
|
|
[gRPC bridge
|
|
filter](https://www.envoyproxy.io/docs/envoy/v1.17.2/configuration/http/http_filters/grpc_http1_bridge_filter)
|
|
that translates HTTP/1.1 calls into gRPC, and instruments
|
|
metrics with `gRPC-status` trailer codes.
|
|
|
|
~> **Note:** The protocol of a service should ideally be configured via the
|
|
[`protocol`](/docs/connect/config-entries/service-defaults#protocol)
|
|
field of a
|
|
[`service-defaults`](/docs/connect/config-entries/service-defaults)
|
|
config entry for the service. Configuring it in a
|
|
proxy config will not fully enable some [L7
|
|
features](/docs/connect/l7-traffic).
|
|
It is supported here for backwards compatibility with Consul versions prior to 1.6.0.
|
|
|
|
- `bind_address` - Override the address Envoy's public listener binds to. By
|
|
default Envoy will bind to the service address or 0.0.0.0 if there is not explicit address on the service registration.
|
|
|
|
- `bind_port` - Override the port Envoy's public listener binds to. By default
|
|
Envoy will bind to the service port.
|
|
|
|
- `local_connect_timeout_ms` - The number of milliseconds allowed to make
|
|
connections to the local application instance before timing out. Defaults to 5000
|
|
(5 seconds).
|
|
|
|
- `local_request_timeout_ms` - In milliseconds, the request timeout for HTTP requests
|
|
to the local application instance. Applies to HTTP based protocols only. If not
|
|
specified, inherits the Envoy default for route timeouts (15s). A value of 0 will
|
|
disable request timeouts.
|
|
|
|
### Proxy Upstream Config Options
|
|
|
|
The following configuration items may be overridden directly in the
|
|
`proxy.upstreams[].config` field of a [proxy service
|
|
definition](/docs/connect/registration/service-registration) or
|
|
[`sidecar_service`](/docs/connect/registration/sidecar-service) block.
|
|
|
|
- `protocol` - Same as above in main config but affects the listener setup for
|
|
the upstream.
|
|
|
|
~> **Note:** The protocol of a service should ideally be configured via the
|
|
[`protocol`](/docs/connect/config-entries/service-defaults#protocol)
|
|
field of a
|
|
[`service-defaults`](/docs/connect/config-entries/service-defaults)
|
|
config entry for the upstream destination service. Configuring it in a
|
|
proxy upstream config will not fully enable some [L7
|
|
features](/docs/connect/l7-traffic).
|
|
It is supported here for backwards compatibility with Consul versions prior to 1.6.0.
|
|
|
|
- `connect_timeout_ms` - The number of milliseconds to allow when making upstream
|
|
connections before timing out. Defaults to 5000
|
|
(5 seconds).
|
|
|
|
~> **Note:** The connection timeout for a service should ideally be
|
|
configured via the
|
|
[`connect_timeout`](/docs/connect/config-entries/service-resolver#connecttimeout)
|
|
field of a
|
|
[`service-resolver`](/docs/connect/config-entries/service-resolver)
|
|
config entry for the upstream destination service. Configuring it in a
|
|
proxy upstream config will override any values defined in config entries.
|
|
It is supported here for backwards compatibility with Consul versions prior to 1.6.0.
|
|
|
|
- `limits` - A set of limits to apply when connecting to the upstream service.
|
|
These limits are applied on a per-service-instance basis. The following
|
|
limits are respected:
|
|
|
|
- `max_connections` - The maximum number of connections a service instance
|
|
will be allowed to establish against the given upstream. Use this to limit
|
|
HTTP/1.1 traffic, since HTTP/1.1 has a request per connection.
|
|
- `max_pending_requests` - The maximum number of requests that will be queued
|
|
while waiting for a connection to be established. For this configuration to
|
|
be respected, a L7 protocol must be defined in the `protocol` field.
|
|
- `max_concurrent_requests` - The maximum number of concurrent requests that
|
|
will be allowed at a single point in time. Use this to limit HTTP/2 traffic,
|
|
since HTTP/2 has many requests per connection. For this configuration to be
|
|
respected, a L7 protocol must be defined in the `protocol` field.
|
|
|
|
- `passive_health_check` - Passive health checks are used to remove hosts from
|
|
the upstream cluster which are unreachable or are returning errors.
|
|
|
|
- `interval` - The time between checks. Each check will cause hosts which
|
|
have exceeded `max_failures` to be removed from the load balancer, and
|
|
any hosts which have passed their ejection time to be returned to the
|
|
load balancer.
|
|
- `max_failures` - The number of consecutive failures which cause a host to be
|
|
removed from the load balancer.
|
|
- `enforcing_consecutive_5xx` - The % chance that a host will be actually ejected
|
|
when an outlier status is detected through consecutive 5xx.
|
|
|
|
### Gateway Options
|
|
|
|
These fields may also be overridden explicitly in the [proxy service
|
|
definition](/docs/connect/registration/service-registration), or defined in
|
|
the [global `proxy-defaults` configuration
|
|
entry](/docs/connect/config-entries/proxy-defaults) to act as
|
|
defaults that are inherited by all services.
|
|
|
|
Prior to 1.8.0 these settings were specific to Mesh Gateways. The deprecated
|
|
names such as `envoy_mesh_gateway_bind_addresses` and `envoy_mesh_gateway_no_default_bind`
|
|
will continue to be supported.
|
|
|
|
- `connect_timeout_ms` - The number of milliseconds to allow when making upstream
|
|
connections before timing out. Defaults to 5000 (5 seconds). If the upstream
|
|
service has the configuration option
|
|
[`connect_timeout_ms`](/docs/connect/config-entries/service-resolver#connecttimeout)
|
|
set for the `service-resolver`, that timeout value will take precedence over
|
|
this gateway option.
|
|
|
|
- `envoy_gateway_bind_tagged_addresses` - Indicates that the gateway
|
|
services tagged addresses should be bound to listeners in addition to the
|
|
default listener address.
|
|
|
|
- `envoy_gateway_bind_addresses` - A map of additional addresses to be bound.
|
|
This map's keys are the name of the listeners to be created and the values are
|
|
a map with two keys, address and port, that combined make the address to bind the
|
|
listener to. These are bound in addition to the default address.
|
|
|
|
- `envoy_gateway_no_default_bind` - Prevents binding to the default address
|
|
of the gateway service. This should be used with one of the other options
|
|
to configure the gateway's bind addresses.
|
|
|
|
- `envoy_dns_discovery_type` - Determines how Envoy will resolve hostnames. Defaults to `LOGICAL_DNS`.
|
|
Must be one of `STRICT_DNS` or `LOGICAL_DNS`. Details for each type are available in
|
|
the [Envoy documentation](https://www.envoyproxy.io/docs/envoy/v1.17.2/intro/arch_overview/upstream/service_discovery).
|
|
This option applies to terminating gateways that route to services
|
|
addressed by a hostname, such as a managed database. It also applies to mesh gateways,
|
|
such as when gateways in other Consul datacenters are behind a load balancer that is addressed by a hostname.
|
|
|
|
## Advanced Configuration
|
|
|
|
To support more flexibility when configuring Envoy, several "lower-level" options exist
|
|
that require knowledge of Envoy's configuration format.
|
|
Many options allow configuring a subsection of either the bootstrap or
|
|
dynamic configuration using your own custom protobuf config.
|
|
|
|
We separate these into two sets, [Advanced Bootstrap
|
|
Options](#advanced-bootstrap-options) and [Escape Hatch
|
|
Overrides](#escape-hatch-overrides). Both require writing Envoy config in the
|
|
protobuf JSON encoding. Advanced options cover smaller chunks that might
|
|
commonly need to be set for tasks like configuring tracing. In contrast, escape hatches
|
|
give almost complete control over the proxy setup, but require operators to
|
|
manually code the entire configuration in protobuf JSON.
|
|
|
|
~> **Advanced Topic!** This section covers options that allow users to take almost
|
|
complete control of Envoy's configuration. We provide these options so users can
|
|
experiment or take advantage of features not yet fully supported in Consul Connect. We
|
|
plan to retain this ability in the future, but it should still be considered
|
|
experimental because it requires in-depth knowledge of Envoy's configuration format.
|
|
Users should consider Envoy version compatibility when using these features because they can configure Envoy in ways that
|
|
are outside of Consul's control. Incorrect configuration could prevent all
|
|
proxies in your mesh from functioning correctly, or bypass the security
|
|
guarantees Connect is designed to enforce.
|
|
|
|
### Configuration Formatting
|
|
|
|
All configurations are specified as strings containing the serialized proto3 JSON encoding
|
|
of the specified Envoy configuration type. They are full JSON types except where
|
|
noted.
|
|
|
|
The JSON supplied may describe a protobuf `types.Any` message with an `@type`
|
|
field set to the appropriate type (for example
|
|
`type.googleapis.com/envoy.config.listener.v3.Listener`).
|
|
|
|
For example, given a tracing config:
|
|
|
|
<CodeBlockConfig heading="Example envoy_tracing_json configuration">
|
|
|
|
```json
|
|
{
|
|
"http": {
|
|
"name": "envoy.tracers.zipkin",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
|
|
"collector_cluster": "zipkin",
|
|
"collector_endpoint_version": "HTTP_JSON",
|
|
"collector_endpoint": "/api/v1/spans",
|
|
"shared_span_context": false
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
JSON escape the value of `tracing` into a string, for example using [https://codebeautify.org/json-escape-unescape](https://codebeautify.org/json-escape-unescape),
|
|
or using [jq](https://stedolan.github.io/jq/).
|
|
|
|
```shell
|
|
$ cat <<EOF | jq '. | @json'
|
|
{
|
|
"http": {
|
|
"name": "envoy.tracers.zipkin",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
|
|
"collector_cluster": "zipkin",
|
|
"collector_endpoint_version": "HTTP_JSON",
|
|
"collector_endpoint": "/api/v1/spans",
|
|
"shared_span_context": false
|
|
}
|
|
}
|
|
}
|
|
EOF
|
|
"{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"zipkin\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v1/spans\",\"shared_span_context\":false}}}"
|
|
```
|
|
|
|
Then use that as the value for `envoy_tracing_json`:
|
|
|
|
```json
|
|
{
|
|
"kind": "proxy-defaults",
|
|
"name": "global",
|
|
"config": {
|
|
"envoy_tracing_json": "{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"zipkin\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v1/spans\",\"shared_span_context\":false}}}\n"
|
|
}
|
|
}
|
|
```
|
|
|
|
If using HCL, this escaping is done automatically:
|
|
|
|
```hcl
|
|
Kind = "proxy-defaults"
|
|
Name = "global"
|
|
Config {
|
|
envoy_tracing_json = <<EOF
|
|
{
|
|
"http": {
|
|
"name": "envoy.tracers.zipkin",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
|
|
"collector_cluster": "zipkin",
|
|
"collector_endpoint_version": "HTTP_JSON",
|
|
"collector_endpoint": "/api/v1/spans",
|
|
"shared_span_context": false
|
|
}
|
|
}
|
|
}
|
|
EOF
|
|
}
|
|
```
|
|
|
|
### Advanced Bootstrap Options
|
|
|
|
Users may add the following configuration items to the [global `proxy-defaults`
|
|
configuration
|
|
entry](/docs/connect/config-entries/proxy-defaults) or
|
|
override them directly in the `proxy.config` field of a [proxy service
|
|
definition](/docs/connect/registration/service-registration) or
|
|
[`sidecar_service`](/docs/connect/registration/sidecar-service) block.
|
|
|
|
- `envoy_extra_static_clusters_json` - Specifies one or more [Envoy clusters][pb-cluster]
|
|
that will be appended to the array of [static
|
|
clusters](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-staticresources-clusters)
|
|
in the bootstrap config. This enables you to add custom clusters for tracing sinks,
|
|
for example. In order to configure a single cluster, specify a single JSON object with the cluster details. For multiple
|
|
clusters, specify objects in a comma-separated list with no trailing comma. The
|
|
cluster objects will be interpolated directly into a JSON array.
|
|
|
|
<CodeBlockConfig heading="Example envoy_extra_static_clusters_json">
|
|
|
|
```json
|
|
{
|
|
"name": "local-service-cluster",
|
|
"load_assignment": {
|
|
"cluster_name": "local-service-cluster",
|
|
"endpoints": [
|
|
{
|
|
"lb_endpoints": [
|
|
{
|
|
"endpoint": {
|
|
"address": {
|
|
"socket_address": {
|
|
"address": "127.0.0.1",
|
|
"port_value": 32769
|
|
}
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
- `envoy_extra_static_listeners_json` - Similar to
|
|
`envoy_extra_static_clusters_json` but appends one or more [Envoy listeners][pb-listener] to the array of [static
|
|
listener](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-staticresources-listeners) definitions.
|
|
Can be used to setup limited access that bypasses Connect mTLS or
|
|
authorization for health checks or metrics.
|
|
|
|
<CodeBlockConfig heading="Example envoy_extra_static_listeners_json">
|
|
|
|
```json
|
|
{
|
|
"name": "test_envoy_mtls_bypass_listener",
|
|
"address": {
|
|
"socket_address": {
|
|
"address": "0.0.0.0",
|
|
"port_value": 20201
|
|
}
|
|
},
|
|
"filter_chains": [
|
|
{
|
|
"filters": [
|
|
{
|
|
"name": "envoy.filters.network.http_connection_manager",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
|
|
"http_filters": [
|
|
{
|
|
"name": "envoy.filters.http.router"
|
|
}
|
|
],
|
|
"route_config": {
|
|
"name": "self_admin_route",
|
|
"virtual_hosts": [
|
|
{
|
|
"name": "self_admin",
|
|
"domains": [
|
|
"*"
|
|
],
|
|
"routes": [
|
|
{
|
|
"match": {
|
|
"path": "/"
|
|
},
|
|
"route": {
|
|
"cluster": "local-service-cluster"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
},
|
|
"stat_prefix": "envoy_mtls_bypass",
|
|
"tracing": {
|
|
"random_sampling": {}
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
```
|
|
</CodeBlockConfig>
|
|
|
|
- `envoy_extra_stats_sinks_json` - Similar to `envoy_extra_static_clusters_json`
|
|
but for [stats sinks](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-sinks).
|
|
These are appended to any sinks defined by use of the
|
|
higher-level [`envoy_statsd_url`](#envoy_statsd_url) or
|
|
[`envoy_dogstatsd_url`](#envoy_dogstatsd_url) config options.
|
|
|
|
<CodeBlockConfig heading="Example envoy_extra_stats_sinks_json">
|
|
|
|
```json
|
|
{
|
|
"name": "envoy.stat_sinks.dog_statsd",
|
|
"typed_config": {
|
|
"@type": "type.googleapis.com/envoy.config.metrics.v3.DogStatsdSink",
|
|
"address": {
|
|
"socket_address": {
|
|
"protocol": "UDP",
|
|
"port_value": 8125,
|
|
"address": "172.31.20.6"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
</CodeBlockConfig>
|
|
|
|
- `envoy_stats_config_json` - The entire [stats
|
|
config](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-stats-config).
|
|
If provided this will override the higher-level
|
|
[`envoy_stats_tags`](#envoy_stats_tags). It allows full control over dynamic
|
|
tag replacements etc.
|
|
|
|
<CodeBlockConfig heading="Example envoy_stats_config_json">
|
|
|
|
```json
|
|
{
|
|
"stats_matcher": {
|
|
"reject_all": true
|
|
},
|
|
"stats_tags": [
|
|
{
|
|
"tag_name": "envoy.http_user_agent",
|
|
"regex": "^http(?=\\.).*?\\.user_agent\\.((.+?)\\.)\\w+?$"
|
|
}
|
|
],
|
|
"use_all_default_tags": false
|
|
}
|
|
```
|
|
</CodeBlockConfig>
|
|
|
|
- `envoy_tracing_json` - The entire [tracing
|
|
config](https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-field-config-bootstrap-v3-bootstrap-tracing).
|
|
Most tracing providers will also require adding static clusters to define the
|
|
endpoints to send tracing data to.
|
|
|
|
<CodeBlockConfig heading="Example envoy_tracing_json">
|
|
|
|
```json
|
|
{
|
|
"http": {
|
|
"name": "envoy.tracers.zipkin",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
|
|
"collector_cluster": "zipkin",
|
|
"collector_endpoint_version": "HTTP_JSON",
|
|
"collector_endpoint": "/api/v1/spans",
|
|
"shared_span_context": false
|
|
}
|
|
}
|
|
}
|
|
```
|
|
</CodeBlockConfig>
|
|
|
|
### Escape-Hatch Overrides
|
|
|
|
Users may add the following configuration items to the [global `proxy-defaults`
|
|
configuration
|
|
entry](/docs/connect/config-entries/proxy-defaults) or
|
|
override them directly in the `proxy.config` field of a [proxy service
|
|
definition](/docs/connect/registration/service-registration) or
|
|
[`sidecar_service`](/docs/connect/registration/sidecar-service) block.
|
|
|
|
- `envoy_bootstrap_json_tpl` - Specifies a template in Go template syntax that
|
|
is used in place of [the default
|
|
template](https://github.com/hashicorp/consul/blob/71d45a34601423abdfc0a64d44c6a55cf88fa2fc/command/connect/envoy/bootstrap_tpl.go#L129)
|
|
when generating bootstrap via [`consul connect envoy`
|
|
command](/commands/connect/envoy). The variables that are available
|
|
to be interpolated are [documented
|
|
here](https://github.com/hashicorp/consul/blob/71d45a34601423abdfc0a64d44c6a55cf88fa2fc/command/connect/envoy/bootstrap_tpl.go#L5).
|
|
This offers complete control of the proxy's bootstrap although major
|
|
deviations from the default template may break Consul's ability to correctly
|
|
manage the proxy or enforce its security model.
|
|
|
|
- `envoy_public_listener_json` - Specifies a complete [Envoy listener][pb-listener]
|
|
to be delivered in place of the main public listener that the proxy used to
|
|
accept inbound connections. This will be used verbatim with the following
|
|
exceptions:
|
|
|
|
- Every `FilterChain` added to the listener will have its `TlsContext`
|
|
overridden by the Connect TLS certificates and validation context. This
|
|
means there is no way to override Connect's mutual TLS for the public
|
|
listener.
|
|
- Every `FilterChain` will have the `envoy.filters.{network|http}.rbac` filter
|
|
prepended to the filters array to ensure that all inbound connections are
|
|
authorized by Connect. Before Consul 1.9.0 `envoy.ext_authz` was inserted instead.
|
|
|
|
<CodeTabs heading="Example envoy_public_listener_json" tabs={[ "HTTP", "TCP" ]}>
|
|
|
|
```json
|
|
{
|
|
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
|
|
"name": "public_listener",
|
|
"address": {
|
|
"socket_address": {
|
|
"address": "127.0.0.1",
|
|
"port_value": 21002
|
|
}
|
|
},
|
|
"filter_chains": [
|
|
{
|
|
"filters": [
|
|
{
|
|
"name": "envoy.filters.network.http_connection_manager",
|
|
"typed_config": {
|
|
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
|
|
"stat_prefix": "ingress_http",
|
|
"http_filters": [
|
|
{
|
|
"name": "envoy.filters.http.router"
|
|
}
|
|
],
|
|
"route_config": {
|
|
"name": "local_route",
|
|
"virtual_hosts": [
|
|
{
|
|
"name": "local_service",
|
|
"domains": ["*"],
|
|
"routes": [
|
|
{
|
|
"match": {
|
|
"prefix": "/"
|
|
},
|
|
"route": {
|
|
"cluster": "local-service-cluster",
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"traffic_direction": "INBOUND"
|
|
}
|
|
```
|
|
|
|
```json
|
|
{
|
|
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
|
|
"name": "public_listener",
|
|
"address": {
|
|
"socket_address": {
|
|
"address": "127.0.0.1",
|
|
"port_value": 21002
|
|
}
|
|
},
|
|
"filter_chains": [
|
|
{
|
|
"filters": [
|
|
{
|
|
"name": "envoy.filters.network.tcp_proxy",
|
|
"typed_config": {
|
|
"@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
|
|
"stat_prefix": "ingress_tcp",
|
|
"cluster": "local-service-cluster"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"traffic_direction": "INBOUND"
|
|
}
|
|
```
|
|
|
|
</CodeTabs>
|
|
|
|
|
|
- `envoy_listener_tracing_json` - Specifies a [tracing
|
|
configuration](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-msg-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-tracing)
|
|
to be inserted in the proxy's public and upstreams listeners.
|
|
|
|
<CodeBlockConfig heading="Example envoy_listener_tracing_json">
|
|
|
|
```json
|
|
{
|
|
"@type" : "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.Tracing",
|
|
"provider" : {
|
|
"name" : "envoy.tracers.zipkin",
|
|
"typed_config" : {
|
|
"@type" : "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
|
|
"collector_cluster" : "otelcolector",
|
|
"collector_endpoint" : "/api/v2/spans",
|
|
"collector_endpoint_version" : "HTTP_JSON",
|
|
"shared_span_context" : false
|
|
}
|
|
},
|
|
"custom_tags" : [
|
|
{
|
|
"tag" : "custom_header",
|
|
"request_header" : {
|
|
"name" : "x-custom-traceid",
|
|
"default_value" : ""
|
|
}
|
|
},
|
|
{
|
|
"tag" : "alloc_id",
|
|
"environment" : {
|
|
"name" : "NOMAD_ALLOC_ID"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
- `envoy_local_cluster_json` - Specifies a complete [Envoy cluster][pb-cluster]
|
|
to be delivered in place of the local application cluster. This allows
|
|
customization of timeouts, rate limits, load balancing strategy etc.
|
|
|
|
<CodeBlockConfig heading="Example envoy_local_cluster_json">
|
|
|
|
```json
|
|
{
|
|
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
|
|
"name": "local_app",
|
|
"type": "STATIC",
|
|
"connect_timeout": "5s",
|
|
"circuit_breakers": {
|
|
"thresholds": [
|
|
{
|
|
"priority": "DEFAULT",
|
|
"max_connections": 2048
|
|
}
|
|
]
|
|
},
|
|
"load_assignment": {
|
|
"cluster_name": "local_app",
|
|
"endpoints": [
|
|
{
|
|
"lb_endpoints": [
|
|
{
|
|
"endpoint": {
|
|
"address": {
|
|
"socket_address": {
|
|
"address": "127.0.0.1",
|
|
"port_value": 8080
|
|
}
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
|
|
</CodeBlockConfig>
|
|
|
|
|
|
The following configuration items may be overridden directly in the
|
|
`proxy.upstreams[].config` field of a [proxy service
|
|
definition](/docs/connect/registration/service-registration) or
|
|
[`sidecar_service`](/docs/connect/registration/sidecar-service) block.
|
|
|
|
~> **Note:** - When a
|
|
[`service-router`](/docs/connect/config-entries/service-router),
|
|
[`service-splitter`](/docs/connect/config-entries/service-splitter), or
|
|
[`service-resolver`](/docs/connect/config-entries/service-resolver) config
|
|
entry exists for a service the below escape hatches are ignored and will log a
|
|
warning.
|
|
|
|
- `envoy_listener_json` - Specifies a complete [Listener][pb-listener]
|
|
to be delivered in place of the upstream listener that the proxy exposes to
|
|
the application for outbound connections. This will be used verbatim with the
|
|
following exceptions:
|
|
|
|
- Every `FilterChain` added to the listener will have its `TlsContext`
|
|
overridden by the Connect TLS certificates and validation context. This
|
|
means there is no way to override Connect's mutual TLS for the public
|
|
listener.
|
|
|
|
<CodeTabs heading="Example upstream envoy_listener_json">
|
|
|
|
```json
|
|
{
|
|
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
|
|
"name": "example-service",
|
|
"address": {
|
|
"socket_address": {
|
|
"address": "0.0.0.0",
|
|
"port_value": 14000
|
|
}
|
|
},
|
|
"filter_chains": [
|
|
{
|
|
"filters": [
|
|
{
|
|
"name": "envoy.filters.network.http_connection_manager",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
|
|
"access_log": [
|
|
{
|
|
"name": "envoy.access_loggers.file",
|
|
"typedConfig": {
|
|
"@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
|
|
"path": "/var/log/envoy-access/example-service.log"
|
|
}
|
|
}
|
|
],
|
|
"http_filters": [
|
|
{
|
|
"name": "envoy.filters.http.router"
|
|
}
|
|
],
|
|
"route_config": {
|
|
"name": "example-service",
|
|
"virtual_hosts": [
|
|
{
|
|
"name": "example-service",
|
|
"domains": [
|
|
"*"
|
|
],
|
|
"routes": [
|
|
{
|
|
"match": {
|
|
"prefix": "/"
|
|
},
|
|
"route": {
|
|
"cluster": "example-service",
|
|
"timeout": "90s",
|
|
"retry_policy": {
|
|
"retry_on": "5xx,connect-failure",
|
|
"num_retries": 2,
|
|
"per_try_timeout": "60s"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
},
|
|
"stat_prefix": "example-service",
|
|
"tracing": {
|
|
"random_sampling": {}
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"traffic_direction": "OUTBOUND"
|
|
}
|
|
```
|
|
</CodeTabs>
|
|
|
|
- `envoy_cluster_json` - Specifies a complete [Envoy cluster][pb-cluster]
|
|
to be delivered in place of the discovered upstream cluster. This allows
|
|
customization of timeouts, circuit breaking, rate limits, load balancing
|
|
strategy etc.
|
|
|
|
<CodeTabs heading="Example upstream envoy_cluster_json">
|
|
|
|
```json
|
|
{
|
|
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
|
|
"name": "example-service",
|
|
"type": "EDS",
|
|
"eds_cluster_config": {
|
|
"eds_config": {
|
|
"ads": {}
|
|
}
|
|
},
|
|
"connect_timeout": "90s",
|
|
"lb_policy": "ROUND_ROBIN",
|
|
"circuit_breakers": {
|
|
"thresholds": [
|
|
{
|
|
"priority": "DEFAULT",
|
|
"max_connections": 1024,
|
|
"max_pending_requests": 1024,
|
|
"max_requests": 1024,
|
|
"max_retries": 3
|
|
}
|
|
]
|
|
}
|
|
}
|
|
```
|
|
</CodeTabs>
|
|
|
|
[protocol]: /docs/connect/config-entries/service-defaults#protocol
|
|
[intentions]: /docs/connect/intentions
|
|
[intentions]: /docs/connect/intentions
|
|
[pb-cluster]: https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/cluster/v3/cluster.proto
|
|
[pb-listener]: https://www.envoyproxy.io/docs/envoy/v1.17.2/api-v3/config/listener/v3/listener.proto
|