open-consul/website/content/commands/connect/envoy.mdx

---
layout: commands
page_title: 'Commands: Connect Envoy'
sidebar_title: envoy
description: The connect envoy subcommand is used to generate a bootstrap configuration for Envoy.
---

# Consul Connect Envoy

Command: `consul connect envoy`

The connect Envoy command is used to generate a bootstrap configuration for
[Envoy proxy](https://envoyproxy.io) for use with [Consul
Connect](/docs/connect/).

The default behavior is to generate the necessary bootstrap configuration for
Envoy based on the environment variables and options provided and by talking to
the local Consul agent. It `exec`s an external Envoy binary with that
configuration leaving the Envoy process running in the foreground. An error is
returned on operating systems other than linux or macOS since Envoy does not
build for other platforms currently.

If the `-bootstrap` option is specified, the bootstrap config is generated in
the same way and then printed to stdout. This allows it to be redirected to a
file and used with `envoy --config-path bootstrap.json`. This works on all operating
systems allowing configuration to be generated on a host that Envoy doesn't
build on but then used in a virtualized environment that can run Envoy.

## Usage

Usage: `consul connect envoy [options] [-- pass-through options]`

#### Envoy Options for both Sidecars and Gateways

- `-proxy-id` - The [proxy service](/docs/connect/registration/service-registration) ID.
  This service ID must already be registered with the local agent unless a gateway is being
  registered with the `-register` flag. As of Consul 1.8.0, this can also be
  specified via the `CONNECT_PROXY_ID` environment variable.

- `-envoy-binary` - The full path to a specific Envoy binary to exec. By
  default the current `$PATH` is searched for `envoy`.

- `-admin-bind` - The `host:port` to bind Envoy's admin HTTP API. Default is
  `localhost:19000`. Envoy requires that this be enabled. The host part must be
  resolvable DNS name or IP address.

- `-bootstrap` - If present, the command will simply output the generated
  bootstrap config to stdout in JSON protobuf form. This can be directed to a
  file and used to start Envoy with `envoy --config-path bootstrap.json`.

  ~> **Security Note:** If ACLs are enabled the bootstrap JSON will contain the
  ACL token from `-token` or the environment and so should be handled as a secret.
  This token grants the identity of any service it has `service:write` permission
  for and so can be used to access any upstream service that that service is
  allowed to access by [Connect intentions](/docs/connect/intentions).

- `-envoy-version` - The version of envoy that is being started. Default is
  `1.19.1`. This is required so that the correct configuration can be generated.

- `-no-central-config` - By default the proxy's bootstrap configuration can be
  customized centrally. This requires that the command run on the same agent
  as the proxy will and that the agent is reachable when the command is run.
  In cases where either assumption is violated this flag will prevent the
  command attempting to resolve config from the local agent.

- `-prometheus-backend-port` - Sets the backend port for the "prometheus_backend"
  cluster that `envoy_prometheus_bind_addr` will point to. Without this flag,
  `envoy_prometheus_bind_addr` would point to the "self_admin" cluster where Envoy metrics
  are exposed. The metrics merging feature in consul-k8s uses this to point to the
  merged metrics endpoint combining Envoy and service metrics.
	Only applicable when `envoy_prometheus_bind_addr` is set in proxy config.

- `-prometheus-scrape-path` - Sets the path where Envoy will expose metrics on the
  `envoy_prometheus_bind_addr` listener. Default is `/metrics`. For example, if `envoy_prometheus_bind_addr`
  is `0.0.0.0:20200`, and this flag is set to `/scrape-metrics`, prometheus metrics would
  be scrapable at `0.0.0.0:20200/scrape-metrics`.
	Only applicable when `envoy_prometheus_bind_addr` is set in proxy config.

- `-prometheus-ca-file` - Path to a CA file for Envoy to use when serving TLS on
  the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
  is set in proxy config.

- `-prometheus-ca-path` - Path to a directory of CA certificates for Envoy to use when
  serving the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
  is set in proxy config.

- `-prometheus-cert-file` - Path to a certificate file for Envoy to use when serving
  TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
  is set in proxy config.

- `-prometheus-key-file` - Path to a private key file for Envoy to use when serving
  TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
  is set in proxy config.

- `-- [pass-through options]` - Any options given after a double dash are passed
  directly through to the `envoy` invocation. See [Envoy's
  documentation](https://www.envoyproxy.io/docs) for more details. The command
  always specifies `--config-file` and `--v2-config-only` and by default passes
  `--disable-hot-restart` see [hot restart](#envoy-hot-restart).

#### Envoy Sidecar Proxy Options

- `-sidecar-for` - The _ID_ (not name if they differ) of the service instance
  this proxy will represent. The target service doesn't need to exist on the
  local agent yet but a [sidecar proxy
  registration](/docs/connect/registration/service-registration) with
  `proxy.destination_service_id` equal to the passed value must be present. If
  multiple proxy registrations targeting the same local service instance are
  present the command will error and `-proxy-id` should be used instead.
  As of Consul 1.8.0, this can also be specified via the `CONNECT_SIDECAR_FOR`
  environment variable.

  -> **Note:** If ACLs are enabled, a token granting `service:write` for the
  _target_ service (configured in `proxy.destination_service_name`) must be
  passed using the `-token` option or `CONSUL_HTTP_TOKEN` environment variable.
  This token authorizes the proxy to obtain TLS certificates representing the
  target service.

#### Envoy Gateway Options

- `-gateway` - Flag to indicate that Envoy should be configured as a Gateway.
  Must be one of: `terminating`, `ingress`, or `mesh`.
  If multiple gateways are managed by the same local agent then
  `-proxy-id` should be used as well to specify the instance this represents.

- `-register` - Indicates that the gateway service should be registered
  with the local agent instead of expecting it to already exist. This flag
  is unused for traditional sidecar proxies.

- `-address` - The address to advertise for services within the local datacenter
  to use to reach the gateway instance. This flag is used in combination with
  `-register`. This takes the form of `<ip address>:<port>` but also supports go-sockaddr
  templates.

  If Envoy is configured as a terminating or mesh gateway, traffic from services
  within the mesh will be received at the specified IP and port.

  If Envoy is configured as an ingress gateway, a `/ready` HTTP endpoint will be
  instantiated at the specified IP and port. Consul uses `/ready` HTTP endpoints
  to check gateway health. The specified IP will also be used by the ingress
  gateway when instantiating user-defined listeners configured in the
  [ingress gateway](/docs/connect/gateways/ingress-gateway) configuration entry.

  ~> **Note**: Ensure that user-defined ingress gateway listeners use a
     different port than the port specified in `-address` so that they do not
     conflict with the health check endpoint.

- `-admin-access-log-path` The path to write the access log for the administration
  server. If no access log is desired specify `/dev/null`. By default it will
  use `/dev/null`.

- `-bind-address` - The bind address to use instead of the default binding rules.
  Specified as `<name>=<ip>:<port>` pairs. This flag may be used multiple times
  to add multiple bind addresses.

- `-expose-servers` - Expose the servers for WAN federation via this mesh
  gateway.

- `-wan-address` - The address to advertise for services within remote datacenters
  to use to reach the gateway instance. This flag is used in combination with
  `-register`. This takes the form of `<ip address>:<port>` but also supports go-sockaddr
  templates.

- `-service` - The name of the gateway service to register. This flag is used
  in combination with `-register`.

- `-deregister-after-critical` - The amount of time the gateway services health check can
  be failing before being deregistered. This flag is used in combination with `-register`

#### Enterprise Options

@include 'http_api_partition_options.mdx'

@include 'http_api_namespace_options.mdx'

#### API Options

The standard API options are used to connect to the local agent to discover the
proxy configuration needed.

- `-grpc-addr=<addr>` - Address of the Consul agent with `grpc` port. This can
  be an IP address or DNS address, but it must include the port. This can also
  be specified via the CONSUL_GRPC_ADDR environment variable. In Consul 1.3 and
  later, the default value is 127.0.0.1:8502, and https can optionally
  be used instead. The scheme can also be set to HTTPS by setting the
  environment variable CONSUL_HTTP_SSL=true. This may be a unix domain socket
  using `unix:///path/to/socket` if the [agent is configured to
  listen](/docs/agent/config/config-files#addresses) that way.

  -> **Note:** gRPC uses the same TLS
  settings as the HTTPS API. If HTTPS is enabled then gRPC will require HTTPS
  as well.

  @include 'http_api_options_client.mdx'

## Examples

Assume a local service instance is registered on the local agent with a
sidecar proxy (using the [sidecar service
registration](/docs/connect/registration/service-registration) helper) as below.

```hcl
service {
  name = "web"
  port = 8080
  connect { sidecar_service {} }
}
```

### Basic Sidecar Proxy

The sidecar Envoy process can be started with.

```shell-session
$ consul connect envoy -sidecar-for web
```

This example assumes that the correct [environment variables](#api-options) are
used to set the local agent connection information and ACL token, or that the
agent is using all-default configuration.

### Additional Envoy Arguments

To pass additional arguments directly to Envoy, for example output logging
level, you can use:

```shell-session
$ consul connect envoy -sidecar-for web -- --log-level debug
```

### Multiple Proxy Instances

To run multiple different proxy instances on the same host, you will
need to use `-admin-bind` on all but one to ensure they don't attempt to bind to
the same port as in the following example.

```shell-session
$ consul connect envoy -sidecar-for db -admin-bind localhost:19001
```

### Mesh Gateways

The mesh gateway Envoy process can be auto-registered and started with the following command.

```shell-session
$ consul connect envoy -gateway=mesh -register \
  -address '{{ GetInterfaceIP "eth0" }}:8443' \
  -wan-address '{{ GetInterfaceIP "eth1" }}:8443'
```

### Terminating Gateways

The terminating gateway Envoy process can be auto-registered and started with the following command.

```shell-session
$ consul connect envoy -gateway=terminating -register -service my-gateway \
  -address '{{ GetInterfaceIP "eth0" }}:8443'
```

### Ingress Gateways

The ingress gateway Envoy process can be auto-registered and started with the following command.

```shell-session
$ consul connect envoy -gateway=ingress -register -service ingress-service \
  -address '{{ GetInterfaceIP "eth0" }}:8888'
```

## Exec Security Details

The command needs to pass the bootstrap config through to Envoy. Envoy currently
only supports passing this as a file path or passing a whole string on the
command line with `--config-yaml`. Since the bootstrap needs to contain the ACL
token to authorize the proxy, this secret needs careful handling.

Passing a secret via command option is unacceptable as on many unix systems
these are readable to any user on the host for example via `/proc` or via a
setuid process like `ps`.

Creating a temporary file is more secure in that it can only be read by the
current user but risks leaving secret material on disk for an unbounded length
of time and in a location that is opaque to the operator.

To work around these issues, the command currently creates a temporary file and
immediately unlinks it so it can't be read by any other process that doesn't
already have the file descriptor. It then writes the bootstrap JSON, and unsets
the CLOEXEC bit on the file handle so that it remains available to the Envoy
process after exec. Finally it `exec`s Envoy with `--config-file /dev/fd/X`
where `X` is the the file descriptor number of the temp file.

This ensures that Envoy can read the file without any other normal user process
being able to (assuming they don't have privileged access to /proc). Once the
Envoy process stops, there is no longer any reference to the file to clean up.

## Envoy Hot Restart

Envoy supports hot restart which requires simple external coordination. By
default, this command will add `--disable-hot-restart` when it runs Envoy.

The reason for this default behavior is to make it easy to test and run local
demonstrations with multiple Envoy instances outside of cgroups or network
namespaces.

To use hot restart, Envoy needs to be started with either the `--restart-epoch`
option. If this command detects that option in the pass-through flags it will
_not_ add `--disable-hot-restart` allowing hot restart to work normally.

The only difference to note over running Envoy directly is that
`--restart-epoch` must be explicitly set to `0` for the initial launch of the
Envoy instance to avoid disabling hot restart entirely. The official
`hot-restarter.py` always sets this option so should work as recommended.