open-consul/website/content/docs/connect/proxies/integrate.mdx

---
layout: docs
page_title: Connect - Proxy Integration
description: >-
  A Connect-aware proxy enables unmodified applications to use Connect. A
  per-service proxy sidecar transparently handles inbound and outbound service
  connections, automatically wrapping and verifying TLS connections.
---

# Connect Custom Proxy Integration

Any proxy can be extended to support Connect. Consul ships with a built-in
proxy for a good development and out of the box experience, but production
users will require other proxy solutions.

A proxy must serve one or both of the following two roles: it must accept
inbound connections or establish outbound connections identified as a
particular service. One or both of these may be implemented depending on the
case, although generally both must be supported for full sidecar functionality.

There are also two different levels of compatibility as a sidecar: L4 or L7.
L4 integration is simpler and adequate to secure all traffic but treats all
traffic as TCP so no advanced routing or metrics features can be supported.
Full L7 support is built on top of L4 support and includes supporting most or
all of the L7 traffic routing features in Connect by dynamically configuring
routing, retries and more L7 features. Currently The built-in proxy only
supports L4 while Envoy supports the full L7 feature set.

Places where the integration approach diverges for L4/L7 support is indicated
below.

## Accepting Inbound Connections

For inbound connections, the proxy must accept TLS connections on some port.
The certificate served should be obtained from the
[`/v1/agent/connect/ca/leaf/`] API endpoint. The client certificate should be
validated against the root certificates provided by the
[`/v1/agent/connect/ca/roots`] endpoint. After validating the client
certificate from the caller, depending upon the [protocol] of the proxied
service service the proxy must either authorize the entire connection (L4) or
each request (L7).

Connection authorization can be performed one of two ways:

1. The first is by calling the
   [`/v1/agent/connect/authorize`](/api/agent/connect) endpoint. The authorize
   endpoint is expected to be called in the connection path, so if the local
   Consul agent is down or unresponsive it will impact the success rate of new
   connections. The agent uses locally cached data to authorize the connection
   and typically responds in microseconds. Therefore, the impact to the TLS
   handshake is typically microseconds.

   ~> **Note:** This endpoint is only suited for networking layer 4 (e.g. TCP)
   integration. The endpoint will always treat intentions with Permissions
   defined (i.e., layer 7 criteria) as deny intentions during evaluation.

2. Alternatively, proxies may list intentions that match the destination by
   querying the [intention match
   API](/api/connect/intentions#list-matching-intentions) endpoint, and
   represent them in the native configuration of the proxy itself (such as RBAC
   for Envoy). For performance and reliability reasons this is the desirable
   method for implementing intention enforcement. The cached intentions should
   be consulted for each incoming connection (L4) or request (L7) to determine
   if the should be accepted or rejected.

All of these API endpoints operate on agent-local data that is updated in the
background. The leaf, roots, and intentions should be updated in the background
by the proxy.

The leaf cert, root cert, and intentions endpoints support [blocking
queries](/api/features/blocking), which should be used to get near-immediate
updates for root key rotations, new leaf certs before expiry, and intention
changes.

Although Consul follows the SPIFFE spec for certificates, some currently
supported CA providers don't allow strict adherence. For example, CA
certificates may not have the correct trust-domain SPIFFE URI SAN for the
cluster. If SPIFFE validation is performed in the proxy, be aware that it
should be possible to opt out, otherwise certain CA providers supported by
Consul will not be compatible with the use of that proxy. Currently neither
Envoy nor the built-in proxy validate the SPIFFE URI of the chain beyond the
leaf certificate.

### Connection Authorization

Authentication is based on "service identity" (TLS), and is implemented at the
transport layer. Depending upon the [protocol] of the proxied service,
authorization is performed either on a per-connection (L4) or per-request (L7)
basis.

-> **Note:** Features like (local) rate limiting or max connections are
configurations that we expect to push into proxies and have them enforce
separately to the AuthZ call based on the state they already have about request
rates etc.

#### Persistent TCP Connections and Intentions

For a proxied service configured with a [protocol] of TCP, potentially
long-lived TCP connections will be authorized only when they are established.
Since many services (e.g. databases) typically use persistent connection pools,
a change in intentions that newly denies access currently does not terminate
existing connections in violation of the updated intention. In this case it may
appear as if the intention is not being enforced.

Consul eventually may support a mechanism for tracking specific connections in
the agent and then allow the agent to tell the proxy to close those connections
when their authorization state changes, but for now that is not on the roadmap.

It is recommended therefore to do one of the following:

1. Have connections terminate after a configurable maximum lifetime of say
   several hours. This balances the overhead of establishing new connections
   while keeping an upper bound on how long after Intention changes existing
   connections remain open.
2. Periodically re-authorize every open connection. The AuthZ call itself is
   not expensive and should be a local, in-memory operation so authorizing
   thousands of open connections once every minute or so is likely to be
   negligible overhead, but enforces a tighter upper bound on how long it takes
   to enforce Intention changes without affecting protocol efficiency of
   persistent connections.

#### Certificate Serial in AuthZ

Intentions currently utilize TLS' URI Subject Alternative Name (SAN) for
enforcement. In the future, Consul will support revoking specific certificates
by serial number. The AuthZ API in the Go SDK has a field to pass the serial
number ([`consul/connect/tls.go`]). Proxies may provide this value during
authorization.

## Establishing Outbound Connections

For outbound connections, the proxy should communicate to a Connect-capable
endpoint for a service and provide a client certificate from the
[`/v1/agent/connect/ca/leaf/`] API endpoint. The certificate served by the
remote endpoint may be verified against the root certificates from the
[`/v1/agent/connect/ca/roots`] endpoint.

## Configuration Discovery

Any proxy can discover proxy configuration registered with a local service
instance using the
[`/v1/agent/service/:service_id`](/api/agent/service#get-service-configuration)
API endpoint.

This endpoint supports hash-based blocking, enabling long-polling for changes
to the registration/configuration. Any changes to the registration/config will
result in the new config being returned immediately. An example implementation
may be found in our [built-in proxy](/docs/connect/proxies/built-in) which
utilizes our Go SDK, and uses the HTTP "pull" API (via our `watch` package):
[`consul/connect/proxy/config.go`].

The [discovery chain] for each upstream service should be fetched from the
[`/v1/discovery-chain/:service_id`](/api/discovery-chain#read-compiled-discovery-chain)
API endpoint. This will return a compiled graph of configurations needed by
sidecars for a particular upstream service. If you are only implementing L4
support in your proxy, set the
[`OverrideProtocol`](/api/discovery-chain#overrideprotocol) value to "tcp" when
fetching the discovery chain so that L7 features such as HTTP routing rules are
not returned.

For each [target](/docs/internals/discovery-chain#targets) in the resulting
discovery chain, a list of healthy, Connect-capable endpoints may be fetched
from the [`/v1/health/connect/:service_id`] API endpoint per the [Service
Discovery](#service-discovery) section below.

The rest of the nodes in the chain include configurations that should be
translated into the nearest equivalent for things like HTTP routing, connection
timeouts, connection pool settings, rate limits, etc. See the full [discovery
chain] documentation and relevant [config entry](/docs/agent/config-entries)
documentation for details of supported configuration parameters.

We expect config here to evolve reasonably rapidly. While we do not intend to
make backwards incompatible API changes, there are likely to be new
configurations and features added regularly. Some proxies may not be able to
support all features or may have differing semantics with the way they support
them. We intend to find a suitable format to document the behavior differences
between proxy implementations as they mature.

### Service Discovery

Proxies can use Consul's service discovery API
[`/v1/health/connect/:service_id`] to return all available, Connect-capable
endpoints for a given service. This endpoint supports a `?cached` parameter
which makes use of [agent caching](/api/features/caching) and thus has
performance benefits. The API package provides a [`UseCache`] query option to
leverage this. In addition to performance improvements, use of the cache makes
the mesh more resilient to Consul server outages - the mesh "fails static" with
the last known set of service instances still used rather than errors on new
connections.

Proxies can decide whether to perform just-in-time queries to the API when a
new connection needs to be routed, or to use blocking queries to load the
current set of endpoints for a service and keep that list updated. The SDK and
built-in proxy currently use just-in-time resolution however many existing
proxies are likely to find it easier to integrate by pulling the set of
endpoints and maintaining it in local memory using blocking queries.

Upstreams can be defined with Prepared Query target types. These upstreams
should use Consul's [prepared query](/api/query) API. It's worth noting that
the PreparedQuery API does not support blocking, so proxies choosing to
populate endpoints in memory will need to poll the endpoint at a suitable and
ideally configurable frequency.

-> **Note:** Long-term the [`service-resolver` config
entries](/docs/connect/config-entries/service-resolver) are intended to replace
Prepared Queries in Consul entirely, but for now these are still used in some
configurations.

## Sidecar instantiation

Consul does not start or manage sidecar proxies processes. Proxies running on a
physical host or VM are designed to be started and run by process supervisor
systems such as init, systemd, supervisord, etc. Or, if deployed within a
cluster scheduler (Kubernetes, Nomad) running as a sidecar container in the
same namespace.

The proxy will use the [`CONSUL_HTTP_TOKEN`](/commands#consul_http_token) and
[`CONSUL_HTTP_ADDR`](/commands#consul_http_addr) environment variables to
contact Consul to fetch certificates, provided the `CONSUL_HTTP_TOKEN`
environment variable contains a Consul ACL that has the necessary permissions
to read configuration for that service. If you use our Go [`api` package] then
those environment variables will be read and the client configured for you
automatically.

The ID of the proxy service comes from the user. See [`consul connect envoy`](/commands/connect/envoy) as an example. You may start it with the
`-proxy-id` flag and pass the ID of the proxy service you registered elsewhere.
A nicer UX is available for end-users using the `-sidecar-for=<service>`
argument, which causes the command to query Consul for a proxy that is
registered as a sidecar for the specified `<service>`. If there is exactly one
such proxy, that ID will be used to start the proxy. Your controller only needs
to accept `-proxy-id` as an argument; the Consul CLI will handle resolving the
ID for the name specified in `-sidecar-for`.

[`/v1/agent/connect/ca/leaf/`]: /api/agent/connect#service-leaf-certificate
[`/v1/agent/connect/ca/roots`]: /api/agent/connect#certificate-authority-ca-roots
[`/v1/health/connect/:service_id`]: /api/health#list-nodes-for-connect-capable-service
[`api` package]: https://github.com/hashicorp/consul/tree/master/api
[`consul/connect/proxy/config.go`]: https://github.com/hashicorp/consul/blob/v1.8.3/connect/proxy/config.go#L187
[`consul/connect/tls.go`]: https://github.com/hashicorp/consul/blob/v1.8.3/connect/tls.go#L232-L237
[discovery chain]: /docs/connect/l7-traffic/discovery-chain
[`usecache`]: https://github.com/hashicorp/consul/blob/v1.8.3/api/api.go#L99-L102
[protocol]: /docs/connect/config-entries/service-defaults#protocol