---
layout: docs
page_title: Custom Proxy Configuration | Service Mesh
description: >-
Consul supports custom proxy integrations for service discovery and sidecar instantiation. Learn about proxy requirements for service mesh operations, as well as how to authorize inbound and outbound connections for your custom proxy.
---
# Custom Proxy Configuration for Service Mesh
This topic describes the process and API endpoints you can use to extend proxies for integration with Consul.
## Overview
You can extend any proxy to support Connect. Consul ships with a built-in
proxy suitable for an out-of-the-box development experience, but you may require a more robust proxy solution for production environments.
The proxy you integrate must be able to accept inbound connections and/or establish outbound connections identified as a particular service. In some cases, either ability may be acceptable, but both are generally required to support for full sidecar functionality.
Sidecar proxies may support L4 or L7 network functionality. L4 integration is simpler and adequate for securing all traffic. L4 treats all traffic as TCP, however, so advanced routing or metrics features are not supported.
Full L7 support is built on top of L4 support. An L7 proxy integration supports most or all of the L7 traffic routing features in Connect by dynamically configuring routing, retries, and other L7 features. The built-in proxy only supports L4, while [Envoy](/consul/docs/connect/proxies/envoy) supports the full L7 feature set.
Areas where the integration approach differs between L4 and L7 are identified in this topic.
## Accepting Inbound Connections
The proxy must accept TLS connections on some port to accept inbound connections.
### Obtaining and validating client certificates
Call the [`/v1/agent/connect/ca/leaf/`] API endpoint to obtain the client certificate, e.g.:
```shell
$ curl http://:8500/v1/agent/connect/ca/leaf/
```
The client certificate from the inbound connection must be validated against the Connect CA root certificates. Call the [`/v1/agent/connect/ca/roots`] endpoint to obtain the root certificates from the Connect CA, e.g.:
```shell
$ curl http://:8500/v1/agent/connect/ca/roots
```
### Authorizing the connection
After validating the client certificate from the caller, the proxy can authorize the entire connection (L4) or each request (L7). Depending upon the [protocol] of the proxied service, authorization is performed either on a per-connection (L4) or per-request (L7) basis. Authentication is based on "service identity" (TLS), and is implemented at the
transport layer.
-> **Note:** Some features, such as (local) rate limiting or max connections, are expected to be proxy-level configurations enforced separately when authorization calls are made. Proxies can enforce the configurations based on information about request rates and other states that should already be available.
The proxy can authorize the connection by either calling the [`/v1/agent/connect/authorize`](/consul/api-docs/agent/connect) API endpoint or by querying the [intention match API](/consul/api-docs/connect/intentions#list-matching-intentions) endpoint.
The [`/v1/agent/connect/authorize`](/consul/api-docs/agent/connect) endpoint should be called in the connection path for each received connection.
If the local Consul agent is down or unresponsive, the success rate of new connections will be compromised.
The agent uses locally-cached data to authorize the connection and typically responds in microseconds. As a result, the TLS handshake typically spans microseconds.
~> **Note:** This endpoint is only suitable for L4 (e.g., TCP) integration. The endpoint always treats intentions with `Permissions` defined (i.e., L7 criteria) as `deny` intentions during evaluation.
The proxy can query the [intention match API](/consul/api-docs/connect/intentions#list-matching-intentions) endpoint on startup to retrieve a list of intentions that match the proxy destination. The matches should be stored in the native filter configuration of the proxy, such as RBAC for Envoy.
For performance and reliability reasons, querying the intention match API endpoint is the recommended method for implementing intention enforcement. The cached intentions should be consulted for each incoming connection (L4) or request (L7) to determine if the connection or request should be accepted or rejected.
#### Persistent TCP connections and intentions
For a proxied service configured with the TCP [protocol], potentially long-lived TCP connections will only be authorized when the connections are initially established. But because many services, such as databases, typically use persistent connection pools, changing intentions to deny access does not terminate existing connections. This behavior violates the updated intention. In these cases, it may appear as if the intention is not being enforced.
Implement one of the following strategies to close connections:
1. **Configure connections to terminate after a maximum lifetime**, e.g., several hours. This balances the overhead of establishing new connections with determining how long existing connections remain open after an intention changes.
1. **Periodically re-authorize every open connection**. The authorization call is inexpensive and should be a local, in-memory operation on the Consul agent. Periodically authorizing thousands of open connections (e.g., once every minute) is likely to be negligible overhead, but doing so enforces a tighter upper boundary on how long it takes to enforce intention changes without affecting the protocol efficiency of persistent connections.
#### Certificate serial in authorization
Intentions currently use TLS URI Subject Alternative Name (SAN) for enforcement. The `AuthZ` API in the Go SDK contains a field for passing the serial number ([`consul/connect/tls.go`]). Proxies may provide this value during authorization.
### Updating data
The API endpoints described in this section operate on agent-local data that is updated in the
background. The leaf, roots, and intentions should be updated in the background
by the proxy.
The leaf cert, root cert, and intentions endpoints support [blocking
queries](/consul/api-docs/features/blocking), which should be used to get near-immediate
updates for root key rotations, new leaf certs before expiry, and intention
changes.
### SPIFFE certificates
Although Consul follows the SPIFFE spec for certificates, some CA providers do not allow strict adherence. For example, CA certificates may not have the correct trust-domain SPIFFE URI SAN for the
cluster. If SPIFFE validation is performed in the proxy, be aware that it
should be possible to opt out, otherwise certain CA providers supported by
Consul will not be compatible with the use of that proxy. Neither
Envoy nor the built-in proxy currently validate the SPIFFE URI of the chain beyond the
leaf certificate.
## Establishing Outbound Connections
For outbound connections, the proxy should communicate with a Connect-capable
endpoint for a service and provide a client certificate from the
[`/v1/agent/connect/ca/leaf/`] API endpoint. The proxy must use the root certificate obtained from the [`/v1/agent/connect/ca/roots`] endpoint to verify the certificate served from the destination endpoint.
## Configuration Discovery
The [`/v1/agent/service/:service_id`](/consul/api-docs/agent/service#get-service-configuration)
API endpoint enables any proxy to discover proxy configurations registered with a local service. This endpoint supports hash-based blocking, which enables long-polling for changes
to the registration/configuration. Any changes to the registration/config will
result in the new config being returned immediately.
Refer to the [built-in proxy](/consul/docs/connect/proxies/built-in) for an example implementation. Using the Go SDK, the proxy calls the HTTP "pull" API via the `watch` package: [`consul/connect/proxy/config.go`].
The [discovery chain] for each upstream service should be fetched from the
[`/v1/discovery-chain/:service_id`](/consul/api-docs/discovery-chain#read-compiled-discovery-chain)
API endpoint. This will return a compiled graph of configurations a sidecar needs for a particular upstream service.
If you are only implementing L4 support in your proxy, set the
[`OverrideProtocol`](/consul/api-docs/discovery-chain#overrideprotocol) value to `tcp` when
fetching the discovery chain so that L7 features, such as HTTP routing rules, are
not returned.
For each [target](/consul/docs/connect/l7-traffic/discovery-chain#targets) in the resulting
discovery chain, a list of healthy, Connect-capable endpoints may be fetched
from the [`/v1/health/connect/:service_id`] API endpoint as described in the [Service
Discovery](#service-discovery) section.
The remaining nodes in the chain include configurations that should be
translated into the nearest equivalent for features, such as HTTP routing, connection
timeouts, connection pool settings, rate limits, etc. See the full [discovery
chain] documentation and relevant [config entry](/consul/docs/agent/config-entries)
documentation for details about supported configuration parameters.
### Service Discovery
Proxies can use Consul's [service discovery API](https://consul.io/%60/v1/health/connect/:service_id%60) to return all available, Connect-capable endpoints for a given service. This endpoint supports a `cached` query parameter, which uses [agent caching](/consul/api-docs/features/caching) to improve
performance. The API package provides a [`UseCache`] query option to leverage caching.
In addition to performance improvements, using the cache makes the mesh more resilient to Consul server outages. This is because the mesh "fails static" with the last known set of service instances still used, rather than errors on new connections.
Proxies can decide whether to perform just-in-time queries to the API when a
new connection needs to be routed, or to use blocking queries to load the
current set of endpoints for a service and keep that list updated. The SDK and
built-in proxy currently use just-in-time resolution however many existing
proxies are likely to find it easier to integrate by pulling the set of
endpoints and maintaining it in local memory using blocking queries.
Upstreams may be defined with the Prepared Query target type. These upstreams
should use Consul's [prepared query](/consul/api-docs/query) API to determine a list of upstream endpoints for the service. Note that the `PreparedQuery` API does not support blocking, so proxies choosing to populate endpoints in memory will need to poll the endpoint at a suitable and, ideally, configurable frequency.
-> **Long-term support for [`service-resolver`](/consul/docs/connect/config-entries/service-resolver) configuration
entries**. The `service-resolver` configuration will completely replace prepared queries in future versions of Consul. In some instances, however, prepared queries are still used.
## Sidecar Instantiation
Consul does not start or manage sidecar proxy processes. Proxies running on a
physical host or VM are designed to be started and run by process supervisor
systems, such as init, systemd, supervisord, etc. If deployed within a
cluster scheduler (Kubernetes, Nomad), proxies should run as a sidecar container in the
same namespace.
Proxies will use the [`CONSUL_HTTP_TOKEN`](/consul/commands#consul_http_token) and
[`CONSUL_HTTP_ADDR`](/consul/commands#consul_http_addr) environment variables to
contact Consul and fetch certificates. This occurs if the `CONSUL_HTTP_TOKEN`
environment variable contains a Consul ACL token that has the necessary permissions
to read configurations for that service. If you use the Go [`api` package], then
the environment variables will be read and the client configured for you
automatically.
Alternatively, you may also use the flags `-token` or `-token-file` to provide the Consul ACL token.
```shell
$ consul connect envoy -sidecar-for "web" -token-file=/etc/consul.d/consul.token
```
```shell
$ consul connect proxy -sidecar-for "web" -token-file=/etc/consul.d/consul.token
```
If TLS is enabled on Consul, you will also need to add the following environment variables _prior_ to starting the proxy:
- [`CONSUL_CACERT`](/consul/commands#consul_cacert)
- [`CONSUL_CLIENT_CERT`](/consul/commands#consul_client_cert)
- [`CONSUL_CLIENT_KEY`](/consul/commands#consul_client_key)
The `CONSUL_CACERT`, `CONSUL_CLIENT_CERT` and `CONSUL_CLIENT_KEY` can also be provided as CLI flags. Refer to the [`consul connect proxy` documentation](/consul/commands/connect/proxy) for details.
The proxy service ID comes from the user. See [`consul connect envoy`](/consul/commands/connect/envoy#examples) for an example. You can use the `-proxy-id` flag to specify the ID of the proxy service you have already registered with the local agent.
Alternatively, you can start the service using the `-sidecar-for=` option. This option queries Consul for a proxy that is registered as a sidecar for the specified ``. If exactly one service associated with the proxy is returned, the ID will be used to start the proxy. Your controller only needs to accept `-proxy-id` as an argument; the Consul CLI will resolve the
ID for the name specified in `-sidecar-for` flag.
[`/v1/agent/connect/ca/leaf/`]: /consul/api-docs/agent/connect#service-leaf-certificate
[`/v1/agent/connect/ca/roots`]: /consul/api-docs/agent/connect#certificate-authority-ca-roots
[`api` package]: https://github.com/hashicorp/consul/tree/main/api
[`consul/connect/proxy/config.go`]: https://github.com/hashicorp/consul/blob/v1.8.3/connect/proxy/config.go#L187
[`consul/connect/tls.go`]: https://github.com/hashicorp/consul/blob/v1.8.3/connect/tls.go#L232-L237
[discovery chain]: /consul/docs/connect/l7-traffic/discovery-chain
[`usecache`]: https://github.com/hashicorp/consul/blob/v1.8.3/api/api.go#L99-L102
[protocol]: /consul/docs/connect/config-entries/service-defaults#protocol