xDS Server

The agent/xds package implements the streaming DeltaAggregatedResources gRPC service used by Envoy to fetch configuration. At the core of the package is delta.go, which contains the implementation of the Incremental ADS protocol variant. With this variant there is a single stream between Consul and an Envoy proxy, and on that stream we send configuration diffs based on Envoy's current state.

The remainder of this package contains the logic necessary for generating xDS configuration such as Clusters, Endpoints, Listeners, and Routes from snapshots generated by proxycfg.

Authorization

The xDS server authorizes requests by looking at the proxy ID in the request and ensuring the ACL token has service:write access to either the destination service (for kind=ConnectProxy), or the gateway service (for other kinds).

This authorization strategy is based on the assumption of how the corresponding proxycfg.ConfigSnapshot is constructed. Most interfaces (HTTP, DNS, RPC) authorize requests by authorizing the data in the response, or by filtering out data that the requester is not authorized to view.

This authorization strategy requires that agent/proxycfg only fetches data using a token with the same permissions, and that it only stores data by proxy ID. We assume that any data in the snapshot was already filtered, which allows this authorization at the xDS server to only perform a shallow check against the proxy ID.

Config Generation

The xDS types that Consul supports as of v1.14 are: Clusters, Endpoints, Listeners, and Routes. For each of these resource types there is a corresponding file such as listeners.go. There, the entry-point will take a proxycfg snapshot and generate xDS configuration depending on the kind of proxy being configured. There are diverging paths depending on whether a sidecar is being configured, or a gateway.

Testing

Testing changes to this package is generally done at two layers:

Against golden files, where each test case tests against a fixed file containing the JSON representation of an xDS resource.
In integration tests, which spin up live instances of Consul and Envoy and make assertions against Envoy's metrics or configuration.

Golden files

Tests against golden files exists in functions with names such as TestAllResourcesFromSnapshot, TestListenersFromSnapshot, etc. These tests generate xDS configuration from a proxycfg.ConfigSnapshot, mimicking how we generate configuration for Envoy.

The primary source for the test snapshots is proxycfg.TestConfigSnapshot. This function will construct a snapshot from a list of events by calling initialize and handleUpdate as we do in production code. You can attach new update events to the snapshot, or override existing events by emitting a replacement event for an existing CorrelationID.

When a new test case is added, the corresponding Golden files can be generated using:

go test ./agent/xds -update -run TestAllResourcesFromSnapshot

The new golden files then must be manually inspected to ensure that the Envoy configuration was generated as expected. Tests against golden files do not assert that the configuration works as intended, but rather that it looks as intended.

Integration tests

New consul-container integration tests

TODO

Legacy bash-driven integration tests

Updating one of these integration tests may be appropriate if fixing a bug in functionality that is tested there, or making an improvement to functionality tested there. If the new test involves significant modifications to the bash helpers you should consider using adding a consul-container integration test instead.

For more information refer to their documentation.

Delta (Incremental) xDS Protocol

Consul's implementation of the incremental xDS protocol exists in the file delta.go. The interactions between Envoy follow the general guidance from Envoy's documentation,

The xDS stream is a bidirectional stream, and messages from Envoy can be either a: subscription request for resources, an ACK for resources it successfully stored, or a NACK for resources that it did not store. ACKs and NACKs can be associated with specific messages by tracking a "nonce", which Consul generates and pushes in every response sent to Envoy. Consul will only ever send xDS resources, into the stream and will only send resources if it believes that Envoy does not already have them.

For mock examples of how interactions over the stream can play out, refer to the tests in delta_test.go.

To achieve the aim of only sending incremental diffs to Envoy there are maps tracking the state of both Envoy and Consul. The xDSDeltaType struct contains this data for each xDS resource type:

subscriptions tracks the names that Envoy is interested in for a given xDS type. For example, if Envoy is subscribed to Routes, then this map would contain the names of the routes that Envoy is interested in receiving updates about.
resourceVersions tracks the hash of resources that Envoy has ACK'd. This way we can avoid sending data that Envoy already has.
pendingUpdates holds a map used for tracking data that has been sent to Envoy but has not been ACK'd yet. Certain xDS types have strict ordering requirements to avoid dropping traffic. Tracking pending updates allows us to block sending another update of the same type or sending resources that depend on a previous one being acknowledged.

5.9 KiB Raw Blame History