Add initial dev docs on config entries (#15717)

docs/service-mesh/README.md

@@ -1,6 +1,7 @@
 # Service Mesh (Connect)
 
 - call out: envoy/proxy is the data plane, Consul is the control plane
+- [Configuration Entries](./config-entries)
 - [xDS Server] - a gRPC service that implements [xDS] and handles requests from an [envoy proxy].
 - [agent/proxycfg]
 - [Certificate Authority](./ca) for issuing TLS certs for services and client agents

docs/service-mesh/config-entries/README.md

@@ -0,0 +1,30 @@
+# Configuration Entries
+## Background
+Consul's catalog was designed to store node, service, and check registrations. An instance of a service is defined by a combination of a node and a service registered against that node. Each of these service registrations is considered to be an _instance_ of a logical service.
+
+Logical services in consul are implicit. There is no true logical service "object" that defines the existence of a service. In Consul, a service is a collection of service instances. When the first instance of a service is registered, the service becomes known, and when the last instance is deregistered then the service is mostly forgotten.
+
+The lack of a first-class service object makes it so that the source of truth for the configuration of a service is the configuration of its service instances. Registering services at the edge has the side-effect of making it more challenging to enforce uniform configuration across service or proxy instances.
+
+## Overview
+Configuration entries are the primary way to apply configuration or policies uniformly across the mesh. They are stored centrally on Consul's servers, and can be scoped to a service, namespace, a cluster, or a federation of datacenters.
+
+With WAN-federation, the scope of config entries spans across Consul datacenters. All config entry writes are forwarded to the primary datacenter, and each secondary datacenter has blocking queries to the primary datacenter so that they can replicate in the latest entries. The implications of their global scope means that a config entry referencing service "web" in namespace "frontend" will apply to **all** services named "web" in a namespace called "frontend"  across federated datacenters.
+
+Since the introduction of admin partitions in Consul v1.11.0, the broadest scope for a config entry is a partition. A service a namespace of one partition is logically distinct from a service with the same name and namespace in another partition.
+
+This restriction also applies when peering two clusters. As of Consul 1.14, configuration entries are not replicated to peer clusters.  Configuration entries applied in one cluster do not apply to services in a peer cluster because services in different clusters that share the same name are considered to be logically different.
+
+The diagram below shows how various resources are scoped:
+* Service and gateway config entries exist within a namespace but can span across WAN federated datacenters, just as namespaces can.
+* The proxy-defaults config entry is not namespaced, but can similarly span across datacenters.
+* Config entries in admin partition "default" are logically distinct from config entries in other admin partitions like "team1".
+
+![World of Data](./world-of-data.png)
+
+For sidecar proxies the fundamental config entries are [service-defaults](https://developer.hashicorp.com/consul/docs/connect/config-entries/service-defaults) and [proxy-defaults](https://developer.hashicorp.com/consul/docs/connect/config-entries/proxy-defaults). These config entries can be thought of as default values for proxy or gateway service registrations. Their data gets merged into proxy service registrations themselves.
+
+As defaults, their data is of lower precedence compared to data stored in proxy registrations. In Consul, data present in individual proxy registrations **always** has a higher precedence than the equivalent stored in a configuration entry.
+
+## Additional Information:
+- [Config Resolution](config-resolution.md): Summary of the mechanics of how configuration entries are resolved for sidecar proxies.

docs/service-mesh/config-entries/config-resolution.md

@@ -0,0 +1,25 @@
+# Configuration Entries
+## Merging Service Config
+Merging data from defaults in config entries is roughly a two-step process:
+1. Flatten central configuration for proxies at a Consul server. Resolving config is done on behalf of a specific service, and this merge considers multiple config entries, such as proxy-defaults and service-defaults.
+2. Merge flattened central defaults into sidecar proxy registrations. This occurs at a client agent in the agentful case, or on a server in the agentless case.
+
+Any time the centralized defaults are updated, internal watches will fire and their data will be merged into the registration for the given sidecar proxy. You can ensure that a config entry has applied by querying the service:
+* Agentful: `/v1/agent/service/:service_id`
+* Agentless: `/v1/health/connect/:service_name?merge-central-config`
+
+### Agentless
+The agentless flow relies on [configentry.MergeNodeServiceWithCentralConfig](https://github.com/hashicorp/consul/blob/0402fd23a349513d3e8d137ddbffcdefcc89838b/agent/configentry/merge_service_config.go) to merge default configuration. This helper is called both when bootstrapping a new dataplane proxy and when an Envoy proxy first dials the `xds` server on a Consul server.
+
+This merge does not directly update the proxy registration in the catalog. The merged proxy registration is passed to the consuming `proxycfg` package.
+
+### Agentful
+The [agent.ServiceManager](https://github.com/hashicorp/consul/blob/0402fd23a349513d3e8d137ddbffcdefcc89838b/agent/service_manager.go#LL18) is responsible for ensuring that central configuration is merged down into proxy registrations on **client agents**.  Any time a proxy is registered or updated, the service manager will set up a watch through the agent cache to fetch the central configuration.
+
+When the central configuration is fetched, the configuration is merged and persisted to the agent state in [serviceConfigWatch.handleUpdate](https://github.com/hashicorp/consul/blob/0402fd23a349513d3e8d137ddbffcdefcc89838b/agent/service_manager.go#L256).
+
+## Consuming Central Config
+Once centralized defaults have been merged into service registrations, the data becomes available to consume by the `proxycfg` package. This package assembles the necessary Consul data to generate Envoy configuration for a given proxy.
+
+The `proxycfg` package does not need explicit watches for the service or proxy-defaults that apply to a sidecar proxy because of how these defaults are merged into proxy registrations. When defaults change, a fresh `proxycfg` snapshot is built based on the latest registration.
+