contributing: start an outline for more docs

Add diagrams for rpc routing and acl entity relationship
contributing: create directory structure for new docs
WIP diagram for catalog entities
Add overview diagram

Co-Authored-By: Kelly Devlin <kdevlin@hashicorp.com>

contributing/README.md

@@ -5,10 +5,25 @@ See [our contributing guide](../.github/CONTRIBUTING.md) to get started.
 This directory contains documentation intended for anyone interested in
 understanding, and contributing changes to, the Consul codebase.
 
+## Overview
+
+This documentation is organized into the following categories. Each category is 
+either a significant architectural layer, or major functional area of Consul.
+
+![Overview](./overview.svg)
+
+<sup>[source](./overview.mmd)</sup>
+
 ## Contents 
 
 1. [Overview](./INTERNALS.md)
-2. [Configuration](./checklist-adding-config-fields.md)
-3. [Streaming](./streaming)
-4. [Network Areas](./network-areas)
-5. [Service Discovery](./service-discovery)
+1. [Agent Configuration](./config)
+1. [RPC](./rpc)
+1. [Cluster Persistence](./persistence)
+1. [Client Agent](./client-agent)
+1. [Service Discovery](./service-discovery)
+1. [Service Mesh (Connect)](./service-mesh)
+1. [Cluster Membership](./cluster-membership)
+1. [Key/Value Store](./kv)
+1. [ACL](./acl)
+1. [Multi-Cluster Federation](./cluster-federation)

contributing/acl/acl-entity-relationship.mmd

@@ -0,0 +1,28 @@
+erDiagram
+
+    Token
+    Policy
+    Role
+    ServiceIdentity
+    NodeIdentity
+    AuthMethod
+    BindingRule
+    Rule
+    // TODO: rules are made up of resources and x (enforcement decision or permission?)
+    // TODO: add Authorizer and Enforcement Decision
+
+    Policy ||--|{ Rule: grants
+    Role ||--|{ Policy: includes
+    Role }|--|{ ServiceIdentity: includes
+    Role }|--|{ NodeIdentity: includes
+
+    Token }|--|{ Policy: includes
+    Token }|--|{ Role: includes
+    Token }|--|{ ServiceIdentity: includes
+    Token }|--|{ NodeIdentity: includes
+
+    AuthMethod ||--|{ BindingRule: defines
+    AuthMethod ||--|{ Token: creates
+
+    ServiceIdentity ||--|{ Rule: implies
+    NodeIdentity ||--|{ Rule: implies

contributing/client-agent/README.md

@@ -0,0 +1,5 @@
+# Client Agent
+
+- agent/cache
+- agent/local (local state)
+- anti-entropy sync

contributing/cluster-federation/README.md

@@ -0,0 +1,4 @@
+# Multi-Cluster Federation
+
+1. [Network Areas](./network-areas)
+

contributing/cluster-membership/README.md

@@ -0,0 +1,7 @@
+# Cluster membership
+   - hashicorp/serf
+   - hashicorp/memberlist
+   - network coordinates
+   - consul events
+   - consul exec
+

contributing/config/README.md

@@ -0,0 +1,3 @@
+# Agent Configuration
+
+- [Checklist for adding a new field](./checklist-adding-config-fields.md)

contributing/consul-mermaid-theme.json

@@ -0,0 +1,4 @@
+{
+  "theme": "default",
+  "themeCSS": ".node rect { fill: rgb(220, 71, 125); stroke-width: 1; stroke: black; } .node .label { color: white; } "
+}

contributing/overview.mmd

@@ -0,0 +1,24 @@
+graph TD
+    
+    ServiceMesh[Sercice Mesh]
+    ServiceDiscovery[Service Discovery]
+    ClusterMembership[Cluster Membership]
+    KV[Key/Value Store]
+    MultiClusterFederation[Multi-Cluster Federation]
+
+    ACL
+    AgentConfiguration[Agent Configuration]
+    ClientAgent[Client Agent]
+    RPC
+    ClusterPersistence[Cluster Persistence]
+
+    AgentConfiguration --> ClientAgent
+    ClientAgent --> RPC
+    ClientAgent --> ACL
+    RPC --> ClusterPersistence
+    RPC --> ACL
+
+    MultiClusterFederation --> ClusterMembership
+    MultiClusterFederation --> RPC
+    ServiceMesh --> ServiceDiscovery
+    

contributing/persistence/README.md

@@ -0,0 +1,8 @@
+# Persistence
+
+- hashicorp/raft
+- state.Store - hashicorp/go-memdb
+- FSM
+- boltdb - https://github.com/boltdb/bolt (https://github.com/etcd-io/bbolt)
+- snapshot and restore
+

contributing/rpc/README.md

@@ -0,0 +1,6 @@
+# RPC
+
+- net/rpc - (in the stdlib)
+- [Streaming](./streaming)
+- routing of "RPC" requests
+

contributing/rpc/rpc-routing.mmd

@@ -0,0 +1,33 @@
+graph LR
+
+    handleConn
+
+    handleConn -->|RPCConsul| handleConsulConn
+    handleConn -->|RPCRaft| raftLayer
+    handleConn -->|RPCTLS| handleConn
+    handleConn -->|RPCMultiplexV2| handleMultiplexV2
+    handleConn -->|RPCSnapshot| handleSnapshotConn
+    handleConn -->|RPCTLSInsecure| handleInsecureConn
+    handleConn -->|RPCGossip| handleGossipConn
+
+    handleConsulConn --> RPCServer
+    handleMultiplexV2 --> handleConsulConn
+
+    %% new after 1.6.9
+
+    handleConn -->|PeekForTLS| handleNativeTLS
+
+    handleNativeTLS -->|ALPN_RPCConsul| handleConsulConn
+    handleNativeTLS -->|ALPN_RPCRaft| raftLayer
+    handleNativeTLS -->|ALPN_RPCMultiplexV2| handleMultiplexV2
+    handleNativeTLS -->|ALPN_RPCSnapshot| handleSnapshotConn
+    handleNativeTLS -->|ALPN_RPCGRPC| grpcHandler
+    handleNativeTLS -->|ALPN_WANGossipPacket| handleWANGossipPacket
+    handleNativeTLS -->|ALPN_WANGossipStream | handleWANGossipStream
+    handleNativeTLS -->|ALPN_RPCGossip| handleGossipConn
+
+    handleMultiplexV2 -->|RPCGossip| handleGossipConn
+    handleConn -->|RPCGRPC| grpcHandler
+
+
+

contributing/service-discovery/README.md

@@ -2,4 +2,6 @@
 
 This section is still a work in progress.
 
+1. [catalog](./catalog.md)
 1. [DNS Interface](./dns.md)
+1. health checking

contributing/service-discovery/catalog-2.mmd

@@ -0,0 +1,36 @@
+erDiagram
+   
+    CheckServiceNode
+    Node
+    NodeService
+    ServiceNode
+    HealthCheck
+
+    CheckServiceNode ||--|| Node: has
+    CheckServiceNode ||--|| NodeService: has
+    CheckServiceNode ||--o{ HealthCheck: has
+
+    Store ||--o{ Node: "stored in the node table"
+    Store ||--o{ ServiceNode: "stored in the service table"
+    Store ||--o{ HealthCheck: "stored in the checks table"
+
+    ServiceNode ||--|| Node: references
+    HealthCheck ||--o| Node: references
+    HealthCheck ||--o| Service: references
+
+    RegisterRequest ||--o| Node: has
+    RegisterRequest ||--o| NodeService: has
+    RegisterRequest ||--o{ HealthCheck: has
+
+
+    CheckDefinition
+    HealthCheckDefinition
+    CheckType
+
+    HealthCheck ||--|| HealthCheckDefinition: has
+
+    ServiceDefinition ||--|| NodeService: "is essentially a"
+    ServiceDefinition ||--o{ CheckType: "has"
+
+    Config ||--o{ CheckDefinition: "has"
+    Config ||--o{ ServiceDefinition: "has"

contributing/service-discovery/catalog.md

@@ -0,0 +1,6 @@
+# Catalog
+
+This section is a work in progress.
+
+The catalog is at the core of both Service Discovery and Service Mesh. It accepts
+registrations and deregistrations of Services, Nodes, and Checks.

contributing/service-discovery/catalog.mmd

@@ -0,0 +1,24 @@
+erDiagram
+   
+    CheckServiceNode
+    Node
+    NodeService
+    ServiceNode
+    HealthCheck
+
+    CheckServiceNode ||--|| Node: has
+    CheckServiceNode ||--|| NodeService: has
+    CheckServiceNode ||--o{ HealthCheck: has
+
+    Store ||--o{ Node: "stored in the node table"
+    Store ||--o{ ServiceNode: "stored in the service table"
+    Store ||--o{ HealthCheck: "stored in the checks table"
+
+    ServiceNode ||--|| Node: references
+    HealthCheck ||--o| Node: references
+    HealthCheck ||--o| Service: references
+
+    RegisterRequest ||--o| Node: has
+    RegisterRequest ||--o| NodeService: has
+    RegisterRequest ||--o{ HealthCheck: has
+

contributing/service-mesh/README.md

@@ -0,0 +1,12 @@
+# Service Mesh (Connect)
+
+- call out: envoy/proxy is the data plane, Consul is the control plane
+- agent/xds - gRPC service that implements
+  [xDS](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol)
+- [agent/proxycfg](https://github.com/hashicorp/consul/blob/master/agent/proxycfg/proxycfg.go)
+- CA Manager - certificate authority
+- command/connect/envoy - bootstrapping and running envoy
+- command/connect/proxy - built-in proxy that is dev-only and not supported 
+  for production.
+- `connect/` - "Native" service mesh
+