luxolus/open-consul
2
0
Fork 0
Code Issues 1 Pull requests Packages Releases Wiki Activity

Merge pull request #10891 from hashicorp/dnephin/docs-remove-old-overview

Browse source 
docs: move the remaining INTERNALS.md content
This commit is contained in:
Daniel Nephin 2021-08-30 14:22:10 -04:00 committed by GitHub
parent 2497afed0d 7a2ff886a8
commit b5b902033b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 57 additions and 48 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
INTERNALS.md
View file

@ -1 +1 @@
Moved to [contributing/INTERNALS.md].
Moved to [contributing/README.md](./contributing/README.md).

42
contributing/INTERNALS.md
View file

@ -1,41 +1 @@
# Consul Internals Guide
This guide is intended to help folks who want to understand more about how Consul works from a code perspective, or who are thinking about contributing to Consul. For a high level overview of Consul's design, please see the [Consul Architecture Guide](https://www.consul.io/docs/internals/architecture.html) as a starting point.
## Architecture Overview
Consul is designed around the concept of a [Consul Agent](https://www.consul.io/docs/agent/basics.html). The agent is deployed as a single Go binary and runs on every node in a cluster.
A small subset of agents, usually 3 to 7, run in server mode and participate in the [Raft Consensus Protocol](https://www.consul.io/docs/internals/consensus.html). The Consul servers hold a consistent view of the state of the cluster, including the [service catalog](https://www.consul.io/api/catalog.html) and the [health state of services and nodes](https://www.consul.io/api/health.html) as well as other items like Consul's [key/value store contents](https://www.consul.io/api/kv.html). An agent in server mode is a superset of the client capabilities that follow.
All the remaining agents in a cluster run in client mode. Applications on client nodes use their local agent in client mode to [register services](https://www.consul.io/api/agent.html) and to discover other services or interact with the key/value store. For the latter queries, the agent sends RPC requests internally to one of the Consul servers for the information. None of the key/value data is on any of the client agents, for example, it's always fetched on the fly from a Consul server.
Both client and server mode agents participate in a [Gossip Protocol](https://www.consul.io/docs/internals/gossip.html) which provides two important mechanisms. First, it allows for agents to learn about all the other agents in the cluster, just by joining initially with a single existing member of the cluster. This allows clients to discover new Consul servers. Second, the gossip protocol provides a distributed failure detector, whereby the agents in the cluster randomly probe each other at regular intervals. Because of this failure detector, Consul can run health checks locally on each agent and just sent edge-triggered updates when the state of a health check changes, confident that if the agent dies altogether then the cluster will detect that. This makes Consul's health checking design very scaleable compared to centralized systems with a central polling type of design.
There are many other aspects of Consul that are well-covered in Consul's [Internals Guides](https://www.consul.io/docs/internals/index.html).
## Source Code Layout
### Shared Components
The components in this section are shared between Consul agents in client and server modes.
| Directory | Contents |
| --------- | -------- |
| [command/agent](https://github.com/hashicorp/consul/tree/main/command/agent) | This contains the actual CLI command implementation for the `consul agent` command, which mostly just invokes an agent object from the `agent` package. |
| [agent](https://github.com/hashicorp/consul/tree/main/agent) | This is where the agent object is defined, and the top level `agent` package has all of the functionality that's common to both client and server agents. This includes things like service registration, the HTTP and DNS endpoints, watches, and top-level glue for health checks. |
| [agent/config](https://github.com/hashicorp/consul/tree/main/agent/config) | This has all the user-facing [configuration](https://www.consul.io/docs/agent/options.html) processing code, as well as the internal configuration structure that's used by the agent. |
| [agent/checks](https://github.com/hashicorp/consul/tree/main/agent/checks) | This has implementations for the different [health check types](https://www.consul.io/docs/agent/checks.html). |
| [agent/ae](https://github.com/hashicorp/consul/tree/main/agent/ae), [agent/local](https://github.com/hashicorp/consul/tree/main/agent/local) | These are used together to power the agent's [Anti-Entropy Sync Back](https://www.consul.io/docs/internals/anti-entropy.html) process to the Consul servers. |
| [agent/router](https://github.com/hashicorp/consul/tree/main/agent/router), [agent/pool](https://github.com/hashicorp/consul/tree/main/agent/pool) | These are used for routing RPC queries to Consul servers and for connection pooling. |
| [agent/structs](https://github.com/hashicorp/consul/tree/main/agent/structs) | This has definitions of all the internal RPC protocol request and response structures. |
### Other Components
There are several other top-level packages used internally by Consul as well as externally by other applications.
| Directory | Contents |
| --------- | -------- |
| [api](https://github.com/hashicorp/consul/tree/main/api) | This `api` package provides an official Go API client for Consul, which is also used by Consul's [CLI](https://www.consul.io/docs/commands/index.html) commands to communicate with the local Consul agent. |
| [api/watch](https://github.com/hashicorp/consul/tree/main/api/watch) | This has implementation details for Consul's [watches](https://www.consul.io/docs/agent/watches.html), used both internally to Consul and by the [watch CLI command](https://www.consul.io/docs/commands/watch.html). |
| [website](https://github.com/hashicorp/consul/tree/main/website) | This has the full source code for [consul.io](https://www.consul.io/). Pull requests can update the source code and Consul's documentation all together. |
Moved to [README.md](./README.md).

31
contributing/README.md
View file

@ -20,7 +20,6 @@ found in the public [user documentation].
## Contents
1. [Overview](./INTERNALS.md)
1. [Command-Line Interface (CLI)](./cli)
1. [HTTP API](./http-api)
1. [Agent Configuration](./config)
@ -36,6 +35,36 @@ found in the public [user documentation].
Also see the [FAQ](./faq.md).
## Important Directories
Most top level directories contain Go source code. The directories listed below
contain other important source related to Consul.
* [ui] contains the source code for the Consul UI.
* [website] contains the source for [consul.io](https://www.consul.io/). A pull requests
can update the source code and Consul's documentation at the same time.
* [.circleci] and [.github] contain the source for our CI and GitHub repository
automation.
* [.changelog] contains markdown files that are used by [hashicorp/go-changelog] to produce the
[CHANGELOG.md].
* [build-support] contains bash functions and scripts used to automate.
development tasks. Generally these scripts are called from the [GNUmakefile].
* [grafana] contains the source for a [Grafana dashboard] that can be used to
monitor Consul.
[ui]: https://github.com/hashicorp/consul/tree/main/ui
[website]: https://github.com/hashicorp/consul/tree/main/website
[.circleci]: https://github.com/hashicorp/consul/tree/main/.circleci
[.github]: https://github.com/hashicorp/consul/tree/main/.github
[.changelog]: https://github.com/hashicorp/consul/tree/main/.changelog
[hashicorp/go-changelog]: https://github.com/hashicorp/go-changelog
[CHANGELOG.md]: https://github.com/hashicorp/consul/blob/main/CHANGELOG.md
[build-support]: https://github.com/hashicorp/consul/tree/main/build-support
[GNUmakefile]: https://github.com/hashicorp/consul/tree/main/GNUmakefile
[Grafana dashboard]: https://grafana.com/grafana/dashboards
[grafana]: https://github.com/hashicorp/consul/tree/main/grafana
## Contributing to these docs
This section is meta documentation about contributing to these docs.

6
contributing/client-agent/README.md
View file

@ -1,5 +1,7 @@
# Client Agent
- agent/cache
- agent/local (local state)
- anti-entropy sync
- [agent/local](https://github.com/hashicorp/consul/tree/main/agent/local)
- anti-entropy sync in [agent/ae](https://github.com/hashicorp/consul/tree/main/agent/ae) powering the [Anti-Entropy Sync Back](https://www.consul.io/docs/internals/anti-entropy.html) process to the Consul servers.
Applications on client nodes use their local agent in client mode to [register services](https://www.consul.io/api/agent.html) and to discover other services or interact with the key/value store.

5
contributing/cluster-membership/README.md
View file

@ -1,7 +1,12 @@
# Cluster membership
This section is a work in progress. It will contain topics like the following:
- hashicorp/serf
- hashicorp/memberlist
- network coordinates
- consul events
- consul exec
Both client and server mode agents participate in a [Gossip Protocol](https://www.consul.io/docs/internals/gossip.html) which provides two important mechanisms. First, it allows for agents to learn about all the other agents in the cluster, just by joining initially with a single existing member of the cluster. This allows clients to discover new Consul servers. Second, the gossip protocol provides a distributed failure detector, whereby the agents in the cluster randomly probe each other at regular intervals. Because of this failure detector, Consul can run health checks locally on each agent and just sent edge-triggered updates when the state of a health check changes, confident that if the agent dies altogether then the cluster will detect that. This makes Consul's health checking design very scaleable compared to centralized systems with a central polling type of design.

8
contributing/http-api/README.md
View file

@ -1,3 +1,9 @@
# HTTP API
Work in progress.
Work in progress. This section will eventually contain docs about:
* the HTTP "framework"
* HTTP endpoints
* the [api](https://github.com/hashicorp/consul/tree/main/api) client library - the `api` package
provides an official Go API client for Consul, which is also used by Consul's
[CLI](https://www.consul.io/docs/commands/index.html) commands to communicate with the local Consul agent.

8
contributing/rpc/README.md
View file

@ -40,11 +40,15 @@ This section is a work in progress, it will eventually cover topics like:
- net/rpc - (in the stdlib)
- new grpc endpoints
- [Streaming](./streaming)
- [agent/structs](https://github.com/hashicorp/consul/tree/main/agent/structs) - contains definitions of all the internal RPC protocol request and response structures.
## RPC connections and load balancing
This section is a work in progress, it will eventually cover topics like:
- agent/router
- agent/pool
Routing RPC request to Consul servers and for connection pooling.
- [agent/router](https://github.com/hashicorp/consul/tree/main/agent/router)
- [agent/pool](https://github.com/hashicorp/consul/tree/main/agent/pool)

3
contributing/service-discovery/health-checks.md
View file

@ -2,6 +2,9 @@
This section is still a work in progress.
[agent/checks](https://github.com/hashicorp/consul/tree/main/agent/checks) contains the logic for
performing active [health checking](https://www.consul.io/docs/agent/checks.html).
## Check Registration flows