From d8123b2c6a9ac1e06f83f7e757d26aac626eedf5 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 6 Jun 2019 12:18:45 -0700 Subject: [PATCH 1/7] docs: initial attempt at developer docs --- README.md | 5 ++++- docs/README.md | 38 ++++++++++++++++++++++++++++++++++++++ docs/checklist-command.md | 32 ++++++++++++++++++++++++++++++++ docs/checklist-jobspec.md | 25 +++++++++++++++++++++++++ 4 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 docs/README.md create mode 100644 docs/checklist-command.md create mode 100644 docs/checklist-jobspec.md diff --git a/README.md b/README.md index 2ad0f8e23..f2207b532 100644 --- a/README.md +++ b/README.md @@ -140,7 +140,10 @@ Who Uses Nomad Contributing to Nomad -------------------- -If you wish to contribute to Nomad, you will need [Go](https://www.golang.org) installed on your machine (version 1.10.2+ is *required*). +If you wish to contribute to Nomad, you will need [Go](https://www.golang.org) +installed on your machine (version 1.10.2+ is *required*). + +See the [`docs`](docs/) directory for more developer documentation. **Developing with Vagrant** There is an included Vagrantfile that can help bootstrap the process. The diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..cb462a23a --- /dev/null +++ b/docs/README.md @@ -0,0 +1,38 @@ +# Nomad Codebase Documentation + +This directory contains some documentation about the Nomad codebase, +aimed at readers who are interested in making code contributions. + +If you're looking for information on _using_ using, please instead refer +to [the main Nomad website](https://nomadproject.io). + +## Architecture + +The code for Nomad's major components is organized as: + +* `api/` provides a Go client for Nomad's HTTP API. +* `client/` contains Nomad's client agent code. +* `command/` contains Nomad's CLI code. +* `nomad/` contains Nomad's server agent code. + +The high level control flow for many Nomad commands are: + +``` +# Read commands: +CLI command -> HTTP API -> RPC -> StateStore + +# Commands that change state: +CLI command -> HTTP API -> RPC -> Raft -> FSM -> StateStore +``` + +## Checklists + +When adding new features to Nomad there are often many places to make changes. +It is difficult to determine where changes must be made and easy to make +mistakes. + +The following checklists are meant to be copied and pasted into PRs to give +developers and reviewers confidence that the proper changes have been made: + +* [New `jobspec` entry](checklist-jobspec.md) +* [New CLI command](checklist-command.md) diff --git a/docs/checklist-command.md b/docs/checklist-command.md new file mode 100644 index 000000000..9cf7de829 --- /dev/null +++ b/docs/checklist-command.md @@ -0,0 +1,32 @@ +# New CLI command + +Subcommands should always be preferred over adding more top-level commands. + +Code flow for commands is generally: + +``` +CLI (command/) -> API Client (api/) -> HTTP API (command/agent) -> RPC (nomad/) +``` + +## Code + +1. [ ] New file in `command/` or in an existing file if a subcommand +2. [ ] Test new command in `command/` package +3. [ ] Implement autocomplete +4. [ ] Implement `-json` (returns raw API response) +5. [ ] Implement `-verbose` (expands truncated UUIDs, adds other detail) +7. [ ] Update help text +7. [ ] Implement and test new HTTP endpoint in `command/agent/_endpoint.go` +8. [ ] Implement and test new RPC endpoint in `nomad/_endpoint.go` +9. [ ] Implement and test new Client RPC endpoint in `client/_endpoint.go` + * For client endpoints only (e.g. Filesystem) +10. [ ] Implement and test new `api/` package Request and Response structs +11. [ ] Implement and test new `api/` package helper methods +12. [ ] Implement and test new `nomad/structs/` package Request and Response structs + +## Docs + +1. [ ] Changelog +2. [ ] API docs https://www.nomadproject.io/api/index.html +3. [ ] CLI docs https://www.nomadproject.io/docs/commands/index.html +4. [ ] Consider if it needs a guide https://www.nomadproject.io/guides/index.html diff --git a/docs/checklist-jobspec.md b/docs/checklist-jobspec.md new file mode 100644 index 000000000..aad500bf0 --- /dev/null +++ b/docs/checklist-jobspec.md @@ -0,0 +1,25 @@ +# New `jobspec` Entry Checklist + +## Code + +1. [ ] Parse in `jobspec/parse.go` +2. [ ] Test in `jobspec/parse_test.go` (preferably with a + `jobspec/text-fixtures/.hcl` test file) +3. [ ] Add structs/fields to `api/` package + * structs usually have Canonicalize, Copy, and Merge methods + * New fields should be added to existing Canonicalize, Copy, and Merge + methods + * Test the struct/field via all methods mentioned above +4. [ ] Add structs/fields to `nomad/structs` package + * Validation happens in this package and must be implemented + * Implement other methods and tests from `api/` package +5. [ ] Add conversion between `api/` and `nomad/structs` in `command/agent/job_endpoint.go` +6. [ ] Test conversion + +## Docs + +1. [ ] Changelog +2. [ ] Jobspec entry https://www.nomadproject.io/docs/job-specification/index.html +3. [ ] Job JSON API entry https://www.nomadproject.io/api/json-jobs.html +4. [ ] Sample Response output in API https://www.nomadproject.io/api/jobs.html +5. [ ] Consider if it needs a guide https://www.nomadproject.io/guides/index.html From 93adf2e286f6a1bdb9e16f6b1a2d302544a24657 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 6 Jun 2019 14:05:13 -0700 Subject: [PATCH 2/7] docs: CLI -> client The UI is a client too! --- docs/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/README.md b/docs/README.md index cb462a23a..8ce6580fa 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,14 +15,14 @@ The code for Nomad's major components is organized as: * `command/` contains Nomad's CLI code. * `nomad/` contains Nomad's server agent code. -The high level control flow for many Nomad commands are: +The high level control flow for many Nomad actions (via the CLI or UI) are: ``` -# Read commands: -CLI command -> HTTP API -> RPC -> StateStore +# Read actions: +Client -> HTTP API -> RPC -> StateStore -# Commands that change state: -CLI command -> HTTP API -> RPC -> Raft -> FSM -> StateStore +# Actions that change state: +Client -> HTTP API -> RPC -> Raft -> FSM -> StateStore ``` ## Checklists From 32f844e470d8373d802a4e6485e1f84bb54b76a0 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Thu, 6 Jun 2019 14:05:32 -0700 Subject: [PATCH 3/7] docs: remove subbullet points They render ambiguously --- docs/checklist-command.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/checklist-command.md b/docs/checklist-command.md index 9cf7de829..9a5448835 100644 --- a/docs/checklist-command.md +++ b/docs/checklist-command.md @@ -18,8 +18,8 @@ CLI (command/) -> API Client (api/) -> HTTP API (command/agent) -> RPC (nomad/) 7. [ ] Update help text 7. [ ] Implement and test new HTTP endpoint in `command/agent/_endpoint.go` 8. [ ] Implement and test new RPC endpoint in `nomad/_endpoint.go` -9. [ ] Implement and test new Client RPC endpoint in `client/_endpoint.go` - * For client endpoints only (e.g. Filesystem) +9. [ ] Implement and test new Client RPC endpoint in + `client/_endpoint.go` (For client endpoints like Filesystem only) 10. [ ] Implement and test new `api/` package Request and Response structs 11. [ ] Implement and test new `api/` package helper methods 12. [ ] Implement and test new `nomad/structs/` package Request and Response structs From 9365253f65654c62f382f54845c13214c5cec0cc Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Fri, 7 Jun 2019 08:33:24 -0700 Subject: [PATCH 4/7] docs: require Go 1.11+ --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f2207b532..ef0eda515 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,7 @@ Contributing to Nomad -------------------- If you wish to contribute to Nomad, you will need [Go](https://www.golang.org) -installed on your machine (version 1.10.2+ is *required*). +installed on your machine (version 1.11+ is *required*). See the [`docs`](docs/) directory for more developer documentation. From 96fa01ac3c210509d29bbb71e7cda8818527228a Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Fri, 7 Jun 2019 08:33:35 -0700 Subject: [PATCH 5/7] docs: mention ui & website as "core components" --- docs/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/README.md b/docs/README.md index 8ce6580fa..555861b85 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,6 +14,8 @@ The code for Nomad's major components is organized as: * `client/` contains Nomad's client agent code. * `command/` contains Nomad's CLI code. * `nomad/` contains Nomad's server agent code. +* `ui/` contains Nomad's UI code. +* `website/` contains Nomad's website and documentation. The high level control flow for many Nomad actions (via the CLI or UI) are: From 6c116add0fedcee7cb21ef1cc6598a30dbd54094 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Fri, 7 Jun 2019 15:41:13 -0700 Subject: [PATCH 6/7] docs: move dev docs to contributing --- README.md | 2 +- {docs => contributing}/README.md | 0 {docs => contributing}/checklist-command.md | 0 {docs => contributing}/checklist-jobspec.md | 0 4 files changed, 1 insertion(+), 1 deletion(-) rename {docs => contributing}/README.md (100%) rename {docs => contributing}/checklist-command.md (100%) rename {docs => contributing}/checklist-jobspec.md (100%) diff --git a/README.md b/README.md index ef0eda515..7e2c18b54 100644 --- a/README.md +++ b/README.md @@ -143,7 +143,7 @@ Contributing to Nomad If you wish to contribute to Nomad, you will need [Go](https://www.golang.org) installed on your machine (version 1.11+ is *required*). -See the [`docs`](docs/) directory for more developer documentation. +See the [`contributing`](contributing/) directory for more developer documentation. **Developing with Vagrant** There is an included Vagrantfile that can help bootstrap the process. The diff --git a/docs/README.md b/contributing/README.md similarity index 100% rename from docs/README.md rename to contributing/README.md diff --git a/docs/checklist-command.md b/contributing/checklist-command.md similarity index 100% rename from docs/checklist-command.md rename to contributing/checklist-command.md diff --git a/docs/checklist-jobspec.md b/contributing/checklist-jobspec.md similarity index 100% rename from docs/checklist-jobspec.md rename to contributing/checklist-jobspec.md From b2d62096d005a8fcfd9ff8ad468ceef8e0db09d3 Mon Sep 17 00:00:00 2001 From: Michael Schurter Date: Wed, 12 Jun 2019 15:35:48 -0700 Subject: [PATCH 7/7] docs: checklists don't support numbered lists The order didn't really make sense anyway. --- contributing/checklist-command.md | 32 +++++++++++++++---------------- contributing/checklist-jobspec.md | 22 ++++++++++----------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/contributing/checklist-command.md b/contributing/checklist-command.md index 9a5448835..6473c4894 100644 --- a/contributing/checklist-command.md +++ b/contributing/checklist-command.md @@ -10,23 +10,23 @@ CLI (command/) -> API Client (api/) -> HTTP API (command/agent) -> RPC (nomad/) ## Code -1. [ ] New file in `command/` or in an existing file if a subcommand -2. [ ] Test new command in `command/` package -3. [ ] Implement autocomplete -4. [ ] Implement `-json` (returns raw API response) -5. [ ] Implement `-verbose` (expands truncated UUIDs, adds other detail) -7. [ ] Update help text -7. [ ] Implement and test new HTTP endpoint in `command/agent/_endpoint.go` -8. [ ] Implement and test new RPC endpoint in `nomad/_endpoint.go` -9. [ ] Implement and test new Client RPC endpoint in +* [ ] New file in `command/` or in an existing file if a subcommand +* [ ] Test new command in `command/` package +* [ ] Implement autocomplete +* [ ] Implement `-json` (returns raw API response) +* [ ] Implement `-verbose` (expands truncated UUIDs, adds other detail) +* [ ] Update help text +* [ ] Implement and test new HTTP endpoint in `command/agent/_endpoint.go` +* [ ] Implement and test new RPC endpoint in `nomad/_endpoint.go` +* [ ] Implement and test new Client RPC endpoint in `client/_endpoint.go` (For client endpoints like Filesystem only) -10. [ ] Implement and test new `api/` package Request and Response structs -11. [ ] Implement and test new `api/` package helper methods -12. [ ] Implement and test new `nomad/structs/` package Request and Response structs +* [ ] Implement and test new `api/` package Request and Response structs +* [ ] Implement and test new `api/` package helper methods +* [ ] Implement and test new `nomad/structs/` package Request and Response structs ## Docs -1. [ ] Changelog -2. [ ] API docs https://www.nomadproject.io/api/index.html -3. [ ] CLI docs https://www.nomadproject.io/docs/commands/index.html -4. [ ] Consider if it needs a guide https://www.nomadproject.io/guides/index.html +* [ ] Changelog +* [ ] API docs https://www.nomadproject.io/api/index.html +* [ ] CLI docs https://www.nomadproject.io/docs/commands/index.html +* [ ] Consider if it needs a guide https://www.nomadproject.io/guides/index.html diff --git a/contributing/checklist-jobspec.md b/contributing/checklist-jobspec.md index aad500bf0..195cb0cb6 100644 --- a/contributing/checklist-jobspec.md +++ b/contributing/checklist-jobspec.md @@ -2,24 +2,24 @@ ## Code -1. [ ] Parse in `jobspec/parse.go` -2. [ ] Test in `jobspec/parse_test.go` (preferably with a +* [ ] Parse in `jobspec/parse.go` +* [ ] Test in `jobspec/parse_test.go` (preferably with a `jobspec/text-fixtures/.hcl` test file) -3. [ ] Add structs/fields to `api/` package +* [ ] Add structs/fields to `api/` package * structs usually have Canonicalize, Copy, and Merge methods * New fields should be added to existing Canonicalize, Copy, and Merge methods * Test the struct/field via all methods mentioned above -4. [ ] Add structs/fields to `nomad/structs` package +* [ ] Add structs/fields to `nomad/structs` package * Validation happens in this package and must be implemented * Implement other methods and tests from `api/` package -5. [ ] Add conversion between `api/` and `nomad/structs` in `command/agent/job_endpoint.go` -6. [ ] Test conversion +* [ ] Add conversion between `api/` and `nomad/structs` in `command/agent/job_endpoint.go` +* [ ] Test conversion ## Docs -1. [ ] Changelog -2. [ ] Jobspec entry https://www.nomadproject.io/docs/job-specification/index.html -3. [ ] Job JSON API entry https://www.nomadproject.io/api/json-jobs.html -4. [ ] Sample Response output in API https://www.nomadproject.io/api/jobs.html -5. [ ] Consider if it needs a guide https://www.nomadproject.io/guides/index.html +* [ ] Changelog +* [ ] Jobspec entry https://www.nomadproject.io/docs/job-specification/index.html +* [ ] Job JSON API entry https://www.nomadproject.io/api/json-jobs.html +* [ ] Sample Response output in API https://www.nomadproject.io/api/jobs.html +* [ ] Consider if it needs a guide https://www.nomadproject.io/guides/index.html