3479e2231f
When a Nomad client that is running an allocation with `max_client_disconnect` set misses a heartbeat the Nomad server will update its status to `disconnected`. Upon reconnecting, the client will make three main RPC calls: - `Node.UpdateStatus` is used to set the client status to `ready`. - `Node.UpdateAlloc` is used to update the client-side information about allocations, such as their `ClientStatus`, task states etc. - `Node.Register` is used to upsert the entire node information, including its status. These calls are made concurrently and are also running in parallel with the scheduler. Depending on the order they run the scheduler may end up with incomplete data when reconciling allocations. For example, a client disconnects and its replacement allocation cannot be placed anywhere else, so there's a pending eval waiting for resources. When this client comes back the order of events may be: 1. Client calls `Node.UpdateStatus` and is now `ready`. 2. Scheduler reconciles allocations and places the replacement alloc to the client. The client is now assigned two allocations: the original alloc that is still `unknown` and the replacement that is `pending`. 3. Client calls `Node.UpdateAlloc` and updates the original alloc to `running`. 4. Scheduler notices too many allocs and stops the replacement. This creates unnecessary placements or, in a different order of events, may leave the job without any allocations running until the whole state is updated and reconciled. To avoid problems like this clients must update _all_ of its relevant information before they can be considered `ready` and available for scheduling. To achieve this goal the RPC endpoints mentioned above have been modified to enforce strict steps for nodes reconnecting: - `Node.Register` does not set the client status anymore. - `Node.UpdateStatus` sets the reconnecting client to the `initializing` status until it successfully calls `Node.UpdateAlloc`. These changes are done server-side to avoid the need of additional coordination between clients and servers. Clients are kept oblivious of these changes and will keep making these calls as they normally would. The verification of whether allocations have been updates is done by storing and comparing the Raft index of the last time the client missed a heartbeat and the last time it updated its allocations. |
||
|---|---|---|
| .changelog | ||
| .circleci | ||
| .github | ||
| .release | ||
| .semgrep | ||
| .tours | ||
| acl | ||
| api | ||
| ci | ||
| client | ||
| command | ||
| contributing | ||
| demo | ||
| dev | ||
| drivers | ||
| e2e | ||
| helper | ||
| integrations | ||
| internal/testing/apitests | ||
| jobspec | ||
| jobspec2 | ||
| lib | ||
| nomad | ||
| plugins | ||
| scheduler | ||
| scripts | ||
| terraform | ||
| testutil | ||
| tools | ||
| ui | ||
| version | ||
| website | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .go-version | ||
| .golangci.yml | ||
| .semgrepignore | ||
| CHANGELOG.md | ||
| CODEOWNERS | ||
| GNUmakefile | ||
| LICENSE | ||
| README.md | ||
| Vagrantfile | ||
| build_linux_arm.go | ||
| go.mod | ||
| go.sum | ||
| main.go | ||
| main_test.go | ||
README.md
Nomad
Nomad is a simple and flexible workload orchestrator to deploy and manage containers (docker, podman), non-containerized applications (executable, Java), and virtual machines (qemu) across on-prem and clouds at scale.
Nomad is supported on Linux, Windows, and macOS. A commercial version of Nomad, Nomad Enterprise, is also available.
- Website: https://nomadproject.io
- Tutorials: HashiCorp Learn
- Forum: Discuss
Nomad provides several key features:
-
Deploy Containers and Legacy Applications: Nomad’s flexibility as an orchestrator enables an organization to run containers, legacy, and batch applications together on the same infrastructure. Nomad brings core orchestration benefits to legacy applications without needing to containerize via pluggable task drivers.
-
Simple & Reliable: Nomad runs as a single binary and is entirely self contained - combining resource management and scheduling into a single system. Nomad does not require any external services for storage or coordination. Nomad automatically handles application, node, and driver failures. Nomad is distributed and resilient, using leader election and state replication to provide high availability in the event of failures.
-
Device Plugins & GPU Support: Nomad offers built-in support for GPU workloads such as machine learning (ML) and artificial intelligence (AI). Nomad uses device plugins to automatically detect and utilize resources from hardware devices such as GPU, FPGAs, and TPUs.
-
Federation for Multi-Region, Multi-Cloud: Nomad was designed to support infrastructure at a global scale. Nomad supports federation out-of-the-box and can deploy applications across multiple regions and clouds.
-
Proven Scalability: Nomad is optimistically concurrent, which increases throughput and reduces latency for workloads. Nomad has been proven to scale to clusters of 10K+ nodes in real-world production environments.
-
HashiCorp Ecosystem: Nomad integrates seamlessly with Terraform, Consul, Vault for provisioning, service discovery, and secrets management.
Quick Start
Testing
See Learn: Getting Started for instructions on setting up a local Nomad cluster for non-production use.
Optionally, find Terraform manifests for bringing up a development Nomad cluster on a public cloud in the terraform directory.
Production
See Learn: Nomad Reference Architecture for recommended practices and a reference architecture for production deployments.
Documentation
Full, comprehensive documentation is available on the Nomad website: https://www.nomadproject.io/docs
Guides are available on HashiCorp Learn.
Contributing
See the contributing directory for more developer documentation.