219 lines
8.8 KiB
Plaintext
219 lines
8.8 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: CNI
|
|
description: |-
|
|
Learn about Nomad's runtime environment variables, interpolation, caveats,
|
|
and more.
|
|
---
|
|
|
|
# Container Network Interface (CNI)
|
|
|
|
Nomad has built-in support for scheduling compute resources such as
|
|
CPU, memory, and networking. Nomad's network plugin support extends
|
|
this to allow scheduling tasks with purpose-created or specialty network
|
|
configurations. Network plugins are third-party plugins that conform to the
|
|
[Container Network Interface (CNI)][cni_spec] specification.
|
|
|
|
Network plugins need to be installed and configured on each client. The [Nomad
|
|
installation instructions][nomad_install] recommend installing the [CNI
|
|
reference plugins][cni_ref] because certain Nomad networking features, like
|
|
`bridge` network mode and Consul service mesh, leverage them to provide an
|
|
operating-system agnostic interface to configure workload networking.
|
|
|
|
Custom networking in Nomad is accomplished with a combination of CNI plugin
|
|
binaries and CNI configuration files.
|
|
|
|
## CNI plugins
|
|
|
|
Spec-compliant plugins should work with Nomad, however, it's possible a plugin
|
|
vendor has implemented their plugin to make non-standard API calls, or it is
|
|
otherwise non-compliant with the CNI specification. In those situations the
|
|
plugin may not function correctly in a Nomad environment. You should verify
|
|
plugin compatibility with Nomad before deploying in production.
|
|
|
|
CNI plugins are installed and configured on a per-client basis. Nomad consults
|
|
the path given in the client's [`cni_path`][] to find CNI plugin executables.
|
|
|
|
## CNI configuration files
|
|
|
|
The CNI specification defines a network configuration format for administrators.
|
|
It contains directives for both the orchestrator and the plugins to consume.
|
|
At plugin execution time, this configuration format is interpreted by Nomad
|
|
and transformed into arguments for the plugins.
|
|
|
|
Nomad reads the following extensions from the [`cni_config_dir`][]—
|
|
`/opt/cni/config` by default:
|
|
|
|
* `.conflist` files are loaded as [network
|
|
configurations][cni_spec_net_config] that contain a list of plugin
|
|
configurations.
|
|
|
|
* `.conf` and `.json` files are loaded as individual [plugin
|
|
configurations][cni_spec_plugin_config] for a specific network.
|
|
|
|
## Using CNI networks with Nomad jobs
|
|
|
|
To specify that a job should use a CNI network, set the task group's
|
|
network [`mode`][] attribute to the value `cni/<your_cni_config_name>`.
|
|
Nomad will then schedule the workload on client nodes that have fingerprinted a
|
|
CNI configuration with the given name. For example, to use the configuration
|
|
named `mynet`, you should set the task group's network mode to `cni/mynet`.
|
|
Nodes that have a network configuration defining a network named `mynet` in
|
|
their [`cni_config_dir`][] will be eligible to run the workload.
|
|
|
|
## Nomad's `bridge` configuration
|
|
|
|
Nomad itself uses CNI plugins and configuration as the underlying implementation
|
|
for the `bridge` network mode, using the [loopback][], [bridge][], [firewall][],
|
|
and [portmap][] CNI plugins configured together to create Nomad's bridge
|
|
network.
|
|
|
|
The following is the configuration template Nomad uses when setting up these
|
|
networks with the template placeholders replaced with the default configuration
|
|
values for [`bridge_network_name`][], [`bridge_network_subnet`][], and an
|
|
internal constant that provides the value for `iptablesAdminChainName`. This is
|
|
a convenient jumping off point to discuss a worked example.
|
|
|
|
```json
|
|
{
|
|
"cniVersion": "0.4.0",
|
|
"name": "nomad",
|
|
"plugins": [
|
|
{
|
|
"type": "loopback"
|
|
},
|
|
{
|
|
"type": "bridge",
|
|
"bridge": "nomad",
|
|
"ipMasq": true,
|
|
"isGateway": true,
|
|
"forceAddress": true,
|
|
"hairpinMode": false,
|
|
"ipam": {
|
|
"type": "host-local",
|
|
"ranges": [
|
|
[
|
|
{
|
|
"subnet": "172.26.64.0/20"
|
|
}
|
|
]
|
|
],
|
|
"routes": [
|
|
{ "dst": "0.0.0.0/0" }
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"type": "firewall",
|
|
"backend": "iptables",
|
|
"iptablesAdminChainName": "NOMAD-ADMIN"
|
|
},
|
|
{
|
|
"type": "portmap",
|
|
"capabilities": {"portMappings": true},
|
|
"snat": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
<!-- Source: https://www.figma.com/file/Ne2qaPUlBTmTYer9biCfK9/Networking?node-id=0%3A1&t=BepgOoQ0kb76GwIr-1 -->
|
|
[![Visual example of Nomad bridge network](/img/nomad-bridge-network.png)](/img/nomad-bridge-network.png)
|
|
|
|
For a more thorough understanding of this configuration, consider each CNI
|
|
plugin's configuration in turn.
|
|
|
|
### loopback
|
|
|
|
The `loopback` plugin sets the default local interface, `lo0`, created inside
|
|
the bridge network's network namespace to **UP**. This allows workload running
|
|
inside the namespace to bind to a namespace-specific loopback interface.
|
|
|
|
### bridge
|
|
|
|
The `bridge` plugin creates a bridge (virtual switch) named `nomad` that resides
|
|
in the host network namespace. Because this bridge is intended to provide
|
|
network connectivity to allocations, it's configured to be a gateway by setting
|
|
`isGateway` to `true`. This tells the plugin to assign an IP address to the
|
|
bridge interface
|
|
|
|
The bridge plugin connects allocations (on the same host) into a bridge (virtual
|
|
switch) that resides in the host network namespace. By default Nomad creates a
|
|
single bridge for each client. Since Nomad's bridge network is designed to
|
|
provide network connectivity to the allocations, it configures the bridge
|
|
interface to be a gateway for outgoing traffic by providing it with an address
|
|
using an `ipam` configuration. The default configuration creates a host-local
|
|
address for the host side of the bridge in the `172.26.64.0/20` subnet at
|
|
`172.26.64.1`. When associating allocations to the bridge, it creates addresses
|
|
for the allocations from that same subnet using the host-local plugin. The
|
|
configuration also specifies a default route for the allocations of the
|
|
host-side bridge address.
|
|
|
|
### firewall
|
|
|
|
The firewall plugin creates firewall rules to allow traffic to/from the
|
|
allocation's IP address via the host network. Nomad uses the iptables backend
|
|
for the firewall plugin. This configuration creates two new iptables chains,
|
|
`CNI-FORWARD` and `NOMAD-ADMIN`, in the filter table and add rules that allow
|
|
the given interface to send/receive traffic.
|
|
|
|
The firewall creates an admin chain using the name provided in the
|
|
`iptablesAdminChainName` attribute. For this case, it's called `NOMAD-ADMIN`.
|
|
The admin chain is a user-controlled chain for custom rules that run before
|
|
rules managed by the firewall plugin. The firewall plugin does not add, delete,
|
|
or modify rules in the admin chain.
|
|
|
|
A new chain, `CNI-FORWARD` is added to the `FORWARD` chain. `CNI-FORWARD` is
|
|
the chain where rules will be added when allocations are created and from where
|
|
rules will be removed when those allocations stop. The `CNI-FORWARD` chain
|
|
first sends all traffic to `NOMAD-ADMIN` chain.
|
|
|
|
You can use the following command to list the iptables rules present in each
|
|
chain.
|
|
|
|
```shell-session
|
|
$ sudo iptables -L
|
|
```
|
|
|
|
### portmap
|
|
|
|
Nomad needs to be able to map specific ports from the host to tasks running in
|
|
the allocation namespace. The `portmap` plugin forwards traffic from one or more
|
|
ports on the host to the allocation using network address translation (NAT)
|
|
rules.
|
|
|
|
The plugin sets up two sequences of chains and rules:
|
|
|
|
- One “primary” `DNAT` (destination NAT) sequence to rewrite the destination.
|
|
- One `SNAT` (source NAT) sequence that will masquerade traffic as needed.
|
|
|
|
You can use the following command to list the iptables rules in the NAT table.
|
|
|
|
```shell-session
|
|
$ sudo iptables -t nat -L
|
|
```
|
|
|
|
## Create your own
|
|
|
|
You can use this template as a basis for your own CNI-based bridge network
|
|
configuration in cases where you need access to an unsupported option in the
|
|
default configuration, like hairpin mode. When making your own bridge network
|
|
based on this template, ensure that you change the `iptablesAdminChainName` to
|
|
a unique value for your configuration.
|
|
|
|
[3rd_party_cni]: https://www.cni.dev/docs/#3rd-party-plugins
|
|
[`bridge_network_name`]: /nomad/docs/configuration/client#bridge_network_name
|
|
[`bridge_network_subnet`]: /nomad/docs/configuration/client#bridge_network_subnet
|
|
[`cni_config_dir`]: /nomad/docs/configuration/client#cni_config_dir
|
|
[`cni_path`]: /nomad/docs/configuration/client#cni_path
|
|
[`mode`]: /nomad/docs/job-specification/network#mode
|
|
[bridge]: https://www.cni.dev/plugins/current/main/bridge/
|
|
[cni_ref]: https://github.com/containernetworking/plugins
|
|
[cni_spec]: https://www.cni.dev/docs/spec/
|
|
[cni_spec_net_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#configuration-format
|
|
[cni_spec_plugin_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#plugin-configuration-objects
|
|
[firewall]: https://www.cni.dev/plugins/current/meta/firewall/
|
|
[loopback]: https://github.com/containernetworking/plugins#main-interface-creating
|
|
[nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps
|
|
[portmap]: https://www.cni.dev/plugins/current/meta/portmap/
|