df587d8263
This PR updates the changelog, adds notes the 1.3 upgrade guide, and updates the connect integration docs with documentation about the new requirement on Consul ACL policies of Consul agent default anonymous ACL tokens.
355 lines
9.7 KiB
Plaintext
355 lines
9.7 KiB
Plaintext
---
|
|
layout: docs
|
|
page_title: Consul Service Mesh
|
|
description: >-
|
|
Learn how to use Nomad with Consul service mesh to enable secure service to service
|
|
communication
|
|
---
|
|
|
|
# Consul Service Mesh
|
|
|
|
~> **Note:** Nomad's service mesh integration requires Linux network namespaces.
|
|
Consul service mesh will not run on Windows or macOS.
|
|
|
|
[Consul service mesh](https://www.consul.io/docs/connect) provides
|
|
service-to-service connection authorization and encryption using mutual
|
|
Transport Layer Security (TLS). Applications can use sidecar proxies in a
|
|
service mesh configuration to automatically establish TLS connections for
|
|
inbound and outbound connections without being aware of the service mesh at all.
|
|
|
|
# Nomad with Consul Service Mesh Integration
|
|
|
|
Nomad integrates with Consul to provide secure service-to-service communication
|
|
between Nomad jobs and task groups. To support Consul service mesh, Nomad
|
|
adds a new networking mode for jobs that enables tasks in the same task group to
|
|
share their networking stack. With a few changes to the job specification, job
|
|
authors can opt into service mesh integration. When service mesh is enabled, Nomad will
|
|
launch a proxy alongside the application in the job file. The proxy (Envoy)
|
|
provides secure communication with other applications in the cluster.
|
|
|
|
Nomad job specification authors can use Nomad's Consul service mesh integration to
|
|
implement [service segmentation](https://www.consul.io/use-cases/multi-platform-service-mesh) in a
|
|
microservice architecture running in public clouds without having to directly
|
|
manage TLS certificates. This is transparent to job specification authors as
|
|
security features in service mesh continue to work even as the application scales up
|
|
or down or gets rescheduled by Nomad.
|
|
|
|
For using the Consul service mesh integration with Consul ACLs enabled, see the
|
|
[Secure Nomad Jobs with Consul Service Mesh](https://learn.hashicorp.com/tutorials/nomad/consul-service-mesh)
|
|
guide.
|
|
|
|
# Nomad Consul Service Mesh Example
|
|
|
|
The following section walks through an example to enable secure communication
|
|
between a web dashboard and a backend counting service. The web dashboard and
|
|
the counting service are managed by Nomad. Nomad additionally configures Envoy
|
|
proxies to run along side these applications. The dashboard is configured to
|
|
connect to the counting service via localhost on port 9001. The proxy is managed
|
|
by Nomad, and handles mTLS communication to the counting service.
|
|
|
|
## Prerequisites
|
|
|
|
### Consul
|
|
|
|
The Consul service mesh integration with Nomad requires [Consul 1.6 or
|
|
later.](https://releases.hashicorp.com/consul/1.6.0/) The Consul agent can be
|
|
run in dev mode with the following command:
|
|
|
|
~> **Note:** Nomad's Consul service mesh integration requires Consul in your `$PATH`
|
|
|
|
```shell-session
|
|
$ consul agent -dev
|
|
```
|
|
|
|
To use service mesh on a non-dev Consul agent, you will minimally need to enable the
|
|
GRPC port and set `connect` to enabled by adding some additional information to
|
|
your Consul client configurations, depending on format.
|
|
|
|
For HCL configurations:
|
|
|
|
```hcl
|
|
# ...
|
|
|
|
ports {
|
|
grpc = 8502
|
|
}
|
|
|
|
connect {
|
|
enabled = true
|
|
}
|
|
```
|
|
|
|
For JSON configurations:
|
|
|
|
```javascript
|
|
{
|
|
// ...
|
|
"ports": {
|
|
"grpc": 8502
|
|
},
|
|
"connect": {
|
|
"enabled": true
|
|
}
|
|
}
|
|
```
|
|
|
|
#### Consul ACLs
|
|
|
|
~> **Note:** Starting in Nomad v1.3.0, Consul Service Identity ACL tokens automatically
|
|
generated by Nomad on behalf of Connect enabled services are now created in [`Local`]
|
|
rather than Global scope, and are no longer replicated globally.
|
|
|
|
To facilitate cross-Consul datacenter requests of Connect services registered by
|
|
Nomad, Consul agents will need to be configured with [default anonymous][anon_token]
|
|
ACL tokens with ACL policies of sufficient permissions to read service and node
|
|
metadata pertaining to those requests. This mechanism is described in Consul [#7414][consul_acl].
|
|
A typical Consul agent anonymous token may contain an ACL policy such as:
|
|
|
|
```hcl
|
|
service_prefix "" { policy = "read" }
|
|
node_prefix "" { policy = "read" }
|
|
```
|
|
|
|
### Nomad
|
|
|
|
Nomad must schedule onto a routable interface in order for the proxies to
|
|
connect to each other. The following steps show how to start a Nomad dev agent
|
|
configured for Consul service mesh.
|
|
|
|
```shell-session
|
|
$ sudo nomad agent -dev-connect
|
|
```
|
|
|
|
### CNI Plugins
|
|
|
|
Nomad uses CNI plugins to configure the network namespace used to secure the
|
|
Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces
|
|
must have CNI plugins installed.
|
|
|
|
The following commands install CNI plugins:
|
|
|
|
```shell-session
|
|
curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)"-v1.0.0.tgz
|
|
sudo mkdir -p /opt/cni/bin
|
|
sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz
|
|
```
|
|
|
|
Ensure the your Linux operating system distribution has been configured to allow
|
|
container traffic through the bridge network to be routed via iptables. These
|
|
tunables can be set as follows:
|
|
|
|
```shell-session
|
|
echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-arptables
|
|
echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-ip6tables
|
|
echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables
|
|
```
|
|
|
|
To preserve these settings on startup of a client node, add a file including the
|
|
following to `/etc/sysctl.d/` or remove the file your Linux distribution puts in
|
|
that directory.
|
|
|
|
```
|
|
net.bridge.bridge-nf-call-arptables = 1
|
|
net.bridge.bridge-nf-call-ip6tables = 1
|
|
net.bridge.bridge-nf-call-iptables = 1
|
|
```
|
|
|
|
## Run the Service Mesh-enabled Services
|
|
|
|
Once Nomad and Consul are running, submit the following service mesh-enabled services
|
|
to Nomad by copying the HCL into a file named `servicemesh.nomad` and running:
|
|
`nomad job run servicemesh.nomad`
|
|
|
|
```hcl
|
|
job "countdash" {
|
|
datacenters = ["dc1"]
|
|
|
|
group "api" {
|
|
network {
|
|
mode = "bridge"
|
|
}
|
|
|
|
service {
|
|
name = "count-api"
|
|
port = "9001"
|
|
|
|
connect {
|
|
sidecar_service {}
|
|
}
|
|
}
|
|
|
|
task "web" {
|
|
driver = "docker"
|
|
|
|
config {
|
|
image = "hashicorpnomad/counter-api:v3"
|
|
}
|
|
}
|
|
}
|
|
|
|
group "dashboard" {
|
|
network {
|
|
mode = "bridge"
|
|
|
|
port "http" {
|
|
static = 9002
|
|
to = 9002
|
|
}
|
|
}
|
|
|
|
service {
|
|
name = "count-dashboard"
|
|
port = "http"
|
|
|
|
connect {
|
|
sidecar_service {
|
|
proxy {
|
|
upstreams {
|
|
destination_name = "count-api"
|
|
local_bind_port = 8080
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
task "dashboard" {
|
|
driver = "docker"
|
|
|
|
env {
|
|
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
|
|
}
|
|
|
|
config {
|
|
image = "hashicorpnomad/counter-dashboard:v3"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
The job contains two task groups: an API service and a web frontend.
|
|
|
|
### API Service
|
|
|
|
The API service is defined as a task group with a bridge network:
|
|
|
|
```hcl
|
|
group "api" {
|
|
network {
|
|
mode = "bridge"
|
|
}
|
|
|
|
# ...
|
|
}
|
|
```
|
|
|
|
Since the API service is only accessible via Consul service mesh, it does not define
|
|
any ports in its network. The service stanza enables service mesh.
|
|
|
|
```hcl
|
|
group "api" {
|
|
|
|
# ...
|
|
|
|
service {
|
|
name = "count-api"
|
|
port = "9001"
|
|
|
|
connect {
|
|
sidecar_service {}
|
|
}
|
|
}
|
|
|
|
# ...
|
|
|
|
}
|
|
```
|
|
|
|
The `port` in the service stanza is the port the API service listens on. The
|
|
Envoy proxy will automatically route traffic to that port inside the network
|
|
namespace. Note that currently this cannot be a named port; it must be a
|
|
hard-coded port value. See [GH-9907].
|
|
|
|
### Web Frontend
|
|
|
|
The web frontend is defined as a task group with a bridge network and a static
|
|
forwarded port:
|
|
|
|
```hcl
|
|
group "dashboard" {
|
|
network {
|
|
mode = "bridge"
|
|
|
|
port "http" {
|
|
static = 9002
|
|
to = 9002
|
|
}
|
|
}
|
|
|
|
# ...
|
|
|
|
}
|
|
```
|
|
|
|
The `static = 9002` parameter requests the Nomad scheduler reserve port 9002 on
|
|
a host network interface. The `to = 9002` parameter forwards that host port to
|
|
port 9002 inside the network namespace.
|
|
|
|
This allows you to connect to the web frontend in a browser by visiting
|
|
`http://<host_ip>:9002` as show below:
|
|
|
|
[![Count Dashboard][count-dashboard]][count-dashboard]
|
|
|
|
The web frontend connects to the API service via Consul service mesh.
|
|
|
|
```hcl
|
|
service {
|
|
name = "count-dashboard"
|
|
port = "http"
|
|
|
|
connect {
|
|
sidecar_service {
|
|
proxy {
|
|
upstreams {
|
|
destination_name = "count-api"
|
|
local_bind_port = 8080
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
The `upstreams` stanza defines the remote service to access (`count-api`) and
|
|
what port to expose that service on inside the network namespace (`8080`).
|
|
|
|
The web frontend is configured to communicate with the API service with an
|
|
environment variable:
|
|
|
|
```hcl
|
|
env {
|
|
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
|
|
}
|
|
```
|
|
|
|
The web frontend is configured via the `$COUNTING_SERVICE_URL`, so you must
|
|
interpolate the upstream's address into that environment variable. Note that
|
|
dashes (`-`) are converted to underscores (`_`) in environment variables so
|
|
`count-api` becomes `count_api`.
|
|
|
|
## Limitations
|
|
|
|
- The minimum Consul version to use Connect with Nomad is Consul v1.8.0.
|
|
- The `consul` binary must be present in Nomad's `$PATH` to run the Envoy
|
|
proxy sidecar on client nodes.
|
|
- Consul service mesh using network namespaces is only supported on Linux.
|
|
- Prior to Consul 1.9, the Envoy sidecar proxy will drop and stop accepting
|
|
connections while the Nomad agent is restarting.
|
|
|
|
[count-dashboard]: /img/count-dashboard.png
|
|
[consul_acl]: https://github.com/hashicorp/consul/issues/7414
|
|
[gh-9907]: https://github.com/hashicorp/nomad/issues/9907
|
|
[`Local`]: https://www.consul.io/docs/security/acl/acl-tokens#token-attributes
|
|
[anon_token]: https://www.consul.io/docs/security/acl/acl-tokens#special-purpose-tokens
|