diff --git a/website/_redirects b/website/_redirects index 2f1457ea0..023d34b9d 100644 --- a/website/_redirects +++ b/website/_redirects @@ -394,6 +394,8 @@ /docs/platform/k8s/uninstalling /docs/k8s/operations/upgrading 301! /docs/platform/k8s/* /docs/k8s/:splat 301! +/docs/nia/installation/configuration /docs/nia/configuration 301! + /docs/partnerships/index.html /docs/partnerships 301! /docs/enterprise/backups/index.html /docs/enterprise/backups 301! /docs/enterprise/upgrades/index.html /docs/enterprise/upgrades 301! diff --git a/website/data/docs-navigation.js b/website/data/docs-navigation.js index ecb4ff27a..68d8f96cc 100644 --- a/website/data/docs-navigation.js +++ b/website/data/docs-navigation.js @@ -189,10 +189,12 @@ export default [ { category: 'installation', name: 'Get Started', - content: ['install', 'requirements', 'configuration', 'run'], + content: ['install', 'requirements', 'configure', 'run'], }, 'architecture', + 'api', 'cli', + 'configuration', 'tasks', 'network-drivers', ], diff --git a/website/pages/docs/nia/api.mdx b/website/pages/docs/nia/api.mdx new file mode 100644 index 000000000..12f04e56c --- /dev/null +++ b/website/pages/docs/nia/api.mdx @@ -0,0 +1,219 @@ +--- +layout: docs +page_title: Consul-Terraform-Sync API +sidebar_title: API +description: >- + How to use the Consul-Terraform-Sync API +--- + +# Consul-Terraform-Sync API + +When running in [daemon mode](/docs/nia/cli#daemon-mode), Consul-Terraform-Sync serves an HTTP API interface. + +### Port + +The API is served at the default port `8558` or a different port if set with [`port` configuration](/docs/nia/configuration#port) + +### Version Prefix + +All API routes are prefixed with `/v1/`. This documentation is for v1 of the API, which is the only version currently. + +Example: `localhost:8558/v1/status` + +### Error + +Successful API requests will receive a 2XX success status code. For other unsuccessful status codes, when possible, more details will be provided in a response body. The response will be a JSON map with an "error" key. + +Example: Status 400 Bad Request +```json +{ + "error": "example error message: unsupported status parameter value" +} +``` + +## Status + +The `/status` endpoints share status-related information for tasks. This information is available for understanding the status of individual tasks and across tasks. + +The health status value is determined by aggregating the success or failure of the event of a task detecting changes in Consul services and then updating network infrastructure. Currently, only the 5 most recent events are stored in Consul-Terraform-Sync. For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). + +### Overall Status + +This endpoint currently returns the overall status information for all tasks. + +| Method | Path | Produces | +| ------ | ------------------- | ------------------ | +| `GET` | `/status` | `application/json` | + +#### Request Parameters +Currently no request parameters are offered for the overall status API. + +#### Response Fields + +* `task_summary` - Summary of the count of tasks for each health status. See [Task Status API](/docs/nia/api#task-status) to learn more about how health status is determined. + * `successful` - (int) The number of tasks that have a 'successful' health status + * `errored` - (int) The number of tasks that have a 'errored' health status + * `critical` - (int) The number of tasks that have a 'critical' health status + + +#### Example + +Request: +```shell-session +$ curl localhost:8558/v1/status +``` + +Response: +```json +{ + "task_summary": { + "successful": 28, + "errored": 5, + "critical": 1 + } +} +``` + +### Task Status + +This endpoint returns the individual task status information for a single specified task or for all tasks. + +Task health status value is determined by the success or failure of all stored [event data](/docs/nia/tasks#event) on the process of updating network infrastructure for a task. Currently only the 5 most recent events are stored per task. + - Successful: The most recent stored event is successful. + - Errored: The most recent stored event is not successful but all previous stored events are successful. + - Critical: The most recent stored event is not successful and one or more previous stored events are also not successful. + - Unknown: No event data is stored for the task. + +| Method | Path | Produces | +| ------ | ------------------- | ------------------ | +| `GET` | `/status/tasks/:task` | `application/json` | + +#### Request Parameters + +* `task` - (string) Option to specify the name of the task to return in the response. If not specified, all tasks are returned in the response. +* `include` - (string) Only accepts the value "events". Use to include stored event information in response. +* `status` - (string) Only accepts health status values "successful", "errored", "critical", or "unknown". Use to filter response by tasks that have the specified health status value. Recommend setting this parameter when requesting all tasks i.e. no `task` parameter is set. + +#### Response Fields + +The response is a JSON map of task name to a status information structure with the following fields. + +* `task_name` - (string) Name that task is configured with in Consul-Terraform-Sync. +* `status` - (string) Values are "successful", "errored", "critical", or "unknown". This is determined by the success or failure of all stored events on the network infrastructure update process for the task, as described earlier. +* `services` - (list[string]) List of the services configured for the task. +* `providers` - (list[string]) List of the providers configured for the task. +* `events_url` - (string) Relative URL to retrieve the event data stored for the task. +* `events` - [(list[Event])](/docs/nia/api#event) - List of stored events that inform the task's status. See section below for information on event data. This field is only included in the response upon request by setting the `?include=events` parameter. The relative URL for the request to include events can be retrieved from the `events_url` field. + +##### Event + +Event represents the process of updating network infrastructure of a task. The data is captured in a JSON structure. For more details on the scope of an event, see [Event](/docs/nia/tasks#event). + +* `id` - (string) UUID to uniquely identify the event. +* `success` - (bool) Indication of whether the event was successful or not. +* `start_time` - (time) Time when the event started. +* `end_time` - (time) Time when the event ended. +* `task_name` - (string) Name that task is configured with in Consul-Terraform-Sync. +* `error` Information when the event fails. Null when successful. + * `message` - (string) Error message that is returned on failure. +* `config` + * `services` - (list[string]) List of the services configured for the task. + * `source` - (string) Source configured for the task. + * `providers` - (list[string]) List of the providers configured for the task. + +#### Example: All Task Statuses + +Request: +```shell-session +$ curl localhost:8558/v1/status/tasks +``` + +Response: +```json +{ + "task_a": { + "task_name": "task_a", + "status": "successful", + "providers": [ + "local" + ], + "services": [ + "api" + ], + "events_url": "/v1/status/tasks/task_a?include=events" + }, + "task_b": { + "task_name": "task_b", + "status": "errored", + "providers": [ + "null" + ], + "services": [ + "web" + ], + "events_url": "/v1/status/tasks/task_b?include=events" + } +} +``` + +#### Example: Individual Task Status with Events + +Request: +```shell-session +$ curl localhost:8558/v1/status/tasks/task_b?include=events +``` + +Response: +```json +{ + "task_b": { + "task_name": "task_b", + "status": "errored", + "providers": [ + "null" + ], + "services": [ + "web", + ], + "events_url": "/v1/status/tasks/task_b?include=events", + "events": [ + { + "id": "44137ba2-8fc9-6cbe-0e0e-e9305ee4f7f9", + "success": false, + "start_time": "2020-11-24T12:06:51.858292-05:00", + "end_time": "2020-11-24T12:06:52.770165-05:00", + "task_name": "task_b", + "error": { + "message": "example error: terraform-apply error" + }, + "config": { + "providers": [ + "null" + ], + "services": [ + "web", + ], + "source": "../modules/test_task" + } + }, + { + "id": "ef202675-502f-431f-b133-ed64d15b0e0e", + "success": true, + "start_time": "2020-11-24T12:04:18.651231-05:00", + "end_time": "2020-11-24T12:04:20.900115-05:00", + "task_name": "task_b", + "error": null, + "config": { + "providers": [ + "null" + ], + "services": [ + "web", + ], + "source": "../modules/test_task" + } + } + ] + } +} +``` diff --git a/website/pages/docs/nia/cli.mdx b/website/pages/docs/nia/cli.mdx index c23a1faa1..32edfc667 100644 --- a/website/pages/docs/nia/cli.mdx +++ b/website/pages/docs/nia/cli.mdx @@ -30,6 +30,22 @@ Behavior: This is the default mode in which Consul-Terraform-Sync passes through Usage: Intended to be run as a long running process after running once-mode successfully for given configuration and tasks. +### Inspect Mode + +Flag: `-inspect` + +Behavior: Consul-Terraform-Sync will display the proposed state changes for all tasks once and exit. No changes are applied in this mode. On encountering an error before completing, Consul-Terraform-Sync will exit with a non-zero status. + +Usage: Intended to be run before daemon-mode in order to confirm configuration is accurate and tasks would update network infrastructure as expected. + +----- + +Flag: `-inspect-task [task-name]` + +Behavior: This has similar behavior as `-inspect` mode for the selected task. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. + +Usage: Useful to debug one or more tasks to confirm configuration is accurate and the selected tasks would update network infrastructure as expected. + ### Once Mode Flag: `-once` diff --git a/website/pages/docs/nia/configuration.mdx b/website/pages/docs/nia/configuration.mdx new file mode 100644 index 000000000..ee11a930d --- /dev/null +++ b/website/pages/docs/nia/configuration.mdx @@ -0,0 +1,322 @@ +--- +layout: docs +page_title: Consul-Terraform-Sync Configuration +sidebar_title: Configuration +description: >- + Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the consul-terraform-sync daemon. +--- + +# Configuration Options for Consul-Terraform-Sync + +The Consul-Terraform-Sync daemon is configured using configuration files and supports [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) (HCL) and JSON file formats. + +## Global Config Options + +Top level options are reserved for configuring Consul-Terraform-Sync. + +```hcl +log_level = "INFO" +port = 8558 +syslog {} +buffer_period { + enabled = true + min = "5s" + max = "20s" +} +``` + +* `buffer_period` - Configures the default buffer period for all [tasks](#task) to dampen the affects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task executions. The default is enabled to reduce the number of times downstream infrastructure is updated within a short period of time. This is useful to enable in systems that have a lot of flapping. + * `enabled` - (bool: true) Enable or disable buffer periods globally. Specifying `min` will also enable it. + * `min` - (string: "5s") The minimum period of time to wait after changes are detected before triggering related tasks. + * `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. +* `log_level` - (string: "WARN") The log level to use for Consul-Terraform-Sync logging. +* `port` - (int: 8558) The port for Consul-Terraform-Sync to use to serve API requests. +* `syslog` - Specifies the syslog server for logging. + * `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging. + * `facility` - (string) Name of the syslog facility to log to. + * `name` - (string: "consul-terraform-sync") Name to use for the daemon process when logging to syslog. + +## Consul + +The `consul` block is used to configure Consul-Terraform-Sync connection with a Consul agent to perform queries to the Consul Catalog and Consul KV pertaining to task execution. + +To read more on suggestions for configuring the Consul agent, see [run an agent](/docs/nia/installation/requirements#run-an-agent). + +```hcl +consul { + address = "consul.example.com" + auth {} + tls {} + token = null + transport {} +} +``` + +* `address` - (string: "localhost:8500") Address is the address of the Consul agent. It may be an IP or FQDN. +* `auth` - Auth is the HTTP basic authentication for communicating with Consul. + * `enabled` - (bool) + * `username` - (string) + * `password` - (string) +* `tls` - Configure TLS to use a secure client connection with Consul. This option is required for Consul-Terraform-Sync when connecting to a [Consul agent with TLS verification enabled for HTTPS connections](/docs/agent/options#verify_incoming). + * `enabled` - (bool) Enable TLS. Specifying any option for TLS will also enable it. + * `verify` - (bool: true) Enables TLS peer verification. The default is enabled, which will check the global CA chain to make sure the given certificates are valid. If you are using a self-signed certificate that you have not added to the CA chain, you may want to disable SSL verification. However, please understand this is a potential security vulnerability. + * `key` - (string) The client key file to use for talking to Consul over TLS. The key also be provided through the `CONSUL_CLIENT_KEY` environment variable. + * `ca_cert` - (string) The CA file to use for talking to Consul over TLS. Can also be provided though the `CONSUL_CACERT` environment variable. + * `ca_path` - (string) The path to a directory of CA certs to use for talking to Consul over TLS. Can also be provided through the `CONSUL_CAPATH` environment variable. + * `cert` - (string) The client cert file to use for talking to Consul over TLS. Can also be provided through the `CONSUL_CLIENT_CERT` environment variable. + * `server_name` - (string) The server name to use as the SNI host when connecting via TLS. Can also be provided through the `CONSUL_TLS_SERVER_NAME` environment variable. +* `token` - (string) The ACL token to use for client communication with the local Consul agent. The token can also be provided through the `CONSUL_TOKEN` or `CONSUL_HTTP_TOKEN` environment variables. +* `transport` - Transport configures the low-level network connection details. + * `dial_keep_alive` - (string: "30s") The amount of time for keep-alives. + * `dial_timeout` - (string: "30s") The amount of time to wait to establish a connection. + * `disable_keep_alives` - (bool) Determines if keep-alives should be used. Disabling this significantly decreases performance. + * `idle_conn_timeout` - (string: "90s") The timeout for idle connections. + * `max_idle_conns` - (int: 100) The maximum number of total idle connections. + * `max_idle_conns_per_host` - (int: 1) The maximum number of idle connections per remote host. + * `tls_handshake_timeout` - (string: "10s") amount of time to wait to complete the TLS handshake. + +## Service + +A `service` block is an optional block to explicitly define configuration of services that Consul-Terraform-Sync monitors. A `service` block is only necessary for services that have non-default values e.g. custom datacenter. Services that do not have a `service` block configured will assume default values. To configure multiple services, specify multiple `service` blocks. For services to be included in task automation, the service must be included in the `task.services` field of a [`task` block](#task). If a `service` block is configured, the service can be referred in `task.services` by service name or ID. If a `service` block is not configured, it can only be referred to by service name. + +```hcl +service { + name = "web" + datacenter = "dc1" + description = "all instances of the service web in datacenter dc1" +} +``` + +* `datacenter` - (string) The datacenter the service is deployed in. +* `description` - (string) The human readable text to describe the service. +* `id` - (string) ID identifies the service for Consul-Terraform-Sync. This is used to explicitly identify the service config for a task to use. If no ID is provided, the service is identified by the service name within a [task definition](#task). +* `name` - (string: required) The Consul logical name of the service (required). +* `namespace` - (string: "default") The namespace of the service. If not provided, the namespace will be inferred from the Consul-Terraform-Sync ACL token, or default to the `default` namespace. +* `tag` - (string) Tag is used to filter nodes based on the tag for the service. + +## Task + +A `task` block configures which task to run in automation for the selected services. The list of services can include services explicitly defined by a `service` block or implicitly declared by the service name. The `task` block may be specified multiple times to configure multiple tasks. + +```hcl +task { + name = "taskA" + description = "" + providers = [] + services = ["web", "api"] + source = "org/example/module" + version = "1.0.0" + variable_files = [] +} +``` + +* `description` - (string) The human readable text to describe the service. +* `name` - (string: required) Name is the unique name of the task (required). A task name must start with a letter or underscore and may contain only letters, digits, underscores, and dashes. +* `providers` - (list[string]) Providers is the list of provider names the task is dependent on. This is used to map [Terraform provider configuration](#terraform-provider) to the task. +* `services` - (list[string]: required) Services is the list of logical service names or service IDs the task executes on. Consul-Terraform-Sync monitors the Consul Catalog for changes to these services and triggers the task to run. Any service value not explicitly defined by a `service` block with a matching ID is assumed to be a logical service name in the default namespace. +* `source` - (string: required) Source is the location the driver uses to fetch task dependencies. The source format is dependent on the driver. For the [Terraform driver](#terraform-driver), the source is the module path (local or remote). Read more on [Terraform module source here](https://www.terraform.io/docs/modules/sources.html). +* `variable_files` - (list[string]) A list of paths to files containing variables for the task. For the [Terraform driver](#terraform-driver), these are used as Terraform [variable defintion (`.tfvars`) files](https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files) and consists of only variable name assignments. The variable assignments must match the corresponding variable declarations available by the Terraform module for the task. Consul-Terraform-Sync will generate the intermediate variable declarations to pass as arguments from the auto-generated root module to the task's module. Variables are loaded in the same order as they appear in the order of the files. Duplicate variables are overwritten with the later value. *Note: unless specified by the module, configure arguments for Terraform providers using [`terraform_provider` blocks](#terraform-provider).* + ```hcl + address_group = "consul-services" + tags = [ + "consul-terraform-sync", + "terraform" + ] + ``` +* `version` - (string) The version of the provided source the task will use. For the [Terraform driver](#terraform-driver), this is the module version. The latest version will be used as the default if omitted. +* `buffer_period` - Configures the buffer period for the task to dampen the affects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task execution. The default is inherited from the top level [`buffer_period` block](#global-config-options). If configured, these values will take precedence over the global buffer period. This is useful to enable for a task that is dependent on services that have a lot of flapping. + * `enabled` - (bool) Enable or disable buffer periods for this task. Specifying `min` will also enable it. + * `min` - (string: "5s") The minimum period of time to wait after changes are detected before triggering related tasks. + * `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. + +## Terraform Driver + +The `driver` block configures the subprocess for Consul-Terraform-Sync to propagate infrastructure change. The Terraform driver is a required configuration for Consul-Terraform-Sync to relay provider discovery and installation information to Terraform, specifically the `required_providers` stanza. Other driver options do not need to be explicitly configured and has reasonable default values. + +```hcl +driver "terraform" { + log = false + persist_log = false + path = "" + working_dir = "" + + backend "consul" { + scheme = "https" + } + + required_providers { + myprovider = { + source = "namespace/myprovider" + version = "1.3.0" + } + } +} +``` + +* `backend` - (obj) The backend stores [Terraform state files](https://www.terraform.io/docs/state/index.html) for each task. This option is similar to the [Terraform backend configuration](https://www.terraform.io/docs/configuration/backend.html). Consul-Terraform-Sync supports Terraform backends used as a state store and currently does not support enhanced backends. If omitted, Consul-Terraform-Sync will generate default values and use configuration from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifer appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. + * Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html) + * Visit the Terraform documentation links above for the specific backend configuration options. +* `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the Consul-Terraform-Sync log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. +* `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the Consul-Terraform-Sync daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path. +* `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. +* `required_providers` - (obj: required) Declare each Terraform provider used across all tasks. This is similar to the [Terraform `terraform.required_providers`](https://www.terraform.io/docs/configuration/provider-requirements.html#requiring-providers) field to specify the source and version for each provider. Consul-Terraform-Sync will process these requirements when preparing each task that uses the provider. +* `version` - (string) The Terraform version to install and run in automation for task execution. If omittied, the driver will install the latest official release of Terraform. To change versions, remove the existing binary or change the path to install the desired version. Verify that the desired Terraform version is compatible across all Terraform modules used for Consul-Terraform-Sync automation. +* `working_dir` - (string: "sync-tasks") The base working directory to manage Terraform configurations all tasks. The full path of each working directory will have the task identifier appended to the end of the path, e.g. `./sync-tasks/task-name`. + +## Terraform Provider + +A `terraform_provider` block configures the options to interface with network infrastructure. Define a block for each provider required by the set of Terraform modules across all tasks. This block resembles [provider blocks for Terraform configuration](https://www.terraform.io/docs/configuration/providers.html). To find details on how to configure a provider, refer to the corresponding documentation for the Terraform provider. The main directory of publicly available providers are hosted on the [Terraform Registry](https://registry.terraform.io/browse/providers). + +The below configuration captures the general design of defining a provider using the [Vault Terraform provider](https://registry.terraform.io/providers/hashicorp/vault/latest/docs) as an example. + +```hcl +driver "terraform" { + required_providers { + vault = { + source = "hashicorp/vault" + version = "2.13.0" + } + } +} + +terraform_provider "vault" { + address = "vault.example.com" +} + +task { + source = "some/source" + providers = ["vault"] + services = ["web", "api"] +} +``` + +### Securely Configure Terraform Providers + +The `terraform_provider` block supports dynamically loading arguments from other sources. This can be used to securely configure your Terraform provider from the shell environment, Consul KV, or Vault. Using the template syntax below, you can avoid including sensitive values or credentials in plain text within configuration files for Consul-Terraform-Sync. + +Template syntax is only supported within the `terraform_provider` block. + +-> **Note**: The dynamic values will be included in the [`terraform.tfvars`](/docs/nia/network-drivers#terraform-tfvars) file for each Terraform workspace that references the provider. To exclude dynamic values from rendering to local files in plain text, check out this section on how to configure [provider environment variables](#provider-environment-variables) using dynamic configuration. + +#### Env + +`env` reads the given environment variable accessible to Consul-Terraform-Sync. + +```hcl +terraform_provider "example" { + address = "{{ env \"EXAMPLE_HOSTNAME\" }}" +} +``` + +#### Consul + +`key` queries the key's value in the KV store of the Consul server configured in the required [`consul` block](#consul). + +```hcl +terraform_provider "example" { + value = "{{ key \"path/example/key\" }}" +} +``` + +#### Vault + +`with secret` queries the [Vault KV secrets engine](https://www.vaultproject.io/api-docs/secret/kv). Vault is an optional source that will require operators to configure the Vault client. Access the secret using template dot notation `Data.data.`. + +```hcl +vault { + address = "vault.example.com" +} + +terraform_provider "example" { + token = "{{ with secret \"secret/my/path\" }}{{ .Data.data.foo }}{{ end }}" +} +``` + +##### Vault Configuration + +* `address` - (string) The URI of the Vault server. This can also be set via the `VAULT_ADDR` environment variable. +* `enabled` - (bool) Enabled controls whether the Vault integration is active. +* `namespace` - (string) Namespace is the Vault namespace to use for reading secrets. This can also be set via the `VAULT_NAMESPACE` environment variable. +* `renew_token` - (bool) Renews the Vault token. This can also be set via the `VAULT_RENEW_TOKEN` environment variable. +* `tls` - [(tls block)](#tls) TLS indicates the client should use a secure connection while talking to Vault. Supports the environment variables: + * `VAULT_CACERT` + * `VAULT_CAPATH` + * `VAULT_CLIENT_CERT` + * `VAULT_CLIENT_KEY` + * `VAULT_SKIP_VERIFY` + * `VAULT_TLS_SERVER_NAME` +* `token` - (string) Token is the Vault token to communicate with for requests. It may be a wrapped token or a real token. This can also be set via the `VAULT_TOKEN` environment variable, or via the `VaultAgentTokenFile`. +* `vault_agent_token_file` - (string) The path of the file that contains a Vault Agent token. If this is specified, Consul-Terraform-Sync will not try to renew the Vault token. +* `transport` - [(transport block)](#transport) Transport configures the low-level network connection details. +* `unwrap_token` - (bool) Unwraps the provided Vault token as a wrapped token. + +~> Note: Vault credentials are not accessible by tasks and the associated Terraform configurations, including automated Terraform modules. If the task requires Vault, you will need to seprately configure the Vault provider and explicitly include it in the `task.providers` list. + +### Provider Environment Variables + +Terraform providers may support shell environment variables as values for some of their arguments. When available, we recommend using environment variables as a way to keep credentials out of plain-text configuration files. Refer to the official provider docs hosted on the [Terraform Registry](https://registry.terraform.io/browse/providers) to find supported environment variables for a provider. By default, Consul-Terraform-Sync enables all Terraform workspaces for each task to inherit from its environment. + +The `task_env` block is a meta-argument available for the `terraform_provider` block that can be used to rename or scope the available environment to a selected set of variables. + +```hcl +terraform_provider "foo" { + // Direct assignment of provider arguments are rendered in plain-text in the + // generated terraform.tfvars file for the corresponding Terraform workspaces. + // token = "" + + // Instead of configuring the token argument directly for the provider, + // use the provider's supported environment variable for the token argument. + // For example, + // $ export FOO_TOKEN = "" + + // Alternatively, the task_env block allows you to rename the shell + // environment to the expected name by the provider. + task_env { + "FOO_TOKEN" = "{{ env \"CTS_FOO_TOKEN\" }}" + } +} +``` + +### Multiple Provider Configurations + +Consul-Terraform-Sync supports the [Terraform feature to define multiple configurations](https://www.terraform.io/docs/configuration/providers.html#alias-multiple-provider-configurations) for the same provider by utilizing the `alias` meta-argument. Define multiple provider blocks with the same provider name and set the `alias` to a unique value across a given provider. Select which provider configuration to use for a task by specifying the configuration with the provider name and alias (`.`) within the list of providers in the [`task.provider`](#task) parameter. A task can use multiple providers, but only one provider instance of a provider is allowed per task. + +The example Consul-Terraform-Sync configuration below defines two similar tasks executing the same module with different instances of the Vault provider. + +```hcl +terraform_provider "vault" { + alias = "a" + address = "vault.example.com" + namespace = "team-a" + task_env { + "VAULT_TOKEN" = "{{ env \"CTS_VAULT_TOKEN_A\" }}" + } +} + +terraform_provider "vault" { + alias = "b" + address = "vault.internal.com" + namespace = "team-b" + task_env { + "VAULT_TOKEN" = "{{ env \"CTS_VAULT_TOKEN_B\" }}" + } +} + +terraform_provider "dns" { + // ... +} + +task { + name = "task-a" + source = "org/module" + providers = ["vault.a", "dns"] + // ... +} + +task { + name = "task-b" + source = "org/module" + providers = ["vault.b", "dns"] + // ... +} +``` diff --git a/website/pages/docs/nia/installation/configuration.mdx b/website/pages/docs/nia/installation/configuration.mdx deleted file mode 100644 index 6f686b160..000000000 --- a/website/pages/docs/nia/installation/configuration.mdx +++ /dev/null @@ -1,264 +0,0 @@ ---- -layout: docs -page_title: Configure Consul-Terraform-Sync -sidebar_title: Configuration -description: >- - Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the consul-terraform-sync daemon. ---- - -# Configuration Options for Consul-Terraform-Sync - -The Consul-Terraform-Sync daemon is configured using configuration files and supports [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) (HCL) and JSON file formats. - -## Example - -An example HCL configuration is shown below to automate a task to execute a Terraform module for 2 services. - -```hcl -log_level = "info" - -consul { - address = "consul.example.com" -} - -buffer_period { - min = "5s" - max = "20s" -} - -task { - name = "website-x" - description = "automate services for website-x" - source = "namespace/example/module" - version = "1.0.0" - providers = ["myprovider"] - services = ["web", "api"] - buffer_period { - min = "10s" - } -} - -driver "terraform" { - required_providers { - myprovider = { - source = "namespace/myprovider" - version = "1.3.0" - } - } -} - -provider "myprovider" { - address = "myprovider.example.com" -} -``` - -## Global Config Options - -Top level options are reserved for configuring Consul-Terraform-Sync. - -```hcl -log_level = "INFO" -syslog {} -buffer_period { - enabled = true - min = "5s" - max = "20s" -} -``` - -* `log_level` - `(string: "WARN")` The log level to use for Consul-Terraform-Sync logging. -* `syslog` - Specifies the syslog server for logging. - * `enabled` - `(bool: false)` Enable syslog logging. Specifying other option also enables syslog logging. - * `facility` - `(string: )` Name of the syslog facility to log to. - * `name` - `(string: "consul-terraform-sync")` Name to use for the daemon process when logging to syslog. -* `buffer_period` - Configures the default buffer period for all [tasks](#task) to dampen the affects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task executions. The default is enabled to reduce the number of times downstream infrastructure is updated within a short period of time. This is useful to enable in systems that have a lot of flapping. - * `enabled` - `(bool: true)` Enable or disable buffer periods globally. Specifying `min` will also enable it. - * `min` - `(string: 5s)` The minimum period of time to wait after changes are detected before triggering related tasks. - * `max` - `(string: 20s)` The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - -## Consul - -The `consul` block is used to configure Consul-Terraform-Sync connection with a Consul agent to perform queries to the Consul Catalog and Consul KV pertaining to task execution. - -```hcl -consul { - address = "consul.example.com" - auth {} - tls {} - token = null - transport {} -} -``` - -* `address` - `(string: "localhost:8500")` Address is the address of the Consul agent. It may be an IP or FQDN. -* `auth` - Auth is the HTTP basic authentication for communicating with Consul. - * `enabled` - `(bool: false)` - * `username` - `(string: )` - * `password` - `(string: )` -* `tls` - Configure TLS to use a secure client connection with Consul. This requires Consul to be configured to serve HTTPS. - * `enabled` - `(bool: false)` Enable TLS. Specifying any option for TLS will also enable it. - * `verify` - `(bool: true)` Enables TLS peer verification. The default is enabled, which will check the global CA chain to make sure the given certificates are valid. If you are using a self-signed certificate that you have not added to the CA chain, you may want to disable SSL verification. However, please understand this is a potential security vulnerability. - * `key` - `(string: )` The client key file to use for talking to Consul over TLS. The key also be provided through the `CONSUL_CLIENT_KEY` environment variable. - * `ca_cert` - `(string: )` The CA file to use for talking to Consul over TLS. Can also be provided though the `CONSUL_CACERT` environment variable. - * `ca_path` - `(string: )` The path to a directory of CA certs to use for talking to Consul over TLS. Can also be provided through the `CONSUL_CAPATH` environment variable. - * `cert` - `(string: )` The client cert file to use for talking to Consul over TLS. Can also be provided through the `CONSUL_CLIENT_CERT` environment variable. - * `server_name` - `(string: )` The server name to use as the SNI host when connecting via TLS. Can also be provided through the `CONSUL_TLS_SERVER_NAME` environment variable. -* `token` - `(string: )` The ACL token to use for client communication with the local Consul agent. The token can also be provided through the `CONSUL_TOKEN` or `CONSUL_HTTP_TOKEN` environment variables. -* `transport` - Transport configures the low-level network connection details. - * `dial_keep_alive` - `(string: "30s")` The amount of time for keep-alives. - * `dial_timeout` - `(string: "30s")` The amount of time to wait to establish a connection. - * `disable_keep_alives` - `(bool: false)` Determines if keep-alives should be used. Disabling this significantly decreases performance. - * `idle_conn_timeout` - `(string: "90s")` The timeout for idle connections. - * `max_idle_conns` - `(int: 100)` The maximum number of total idle connections. - * `max_idle_conns_per_host` - `(int: 1)` The maximum number of idle connections per remote host. - * `tls_handshake_timeout` - `(string: "10s")` amount of time to wait to complete the TLS handshake. - -## Service - -A `service` block is an optional block to explicitly define configuration of services that Consul-Terraform-Sync monitors. A `service` block is only necessary for services that have non-default values e.g. custom datacenter. Services that do not have a `service` block configured will assume default values. To configure multiple services, specify multiple `service` blocks. For services to be included in task automation, the service must be included in the `task.services` field of a [`task` block](#task). If a `service` block is configured, the service can be referred in `task.services` by service name or ID. If a `service` block is not configured, it can only be referred to by service name. - -```hcl -service { - name = "web" - datacenter = "dc1" - description = "all instances of the service web in datacenter dc1" -} -``` - -* `datacenter` - `(string: )` The datacenter the service is deployed in. -* `description` - `(string: )` The human readable text to describe the service. -* `id` - `(string: )` ID identifies the service for Consul-Terraform-Sync. This is used to explicitly identify the service config for a task to use. If no ID is provided, the service is identified by the service name within a [task definition](#task). -* `name` - `(string: )` The Consul logical name of the service (required). -* `namespace` - `(string: "default")` The namespace of the service. If not provided, the namespace will be inferred from the Consul-Terraform-Sync ACL token, or default to the `default` namespace. -* `tag` - `(string: )` Tag is used to filter nodes based on the tag for the service. - -## Task - -A `task` block configures which task to run in automation for the selected services. The list of services can include services explicitly defined by a `service` block or implicitly declared by the service name. The `task` block may be specified multiple times to configure multiple tasks. - -```hcl -task { - name = "taskA" - description = "" - providers = [] - services = ["web", "api"] - source = "org/example/module" - version = "1.0.0" - variable_files = [] -} -``` - -* `description` - `(string: )` The human readable text to describe the service. -* `name` - `(string: )` Name is the unique name of the task (required). A task name must start with a letter or underscore and may contain only letters, digits, underscores, and dashes. -* `providers` - `(list(string): [])` Providers is the list of provider names the task is dependent on. This is used to map [provider configuration](#provider) to the task. -* `services` - `(list(string): [])` Services is the list of logical service names or service IDs the task executes on. Consul-Terraform-Sync monitors the Consul Catalog for changes to these services and triggers the task to run. Any service value not explicitly defined by a `service` block with a matching ID is assumed to be a logical service name in the default namespace. -* `source` - `(string: )` Source is the location the driver uses to fetch task dependencies. The source format is dependent on the driver. For the [Terraform driver](#terraform-driver), the source is the module path (local or remote). Read more on [Terraform module source here](https://www.terraform.io/docs/modules/sources.html). -* `variable_files` - `(list(string): [])` A list of paths to files containing variables for the task. For the [Terraform driver](#terraform-driver), these are used as Terraform [variable defintion (`.tfvars`) files](https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files) and consists of only variable name assignments. The variable assignments must match the corresponding variable declarations available by the Terraform module for the task. Consul-Terraform-Sync will generate the intermediate variable declarations to pass as arguments from the auto-generated root module to the task's module. Variables are loaded in the same order as they appear in the order of the files. Duplicate variables are overwritten with the later value. *Note: unless specified by the module, configure arguments for providers using [provider blocks](#provider).* - ```hcl - address_group = "consul-services" - tags = [ - "consul-terraform-sync", - "terraform" - ] - ``` -* `version` - `(string: )` The version of the provided source the task will use. For the [Terraform driver](#terraform-driver), this is the module version. The latest version will be used as the default if omitted. -* `buffer_period` - Configures the buffer period for the task to dampen the affects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task execution. The default is inherited from the top level [`buffer_period` block](#global-config-options). If configured, these values will take precedence over the global buffer period. This is useful to enable for a task that is dependent on services that have a lot of flapping. - * `enabled` - `(bool: false)` Enable or disable buffer periods for this task. Specifying `min` will also enable it. - * `min` - `(string: 5s)` The minimum period of time to wait after changes are detected before triggering related tasks. - * `max` - `(string: 20s)` The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - -## Terraform Driver - -The `driver` block configures the subprocess for Consul-Terraform-Sync to propagate infrastructure change. The Terraform driver does not need to be explicitly configured and has reasonable default values. - -```hcl -driver "terraform" { - log = false - persist_log = false - path = "" - working_dir = "" - - backend "consul" { - scheme = "https" - } - - required_providers { - myprovider = { - source = "namespace/myprovider" - version = "1.3.0" - } - } -} -``` - -* `backend` - `(obj: optional)` The backend stores [Terraform state files](https://www.terraform.io/docs/state/index.html) for each task. This option is similar to the [Terraform backend configuration](https://www.terraform.io/docs/backends/types/consul.html). Consul backend is the only supported backend at this time. If omitted, Consul-Terraform-Sync will generate default values and use configuration from the [`consul` block](#consul). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifer appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. -* `log` - `(bool: false)` Enable all Terraform output (stderr and stdout) to be included in the Consul-Terraform-Sync log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. -* `path` - `(string: optional)` The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the Consul-Terraform-Sync daemon. -* `persist_log` - `(bool: false)` Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. -* `required_providers` - `(obj)` Declare each Terraform provider used across all tasks. This is similar to the [Terraform `terraform.required_providers`](https://www.terraform.io/docs/configuration/provider-requirements.html#requiring-providers) field to specify the source and version for each provider. Consul-Terraform-Sync will process these requirements when preparing each task that uses the provider. -* `working_dir` - `(string: "sync-tasks")` The base working directory to manage Terraform configurations all tasks. The full path of each working directory will have the task identifier appended to the end of the path, e.g. `./sync-tasks/task-name`. - -## Provider - -A `provider` block configures the options to interface with network infrastructure. Define a block for each provider required by the set of Terraform modules across all tasks. This block resembles [provider blocks for Terraform configuration](https://www.terraform.io/docs/configuration/providers.html). To find details on how to configure a provider, refer to the corresponding documentation for the Terraform provider. The main directory of publicly available providers are hosted on the [Terraform Registry](https://registry.terraform.io/browse/providers). - -The below configuration captures the general design of defining a provider using the [Vault Terraform provider](https://registry.terraform.io/providers/hashicorp/vault/latest/docs) as an example. - -```hcl -driver "terraform" { - required_providers { - vault = { - source = "hashicorp/vault" - version = "2.13.0" - } - } -} - -provider "vault" { - address = "vault.example.com" -} - -task { - source = "some/source" - providers = ["vault"] - services = ["web", "api"] -} -``` - -### Multiple Provider Configurations - -Consul-Terraform-Sync supports the [Terraform feature to define multiple configurations](https://www.terraform.io/docs/configuration/providers.html#alias-multiple-provider-configurations) for the same provider by utilizing the `alias` meta-argument. Define multiple provider blocks with the same provider name and set the `alias` to a unique value across a given provider. Select which provider configuration to use for a task by specifying the configuration with the provider name and alias (`.`) within the list of providers in the [`task.provider`](#task) parameter. A task can use multiple providers, but only one provider instance of a provider is allowed per task. - -The example Consul-Terraform-Sync configuration below defines two similar tasks executing the same module with different instances of the Vault provider. - -```hcl -provider "vault" { - alias = "a" - address = "vault.example.com" - namespace = "team-a" -} - -provider "vault" { - alias = "b" - address = "vault.internal.com" - namespace = "team-b" -} - -provider "dns" { - // ... -} - -task { - name = "task-a" - source = "org/module" - providers = ["vault.a", "dns"] - // ... -} - -task { - name = "task-b" - source = "org/module" - providers = ["vault.b", "dns"] - // ... -} -``` diff --git a/website/pages/docs/nia/installation/configure.mdx b/website/pages/docs/nia/installation/configure.mdx new file mode 100644 index 000000000..7621469ac --- /dev/null +++ b/website/pages/docs/nia/installation/configure.mdx @@ -0,0 +1,100 @@ +--- +layout: docs +page_title: Configure Consul-Terraform-Sync +sidebar_title: Configure +description: >- + A high level guide to configure Consul-Terraform-Sync. +--- + +# Configure Consul-Terraform-Sync + +The page will cover the main components for configuring your Network Infrastructure Automation with Consul at a high level. For the full list of configuration options, visit the [Consul-Terraform-Sync Configuration page](/docs/nia/configuration). + +## Tasks + +A task captures a network automation process by defining which network resources to update for a set of services as those services change over time. Configure Consul-Terraform-Sync with one or more tasks that contain a list of Consul services, a Terraform module, and various Terraform providers. + +Within the [`task` block](/docs/nia/configuration#task), the list of services for a task represents the service layer that drives network automation. The `source` is the discovery location of the Terraform module that defines the network automation process for the task. + +Review the Terraform module to be used for network automation and identify the Terraform providers required by the module. If the module depends on a set of providers, include the list of provider names in the `providers` field to associate the corresponding provider configuration with the task. These providers will need to be configured later in a separate block. + +```hcl +task { + name = "website-x" + description = "automate services for website-x" + source = "namespace/example/module" + version = "1.0.0" + providers = ["myprovider"] + services = ["web", "api"] +} +``` + +## Terraform Providers + +Configuring Terraform providers within Consul-Terraform-Sync requires 2 config components. The first component is required within the [`driver.terraform` block](/docs/nia/configuration#terraform-driver). All providers configured for Consul-Terraform-Sync must be listed within the `required_providers` stanza to satisfy a [Terraform v0.13+ requirement](https://www.terraform.io/docs/configuration/provider-requirements.html#requiring-providers) for Terraform to discover and install them. The providers listed are later organized by Consul-Terraform-Sync to be included in the appropriate Terraform configuration files for each task. + +```hcl +driver "terraform" { + required_providers { + myprovider = { + source = "namespace/myprovider" + version = "1.3.0" + } + } +} +``` + +The second component for configuring a provider is the [`terraform_provider` block](/docs/nia/configuration#terraform-provider). This block resembles [provider blocks for Terraform configuration](https://www.terraform.io/docs/configuration/providers.html) and has the same responsibility for understanding API interactions and exposing resources for a specific infrastructure platform. + +Terraform modules configured for task automation may require configuring the referenced providers. For example, configuring the host address and authentication to interface with your network infrastructure. Refer to the Terraform provider documentation hosted on the [Terraform Registry](https://registry.terraform.io/browse/providers) to find available options. The `terraform_provider` block is loaded by Consul-Terraform-Sync during runtime and processed to be included in [autogenerated Terraform configuration files](/docs/nia/network-drivers#provider) used for task automation. Omitting the `terraform_provider` block for a provider will defer to the Terraform behavior assuming an empty default configuration. + +```hcl +terraform_provider "myprovider" { + address = "myprovider.example.com" +} +``` + +## Summary + +Piecing it all together, the configuration file for Consul-Terraform-Sync will have several HCL blocks in addition to other options for configuring the Consul-Terraform-Sync daemon: `task`, `driver.terraform`, and `terraform_provider` blocks. + +An example HCL configuration file is shown below to automate one task to execute a Terraform module for 2 services. + +```hcl +log_level = "info" + +syslog { + enabled = true +} + +consul { + address = "consul.example.com" +} + +task { + name = "website-x" + description = "automate services for website-x" + source = "namespace/example/module" + version = "1.0.0" + providers = ["myprovider"] + services = ["web", "api"] + buffer_period { + min = "10s" + } +} + +driver "terraform" { + log = true + + required_providers { + myprovider = { + source = "namespace/myprovider" + version = "1.3.0" + } + } +} + +terraform_provider "myprovider" { + address = "myprovider.example.com" +} +``` diff --git a/website/pages/docs/nia/installation/install.mdx b/website/pages/docs/nia/installation/install.mdx index b980f5cc0..6b1377b33 100644 --- a/website/pages/docs/nia/installation/install.mdx +++ b/website/pages/docs/nia/installation/install.mdx @@ -40,7 +40,7 @@ Usage of consul-terraform-sync: ## Connect your Consul Cluster -Consul-Terraform-Sync connects with your Consul cluster in order to monitor the Consul catalog for service changes. These service changes lead to downstream updates to your network devices. You can configure your Consul cluster in Consul-Terraform-Sync with the [Consul block](/docs/nia/installation/configuration#consul). Below is an example: +Consul-Terraform-Sync connects with your Consul cluster in order to monitor the Consul catalog for service changes. These service changes lead to downstream updates to your network devices. You can configure your Consul cluster in Consul-Terraform-Sync with the [Consul block](/docs/nia/configuration#consul). Below is an example: ```hcl consul { @@ -53,10 +53,10 @@ consul { Consul-Terraform-Sync interacts with your network device through a network driver. For the Terraform network driver, Consul-Terraform-Sync uses Terraform providers to make changes to your network infrastructure resources. You can reference existing provider docs on the Terraform Registry to configure each provider or create a new Terraform provider. -Once you have identified a Terraform provider for all of your network devices, you can configure them in Consul-Terraform-Sync with a [provider block](/docs/nia/installation/configuration#provider) for each network device. Below is an example: +Once you have identified a Terraform provider for all of your network devices, you can configure them in Consul-Terraform-Sync with a [`terraform_provider` block](/docs/nia/configuration#terraform-provider) for each network device. Below is an example: ```hcl -provider "fake-firewall" { +terraform_provider "fake-firewall" { address = "10.10.10.10" username = "admin" password = "password123" @@ -67,4 +67,4 @@ This provider is then used by task(s) to execute a Terraform module that will up ### Multiple Instances per Provider -You might have multiple instances of the same type of network device; for example, multiple instances of a firewall or load balancer. You can configure each instance with its own provider block and distinguish it by the `alias` meta-argument. See [multiple provider configurations](/docs/nia/installation/configuration#multiple-provider-configurations) for more details and an example of the configuration. +You might have multiple instances of the same type of network device; for example, multiple instances of a firewall or load balancer. You can configure each instance with its own provider block and distinguish it by the `alias` meta-argument. See [multiple provider configurations](/docs/nia/configuration#multiple-provider-configurations) for more details and an example of the configuration. diff --git a/website/pages/docs/nia/installation/requirements.mdx b/website/pages/docs/nia/installation/requirements.mdx index f5e1fed13..9a4a1843d 100644 --- a/website/pages/docs/nia/installation/requirements.mdx +++ b/website/pages/docs/nia/installation/requirements.mdx @@ -24,6 +24,8 @@ To install a local Consul agent, refer to the [Getting Started: Install Consul T The Consul agent must be running in order to dynamically update network devices. To run the local Consul agent, you can run Consul in development mode which can be started with `consul agent -dev` for simplicity. For more details on running Consul agent, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). +When running a Consul agent with Consul-Terraform-Sync in production, we suggest to keep a few considerations in mind. Consul-Terraform-Sync uses [blocking queries](/api/features/blocking) to monitor task dependencies, like changes to registered services. This results in multiple long running TCP connections between Consul-Terraform-Sync and the agent to poll changes for each dependency. Monitoring a high number of services may quickly hit the default Consul agent connection limits. Configure [`limits.http_max_conns_per_client`](/docs/agent/options#http_max_conns_per_client) for the agent to a reasonable value proportional to the number of services monitored by Consul-Terraform-Sync. + ### Register Services Consul-Terraform-Sync monitors Consul catalog for service changes which lead to downstream changes to your network devices. Without services, your Consul-Terraform-Sync daemon will be operational but idle. You can register services with your Consul agent either by loading a service definition or by HTTP API request. @@ -41,7 +43,7 @@ $ echo '{ $ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register ``` -The above example registers a service named "web" with your Consul agent. This represents a non-existent web service running at 10.10.10.10:8000. Your web service is now available for Consul-Terraform-Sync to consume. In Consul-Terraform-Sync, you can optionally configure the web service with a [service block](/docs/nia/installation/configuration#service) if it has any non-default values. You can also have Consul-Terraform-Sync monitor the web service to execute a task and update network device(s) by configuring "web" in [`task.services`](/docs/nia/installation/configuration#services) of a task block. +The above example registers a service named "web" with your Consul agent. This represents a non-existent web service running at 10.10.10.10:8000. Your web service is now available for Consul-Terraform-Sync to consume. In Consul-Terraform-Sync, you can optionally configure the web service with a [service block](/docs/nia/configuration#service) if it has any non-default values. You can also have Consul-Terraform-Sync monitor the web service to execute a task and update network device(s) by configuring "web" in [`task.services`](/docs/nia/configuration#services) of a task block. For more details on registering a service by HTTP API request, refer to the [register service API docs](https://www.consul.io/api-docs/agent/service#register-service). @@ -69,7 +71,7 @@ If there is no existing Terraform provider, a new Terraform provider can be [cre Working with a Terraform provider, you can write an integration task for Consul-Terraform-Sync by creating a Terraform module that is compatible with the Terraform driver. --> **Note:** [Release 0.1.0-techpreview1](https://github.com/hashicorp/consul-terraform-sync/releases/tag/v0.1.0-techpreview1) is compatible with Terraform modules with syntax supported by [Terraform version 0.13](https://github.com/hashicorp/terraform/blob/v0.13/CHANGELOG.md). +-> **Note:** [Release 0.1.0-techpreview2](https://github.com/hashicorp/consul-terraform-sync/releases/tag/v0.1.0-techpreview2) is compatible with Terraform modules with syntax supported by Terraform version [0.13](https://github.com/hashicorp/terraform/blob/v0.13/CHANGELOG.md) - [0.14](https://github.com/hashicorp/terraform/blob/v0.14/CHANGELOG.md). ### Using Terraform Modules @@ -162,7 +164,7 @@ Network infrastructure differs vastly across teams and organizations, and the au 3. Include descriptions to capture what the variables are and how they are used, and specify [custom validation rules for variables](https://www.terraform.io/docs/configuration/variables.html#custom-validation-rules) to provide context to users the expected format and conditions for the variables. 4. Set reasonable default values for variables that are optional, and omit default values for variables that are required module arguments. -Terraform is an explicit configuration language and requires variables to be declared, typed, and passed explicitly through as module arguments. Consul-Terraform-Sync abstracts this by creating intermediate variables at the root level from values intended for the module. These values are configured by practitioners within the [`task` block](/docs/nia/installation/configuration#variable_files). Value assignments are parsed to interpolate the corresponding variable declaration and are written to the appropriate Terraform files. A few assumptions are made for the intermediate variables: the variables users provide Consul-Terraform-Sync are declared and supported by the module, matching name and type. +Terraform is an explicit configuration language and requires variables to be declared, typed, and passed explicitly through as module arguments. Consul-Terraform-Sync abstracts this by creating intermediate variables at the root level from values intended for the module. These values are configured by practitioners within the [`task` block](/docs/nia/configuration#variable_files). Value assignments are parsed to interpolate the corresponding variable declaration and are written to the appropriate Terraform files. A few assumptions are made for the intermediate variables: the variables users provide Consul-Terraform-Sync are declared and supported by the module, matching name and type. #### Module Guidelines diff --git a/website/pages/docs/nia/installation/run.mdx b/website/pages/docs/nia/installation/run.mdx index e6b217b07..3974ffbe8 100644 --- a/website/pages/docs/nia/installation/run.mdx +++ b/website/pages/docs/nia/installation/run.mdx @@ -10,14 +10,20 @@ description: >- 1. Move the `consul-terraform-sync` binary to a location available on your `PATH`. -``` -$ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync -``` + ```shell-session + $ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync + ``` 2. Create the config.hcl file, all the options are available [here](/docs/nia/installation/configuration). 3. Run consul-terraform-sync. -``` -./consul-terraform-sync -config-file -``` + ```shell-session + $ consul-terraform-sync -config-file + ``` + +4. Check status of tasks. Replace port number if configured in Step 2. See additional API endpoints [here](/docs/nia/api) + + ```shell-session + $ curl localhost:8558/status/tasks + ``` diff --git a/website/pages/docs/nia/network-drivers.mdx b/website/pages/docs/nia/network-drivers.mdx index 7d9de4a75..fc2942e4c 100644 --- a/website/pages/docs/nia/network-drivers.mdx +++ b/website/pages/docs/nia/network-drivers.mdx @@ -26,7 +26,7 @@ Within the Consul-Terraform-Sync configuration for a task, practitioners can sel The root module is simple in structure and proxies Consul service information, configuration, and other variables to the Terraform module for the task. The content of the files that make up the root module are sourced from Consul-Terraform-Sync configuration, information for task's module to use as the automation playbook, and the Consul catalog for discovering service information. -Autogenerated root modules for tasks are stored in local subdirectories of the Terraform working directory. By default, the working directory `sync-tasks` is created in the current directory. To configure where Terraform configuration files are stored, set [`working_dir`](/docs/nia/installation/configuration#working_dir) for the Terraform driver to the desired path. +Autogenerated root modules for tasks are stored in local subdirectories of the Terraform working directory. By default, the working directory `sync-tasks` is created in the current directory. To configure where Terraform configuration files are stored, set [`working_dir`](/docs/nia/configuration#working_dir) for the Terraform driver to the desired path. A working directory with one task named "my-task" would have the folder structure below. @@ -42,10 +42,10 @@ sync-tasks/ The following files of the root module are generated for each task. An [example of a root module created by Consul-Terraform-Sync](https://github.com/hashicorp/consul-terraform-sync/tree/master/examples) can be found in the project repository. * `main.tf` - The main file contains the terraform block, provider blocks, and a module block calling the module configured for the task. * `terraform` block - The corresponding provider source and versions for the task from the configuration files are placed into this block for the root module. The Terraform backend from the configuration is also templated here. - * `provider` blocks - The provider blocks generated in the root module resemble the provider blocks in the configuration. They have identical arguments present and are set from the intermediate variable created per provider. + * `provider` blocks - The provider blocks generated in the root module resemble the `terraform_provider` blocks in the configuration. They have identical arguments present and are set from the intermediate variable created per provider. * `module` block - The module block is where the task's module is called as a [child module](https://www.terraform.io/docs/configuration/modules.html#calling-a-child-module). The child module contains the core logic for automation. Required and optional input variables are passed as arguments to the module. * `variables.tf` - This file contains 2 types of variable declarations. The required `services` input variable which determines module compatibility with Consul-Terraform Sync (read more on [compatible Terraform modules](/docs/nia/installation/requirements#how-to-create-a-compatible-terraform-module) for more details) and various intermediate variables used to dynamically configure providers. These intermediate provider variables are interpolated from the provider blocks and arguments configured in the Consul-Terraform-Sync configuration. -* `variables.module.tf` - This file is conditionally created if there are [variables configured for the task](/docs/nia/installation/configuration#variable_files) and contains the interpolated variable declarations that match the variables from configuration. These are then used to proxy the configured variables to the module through explicit assignment in the module block. +* `variables.module.tf` - This file is conditionally created if there are [variables configured for the task](/docs/nia/configuration#variable_files) and contains the interpolated variable declarations that match the variables from configuration. These are then used to proxy the configured variables to the module through explicit assignment in the module block. * `terraform.tfvars` - The variable definitions file is where the services input variable is assigned values from the Consul catalog. It is periodically updated to reflect the current state of the configured set of services for the task. * `terraform.tfvars.tmpl` - The template file is used by Consul-Terraform-Sync to template service information from the Consul catalog by using the HashiCorp configuration and templating library ([hashicorp/hcat](https://github.com/hashicorp/hcat)). diff --git a/website/pages/docs/nia/tasks.mdx b/website/pages/docs/nia/tasks.mdx index 8036c19f0..17a1adb49 100644 --- a/website/pages/docs/nia/tasks.mdx +++ b/website/pages/docs/nia/tasks.mdx @@ -23,9 +23,9 @@ task { } ``` -In the example task above, the "fake-firewall" and "null" providers, listed in the `providers` field, are used. These providers themselves should be configured in their own separate [provider blocks](/docs/nia/installation/configuration#provider). These providers are used in the Terraform module "example/firewall-policy/module", configured in the `source` field, to create, update, and destroy resources. This module may do something like use the providers to create and destroy firewall policy objects based on IP addresses. The IP addresses come from the "web" and "image" service instances configured in the `services` field. This service-level information is retrieved by Consul-Terraform-Sync which watches Consul catalog for changes. +In the example task above, the "fake-firewall" and "null" providers, listed in the `providers` field, are used. These providers themselves should be configured in their own separate [`terraform_provider` blocks](/docs/nia/configuration#terraform-provider). These providers are used in the Terraform module "example/firewall-policy/module", configured in the `source` field, to create, update, and destroy resources. This module may do something like use the providers to create and destroy firewall policy objects based on IP addresses. The IP addresses come from the "web" and "image" service instances configured in the `services` field. This service-level information is retrieved by Consul-Terraform-Sync which watches Consul catalog for changes. -See [task configuration](/docs/nia/installation/configuration#task) for more details on how to configure a task. +See [task configuration](/docs/nia/configuration#task) for more details on how to configure a task. ## Task Execution @@ -54,4 +54,72 @@ Consul-Terraform-Sync automatically generates any files needed to execute the ne Consul-Terraform-Sync will attempt to execute each task once upon startup to synchronize infrastructure with the current state of Consul. The daemon will stop and exit if any error occurs while preparing the automation environment or executing a task for the first time. This helps ensure all tasks have proper configuration and are executable before the daemon transitions into running tasks in full automation as service changes are discovered over time. After all tasks have successfully executed once, task failures during automation will be logged and retried or attempted again after a subsequent change. -Tasks are executed near-real time when service changes are detected. For services or environments that are prone to flapping, it may be useful to configure a [buffer period](/docs/nia/installation/configuration#buffer_period-1) for a task to accumulate changes before it is executed. The buffer period would reduce the number of consecutive network calls to infrastructure by batching changes for a task over a short duration of time. +Tasks are executed near-real time when service changes are detected. For services or environments that are prone to flapping, it may be useful to configure a [buffer period](/docs/nia/configuration#buffer_period-1) for a task to accumulate changes before it is executed. The buffer period would reduce the number of consecutive network calls to infrastructure by batching changes for a task over a short duration of time. + +## Status Information + +Status-related information is collected and offered via [status API](/docs/nia/api#status) to provide visibility into what and how the tasks are running. Information is offered in three-levels (lowest to highest): + - Event data + - Task status + - Overall status + +These three levels form a hierarchy where each level of data informs the one higher. The lowest-level, event data, is collected each time a task runs to update network infrastructure. This event data is then aggregated to inform individual task statuses. The count distribution of all the task statuses inform the overall status's task summary. + +### Event + +Each time a task's services has an update, Consul-Terraform-Sync takes a series of steps in order to update network infrastructure. This process starts with updating the task's templates to fetch new service data from Consul and ends with any post-actions after modifying network infrastructure. An event is a data structure that captures information on this process of updating network infrastructure. It stores information to help understand if the update to network infrastructure was successful or not, and it stores any errors that occurred. + +Sample event: +```json +{ + "id": "ef202675-502f-431f-b133-ed64d15b0e0e", + "success": false, + "start_time": "2020-11-24T12:05:18.651231-05:00", + "end_time": "2020-11-24T12:05:20.900115-05:00", + "task_name": "task_b", + "error": { + "message": "example error: error while doing terraform-apply" + }, + ... +} +``` + +For complete information on the event structure, see [events in our API documentation](/docs/nia/api#event). Event information can be retrieved by using the [`include=events` parameter](/docs/nia/api#include) with the [task status API](/docs/nia/api#task-status). + +### Task Status + +Each time a task runs to update network infrastructure, event data is stored for that run. 5 most recent events are stored for each task, and these stored events are used to determine task status. For example, if the most recent stored event is not successful but the others are, then the task's health status is "errored". + +Sample task status: +```json +{ + "task_name": "task_b", + "status": "errored", + "providers": [ + "null" + ], + "services": [ + "web", + ], + "events_url": "/v1/status/tasks/task_b?include=events", +} +``` + +Task status information can be retrieved with [task status API](/docs/nia/api#task-status). The API documentation includes details on what health statuses are available and how it is calculated based on events' success/failure information. + +### Overall Status + +Overall status returns a summary of the health statuses across all tasks. The summary is the count of tasks in each health status category. + +Sample overall status: +```json +{ + "task_summary": { + "successful": 28, + "errored": 5, + "critical": 1 + } +} +``` + +Overall status information can be retrieved with [overall status API](/docs/nia/api#overall-status). The API documentation includes details on what health statuses are available and how it is calculated based on task statuses' health status information.