luxolus
/
open-consul
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity

docs(access logs): new docs for access logging (#15948) 

Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
This commit is contained in:
Dan Stough 2023-01-11 11:41:02 -05:00 committed by GitHub
parent 554f1e6fee
commit 797bfb1677
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 401 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.changelog/15864.txt
View File

@ -1,3 +1,3 @@
```release-note:feature
connect: adds support for Envoy access logging. Access logging can be enabled using the [`proxy-defaults`](https://developer.hashicorp.com/consul/docs/connect/config-entries/proxy-defaults#accesslogs) config entry.
connect: adds support for Envoy [access logging](https://developer.hashicorp.com/consul/docs/connect/observability/access-logs). Access logging can be enabled using the [`proxy-defaults`](https://developer.hashicorp.com/consul/docs/connect/config-entries/proxy-defaults#accesslogs) config entry.
```

4
website/content/commands/connect/envoy.mdx
View File

@ -161,7 +161,9 @@ compatibility with Envoy and prevent potential issues. Default is `false`.
different port than the port specified in `-address` so that they do not
conflict with the health check endpoint.
- `-admin-access-log-path` The path to write the access log for the administration
- `-admin-access-log-path` -
**Deprecated in Consul 1.15.0 in favor of [`proxy-defaults` access logs](/docs/connect/config-entries/proxy-defaults#accesslogs).**
The path to write the access log for the administration
server. If no access log is desired specify `/dev/null`. By default it will
use `/dev/null`.

150
website/content/docs/connect/config-entries/proxy-defaults.mdx
View File

@ -2,20 +2,20 @@
layout: docs
page_title: Proxy Defaults - Configuration Entry Reference
description: >-
The proxy defaults configuration entry kind defines default behaviors for sidecar proxies in the service mesh. Use the reference guide to learn about `""proxy-defaults""` config entry parameters and how to expose HTTP paths through Envoy.
The proxy defaults configuration entry kind defines default behaviors for proxies in the service mesh. Use the reference guide to learn about `""proxy-defaults""` config entry parameters.
---
# Proxy Defaults Configuration Entry
The `proxy-defaults` configuration entry (`ProxyDefaults` on Kubernetes) allows you
to configure global defaults across all services for Connect proxy
configurations. Only one global entry is supported.
The `proxy-defaults` configuration entry (`ProxyDefaults` on Kubernetes) allows you to globally configure passthrough Envoy settings for proxies in the service mesh, including both sidecars and gateways.
It is different from the [`mesh` configuration entry](/docs/connect/config-entries/mesh), which sets Consul features for cluster peering, transparent proxy, and TLS behavior that also affect Consul servers.
Only one global entry is supported.
For Consul Enterprise, only the global entry in the `default` partition is recognized.
## Introduction
You can customize some service registration settings for service mesh sidecar
proxies centrally using the `proxy-defaults` configuration entry in the `kind`
field.
You can customize some service registration settings for service mesh proxies centrally using the `proxy-defaults` configuration entry in the `kind` field.
You can still override this centralized configuration for specific services
with the [`service-defaults`](/docs/connect/config-entries/service-defaults)
@ -74,6 +74,14 @@ Expose {
}
]
}
AccessLogs {
Enabled = < true | false >
DisableListenerLogs = < true | false , disables listener access logs for unrecognized traffic>
Type = "< file | stdout | stdout, the destination for access logs >"
Path = "< set the output path for 'file' based access logs >"
JSONFormat = "< json representation of access log format >"
TextFormat = "< text representation of access log format >"
}
```
```yaml
@ -99,6 +107,13 @@ spec:
localPathPort: <port where the local service is listening for connections to the path>
listenerPort: <port where the proxy will listen for connections>
protocol:= <protocol of the listener>
accessLogs:
enabled: < true | false >
disableListenerLogs: < true | false , disables listener access logs for unrecognized traffic>
type: < file | stdout | stdout, the destination for access logs >
path: < set the output path for 'file' based access logs >
jsonFormat: < json representation of access log format >
textFormat: < text representation of access log format >
```
```json
@ -129,6 +144,14 @@ spec:
"Protocol": "<protocol of the listener>"
}
]
},
"AccessLogs": {
"Enabled": < true | false >,
"DisableListenerLogs": < true | false , disables listener access logs for unrecognized traffic>,
"Type": "< file | stdout | stdout, the destination for access logs >",
"Path": "< set the output path for 'file' based access logs >",
"JSONFormat": "< json representation of access log format >",
"TextFormat": "< text representation of access log format >"
}
}
```
@ -173,6 +196,14 @@ Expose {
}
]
}
AccessLogs {
Enabled = < true | false >
DisableListenerLogs = < true | false , disables listener access logs for unrecognized traffic>
Type = "< file | stdout | stdout, the destination for access logs >"
Path = "< set the output path for 'file' based access logs >"
JSONFormat = "< json representation of access log format >"
TextFormat = "< text representation of access log format >"
}
```
```yaml
@ -199,6 +230,13 @@ spec:
localPathPort: <port where the local service is listening for connections to the path>
listenerPort: <port where the proxy will listen for connections>
protocol:= <protocol of the listener>
accessLogs:
enabled: < true | false >
disableListenerLogs: < true | false , disables listener access logs for unrecognized traffic>
type: < file | stdout | stdout, the destination for access logs >
path: < set the output path for 'file' based access logs >
jsonFormat: < json representation of access log format >
textFormat: < text representation of access log format >
```
```json
@ -230,6 +268,14 @@ spec:
"Protocol": "<protocol of the listener>"
}
]
},
"AccessLogs": {
"Enabled": < true | false >,
"DisableListenerLogs": < true | false , disables listener access logs for unrecognized traffic>,
"Type": "< file | stdout | stdout, the destination for access logs >",
"Path": "< set the output path for 'file' based access logs >",
"JSONFormat": "< json representation of access log format >",
"TextFormat": "< text representation of access log format >"
}
}
```
@ -431,6 +477,53 @@ spec:
},
],
},
{
name: 'AccessLogs',
type: 'AccessLogsConfig: <optional>',
description: `Controls the configuration of [Envoy's access logging](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/access_logging.html?highlight=access%20logs)
for all proxies in the mesh, including gateways. It also configures access logs on [Envoy's administration interface](https://www.envoyproxy.io/docs/envoy/latest/operations/admin.html?highlight=administration%20logs).`,
children: [
{
name: 'Enabled',
type: 'bool: false',
description: 'When enabled, access logs are emitted for all proxies in the mesh, including gateways.',
},
{
name: 'DisableListenerLogs',
type: 'bool: false',
description: `When enabled, access logs for traffic rejected at the listener-level are not emitted.
This traffic includes connections that do not match any of Envoy's configured filters, such as Consul upstream services.
Set this option to \`true\` if you do not want to log unknown requests that Envoy is not forwarding`,
},
{
name: 'Type',
type: 'string: "stdout"',
description: 'The destination for access logs. One of \`stdout\`, \`stderr\`, or \`file\`.',
},
{
name: 'Path',
type: 'string: ""',
description: 'The destination file for access logs. Only valid with \`Type\` set to \`file\`.',
},
{
name: 'JSONFormat',
type: 'string: (default as follows)',
description: `A JSON-formatted string that represents the format of each emitted access log.
By default, it is set to the [default access log format](/consul/docs/connect/observability/access-logs#default-log-format).
You can use Envoy [command operators](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#command-operators) to customize the emitted data.
Nesting is supported.
Invalid if a custom format is specified with TextFormat.`,
},
{
name: 'TextFormat',
type: 'string: ""',
description: `A formatted string that represents the format of each emitted access log.
Envoy [command operators](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#command-operators) can be used to customize the data emitted.
A new line is added to the string automatically.
Invalid when a custom JSONFormat is already specified.`,
}
],
},
]}
/>
@ -438,7 +531,7 @@ spec:
### Default protocol
The following example configures the default protocol for all sidecar proxies.
The following example configures the default protocol for all proxies.
<Tabs>
<Tab heading="Consul OSS">
@ -521,7 +614,7 @@ spec:
### Prometheus
The following example configures all sidecar proxies to expose Prometheus metrics.
The following example configures all proxies to expose Prometheus metrics.
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>
@ -555,9 +648,46 @@ spec:
</CodeTabs>
### Access Logs
The following example is a minimal configuration for enabling access logs for all proxies.
Refer to [access logs](/docs/connect/observability/access-logs) for advanced configurations.
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>
```hcl
Kind = "proxy-defaults"
Name = "global"
AccessLogs {
Enabled = true
}
```
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
accessLogs:
enabled: true
```
```json
{
"Kind": "proxy-defaults",
"Name": "global",
"AccessLogs": {
"Enabled": true
}
}
```
</CodeTabs>
### Proxy-specific defaults
The following example configures some custom default values for all sidecar proxies.
The following example configures some custom default values for all proxies.
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>

253
website/content/docs/connect/observability/access-logs.mdx Normal file
View File

@ -0,0 +1,253 @@
---
layout: docs
page_title: Service Mesh Observability - Access Logs
description: >-
Consul can emit access logs for application connections and requests that pass through Envoy proxies in the service mesh. Learn how to configure access logs, including minimum configuration requirements and the default log format.
---
# Access Logs
This topic describes configuration and usage for access logs. Consul can emit access logs to record application connections and requests that pass through proxies in a service mesh, including sidecar proxies and gateways.
You can use the application traffic records in access to logs to help you performance the following operations:
- **Diagnosing and Troubleshooting Issues**: Operators and application owners can identify configuration issues in the service mesh or the application by analyzing failed connections and requests.
- **Threat Detection**: Operators can review details about unauthorized attempts to access the service mesh and their origins.
- **Audit Compliance**: Operators can use access less for security compliance requirements for traffic entering and exiting the service mesh through gateways.
Consul supports access logs capture through Envoy proxies started through the [`consul connect envoy`](/consul/commands/connect/envoy) CLI command and [`consul-dataplane`](/consul/docs/connect/dataplane). Other proxies are not supported.
## Enable access logs
Access logs configurations are defined globally in the [`proxy-defaults`](/docs/connect/config-entries/proxy-defaults#accesslogs) configuration entry.
The following example is a minimal configuration for enabling access logs:
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>
```hcl
Kind = "proxy-defaults"
Name = "global"
AccessLogs {
Enabled = true
}
```
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
accessLogs:
enabled: true
```
```json
{
"Kind": "proxy-defaults",
"Name": "global",
"AccessLogs": {
"Enabled": true
}
}
```
</CodeTabs>
All proxies, including sidecars and gateways, emit access logs when the behavior is enabled.
Both inbound and outbound traffic through the proxy are logged, including requests made directly to [Envoy's administration interface](https://www.envoyproxy.io/docs/envoy/latest/operations/admin.html?highlight=administration%20logs#administration-interface).
If you enable access logs after the Envoy proxy was started, access logs for the administration interface are not captured until you restart the proxy.
## Default log format
Access logs use the following format when no additional customization is provided:
~> **Security warning:** The following log format contains IP addresses which may be a data compliance issue, depending on your regulatory environment.
Operators should carefully inspect their chosen access log format to prevent leaking sensitive or personally identifiable information.
```json
{
"start_time": "%START_TIME%",
"route_name": "%ROUTE_NAME%",
"method": "%REQ(:METHOD)%",
"path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
"protocol": "%PROTOCOL%",
"response_code": "%RESPONSE_CODE%",
"response_flags": "%RESPONSE_FLAGS%",
"response_code_details": "%RESPONSE_CODE_DETAILS%",
"connection_termination_details": "%CONNECTION_TERMINATION_DETAILS%",
"bytes_received": "%BYTES_RECEIVED%",
"bytes_sent": "%BYTES_SENT%",
"duration": "%DURATION%",
"upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
"x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
"user_agent": "%REQ(USER-AGENT)%",
"request_id": "%REQ(X-REQUEST-ID)%",
"authority": "%REQ(:AUTHORITY)%",
"upstream_host": "%UPSTREAM_HOST%",
"upstream_cluster": "%UPSTREAM_CLUSTER%",
"upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
"downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
"downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
"requested_server_name": "%REQUESTED_SERVER_NAME%",
"upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%"
}
```
Depending on the connection type, such TCP or HTTP, some of these fields may be empty.
## Custom log format
Envoy uses [command operators](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#command-operators) to expose information about application traffic.
You can use these fields to customize the access logs that proxies emit.
Custom logs can be either JSON format or text format.
### JSON format
You can format access logs in JSON so that you can parse them with Application Monitoring Platforms (APMs).
To use a custom access log, in the `proxy-defaults` configuration entry, set [`JSONFormat`](/docs/connect/config-entries/proxy-defaults#jsonformat) to the string representation of the desired JSON.
Nesting is supported.
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>
```hcl
Kind = "proxy-defaults"
Name = "global"
AccessLogs {
Enabled = true
JSONFormat = <<EOF
{
"myCustomKey" : {
"myStartTime" : "%START_TIME%",
"myProtocol" : "%PROTOCOL%"
}
}
EOF
}
```
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
accessLogs:
enabled: true
jsonFormat: |
{
"myCustomKey" : {
"myStartTime" : "%START_TIME%",
"myProtocol" : "%PROTOCOL%"
}
}
```
```json
{
"Kind": "proxy-defaults",
"Name": "global",
"AccessLogs": {
"Enabled": true,
"JSONFormat": "{ \"myCustomKey\" : { \"myStartTime\" : \"%START_TIME%\", \"myProtocol\" : \"%PROTOCOL%\"} }"
}
}
```
</CodeTabs>
### Text format
To use a custom access log formatted in plaintext, in the `proxy-defaults` configuration entry, set [`TextFormat`](/docs/connect/config-entries/proxy-defaults#textformat) to the desired customized string.
New lines are automatically added to the end of the log to keep each access log on its own line in the output.
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>
```hcl
Kind = "proxy-defaults"
Name = "global"
AccessLogs {
Enabled = true
TextFormat = "MY START TIME: %START_TIME%, THIS CONNECTIONS PROTOCOL IS %PROTOCOL%"
}
```
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
accessLogs:
enabled: true
textFormat: "MY START TIME: %START_TIME%, THIS CONNECTIONS PROTOCOL IS %PROTOCOL%"
```
```json
{
"Kind": "proxy-defaults",
"Name": "global",
"AccessLogs": {
"Enabled": true,
"JSONFormat": "MY START TIME: %START_TIME%, THIS CONNECTIONS PROTOCOL IS %PROTOCOL%"
}
}
```
</CodeTabs>
## Kubernetes
As part of its normal operation, the Envoy debugging logs for the `consul-dataplane`, `envoy`, or `envoy-sidecar` containers are written to `stderr`.
The access log [`Type`](/docs/connect/config-entries/proxy-defaults#type) is set to `stdout` by default for access logs when enabled.
Use a log aggregating solution to separate the machine-readable access logs from the Envoy process debug logs.
## Write to a file
You can configure Consul to write access logs to a file on the host where Envoy runs.
Envoy does not rotate log files. A log rotation solution, such as [logrotate](https://www.redhat.com/sysadmin/setting-logrotate), can prevent access logs from consuming too much of the host's disk space when writing to a file.
<CodeTabs tabs={[ "HCL", "Kubernetes YAML", "JSON" ]}>
```hcl
Kind = "proxy-defaults"
Name = "global"
AccessLogs {
Enabled = true
Type = "file"
Path = "/var/log/envoy/access-logs.txt"
}
```
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
accessLogs:
enabled: true
type: file
path: "/var/log/envoy/access-logs.txt"
```
```json
{
"Kind": "proxy-defaults",
"Name": "global",
"AccessLogs": {
"Enabled": true,
"Type": "file",
"Path": "/var/log/envoy/access-logs.txt"
}
}
```
</CodeTabs>

4
website/data/docs-nav-data.json
View File

@ -441,6 +441,10 @@
"title": "Overview",
"path": "connect/observability"
},
{
"title": "Access Logs",
"path": "connect/observability/access-logs"
},
{
"title": "UI Visualization",
"path": "connect/observability/ui-visualization"