Commit Graph

198 Commits

Author SHA1 Message Date
Dan Upton 9fe6c33c0d
xDS Load Balancing (#14397)
Prior to #13244, connect proxies and gateways could only be configured by an
xDS session served by the local client agent.

In an upcoming release, it will be possible to deploy a Consul service mesh
without client agents. In this model, xDS sessions will be handled by the
servers themselves, which necessitates load-balancing to prevent a single
server from receiving a disproportionate amount of load and becoming
overwhelmed.

This introduces a simple form of load-balancing where Consul will attempt to
achieve an even spread of load (xDS sessions) between all healthy servers.
It does so by implementing a concurrent session limiter (limiter.SessionLimiter)
and adjusting the limit according to autopilot state and proxy service
registrations in the catalog.

If a server is already over capacity (i.e. the session limit is lowered),
Consul will begin draining sessions to rebalance the load. This will result
in the client receiving a `RESOURCE_EXHAUSTED` status code. It is the client's
responsibility to observe this response and reconnect to a different server.

Users of the gRPC client connection brokered by the
consul-server-connection-manager library will get this for free.

The rate at which Consul will drain sessions to rebalance load is scaled
dynamically based on the number of proxies in the catalog.
2022-09-09 15:02:01 +01:00
Derek Menteer 8efe862b76 Merge branch 'main' of github.com:hashicorp/consul into derekm/split-grpc-ports 2022-09-08 14:53:08 -05:00
Krastin Krastev e2a6c2e932
docs: licensing improvements (#14488)
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
2022-09-06 18:39:07 -07:00
Derek Menteer b50bc443f3 Merge branch 'main' of github.com:hashicorp/consul into derekm/split-grpc-ports 2022-09-06 10:51:04 -05:00
Max Bowsher 3aefc4123f Merge branch 'main' into fix-kv_entries-metric 2022-08-29 22:22:10 +01:00
Derek Menteer 1233680362 Add documentation and changelog entry. 2022-08-29 13:43:49 -05:00
Pablo Ruiz García 4188769c32
Added new auto_encrypt.grpc_server_tls config option to control AutoTLS enabling of GRPC Server's TLS usage
Fix for #14253

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2022-08-24 12:31:38 -04:00
Jared Kirschner 92b718ad96
Merge pull request #14259 from hashicorp/docs/1-13-upgrade-considerations
docs: add 1.13 upgrade considerations
2022-08-18 18:18:33 -04:00
Jared Kirschner 18bb45db75 docs: add 1.13 upgrade considerations
Adds guidance when upgrading a Consul service mesh deployment to 1.13 and:
- using auto-encrypt or auto-config; or
- the HTTPS port is not enabled on Consul agents
2022-08-18 15:13:21 -07:00
Jared Kirschner 149f6a610d
docs: fix broken markdown 2022-08-16 23:08:09 -04:00
Max Bowsher 8d68284491 Correct problem with merge from master, including reformat of table 2022-08-14 16:16:41 +01:00
cskh e7b5baa3cc
feat(telemetry): add labels to serf and memberlist metrics (#14161)
* feat(telemetry): add labels to serf and memberlist metrics
* changelog
* doc update

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-08-11 22:09:56 -04:00
Luke Kysow e9960dfdf3
peering: default to false (#13963)
* defaulting to false because peering will be released as beta
* Ignore peering disabled error in bundles cachetype

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
Co-authored-by: freddygv <freddy@hashicorp.com>
Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>
2022-08-01 15:22:36 -04:00
Freddy a54903b0f4
Merge branch 'main' into fix-kv_entries-metric 2022-08-01 13:19:27 -06:00
Ashwin Venkatesh e4aaf467eb
Add peer counts to emitted metrics. (#13930) 2022-07-27 18:34:04 -04:00
trujillo-adam fb461995c0 fixed typo 2022-07-25 14:32:33 -07:00
David Yu 56a25ab6cc
docs: followup on grammar and typo for latency requirements (#13888) 2022-07-25 12:50:11 -07:00
alex 79bd7d1817
docs: add peering metric doc (#13862)
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2022-07-25 12:43:06 -07:00
David Yu 706808dd84
docs: add details around Consul latency requirements (#13881)
* docs: add details around Consul latency requirements
2022-07-25 11:02:31 -07:00
Luke Kysow 5263980884
Re-document peering disabled (#13879)
Change wording because it does have effect on clients because it
disables peering in the UI served from that client.
2022-07-25 09:30:37 -07:00
Luke Kysow d21f793b74
peering: add config to enable/disable peering (#13867)
* peering: add config to enable/disable peering

Add config:

```
peering {
  enabled = true
}
```

Defaults to true. When disabled:
1. All peering RPC endpoints will return an error
2. Leader won't start its peering establishment goroutines
3. Leader won't start its peering deletion goroutines
2022-07-22 15:20:21 -07:00
Jared Kirschner 7a58a4df96 docs: suggest using token header, not query param 2022-07-20 15:16:27 -07:00
R.B. Boyer 61ebb38092
server: ensure peer replication can successfully use TLS over external gRPC (#13733)
Ensure that the peer stream replication rpc can successfully be used with TLS activated.

Also:

- If key material is configured for the gRPC port but HTTPS is not
  enabled now TLS will still be activated for the gRPC port.

- peerstream replication stream opened by the establishing-side will now
  ignore grpc.WithBlock so that TLS errors will bubble up instead of
  being awkwardly delayed or suppressed
2022-07-15 13:15:50 -05:00
alex 04ec093130
no 1.9 style metrics (#13532)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-29 09:46:37 -07:00
Max Bowsher 728cd03e24
Merge branch 'main' into fix-kv_entries-metric 2022-06-27 18:57:03 +01:00
Max Bowsher 57cdbc6355 Fix use of trailing dots on metric names in telemetry.mdx
In the metric documentation, some metrics are written with an extra
trailing dot character. Often this is just spurious. In some cases, it
is an allusion to the metric name having various variable dot-separated
parts (which really ought to be labels, but that's another issue).

1) Trim all the dots which are without meaning.

2) Explicitly show with placeholders in angle brackets, where there are
   variable parts.

3) Remove a `json` type indicator from a code block which was not JSON
   (rather Prometheus metrics) format, since my IDE was reporting the
   non-JSON syntax as an error.

4) Reformat each Markdown table which I had touched with other changes
   to have consistent column spacing in the source representation. As
   a result, this PR has a lot of whitespace changes, so please make use
   of GitHub's "ignore whitespace" checkbox in the PR diff viewer.

Let me know if you would prefer:

- to not accept the table spacing reformat in the same PR as other
  changes

- to not accept the table spacing reformat ever

- to have all tables in the file reformatted, not just ones touched with
  other changes
2022-06-21 04:37:08 +01:00
David Yu b724f8b3ab
docs: Use "error" to use standard log level value (#13507)
* docs: Use "error" to use standard log level value
2022-06-20 16:07:38 -07:00
Max Bowsher 3f93b3cc88 Fix incorrect name and doc for kv_entries metric
The name of the metric as registered with the metrics library to provide
the help string, was incorrect compared with the actual code that sets
the metric value - bring them into sync.

Also, the help message was incorrect. Rather than copy the help message
from telemetry.mdx, which was correct, but felt a bit unnatural in the
way it was worded, update both of them to a new wording.
2022-06-19 11:58:23 +01:00
Luke Kysow d8a2825361
Add type info to options (#13477) 2022-06-16 10:09:39 -07:00
Eric Haberkorn eb9c341f5e
Lambda Beta Documentation (#13426)
* Document the `enable_serverless_plugin` Agent Configuration Option (#13372)
* Initial AWS Lambda documentation (#13245)
2022-06-15 11:14:16 -04:00
cskh bd3a4dfeab
Add isLeader metric to track if a server is a leader (#13304)
CTIA-21: sdd is_leader metric to track if a server is a leader

Co-authored-by: alex <8968914+acpana@users.noreply.github.com>
2022-06-03 13:07:37 -04:00
Blake Covarrubias a74710fd45
docs: Remove unnecessary use of CodeBlockConfig (#12974)
Remove empty CodeBlockConfig elements. These elements are not
providing any benefit for the enclosed code blocks. This PR removes
the elements so so that the source is easier to read.
2022-05-11 15:37:02 -07:00
Blake Covarrubias 13ac34c08b
docs: Fix spelling errors across site (#12973) 2022-05-10 07:28:33 -07:00
Blake Covarrubias f315eee98f
docs: Restore agent config docs removed in PR #12562 (#12907)
* docs: Re-add config file content removed in PR #12562

Re-add agent config option content that was erroneously removed in #12562 with
commit f4c03d234.

* docs: Re-add CLI flag content removed in PR #12562

Re-add CLI flag content that was erroneously removed in #12562 with
commit c5220fd18.

* Update website/content/docs/agent/config/cli-flags.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2022-05-05 10:08:15 -07:00
Dan Upton e4008124bd
Upgrade Raft to v1.3.9 for saturation metrics (#12865) 2022-04-27 17:17:31 +01:00
Karl Cardenas b0b197964c
Merge pull request #12562 from hashicorp/docs/blake-agent-config
docs: Agent configuration hierarchy reorganization
2022-04-12 12:33:42 -07:00
FFMMM 0c05f6d2e3
add docs for new labels (#12757) 2022-04-12 11:53:30 -07:00
Natalie Smith b9ec2222db docs: simplify agent docs slugs 2022-04-11 17:38:47 -07:00
Natalie Smith bea810cf62 docs: pr feedback 2022-04-11 17:38:17 -07:00
Natalie Smith a00492e622 chore: rebase updates 2022-04-11 17:38:17 -07:00
Natalie Smith cd73f27c84 docs: fix external links to agent config pages 2022-04-11 17:38:11 -07:00
Natalie Smith 9f693afcba docs: fix agent config links 2022-04-11 16:07:09 -07:00
Natalie Smith 02dc86cad1 docs: arrange agent configuration file parameters into logical groups 2022-04-11 16:06:54 -07:00
Blake Covarrubias 7a1d4f0ec5 docs: move configuration files content from agent/config/index to agent/config/agent-config-files 2022-04-11 16:06:20 -07:00
Blake Covarrubias 84123368db docs: move cli content from agent/config/index to agent/config/agent-config-cli
And add sections for logical groupings of options
2022-04-11 16:05:48 -07:00
Blake Covarrubias f7edcdc6b9 docs: move agent/options.mdx into agent/config/index.mdx and add placeholder .mdx files for cli/files
Also update nav data
2022-04-11 16:05:21 -07:00
Matt Keeler 3447880091
Enable running autopilot state updates on all servers (#12617)
* Fixes a lint warning about t.Errorf not supporting %w

* Enable running autopilot on all servers

On the non-leader servers all they do is update the state and do not attempt any modifications.

* Fix the RPC conn limiting tests

Technically they were relying on racey behavior before. Now they should be reliable.
2022-04-07 10:48:48 -04:00
Dhia Ayachi a6e7195bdf
documentation for config auto reload feature (#12548)
* add config watcher to the config package

* add logging to watcher

* add test and refactor to add WatcherEvent.

* add all API calls and fix a bug with recreated files

* add tests for watcher

* remove the unnecessary use of context

* Add debug log and a test for file rename

* use inode to detect if the file is recreated/replaced and only listen to create events.

* tidy ups (#1535)

* tidy ups

* Add tests for inode reconcile

* fix linux vs windows syscall

* fix linux vs windows syscall

* fix windows compile error

* increase timeout

* use ctime ID

* remove remove/creation test as it's a use case that fail in linux

* fix linux/windows to use Ino/CreationTime

* fix the watcher to only overwrite current file id

* fix linter error

* fix remove/create test

* set reconcile loop to 200 Milliseconds

* fix watcher to not trigger event on remove, add more tests

* on a remove event try to add the file back to the watcher and trigger the handler if success

* fix race condition

* fix flaky test

* fix race conditions

* set level to info

* fix when file is removed and get an event for it after

* fix to trigger handler when we get a remove but re-add fail

* fix error message

* add tests for directory watch and fixes

* detect if a file is a symlink and return an error on Add

* rename Watcher to FileWatcher and remove symlink deref

* add fsnotify@v1.5.1

* fix go mod

* do not reset timer on errors, rename OS specific files

* rename New func

* events trigger on write and rename

* add missing test

* fix flaking tests

* fix flaky test

* check reconcile when removed

* delete invalid file

* fix test to create files with different mod time.

* back date file instead of sleeping

* add watching file in agent command.

* fix watcher call to use new API

* add configuration and stop watcher when server stop

* add certs as watched files

* move FileWatcher to the agent start instead of the command code

* stop watcher before replacing it

* save watched files in agent

* add add and remove interfaces to the file watcher

* fix remove to not return an error

* use `Add` and `Remove` to update certs files

* fix tests

* close events channel on the file watcher even when the context is done

* extract `NotAutoReloadableRuntimeConfig` is a separate struct

* fix linter errors

* add Ca configs and outgoing verify to the not auto reloadable config

* add some logs and fix to use background context

* add tests to auto-config reload

* remove stale test

* add tests to changes to config files

* add check to see if old cert files still trigger updates

* rename `NotAutoReloadableRuntimeConfig` to `StaticRuntimeConfig`

* fix to re add both key and cert file. Add test to cover this case.

* review suggestion

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add check to static runtime config changes

* fix test

* add changelog file

* fix review comments

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* update flag description

Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>

* fix compilation error

* add static runtime config support

* fix test

* fix review comments

* fix log test

* Update .changelog/12329.txt

Co-authored-by: Dan Upton <daniel@floppy.co>

* transfer tests to runtime_test.go

* fix filewatcher Replace to not deadlock.

* avoid having lingering locks

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* split ReloadConfig func

* fix warning message

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* convert `FileWatcher` into an interface

* fix compilation errors

* fix tests

* extract func for adding and removing files

* add a coalesceTimer with a very small timer

* extract coaelsce Timer and add a shim for testing

* add tests to coalesceTimer fix to send remaining events

* set `coalesceTimer` to 1 Second

* support symlink, fix a nil deref.

* fix compile error

* fix compile error

* refactor file watcher rate limiting to be a Watcher implementation

* fix linter issue

* fix runtime config

* fix runtime test

* fix flaky tests

* fix compile error

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* fix agent New to return an error if File watcher New return an error

* add a coalesceTimer with a very small timer

* extract coaelsce Timer and add a shim for testing

* set `coalesceTimer` to 1 Second

* add flag description to agent command docs

* fix link

* add Static runtime config docs

* fix links and alignment

* fix typo

* Revert "add a coalesceTimer with a very small timer"

This reverts commit d9db2fcb8213a81ac761f04b458091409c5fb1ee.

* Revert "extract coaelsce Timer and add a shim for testing"

This reverts commit 0ab86012a415ffeb452acf58e52c9f37c9f49254.

* Apply suggestions from code review

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>
Co-authored-by: Daniel Upton <daniel@floppy.co>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2022-04-04 12:01:38 -04:00
FFMMM b0cba2ec03
mark disable_compat_1.9 to deprecate in 1.13, change default to true (#12675)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-04-01 10:35:56 -07:00
FFMMM 8dd4e609c1
docs: new rpc metric (#12608) 2022-03-31 13:04:33 -07:00