before:
$ go test ./agent/consul -run TestLeader_ReapOrLeftMember_IgnoreSelf
ok github.com/hashicorp/consul/agent/consul 21.147s
after:
$ go test ./agent/consul -run TestLeader_ReapOrLeftMember_IgnoreSelf
ok github.com/hashicorp/consul/agent/consul 5.402s
Previously we were using two different criteria to decide where to run a
test. The main `go-test` job would skip Vault tests based on the
presence of the `vault` binary, but the `test-connect-ca-providers` job
would run tests based on the name.
This led to a scenario where a test may never run in CI.
To fix this problem I added a name check to the function we use to skip
the test. This should ensure that any test that requires vault is named
correctly to be run as part of the `test-connect-ca-providers` job.
At the same time I relaxed the regex we use. I verified this runs the
same tests using `go test --list Vault`. I made this change because a
bunch of tests in `agent/connect/ca` used `Vault` in the name, without
the underscores. Instead of changing a bunch of test names, this seemed
easier.
With this approach, the worst case is that we run a few extra tests in
the `test-connect-ca-providers` job, which doesn't seem like a problem.
Starting from and extending the mechanism introduced in #12110 we can specially handle the 3 main special Consul RPC endpoints that react to many config entries in a single blocking query in Connect:
- `DiscoveryChain.Get`
- `ConfigEntry.ResolveServiceConfig`
- `Intentions.Match`
All of these will internally watch for many config entries, and at least one of those will likely be not found in any given query. Because these are blends of multiple reads the exact solution from #12110 isn't perfectly aligned, but we can tweak the approach slightly and regain the utility of that mechanism.
### No Config Entries Found
In this case, despite looking for many config entries none may be found at all. Unlike #12110 in this scenario we do not return an empty reply to the caller, but instead synthesize a struct from default values to return. This can be handled nearly identically to #12110 with the first 1-2 replies being non-empty payloads followed by the standard spurious wakeup suppression mechanism from #12110.
### No Change Since Last Wakeup
Once a blocking query loop on the server has completed and slept at least once, there is a further optimization we can make here to detect if any of the config entries that were present at specific versions for the prior execution of the loop are identical for the loop we just woke up for. In that scenario we can return a slightly different internal sentinel error and basically externally handle it similar to #12110.
This would mean that even if 20 discovery chain read RPC handling goroutines wakeup due to the creation of an unrelated config entry, the only ones that will terminate and reply with a blob of data are those that genuinely have new data to report.
### Extra Endpoints
Since this pattern is pretty reusable, other key config-entry-adjacent endpoints used by `agent/proxycfg` also were updated:
- `ConfigEntry.List`
- `Internal.IntentionUpstreams` (tproxy)
This adds a new component blueprint for all our components. We've been
using README.mdx files for quite some time to document our components
for other engineers. This adds a default file to help new engineers get
started writing useful documentation. These README.mdx file are also
very useful for building out components in isolation from scratch and
and some point will be used to run automated component testing.
Many places in consul already treated node names case insensitively.
The state store indexes already do it, but there are a few places that
did a direct byte comparison which have now been corrected.
One place of particular consideration is ensureCheckIfNodeMatches
which is executed during snapshot restore (among other places). If a
node check used a slightly different casing than the casing of the node
during register then the snapshot restore here would deterministically
fail. This has been fixed.
Primary approach:
git grep -i "node.*[!=]=.*node" -- ':!*_test.go' ':!docs'
git grep -i '\[[^]]*member[^]]*\]
git grep -i '\[[^]]*\(member\|name\|node\)[^]]*\]' -- ':!*_test.go' ':!website' ':!ui' ':!agent/proxycfg/testing.go:' ':!*.md'
There are some cross-config-entry relationships that are enforced during
"graph validation" at persistence time that are required to be
maintained. This means that config entries may form a digraph at times.
Config entry replication procedes in a particular sorted order by kind
and name.
Occasionally there are some fixups to these digraphs that end up
replicating in the wrong order and replicating the leaves
(ingress-gateway) before the roots (service-defaults) leading to
replication halting due to a graph validation error related to things
like mismatched service protocol requirements.
This PR changes replication to give each computed change (upsert/delete)
a fair shot at being applied before deciding to terminate that round of
replication in error. In the case where we've simply tried to do the
operations in the wrong order at least ONE of the outstanding requests
will complete in the right order, leading the subsequent round to have
fewer operations to do, with a smaller likelihood of graph validation
errors.
This does not address all scenarios, but for scenarios where the edits
are being applied in the wrong order this should avoid replication
halting.
Fixes#9319
The scenario that is NOT ADDRESSED by this PR is as follows:
1. create: service-defaults: name=new-web, protocol=http
2. create: service-defaults: name=old-web, protocol=http
3. create: service-resolver: name=old-web, redirect-to=new-web
4. delete: service-resolver: name=old-web
5. update: service-defaults: name=old-web, protocol=grpc
6. update: service-defaults: name=new-web, protocol=grpc
7. create: service-resolver: name=old-web, redirect-to=new-web
If you shutdown dc2 just before (4) and turn it back on after (7)
replication is impossible as there is no single edit you can make to
make forward progress.
* docs/nia: new configuration for services condition & source_input (#11646)
* docs/nia: new configuration for services condition
* docs/nia: new configuration for services source_input
* reword filter and cts_user_defined_meta
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
* Update service block config to table format
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
* Remove deprecated driver.working_dir (#11831)
* Deprecate workspace_prefix for now workspaces.prefix (#11836)
* docs/nia: new config field names for services condition/source_input (#11896)
* docs/nia: new config field `names` for services condition/source_input
* Remove language about 'default condition' and services condition relation to services list
Context:
- Added a new `names` field to condition/source_input "services"
- `names` or `regexp` must be configured for condition/source_input "services"
This therefore:
- Removed relationship between condition/source_input "services" and
task.services list
- Removed concept of "default condition" i.e. condition "services" must be
configured with `names` or `regexp`, there is no meaningful unconfigured default
Change: remove language regarding "default condition" and relationship with services list
* docs/nia: Update paramters to table format
Changes from a bulleted list to a table. Also adds the possible response codes
and fixes the update example response to include the inspect object.
* docs/nia: Delete task API and CLI
* docs/nia: Update wording for run values
Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com>
* docs/nia: require condition "catalog-services" block's regexp to be configured (#11915)
Changes:
- Update Catalog Services Condition configuration docs to new table format
- Rewrite `regexp` field docs to be required, no longer optional
- Remove details about `regexp` field's original default behavior when the
field was optional
* docs/nia: Update status API docs to table format
* Cleaner wording for response descriptions
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
* docs/nia - 'source_includes_var' changes (#11939)
* docs/nia - condition "services" new field source_includes_var
- Add new configuration details for condition "services" block's
`source_includes_var` field.
- Note: this field's description is worded differently from condition type's
`source_includes_var` since a services variable is always required (unlike
other vars) for CTS modules.
- Also worded in a way to anticipate renaming to `use_as_module_input`
* docs/nia - change 'source_includes_var' default value from false to true
- Update configs
- Table-ify Consul-KV condition (reuse wording from Consul-KV source input)
* docs/nia - reword task execution page for source_includes_var changes
- Note: switched to using "module input" language over "source input" language.
Separate PR will make a mass change across docs
- Slim down general task condition section to have fewer details on module input
- Updated services, catalog-services, and consul-kv condition sections for
source_includes_var
- Add config page links for details
* Improve CTS acronym usage
- Use Consul-Terraform-Sync at the first instance with CTS in brackets - Consul-Terraform-Sync (CTS) and then CTS for all following instances on a per-page basis.
- some exceptions: left usage of the term `Consul-Terraform-Sync` in config examples and where it made sense for hyperlinking
* Improve CTS acronym usage (part 2) (#11991)
Per page:
- At first instance in text, use "Consul-Terraform-Sync (CTS)"
- Subsequent instances in text, use "CTS"
* Update schedule condition config to table format
* Update config tables with type column
* docs/nia: Update required fields values
Standardizing Required/Optional over boolean values.
* docs/nia: Standardize order of columns
Updated Required to come before Type, which is how the configurations are formatted. Also
changed the empty strings to "none" for default values.
* Deprecate port CLI option for CTS and updated example usage
* docs/nia cts multiple source input configuration updates (#12158)
* docs/nia cts multiple source input configuration updates
CTS expanded its usage of `source_input` block configurations and added
some restrictions. This change accounts for the following changes:
- `source_input` block can be configured for a task. No longer restricting to
scheduled task
- Multiple `source_input` blocks can be configured for a task. No longer
restricting to one
- Task cannot have multiple configurations defining the same variable type
Future work: We're planning to do some renaming from "source" to "module" for
v0.5. These changes are made in the code and not yet in the docs. These will be
taken care of across our docs in a separate PR. Perpetuating "source" in this
PR to reduce confusion.
* Apply suggestions from code review
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
* Apply suggestions from code review
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
* code review feedback
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
* Add "Consul object" glossary entry
Changes:
- Add "Consul object" to CTS glossary
- Format glossary terms so that they can be linked
- Add link to "Consul object" glossary entry
* Reorganize source_input limitations section
Co-authored-by: findkim <6362111+findkim@users.noreply.github.com>
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: findkim <6362111+findkim@users.noreply.github.com>
* docs/nia: overview of config streamlining deprecations (#12193)
* docs/nia: overview of config streamlining deprecations
* Update config snippets to use CodeTabs
* Apply code review feedback suggestions
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
* Apply suggestions from code review
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
* Clarify source table language
* Add use_as_module_input callout
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
* docs/nia: deprecate "services" field and "service" block (#12234)
* Deprecate `services` field
Did a search on "`services`", "`task.services`", "services list", and "services
field"
Changes:
- In config docs, mark `services` field as deprecated and `condition` block
as required.
- For necessary references to `services` field, mark with "(deprecated)" e.g.
when listing all options for source input
- Remove unnecessary references to `services` field from docs e.g. any docs
encouraging use of `services`
- Replace `services` field with `condition` / `module_input` "services" in
config snippets and explanations
* Deprecate `service` block
Did a search for "service block", "`service`", and "service {"
Changes:
- In config docs, mark `service` block as deprecated
- For necessary references to `service` block, mark with "(deprecated)"
- Remove unnecessary references to `service` block from docs
* Fix service block typos in config snippet
service block is singular and not plural
* docs/nia: deprecate "source includes var" and "source input" (#12244)
* Deprecate `source_includes_var` field
Did a search for "source_includes_var" and an audit of "include"
Changes
- In config docs, mark `source_includes_var` field as deprecated
- In config docs, add new field for `use_as_module_input`
- For necessary references to `source_includes_var`, mark with "(deprecated)"
- Audit and update "include" language
* Deprecate `source_input` field and language
Did a search and replace for "source_input", "source-input", "source input"
Changes:
- In config docs, mark `source_input` field as deprecated
- In config docs, add new entry for `module_input`
- For necessary references to `source_input`, mark with "(deprecated)"
- Remove or replace "source*input" with "module*input"
Note: added an anchor link alias e.g. `# Module Input ((#source-input))` for
headers that were renamed from "Source Input" so that bookmarked links won't
break
* Update config streamlining release removal version to 0.8
* remove duplicate bullet
* docs/nia: deprecate `source` (#12245)
* Update "source" field in config snippets to "module"
* Deprecate task config `source` field
Did a search and replace for "source" and "src"
Changes:
- In config docs, mark `source` field as deprecated
- In config docs, add new entry for `module`
- Remove or replace "source" with "module"
* Deprecate Status API Event `source` field
Changes:
- Mark `source` field as deprecated
- Add new entry for `module`
* docs/nia - Get Task API docs & Task Status API deprecations (#12303)
* docs/nia - Get Task API
Added a Task Object section intended to be shared with the Create Task API
* docs/nia - Deprecate non-status fields from Task Status API
Deprecate the fields that Get Task API replaces
* docs/nia - Align API docs on `:task_name` request resource
Followed a convention found in Nomad docs
* docs/nia - misc fixes
Context for some:
- remove "" from license_path for consistency - do not specify the default
value when empty string
- remove "optional" language from task condition. we want to move towards it
being required
* docs/nia - add new columns to API Task Object
* Added Create Task API documentation
* Added create task CLI documentation
* addressed code review comments
* fixed example
* docs/nia: Update task delete with async behavior
CTS delete task command is now asynchronous, so updating docs to reflect
this new behavior.
* update create task CLI with new changes from code
* update create task api and cli
- update curl command to include the json header
- update example task names to use 'task_a' to conform with other examples
* docs/nia: Fix hyphens in CTS CLI output
* docs/nia: Add auto-approve option in CLI
* docs/nia: Clarify infrastructure is not destroyed on task deletion
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Kim Ngo <6362111+findkim@users.noreply.github.com>
Co-authored-by: Melissa Kam <mkam@hashicorp.com>
Co-authored-by: Melissa Kam <3768460+mkam@users.noreply.github.com>
Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com>
Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
Co-authored-by: Michael Wilkerson <mwilkerson@hashicorp.com>
Co-authored-by: AJ Jwair <aj.jwair@hashicorp.com>
Eddie Rowe