Commit Graph

2344 Commits

Author SHA1 Message Date
hc-github-team-consul-core 0846916941
Backport of fix a panic in the CLI when deleting an acl policy with an unknown name into release/1.16.x (#19691)
* backport of commit e07f4da212b2957e90550b7de11f6a783e0c374b

* backport of commit eecf61b67f4bfe14007d3057ed174fe7c185fac9

---------

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
2023-11-20 15:01:28 +00:00
hc-github-team-consul-core 8c6005a245
Backport of test: update certs for 10 year expiry into release/1.16.x (#19560)
backport of commit d6283a9352fe46e20206a4ab9ed76ff0c04bde53

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-11-07 20:12:16 +00:00
Ashesh Vidyut 99eb076eae
NET-4519 Collecting journald logs in "consul debug" bundle (#18797) (#18884)
NET-4519 Collecting journald logs in "consul debug" bundle (#18797)

* debug since

* fix docs

* chagelog added

* fix go mod

* debug test fix

* fix test

* tabs test fix

* Update .changelog/18797.txt



---------

Co-authored-by: Ganesh S <ganesh.seetharaman@hashicorp.com>
2023-09-19 17:37:11 +05:30
hc-github-team-consul-core 9a6e95f5e2
Backport of Remove flaky test assertions into release/1.16.x (#18871)
backport of commit 4dfc0c58b0db0f72c90afbef8f186a46005e510e

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2023-09-18 23:10:50 +00:00
hc-github-team-consul-core 5d4f7d2a4f
Backport of Removed nodename and status from consul snapshot save -append-filename command and using leader version in version into release/1.16.x (#18686)
* backport of commit e8f4bfe7a034e9f187790b7aa5f08f7d197e90ec

* backport of commit 5072958f6394dbf4318df549dec061d8e2ca8a5f

* Removed nodename and status from consul snapshot save -append-filename command and using leader version in version (#18680)

* init

* fix tests

* fix tests lint

* fix api call inside dc

* updated doc

* address comments

---------

Co-authored-by: absolutelightning <ashesh.vidyut@hashicorp.com>
Co-authored-by: Ashesh Vidyut <134911583+absolutelightning@users.noreply.github.com>
2023-09-06 10:56:39 +00:00
Ashesh Vidyut d60d408c1a
Backport 1.16 fix test snapshot save (#18665)
test: fix snapshot save w/ append test for ENT (#18656)

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
2023-09-04 08:13:49 +00:00
hc-github-team-consul-core 7ce902c8d3
Backport of NET 1594 - Snapshot Agent Filename Should Include Consul Version / Datacenter into release/1.16.x (#18638)
* backport of commit 5e1bae53412b563912a5c3042c6e48bffd09d75d

* backport of commit 8805feb78ddc01a7b0a6ae0a3d16968ef446ac3d

* backport of commit 99443a27650a39fd62dab512eb93aa4337ef0899

* backport of commit 37155cdc2ae636ad1a43e659649823a3db815cee

* backport of commit c7971c9884fa870dd78ca8758fa2180e8d257b4b

* backport of commit 2b66e28fc2e6c43ff5113931a74803ad98b9934e

* backport of commit 2c9f48e0c2b5b4fa2bb2131374b8b32d4d9c5654

* backport of commit 0130674592727129bf247815d5b7236c693278e3

* backport of commit 171e0a5ba76a61530e33e7b5ce7c484f4884d05e

* NET 1594 - Snapshot Agent Filename Should Include Consul Version / Datacenter (#18625)

* init

* tests added and few fixes

* revert arg message

* changelog added

* removed var declaration

* fix CI

* fix test

* added node name and status

* updated save.mdx

* added example

* fix tense

* fix description

---------

Co-authored-by: absolutelightning <ashesh.vidyut@hashicorp.com>
Co-authored-by: Ashesh Vidyut <134911583+absolutelightning@users.noreply.github.com>
2023-09-04 06:49:09 +00:00
Semir Patel 3fb8dda960
[BACKPORT] 1.16.x manual backport of OSS->CE branch (#18549) 2023-08-23 11:53:44 -05:00
hc-github-team-consul-core 239d703272
Backport of [NET-5217] [OSS] Derive sidecar proxy locality from parent service into release/1.16.x (#18438)
backport of commit 05604eeec1186a4c8a6156e7519a579171659c1a

Partial backport to include fix without logging changes (logging targets
code not included in 1.16.x).

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
2023-08-10 18:47:38 +00:00
hc-github-team-consul-core 44ef42b4c2
Backport of Fix policy lookup to allow for slashes into release/1.16.x (#18372)
* backport of commit 1a9cded960965d615a3a2146a4fef656839b1e34

* backport of commit cfec746d8b2d0b47b4aa66c951defa135f8791de

* backport of commit 8a9db4cffc557641f9209770c6134086f25322af

* backport of commit ac13bf16d6b5f853c26003cf088ff03988d8e235

---------

Co-authored-by: Jeremy Jacobson <jeremy.jacobson@hashicorp.com>
2023-08-03 20:43:45 +00:00
hc-github-team-consul-core 74e6e8f66a
Backport of member cli: add -filter expression to flags into release/1.16.x (#18276)
* backport of commit fe571c6c3e88e828e68172c1eb22b338e614635b

* backport of commit 779901b6cd428e1b5116a9fe33ff5f88ce27e88b

* backport of commit adfa76146adabd4a6ed54f067e6e3e886d5c198f

---------

Co-authored-by: cskh <hui.kang@hashicorp.com>
2023-07-25 14:57:08 -04:00
hc-github-team-consul-core 42a6d1e70f
Backport of [OSS] Fix initial_fetch_timeout to wait for all xDS resources into release/1.16.x (#18065)
* backport of commit 8a2f60ddae1a6ac561544e9cae80e9a037ad06d5

* backport of commit e17e53c93373fadedd61e904949e87c0c7d5ed26

* backport of commit d919d55c2eb4f206840f8d880edda8d5ad8c5fb4

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-07-10 21:27:56 +00:00
hc-github-team-consul-core a83bd1c1dc
Backport of [OSS] Improve Gateway Test Coverage of Catalog Health into release/1.16.x (#18014)
* backport of commit 954bd6ab1f1a2a00f549b10ad435cdead8d2cae2

* backport of commit 85c32d8f2e7e2c3a2855fe7a8fc4d10e3865b81f

* backport of commit 7ea3d622d75b4a69b8fc51d181b79c6b170ea47a

* backport of commit 127ae69c6dc967d575929e920813e7fe0d3fdef1

* backport of commit e04099b6cdd5dc20a36a19897816069669b2ef92

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-07-05 15:36:32 -04:00
hc-github-team-consul-core 4045bcfef7
Backport of feat: include nodes count in operator usage endpoint and cli command into release/1.16.x (#18012)
* backport of commit 54cdccd019ce32227f679b3fdca499283fdbdf5e

* backport of commit e543f716937fabed12ae2872d242a99416846d86

---------

Co-authored-by: Poonam Jadhav <poonam.jadhav@hashicorp.com>
2023-07-05 15:37:51 +00:00
hc-github-team-consul-core 1109d20262
Backport of watch: support -filter for consul watch: checks, services, nodes, service into release/1.16.x (#17965)
* backport to 1.16.x

---------

Co-authored-by: cskh <hui.kang@hashicorp.com>
2023-06-30 12:59:28 -07:00
hc-github-team-consul-core 4da9985ebf
Backport of Add emit_tags_as_labels to envoy bootstrap config when using Consul Telemetry Collector into release/1.16.x (#17901)
* backport of commit 94584cf5402c11e5a3c93467d43a8ad97af8d6b3

* backport of commit aed6a006935182f2ce0efd14c3cbef8168c78975

* backport of commit 2eb7a2a0670ce5ad69f7855615514722e2b4ace3

---------

Co-authored-by: josh <josh.timmons@hashicorp.com>
2023-06-27 16:51:00 +00:00
hc-github-team-consul-core 3bed38c630
Backport of [NET-3865] [Supportability] Additional Information in the output of 'consul operator raft list-peers' into release/1.16.x (#17738)
* backport of commit 2f94024b356f774d0032ad069babb1a3fce6cfbd

* backport of commit 7626d0992dd59f424d893dd462bbde3a9cb0646c

* backport of commit 79aabc958aed60b2bd97c836529d53a627262124

* backport of commit 44eee4168b5a0d405636dbcb5b14ee7959334550

* cherry-pick

* removed detaied flag

* fix change

* removed autopilot health

* removed tests

* removed detailed flag

---------

Co-authored-by: Ashesh Vidyut <ashesh.vidyut@hashicorp.com>
Co-authored-by: David Yu <dyu@hashicorp.com>
Co-authored-by: Ashesh Vidyut <134911583+absolutelightning@users.noreply.github.com>
2023-06-21 11:26:48 +05:30
hc-github-team-consul-core 72409898bc
Backport of [NET-4107][Supportability] Log Level set to TRACE and duration set to 5m for consul-debug into release/1.16.x (#17728)
* backport of commit 9d72a262f357ef0ee6409acae7d3c24d8f87caec
* backport of commit 9b9bb8d206bd68721bc89a9cdf488e8bba38f4d2
* backport of commit ba448092974b8f0439e64e338267835a02480f73

---------

Co-authored-by: Ashesh Vidyut <ashesh.vidyut@hashicorp.com>
2023-06-20 13:48:33 -07:00
hc-github-team-consul-core 58437bfdc8
backport of commit 6110cd6c7ccee249065672f87248e4b07fee4556 (#17662)
Co-authored-by: Poonam Jadhav <poonam.jadhav@hashicorp.com>
2023-06-12 16:47:03 +00:00
Dan Stough b068b41288
Revert "fix(connect envoy): set initial_fetch_timeout to wait for initial xDS… (#17317)" (#17540)
This reverts commit be7d2a4d842a8cbab7ce6161e88807a1f0a946b0.
2023-06-01 13:10:41 -04:00
Nathan Coleman 22cf348c24
Export peering cli (#15654)
* Sujata's peering-cli branch

* Added error message for connecting to cluster

* We can export service to peer

* export handling multiple peers

* export handles multiple peers

* export now can handle multiple services

* Export after 1st cleanup

* Successful export

* Added the namespace option

* Add .changelog entry

* go mod tidy

* Stub unit tests for peering export command

* added export in peering.go

* Adding export_test

* Moved the code to services from peers and cleaned the serviceNamespace

* Added support for exporting to partitions

* Fixed partition bug

* Added unit tests for export command

* Add multi-tenancy flags

* gofmt

* Add some helpful comments

* Exclude namespace + partition flags when running OSS

* cleaned up partition stuff

* Validate required flags differently for OSS vs. ENT

* Update success output to include only the requested consumers

* cleaned up

* fixed broken test

* gofmt

* Include all flags in OSS build

* Remove example previously added to peering command

* Move stray import into correct block

* Update changelog entry to include support for exporting to a partition

* Add required-ness label to consumer-peers flag description

* Update command/services/export/export.go

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* Add docs placeholder for new services export command

* Moved piece of code to OSS

* Break config entry init + update into separate functions

* fixed

* Vary existing service export comparison for OSS vs. ENT

* Move OSS-specific test to export_oss_test.go

* Set config entry name based on partition being exported from

* Set namespace on added services

* Adding namespace

* Remove export documentation

We will include documentation in a followup PR

* Consolidate code from export_oss into export.go

* Consolidated export_oss_test.go and export_test.go

* Add example of partition export to command synopsis

* Allow empty peers flag if partitions flag provided

* Add test coverage for -consumer-partitions flag

* Update command/services/export/export.go

Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>

* Update command/services/export/export.go

Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>

* Update changelog entry

* Use "cluster peers" to clear up any possible confusion

* Update test assertions

---------

Co-authored-by: 20sr20 <sujata@hashicorp.com>
Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
2023-05-31 14:27:35 -04:00
Jared Kirschner 303f1cfd08
Accept ap, datacenter, and namespace query params (#17525)
This commit only contains the OSS PR (datacenter query param support).
A separate enterprise PR adds support for ap and namespace query params.

Resources in Consul can exists within scopes such as datacenters, cluster
peers, admin partitions, and namespaces. You can refer to those resources from
interfaces such as the CLI, HTTP API, DNS, and configuration files.

Some scope levels have consistent naming: cluster peers are always referred to
as "peer".

Other scope levels use a short-hand in DNS lookups...
- "ns" for namespace
- "ap" for admin partition
- "dc" for datacenter

...But use long-hand in CLI commands:
- "namespace" for namespace
- "partition" for admin partition
- and "datacenter"

However, HTTP API query parameters do not follow a consistent pattern,
supporting short-hand for some scopes but long-hand for others:
- "ns" for namespace
- "partition" for admin partition
- and "dc" for datacenter.

This inconsistency is confusing, especially for users who have been exposed to
providing scope names through another interface such as CLI or DNS queries.

This commit improves UX by consistently supporting both short-hand and
long-hand forms of the namespace, partition, and datacenter scopes in HTTP API
query parameters.
2023-05-31 11:50:24 -04:00
Ashvitha 077a755fe8
HCP Telemetry Feature (#17460)
* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* [HCP Observability] Init OTELSink in Telemetry (#17162)

* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Initialize OTELSink with sync.Map for all the instrument stores.

* Added telemetry agent to client and init sink in deps

* Fixed client

* Initalize sink in deps

* init sink in telemetry library

* Init deps before telemetry

* Use concrete telemetry.OtelSink type

* add /v1/metrics

* Avoid returning err for telemetry init

* move sink init within the IsCloudEnabled()

* Use HCPSinkOpts in deps instead

* update golden test for configuration file

* Switch to using extra sinks in the telemetry library

* keep name MetricsConfig

* fix log in verifyCCMRegistration

* Set logger in context

* pass around MetricSink in deps

* Fix imports

* Rebased onto otel sink pr

* Fix URL in test

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* pass extraSinks as function param instead

* Add default interval as package export

* remove verifyCCM func

* Add clusterID

* Fix import and add t.Parallel() for missing tests

* Kick Vercel CI

* Remove scheme from endpoint path, and fix error logging

* return metrics.MetricSink for sink method

* Update SDK

* [HCP Observability] Metrics filtering and Labels in Go Metrics sink (#17184)

* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Initialize OTELSink with sync.Map for all the instrument stores.

* Added telemetry agent to client and init sink in deps

* Fixed client

* Initalize sink in deps

* init sink in telemetry library

* Init deps before telemetry

* Use concrete telemetry.OtelSink type

* add /v1/metrics

* Avoid returning err for telemetry init

* move sink init within the IsCloudEnabled()

* Use HCPSinkOpts in deps instead

* update golden test for configuration file

* Switch to using extra sinks in the telemetry library

* keep name MetricsConfig

* fix log in verifyCCMRegistration

* Set logger in context

* pass around MetricSink in deps

* Fix imports

* Rebased onto otel sink pr

* Fix URL in test

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* pass extraSinks as function param instead

* Add default interval as package export

* remove verifyCCM func

* Add clusterID

* Fix import and add t.Parallel() for missing tests

* Kick Vercel CI

* Remove scheme from endpoint path, and fix error logging

* return metrics.MetricSink for sink method

* Update SDK

* Added telemetry agent to client and init sink in deps

* Add node_id and __replica__ default labels

* add function for default labels and set x-hcp-resource-id

* Fix labels tests

* Commit suggestion for getDefaultLabels

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>

* Fixed server.id, and t.Parallel()

* Make defaultLabels a method on the TelemetryConfig object

* Rename FilterList to lowercase filterList

* Cleanup filter implemetation by combining regex into a single one, and making the type lowercase

* Fix append

* use regex directly for filters

* Fix x-resource-id test to use mocked value

* Fix log.Error formats

* Forgot the len(opts.Label) optimization)

* Use cfg.NodeID instead

---------

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>

* remove replic tag (#17484)

* [HCP Observability] Add custom metrics for OTEL sink, improve logging, upgrade modules and cleanup metrics client (#17455)

* Add custom metrics for Exporter and transform operations

* Improve deps logging

Run go mod tidy

* Upgrade SDK and OTEL

* Remove the partial success implemetation and check for HTTP status code in metrics client

* Add x-channel

* cleanup logs in deps.go based on PR feedback

* Change to debug log and lowercase

* address test operation feedback

* use GetHumanVersion on version

* Fix error wrapping

* Fix metric names

* [HCP Observability] Turn off retries for now until dynamically configurable (#17496)

* Remove retries for now until dynamic configuration is possible

* Clarify comment

* Update changelog

* improve changelog

---------

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>
2023-05-29 16:11:08 -04:00
skpratt 6db3842ca6
fips oss version changes (#17500) 2023-05-29 13:02:06 -05:00
John Landa 4859cfb47b
Add ACLs Enabled field to consul agent startup status message (#17086)
* Add ACLs Enabled field to consul agent startup status message

* Add changelog

* Update startup messages to include default ACL policy configuration

* Correct import groupings
2023-05-16 13:47:02 -05:00
Connor 6532ede487
Rename hcp-metrics-collector to consul-telemetry-collector (#17327)
* Rename hcp-metrics-collector to consul-telemetry-collector

* Fix docs

* Fix doc comment

---------

Co-authored-by: Ashvitha Sridharan <ashvitha.sridharan@hashicorp.com>
2023-05-16 14:36:05 -04:00
Dan Stough 4a245b2bff
fix(connect envoy): set initial_fetch_timeout to wait for initial xDS… (#17317)
* fix(connect envoy): set initial_fetch_timeout to wait for initial xDS indefinitely

---------

Co-authored-by: Kiril Angov <kiril.angov@gmail.com>
2023-05-15 10:45:16 -04:00
Freddy 0459069523
Hash namespace+proxy ID when creating socket path (#17204)
UNIX domain socket paths are limited to 104-108 characters, depending on
the OS. This limit was quite easy to exceed when testing the feature on
Kubernetes, due to how proxy IDs encode the Pod ID eg:
metrics-collector-59467bcb9b-fkkzl-hcp-metrics-collector-sidecar-proxy

To ensure we stay under that character limit this commit makes a
couple changes:
- Use a b64 encoded SHA1 hash of the namespace + proxy ID to create a
  short and deterministic socket file name.
- Add validation to proxy registrations and proxy-defaults to enforce a
  limit on the socket directory length.
2023-05-09 12:20:26 -06:00
Freddy a90fff4789
Revert "[CC-4519] Include Consul NodeID in Envoy bootstrap metadata" (#17191) 2023-04-28 15:23:55 -06:00
Paul Glass e1cff98a8f
Permissive mTLS: Config entry filtering and CLI warnings (#17183)
This adds filtering for service-defaults: consul config list -filter 'MutualTLSMode == "permissive"'.

It adds CLI warnings when the CLI writes a config entry and sees that either service-defaults or proxy-defaults contains MutualTLSMode=permissive, or sees that the mesh config entry contains AllowEnablingPermissiveMutualTLSMode=true.
2023-04-28 12:51:36 -05:00
John Landa b9cf6579e6
Remove artificial ACLTokenMaxTTL limit for configuring acl token expiry (#17066)
* Remove artificial ACLTokenMaxTTL limit for configuring acl token expiry

* Add changelog

* Remove test on default MaxTokenTTL

* Change to imperitive tense for changelog entry
2023-04-28 10:57:30 -05:00
Freddy 29d5811f0d
Update HCP bootstrapping to support existing clusters (#16916)
* Persist HCP management token from server config

We want to move away from injecting an initial management token into
Consul clusters linked to HCP. The reasoning is that by using a separate
class of token we can have more flexibility in terms of allowing HCP's
token to co-exist with the user's management token.

Down the line we can also more easily adjust the permissions attached to
HCP's token to limit it's scope.

With these changes, the cloud management token is like the initial
management token in that iit has the same global management policy and
if it is created it effectively bootstraps the ACL system.

* Update SDK and mock HCP server

The HCP management token will now be sent in a special field rather than
as Consul's "initial management" token configuration.

This commit also updates the mock HCP server to more accurately reflect
the behavior of the CCM backend.

* Refactor HCP bootstrapping logic and add tests

We want to allow users to link Consul clusters that already exist to
HCP. Existing clusters need care when bootstrapped by HCP, since we do
not want to do things like change ACL/TLS settings for a running
cluster.

Additional changes:

* Deconstruct MaybeBootstrap so that it can be tested. The HCP Go SDK
  requires HTTPS to fetch a token from the Auth URL, even if the backend
  server is mocked. By pulling the hcp.Client creation out we can modify
  its TLS configuration in tests while keeping the secure behavior in
  production code.

* Add light validation for data received/loaded.

* Sanitize initial_management token from received config, since HCP will
  only ever use the CloudConfig.MangementToken.

* Add changelog entry
2023-04-27 22:27:39 +02:00
Freddy 8082ca612c
[CC-4519] Include Consul NodeID in Envoy bootstrap metadata (#17139)
This is being added so that metrics sent to HCP can be augmented with the source node's ID.

Opting not to add this to stats_tag out of caution, since it would increase the cardinality of metrics emitted by Envoy for all users.

There is no functional impact to Envoy expected from this change.
2023-04-26 10:04:57 -06:00
Dan Upton 52ce151221
Raft storage backend (#16619) 2023-04-04 17:30:06 +01:00
Ronald 7a5c8dc1eb
Copyright headers for command folder (#16705)
* copyright headers for agent folder

* Ignore test data files

* fix proto files and remove headers in agent/uiserver folder

* ignore deep-copy files

* copyright headers for agent folder

* Copyright headers for command folder

* fix merge conflicts
2023-03-28 15:12:30 -04:00
Eric Haberkorn 672c1eea41
add enterprise xds tests (#16738) 2023-03-22 14:56:18 -04:00
Ashvitha f514182f3e
Allow HCP metrics collection for Envoy proxies
Co-authored-by: Ashvitha Sridharan <ashvitha.sridharan@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>

Add a new envoy flag: "envoy_hcp_metrics_bind_socket_dir", a directory
where a unix socket will be created with the name
`<namespace>_<proxy_id>.sock` to forward Envoy metrics.

If set, this will configure:
- In bootstrap configuration a local stats_sink and static cluster.
  These will forward metrics to a loopback listener sent over xDS.

- A dynamic listener listening at the socket path that the previously
  defined static cluster is sending metrics to.

- A dynamic cluster that will forward traffic received at this listener
  to the hcp-metrics-collector service.


Reasons for having a static cluster pointing at a dynamic listener:
- We want to secure the metrics stream using TLS, but the stats sink can
  only be defined in bootstrap config. With dynamic listeners/clusters
  we can use the proxy's leaf certificate issued by the Connect CA,
  which isn't available at bootstrap time.

- We want to intelligently route to the HCP collector. Configuring its
  addreess at bootstrap time limits our flexibility routing-wise. More
  on this below.

Reasons for defining the collector as an upstream in `proxycfg`:
- The HCP collector will be deployed as a mesh service.

- Certificate management is taken care of, as mentioned above.

- Service discovery and routing logic is automatically taken care of,
  meaning that no code changes are required in the xds package.

- Custom routing rules can be added for the collector using discovery
  chain config entries. Initially the collector is expected to be
  deployed to each admin partition, but in the future could be deployed
  centrally in the default partition. These config entries could even be
  managed by HCP itself.
2023-03-10 13:52:54 -07:00
Ronald 0558cbc5ee
Fix flakey tests related to ACL token updates (#16545)
* Fix flakey tests related to ACL token updates

* update all acl token update tests

* extra create_token function to its own thing
2023-03-06 23:14:06 +00:00
Chris S. Kim 39dc305143
Follow-up fixes to consul connect envoy command (#16530) 2023-03-06 10:32:06 -05:00
Ronald 7f6f12089f
Improve ux around ACL token to help users avoid overwriting node/service identities (#16506)
* Deprecate merge-node-identities and merge-service-identities flags

* added tests for node identities changes

* added changelog file and docs
2023-03-06 15:00:39 +00:00
Michael Wilkerson 87c5c41a25
modified unsupported envoy version error (#16518)
- When an envoy version is out of a supported range, we now return the envoy version being used as `major.minor.x` to indicate that it is the minor version at most that is incompatible
- When an envoy version is in the list of unsupported envoy versions we return back the envoy version in the error message as `major.minor.patch` as now the exact version matters.
2023-03-03 10:29:34 -08:00
Ronald c898a26ba0
Improve ux to help users avoid overwriting fields of ACL tokens, roles and policies (#16288)
* Deprecate merge-policies and add options add-policy-name/add-policy-id to improve CLI token update command

* deprecate merge-roles fields

* Fix potential flakey tests and update ux to remove 'completely' + typo fixes
2023-03-01 15:00:37 -05:00
cskh 452ec19a42
fix (cli): return error msg if acl policy not found (#16485)
* fix: return error msg if acl policy not found

* changelog

* add test
2023-03-01 19:50:03 +00:00
R.B. Boyer 7906f52b89
cli: ensure acl token read -self works (#16445)
Fixes a regression in #16044

The consul acl token read -self cli command should not require an -accessor-id because typically the persona invoking this would not already know the accessor id of their own token.
2023-02-28 10:58:29 -06:00
Kyle Havlovitz 18954ccd3e
Fix a couple inconsistencies in `operator usage instances` command (#16260) 2023-02-24 17:51:09 +00:00
Chris S. Kim 652b74dd37
Fix various flaky tests (#16396) 2023-02-23 14:52:18 -05:00
Dan Stough 29497be7e8
[OSS] security: update go to 1.20.1 (#16263)
* security: update go to 1.20.1
2023-02-17 15:04:12 -05:00
Nitya Dhanushkodi 9d255fe057
troubleshoot: fixes and updated messages (#16294) 2023-02-17 07:43:05 -08:00
Nitya Dhanushkodi d3591b687a
troubleshoot: make output have tables and colors (#16235)
Adds tables and colors using libraries used in consul-k8s. It doesn't add the full `terminal` UI package that consul-k8s uses since there is an existing UI in Consul that I didn't want to affect too much. So instead this adds to the existing UI.
2023-02-10 11:12:13 -08:00
Andrew Stucki 8073d1d16e
Add some fixes to allow for registering via consul connect envoy -gateway api (#16219)
* Add some fixes to allow for registering via consul connect envoy -gateway api

* Fix infinite recursion

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-02-09 20:28:04 +00:00