Commit Graph

2629 Commits

Author SHA1 Message Date
hashicorp-ci e11b76ce80 auto-updated agent/uiserver/bindata_assetfs.go from commit f4208b5fb 2020-10-27 14:56:48 +00:00
hashicorp-ci e5b54d9be3 auto-updated agent/uiserver/bindata_assetfs.go from commit 30da884d5 2020-10-27 14:31:16 +00:00
hashicorp-ci 707ef19716 auto-updated agent/uiserver/bindata_assetfs.go from commit ed6a2c150 2020-10-27 14:09:17 +00:00
hashicorp-ci 338002054f auto-updated agent/uiserver/bindata_assetfs.go from commit 827e53694 2020-10-27 13:36:44 +00:00
hashicorp-ci 5ae5a4b800 auto-updated agent/uiserver/bindata_assetfs.go from commit 52d7283cd 2020-10-26 19:56:11 +00:00
Paul Banks df1eec292f
UI metrics provider dc (#9001)
* Plumb Datacenter and Namespace to metrics provider in preparation for them being usable.

* Move metrics loader/status to a new component and show reason for being disabled.

* Remove stray console.log

* Rebuild AssetFS to resolve conflicts

* Yarn upgrade

* mend
2020-10-26 19:48:23 +00:00
hashicorp-ci fea7f184ea auto-updated agent/uiserver/bindata_assetfs.go from commit c4f027fa0 2020-10-26 16:55:52 +00:00
Daniel Nephin 68a0fd3f8c streaming: disable streaming when requesting connect events
Until the correct events are created for terminating gateways.
2020-10-26 11:55:49 -04:00
Daniel Nephin abd8cfcfe9 state: disable streaming connect topic 2020-10-26 11:49:47 -04:00
hashicorp-ci 03212abfca auto-updated agent/uiserver/bindata_assetfs.go from commit 948917c6b 2020-10-26 09:34:58 +00:00
Daniel Nephin 1dcbfd17c3 health: change the name of UseStreamingBackend config
Remove it from the cache section, and update the docs.
2020-10-23 17:47:01 -04:00
Daniel Nephin 6650146a74 health: use streaming, even when cache=1 is not set 2020-10-23 17:39:55 -04:00
R.B. Boyer 0a80e82f21
server: config entry replication now correctly uses namespaces in comparisons (#9024)
Previously config entries sharing a kind & name but in different
namespaces could occasionally cause "stuck states" in replication
because the namespace fields were ignored during the differential
comparison phase.

Example:

Two config entries written to the primary:

    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo

Under the covers these both get saved to memdb, so they are sorted by
all 3 components (kind,name,namespace) during natural iteration. This
means that before the replication code does it's own incomplete sort,
the underlying data IS sorted by namespace ascending (bar comes before
foo).

After one pass of replication the primary and secondary datacenters have
the same set of config entries present. If
"kind=A,name=web,namespace=bar" were to be deleted, then things get
weird. Before replication the two sides look like:

primary: [
    kind=A,name=web,namespace=foo
]
secondary: [
    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo
]

The differential comparison phase walks these two lists in sorted order
and first compares "kind=A,name=web,namespace=foo" vs
"kind=A,name=web,namespace=bar" and falsely determines they are the SAME
and are thus cause an update of "kind=A,name=web,namespace=foo". Then it
compares "<nothing>" with "kind=A,name=web,namespace=foo" and falsely
determines that the latter should be DELETED.

During reconciliation the deletes are processed before updates, and so
for a brief moment in the secondary "kind=A,name=web,namespace=foo" is
erroneously deleted and then immediately restored.

Unfortunately after this replication phase the final state is identical
to the initial state, so when it loops around again (rate limited) it
repeats the same set of operations indefinitely.
2020-10-23 13:41:54 -05:00
Daniel Nephin f9b2834171 state: convert the remaining functions to ReadTxn
Required also converting some of the transaction functions to WriteTxn
because TxnRO() called the same helper as TxnRW.

This change allows us to return a memdb.Txn for read-only txn instead of
wrapping them with state.txn.
2020-10-23 14:29:22 -04:00
Daniel Nephin 26387cdc0e
Merge pull request #8975 from hashicorp/dnephin/stream-close-on-unsub
stream: close the subscription on Unsubscribe
2020-10-23 12:58:12 -04:00
Freddy d23038f94f
Add HasExact to topology endpoint (#9010) 2020-10-23 10:45:41 -06:00
hashicorp-ci 8fa2b814ad auto-updated agent/uiserver/bindata_assetfs.go from commit eedee07e7 2020-10-23 16:32:06 +00:00
hashicorp-ci d1a6880699 auto-updated agent/uiserver/bindata_assetfs.go from commit 7559f64d0 2020-10-23 14:45:43 +00:00
hashicorp-ci e174a0c274 auto-updated agent/uiserver/bindata_assetfs.go from commit 2a8a80638 2020-10-23 13:50:57 +00:00
hashicorp-ci 0b7ee99f23 auto-updated agent/uiserver/bindata_assetfs.go from commit 45554e4e9 2020-10-23 08:32:18 +00:00
R.B. Boyer 2183842f0e
connect: add support for envoy 1.16.0, drop support for 1.12.x, and bump point releases as well (#8944)
Supported versions will be: "1.16.0", "1.15.2", "1.14.5", "1.13.6"
2020-10-22 13:46:19 -05:00
Daniel Nephin 56050a1dd1
Merge pull request #8973 from hashicorp/dnephin/resolver-balance
agent/router: refactor calculation of delay between rebalances.
2020-10-22 13:52:10 -04:00
Daniel Nephin fb8b68a6ec stream: close the subscription on Unsubscribe 2020-10-22 13:39:27 -04:00
Daniel Nephin 1d03a7a8a4
Merge pull request #8924 from ShimmerGlass/fix-sidecar-deregister-after-restart
Fix: service LocallyRegisteredAsSidecar property is not persisted
2020-10-22 13:26:55 -04:00
Daniel Nephin 193c89429d
Merge pull request #9004 from hashicorp/dnephin/dns-small-cleanup-1
dns: remove goto INVALID and a naked return
2020-10-21 18:41:11 -04:00
Daniel Nephin 19cffcb1f3
Merge pull request #8970 from hashicorp/dnephin/resolver-balance-test
agent/router: improve  the test for refreshServerRebalanceTimer
2020-10-21 18:40:32 -04:00
Daniel Nephin 1e1f963064
Merge pull request #8771 from amenzhinsky/fix-grpc-use-tls-mapping
Fix GRPCUseTLS flag HTTP API mapping
2020-10-21 18:37:11 -04:00
Daniel Nephin aa06ae91ce dns: remove goto INVALID and a naked return
This commit is one small step toward modernizing and making the DNSServer readable.
2020-10-21 15:16:03 -04:00
Daniel Nephin 09d62f1df0 lib/ttlcache: unexport key and additional godoc 2020-10-20 19:16:03 -04:00
Daniel Nephin 2601998766 lib/ttlcache: add a constant for NotIndexed 2020-10-20 19:10:20 -04:00
Daniel Nephin 0beaced90f cache: fix a bug with Prepopulate
Prepopulate was setting entry.Expiry.HeapIndex to 0. Previously this would result in a call to heap.Fix(0)
which wasn't correct, but was also not really a problem because at worse it would re-notify.

With the recent change to extract cachettl it was changed to call Update(idx), which would have updated
the wrong entry.

A previous commit removed the setting of entry.Expiry so that the HeapIndex would be reported
as -1, and this commit adds a test and handles the -1 heap index.
2020-10-20 19:10:20 -04:00
Daniel Nephin 9d5b738cdb lib/ttlcache: extract package from agent/cache 2020-10-20 19:10:20 -04:00
Daniel Nephin 909b8e674e cache: export ExpiryHeap
and hide internal methods on an unexported type, so that when it is extrated those methods are not exported.
2020-10-20 19:10:20 -04:00
Daniel Nephin d3742a1d0e cache: Refactor heap.notify to make it more explicit.
And remove duplicate notifications.

Instead of performing the check in the heap implementation, check the
index in the higher level interface (Add,Remove,Update) and notify if one
of the relevant indexes is 0.
2020-10-20 19:10:20 -04:00
Daniel Nephin a96646c562 cache: Move more of the expiryLoop into the Heap 2020-10-20 19:10:20 -04:00
Daniel Nephin b6f24c6554 cache: extract cache eviction heap
Start creating an interface that doesn't require using heap and hides more of the
entry internals.
2020-10-20 19:10:19 -04:00
Daniel Nephin 312a3bb9b1 streaming: apply filter to a single item
Instead of the whole map. This should save a lot of time performing reflecting on a large map.
The filter does not change, so there is no reason to re-apply it to older entries.
2020-10-19 18:24:02 -04:00
hashicorp-ci 5de8c073cc auto-updated agent/uiserver/bindata_assetfs.go from commit 2773daedc 2020-10-19 16:35:48 +00:00
hashicorp-ci 054483d5cf auto-updated agent/uiserver/bindata_assetfs.go from commit b24c2227d 2020-10-19 16:19:40 +00:00
hashicorp-ci f2d511b9e5 auto-updated agent/uiserver/bindata_assetfs.go from commit 02dcd422f 2020-10-19 14:50:52 +00:00
hashicorp-ci ef43c1a09e auto-updated agent/uiserver/bindata_assetfs.go from commit bf8bb7415 2020-10-19 14:25:54 +00:00
Daniel Nephin a3f922249e agent/router: refactor calculation of delay between rebalances.
This change attempts to make the delay logic more obvious by:

* remove indirection, inline a bunch of function calls
* move all the code and constants next to each other
* replace the two constant values with a single value
* reword the comments.
2020-10-15 15:59:36 -04:00
Daniel Nephin da1e45745c agent/router: Add bounds test cases 2020-10-15 14:43:29 -04:00
Daniel Nephin 9bbfc53685 router: organize the test by number of servers
And adddd some additional cases to show where the minimum value stops being used
2020-10-15 13:53:37 -04:00
Daniel Nephin 2ac2503604 router: make refreshServerRebalanceTimer test a lot more strict 2020-10-15 12:05:07 -04:00
hashicorp-ci d15d2b5b2c auto-updated agent/uiserver/bindata_assetfs.go from commit d08e13e5b 2020-10-15 14:49:01 +00:00
hashicorp-ci e5ace85562 auto-updated agent/uiserver/bindata_assetfs.go from commit 5fd79ba40 2020-10-14 08:07:29 +00:00
Mathilde Gilles a1ec792acc Fix: service LocallyRegisteredAsSidecar property is not persisted
When a service is deregistered, we check whever matching services were
registered as sidecar along with it and deregister them as well.
To determine if a service is indeed a sidecar we check the
structs.ServiceNode.LocallyRegisteredAsSidecar property. However, to
avoid interal API leakage, it is excluded from JSON serialization,
meaning it is not persisted to disk either.
When the agent is restarted, this property lost and sidecars are no
longer deregistered along with their parent service.
To fix this, we now specifically save this property in the persisted
service file.
2020-10-13 19:38:58 +02:00
hashicorp-ci 646e1f369c auto-updated agent/uiserver/bindata_assetfs.go from commit 7afcd7755 2020-10-13 16:43:31 +00:00
Mike Morris 528b0f4621 chore: regenerate bindata_assetfs 2020-10-12 15:44:33 -04:00
Mike Morris f2443e0745 Merge branch 'master' into release/1.9.0-beta1 2020-10-12 15:32:58 -04:00
hashicorp-ci d2f62d64cb
update bindata_assetfs.go 2020-10-12 18:30:30 +00:00
hashicorp-ci 72988bae34 auto-updated agent/uiserver/bindata_assetfs.go from commit 868cd47a6 2020-10-12 16:28:44 +00:00
Pierre Souchay 54f9f247f8
Consul Service meta wrongly computes and exposes non_voter meta (#8731)
* Consul Service meta wrongly computes and exposes non_voter meta

In Serf Tags, entreprise members being non-voters use the tag
`nonvoter=1`, not `non_voter = false`, so non-voters in members
were wrongly displayed as voter.

Demonstration:

```
consul members -detailed|grep voter
consul20-hk5 10.200.100.110:8301   alive   acls=1,build=1.8.4+ent,dc=hk5,expect=3,ft_fs=1,ft_ns=1,id=xxxxxxxx-5629-08f2-3a79-10a1ab3849d5,nonvoter=1,port=8300,raft_vsn=3,role=consul,segment=<all>,use_tls=1,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302
```

* Added changelog

* Added changelog entry
2020-10-09 17:18:24 -04:00
hashicorp-ci f58806fbab auto-updated agent/uiserver/bindata_assetfs.go from commit 851705e93 2020-10-09 21:01:02 +00:00
hashicorp-ci 89977d1467 auto-updated agent/uiserver/bindata_assetfs.go from commit 27048a061 2020-10-09 20:35:59 +00:00
Paul Banks 40695d5919
Add metrics rendering to the new topology view. (#8858)
* Remove unused StatsCard component

* Create Card and Stats contextual components with styling

* Send endpoint, item, and protocol to Stats as props

* WIP basic plumbing for metrics in Ember

* WIP metrics data source now works for different protocols and produces reasonable mock responses

* WIP sparkline component

* Mostly working metrics and graphs in topology

* Fix date in tooltip to actually be correct

* Clean up console.log

* Add loading frame and create a style sheet for Stats

* Various polish fixes:

 - Loading state for graph
 - Added fake latency cookie value to test loading
 - If metrics provider has no series/stats for the service show something that doesn't look broken
 - Graph hover works right to the edge now
 - Stats boxes now wrap so they are either shown or not as will fit not cut off
 - Graph resizes when browser window size changes
 - Some tweaks to number formats and stat metrics to make them more compact/useful

* Thread Protocol through topology model correctly

* Rebuild assetfs

* Fix failing tests and remove stats-card now it's changed and become different

* Fix merge conflict

* Update api doublt

* more merge fixes

* Add data-permission and id attr to Card

* Run JS linter

* Move things around so the tests run with everything available

* Get tests passing:

1. Remove fakeLatency setTimeout (will be replaced with CONSUL_LATENCY
in mocks)
2. Make sure any event handlers are removed

* Make sure the Consul/scripts are available before the app

* Make sure interval gets set if there is no cookie value

* Upgrade mocks so we can use CONSUL_LATENCY

* Fix handling of no series values from Prometheus

* Update assetfs and fix a comment

* Rebase and rebuild assetfs; fix tcp metric series units to be bits not bytes

* Rebuild assetfs

* Hide stats when provider is not configured

Co-authored-by: kenia <keniavalladarez@gmail.com>
Co-authored-by: John Cowen <jcowen@hashicorp.com>
2020-10-09 21:31:15 +01:00
s-christoff a62705101f
Enhance the output of consul snapshot inspect (#8787) 2020-10-09 14:57:29 -05:00
Kyle Havlovitz 707f4a8d26 Stop intermediate renew routine on leader stop 2020-10-09 12:30:57 -07:00
Kyle Havlovitz 926a393a5c
Merge pull request #8784 from hashicorp/renew-intermediate-primary
connect: Enable renewing the intermediate cert in the primary DC
2020-10-09 12:18:59 -07:00
hashicorp-ci 2facfdf9a1 auto-updated agent/uiserver/bindata_assetfs.go from commit 766b28b8e 2020-10-09 18:44:15 +00:00
Daniel Nephin dd0e8d42c4
Merge pull request #8825 from hashicorp/streaming/add-config
streaming: add config and docs
2020-10-09 14:33:58 -04:00
Daniel Nephin 195511140f config: add field for enabling streaming in the client
agent: register the new streaming cache-type
2020-10-09 14:11:34 -04:00
Daniel Nephin c1b832c2e6
Merge pull request #8893 from hashicorp/dnephin/add-steps-to-subscribe-tests
subscribe: add steps to long test cases, and add new cases for converting Events
2020-10-09 13:54:59 -04:00
Daniel Nephin 1d41d78338
Merge pull request #8896 from hashicorp/dnephin/go-test-race-more-pkgs
ci: go test -race more packages
2020-10-09 13:48:56 -04:00
Chris Piraino 4f77f87065
Emit service usage metrics with correct labeling strategy (#8856)
Previously, we would emit service usage metrics both with and without a
namespace label attached. This is problematic in the case when you want
to aggregate metrics together, i.e. "sum(consul.state.services)". This
would cause services to be counted twice in that aggregate, once via the
metric emitted with a namespace label, and once in the metric emited
without any namespace label.
2020-10-09 11:01:45 -05:00
Kyle Havlovitz 50543d678e Fix intermediate refresh test comments 2020-10-09 08:53:33 -07:00
Matt Keeler 44017f819e
Remove oss/ent divergence of runtime_test.go (#8903) 2020-10-09 10:45:11 -04:00
Matt Keeler 891d05fada
Add capability for the v1/connect/ca/roots endpoint to return a PEM encoded certificate chain (#8774)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-09 10:43:33 -04:00
R.B. Boyer d2f09ca306
upstream some differences from enterprise (#8902) 2020-10-09 09:42:53 -05:00
hashicorp-ci b55c4d90b2 auto-updated agent/uiserver/bindata_assetfs.go from commit b6f686fec 2020-10-09 13:37:35 +00:00
R.B. Boyer b4bf092db3
uiserver: upstream refactors done elsewhere (#8891) 2020-10-09 08:32:39 -05:00
Kyle Havlovitz 968fd8660d Update CI for leader renew CA test using Vault 2020-10-09 05:48:15 -07:00
Kyle Havlovitz 62270c3f9a
Merge branch 'master' into renew-intermediate-primary 2020-10-09 04:40:34 -07:00
Kyle Havlovitz b78f618beb connect: Check for expired root cert when cross-signing 2020-10-09 04:35:56 -07:00
Paul Banks fb87e218ab
Actually proxy the query string too 2020-10-09 12:27:24 +01:00
Daniel Nephin 6a8eac77af cache-types: skip tests with races 2020-10-08 20:15:13 -04:00
Daniel Nephin 022744699f grpc: fix data rate in stats handler test 2020-10-08 19:43:49 -04:00
Freddy 89d52f41c4
Add protocol to the topology endpoint response (#8868) 2020-10-08 17:31:54 -06:00
Daniel Nephin f06fd96d3a subscribe: add test cases for newEventFromStreamEvent 2020-10-08 18:48:17 -04:00
Kit Patella 40b9769b1f
Merge pull request #8877 from hashicorp/mkcp/telemetry/consul.api.http
Add flag for disabling 1.9 metrics backwards compatibility and warnings when set to default
2020-10-08 13:22:37 -07:00
R.B. Boyer 69af49441a
agent: allow the /v1/connect/intentions/match endpoint to use the agent cache (#8875)
This is the recommended proxy integration API for listing intentions
which should not require an active connection to the servers to resolve
after the initial cache filling.
2020-10-08 14:51:53 -05:00
Daniel Nephin ea95908f63 subscribe: Add steps to rpc/subscribe tests
To make them easier to follow
2020-10-08 15:38:01 -04:00
hashicorp-ci 143bfb7462 auto-updated agent/uiserver/bindata_assetfs.go from commit 13dfde75a 2020-10-08 19:36:48 +00:00
Matt Keeler 141eb60f06
Add per-agent reconnect timeouts (#8781)
This allows for client agent to be run in a more stateless manner where they may be abruptly terminated and not expected to come back. If advertising a per-agent reconnect timeout using the advertise_reconnect_timeout configuration when that agent leaves, other agents will wait only that amount of time for the agent to come back before reaping it.

This has the advantageous side effect of causing servers to deregister the node/services/checks for that agent sooner than if the global reconnect_timeout was used.
2020-10-08 15:02:19 -04:00
Paul Banks d9818ed1e0
Add /v1/internal/ui/metrics-proxy API endpoint that proxies to a configured metrics provider backend. 2020-10-08 17:32:29 +01:00
Paul Banks 42d5e75dc5
Fix merge conflicts 2020-10-08 17:31:36 +01:00
Daniel Nephin 05df7b18a9 config: add field for enabling streaming RPC endpoint 2020-10-08 12:11:20 -04:00
Daniel Nephin 3483e2fb89 streaming: Use a shorter LastGetTTL for the cache 2020-10-08 12:11:20 -04:00
hashicorp-ci fdfa9c610c auto-updated agent/uiserver/bindata_assetfs.go from commit b373456c7 2020-10-08 15:56:05 +00:00
Freddy de4af766f3
Support ingress gateways in mesh viz endpoint (#8864)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2020-10-08 09:47:09 -06:00
hashicorp-ci 75847b0f11 auto-updated agent/uiserver/bindata_assetfs.go from commit ef7b1f8a4 2020-10-08 15:41:40 +00:00
hashicorp-ci 448febe0be auto-updated agent/uiserver/bindata_assetfs.go from commit d849f025c 2020-10-08 15:08:16 +00:00
Daniel Nephin 0b3f438703
Merge pull request #8841 from hashicorp/streaming/materialize-view-fix-service-unique-id
streaming: Use an ID that includes namespace to store services in the materialized view
2020-10-07 21:28:53 -04:00
Daniel Nephin a94fe054f0
Merge pull request #8809 from hashicorp/streaming/materialize-view
Add StreamingHealthServices cache-type
2020-10-07 21:26:38 -04:00
Daniel Nephin e0236b5a9f
Merge pull request #8818 from hashicorp/streaming/add-subscribe-service-batch-events
stream: handle batch events as a special case of Event
2020-10-07 21:25:32 -04:00
Daniel Nephin 783627aeef
Merge pull request #8768 from hashicorp/streaming/add-subscribe-service
subscribe: add subscribe service for streaming change events
2020-10-07 21:24:03 -04:00
Freddy 7d1f50d2e6
Return intention info in svc topology endpoint (#8853) 2020-10-07 18:35:34 -06:00
Kit Patella 328036dd37 add config flag to disable 1.9 metrics backwards compatibility. Add warnings on start and reload on default value 2020-10-07 17:12:52 -07:00
Daniel Nephin eb6f2a8d72 structs: add CheckServiceNode.CanRead
And use it from the subscribe endpoint.
2020-10-07 18:15:13 -04:00