Commit Graph

12708 Commits

Author SHA1 Message Date
Daniel Nephin 05df7b18a9 config: add field for enabling streaming RPC endpoint 2020-10-08 12:11:20 -04:00
Daniel Nephin 3483e2fb89 streaming: Use a shorter LastGetTTL for the cache 2020-10-08 12:11:20 -04:00
Daniel Nephin 0b3f438703
Merge pull request #8841 from hashicorp/streaming/materialize-view-fix-service-unique-id
streaming: Use an ID that includes namespace to store services in the materialized view
2020-10-07 21:28:53 -04:00
Daniel Nephin a94fe054f0
Merge pull request #8809 from hashicorp/streaming/materialize-view
Add StreamingHealthServices cache-type
2020-10-07 21:26:38 -04:00
Daniel Nephin e0236b5a9f
Merge pull request #8818 from hashicorp/streaming/add-subscribe-service-batch-events
stream: handle batch events as a special case of Event
2020-10-07 21:25:32 -04:00
Daniel Nephin 783627aeef
Merge pull request #8768 from hashicorp/streaming/add-subscribe-service
subscribe: add subscribe service for streaming change events
2020-10-07 21:24:03 -04:00
Freddy 7d1f50d2e6
Return intention info in svc topology endpoint (#8853) 2020-10-07 18:35:34 -06:00
Daniel Nephin eb6f2a8d72 structs: add CheckServiceNode.CanRead
And use it from the subscribe endpoint.
2020-10-07 18:15:13 -04:00
Mike Morris 80aef1ca1d
test(vault): generate test coverage and upload to CodeCov (#8870) 2020-10-07 16:32:25 -04:00
R.B. Boyer eb23a7d5dd
add testrpc.WaitForServiceIntentions to help unflake tests that manipulate intentions (#8867) 2020-10-07 14:20:25 -05:00
R.B. Boyer 140c88a9f4
api: unflake some intention-related api tests (#8857) 2020-10-07 13:32:53 -05:00
Jono Sosulska b596538397
Turns PR labeler to only trigger on create (#8838) 2020-10-07 13:45:17 -04:00
Mike Morris 1d4f3166fb
chore(deps): update gopsutil to v2.20.9 (#8843)
* core(deps): bump golang.org/x/sys

To resolve /go/pkg/mod/github.com/shirou/gopsutil@v2.20.9+incompatible/host/host_bsd.go:20:13: undefined: unix.SysctlTimeval

* chore(deps): make update-vendor
2020-10-07 12:57:18 -04:00
R.B. Boyer 7d18407e6a
command: remove conditional envoy bootstrap generation for versions <=1.10.0 since those are not supported (#8855) 2020-10-07 10:53:23 -05:00
R.B. Boyer d257b49601 add missing changelog entry for #8839 2020-10-07 10:22:40 -05:00
hashicorp-ci 7af06157eb auto-updated agent/uiserver/bindata_assetfs.go from commit 0d0de4309 2020-10-07 14:13:29 +00:00
John Cowen 8b0ed0ff01
ui: Change query param name for service instance listing from nspace to ns (#8852) 2020-10-07 15:08:13 +01:00
hashicorp-ci 50efe14f3e auto-updated agent/uiserver/bindata_assetfs.go from commit a61b6c53d 2020-10-07 13:50:39 +00:00
John Cowen b33a01f8dd
ui: Move node listing copy button to the detail rather than the title (#8850) 2020-10-07 14:45:52 +01:00
hashicorp-ci a7a30d8f5d auto-updated agent/uiserver/bindata_assetfs.go from commit 222555c4c 2020-10-07 08:09:54 +00:00
John Cowen ccf0e257ea
ui: Fix up Service filtering by whether a Service is in the mesh or not in the mesh (#8836)
* Add MeshEnabled, InMesh properties and add Proxy back in

* Change query param to in-mesh/not-in-mesh

* Use new computed properties
2020-10-07 09:04:55 +01:00
R.B. Boyer 35c4efd220
connect: support defining intentions using layer 7 criteria (#8839)
Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).
2020-10-06 17:09:13 -05:00
Pierre Souchay a12056f57c Added changelog for merged PR #8221 2020-10-06 17:15:33 -04:00
Daniel Nephin dbfa6530f1 streaming: store services with a unique ID that includes namespace 2020-10-06 16:54:56 -04:00
Daniel Nephin 9ace9a5d2e proto: add Namespace to EnterpriseMeta
By adding Namespace to the OSS EnterpriseMeta we reduce the differences between OSS and Enterprise
which simplifies the code quite a bit.
2020-10-06 16:42:09 -04:00
Mike Morris 5d273bd910 Update CHANGELOG.md 2020-10-06 16:27:38 -04:00
Alvin Huang b2a66b7426
add ui test-node (#8820) 2020-10-06 15:58:12 -04:00
R.B. Boyer d6dce2332a
connect: intentions are now managed as a new config entry kind "service-intentions" (#8834)
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
Daniel Nephin 83401194ab streaming: improve godoc for cache-type
And fix a bug where any error that implemented the temporary interface was considered
a temporary error, even when the method would return false.
2020-10-06 13:52:02 -04:00
Daniel Nephin bda19cb71e submatview: remove notifyUpdateLocked from reset
This call appears to only be necessary because reset() was called from
NewMaterializer.

This commit has the constructor set a default value for updateCh, and
removes both the call to reset() from New(), and the call to
notifyUpdateLocked() from reset().

This should ensure that we do not notify the Fetch() call before we have new
values to report.
2020-10-06 13:22:02 -04:00
Daniel Nephin e5813bd9d6 submatview: remove method receiver from handlers 2020-10-06 13:22:02 -04:00
Daniel Nephin f857aef4a8 submatview: add a test for handling of NewSnapshotToFollow
Also add some godoc
Rename some vars and functions
Fix a data race in the new cache test for entry closing.
2020-10-06 13:22:02 -04:00
Daniel Nephin 58cf09247b submatview: refactor Materializer
Refactor of Materializer.Run
Use handlers to manage state in Materializer
Rename Materializer receiver
rename m.l to m.lock, and flip some conditionals to remove the negative.
Improve godoc, rename Deps, move resetErr, and pass err into notifyUpdate
Update for NewSnapshotToFollow events
Refactor to move context cancel out of Materializer
2020-10-06 13:22:02 -04:00
Daniel Nephin e8c7881196 submatview: Move the 'use materialize from result.State' logic
No need to do all this other work if we have one already.

This logic moved closer to this call site 3 times during the process
of refactoring.
2020-10-06 13:22:02 -04:00
Daniel Nephin 3bb252888b submatview: Move Materializer to submatview package 2020-10-06 13:22:02 -04:00
Daniel Nephin d24e243f70 submatview: Refactor MaterializeView
Replace InitFilter with Reset.
Removes the need to store a fatalErr and the cache-type, and removes the need to recreate the filter
each time.
Pass dependencies into MaterializedView.
Remove context from MaterializedView.
Rename state to view.
Rename MaterialziedView to Materialzier.
Rename to NewMaterializer
Pass in retry.Waiter
2020-10-06 13:22:02 -04:00
Daniel Nephin 50846a96ff cache-types: Update Streaming health cache-type
To use latest protobuf types
2020-10-06 13:22:02 -04:00
Daniel Nephin e5d37bdf23 agent/cache: Add cache-type and materialized view for streaming health
Extracted from d97412ce4c399a35b41bbdae2716f0e32dce80bf

Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 13:21:57 -04:00
Daniel Nephin ad29cf4f94 stream: Return a single event from a subscription.Next
Handle batch events as a single event
2020-10-06 13:18:20 -04:00
Jono Sosulska a53730362c
Remove label trigger on edit (#8832) 2020-10-06 13:16:14 -04:00
Daniel Nephin 3183b9ebb3 subscribe: update to use NewSnapshotToFollow event 2020-10-06 12:49:35 -04:00
Daniel Nephin fa115c6249 Move agent/subscribe -> agent/rpc/subscribe 2020-10-06 12:49:35 -04:00
Daniel Nephin 011109a6f6 subscirbe: extract streamID and logging from Subscribe
By extracting all of the tracing logic the core logic of the Subscribe
endpoint is much easier to read.
2020-10-06 12:49:35 -04:00
Daniel Nephin 4c4441997a subscribe: add integration test for acl token updates 2020-10-06 12:49:35 -04:00
Daniel Nephin 3deccdde56 subscribe: add integration test for filtering events by acl 2020-10-06 12:49:35 -04:00
Daniel Nephin cb6ce4ab0c subscribe: Add an integration test for forward to DC 2020-10-06 12:49:35 -04:00
Daniel Nephin f5f4a3d803 subscribe: add first integration test for Server 2020-10-06 12:49:35 -04:00
Daniel Nephin 371ec2d70a subscribe: add a stateless subscribe service for the gRPC server
With a Backend that provides access to the necessary dependencies.
2020-10-06 12:49:35 -04:00
Daniel Nephin 2f6c98ee0e subscribe: add commented out test cases
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00
Daniel Nephin f73432be66 subscribe: add initial impl from streaming-rpc-final branch
Co-authored-by: Paul Banks <banks@banksco.de>
2020-10-06 12:49:35 -04:00