Commit Graph

1053 Commits

Author SHA1 Message Date
Paul Banks dbcf286d4c
Ooops remove the CA stuff from actual server defaults and make it test server only 2018-06-14 09:42:16 -07:00
Paul Banks 834ed1d25f
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes. 2018-06-14 09:42:16 -07:00
Paul Banks bdd30b191b
Comment cleanup 2018-06-14 09:42:16 -07:00
Paul Banks 5abf47472d
Verify trust domain on /authorize calls 2018-06-14 09:42:16 -07:00
Paul Banks 30d90b3be4
Generate CSR using real trust-domain 2018-06-14 09:42:16 -07:00
Paul Banks 5a1408f186
Add CSR signing verification of service ACL, trust domain and datacenter. 2018-06-14 09:42:16 -07:00
Paul Banks c808833a78
Return TrustDomain from CARoots RPC 2018-06-14 09:42:15 -07:00
Kyle Havlovitz d1265bc38b
Rename some of the CA structs/files 2018-06-14 09:42:15 -07:00
Kyle Havlovitz 1660f9ebab
Add more metadata to structs.CARoot 2018-06-14 09:42:15 -07:00
Kyle Havlovitz baf4db1c72
Use provider state table for a global serial index 2018-06-14 09:42:15 -07:00
Kyle Havlovitz 5998623c44
Add test for ca config http endpoint 2018-06-14 09:42:15 -07:00
Kyle Havlovitz c90b353eea
Move connect CA provider to separate package 2018-06-14 09:42:15 -07:00
Mitchell Hashimoto 4bb745a2d4
agent/cache: change uint8 to uint 2018-06-14 09:42:15 -07:00
Mitchell Hashimoto 6cf2e1ef1a
agent/cache: string through attempt rather than storing on the entry 2018-06-14 09:42:15 -07:00
Mitchell Hashimoto c42510e1ec
agent/cache: implement refresh backoff 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto 54a1662da8
agent/consul: change provider wait from goto to a loop 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto 749f81373f
agent/consul: check nil on getCAProvider result 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto c57405b323
agent/consul: retry reading provider a few times 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto b4f990bc6c
agent: verify local proxy tokens for CA leaf + tests 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto 8f7b5f93cd
agent: verify proxy token for ProxyConfig endpoint + tests 2018-06-14 09:42:14 -07:00
Mitchell Hashimoto 3a7aaa63bc
agent/proxy: pass proxy ID as an env var 2018-06-14 09:42:13 -07:00
Mitchell Hashimoto f69c8b85ef
agent/config: add managed proxy upstreams config to skip
agent/config will turn [{}] into {} (single element maps into a single
map) to work around HCL issues. These are resolved in HCL2 which I'm
sure Consul will switch to eventually.

This breaks the connect proxy configuration in service definition FILES
since we call this patch function. For now, let's just special-case skip
this. In the future we maybe Consul will adopt HCL2 and fix it, or we
can do something else if we want. This works and is tested.
2018-06-14 09:42:13 -07:00
Mitchell Hashimoto 662f38c625
agent/structs: validate service definitions, port required for proxy 2018-06-14 09:42:13 -07:00
Mitchell Hashimoto 498c63a6f1
agent/config: default connect enabled in dev mode
This enables `consul agent -dev` to begin using Connect features with
the built-in CA. I think this is expected behavior since you can imagine
that new users would want to try.

There is no real downside since we're just using the built-in CA.
2018-06-14 09:42:13 -07:00
Paul Banks 954d286d73
Make CSR work with jank domain 2018-06-14 09:42:13 -07:00
Mitchell Hashimoto 257d31e319
agent/proxy: delete pid file on Stop 2018-06-14 09:42:13 -07:00
Mitchell Hashimoto 1dfb4762f5
agent: increase timer for blocking cache endpoints 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto a89238a9d3
agent/proxy: address PR feedback 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto 7bb13246a8
agent: clarify why we Kill still 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto 147b066c67
agent: restore proxy snapshot but still Kill proxies 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto 1d24df3827
agent/proxy: check if process is alive in addition to Wait 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto 4301f7f1f5
agent: only set the proxy manager data dir if its set 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto e3be9f7a02
agent/proxy: improve comments on snapshotting 2018-06-14 09:42:12 -07:00
Mitchell Hashimoto eb31827fac
agent/proxy: implement periodic snapshotting in the manager 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto 64fc9e0218
agent/proxy: check if process is alive 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto a3a0bc7b13
agent/proxy: implement snapshotting for daemons 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto 9675ed626d
agent/proxy: manager configures the daemon pid path to write pids 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto 5e0f0ba178
agent/proxy: write pid file whenever the daemon process changes 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto 09093a1a1a
agent/proxy: change LogDir to DataDir to reuse for other things 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto e2133fd391
agent/proxy: make the logs test a bit more robust by waiting for file 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto d019d33bc6
agent/proxy: don't create the directory in newProxy 2018-06-14 09:42:11 -07:00
Mitchell Hashimoto 49bc7181a4
agent/proxy: send logs to the correct location for daemon proxies 2018-06-14 09:42:10 -07:00
Mitchell Hashimoto 515c47be7d
agent: add additional tests for defaulting in AddProxy 2018-06-14 09:42:10 -07:00
Mitchell Hashimoto 52665f7d23
agent: clean up defaulting of proxy configuration
This cleans up and unifies how proxy settings defaults are applied.
2018-06-14 09:42:10 -07:00
Mitchell Hashimoto ed14e9edf8
agent: resolve some conflicts and fix tests 2018-06-14 09:42:10 -07:00
Mitchell Hashimoto 657c09133a
agent/local: clarify the non-risk of a full buffer 2018-06-14 09:42:10 -07:00
Mitchell Hashimoto 31b09c0674
agent/local: remove outdated comment 2018-06-14 09:42:10 -07:00
Mitchell Hashimoto bae428326a
agent: use os.Executable 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 6a78ecea57
agent/proxy: local state event coalescing 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 16f529a13c
agent/proxy: implement force kill of unresponsive proxy process 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 4722e3ef76
agent: fix crash that could happen if proxy was nil on load 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 10fe87bd4a
agent/proxy: pull exit status extraction to constrained file 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 669268f85c
agent: start proxy manager 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 6884654c9d
agent/proxy: detect config change to stop/start proxies 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto 8ce3deac5d
agent/proxy: test removing proxies and stopping them 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto a2167a7fd1
agent/proxy: manager and basic tests, not great coverage yet coming soon 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto fae8dc8951
agent/local: add Notify mechanism for proxy changes 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto f64a002f68
agent: start/stop proxies 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto 93cdd3f206
agent/proxy: clean up usage, can't be restarted 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto 536f31571b
agent: change connect command paths to be slices, not strings
This matches other executable configuration and allows us to cleanly
separate executable from arguments without trying to emulate shell
parsing.
2018-06-14 09:42:08 -07:00
Mitchell Hashimoto 76c6849ffe
agent/local: store proxy on local state, wip, not working yet 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto 659ab7ee2d
agent/proxy: exponential backoff on restarts 2018-06-14 09:42:07 -07:00
Mitchell Hashimoto c2f50f1688
agent/proxy: Daemon works, tests cover it too 2018-06-14 09:42:07 -07:00
Mitchell Hashimoto c47ad68f25
wip 2018-06-14 09:42:07 -07:00
Paul Banks 02ab461dae
TLS watching integrated into Service with some basic tests.
There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.
2018-06-14 09:42:07 -07:00
Paul Banks dcd277de8a
Wire up agent leaf endpoint to cache framework to support blocking. 2018-06-14 09:42:07 -07:00
Kyle Havlovitz b28e11fdd3
Fill out connect CA rpc endpoint tests 2018-06-14 09:42:06 -07:00
Kyle Havlovitz 0e184f3f5b
Fix config tests 2018-06-14 09:42:06 -07:00
Kyle Havlovitz 7c0976208d
Add tests for the built in CA's state store table 2018-06-14 09:42:06 -07:00
Kyle Havlovitz 19b9399f2f
Add more tests for built-in provider 2018-06-14 09:42:06 -07:00
Kyle Havlovitz a29f3c6b96
Fix some inconsistencies around the CA provider code 2018-06-14 09:42:06 -07:00
Paul Banks 153808db7c
Don't allow connect watches in agent/cli yet 2018-06-14 09:42:06 -07:00
Paul Banks 072b2a79ca
Support legacy watch.HandlerFunc type for backward compat reduces impact of change 2018-06-14 09:42:05 -07:00
Paul Banks 6f566f750e
Basic `watch` support for connect proxy config and certificate endpoints.
- Includes some bug fixes for previous `api` work and `agent` that weren't tested
 - Needed somewhat pervasive changes to support hash based blocking - some TODOs left in our watch toolchain that will explicitly fail on hash-based watches.
 - Integration into `connect` is partially done here but still WIP
2018-06-14 09:42:05 -07:00
Kyle Havlovitz 2167713226
Add CA config to connect section of agent config 2018-06-14 09:42:05 -07:00
Kyle Havlovitz 02fef5f9a2
Move ConsulCAProviderConfig into structs package 2018-06-14 09:42:04 -07:00
Kyle Havlovitz 887cc98d7e
Simplify the CAProvider.Sign method 2018-06-14 09:42:04 -07:00
Kyle Havlovitz 44b30476cb
Simplify the CA provider interface by moving some logic out 2018-06-14 09:42:04 -07:00
Kyle Havlovitz aa10fb2f48
Clarify some comments and names around CA bootstrapping 2018-06-14 09:42:04 -07:00
Mitchell Hashimoto 5abd43a567
agent: resolve flaky test by checking cache hits increase, rather than
exact
2018-06-14 09:42:04 -07:00
Mitchell Hashimoto 73838c9afa
agent: use helper/retry instead of timing related tests 2018-06-14 09:42:04 -07:00
Mitchell Hashimoto dcb2671d10
agent/cache: address PR feedback, lots of typos 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto 07d878a157
agent/cache: address feedback, clarify comments 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto ad3928b6bd
agent/cache: don't every block on NotifyCh 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto 3f80a9f330
agent/cache: unit tests for ExpiryHeap, found a bug! 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto 1c31e34e5b
agent/cache: send the total entries count on eviction to go-metrics 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto ec559d77bd
agent/cache: make edge case with prev/next idx == 0 handled better 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto b319d06276
agent/cache: rework how expiry data is stored to be more efficient 2018-06-14 09:42:03 -07:00
Mitchell Hashimoto 449bbd817d
agent/cache: initial TTL work 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto 3c6acbda5d
agent/cache: send the RefreshTimeout into the backend fetch 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto 257fc34e51
agent/cache: on error, return from Get immediately, don't block forever 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto e9d58ca219
agent/cache: lots of comment/doc updates 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto a1f8cb9570
agent: augment /v1/connect/authorize to cache intentions 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto 56774f24d0
agent/cache-types: support intention match queries 2018-06-14 09:42:02 -07:00
Mitchell Hashimoto 109bb946e9
agent/cache: return the error as part of Get 2018-06-14 09:42:01 -07:00
Mitchell Hashimoto 6ecc2da7ff
agent/cache: integrate go-metrics so the cache is debuggable 2018-06-14 09:42:01 -07:00
Mitchell Hashimoto 3b6c46b7d7
agent/structs: DCSpecificRequest sets all the proper fields for
CacheInfo
2018-06-14 09:42:01 -07:00
Mitchell Hashimoto ccd7eeef1a
agent/cache-types/ca-leaf: proper result for timeout, race on setting CA 2018-06-14 09:42:01 -07:00
Mitchell Hashimoto 4509589427
agent/cache: support timeouts for cache reads and empty fetch results 2018-06-14 09:42:01 -07:00
Mitchell Hashimoto b0f70f17db
agent/cache-types: rename to separate root and leaf cache types 2018-06-14 09:42:01 -07:00
Mitchell Hashimoto e3b1c400e5
agent/cache-types: got basic CA leaf caching work, major problems still 2018-06-14 09:42:01 -07:00
Mitchell Hashimoto 9e44a319d3
agent: check cache hit count to verify CA root caching, background update 2018-06-14 09:42:00 -07:00
Mitchell Hashimoto 8bb4fd95a6
agent: initialize the cache and cache the CA roots 2018-06-14 09:42:00 -07:00
Mitchell Hashimoto 286217cbd8
agent/cache: partition by DC/ACL token 2018-06-14 09:42:00 -07:00
Mitchell Hashimoto 72c82a9b29
agent/cache: Reorganize some files, RequestInfo struct, prepare for partitioning 2018-06-14 09:42:00 -07:00
Mitchell Hashimoto ecc789ddb5
agent/cache: ConnectCA roots caching type 2018-06-14 09:42:00 -07:00
Mitchell Hashimoto c69df79e0c
agent/cache: blank cache key means to always fetch 2018-06-14 09:42:00 -07:00
Mitchell Hashimoto 8584e9262e
agent/cache: initial kind-of working cache 2018-06-14 09:42:00 -07:00
Kyle Havlovitz 43f13d5a0b
Add cross-signing mechanism to root rotation 2018-06-14 09:42:00 -07:00
Kyle Havlovitz bbfcb278e1
Add the root rotation mechanism to the CA config endpoint 2018-06-14 09:41:59 -07:00
Kyle Havlovitz a585a0ba10
Have the built in CA store its state in raft 2018-06-14 09:41:59 -07:00
Kyle Havlovitz 80eddb0bfb
Fix the testing endpoint's root set op 2018-06-14 09:41:59 -07:00
Kyle Havlovitz 9fefac745e
Update the CA config endpoint to enable GETs 2018-06-14 09:41:59 -07:00
Kyle Havlovitz fc9ef9741b
Hook the CA RPC endpoint into the provider interface 2018-06-14 09:41:59 -07:00
Kyle Havlovitz a40db26ffe
Add CA bootstrapping on establishing leadership 2018-06-14 09:41:59 -07:00
Kyle Havlovitz e26819ed9c
Add the bootstrap config for the CA 2018-06-14 09:41:59 -07:00
Kyle Havlovitz 4d0713d5bb
Add the CA provider interface and built-in provider 2018-06-14 09:41:58 -07:00
Kyle Havlovitz ebdda17a30
Add CA config set to fsm operations 2018-06-14 09:41:58 -07:00
Kyle Havlovitz f7ff16669f
Add the Connect CA config to the state store 2018-06-14 09:41:58 -07:00
Paul Banks a90f69faa4
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list. 2018-06-14 09:41:58 -07:00
Paul Banks 9d11cd9bf4
Fix various test failures and vet warnings.
Intention de-duplication in previously merged PR actualy failed some tests that were not caught be me or CI. I ran the test files for state changes but they happened not to trigger this case so I made sure they did first and then fixed. That fixed some upstream intention endpoint tests that I'd not run as part of testing the previous fix.
2018-06-14 09:41:58 -07:00
Paul Banks 8a4410b549
Refactor localBlockingQuery to use memdb.WatchSet. Much simpler and correct as a bonus! 2018-06-14 09:41:58 -07:00
Paul Banks aed5e5b03e
Super ugly hack to get TeamCity build to work for this PR without adding a vendor that is being added elsewhere and will conflict... 2018-06-14 09:41:58 -07:00
Paul Banks cbd8606651
Add X-Consul-ContentHash header; implement removing all proxies; add load/unload test. 2018-06-14 09:41:57 -07:00
Paul Banks 44afb5c699
Agent Connect Proxy config endpoint with hash-based blocking 2018-06-14 09:41:57 -07:00
Paul Banks c2266b134a
HTTP agent registration allows proxy to be defined. 2018-06-14 09:41:57 -07:00
Paul Banks 78e48fd547
Added connect proxy config and local agent state setup on boot. 2018-06-14 09:41:57 -07:00
Paul Banks 280382c25f
Add tests all the way up through the endpoints to ensure duplicate src/destination is supported and so ultimately deny/allow nesting works.
Also adds a sanity check test for `api.Agent().ConnectAuthorize()` and a fix for a trivial bug in it.
2018-06-14 09:41:57 -07:00
Paul Banks adc5589329
Allow duplicate source or destination, but enforce uniqueness across all four. 2018-06-14 09:41:57 -07:00
Paul Banks 51b1bc028d
Rework connect/proxy and command/connect/proxy. End to end demo working again 2018-06-14 09:41:57 -07:00
Paul Banks 2d6a2ce1e3
connect.Service based implementation after review feedback. 2018-06-14 09:41:56 -07:00
Mitchell Hashimoto 62b746c380
agent: rename authorize param ClientID to ClientCertURI 2018-06-14 09:41:56 -07:00
Mitchell Hashimoto 94e7a0a3c1
agent: add TODO for verification 2018-06-14 09:41:55 -07:00
Mitchell Hashimoto f983978fb8
acl: IntentionDefault => IntentionDefaultAllow 2018-06-14 09:41:55 -07:00
Mitchell Hashimoto b3584b6355
agent: ACL checks for authorize, default behavior 2018-06-14 09:41:55 -07:00
Mitchell Hashimoto 3e0e0a94a7
agent/structs: String format for Intention, used for logging 2018-06-14 09:41:55 -07:00
Mitchell Hashimoto 3f80808379
agent: bolster commenting for clearer understandability 2018-06-14 09:41:55 -07:00
Mitchell Hashimoto c6269cda37
agent: default deny on connect authorize endpoint 2018-06-14 09:41:54 -07:00
Mitchell Hashimoto 5364a8cd90
agent: /v1/agent/connect/authorize is functional, with tests 2018-06-14 09:41:54 -07:00
Mitchell Hashimoto 7af99667b6
agent/connect: Authorize for CertURI 2018-06-14 09:41:54 -07:00
Mitchell Hashimoto 68fa4a83b1
agent: get rid of method checks since they're done in the http layer 2018-06-14 09:41:54 -07:00
Paul Banks 894ee3c5b0
Add Connect agent, catalog and health endpoints to api Client 2018-06-14 09:41:54 -07:00
Mitchell Hashimoto 1985655dff
agent/consul/state: ensure exactly one active CA exists when setting 2018-06-14 09:41:54 -07:00
Mitchell Hashimoto 9d93c52098
agent/connect: support any values in the URL 2018-06-14 09:41:54 -07:00
Mitchell Hashimoto 8934f00d03
agent/connect: support SpiffeIDSigning 2018-06-14 09:41:53 -07:00
Mitchell Hashimoto da1bc48372
agent/connect: rename SpiffeID to CertURI 2018-06-14 09:41:53 -07:00
Mitchell Hashimoto b0315811b9
agent/connect: use proper keyusage fields for CA and leaf 2018-06-14 09:41:53 -07:00
Mitchell Hashimoto 434d8750ae
agent/connect: address PR feedback for the CA.go file 2018-06-14 09:41:53 -07:00
Mitchell Hashimoto e0562f1c21
agent: implement an always-200 authorize endpoint 2018-06-14 09:41:53 -07:00
Mitchell Hashimoto 2026cf3753
agent/consul: encode issued cert serial number as hex encoded 2018-06-14 09:41:53 -07:00