2df422e1e5
Disable TestAgent proxy execution properly
2018-06-25 12:25:38 -07:00
81bd1b43a3
Fix hot loop in cache for RPC returning zero index.
2018-06-25 12:25:37 -07:00
3d51c2aeac
Get agent cache tests passing without global hit count (which is racy).
...
Few other fixes in here just to get a clean run locally - they are all also fixed in other PRs but shouldn't conflict.
This should be robust to timing between goroutines now.
2018-06-25 12:25:37 -07:00
3efa77b912
Update UI for beta3
2018-06-25 12:25:16 -07:00
29af8fb5ab
agent/cache: always schedule the refresh
2018-06-25 12:25:14 -07:00
63047f9434
agent: clarify comment
2018-06-25 12:25:14 -07:00
1f3d2701f3
agent: add additional assertion to test
2018-06-25 12:25:13 -07:00
8f26c9c3b9
More test tweaks
2018-06-25 12:25:13 -07:00
d6b13463ed
Fix misc test failures (some from other PRs)
2018-06-25 12:25:13 -07:00
1283373a64
Only set precedence on write path
2018-06-25 12:25:13 -07:00
22b95283e9
Fix some tests failures caused by the sorting change and some cuased by previous UpdatePrecedence() change
2018-06-25 12:25:13 -07:00
e2938138f6
Sort intention list by precedence
2018-06-25 12:25:13 -07:00
efa2bdb88b
agent: intention update/delete responess match ACL/KV behavior
2018-06-25 12:25:12 -07:00
93037b0607
agent/structs: JSON marshal the configuration for a managed proxy
2018-06-25 12:25:12 -07:00
8cb57b9316
agent: disallow deregistering a managed proxy directly
2018-06-25 12:25:12 -07:00
47c0e0dde6
agent: deregister service deregisters the proxy along with it
2018-06-25 12:25:12 -07:00
0d457a3e71
agent: RemoveProxy also removes the proxy service
2018-06-25 12:25:12 -07:00
8c349a2b24
Fix broken tests from PR merge related to proxy secure defaults
2018-06-25 12:25:12 -07:00
a3ef9c2308
agent/cache: always fetch with minimum index of 1 at least
2018-06-25 12:25:12 -07:00
164da57afb
agent/proxy: remove debug println
2018-06-25 12:25:11 -07:00
f551413714
agent: disallow API registration with managed proxy if not enabled
2018-06-25 12:25:11 -07:00
a8ec3064f5
agent/config: AllowManagedAPIRegistration
2018-06-25 12:25:11 -07:00
c30affa4b6
agent/proxy: AllowRoot to disable executing managed proxies when root
2018-06-25 12:25:11 -07:00
be83efe61e
agent/proxy: set the proper arguments so we only run the helper process
2018-06-25 12:25:11 -07:00
a7690301f9
agent/config: add AllowManagedRoot
2018-06-25 12:25:11 -07:00
549dc22944
connect: fix two CA tests that were broken in a previous PR ( #60 )
2018-06-25 12:25:10 -07:00
3433020fa6
Fix roots race with CA setup hammering bug and defensive nil check hit during obscure upgrade scenario
2018-06-25 12:25:10 -07:00
1ce8361aa2
agent: format all CA config fields
2018-06-25 12:25:09 -07:00
a242e5b130
agent: update accepted CA config fields and defaults
2018-06-25 12:25:09 -07:00
7846206753
agent/proxy: fix build on Windows
2018-06-25 12:24:18 -07:00
6c77f7883e
Misc comment cleanups
2018-06-25 12:24:16 -07:00
d0674cdd7a
Warn about killing proxies in dev mode
2018-06-25 12:24:16 -07:00
4ebddd6adb
agent/consul: set precedence value on struct itself
2018-06-25 12:24:16 -07:00
61c7e33a22
agent/config: move ports to `ports` structure, update docs
2018-06-25 12:24:15 -07:00
d140612350
Fixs a few issues that stopped this working in real life but not caught by tests:
...
- Dev mode assumed no persistence of services although proxy state is persisted which caused proxies to be killed on startup as their services were no longer registered. Fixed.
- Didn't snapshot the ProxyID which meant that proxies were adopted OK from snapshot but failed to restart if they died since there was no proxyID in the ENV on restart
- Dev mode with no persistence just kills all proxies on shutdown since it can't recover them later
- Naming things
2018-06-25 12:24:14 -07:00
3df45ac7f1
Don't kill proxies on agent shutdown; backport manager close fix
2018-06-25 12:24:13 -07:00
877390cd28
Test for adopted process Stop race and fix
2018-06-25 12:24:13 -07:00
e016f37ae7
agent: accept connect param for execute
2018-06-25 12:24:12 -07:00
52c10d2208
agent/consul: support a Connect option on prepared query request
2018-06-25 12:24:12 -07:00
e8c899b1b8
agent/consul: prepared query supports "Connect" field
2018-06-25 12:24:11 -07:00
e3562e39cc
agent: intention create returns 500 for bad body
2018-06-25 12:24:10 -07:00
ad382d7351
agent: switch ConnectNative to an embedded struct
2018-06-25 12:24:10 -07:00
1e5a2561b6
Make tests pass and clean proxy persistence. No detached child changes yet.
...
This is a good state for persistence stuff to re-start the detached child work that got mixed up last time.
2018-06-25 12:24:10 -07:00
3bac52480e
Abandon daemonize for simpler solution (preserving history):
...
Reverts:
- bdb274852ae469c89092d6050697c0ff97178465
- 2c689179c4f61c11f0016214c0fc127a0b813bfe
- d62e25c4a7ab753914b6baccd66f88ffd10949a3
- c727ffbcc98e3e0bf41e1a7bdd40169bd2d22191
- 31b4d18933fd0acbe157e28d03ad59c2abf9a1fb
- 85c3f8df3eabc00f490cd392213c3b928a85aa44
2018-06-25 12:24:10 -07:00
9ef748157a
WIP
2018-06-25 12:24:09 -07:00
9cea27c66e
Sanity check that we are never trying to self-exec a test binary. Add daemonize bypass for TestAgent so that we don't have to jump through ridiculous self-execution hooks for every package that might possibly invoke a managed proxy
2018-06-25 12:24:09 -07:00
56f5924f3e
agent/proxy: Manager.Close also has to stop all proxy watchers
2018-06-25 12:24:09 -07:00
18e64dafbc
Fix import tooling fail
2018-06-25 12:24:09 -07:00
e1aca748c4
Make daemoinze an option on test binary without hacks. Misc fixes for racey or broken tests. Still failing on several though.
2018-06-25 12:24:09 -07:00
c97db00903
Run daemon processes as a detached child.
...
This turns out to have a lot more subtelty than we accounted for. The test suite is especially prone to races now we can only poll the child and many extra levels of indirectoin are needed to correctly run daemon process without it becoming a Zombie.
I ran this test suite in a loop with parallel enabled to verify for races (-race doesn't find any as they are logical inter-process ones not actual data races). I made it through ~50 runs before hitting an error due to timing which is much better than before. I want to go back and see if we can do better though. Just getting this up.
2018-06-25 12:24:08 -07:00
3a00574a13
Persist proxy state through agent restart
2018-06-25 12:24:08 -07:00
a3e0ac1ee3
agent/consul/state: support querying by Connect native
2018-06-25 12:24:08 -07:00
bb98686ec8
agent/cache: update comment from PR review to clarify
2018-06-25 12:24:08 -07:00
418ed161dc
agent: agent service registration supports Connect native services
2018-06-25 12:24:08 -07:00
8e02bbc897
agent/consul: support catalog registration with Connect native
2018-06-25 12:24:07 -07:00
55b3d5d6f4
agent/cache: update comments
2018-06-25 12:24:07 -07:00
ea0270e6aa
agent/cache: correct test name
2018-06-25 12:24:07 -07:00
b5201276bc
agent/cache: change behavior to return error rather than retry
...
The cache behavior should not be to mask errors and retry. Instead, it
should aim to return errors as quickly as possible. We do that here.
2018-06-25 12:24:07 -07:00
778e318a52
agent/cache: perform backoffs on error retries on blocking queries
2018-06-25 12:24:06 -07:00
95f0e8815d
Merge pull request #4234 from hashicorp/feature/default-new-ui
...
Switch over to defaulting to the new UI
2018-06-20 09:10:08 -04:00
6ccc4f39db
Merge pull request #4216 from hashicorp/rpc-limiting
...
Make RPC limits reloadable
2018-06-20 09:05:28 -04:00
426211fad6
Merge pull request #4215 from hashicorp/feature/config-node-meta-dns-txt
...
Add configuration entry to control including TXT records for node meta in DNS responses
2018-06-20 08:53:04 -04:00
bfe2fcbdf1
Update the runtime tests
2018-06-19 13:59:26 -04:00
b9d1e7042a
Make filtering out TXT RRs only apply when they would end up in Additional section
...
ANY queries are no longer affected.
2018-06-19 10:08:16 -04:00
9cb81dc47e
Switch over to defaulting to the new UI
2018-06-15 09:20:13 -04:00
54bc937fed
Re-use uint8ToString
2018-06-14 09:42:23 -07:00
4d46bba2c4
Support giving the duration as a string in CA config
2018-06-14 09:42:22 -07:00
771842255a
address comment feedback
2018-06-14 09:42:22 -07:00
9249662c6c
agent: leaf endpoint accepts name, not service ID
...
This change is important so that requests can made representing a
service that may not be registered with the same local agent.
2018-06-14 09:42:20 -07:00
787ce3b269
agent: address feedback
2018-06-14 09:42:20 -07:00
b5b29cd6af
agent: rename test to check
2018-06-14 09:42:18 -07:00
b961bab08c
agent: implement HTTP endpoint
2018-06-14 09:42:18 -07:00
a48ff54318
agent/consul: forward request if necessary
2018-06-14 09:42:17 -07:00
b02502be73
agent: comments to point to differing logic
2018-06-14 09:42:17 -07:00
526cfc34bd
agent/consul: implement Intention.Test endpoint
2018-06-14 09:42:17 -07:00
bd5e569dc7
Make invalid clusterID be fatal
2018-06-14 09:42:17 -07:00
919fd3e148
Fix logical conflicts with CA refactor
2018-06-14 09:42:17 -07:00
73f2a49ef1
Fix broken api test for service Meta (logical conflict rom OSS). Add test that would make this much easier to catch in future.
2018-06-14 09:42:17 -07:00
bd5eb8b749
Add default CA config back - I didn't add it and causes nil panics
2018-06-14 09:42:17 -07:00
dbcf286d4c
Ooops remove the CA stuff from actual server defaults and make it test server only
2018-06-14 09:42:16 -07:00
834ed1d25f
Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.
2018-06-14 09:42:16 -07:00
bdd30b191b
Comment cleanup
2018-06-14 09:42:16 -07:00
5abf47472d
Verify trust domain on /authorize calls
2018-06-14 09:42:16 -07:00
30d90b3be4
Generate CSR using real trust-domain
2018-06-14 09:42:16 -07:00
5a1408f186
Add CSR signing verification of service ACL, trust domain and datacenter.
2018-06-14 09:42:16 -07:00
c808833a78
Return TrustDomain from CARoots RPC
2018-06-14 09:42:15 -07:00
d1265bc38b
Rename some of the CA structs/files
2018-06-14 09:42:15 -07:00
1660f9ebab
Add more metadata to structs.CARoot
2018-06-14 09:42:15 -07:00
baf4db1c72
Use provider state table for a global serial index
2018-06-14 09:42:15 -07:00
5998623c44
Add test for ca config http endpoint
2018-06-14 09:42:15 -07:00
c90b353eea
Move connect CA provider to separate package
2018-06-14 09:42:15 -07:00
4bb745a2d4
agent/cache: change uint8 to uint
2018-06-14 09:42:15 -07:00
6cf2e1ef1a
agent/cache: string through attempt rather than storing on the entry
2018-06-14 09:42:15 -07:00
c42510e1ec
agent/cache: implement refresh backoff
2018-06-14 09:42:14 -07:00
54a1662da8
agent/consul: change provider wait from goto to a loop
2018-06-14 09:42:14 -07:00
749f81373f
agent/consul: check nil on getCAProvider result
2018-06-14 09:42:14 -07:00
c57405b323
agent/consul: retry reading provider a few times
2018-06-14 09:42:14 -07:00
b4f990bc6c
agent: verify local proxy tokens for CA leaf + tests
2018-06-14 09:42:14 -07:00
8f7b5f93cd
agent: verify proxy token for ProxyConfig endpoint + tests
2018-06-14 09:42:14 -07:00
3a7aaa63bc
agent/proxy: pass proxy ID as an env var
2018-06-14 09:42:13 -07:00
f69c8b85ef
agent/config: add managed proxy upstreams config to skip
...
agent/config will turn [{}] into {} (single element maps into a single
map) to work around HCL issues. These are resolved in HCL2 which I'm
sure Consul will switch to eventually.
This breaks the connect proxy configuration in service definition FILES
since we call this patch function. For now, let's just special-case skip
this. In the future we maybe Consul will adopt HCL2 and fix it, or we
can do something else if we want. This works and is tested.
2018-06-14 09:42:13 -07:00
662f38c625
agent/structs: validate service definitions, port required for proxy
2018-06-14 09:42:13 -07:00
498c63a6f1
agent/config: default connect enabled in dev mode
...
This enables `consul agent -dev` to begin using Connect features with
the built-in CA. I think this is expected behavior since you can imagine
that new users would want to try.
There is no real downside since we're just using the built-in CA.
2018-06-14 09:42:13 -07:00
954d286d73
Make CSR work with jank domain
2018-06-14 09:42:13 -07:00
257d31e319
agent/proxy: delete pid file on Stop
2018-06-14 09:42:13 -07:00
1dfb4762f5
agent: increase timer for blocking cache endpoints
2018-06-14 09:42:12 -07:00
a89238a9d3
agent/proxy: address PR feedback
2018-06-14 09:42:12 -07:00
7bb13246a8
agent: clarify why we Kill still
2018-06-14 09:42:12 -07:00
147b066c67
agent: restore proxy snapshot but still Kill proxies
2018-06-14 09:42:12 -07:00
1d24df3827
agent/proxy: check if process is alive in addition to Wait
2018-06-14 09:42:12 -07:00
4301f7f1f5
agent: only set the proxy manager data dir if its set
2018-06-14 09:42:12 -07:00
e3be9f7a02
agent/proxy: improve comments on snapshotting
2018-06-14 09:42:12 -07:00
eb31827fac
agent/proxy: implement periodic snapshotting in the manager
2018-06-14 09:42:11 -07:00
64fc9e0218
agent/proxy: check if process is alive
2018-06-14 09:42:11 -07:00
a3a0bc7b13
agent/proxy: implement snapshotting for daemons
2018-06-14 09:42:11 -07:00
9675ed626d
agent/proxy: manager configures the daemon pid path to write pids
2018-06-14 09:42:11 -07:00
5e0f0ba178
agent/proxy: write pid file whenever the daemon process changes
2018-06-14 09:42:11 -07:00
09093a1a1a
agent/proxy: change LogDir to DataDir to reuse for other things
2018-06-14 09:42:11 -07:00
e2133fd391
agent/proxy: make the logs test a bit more robust by waiting for file
2018-06-14 09:42:11 -07:00
d019d33bc6
agent/proxy: don't create the directory in newProxy
2018-06-14 09:42:11 -07:00
49bc7181a4
agent/proxy: send logs to the correct location for daemon proxies
2018-06-14 09:42:10 -07:00
515c47be7d
agent: add additional tests for defaulting in AddProxy
2018-06-14 09:42:10 -07:00
52665f7d23
agent: clean up defaulting of proxy configuration
...
This cleans up and unifies how proxy settings defaults are applied.
2018-06-14 09:42:10 -07:00
ed14e9edf8
agent: resolve some conflicts and fix tests
2018-06-14 09:42:10 -07:00
657c09133a
agent/local: clarify the non-risk of a full buffer
2018-06-14 09:42:10 -07:00
31b09c0674
agent/local: remove outdated comment
2018-06-14 09:42:10 -07:00
bae428326a
agent: use os.Executable
2018-06-14 09:42:09 -07:00
6a78ecea57
agent/proxy: local state event coalescing
2018-06-14 09:42:09 -07:00
16f529a13c
agent/proxy: implement force kill of unresponsive proxy process
2018-06-14 09:42:09 -07:00
4722e3ef76
agent: fix crash that could happen if proxy was nil on load
2018-06-14 09:42:09 -07:00
10fe87bd4a
agent/proxy: pull exit status extraction to constrained file
2018-06-14 09:42:09 -07:00
669268f85c
agent: start proxy manager
2018-06-14 09:42:09 -07:00
6884654c9d
agent/proxy: detect config change to stop/start proxies
2018-06-14 09:42:09 -07:00
8ce3deac5d
agent/proxy: test removing proxies and stopping them
2018-06-14 09:42:08 -07:00
a2167a7fd1
agent/proxy: manager and basic tests, not great coverage yet coming soon
2018-06-14 09:42:08 -07:00
fae8dc8951
agent/local: add Notify mechanism for proxy changes
2018-06-14 09:42:08 -07:00
f64a002f68
agent: start/stop proxies
2018-06-14 09:42:08 -07:00
93cdd3f206
agent/proxy: clean up usage, can't be restarted
2018-06-14 09:42:08 -07:00
536f31571b
agent: change connect command paths to be slices, not strings
...
This matches other executable configuration and allows us to cleanly
separate executable from arguments without trying to emulate shell
parsing.
2018-06-14 09:42:08 -07:00
76c6849ffe
agent/local: store proxy on local state, wip, not working yet
2018-06-14 09:42:08 -07:00
659ab7ee2d
agent/proxy: exponential backoff on restarts
2018-06-14 09:42:07 -07:00
c2f50f1688
agent/proxy: Daemon works, tests cover it too
2018-06-14 09:42:07 -07:00
c47ad68f25
wip
2018-06-14 09:42:07 -07:00
02ab461dae
TLS watching integrated into Service with some basic tests.
...
There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.
2018-06-14 09:42:07 -07:00
dcd277de8a
Wire up agent leaf endpoint to cache framework to support blocking.
2018-06-14 09:42:07 -07:00
b28e11fdd3
Fill out connect CA rpc endpoint tests
2018-06-14 09:42:06 -07:00
0e184f3f5b
Fix config tests
2018-06-14 09:42:06 -07:00
7c0976208d
Add tests for the built in CA's state store table
2018-06-14 09:42:06 -07:00
19b9399f2f
Add more tests for built-in provider
2018-06-14 09:42:06 -07:00
a29f3c6b96
Fix some inconsistencies around the CA provider code
2018-06-14 09:42:06 -07:00
153808db7c
Don't allow connect watches in agent/cli yet
2018-06-14 09:42:06 -07:00
072b2a79ca
Support legacy watch.HandlerFunc type for backward compat reduces impact of change
2018-06-14 09:42:05 -07:00
6f566f750e
Basic `watch` support for connect proxy config and certificate endpoints.
...
- Includes some bug fixes for previous `api` work and `agent` that weren't tested
- Needed somewhat pervasive changes to support hash based blocking - some TODOs left in our watch toolchain that will explicitly fail on hash-based watches.
- Integration into `connect` is partially done here but still WIP
2018-06-14 09:42:05 -07:00
2167713226
Add CA config to connect section of agent config
2018-06-14 09:42:05 -07:00
02fef5f9a2
Move ConsulCAProviderConfig into structs package
2018-06-14 09:42:04 -07:00
887cc98d7e
Simplify the CAProvider.Sign method
2018-06-14 09:42:04 -07:00
44b30476cb
Simplify the CA provider interface by moving some logic out
2018-06-14 09:42:04 -07:00
aa10fb2f48
Clarify some comments and names around CA bootstrapping
2018-06-14 09:42:04 -07:00
5abd43a567
agent: resolve flaky test by checking cache hits increase, rather than
...
exact
2018-06-14 09:42:04 -07:00
73838c9afa
agent: use helper/retry instead of timing related tests
2018-06-14 09:42:04 -07:00
dcb2671d10
agent/cache: address PR feedback, lots of typos
2018-06-14 09:42:03 -07:00
07d878a157
agent/cache: address feedback, clarify comments
2018-06-14 09:42:03 -07:00
ad3928b6bd
agent/cache: don't every block on NotifyCh
2018-06-14 09:42:03 -07:00
3f80a9f330
agent/cache: unit tests for ExpiryHeap, found a bug!
2018-06-14 09:42:03 -07:00
1c31e34e5b
agent/cache: send the total entries count on eviction to go-metrics
2018-06-14 09:42:03 -07:00
ec559d77bd
agent/cache: make edge case with prev/next idx == 0 handled better
2018-06-14 09:42:03 -07:00
b319d06276
agent/cache: rework how expiry data is stored to be more efficient
2018-06-14 09:42:03 -07:00
449bbd817d
agent/cache: initial TTL work
2018-06-14 09:42:02 -07:00
3c6acbda5d
agent/cache: send the RefreshTimeout into the backend fetch
2018-06-14 09:42:02 -07:00
257fc34e51
agent/cache: on error, return from Get immediately, don't block forever
2018-06-14 09:42:02 -07:00
e9d58ca219
agent/cache: lots of comment/doc updates
2018-06-14 09:42:02 -07:00
a1f8cb9570
agent: augment /v1/connect/authorize to cache intentions
2018-06-14 09:42:02 -07:00
56774f24d0
agent/cache-types: support intention match queries
2018-06-14 09:42:02 -07:00
109bb946e9
agent/cache: return the error as part of Get
2018-06-14 09:42:01 -07:00
6ecc2da7ff
agent/cache: integrate go-metrics so the cache is debuggable
2018-06-14 09:42:01 -07:00
3b6c46b7d7
agent/structs: DCSpecificRequest sets all the proper fields for
...
CacheInfo
2018-06-14 09:42:01 -07:00
ccd7eeef1a
agent/cache-types/ca-leaf: proper result for timeout, race on setting CA
2018-06-14 09:42:01 -07:00
4509589427
agent/cache: support timeouts for cache reads and empty fetch results
2018-06-14 09:42:01 -07:00
b0f70f17db
agent/cache-types: rename to separate root and leaf cache types
2018-06-14 09:42:01 -07:00
e3b1c400e5
agent/cache-types: got basic CA leaf caching work, major problems still
2018-06-14 09:42:01 -07:00
9e44a319d3
agent: check cache hit count to verify CA root caching, background update
2018-06-14 09:42:00 -07:00
8bb4fd95a6
agent: initialize the cache and cache the CA roots
2018-06-14 09:42:00 -07:00
286217cbd8
agent/cache: partition by DC/ACL token
2018-06-14 09:42:00 -07:00
72c82a9b29
agent/cache: Reorganize some files, RequestInfo struct, prepare for partitioning
2018-06-14 09:42:00 -07:00
ecc789ddb5
agent/cache: ConnectCA roots caching type
2018-06-14 09:42:00 -07:00
c69df79e0c
agent/cache: blank cache key means to always fetch
2018-06-14 09:42:00 -07:00
8584e9262e
agent/cache: initial kind-of working cache
2018-06-14 09:42:00 -07:00
43f13d5a0b
Add cross-signing mechanism to root rotation
2018-06-14 09:42:00 -07:00
bbfcb278e1
Add the root rotation mechanism to the CA config endpoint
2018-06-14 09:41:59 -07:00
a585a0ba10
Have the built in CA store its state in raft
2018-06-14 09:41:59 -07:00
80eddb0bfb
Fix the testing endpoint's root set op
2018-06-14 09:41:59 -07:00
9fefac745e
Update the CA config endpoint to enable GETs
2018-06-14 09:41:59 -07:00
fc9ef9741b
Hook the CA RPC endpoint into the provider interface
2018-06-14 09:41:59 -07:00
a40db26ffe
Add CA bootstrapping on establishing leadership
2018-06-14 09:41:59 -07:00
e26819ed9c
Add the bootstrap config for the CA
2018-06-14 09:41:59 -07:00
4d0713d5bb
Add the CA provider interface and built-in provider
2018-06-14 09:41:58 -07:00
ebdda17a30
Add CA config set to fsm operations
2018-06-14 09:41:58 -07:00
f7ff16669f
Add the Connect CA config to the state store
2018-06-14 09:41:58 -07:00
a90f69faa4
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list.
2018-06-14 09:41:58 -07:00
9d11cd9bf4
Fix various test failures and vet warnings.
...
Intention de-duplication in previously merged PR actualy failed some tests that were not caught be me or CI. I ran the test files for state changes but they happened not to trigger this case so I made sure they did first and then fixed. That fixed some upstream intention endpoint tests that I'd not run as part of testing the previous fix.
2018-06-14 09:41:58 -07:00
Paul Banks