Commit Graph

1842 Commits

Author SHA1 Message Date
hashicorp-ci 8d53a33bf0
update bindata_assetfs.go 2019-12-20 17:16:51 +00:00
R.B. Boyer 42f80367be
Restore a few more service-kind index updates so blocking in ServiceDump works in more cases (#6948)
Restore a few more service-kind index updates so blocking in ServiceDump works in more cases

Namely one omission was that check updates for dumped services were not
unblocking.

Also adds a ServiceDump state store test and also fix a watch bug with the
normal dump.

Follow-on from #6916
2019-12-19 10:15:37 -06:00
Matt Keeler 6de4eb8569
OSS changes for implementing token based namespace inferencing
remove debug log
2019-12-18 14:07:08 -05:00
Matt Keeler 185654b075
Unflake the TestACLEndpoint_TokenList test
In order to do this I added a waitForLeaderEstablishment helper which does the right thing to ensure that leader establishment has finished.

fixup
2019-12-18 14:07:07 -05:00
Matt Keeler 8af12bf4f4
Miscellaneous acl package cleanup
• Renamed EnterpriseACLConfig to just Config
• Removed chained_authorizer_oss.go as it was empty
• Renamed acl.go to errors.go to more closely describe its contents
2019-12-18 13:44:32 -05:00
Matt Keeler bdf025a758
Rename EnterpriseAuthorizerContext -> AuthorizerContext 2019-12-18 13:43:24 -05:00
Matt Keeler af1d101937
OSS changes to allow for parsing the enterprise DNS config prop… (#6959) 2019-12-18 10:16:35 -05:00
Preetha f607a00138 autopilot: fix dead server removal condition to use correct failure tolerance (#4017)
* Make dead server removal condition in autopilot use correct failure tolerance rules
* Introduce func with explanation
2019-12-16 23:35:13 +01:00
Wim e3e56ff3c2 dns: fix memoryleak by upgrading outdated miekg/dns (#6748)
* Add updated github.com/miekg/dns to go modules
* Add updated github.com/miekg/dns to vendor
* Fix github.com/miekg/dns api breakage
* Decrease size when trimming UDP packets
Need more room for the header(?), if we don't decrease the size we get an
"overflow unpacking uint32" from the dns library
* Fix dns truncate tests with api changes
* Make windows build working again. Upgrade x/sys and x/crypto and vendor
This upgrade is needed because of API breakage in x/sys introduced
by the minimal x/sys dependency of miekg/dns

This API breakage has been fixed in commit
855e68c859
2019-12-16 22:31:27 +01:00
Hans Hasselberg ae23376218
acl: use constant time comparing to check token (#6943) 2019-12-16 21:54:52 +01:00
Matt Keeler 0c1a7970e3 ui: feature support templating for index.html (#6921) 2019-12-13 14:50:07 -05:00
hashicorp-ci 30b513fe19
update bindata_assetfs.go 2019-12-10 19:12:06 +00:00
Matt Keeler 9812b32155
Fix blocking for ServiceDumping by kind (#6919) 2019-12-10 13:58:30 -05:00
Matt Keeler 442924c35a
Sync of OSS changes to support namespaces (#6909) 2019-12-09 21:26:41 -05:00
rerorero 3653855f13 [ci] fix: go-fmt fails on master branch (#6906) 2019-12-08 20:30:46 -05:00
Matt Keeler 81b5f9df02
Fix the TestAPI_CatalogRegistration test 2019-12-06 15:47:41 -05:00
Hans Hasselberg 368d5c643f
tls: auto_encrypt and verify_incoming (#6811) (#6899)
* relax requirements for auto_encrypt on server
* better error message when auto_encrypt and verify_incoming on
* docs: explain verify_incoming on Consul clients.
2019-12-06 21:36:13 +01:00
Hans Hasselberg a36e58c964
agent: fewer file local differences between enterprise and oss (#6820) (#6898)
* Increase number to test ignore. Consul Enterprise has more flags and since we are trying to reduce the differences between both code bases, we are increasing the number in oss. The semantics don't change, it is just a cosmetic thing.
* Introduce agent.initEnterprise for enterprise related hooks.
* Sync test with ent version.
* Fix import order.
* revert error wording.
2019-12-06 21:35:58 +01:00
Matt Keeler 609c9dab02
Miscellaneous Fixes (#6896)
Ensure we close the Sentinel Evaluator so as not to leak go routines

Fix a bunch of test logging so that various warnings when starting a test agent go to the ltest logger and not straight to stdout.

Various canned ent meta types always return a valid pointer (no more nils). This allows us to blindly deref + assign in various places.

Update ACL index tracking to ensure oss -> ent upgrades will work as expected.

Update ent meta parsing to include function to disallow wildcarding.
2019-12-06 14:01:34 -05:00
Matt Keeler b9996e6bbe
Add Namespace support to the API module and the CLI commands (#6874)
Also update the Docs and fixup the HTTP API to return proper errors when someone attempts to use Namespaces with an OSS agent.

Add Namespace HTTP API docs

Make all API endpoints disallow unknown fields
2019-12-06 11:14:56 -05:00
Matt Keeler c15c81a7ed
[Feature] API: Add a internal endpoint to query for ACL authori… (#6888)
* Implement endpoint to query whether the given token is authorized for a set of operations

* Updates to allow for remote ACL authorization via RPC

This is only used when making an authorization request to a different datacenter.
2019-12-06 09:25:26 -05:00
Matt Keeler 53d9319a82
Fix the TestLeader_SecondaryCA_IntermediateRefresh test flakine… (#6885)
Fix the TestLeader_SecondaryCA_IntermediateRefresh test flakiness
2019-12-05 09:35:45 -05:00
Hans Hasselberg ad65068d6e
tests: increase TLSHandshakeTimeout to help slow tests (#6864)
Fixes https://github.com/hashicorp/consul/issues/6858.
2019-12-05 13:20:07 +01:00
Matt Keeler f30af37d11
Fix the TestLeader_SecondaryCA_IntermediateRefresh test flakiness 2019-12-04 19:19:55 -05:00
Mike Morris 1fe6da2ad6
Bump go-discover to support EC2 Metadata Service v2 (#6865)
Refs https://github.com/hashicorp/go-discover/pull/128

* deps: add replace directive for gocheck

Transitive dep, source at https://launchpad.net/gocheck indicates
project moved. This also avoids a dependency on bzr when fetching
modules. Refs https://github.com/hashicorp/consul/pull/6818

* deps: make update-vendor

* test: update retry-join expected names from go-discover
2019-12-04 11:59:16 -05:00
Chris Piraino 2a95701341
Allow configuration of upstream connection limits in Envoy (#6829)
* Adds 'limits' field to the upstream configuration of a connect proxy

This allows a user to configure the envoy connect proxy with
'max_connections', 'max_queued_requests', and 'max_concurrent_requests'. These
values are defined in the local proxy on a per-service instance basis
and should thus NOT be thought of as a global-level or even service-level value.
2019-12-03 14:13:33 -06:00
Matt Keeler 7557225272
Fix dns service SRV lookup when service address is a fqdn (#6792)
Fix dns service SRV lookup when service address is a fqdn
2019-12-03 10:15:19 -05:00
Sarah Adams 1f5b333290
give feedback to CLI user on forceleave command if node does not exist (#6841) 2019-12-02 11:06:15 -08:00
R.B. Boyer a9343db838
xds: mesh gateway CDS requests are now allowed to receive an empty CDS reply (#6787)
This is the rest of the fix for #6543 that was incompletely fixed in #6576.
2019-11-26 15:55:13 -06:00
Matt Keeler 90ae4a1f1e
OSS KV Modifications to Support Namespaces 2019-11-25 12:57:35 -05:00
Matt Keeler 68d79142c4
OSS Modifications necessary for sessions namespacing 2019-11-25 12:07:04 -05:00
Paul Banks a84b82b3df
connect: Add AWS PCA provider (#6795)
* Update AWS SDK to use PCA features.

* Add AWS PCA provider

* Add plumbing for config, config validation tests, add test for inheriting existing CA resources created by user

* Unparallel the tests so we don't exhaust PCA limits

* Merge updates

* More aggressive polling; rate limit pass through on sign; Timeout on Sign and CA create

* Add AWS PCA docs

* Fix Vault doc typo too

* Doc typo

* Apply suggestions from code review

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>
Co-Authored-By: kaitlincarter-hc <43049322+kaitlincarter-hc@users.noreply.github.com>

* Doc fixes; tests for erroring if State is modified via API

* More review cleanup

* Uncomment tests!

* Minor suggested clean ups
2019-11-21 17:40:29 +00:00
Chris Piraino 9a85452787 test: unflake two TestHealthServiceNode_* tests
Replaces WaitForLeader with WaitForTestAgent. This waits to make sure
that the node itself is correctly registered in the catalog before
attempting additional registrations.
2019-11-18 16:21:01 -06:00
Chris Piraino 8c25fff329 test: unflake TestDNS_ServiceLookup_WanTranslation
Use retry.R struct to check length of WANMembers so that retries can
work appropriately.
2019-11-18 16:21:01 -06:00
Chris Piraino d7fc9f3d34 test: unflake TestCatalogServiceNodes_DistanceSort
Remove a time.Sleep and replace with retry.Run around call to
CatalogServiceNodes.
2019-11-18 16:21:01 -06:00
Paul Banks 9e17aa3b41
Change CA Configure struct to pass Datacenter through (#6775)
* Change CA Configure struct to pass Datacenter through

* Remove connect/ca/plugin as we don't have immediate plans to use it.

We still intend to one day but there are likely to be several changes to the CA provider interface before we do so it's better to rebuild from history when we do that work properly.

* Rename PrimaryDC; fix endpoint in secondary DCs
2019-11-18 14:22:19 +00:00
Matt Keeler b8391fa760
Finish the comment 2019-11-15 10:33:21 -05:00
Matt Keeler 036ab56f17
Track the correct check id for idempotent service/check updates 2019-11-14 11:30:44 -05:00
Nicolas Benoit e3e0b8c95d Fix dns service SRV lookup when service address is a fqdn
Refactor dns to have same behavior between A and SRV.
Current implementation returns the node name instead of the service
address.
With this fix when querying for SRV record service address is return in
the SRV record.
And when performing a simple dns lookup it returns a CNAME to the
service address.
2019-11-14 16:50:05 +01:00
Paul Banks 1197b43c7b
Support Connect CAs that can't cross sign (#6726)
* Support Connect CAs that can't cross sign

* revert spurios mod changes from make tools

* Add log warning when forcing CA rotation

* Fixup SupportsCrossSigning to report errors and work with Plugin interface (fixes tests)

* Fix failing snake_case test

* Remove misleading comment

* Revert "Remove misleading comment"

This reverts commit bc4db9cabed8ad5d0e39b30e1fe79196d248349c.

* Remove misleading comment

* Regen proto files messed up by rebase
2019-11-11 21:36:22 +00:00
Paul Banks ca96d5fa72
connect: Allow CA Providers to store small amount of state (#6751)
* pass logger through to provider

* test for proper operation of NeedsLogger

* remove public testServer function

* Ooops actually set the logger in all the places we need it - CA config set wasn't and causing segfault

* Fix all the other places in tests where we set the logger

* Allow CA Providers to persist some state

* Update CA provider plugin interface

* Fix plugin stubs to match provider changes

* Update agent/connect/ca/provider.go

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>

* Cleanup review comments
2019-11-11 20:57:16 +00:00
Todd Radel 19a3892f71 connect: Implement NeedsLogger interface for CA providers (#6556)
* add NeedsLogger to Provider interface

* implements NeedsLogger in default provider

* pass logger through to provider

* test for proper operation of NeedsLogger

* remove public testServer function

* Switch test to actually assert on logging output rather than reflection.

--amend

* Ooops actually set the logger in all the places we need it - CA config set wasn't and causing segfault

* Fix all the other places in tests where we set the logger

* Add TODO comment
2019-11-11 20:30:01 +00:00
Todd Radel e100fda218 Make all Connect Cert Common Names valid FQDNs (#6423) 2019-11-11 17:11:54 +00:00
Matt Keeler 7081643191
Fill the Authz Context with a Sentinel Scope (#6729) 2019-11-01 17:05:22 -04:00
Matt Keeler ba9871d1c2
Fix type name (#6728) 2019-11-01 16:58:00 -04:00
Matt Keeler 7a2cee53c9
Add DirEntry method to fill enterprise authz context 2019-11-01 16:48:44 -04:00
Matt Keeler c71ea7056f
Miscellaneous fixes (#6727) 2019-11-01 16:11:44 -04:00
Ferenc Fabian 3ad20d8d5b Case sensitive Authorization header with lower-cased scheme in… (#6724) 2019-11-01 09:56:41 -04:00
Paul Banks 5f405c3277
Fix support for RSA CA keys in Connect. (#6638)
* Allow RSA CA certs for consul and vault providers to correctly sign EC leaf certs.

* Ensure key type ad bits are populated from CA cert and clean up tests

* Add integration test and fix error when initializing secondary CA with RSA key.

* Add more tests, fix review feedback

* Update docs with key type config and output

* Apply suggestions from code review

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>
2019-11-01 13:20:26 +00:00
Matt Keeler a338357fa3
Fix the Synthetic Policy Tests (#6715) 2019-10-30 15:15:14 -04:00
Matt Keeler 21f98f426e
Add hook for validating the enterprise meta attached to a reque… (#6695) 2019-10-30 12:42:39 -04:00
Matt Keeler ae57e736d2
Add note about RPC multiplexing and TLS content type mutual exc… (#6698) 2019-10-30 09:24:30 -04:00
Matt Keeler c2d9041c0f
PreVerify acl:read access for listing endpoints (#6696)
We still will need to filter results based on the authorizer too but this helps to give an early 403.
2019-10-30 09:10:11 -04:00
Sarah Adams 7a4be7863d
Use encoding/json as JSON decoder instead of mapstructure (#6680)
Fixes #6147
2019-10-29 11:13:36 -07:00
Sarah Christoff 86b30bbfbe
Set MinQuorum variable in Autopilot (#6654)
* Add MinQuorum to Autopilot
2019-10-29 09:04:41 -05:00
Matt Keeler 0fc2c95255
More Replication Abstractions (#6689)
Also updated ACL replication to use a function to fill in the desired enterprise meta for all remote listing RPCs.
2019-10-28 13:49:57 -04:00
Matt Keeler 87c44a3b8d
Ensure that cache entries for tokens are prefixed “token-secret… (#6688)
This will be necessary once we store other types of identities in here.
2019-10-25 13:05:43 -04:00
Matt Keeler a688ea952d
Update the ACL Resolver to allow for Consul Enterprise specific hooks. (#6687) 2019-10-25 11:06:16 -04:00
Matt Keeler 1270a93274
Updates to allow for Namespacing ACL resources in Consul Enterp… (#6675)
Main Changes:

• method signature updates everywhere to account for passing around enterprise meta.
• populate the EnterpriseAuthorizerContext for all ACL related authorizations.
• ACL resource listings now operate like the catalog or kv listings in that the returned entries are filtered down to what the token is allowed to see. With Namespaces its no longer all or nothing.
• Modified the acl.Policy parsing to abstract away basic decoding so that enterprise can do it slightly differently. Also updated method signatures so that when parsing a policy it can take extra ent metadata to use during rules validation and policy creation.

Secondary Changes:

• Moved protobuf encoding functions out of the agentpb package to eliminate circular dependencies.
• Added custom JSON unmarshalers for a few ACL resource types (to support snake case and to get rid of mapstructure)
• AuthMethod validator cache is now an interface as these will be cached per-namespace for Consul Enterprise.
• Added checks for policy/role link existence at the RPC API so we don’t push the request through raft to have it fail internally.
• Forward ACL token delete request to the primary datacenter when the secondary DC doesn’t have the token.
• Added a bunch of ACL test helpers for inserting ACL resource test data.
2019-10-24 14:38:09 -04:00
Sarah Adams 911bb78296
regression tests for existing agent/ decoding behavior (#6624)
tests for existing JSON decoding behavior
2019-10-22 15:26:24 -07:00
rerorero e210b0c854 fix: incorrect struct tag and WaitGroup usage (#6649)
* remove duplicated json tag

* fix: incorrect wait group usage
2019-10-18 13:59:29 -04:00
R.B. Boyer b091647090
agent: allow mesh gateways to initialize even if there are no connect services registered yet (#6576)
Fixes #6543

Also improved some of the proxycfg tests to cover snapshot validity
better.
2019-10-17 16:46:49 -05:00
R.B. Boyer 1ab04a8b6a
xds: tcp services using the discovery chain should not assume RDS during LDS (#6623)
Previously the logic for configuring RDS during LDS for L7 upstreams was
overapplied to TCP proxies resulting in a cluster name of <emptystring>
being used incorrectly.

Fixes #6621
2019-10-17 16:44:59 -05:00
Freddy caf658d0d3
Store check type in catalog (#6561) 2019-10-17 20:33:11 +02:00
R.B. Boyer e74a6c44f1
server: ensure the primary dc and ACL dc match (#6634)
This is mostly a sanity check for server tests that skip the normal
config builder equivalent fixup.
2019-10-17 10:57:17 -05:00
R.B. Boyer bc22eb8090
unflake TestLeader_SecondaryCA_Initialize (#6631) 2019-10-16 16:49:01 -05:00
R.B. Boyer 3ae748c7a4
fix flaky multidc acl tests that failed to wait for token replication (#6628)
If acls have not yet replicated to the secondary then authz requests
will be remotely resolved by the primary. Now these tests explicitly
wait until replication has caught up first.
2019-10-16 12:24:29 -05:00
R.B. Boyer a4c5b8e85c
appease the retry linter (#6629) 2019-10-16 11:39:22 -05:00
Paul Banks 979ad7fecb
Allow time for secondary CA to initialize (#6627) 2019-10-16 17:03:31 +01:00
Matt Keeler f9a43a1e2d
ACL Authorizer overhaul (#6620)
* ACL Authorizer overhaul

To account for upcoming features every Authorization function can now take an extra *acl.EnterpriseAuthorizerContext. These are unused in OSS and will always be nil.

Additionally the acl package has received some thorough refactoring to enable all of the extra Consul Enterprise specific authorizations including moving sentinel enforcement into the stubbed structs. The Authorizer funcs now return an acl.EnforcementDecision instead of a boolean. This improves the overall interface as it makes multiple Authorizers easily chainable as they now indicate whether they had an authoritative decision or should use some other defaults. A ChainedAuthorizer was added to handle this Authorizer enforcement chain and will never itself return a non-authoritative decision.

* Include stub for extra enterprise rules in the global management policy

* Allow for an upgrade of the global-management policy
2019-10-15 16:58:50 -04:00
PHBourquin 16ca8340c1 Checks to passing/critical only after reaching a consecutive success/failure threshold (#5739)
A check may be set to become passing/critical only if a specified number of successive
checks return passing/critical in a row. Status will stay identical as before until
the threshold is reached.
This feature is available for HTTP, TCP, gRPC, Docker & Monitor checks.
2019-10-14 21:49:49 +01:00
Sarah Christoff 6247ca7f0d
ui_content_path config option fix (#6601)
* fix ui-content-path config option
2019-10-09 09:14:48 -05:00
Hans Hasselberg 7e4017551a Do not surface left servers (#6420)
* do not surface left servers in catalog
2019-10-08 22:16:00 -05:00
R.B. Boyer 9a51ecc98b
agent: clients should only attempt to remove pruned nodes once per call (#6591) 2019-10-07 16:15:23 -05:00
Sarah Christoff 9b93dd93c9
Prune Unhealthy Agents (#6571)
* Add -prune flag to ForceLeave
2019-10-04 16:10:02 -05:00
R.B. Boyer fa9c567278
agent: updates to the agent token trigger anti-entropy full syncs (#6577) 2019-10-04 13:37:34 -05:00
Matt Keeler b0b57588d1
Implement Leader Routine Management (#6580)
* Implement leader routine manager

Switch over the following to use it for go routine management:

• Config entry Replication
• ACL replication - tokens, policies, roles and legacy tokens
• ACL legacy token upgrade
• ACL token reaping
• Intention Replication
• Secondary CA Roots Watching
• CA Root Pruning

Also added the StopAll call into the Server Shutdown method to ensure all leader routines get killed off when shutting down.

This should be mostly unnecessary as `revokeLeadership` should manually stop each one but just in case we really want these to go away (eventually).
2019-10-04 13:08:45 -04:00
Matt Keeler 29f0616708
Use encoding/json instead of jsonpb even for protobuf types (#6572)
This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 15:32:15 -04:00
Matt Keeler 9bd378a95c
Add EnterpriseConfig stubs (#6566) 2019-10-01 14:34:55 -04:00
Matt Keeler cfa879d63c
Generate JSON and Binary Marshalers for Protobuf Types (#6564)
* Add JSON and Binary Marshaler Generators for Protobuf Types

* Generate files with the correct version of gogo/protobuf

I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date.

The version at the time of this commit we are using is v1.2.1

* Fixup some shell output

* Update how we determine the version of gogo
This just greps the go.mod file instead of expecting the go mod cache to already be present

* Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 15:39:20 -04:00
John Cowen 338812f5c2
ui: UI Release Merge (ui-staging merge) (#6527)
## HTTPAdapter (#5637)

## Ember upgrade 2.18 > 3.12 (#6448)

### Proxies can no longer get away with not calling _super

This means that we can't use create anymore to define dynamic methods.
Therefore we dynamically make 2 extended Proxies on demand, and then
create from those. Therefore we can call _super in the init method of
the extended Proxies.

### We aren't allowed to reset a service anymore

We never actually need to now anyway, this is a remnant of the refactor
from browser based confirmations. We fix it as simply as possible here
but will revisit and remove the old browser confirm functionality at a
later date

### Revert classes to use ES5 style to workaround babel transp. probs

Using a mixture of ES6 classes (and hence super) and arrow functions
means that when babel transpiles the arrow functions down to ES5, a
reference to this is moved before the call to super, hence causing a js
error.

Furthermore, we the testing environment no longer lets use use
apply/call on the constructor.

These errors only manifests during testing (only in the testing
environment), the application itself runs fine with no problems without
this change.

Using ES5 style class definitions give us freedom to do all of the above
without causing any errors, so we reverted these classes back to ES5
class definitions

### Skip test that seems to have changed due to a change in RSVP timing

This test tests a usecase/area of the API that will probably never ever
be used, it was more testing out the API. We've skipped the test for now
as this doesn't affect the application itself, but left a note to come
back here later to investigate further

### Remove enumerableContentDidChange

Initial testing looks like we don't need to call this function anymore,
the function no longer exists

### Rework Changeset.isSaving to take into account new ember APIs

Setting/hanging a computedProperty of an instantiated object no longer
works. Move to setting it on the prototype/class definition instead

### Change how we detect whether something requires listening

New ember API's have changed how you can detect whether something is a
computedProperty or not. It's not immediately clear if its even possible
now. Therefore we change how we detect whether something should be
listened to or not by just looking for presence of `addEventListener`

### Potentially temporary change of ci test scripts to ensure deps exist

All our tooling scripts run through a Makefile (for people familiar with
only using those), which then call yarn scripts which can be called
independently (for people familar with only using yarn).

The Makefile targets always check to make sure all the dependencies are
installed before running anything that requires them (building, testing
etc).

The CI scripts/targets didn't follow this same route and called the yarn
scripts directly (usually CI builds a cache of the dependencies first).

For some reason this cache isn't doing what it usually does, and it
looks as though, in CI, ember isn't installed.

This commit makes the CI scripts consistently use the same method as all
of the other tooling scripts (Makefile target > Install Deps if
required > call yarn script). This should install the dependencies if
for some reason the CI cache building doesn't complete/isn't successful.

Potentially this commit may be reverted if, the root of the problem is
elsewhere, although consistency is always good, so it might be a good
idea to leave this commit as is even if we need to debug and fix things
elsewhere.

### Make test-parallel consistent with the rest of the tooling scripts

As we are here making changes for CI purposes (making test-ci
consistent), we spotted that test-parallel is also inconsistent and also
the README manual instructions won't work without `ember` installed
globally.

This commit makes everything consistent and changes the manual
instructions to use the local ember instance that gets installed via
yarn

### Re-wrangle catchable to fit with new ember 3.12 APIs

In the upgrade from ember 3.8 > 3.12 the public interfaces for
ComputedProperties have changed slightly. `meta` is no longer a public
property of ComputedProperty but of a ComputedDecoratorImpl mixin
instead.

7e4ba1096e/packages/%40ember/-internals/metal/lib/computed.ts (L725)

There seems to be no way, by just using publically available
methods, to replicate this behaviour so that we can create our own
'ComputedProperty` factory via injecting the ComputedProperty class as
we did previously.

3f333bada1/ui-v2/app/utils/computed/factory.js (L1-L18)

Instead we dynamically hang our `Catchable` `catch` method off the
instantiated ComputedProperty. In doing it like this `ComputedProperty`
has already has its `meta` method mixed in so we don't have to manually
mix it in ourselves (which doesn't seem possible)

This functionality is only used during our work in trying to ensure
our EventSource/BlockingQuery work was as 'ember-like' as possible (i.e.
using the traditional Route.model hooks and ember-like Controller
properties). Our ongoing/upcoming work on a componentized approach to
data a.k.a `<DataSource />` means we will be able to remove the majority
of the code involved here now that it seems to be under an amount of
flux in ember.

### Build bindata_assetfs.go with new UI changes
2019-09-30 14:47:49 +01:00
Matt Keeler 04dbd48ce5
Add support for parameterizing the ACL config used with a TestA… (#6559)
* Add support for parameterizing the ACL config used with a TestAgent

Using tokens that are UUIDs will get rid of some warnings

* Refactor to allow setting all tokens and change the template to ignore unset values.
2019-09-27 17:06:43 -04:00
R.B. Boyer 8433ef02a8
connect: connect CA Roots in secondary datacenters should use a SigningKeyID derived from their local intermediate (#6513)
This fixes an issue where leaf certificates issued in secondary
datacenters would be reissued very frequently (every ~20 seconds)
because the logic meant to detect root rotation was errantly triggering
because a hash of the ultimate root (in the primary) was being compared
against a hash of the local intermediate root (in the secondary) and
always failing.
2019-09-26 11:54:14 -05:00
R.B. Boyer 55fdae203f
agent: cache notifications work after error if the underlying RPC returns index=1 (#6547)
Fixes #6521

Ensure that initial failures to fetch an agent cache entry using the
notify API where the underlying RPC returns a synthetic index of 1
correctly recovers when those RPCs resume working.

The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the
index for the next query from 0 to 1 for all queries, when it should
have not done so for queries that errored.

Also fixed some things that made debugging difficult:

- config entry read/list endpoints send back QueryMeta headers
- xds event loops don't swallow the cache notification errors
2019-09-26 10:42:17 -05:00
Matt Keeler 5b83f589da
Expand the QueryOptions and QueryMeta interfaces (#6545)
In a previous PR I made it so that we had interfaces that would work enough to allow blockingQueries to work. However to complete this we need all fields to be settable and gettable.

Notes:
   • If Go ever gets contracts/generics then we could get rid of all the Getters/Setters
   • protoc / protoc-gen-gogo are going to generate all the getters for us.
   • I copied all the getters/setters from the protobuf funcs into agent/structs/protobuf_compat.go
   • Also added JSON marshaling funcs that use jsonpb for protobuf types.
2019-09-26 09:55:02 -04:00
Freddy 5eace88ce2
Expose HTTP-based paths through Connect proxy (#6446)
Fixes: #5396

This PR adds a proxy configuration stanza called expose. These flags register
listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only
listening on the loopback interface, while still accepting traffic from non
Connect-enabled services.

Under expose there is a boolean checks flag that would automatically expose all
registered HTTP and gRPC check paths.

This stanza also accepts a paths list to expose individual paths. The primary
use case for this functionality would be to expose paths for third parties like
Prometheus or the kubelet.

Listeners for requests to exposed paths are be configured dynamically at run
time. Any time a proxy, or check can be registered, a listener can also be
created.

In this initial implementation requests to these paths are not
authenticated/encrypted.
2019-09-25 20:55:52 -06:00
R.B. Boyer 682b5370c9
agent: tolerate more failure scenarios during service registration with central config enabled (#6472)
Also:

* Finished threading replaceExistingChecks setting (from GH-4905)
  through service manager.

* Respected the original configSource value that was used to register a
  service or a check when restoring persisted data.

* Run several existing tests with and without central config enabled
  (not exhaustive yet).

* Switch to ioutil.ReadFile for all types of agent persistence.
2019-09-24 10:04:48 -05:00
Matt Keeler 8885c8d318
Allow for enterprise only leader routines (#6533)
Eventually I am thinking we may need a way to register these at different priority levels but for now sticking this here is fine
2019-09-23 20:09:56 -04:00
R.B. Boyer cc889443a5
connect: don't colon-hex-encode the AuthorityKeyId and SubjectKeyId fields in connect certs (#6492)
The fields in the certs are meant to hold the original binary
representation of this data, not some ascii-encoded version.

The only time we should be colon-hex-encoding fields is for display
purposes or marshaling through non-TLS mediums (like RPC).
2019-09-23 12:52:35 -05:00
R.B. Boyer 1d54909333
connect: intermediate CA certs generated with the vault provider lack URI SANs (#6491)
This only affects vault versions >=1.1.1 because the prior code
accidentally relied upon a bug that was fixed in
https://github.com/hashicorp/vault/pull/6505

The existing tests should have caught this, but they were using a
vendored copy of vault version 0.10.3. This fixes the tests by running
an actual copy of vault instead of an in-process copy. This has the
added benefit of changing the dependency on vault to just vault/api.

Also update VaultProvider to use similar SetIntermediate validation code
as the ConsulProvider implementation.
2019-09-23 12:04:40 -05:00
Matt Keeler 8431c5f533
Add support for implementing new requests with protobufs instea… (#6502)
* Add build system support for protobuf generation

This is done generically so that we don’t have to keep updating the makefile to add another proto generation.

Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date.

If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild>

Providing the PROTOFILES var will override the default behavior of finding all the .proto files.

* Start adding types to the agent/proto package

These will be needed for some other work and are by no means comprehensive.

* Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store.

* Use protobuf marshalling of raft requests instead of msgpack for protoc generated types.

This does not change any encoding of existing types.

* Removed structs package automatically encoding with protobuf marshalling

Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf`

* Run update-vendor to fixup modules.txt

Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical.

* Rename some things and implement the structs.RPCInfo interface bits

agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth.

* Use the right encoding function.

* Renamed agent/proto package to agent/agentpb to prevent package name conflicts

* Update modules.txt to fix ordering

* Change blockingQuery to take in interfaces for the query options and meta

* Add %T to error output.

* Add/Update some comments
2019-09-20 14:37:22 -04:00
R.B. Boyer 5c5f21088c sdk: add freelist tracking and ephemeral port range skipping to freeport
This should cut down on test flakiness.

Problems handled:

- If you had enough parallel test cases running, the former circular
approach to handling the port block could hand out the same port to
multiple cases before they each had a chance to bind them, leading to
one of the two tests to fail.

- The freeport library would allocate out of the ephemeral port range.
This has been corrected for Linux (which should cover CI).

- The library now waits until a formerly-in-use port is verified to be
free before putting it back into circulation.
2019-09-17 14:30:43 -05:00
R.B. Boyer edf5347d3c fix typo of 'unknown' in log messages 2019-09-13 15:59:49 -05:00
R.B. Boyer c17e417cc8 cache: remove data race in agent cache
In normal operations there is a read/write race related to request
QueryOptions fields. An example race:

    WARNING: DATA RACE
    Read at 0x00c000836950 by goroutine 30:
      github.com/hashicorp/consul/agent/structs.(*ServiceConfigRequest).CacheInfo()
          /go/src/github.com/hashicorp/consul/agent/structs/config_entry.go:506 +0x109
      github.com/hashicorp/consul/agent/cache.(*Cache).getWithIndex()
          /go/src/github.com/hashicorp/consul/agent/cache/cache.go:262 +0x5c
      github.com/hashicorp/consul/agent/cache.(*Cache).notifyBlockingQuery()
          /go/src/github.com/hashicorp/consul/agent/cache/watch.go:89 +0xd7

    Previous write at 0x00c000836950 by goroutine 147:
      github.com/hashicorp/consul/agent/cache-types.(*ResolvedServiceConfig).Fetch()
          /go/src/github.com/hashicorp/consul/agent/cache-types/resolved_service_config.go:31 +0x219
      github.com/hashicorp/consul/agent/cache.(*Cache).fetch.func1()
          /go/src/github.com/hashicorp/consul/agent/cache/cache.go:495 +0x112

This patch does a lightweight copy of the request struct so that the
embedded QueryOptions fields that are mutated during Fetch() are scoped
to just that one RPC.
2019-09-12 16:18:01 -05:00
hashicorp-ci 52bd29b337
update bindata_assetfs.go 2019-09-12 19:39:58 +00:00
Hans Hasselberg f025a7440d
agent: handleEnterpriseLeave (#6453) 2019-09-11 11:01:37 +02:00
R.B. Boyer 4aaaad089f
test: actually wait for the TestAgent to be fully shutdown (#6441) 2019-09-05 13:36:26 -05:00
Sarah Adams 8e673371df
test: ensure all TestAgent constructions use a constructor (#6443)
ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place

Fixes #6435
2019-09-05 10:24:36 -07:00
Sarah Adams c6c5f9c494
remove funky panic/recover in agent tests (#6442) 2019-09-04 13:59:11 -07:00
Sarah Adams f8fa10fecb
refactor & add better retry logic to NewTestAgent (#6363)
Fixes #6361
2019-09-03 15:05:51 -07:00
Pierre Souchay 6d13efa828 Distinguish between DC not existing and not being available (#6399) 2019-09-03 09:46:24 -06:00
Aestek 19c4459d19 Add option to register services and their checks idempotently (#4905) 2019-09-02 09:38:29 -06:00
Sarah Adams 2254633d93
txn: don't try to decode request bodies > raft.SuggestedMaxDataSize (#6422)
txn: don't try to decode request bodies > raft.SuggestedMaxDataSize
2019-08-30 10:41:25 -07:00
Matt Keeler 31d9d2e557
Store primaries root in secondary after intermediate signature (#6333)
* Store primaries root in secondary after intermediate signature

This ensures that the intermediate exists within the CA root stored in raft and not just in the CA provider state. This has the very nice benefit of actually outputting the intermediate cert within the ca roots HTTP/RPC endpoints.

This change means that if signing the intermediate fails it will not set the root within raft. So far I have not come up with a reason why that is bad. The secondary CA roots watch will pull the root again and go through all the motions. So as soon as getting an intermediate CA works the root will get set.

* Make TestAgentAntiEntropy_Check_DeferSync less flaky

I am not sure this is the full fix but it seems to help for me.
2019-08-30 11:38:46 -04:00
R.B. Boyer c5e1faaddb
test: ensure the node name is a valid dns name (#6424)
The space in the node name was making every test emit a useless warning.
2019-08-29 16:52:13 -05:00
R.B. Boyer 4a2867a814
test: explicitly run the pprof tests for 1s instead of the 30s default (#6421) 2019-08-29 12:06:50 -05:00
R.B. Boyer 0f29543315
test: add additional http status code assertions in coordinate HTTP API tests (#6410)
When this test flakes sometimes this happens:

    --- FAIL: TestCoordinate_Node (1.69s)
    panic: interface conversion: interface {} is nil, not structs.Coordinates [recovered]
    FAIL    github.com/hashicorp/consul/agent       19.999s
    Exit code: 1
    panic: interface conversion: interface {} is nil, not structs.Coordinates [recovered]
            panic: interface conversion: interface {} is nil, not structs.Coordinates

There is definitely a bug lurking, but the code seems to imply this can
only return nil on 404. The tests previously were not checking the
status code.

The underlying cause of the flake is unknown, but this should turn the
failure into a more normal test failure.
2019-08-29 09:55:05 -05:00
Pierre Souchay 35d90fc899 Display IPs of machines when node names conflict to ease troubleshooting
When there is an node name conflicts, such messages are displayed within Consul:

`consul.fsm: EnsureRegistration failed: failed inserting node: Error while renaming Node ID: "e1d456bc-f72d-98e5-ebb3-26ae80d785cf": Node name node001 is reserved by node 05f10209-1b9c-b90c-e3e2-059e64556d4a with name node001`

While it is easy to find the node that has reserved the name, it is hard to find
the node trying to aquire the name since it is not registered, because it
is not part of `consul members` output

This PR will display the IP of the offender and solve far more easily those issues.
2019-08-28 15:57:05 -04:00
Alvin Huang e4e9381851
revert commits on master (#6413) 2019-08-27 17:45:58 -04:00
tradel 2838a1550a update tests to match new method signatures 2019-08-27 14:16:39 -07:00
tradel 93c839b76c confi\gure providers with DC and domain 2019-08-27 14:16:25 -07:00
tradel 1acde6e30a create a common name for autoTLS agent certs 2019-08-27 14:15:53 -07:00
tradel 82544b64e5 add subject names to issued certs 2019-08-27 14:15:10 -07:00
tradel 1c9b271731 construct a common name for each CSR 2019-08-27 14:12:56 -07:00
tradel 8c733260cd add serviceID to leaf cert request 2019-08-27 14:12:22 -07:00
tradel b0bbcd8b94 add domain and nodeName to agent cert request 2019-08-27 14:11:40 -07:00
tradel 3dc47a9251 Added DC and domain args to Configure method 2019-08-27 14:09:01 -07:00
R.B. Boyer 1b3d066f90
test: send testagent logs through testing.Logf (#6411) 2019-08-27 12:21:30 -05:00
R.B. Boyer 0c5409d172
test: fix TestAgent.Start() to not segfault if the DNSServer cannot ListenAndServe (#6409)
The embedded `Server` field on a `DNSServer` is only set inside of the
`ListenAndServe` method. If that method fails for reasons like the
address being in use and is not bindable, then the `Server` field will
not be set and the overall `Agent.Start()` will fail.

This will trigger the inner loop of `TestAgent.Start()` to invoke
`ShutdownEndpoints` which will attempt to pretty print the DNS servers
using fields on that inner `Server` field. Because it was never set,
this causes a nil pointer dereference and crashes the test.
2019-08-27 10:45:05 -05:00
Alvin Huang 9662b7c01a
add nil pointer check for pointer to ACLToken struct (#6407) 2019-08-27 11:23:28 -04:00
Hans Hasselberg 3314dfd4ec
make sure auto_encrypt has private key type and bits (#6392) 2019-08-27 14:37:56 +02:00
Hans Hasselberg dee5a4ac51
auto_encrypt: verify_incoming_rpc is good enough for auto_encrypt.allow_tls (#6376)
Previously `verify_incoming` was required when turning on `auto_encrypt.allow_tls`, but that doesn't work together with HTTPS UI in some scenarios. Adding `verify_incoming_rpc` to the allowed configurations.
2019-08-27 14:36:36 +02:00
R.B. Boyer 09ce7e1220
test: don't leak agent goroutines in TestAgent_sidecarServiceFromNodeService (#6396)
A goroutine dump using runtime.Stack() before/after shows a drop from 121 => 4.
2019-08-26 15:19:59 -05:00
Hans Hasselberg 4f7a3e8fa8 make sure auto_encrypt has private key type and bits 2019-08-26 13:09:50 +02:00
hashicorp-ci a3ac04526a
update bindata_assetfs.go 2019-08-23 22:10:50 +00:00
R.B. Boyer 2d4a3b51d0
Merge pull request #6388 from hashicorp/release/1-6
merging release/1-6 into master
2019-08-23 13:44:46 -05:00
Matt Keeler 89ac998e8b
Secondary CA `establishLeadership` fix (#6383)
This prevents ACL issues (or other issues) during intermediate CA cert signing from failing leader establishment.
2019-08-23 11:32:37 -04:00
Hans Hasselberg aada537d87
auto_encrypt: use server-port (#6287)
AutoEncrypt needs the server-port because it wants to talk via RPC. Information from gossip might not be available at that point and thats why the server-port is being used.
2019-08-23 10:18:46 +02:00
R.B. Boyer ea65298070
connect: allow 'envoy_cluster_json' escape hatch to continue to function (#6378) 2019-08-22 15:11:56 -05:00
R.B. Boyer e7b8032b48
docs: document how envoy escape hatches work with the discovery chain (#6350)
- Bootstrap escape hatches are OK.
- Public listener/cluster escape hatches are OK.
- Upstream listener/cluster escape hatches are not supported.

If an unsupported escape hatch is configured and the discovery chain is
activated log a warning and act like it was not configured.

Fixes #6160
2019-08-21 15:10:12 -05:00
Matt Keeler 8cb0560f52
Ensure that config entry writes are forwarded to the primary DC (#6339) 2019-08-20 12:01:13 -04:00
R.B. Boyer 94c473fa5f
connect: ensure time.Duration fields retain their human readable forms in the API (#6348)
This applies for both config entries and the compiled discovery chain.

Also omit some other config entries fields when empty.
2019-08-19 15:31:05 -05:00
R.B. Boyer 0675e0606e
connect: generate the full SNI names for discovery targets in the compiler rather than in the xds package (#6340) 2019-08-19 13:03:03 -05:00
R.B. Boyer d6456fddeb
connect: introduce ExternalSNI field on service-defaults (#6324)
Compiling this will set an optional SNI field on each DiscoveryTarget.
When set this value should be used for TLS connections to the instances
of the target. If not set the default should be used.

Setting ExternalSNI will disable mesh gateway use for that target. It also 
disables several service-resolver features that do not make sense for an 
external service.
2019-08-19 12:19:44 -05:00
R.B. Boyer f84f509ce4
connect: updating a service-defaults config entry should leave an unset protocol alone (#6342)
If the entry is updated for reasons other than protocol it is surprising
that the value is explicitly persisted as 'tcp' rather than leaving it
empty and letting it fall back dynamically on the proxy-defaults value.
2019-08-19 10:44:06 -05:00
Matt Keeler 73888eed36
Filter out left/leaving serf members when determining if new AC… (#6332) 2019-08-16 10:34:18 -04:00
R.B. Boyer 91df06098c
xds: improve how envoy metrics are emitted (#6312)
Since generated envoy clusters all are named using (mostly) SNI syntax
we can have envoy read the various fields out of that structure and emit
it as stats labels to the various telemetry backends.

I changed the delimiter for the 'customization hash' from ':' to '~'
because ':' is always reencoded by envoy as '_' when generating metrics
keys.
2019-08-16 09:30:17 -05:00
R.B. Boyer 22ee60d1ba
agent: blocking central config RPCs iterations should not interfere with each other (#6316) 2019-08-14 09:08:46 -05:00
hashicorp-ci 33cab6ae72
update bindata_assetfs.go 2019-08-13 15:28:06 +00:00
hashicorp-ci 29767157ed Merge Consul OSS branch 'master' at commit 8f7586b339dbb518eff3a2eec27d7b8eae7a3fbb 2019-08-13 02:00:43 +00:00
Sarah Adams 2f7a90bc52
add flag to allow /operator/keyring requests to only hit local servers (#6279)
Add parameter local-only to operator keyring list requests to force queries to only hit local servers (no WAN traffic).

HTTP API: GET /operator/keyring?local-only=true
CLI: consul keyring -list --local-only

Sending the local-only flag with any non-GET/list request will result in an error.
2019-08-12 11:11:11 -07:00
Mike Morris e637b48117
snapshot: add TLS support to HalfCloser interface (#6216)
Calls net.TCPConn.CloseWrite or mtls.Conn.CloseWrite, which was added in https://go-review.googlesource.com/c/go/+/31318/
2019-08-12 12:47:02 -04:00
Matt Keeler 0e1b7038cc
Update the v1/agent/service/:service endpoint to output tagged… (#6304) 2019-08-10 09:15:19 -04:00
R.B. Boyer 3b30dfd3ee
connect: allow mesh gateways to use central config (#6302) 2019-08-09 15:07:01 -05:00
Mike Morris 88df658243
connect: remove managed proxies (#6220)
* connect: remove managed proxies implementation and all supporting config options and structs

* connect: remove deprecated ProxyDestination

* command: remove CONNECT_PROXY_TOKEN env var

* agent: remove entire proxyprocess proxy manager

* test: remove all managed proxy tests

* test: remove irrelevant managed proxy note from TestService_ServerTLSConfig

* test: update ContentHash to reflect managed proxy removal

* test: remove deprecated ProxyDestination test

* telemetry: remove managed proxy note

* http: remove /v1/agent/connect/proxy endpoint

* ci: remove deprecated test exclusion

* website: update managed proxies deprecation page to note removal

* website: remove managed proxy configuration API docs

* website: remove managed proxy note from built-in proxy config

* website: add note on removing proxy subdirectory of data_dir
2019-08-09 15:19:30 -04:00
R.B. Boyer 357ca39868
connect: ensure intention replication continues to work when the replication ACL token changes (#6288) 2019-08-07 11:34:09 -05:00
hashicorp-ci 3ac803da5e Merge Consul OSS branch 'master' at commit d84863799deca45ccf4bec5ab9f645ccae6b3aeb 2019-08-06 02:00:30 +00:00
Sarah Adams 9ed3e64510
fallback to proxy config global protocol when upstream services' protocol is unset (#6277)
fallback to proxy config global protocol when upstream services' protocol is unset

Fixes #5857
2019-08-05 12:52:35 -07:00
R.B. Boyer 64fc002e03
connect: fix failover through a mesh gateway to a remote datacenter (#6259)
Failover is pushed entirely down to the data plane by creating envoy
clusters and putting each successive destination in a different load
assignment priority band. For example this shows that normally requests
go to 1.2.3.4:8080 but when that fails they go to 6.7.8.9:8080:

- name: foo
  load_assignment:
    cluster_name: foo
    policy:
      overprovisioning_factor: 100000
    endpoints:
    - priority: 0
      lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: 1.2.3.4
              port_value: 8080
    - priority: 1
      lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: 6.7.8.9
              port_value: 8080

Mesh gateways route requests based solely on the SNI header tacked onto
the TLS layer. Envoy currently only lets you configure the outbound SNI
header at the cluster layer.

If you try to failover through a mesh gateway you ideally would
configure the SNI value per endpoint, but that's not possible in envoy
today.

This PR introduces a simpler way around the problem for now:

1. We identify any target of failover that will use mesh gateway mode local or
   remote and then further isolate any resolver node in the compiled discovery
   chain that has a failover destination set to one of those targets.

2. For each of these resolvers we will perform a small measurement of
   comparative healths of the endpoints that come back from the health API for the
   set of primary target and serial failover targets. We walk the list of targets
   in order and if any endpoint is healthy we return that target, otherwise we
   move on to the next target.

3. The CDS and EDS endpoints both perform the measurements in (2) for the
   affected resolver nodes.

4. For CDS this measurement selects which TLS SNI field to use for the cluster
   (note the cluster is always going to be named for the primary target)

5. For EDS this measurement selects which set of endpoints will populate the
   cluster. Priority tiered failover is ignored.

One of the big downsides to this approach to failover is that the failover
detection and correction is going to be controlled by consul rather than
deferring that entirely to the data plane as with the prior version. This also
means that we are bound to only failover using official health signals and
cannot make use of data plane signals like outlier detection to affect
failover.

In this specific scenario the lack of data plane signals is ok because the
effectiveness is already muted by the fact that the ultimate destination
endpoints will have their data plane signals scrambled when they pass through
the mesh gateway wrapper anyway so we're not losing much.

Another related fix is that we now use the endpoint health from the
underlying service, not the health of the gateway (regardless of
failover mode).
2019-08-05 13:30:35 -05:00
R.B. Boyer 0165e93517
connect: expose an API endpoint to compile the discovery chain (#6248)
In addition to exposing compilation over the API cleaned up the structures that would be exchanged to be cleaner and easier to support and understand.

Also removed ability to configure the envoy OverprovisioningFactor.
2019-08-02 15:34:54 -05:00