Commit Graph

108 Commits

Author SHA1 Message Date
Todd Radel 19a3892f71 connect: Implement NeedsLogger interface for CA providers (#6556)
* add NeedsLogger to Provider interface

* implements NeedsLogger in default provider

* pass logger through to provider

* test for proper operation of NeedsLogger

* remove public testServer function

* Switch test to actually assert on logging output rather than reflection.

--amend

* Ooops actually set the logger in all the places we need it - CA config set wasn't and causing segfault

* Fix all the other places in tests where we set the logger

* Add TODO comment
2019-11-11 20:30:01 +00:00
Todd Radel e100fda218 Make all Connect Cert Common Names valid FQDNs (#6423) 2019-11-11 17:11:54 +00:00
Paul Banks 5f405c3277
Fix support for RSA CA keys in Connect. (#6638)
* Allow RSA CA certs for consul and vault providers to correctly sign EC leaf certs.

* Ensure key type ad bits are populated from CA cert and clean up tests

* Add integration test and fix error when initializing secondary CA with RSA key.

* Add more tests, fix review feedback

* Update docs with key type config and output

* Apply suggestions from code review

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>
2019-11-01 13:20:26 +00:00
Matt Keeler 29f0616708
Use encoding/json instead of jsonpb even for protobuf types (#6572)
This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 15:32:15 -04:00
Matt Keeler cfa879d63c
Generate JSON and Binary Marshalers for Protobuf Types (#6564)
* Add JSON and Binary Marshaler Generators for Protobuf Types

* Generate files with the correct version of gogo/protobuf

I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date.

The version at the time of this commit we are using is v1.2.1

* Fixup some shell output

* Update how we determine the version of gogo
This just greps the go.mod file instead of expecting the go mod cache to already be present

* Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 15:39:20 -04:00
R.B. Boyer cc889443a5
connect: don't colon-hex-encode the AuthorityKeyId and SubjectKeyId fields in connect certs (#6492)
The fields in the certs are meant to hold the original binary
representation of this data, not some ascii-encoded version.

The only time we should be colon-hex-encoding fields is for display
purposes or marshaling through non-TLS mediums (like RPC).
2019-09-23 12:52:35 -05:00
R.B. Boyer 1d54909333
connect: intermediate CA certs generated with the vault provider lack URI SANs (#6491)
This only affects vault versions >=1.1.1 because the prior code
accidentally relied upon a bug that was fixed in
https://github.com/hashicorp/vault/pull/6505

The existing tests should have caught this, but they were using a
vendored copy of vault version 0.10.3. This fixes the tests by running
an actual copy of vault instead of an in-process copy. This has the
added benefit of changing the dependency on vault to just vault/api.

Also update VaultProvider to use similar SetIntermediate validation code
as the ConsulProvider implementation.
2019-09-23 12:04:40 -05:00
Matt Keeler 8431c5f533
Add support for implementing new requests with protobufs instea… (#6502)
* Add build system support for protobuf generation

This is done generically so that we don’t have to keep updating the makefile to add another proto generation.

Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date.

If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild>

Providing the PROTOFILES var will override the default behavior of finding all the .proto files.

* Start adding types to the agent/proto package

These will be needed for some other work and are by no means comprehensive.

* Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store.

* Use protobuf marshalling of raft requests instead of msgpack for protoc generated types.

This does not change any encoding of existing types.

* Removed structs package automatically encoding with protobuf marshalling

Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf`

* Run update-vendor to fixup modules.txt

Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical.

* Rename some things and implement the structs.RPCInfo interface bits

agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth.

* Use the right encoding function.

* Renamed agent/proto package to agent/agentpb to prevent package name conflicts

* Update modules.txt to fix ordering

* Change blockingQuery to take in interfaces for the query options and meta

* Add %T to error output.

* Add/Update some comments
2019-09-20 14:37:22 -04:00
Alvin Huang e4e9381851
revert commits on master (#6413) 2019-08-27 17:45:58 -04:00
tradel 82544b64e5 add subject names to issued certs 2019-08-27 14:15:10 -07:00
tradel 3dc47a9251 Added DC and domain args to Configure method 2019-08-27 14:09:01 -07:00
R.B. Boyer 0675e0606e
connect: generate the full SNI names for discovery targets in the compiler rather than in the xds package (#6340) 2019-08-19 13:03:03 -05:00
Paul Banks a5c70d79d0 Revert "connect: support AWS PCA as a CA provider" (#6251)
This reverts commit 3497b7c00d49c4acbbf951d84f2bba93f3da7510.
2019-07-31 09:08:10 -04:00
Todd Radel d3b7fd83fe
connect: support AWS PCA as a CA provider (#6189)
Port AWS PCA provider from consul-ent
2019-07-30 22:57:51 -04:00
Todd Radel 1b14d6595e
connect: Support RSA keys in addition to ECDSA (#6055)
Support RSA keys in addition to ECDSA
2019-07-30 17:47:39 -04:00
Christian Muehlhaeuser 2602f6907e Simplified code in various places (#6176)
All these changes should have no side-effects or change behavior:

- Use bytes.Buffer's String() instead of a conversion
- Use time.Since and time.Until where fitting
- Drop unnecessary returns and assignment
2019-07-20 09:37:19 -04:00
Hans Hasselberg 73c4e9f07c
tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597) 2019-06-27 22:22:07 +02:00
R.B. Boyer 91e78e00c7
fix typos reported by golangci-lint:misspell (#5434) 2019-03-06 11:13:28 -06:00
R.B. Boyer 23f824fb60 test: fix concurrent map access when setting up test vault 2019-03-01 14:30:19 -06:00
R.B. Boyer 28b87063e3 fix a few leap-year related clock math inaccuracies and failing tests 2019-03-01 13:51:49 -06:00
Kyle Havlovitz 51d5d6b1d4
connect/ca: fix a potential panic in the Consul provider 2019-02-07 10:43:54 -08:00
Kyle Havlovitz 5a5436380b
connect/ca: return a better error message if the CA isn't fully initialized when signing 2019-01-22 11:15:09 -08:00
Hans Hasselberg 092907077d
connect: add tls config for vault connect ca provider (#5125)
* add tlsconfig for vault connect ca provider.
* add options to the docs
* add tests for new configuration
2019-01-08 17:09:22 +01:00
Mitchell Hashimoto 5452c32f50 CA Provider Plugins (#4751)
This adds the `agent/connect/ca/plugin` library for consuming/serving Connect CA providers as [go-plugin](https://github.com/hashicorp/go-plugin) plugins. This **does not** wire this up in any way to Consul itself, so this will not enable using these plugins yet. 

## Why?

We want to enable CA providers to be pluggable without modifying Consul so that any CA or PKI system can potentially back the Connect certificates. This CA system may also be used in the future for easier bootstrapping and internal cluster security.

### go-plugin

The benefit of `go-plugin` is that for the plugin consumer, the fact that the interface implementation is communicating over multi-process RPC is invisible. Internals of Consul will continue to just use `ca.Provider` interface implementations as if they're local. For plugin _authors_, they simply have to implement the interface. The network/transport/process management issues are handled by go-plugin itself.

The CA provider plugins support both `net/rpc` and gRPC transports. This enables easy authoring in any language. go-plugin handles the actual protocol handshake and connection. This is just a feature of go-plugin. 

`go-plugin` is already in production use for years by Packer, Terraform, Nomad, Vault, and Sentinel. We've shown stability for both desktop and server-side software. It is very mature.

## Implementation Details

### `map[string]interface{}`

The `Configure` method passes a `map[string]interface{}`. This map contains only Go primitives and containers of primitives (no funcs, chans, etc.). For `net/rpc` we encode as-is using Gob. For gRPC we marshal to JSON and transmit as a `bytes` type. This is the same approach we take with Vault and other software.

Note that this is just the transport protocol, the end software views it fully decoded.

### `x509.Certificate` and `CertificateRequest`

We transmit the raw ASN.1  bytes and decode on the other side. Unit tests are verifying we get the same cert/csrs across the wire.

### Testing

`go-plugin` exposes test helpers that enable testing the full plugin RPC over real loopback network connections. We test all endpoints for success and error for both `net/rpc` and gRPC.

### Vendoring

This PR doesn't introduce vendoring for two reasons:

  1. @banks's `f-envoy` branch introduces a lot of these and I didn't want conflict.
  2. The library isn't actually used yet so it doesn't introduce compile-time errors (it does introduce test errors).

## Next Steps

With this in place, we need to figure out the proper way to actually hook these up to Consul, load them, etc. This discussion can happen elsewhere, since regardless of approach this plugin library implementation is the exact same.
2019-01-07 12:48:44 -05:00
Kyle Havlovitz 1a4204f363
agent: fix formatting 2018-11-07 02:16:03 -08:00
Aestek 260a9880ae [Security] Add finer control over script checks (#4715)
* Add -enable-local-script-checks options

These options allow for a finer control over when script checks are enabled by
giving the option to only allow them when they are declared from the local
file system.

* Add documentation for the new option

* Nitpick doc wording
2018-10-11 13:22:11 +01:00
Kyle Havlovitz 6301f763df connect/ca: tighten up the intermediate signing verification 2018-09-14 16:08:54 -07:00
Kyle Havlovitz ba1d7201a0 connect/ca: add intermediate functions to Vault ca provider 2018-09-13 13:38:32 -07:00
Kyle Havlovitz 138a39026b connect/ca: add intermediate functions to Consul CA provider 2018-09-13 13:09:21 -07:00
Kyle Havlovitz 9b8f8975c6
Merge pull request #4644 from hashicorp/ca-refactor
connect/ca: rework initialization/root generation in providers
2018-09-13 13:08:34 -07:00
Paul Banks 09e4c2995b
Fix CA pruning when CA config uses string durations. (#4669)
* Fix CA pruning when CA config uses string durations.

The tl;dr here is:

 - Configuring LeafCertTTL with a string like "72h" is how we do it by default and should be supported
 - Most of our tests managed to escape this by defining them as time.Duration directly
 - Out actual default value is a string
 - Since this is stored in a map[string]interface{} config, when it is written to Raft it goes through a msgpack encode/decode cycle (even though it's written from server not over RPC).
 - msgpack decode leaves the string as a `[]uint8`
 - Some of our parsers required string and failed
 - So after 1 hour, a default configured server would throw an error about pruning old CAs
 - If a new CA was configured that set LeafCertTTL as a time.Duration, things might be OK after that, but if a new CA was just configured from config file, intialization would cause same issue but always fail still so would never prune the old CA.
 - Mostly this is just a janky error that got passed tests due to many levels of complicated encoding/decoding.

tl;dr of the tl;dr: Yay for type safety. Map[string]interface{} combined with msgpack always goes wrong but we somehow get bitten every time in a new way :D

We already fixed this once! The main CA config had the same problem so @kyhavlov already wrote the mapstructure DecodeHook that fixes it. It wasn't used in several places it needed to be and one of those is notw in `structs` which caused a dependency cycle so I've moved them.

This adds a whole new test thta explicitly tests the case that broke here. It also adds tests that would have failed in other places before (Consul and Vaul provider parsing functions). I'm not sure if they would ever be affected as it is now as we've not seen things broken with them but it seems better to explicitly test that and support it to not be bitten a third time!

* Typo fix

* Fix bad Uint8 usage
2018-09-13 15:43:00 +01:00
Kyle Havlovitz 70c43a27c3 connect/ca: hash the consul provider ID and include isRoot 2018-09-12 13:44:15 -07:00
Kyle Havlovitz 8fc2c77fdf
connect/ca: some cleanup and reorganizing of the new methods 2018-09-11 16:43:04 -07:00
Kyle Havlovitz e184a18e4b
connect/ca: add Configure/GenerateRoot to provider interface 2018-09-06 19:18:59 -07:00
Siva Prasad cfa436dc16
Revert "CA initialization while boostrapping and TestLeader_ChangeServerID fix." (#4497)
* Revert "BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472)"

This reverts commit cec5d7239621e0732b3f70158addb1899442acb3.

* Revert "CA initialization while boostrapping and TestLeader_ChangeServerID fix. (#4493)"

This reverts commit 589b589b53e56af38de25db9b56967bdf1f2c069.
2018-08-07 08:29:48 -04:00
Siva Prasad 29c181f5fa
CA initialization while boostrapping and TestLeader_ChangeServerID fix. (#4493)
* connect: fix an issue with Consul CA bootstrapping being interrupted

* streamline change server id test
2018-08-06 16:15:24 -04:00
Kyle Havlovitz 68d7a9fbd3
connect/ca: simplify passing of leaf cert TTL 2018-07-25 17:51:45 -07:00
Kyle Havlovitz a125735d76
connect/ca: check LeafCertTTL when rotating expired roots 2018-07-20 16:04:04 -07:00
Kyle Havlovitz 45ec8849f3
connect/ca: add configurable leaf cert TTL 2018-07-16 13:33:37 -07:00
Matt Keeler b3ba709b3d Remove x509 name constraints
These were only added as SPIFFE intends to use the in the future but currently does not mandate their usage due to patch support in common TLS implementations and some ambiguity over how to use them with URI SAN certificates. We included them because until now everything seem fine with it, however we've found the latest version of `openssl` (1.1.0h) fails to validate our certificats if its enabled. LibreSSL as installed on OS X by default doesn’t have these issues. For now it's most compatible not to have them and later we can find ways to add constraints with wider compatibility testing.
2018-06-25 12:26:10 -07:00
Kyle Havlovitz a67bfa2c1b connect/ca: use weak type decoding in the Vault config parsing 2018-06-25 12:25:42 -07:00
Kyle Havlovitz f3089a6647 connect/ca: undo the interface changes and use sign-self-issued in Vault 2018-06-25 12:25:42 -07:00
Kyle Havlovitz f79e3e3fa5 connect/ca: add leaf verify check to cross-signing tests 2018-06-25 12:25:41 -07:00
Kyle Havlovitz cea94d0bcf connect/ca: update Consul provider to use new cross-sign CSR method 2018-06-25 12:25:41 -07:00
Kyle Havlovitz 675555c4ff connect/ca: update Vault provider to add cross-signing methods 2018-06-25 12:25:41 -07:00
Kyle Havlovitz a97c44c1ba connect/ca: add URI SAN support to the Vault provider 2018-06-25 12:25:41 -07:00
Kyle Havlovitz 7b0845ccde connect/ca: fix vault provider URI SANs and test 2018-06-25 12:25:41 -07:00
Kyle Havlovitz a98b85b25c connect/ca: add the Vault CA provider 2018-06-25 12:25:41 -07:00
Paul Banks 6ecc0c8099 Sign certificates valid from 1 minute earlier to avoid failures caused by clock drift 2018-06-25 12:25:41 -07:00
Paul Banks 824a9b4943 Actually return Intermediate certificates bundled with a leaf! 2018-06-25 12:25:40 -07:00
Kyle Havlovitz 54bc937fed
Re-use uint8ToString 2018-06-14 09:42:23 -07:00
Kyle Havlovitz 4d46bba2c4
Support giving the duration as a string in CA config 2018-06-14 09:42:22 -07:00
Paul Banks 30d90b3be4
Generate CSR using real trust-domain 2018-06-14 09:42:16 -07:00
Paul Banks c808833a78
Return TrustDomain from CARoots RPC 2018-06-14 09:42:15 -07:00
Kyle Havlovitz d1265bc38b
Rename some of the CA structs/files 2018-06-14 09:42:15 -07:00
Kyle Havlovitz baf4db1c72
Use provider state table for a global serial index 2018-06-14 09:42:15 -07:00
Kyle Havlovitz 5998623c44
Add test for ca config http endpoint 2018-06-14 09:42:15 -07:00
Kyle Havlovitz c90b353eea
Move connect CA provider to separate package 2018-06-14 09:42:15 -07:00