Commit Graph

2316 Commits

Author SHA1 Message Date
Chris S. Kim ffa7d737af
Refactor some functions for better enterprise use (#13280) 2022-05-30 09:46:55 -04:00
Chris S. Kim e1b376f377
Add build tag for oss (#13279) 2022-05-27 11:39:58 -04:00
Mathew Estafanous 0481a99a70
Replace CLI command registry with a new pattern. (#12729) 2022-05-27 11:33:27 -04:00
DanStough 2c8ca25d8a chore(test): Update bats version 2022-05-24 11:56:08 -04:00
Mark Anderson 1497421b65
Merge pull request #12878 from hashicorp/ma/x-forwarded-client-cert
Support x-forwarded-client-cert
2022-05-04 11:05:44 -07:00
Dan Upton 6bfdb48560
acl: gRPC login and logout endpoints (#12935)
Introduces two new public gRPC endpoints (`Login` and `Logout`) and
includes refactoring of the equivalent net/rpc endpoints to enable the
majority of logic to be reused (i.e. by extracting the `Binder` and
`TokenWriter` types).

This contains the OSS portions of the following enterprise commits:

- 75fcdbfcfa6af21d7128cb2544829ead0b1df603
- bce14b714151af74a7f0110843d640204082630a
- cc508b70fbf58eda144d9af3d71bd0f483985893
2022-05-04 17:38:45 +01:00
Mark Anderson db0c61303f Update mesh config tests
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-05-04 08:50:59 -07:00
R.B. Boyer c949de9f50
structs: ensure exported-services PeerName field can be addressed as peer_name (#12862) 2022-04-27 10:27:21 -05:00
Evan Culver 9d0b5bf8e9
connect: Add Envoy 1.22 to integration tests, remove Envoy 1.18 (#12805)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-04-18 09:36:07 -07:00
Kyle Havlovitz 199f1c7200
Fix namespace default field names in expanded token output 2022-04-13 16:46:39 -07:00
Mark Anderson ed3e42296d Fixup acl.EnterpriseMeta
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-04-05 15:11:49 -07:00
R.B. Boyer 12523197f9
syncing changes back from enterprise (#12701) 2022-04-05 15:46:56 -05:00
Kyle Havlovitz c2a48990cf
Merge pull request #12670 from hashicorp/token-read-expanded
oss: Add expanded token read flag and endpoint option
2022-03-31 12:24:11 -07:00
Dhia Ayachi 8552efa955
auto-reload configuration when config files change (#12329)
* add config watcher to the config package

* add logging to watcher

* add test and refactor to add WatcherEvent.

* add all API calls and fix a bug with recreated files

* add tests for watcher

* remove the unnecessary use of context

* Add debug log and a test for file rename

* use inode to detect if the file is recreated/replaced and only listen to create events.

* tidy ups (#1535)

* tidy ups

* Add tests for inode reconcile

* fix linux vs windows syscall

* fix linux vs windows syscall

* fix windows compile error

* increase timeout

* use ctime ID

* remove remove/creation test as it's a use case that fail in linux

* fix linux/windows to use Ino/CreationTime

* fix the watcher to only overwrite current file id

* fix linter error

* fix remove/create test

* set reconcile loop to 200 Milliseconds

* fix watcher to not trigger event on remove, add more tests

* on a remove event try to add the file back to the watcher and trigger the handler if success

* fix race condition

* fix flaky test

* fix race conditions

* set level to info

* fix when file is removed and get an event for it after

* fix to trigger handler when we get a remove but re-add fail

* fix error message

* add tests for directory watch and fixes

* detect if a file is a symlink and return an error on Add

* rename Watcher to FileWatcher and remove symlink deref

* add fsnotify@v1.5.1

* fix go mod

* do not reset timer on errors, rename OS specific files

* rename New func

* events trigger on write and rename

* add missing test

* fix flaking tests

* fix flaky test

* check reconcile when removed

* delete invalid file

* fix test to create files with different mod time.

* back date file instead of sleeping

* add watching file in agent command.

* fix watcher call to use new API

* add configuration and stop watcher when server stop

* add certs as watched files

* move FileWatcher to the agent start instead of the command code

* stop watcher before replacing it

* save watched files in agent

* add add and remove interfaces to the file watcher

* fix remove to not return an error

* use `Add` and `Remove` to update certs files

* fix tests

* close events channel on the file watcher even when the context is done

* extract `NotAutoReloadableRuntimeConfig` is a separate struct

* fix linter errors

* add Ca configs and outgoing verify to the not auto reloadable config

* add some logs and fix to use background context

* add tests to auto-config reload

* remove stale test

* add tests to changes to config files

* add check to see if old cert files still trigger updates

* rename `NotAutoReloadableRuntimeConfig` to `StaticRuntimeConfig`

* fix to re add both key and cert file. Add test to cover this case.

* review suggestion

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add check to static runtime config changes

* fix test

* add changelog file

* fix review comments

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* update flag description

Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>

* fix compilation error

* add static runtime config support

* fix test

* fix review comments

* fix log test

* Update .changelog/12329.txt

Co-authored-by: Dan Upton <daniel@floppy.co>

* transfer tests to runtime_test.go

* fix filewatcher Replace to not deadlock.

* avoid having lingering locks

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* split ReloadConfig func

* fix warning message

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* convert `FileWatcher` into an interface

* fix compilation errors

* fix tests

* extract func for adding and removing files

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>
Co-authored-by: Daniel Upton <daniel@floppy.co>
2022-03-31 15:11:49 -04:00
Kyle Havlovitz 9a2474381a Add expanded token read flag and endpoint option 2022-03-31 10:49:49 -07:00
Paul Glass aae6d8080d
Add IAM Auth Method (#12583)
This adds an aws-iam auth method type which supports authenticating to Consul using AWS IAM identities.

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-03-31 10:18:48 -05:00
R.B. Boyer e9230e93d8
xds: adding control of the mesh-wide min/max TLS versions and cipher suites from the mesh config entry (#12601)
- `tls.incoming`: applies to the inbound mTLS targeting the public
  listener on `connect-proxy` and `terminating-gateway` envoy instances

- `tls.outgoing`: applies to the outbound mTLS dialing upstreams from
  `connect-proxy` and `ingress-gateway` envoy instances

Fixes #11966
2022-03-30 13:43:59 -05:00
Dan Upton 57f0f42733
Support per-listener TLS configuration ⚙️ (#12504)
Introduces the capability to configure TLS differently for Consul's
listeners/ports (i.e. HTTPS, gRPC, and the internal multiplexed RPC
port) which is useful in scenarios where you may want the HTTPS or
gRPC interfaces to present a certificate signed by a well-known/public
CA, rather than the certificate used for internal communication which
must have a SAN in the form `server.<dc>.consul`.
2022-03-18 10:46:58 +00:00
R.B. Boyer a97d20cf63
catalog: compare node names case insensitively in more places (#12444)
Many places in consul already treated node names case insensitively.
The state store indexes already do it, but there are a few places that
did a direct byte comparison which have now been corrected.

One place of particular consideration is ensureCheckIfNodeMatches
which is executed during snapshot restore (among other places). If a
node check used a slightly different casing than the casing of the node
during register then the snapshot restore here would deterministically
fail. This has been fixed.

Primary approach:

    git grep -i "node.*[!=]=.*node" -- ':!*_test.go' ':!docs'
    git grep -i '\[[^]]*member[^]]*\]
    git grep -i '\[[^]]*\(member\|name\|node\)[^]]*\]' -- ':!*_test.go' ':!website' ':!ui' ':!agent/proxycfg/testing.go:' ':!*.md'
2022-02-24 16:54:47 -06:00
Daniel Nephin c6993bda15 debug: update CLI docs
To clarify how trace is captured.

Also remove the minimum seconds check, because that is already done in prepare()
2022-02-15 18:16:12 -05:00
Daniel Nephin 5bd73fc218 debug: limit the size of the trace
We've noticed that a trace that is captured over the full duration is
too large to open on most machines. A trace.out captured over just the
interval period (30s by default) should be a more than enough time to
capture trace data.
2022-02-15 14:15:34 -05:00
FFMMM 1f8fb17be7
Vendor in rpc mono repo for net/rpc fork, go-msgpack, msgpackrpc. (#12311)
This commit syncs ENT changes to the OSS repo.

Original commit details in ENT:

```
commit 569d25f7f4578981c3801e6e067295668210f748
Author: FFMMM <FFMMM@users.noreply.github.com>
Date:   Thu Feb 10 10:23:33 2022 -0800

    Vendor fork net rpc (#1538)

    * replace net/rpc w consul-net-rpc/net/rpc

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

    * replace msgpackrpc and go-msgpack with fork from mono repo

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

    * gofmt all files touched

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
```

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2022-02-14 09:45:45 -08:00
R.B. Boyer 05c7373a28 bulk rewrite using this script
set -euo pipefail

    unset CDPATH

    cd "$(dirname "$0")"

    for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do
        echo "=== require: $f ==="
        sed -i '/require := require.New(t)/d' $f
        # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah)
        sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f
        # require.XXX(tblah) but not require.XXX(t, blah)
        sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f
        # require.XXX(rblah) but not require.XXX(r, blah)
        sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f
        gofmt -s -w $f
    done

    for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do
        echo "=== assert: $f ==="
        sed -i '/assert := assert.New(t)/d' $f
        # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah)
        sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f
        # assert.XXX(tblah) but not assert.XXX(t, blah)
        sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f
        # assert.XXX(rblah) but not assert.XXX(r, blah)
        sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f
        gofmt -s -w $f
    done
2022-01-20 10:46:23 -06:00
R.B. Boyer c12b0ee3d2 test: normalize require.New and assert.New syntax 2022-01-20 10:45:56 -06:00
Mike Morris 277c41d336
ingress: allow setting TLS min version and cipher suites in ingress gateway config entries (#11576)
* xds: refactor ingress listener SDS configuration

* xds: update resolveListenerSDS call args in listeners_test

* ingress: add TLS min, max and cipher suites to GatewayTLSConfig

* xds: implement envoyTLSVersions and envoyTLSCipherSuites

* xds: merge TLS config

* xds: configure TLS parameters with ingress TLS context from leaf

* xds: nil check in resolveListenerTLSConfig validation

* xds: nil check in makeTLSParameters* functions

* changelog: add entry for TLS params on ingress config entries

* xds: remove indirection for TLS params in TLSConfig structs

* xds: return tlsContext, nil instead of ambiguous err

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>

* xds: switch zero checks to types.TLSVersionUnspecified

* ingress: add validation for ingress config entry TLS params

* ingress: validate listener TLS config

* xds: add basic ingress with TLS params tests

* xds: add ingress listeners mixed TLS min version defaults precedence test

* xds: add more explicit tests for ingress listeners inheriting gateway defaults

* xds: add test for single TLS listener on gateway without TLS defaults

* xds: regen golden files for TLSVersionInvalid zero value, add TLSVersionAuto listener test

* types/tls: change TLSVersion to string

* types/tls: update TLSCipherSuite to string type

* types/tls: implement validation functions for TLSVersion and TLSCipherSuites, make some maps private

* api: add TLS params to GatewayTLSConfig, add tests

* api: add TLSMinVersion to ingress gateway config entry test JSON

* xds: switch to Envoy TLS cipher suite encoding from types package

* xds: fixup validation for TLSv1_3 min version with cipher suites

* add some kitchen sink tests and add a missing struct tag

* xds: check if mergedCfg.TLSVersion is in TLSVersionsWithConfigurableCipherSuites

* xds: update connectTLSEnabled comment

* xds: remove unsued resolveGatewayServiceTLSConfig function

 * xds: add makeCommonTLSContextFromLeafWithoutParams

* types/tls: add LessThan comparator function for concrete values

* types/tls: change tlsVersions validation map from string to TLSVersion keys

* types/tls: remove unused envoyTLSCipherSuites

* types/tls: enable chacha20 cipher suites for Consul agent

* types/tls: remove insecure cipher suites from allowed config

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 are both explicitly listed as insecure and disabled in the Go source.

Refs https://cs.opensource.google/go/go/+/refs/tags/go1.17.3:src/crypto/tls/cipher_suites.go;l=329-330

* types/tls: add ValidateConsulAgentCipherSuites function, make direct lookup map private

* types/tls: return all unmatched cipher suites in validation errors

* xds: check that Envoy API value matching TLS version is found when building TlsParameters

* types/tls: check that value is found in map before appending to slice in MarshalEnvoyTLSCipherSuiteStrings

* types/tls: cast to string rather than fmt.Printf in TLSCihperSuite.String()

* xds: add TLSVersionUnspecified to list of configurable cipher suites

* structs: update note about config entry warning

* xds: remove TLS min version cipher suite unconfigurable test placeholder

* types/tls: update tests to remove assumption about private map values

Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-01-11 11:46:42 -05:00
Daniel Nephin fd084c15c3 cli: use file mode 0600 when saving a snapshot
So that other users on the machine can not access the snapshot data.
2022-01-05 17:51:03 -05:00
Blake Covarrubias 4a36e4ee24
cli: Show node identities in acl token list output (#11926)
Fix the pretty CLI output of `consul acl token list` so that it
properly displays node identities that are associated with a token.
2022-01-04 12:44:43 -08:00
Daniel Nephin de0b51ae8c
Merge pull request #11884 from assareh/patch-1
consul pls cert create usage example provided in CLI help shows outdated arguments
2022-01-04 15:17:32 -05:00
Daniel Nephin f3c385391a
Merge pull request #11781 from marco-m/private-key-0600-permission
cli: consul tls: create private keys with mode 0600
2021-12-21 17:29:14 -05:00
Andy Assareh 2182a6f0a3 usage example given uses outdated arguments
the usage example shows incorrect arguments for ca cert and key. the correct arguments are now -ca and -key
2021-12-21 13:20:59 -08:00
freddygv 6f09051ded Update golden files 2021-12-14 08:57:09 -07:00
freddygv 9d4c8db0d7 Get partition label from upstream metrics 2021-12-14 08:40:47 -07:00
Evan Culver db7c814722
connect: update SNI label extraction to support new taxonomy for partitions (#11786) 2021-12-10 10:26:22 -08:00
Chris S. Kim ead530bc48
Add partitions to prettyformatters (#11789) 2021-12-09 15:58:45 -05:00
Marco Molteni db5f4eaadc cli: consul tls: create private keys with mode 0600
This applies to

consul tls ca create
consul tls cert create -client
consul tls cert create -server

Closes: #11741
2021-12-08 20:16:36 +01:00
Dan Upton 4192468358
Remove references to "master" ACL tokens in tests (#11751) 2021-12-07 12:48:50 +00:00
freddygv 7c326d2a0c Update api module and decoding tests 2021-12-06 12:32:29 -07:00
freddygv 02fb323652 Rename partition-exports to exported-services
Using a name less tied to partitions gives us more flexibility to use
this config entry in OSS for exports between datacenters/meshes.
2021-12-03 17:47:31 -07:00
R.B. Boyer 6ec84cfbe2
agent: add variation of force-leave that exclusively works on the WAN (#11722)
Fixes #6548
2021-12-02 17:15:10 -06:00
Daniel Nephin a0160f7426
Merge pull request #11677 from hashicorp/dnephin/freeport-interface
sdk: use t.Cleanup in freeport and remove unnecessary calls
2021-12-02 15:58:41 -05:00
Dan Upton eff3dc09b6
Rename `agent_master` ACL token in the API and CLI (#11669) 2021-12-02 17:05:27 +00:00
Daniel Nephin 4f0d092c95 testing: remove unnecessary calls to freeport
Previously we believe it was necessary for all code that required ports
to use freeport to prevent conflicts.

https://github.com/dnephin/freeport-test shows that it is actually save
to use port 0 (`127.0.0.1:0`) as long as it is passed directly to
`net.Listen`, and the listener holds the port for as long as it is
needed.

This works because freeport explicitly avoids the ephemeral port range,
and port 0 always uses that range. As you can see from the test output
of https://github.com/dnephin/freeport-test, the two systems never use
overlapping ports.

This commit converts all uses of freeport that were being passed
directly to a net.Listen to use port 0 instead. This allows us to remove
a bit of wrapping we had around httptest, in a couple places.
2021-11-29 12:19:43 -05:00
Daniel Nephin 59a7fb5ca9 testing: use httptest with freeport 2021-11-29 12:01:29 -05:00
Daniel Nephin 2cf41e4dc8 go-sso: remove returnFunc now that freeport handles return 2021-11-27 15:29:38 -05:00
R.B. Boyer 83bf7ab3ff
re-run gofmt on 1.17 (#11579)
This should let freshly recompiled golangci-lint binaries using Go 1.17
pass 'make lint'
2021-11-16 12:04:01 -06:00
R.B. Boyer 086ff42b56
partitions: various refactors to support partitioning the serf LAN pool (#11568) 2021-11-15 09:51:14 -06:00
freddygv e9d9814360 Fix test 2021-11-12 14:59:29 -07:00
freddygv 29cfc23a27 Support partitions in connect expose cmd 2021-11-12 14:45:32 -07:00
freddygv e71e5efa5c Fixup shared oss/ent tests 2021-11-12 14:34:31 -07:00
Nitya Dhanushkodi d396faf246
command/redirect_traffic: Redirect DNS requests to Consul if -consul-dns-ip is passed in (#11480)
* command/redirect_traffic: add rules to redirect DNS to Consul. Currently uses a hack to get the consul dns service ip, and this hack only works when the service is deployed in the same namespace as consul.
* command/redirect_traffic: redirect DNS to Consul when -consul-dns-ip is passed in
* Add unit tests to Consul DNS IP table redirect rules

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
2021-11-10 09:10:48 -05:00
Dhia Ayachi dfafd4e38c
KV refactoring, part 2 (#11512)
* add partition to the kv get pretty print

* fix failing test

* add test for kvs RPC endpoint
2021-11-08 11:43:21 -05:00
Daniel Upton a620b6be2e
Support Check-And-Set deletion of config entries (#11419)
Implements #11372
2021-11-01 16:42:01 +00:00
R.B. Boyer 2353d59413
cli: update consul members output to display partitions and sort the results usefully (#11446) 2021-10-28 17:27:31 -05:00
R.B. Boyer d40d098321
agent: for various /v1/agent endpoints parse the partition parameter on the request (#11444)
Also update the corresponding CLI commands to send the parameter
appropriately.

NOTE: Behavioral changes are not happening in this PR.
2021-10-28 16:44:38 -05:00
Freddy 8e23a6a0cc
Merge pull request #11416 from hashicorp/ap/exports-update
Rename service-exports to partition-exports
2021-10-27 11:27:31 -06:00
R.B. Boyer e27e58c6cc
agent: refactor the agent delegate interface to be partition friendly (#11429) 2021-10-26 15:08:55 -05:00
freddygv c3e381b4c1 Rename service-exports to partition-exports
Existing config entries prefixed by service- are specific to individual
services. Since this config entry applies to partitions it is being
renamed.

Additionally, the Partition label was changed to Name because using
Partition at the top-level and in the enterprise meta was leading to the
enterprise meta partition being dropped by msgpack.
2021-10-25 17:58:48 -06:00
Kyle Havlovitz 752a285552 Add new service-exports config entry 2021-10-20 12:24:18 -07:00
Oleg Butuzov 942c91b965 refactor: replace (*bytes.Buffer).WriteString with (*bytes.Buffer).Write
This PR change one method of bytes.Buffer struct package with a similar
one, as result - code produce less allocations on heap.
2021-10-20 09:36:53 +03:00
Evan Culver 9e949abe3e
connect: Remove envoy_version from bootstrap template (#11215) 2021-10-11 19:18:56 -07:00
Evan Culver d279c60010
Merge pull request #11115 from hashicorp/eculver/envoy-1.19.1
Add support for Envoy 1.19.1
2021-10-04 23:13:26 +02:00
Daniel Nephin bd28d23b55 command/envoy: stop using the DebugConfig from Self endpoint
The DebugConfig in the self endpoint can change at any time. It's not a stable API.

This commit adds the XDSPort to a stable part of the XDS api, and changes the envoy command to read
this new field.

It includes support for the old API as well, in case a newer CLI is used with an older API, and
adds a test for both cases.
2021-09-29 13:21:28 -04:00
Daniel Nephin 402d3792b6 Revert "Merge pull request #10588 from hashicorp/dnephin/config-fix-ports-grpc"
This reverts commit 74fb650b6b966588f8faeec26935a858af2b8bb5, reversing
changes made to 58bd8173364effb98b9fd9f9b98d31dd887a9bac.
2021-09-29 12:28:41 -04:00
Evan Culver 807871224a
Merge branch 'main' into eculver/envoy-1.19.1 2021-09-28 15:58:20 -07:00
Bisakh 85b2855cdf
acl: update GetPolicyByName method implementation (#11055) 2021-09-28 10:46:27 -04:00
Evan Culver 080e8d2c90
regenerate more envoy golden files 2021-09-22 09:30:23 -07:00
freddygv 3eab89cc2d Fixup api config and Envoy test 2021-09-15 17:25:21 -06:00
freddygv c5d5b44faa Bring back entmeta args defaulting 2021-09-15 17:23:52 -06:00
freddygv 3906d0b973 Ensure Envoy can subscribe to non-default partition 2021-09-15 17:23:48 -06:00
Freddy 11672defaf
connect: update envoy supported versions to latest patch release
(#10961)

Relevant advisory: 
https://github.com/envoyproxy/envoy/security/advisories/GHSA-6g4j-5vrw-2m8h
2021-08-31 10:39:18 -06:00
Dhia Ayachi 8e2de60b87
add partition flag to catalog commands (#10949)
* add partition flag to catalog commands

* add missing files
2021-08-27 16:34:45 -04:00
R.B. Boyer 61f1c01b83
agent: ensure that most agent behavior correctly respects partition configuration (#10880) 2021-08-19 15:09:42 -05:00
Daniel Nephin e2a19b1799 debug: use human readable dates for filenames
The unix timestamps that were used make the debug data a little bit more
difficult to consume. By using human readable dates we can easily see
when the profile data was collected.

This commit also improves the test coverage. Two test cases are removed
and the assertions from those cases are moved to TestDebugCommand.

Now TestDebugCommand is able to validate the contents of all files. This
change reduces the test runtime of the command/debug package by almost
50%. It also makes much more strict assertions about the contents by
using gotest.tools/v3/fs.
2021-08-18 13:06:57 -04:00
Daniel Nephin dccaf95cc8 debug: small cleanup
Use the new WriteJsonFile function to write index.json
Remove .String() from time.local() since that is done by %s
Remove an unused field.
2021-08-18 12:30:59 -04:00
Daniel Nephin 064c43ee69 debug: restore cancel on SigInt
Some previous changes broke interrupting the debug on SigInterupt. This change restores
the original behaviour by passing a context to requests.

Since a new API client function was required to pass the context, I had
it also return an io.ReadCloser, so that output can be streamed to files
instead of fully buffering in process memory.
2021-08-18 12:29:34 -04:00
Daniel Nephin d2f5b4d335 debug: improve a couple of the test cases
Use gotest.tools/v3/fs to make better assertions about the files

Remove the TestAgent from TestDebugCommand_Prepare_ValidateTiming, since we can test that validation
without making any API calls.
2021-08-18 12:29:34 -04:00
Daniel Nephin bf30404412 debug: rename cluster target to members
The API is called members. Using the same name as the API should help describe the contents
of the file.
2021-08-18 12:29:34 -04:00
Daniel Nephin e1eab6509c debug: remove unused 2021-08-18 12:29:33 -04:00
Daniel Nephin d77ea40203 cli: remove a test case for updating a legacy token
Legacy tokens are no longer accepted, so we don't need to test their upgrade path.
2021-08-17 13:09:30 -04:00
Mark Anderson 03a3ec2b55
Fixup to support unix domain socket via command line (#10758)
Missed the need to add support for unix domain socket config via
api/command line. This is a variant of the problems described in
it is easy to drop one.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-08-12 10:05:22 -07:00
Blake Covarrubias 41b2f08695
cli: Fix broken KV import on Windows (#10820)
Consul 1.10 (PR #9792) introduced the ability to specify a prefix when
importing KV's. This however introduced a regression on Windows
systems which breaks `kv import`. The key name is joined with
specified`-prefix` using `filepath.Join()` which uses a forward slash
(/) to delimit values on Unix-based systems, and a backslash (\) to
delimit values on Windows – the latter of which is incompatible with
Consul KV paths.

This commit replaces filepath.Join() with path.Join() which uses a
forward slash as the delimiter, providing consistent key join behavior
across supported operating systems.

Fixes #10583
2021-08-10 14:42:05 -07:00
Blake Covarrubias 82565fdf55
cli: Use admin bind address in self_admin cluster (#10757)
Configure the self_admin cluster to use the admin bind address
provided when starting Envoy.

Fixes #10747
2021-08-09 17:10:32 -07:00
Blake Covarrubias 00b0633bda
cli: Test API access using /status/leader in consul watch (#10795)
Replace call to /agent/self with /status/leader to verify agent
reachability before initializing a watch. This endpoint is not guarded
by ACLs, and as such can be queried by any API client regardless of
their permissions.

Fixes #9353
2021-08-09 09:00:33 -07:00
Daniel Nephin 2503f27a36 acl: remove rule == nil checks 2021-07-30 13:58:35 -04:00
Evan Culver 53465764df Fix maint test 2021-07-30 12:58:46 -04:00
Daniel Nephin cf2e25c6bb http: emit indented JSON in the metrics stream endpoint
To remove the need to decode and re-encode in the CLI
2021-07-26 17:53:33 -04:00
Daniel Nephin d716f709fd debug: use the new metrics stream in debug command 2021-07-26 17:53:32 -04:00
Dhia Ayachi c4911cc3ba
add http flag for admin partition (#10683) 2021-07-22 16:46:02 -04:00
R.B. Boyer c9c80b5ef6
add partition cli flag to all cli commands that have namespace flag (#10668) 2021-07-21 14:45:24 -05:00
Blake Covarrubias a5ab658bf8
cli: Document pass-through option for `consul connect envoy` (#10666)
Update help text of `consul connect envoy` command to mention the
ability to provide pass-through options.
2021-07-21 10:43:10 -07:00
Evan Culver 521c423075
acls: Show `AuthMethodNamespace` when reading/listing ACL token meta (#10598) 2021-07-15 10:38:52 -07:00
Daniel Nephin 57c5a40869
Merge pull request #10588 from hashicorp/dnephin/config-fix-ports-grpc
config: rename `ports.grpc` to `ports.xds`
2021-07-13 13:11:38 -04:00
Daniel Nephin 25dc14f036 Apply suggestions from code review
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-07-13 12:31:49 -04:00
Daniel Nephin a5f93e5596 command/envoy: stop using the DebugConfig from Self endpoint
The DebugConfig in the self endpoint can change at any time. It's not a stable API.

With the previous change to rename GRPCPort to XDSPort this command would have broken.
This commit adds the XDSPort to a stable part of the XDS api, and changes the envoy command to read
this new field.

It includes support for the old API as well, in case a newer CLI is used with an older API, and
adds a test for both cases.
2021-07-13 12:31:49 -04:00
Iryna Shustava a75a860174
cli/sdk: Allow applying redirect-traffic rules in a provided Linux namespace (#10564) 2021-07-13 10:05:48 -06:00
Daniel Nephin 2ab6be6a88 config: update GRPCPort and addr in runtime config 2021-07-09 12:31:53 -04:00
Evan Culver 5ff191ad99
Add support for returning ACL secret IDs for accessors with acl:write (#10546) 2021-07-08 15:13:08 -07:00
R.B. Boyer 0e6a482b76
config: add agent config flag for enterprise clients to indicate they wish to join a particular partition (#10572) 2021-07-08 10:03:38 -05:00
Daniel Nephin 14527dd005
Merge pull request #10552 from hashicorp/dnephin/ca-remove-rotation-period
ca: remove unused RotationPeriod field
2021-07-06 18:49:33 -04:00
jkirschner-hashicorp 31bbab8ae7
Merge pull request #10560 from jkirschner-hashicorp/change-sane-to-reasonable
Replace use of 'sane' where appropriate
2021-07-06 11:46:04 -04:00
Daniel Nephin b4a10443d1 ca: remove unused RotationPeriod field
This field was never used. Since it is persisted as part of a map[string]interface{} it
is pretty easy to remove it.
2021-07-05 19:15:44 -04:00
Jared Kirschner 4c3b1b8b7b Replace use of 'sane' where appropriate
HashiCorp voice, style, and language guidelines recommend avoiding ableist
language unless its reference to ability is accurate in a particular use.
2021-07-02 12:18:46 -04:00
Daniel Nephin 72ea979c39
Merge pull request #10515 from hashicorp/dnephin/fix-arm32-atomic-aligment
Fix panic on 32-bit platforms
2021-06-30 16:40:20 -04:00
Daniel Nephin c2cc9b6bda Update references to the main branch
The main branch is being renamed from master->main. This commit should
update all references to the main branch to the new name.

Co-Authored-By: Mike Morris <mikemorris@users.noreply.github.com>
2021-06-29 17:17:38 -04:00
Daniel Nephin 843e08bb23 testing: fix a test for 32-bit
The hcl decoding apparently uses strconv.ParseInt, which fails to parse a 64bit int.
Since hcl v1 is basically EOl, it seems unlikely we'll fix this in hcl.

Since this test is only about loading values from config files, the extra large number
doesn't seem important. Trim a few zeros from the numbers so that they parse
properly on 32bit platforms.

Also skip a slow test when -short is used.
2021-06-29 16:10:21 -04:00
Kyle Nusbaum 4ad13c37db command/agent: change io.Discard to ioutil.Discard 2021-06-23 11:45:40 -05:00
Freddy f399fd2add
Rename CatalogDestinationsOnly (#10397)
CatalogDestinationsOnly is a passthrough that would enable dialing
addresses outside of Consul's catalog. However, when this flag is set to
true only _connect_ endpoints for services can be dialed.

This flag is being renamed to signal that non-Connect endpoints can't be
dialed by transparent proxies when the value is set to true.
2021-06-14 14:15:09 -06:00
Freddy 61ae2995b7
Add flag for transparent proxies to dial individual instances (#10329) 2021-06-09 14:34:17 -06:00
Dhia Ayachi e3dd0f9a44
generate a single debug file for a long duration capture (#10279)
* debug: remove the CLI check for debug_enabled

The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.

Also:
- fix the API client to return errors on non-200 status codes for debug
  endpoints
- improve the failure messages when pprof data can not be collected

Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>

* remove parallel test runs

parallel runs create a race condition that fail the debug tests

* snapshot the timestamp at the beginning of the capture

- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist

* Revert "snapshot the timestamp at the beginning of the capture"

This reverts commit c2d03346

* Refactor captureDynamic to extract capture logic for each item in a different func

* snapshot the timestamp at the beginning of the capture

- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist

* Revert "snapshot the timestamp at the beginning of the capture"

This reverts commit c2d03346

* Refactor captureDynamic to extract capture logic for each item in a different func

* extract wait group outside the go routine to avoid a race condition

* capture pprof in a separate go routine

* perform a single capture for pprof data for the whole duration

* add missing vendor dependency

* add a change log and fix documentation to reflect the change

* create function for timestamp dir creation and simplify error handling

* use error groups and ticker to simplify interval capture loop

* Logs, profile and traces are captured for the full duration. Metrics, Heap and Go routines are captured every interval

* refactor Logs capture routine and add log capture specific test

* improve error reporting when log test fail

* change test duration to 1s

* make time parsing in log line more robust

* refactor log time format in a const

* test on log line empty the earliest possible and return

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* rename function to captureShortLived

* more specific changelog

Co-authored-by: Paul Banks <banks@banksco.de>

* update documentation to reflect current implementation

* add test for behavior when invalid param is passed to the command

* fix argument line in test

* a more detailed description of the new behaviour

Co-authored-by: Paul Banks <banks@banksco.de>

* print success right after the capture is done

* remove an unnecessary error check

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* upgraded github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57 => v0.0.0-20210601050228-01bbb1931b22

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: Paul Banks <banks@banksco.de>
2021-06-07 13:00:51 -04:00
Dhia Ayachi 60d51a50d1
fix monitor to only start the monitor in json format when requested (#10358)
* fix monitor to only start the monitor in json format when requested

* add release notes

* add test to validate json format when requested
2021-06-07 12:08:48 -04:00
Daniel Nephin b93d63b8d2 cmd: remove unnecessary GatedUi
The intent of this struct was to prevent non-json output to stdout. With
the previous cleanup, this can now be done by simply changing the stdout
stream to io.Discard.

This is just one example of why passing around io.Writers for the
streams is better than the UI interface.
2021-06-02 17:33:20 -04:00
Daniel Nephin e207a5de05 cmd: move agent running message to logs
Previously this line was mixed up with logging, which made the output
quite ugly. Use the logger to output this message, instead of printing
directly to stdout.

This has the advantage that the message will be visible when json logs
are enabled.
2021-06-02 17:17:43 -04:00
Daniel Nephin 63c737017d agent: fix agent logging
Remove the leading whitespace on every log line. This was causing problems for
a customer because their logging system was interpretting the logs as a single
multi-line log.
2021-06-02 17:15:12 -04:00
Daniel Nephin 5086baeb7f cmd: introduce a shim to expose Stdout/Stderr writers
This will allow commands to do the right thing, and write to the proper
output stream.
2021-06-02 16:51:34 -04:00
Daniel Nephin 9f83bc97d6 cmd: remove unnecessary args to agent.New
The version args are static and passed in from the caller. Instead read
the static values in New.

The shutdownCh was never closed, so did nothing. Remove it as a field
and an arg.
2021-06-02 16:29:29 -04:00
Daniel Nephin 43408eddb7
Merge pull request #10324 from hashicorp/dnephin/fix-envoy-bootstrap-exec
envoy: fix deadlock when input is larger than named pipe buffer size
2021-06-01 13:02:51 -04:00
Dhia Ayachi 9f2f9ac3a5
make tests use a dummy node_name to avoid environment related failures (#10262)
* fix tests to use a dummy nodeName and not fail when hostname is not a valid nodeName

* remove conditional testing

* add test when node name is invalid
2021-06-01 11:58:03 -04:00
Daniel Nephin 3fd67dc611 envoy: improve comments 2021-06-01 11:35:32 -04:00
Daniel Nephin 0a39ba2c54 envoy: fix bootstrap deadlock caused by a full named pipe
Normally the named pipe would buffer up to 64k, but in some cases when a
soft limit is reached, they will start only buffering up to 4k.
In either case, we should not deadlock.

This commit changes the pipe-bootstrap command to first buffer all of
stdin into the process, before trying to write it to the named pipe.
This allows the process memory to act as the buffer, instead of the
named pipe.

Also changed the order of operations in `makeBootstrapPipe`. The new
test added in this PR showed that simply buffering in the process memory
was not enough to fix the issue. We also need to ensure that the
`pipe-bootstrap` process is started before we try to write to its
stdin. Otherwise the write will still block.

Also set stdout/stderr on the subprocess, so that any errors are visible
to the user.
2021-05-31 18:53:17 -04:00
Daniel Nephin 177a504e9f envoy: start timeout func after validation
This removes the need to check arg length in the timeout function.
2021-05-31 17:37:58 -04:00
Dhia Ayachi 00f7e0772a
debug: remove the CLI check for debug_enabled (#10273)
* debug: remove the CLI check for debug_enabled

The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.

Also:
- fix the API client to return errors on non-200 status codes for debug
  endpoints
- improve the failure messages when pprof data can not be collected

Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>

* remove parallel test runs

parallel runs create a race condition that fail the debug tests

* Add changelog

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-05-27 09:41:53 -04:00
Daniel Nephin f079cef352 redirecttraffic: fix a flaky test
https://app.circleci.com/pipelines/github/hashicorp/consul-enterprise/6408/workflows/e90b1140-daa2-458f-8197-d1821e3693e3/jobs/94694/tests#failed-test-0

The ExcludeInboundPorts can be in a different order sometimes, although I'm not sure how.

Also removes t.Parallel. We should only need to use t.Parallel in large packages with many slow tests.
In this case there is only a single slow test so we don't get any benefit.

Also add struct field names to the testcases list, so that it is easier to see what
each value is doing, and to make it easier to add new fields in the future.
2021-05-18 12:36:46 -04:00
Iryna Shustava 7a41dbd9b6
Save exposed ports in agent's store and expose them via API (#10173)
* Save exposed HTTP or GRPC ports to the agent's store
* Add those the health checks API so we can retrieve them from the API
* Change redirect-traffic command to also exclude those ports from inbound traffic redirection when expose.checks is set to true.
2021-05-12 13:51:39 -07:00
R.B. Boyer 05b52a3d63
connect: update supported envoy versions to 1.18.3, 1.17.3, 1.16.4, and 1.15.5 (#10231) 2021-05-12 14:06:06 -05:00
Daniel Nephin 55f620d636
Merge pull request #10155 from hashicorp/dnephin/config-entry-remove-fields
config-entry: remove Kind and Name field from Mesh config entry
2021-05-04 17:27:56 -04:00
Mark Anderson 626b27a874 Continue working through proxy and agent
Rework/listeners, rename makeListener

Refactor, tests pass

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Luke Kysow eb84a856c4
Give descriptive error if auth method not found (#10163)
* Give descriptive error if auth method not found

Previously during a `consul login -method=blah`, if the auth method was not found, the
error returned would be "ACL not found". This is potentially confusing
because there may be many different ACLs involved in a login: the ACL of
the Consul client, perhaps the binding rule or the auth method.

Now the error will be "auth method blah not found", which is much easier
to debug.
2021-05-03 13:39:13 -07:00
Daniel Nephin fcfd244269 config-entry: use custom MarshalJSON for mesh type
So that the Kind field is added to the JSON object.
2021-04-29 17:49:43 -04:00
Daniel Nephin bf4c289804 config-entry: remove Kind and Name field from Mesh config entry
No config entry needs a Kind field. It is only used to determine the Go type to
target. As we introduce new config entries (like this one) we can remove the kind field
and have the GetKind method return the single supported value.

In this case (similar to proxy-defaults) the Name field is also unnecessary. We always
use the same value. So we can omit the name field entirely.
2021-04-29 17:11:21 -04:00
R.B. Boyer 97e57aedfb
connect: update supported envoy versions to 1.18.2, 1.17.2, 1.16.3, and 1.15.4 (#10101)
The only thing that needed fixing up pertained to this section of the 1.18.x release notes:

> grpc_stats: the default value for stats_for_all_methods is switched from true to false, in order to avoid possible memory exhaustion due to an untrusted downstream sending a large number of unique method names. The previous default value was deprecated in version 1.14.0. This only changes the behavior when the value is not set. The previous behavior can be used by setting the value to true. This behavior change by be overridden by setting runtime feature envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default.

For now to maintain status-quo I'm explicitly setting `stats_for_all_methods=true` in all versions to avoid relying upon the default.

Additionally the naming of the emitted metrics for these gRPC requests changed slightly so the integration test assertions for `case-grpc` needed adjusting.
2021-04-29 15:22:03 -05:00
R.B. Boyer 91bee6246f
Support Incremental xDS mode (#9855)
This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged.

Union of all commit messages follows to give an overarching summary:

xds: exclusively support incremental xDS when using xDS v3

Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail.
Work around a strange older envoy behavior involving empty CDS responses over incremental xDS.
xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support

Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead.
Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client.
xds: pull out checkStreamACLs method in advance of a later commit

xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings

In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty.

This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10.

xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time
2021-04-29 13:54:05 -05:00
Iryna Shustava e7dcf9acd0
Implement traffic redirection exclusion based on proxy config and user-provided values (#10134)
* Use proxy outbound port from TransparentProxyConfig if provided
* If -proxy-id is provided to the redirect-traffic command, exclude any listener ports
  from inbound traffic redirection. This includes envoy_prometheus_bind_addr,
  envoy_stats_bind_addr, and the ListenerPort from the Expose configuration.
* Allow users to provide additional inbound and outbound ports, outbound CIDRs
  and additional user IDs to be excluded from traffic redirection.
  This affects both the traffic-redirect command and the iptables SDK package.
2021-04-29 09:21:15 -07:00
Freddy 401f3010e0
Rename "cluster" config entry to "mesh" (#10127)
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.

Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
2021-04-28 16:13:29 -06:00
Paul Banks bbe0d6aa30
Fix panic bug in snapshot inspect (#10091)
* Fix panic bug in snapshot inspect

* Add changelog entry

* Update .changelog/10091.txt

* Undo bad GitHub UI merge

* Undo bad GitHub UI merge
2021-04-23 20:48:10 +01:00
Paul Banks 5c409739c7
CLI: Allow snapshot inspect to work on internal raft snapshots directly. (#10089)
* CLI: Add support for reading internal raft snapshots to snapshot inspect

* Add snapshot inspect test for raw state files

* Add changelog entry

* Update .changelog/10089.txt
2021-04-23 16:17:08 +01:00
R.B. Boyer c88512fe14
connect: update centralized upstreams representation in service-defaults (#10015) 2021-04-15 14:21:44 -05:00
freddygv e175b309fb Fixup tests 2021-04-13 16:08:41 -06:00
freddygv 50c7915156 Convert new tproxy structs in api module into ptrs
This way we avoid serializing these when empty. Otherwise users of the
latest version of the api submodule cannot interact with older versions
of Consul, because a new api client would send keys that the older Consul
doesn't recognize yet.
2021-04-13 12:44:25 -06:00
freddygv eeccba945d Replace TransparentProxy bool with ProxyMode
This PR replaces the original boolean used to configure transparent
proxy mode. It was replaced with a string mode that can be set to:

- "": Empty string is the default for when the setting should be
defaulted from other configuration like config entries.
- "direct": Direct mode is how applications originally opted into the
mesh. Proxy listeners need to be dialed directly.
- "transparent": Transparent mode enables configuring Envoy as a
transparent proxy. Traffic must be captured and redirected to the
inbound and outbound listeners.

This PR also adds a struct for transparent proxy specific configuration.
Initially this is not stored as a pointer. Will revisit that decision
before GA.
2021-04-12 09:35:14 -06:00
Iryna Shustava ff2e70f4ce
cli: Add new `consul connect redirect-traffic` command for applying traffic redirection rules when Transparent Proxy is enabled. (#9910)
* Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled.
* Add new iptables package for applying traffic redirection rules with iptables.
2021-04-09 11:48:10 -07:00
Freddy 920ba3db39
Merge pull request #9976 from hashicorp/centralized-upstream-fixups 2021-04-08 12:26:56 -06:00
freddygv 43091aae60 Avoid sending zero-value upstream defaults from api 2021-04-07 15:03:42 -06:00
R.B. Boyer 36c74bf865
command: when generating envoy bootstrap configs to stdout do not mix informational logs into the json (#9980)
Fixes #9921
2021-04-07 14:22:52 -05:00
R.B. Boyer 82245585c6
connect: add toggle to globally disable wildcard outbound network access when transparent proxy is enabled (#9973)
This adds a new config entry kind "cluster" with a single special name "cluster" where this can be controlled.
2021-04-06 13:19:59 -05:00
Hans Hasselberg 052662bcf9
introduce certopts (#9606)
* introduce cert opts

* it should be using the same signer

* lint and omit serial
2021-03-22 10:16:41 +01:00
woz5999 1585ea3734 support env var expansion in envoy statsd urls
Fixes #8561
2021-03-18 18:57:28 -04:00
Freddy 63dcb7fa76
Add TransparentProxy option to proxy definitions 2021-03-17 17:01:45 -06:00
Freddy fb252e87a4
Add per-upstream configuration to service-defaults 2021-03-17 16:59:51 -06:00
freddygv d90240d367 Restore old Envoy prefix on escape hatches
This is done because after removing ID and NodeName from
ServiceConfigRequest we will no longer know whether a request coming in
is for a Consul client earlier than v1.10.
2021-03-15 14:12:57 -06:00
Kyle Havlovitz 237b41ac8f
Merge pull request #9672 from hashicorp/ca-force-skip-xc
connect/ca: Allow ForceWithoutCrossSigning for all providers
2021-03-11 11:49:15 -08:00