Commit graph

119 commits

Author SHA1 Message Date
Alexander Scheel e9768b6bc6
Fix radiusd network connection limitations (#17049)
* Allow exposing access to the underlying container

This exposes the Container response from the Docker API, allowing
consumers of the testhelper to interact with the newly started running
container instance. This will be useful for two reasons:

 1. Allowing radiusd container to start its own daemon after modifying
    its configuration.
 2. For loading certificates into a future similar integration test
    using the PKI secrets engine.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow any client to connect to test radiusd daemon

This fixes test failures of the following form:

> 2022-09-07T10:46:19.332-0400 [TRACE] core: adding local paths: paths=[]
> 2022-09-07T10:46:19.333-0400 [INFO]  core: enabled credential backend: path=mnt/ type=test
> 2022-09-07T10:46:19.334-0400 [WARN]  Executing test step: step_number=1
> 2022-09-07T10:46:19.334-0400 [WARN]  Executing test step: step_number=2
> 2022-09-07T10:46:29.334-0400 [WARN]  Executing test step: step_number=3
> 2022-09-07T10:46:29.335-0400 [WARN]  Executing test step: step_number=4
> 2022-09-07T10:46:39.336-0400 [WARN]  Requesting RollbackOperation
> --- FAIL: TestBackend_acceptance (28.56s)
>     testing.go:364: Failed step 4: erroneous response:
>
>         &logical.Response{Secret:<nil>, Auth:<nil>, Data:map[string]interface {}{"error":"context deadline exceeded"}, Redirect:"", Warnings:[]string(nil), WrapInfo:(*wrapping.ResponseWrapInfo)(nil), Headers:map[string][]string(nil)}
> FAIL
> FAIL	github.com/hashicorp/vault/builtin/credential/radius	29.238s

In particular, radiusd container ships with a default clients.conf which
restricts connections to ranges associated with the Docker daemon. When
creating new networks (such as in CircleCI) or when running via Podman
(which has its own set of network ranges), this initial config will no
longer be applicable. We thus need to write a new config into the image;
while we could do this by rebuilding a new image on top of the existing
layers (provisioning our config), we then need to manage these changes
and give hooks for the service setup to build it.

Thus, post-startup modification is probably easier to execute in our
case.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-09-07 13:43:22 -04:00
Nick Cabatoff df61151034
Wait for standby to have a working grpc connection before we try to use it (#16905)
Also teach WaitForStandbyNode to do a better job waiting for standbys to be healthy.
2022-08-26 12:50:10 -04:00
Scott Miller 3bd38fd5dc
OSS portion of wrapper-v2 (#16811)
* OSS portion of wrapper-v2

* Prefetch barrier type to avoid encountering an error in the simple BarrierType() getter

* Rename the OveriddenType to WrapperType and use it for the barrier type prefetch

* Fix unit test
2022-08-23 15:37:16 -04:00
Mike Palmiotto cd1157a905
Vault 7338/fix retry join (#16550)
* storage/raft: Fix cluster init with retry_join

Commit 8db66f4853abce3f432adcf1724b1f237b275415 introduced an error
wherein a join() would return nil (no error) with no information on its
channel if a joining node had been initialized. This was not handled
properly by the caller and resulted in a canceled `retry_join`.

Fix this by handling the `nil` channel respone by treating it as an
error and allowing the existing mechanics to work as intended.

* storage/raft: Improve retry_join go test

* storage/raft: Make VerifyRaftPeers pollable

* storage/raft: Add changelog entry for retry_join fix

* storage/raft: Add description to VerifyRaftPeers
2022-08-03 20:44:57 -05:00
Robert 7f8c849b35
Update Consul bootstrap test case to conditionally add token to config (#16560)
* Fix bootstrap test to conditionally add Consul token

* Refactor bootstrap variable name to be more clear
2022-08-03 13:43:43 -05:00
swayne275 4632a26a09
Use %q for quoted strings where appropriate (#15216)
* change '%s' to %q where single vs double quotes shouldn't matter

* replace double quotes with %q in logs and errors
2022-08-03 12:32:45 -06:00
Mike Palmiotto 439e35f50f
Vault 6773/raft rejoin nonvoter (#16324)
* raft: Ensure init before setting suffrage

As reported in https://hashicorp.atlassian.net/browse/VAULT-6773:

	The /sys/storage/raft/join endpoint is intended to be unauthenticated. We rely
	on the seal to manage trust.

	It’s possible to use multiple join requests to switch nodes from voter to
	non-voter. The screenshot shows a 3 node cluster where vault_2 is the leader,
	and vault_3 and vault_4 are followers with non-voters set to false.  sent two
	requests to the raft join endpoint to have vault_3 and vault_4 join the cluster
	with non_voters:true.

This commit fixes the issue by delaying the call to SetDesiredSuffrage until after
the initialization check, preventing unauthenticated mangling of voter status.

Tested locally using
https://github.com/hashicorp/vault-tools/blob/main/users/ncabatoff/cluster/raft.sh
and the reproducer outlined in VAULT-6773.

* raft: Return join err on failure

This is necessary to correctly distinguish errors returned from the Join
workflow. Previously, errors were being masked as timeouts.

* raft: Default autopilot parameters in teststorage

Change some defaults so we don't have to pass in parameters or set them
in the originating tests. These storage types are only used in two
places:

1) Raft HA testing
2) Seal migration testing

Both consumers have been tested and pass with this change.

* changelog: Unauthn voter status change bugfix
2022-07-18 14:37:12 -04:00
Jim Kalafut a3b0b60a73
postgres: replace the package lib/pq with pgx (#15343)
* WIP replacing lib/pq

* change timezome param to be URI format

* add changelog

* add changelog for redshift

* update changelog

* add test for DSN style connection string

* more parseurl and quoteidentify to sdk; include copyright and license

* call dbutil.ParseURL instead, fix import ordering

Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
2022-05-23 12:49:18 -07:00
Robert 738753b187
secrets/consul: Add support for generating tokens with service and node identities (#15295)
Co-authored-by: Thomas L. Kula <kula@tproa.net>
2022-05-09 20:07:35 -05:00
Nick Cabatoff c5928c1d15
Raft: use a larger initial heartbeat/election timeout (#15042) 2022-04-29 08:32:16 -04:00
Rémi Lapeyre bf4c4595f3
secrets/consul: Add support to auto-bootstrap Consul ACL system (#10751)
* Automatically bootstraps the Consul ACL system if no management token is given on the access config
2022-04-20 17:16:15 -05:00
Ben Ash 287bb77abc
Ensure that URL encoded passwords are properly redacted. (#14744)
The URL password redaction operation did not handle the case where the
database connection URL was provided as a percent-encoded string, and
its password component contained reserved characters. It attempted to
redact the password by replacing the unescaped password in the
percent-encoded URL. This resulted in the password being revealed when
reading the configuration from Vault.
2022-03-29 10:33:55 -04:00
Robert 91f5069c03
secret/consul: Add Consul ACL roles support (#14014)
Co-authored-by: Brandon Ingalls <brandon@ingalls.io>
2022-02-16 19:31:08 -06:00
Robert 74bdf5f006
Formatting: Remove blank line (#13994) 2022-02-09 18:09:20 -06:00
Robert d0832a1993
secret/consul: Add support for consul namespaces and admin partitions (#13850)
* Add support for consul namespaces and admin partitions
2022-02-09 15:44:00 -06:00
Josh Black d249fad2df
reformat using 'make fmt' (#13794) 2022-01-27 10:06:34 -08:00
Jason O'Donnell 974dbf6082
auth/ldap: Add username to alias.metadata.name (#13669)
* Fix upndomain bug causing alias name to change

* Fix nil map

* Add changelog

* revert

* Update changelog

* Add test for alias metadata name

* Fix code comment
2022-01-20 12:30:26 -05:00
Nick Cabatoff 400996ef0d
Parallel retry join (#13606) 2022-01-17 10:33:03 -05:00
Scott Miller 89f617a97c
Convert to Go 1.17 go:build directive (#13579) 2022-01-05 12:02:03 -06:00
Pavlos Tzianos 0abc8f43fa
Add helper for encoding/decoding root tokens and OTP generation in SDK module (#10504) (#10505) 2021-12-01 08:05:49 -05:00
Nick Cabatoff c4764433a8
Add ability to capture container logs, and have mssql test helper use it (#13272) 2021-11-24 12:01:38 -05:00
swayne275 06f83ba8e6
enhance test metrics (#13262) 2021-11-23 14:56:30 -07:00
Nick Cabatoff d89c7b3d7e
Add a little test helper for polling (#13082) 2021-11-08 15:24:06 -05:00
Steven Clark 687701d9a3
Attempt to fix the flaky TestDeleteUser/TestUpdateUser mssql tests (#13071)
- Add a 'Connect Timeout' query parameter to the test helper to set
   a timeout value of 30 seconds in an attempt to address the following
   failure we see at times in TestDeleteUser and TestUpdateUser

   mssql_test.go:253: Failed to initialize: error verifying connection: TLS Handshake failed: cannot read handshake packet: EOF
2021-11-05 14:53:37 -04:00
Ben Ash 6769ab37ea
fix: upgrade vault-plugin-auth-kerberos to v0.5.0 (#12930)
* Update docker helper code to match deps.
* Fix container variable name collides with import.
* Update vault-testing-stepwise to v0.1.2
2021-10-27 10:01:34 -04:00
Guillaume b9b7f5a9a3
Added support for a LDAP user search filter. Documentation, tests and UI included (#11000) 2021-10-26 10:39:12 -07:00
swayne275 fe9da20d67
oss components of vault-3372 (#12898) 2021-10-22 14:22:49 -06:00
Tero Saarni 944332d12d
Update Go client libraries for etcd (#11980)
* Update Go client libraries for etcd

* Added etcd server container to run etcd3 tests automatically.

* Removed etcd2 test case: it fails the backend tests but the failure is
  unrelated to the uplift.  The etcd2 backend implementation does not
  remove empty nested nodes when removing leaf (see comments in #11980).
2021-09-29 14:28:13 -04:00
Hridoy Roy f3895f7cce
oss part of license diagnose (#11939) 2021-06-25 14:18:34 -07:00
Michael Golowka 7f6a1739a3
Cassandra: Refactor PEM parsing logic (#11861)
* Refactor TLS parsing

The ParsePEMBundle and ParsePKIJSON functions in the certutil package assumes
both a client certificate and a custom CA are specified. Cassandra needs to
allow for either a client certificate, a custom CA, or both. This revamps the
parsing of pem_json and pem_bundle to accomodate for any of these configurations
2021-06-21 11:38:08 -06:00
Josh Black ec105f288f
Switch to shared raft-boltdb library and add metrics (#11269) 2021-04-26 16:01:26 -07:00
Nick Cabatoff e3a88c59e9
Add metrics for perf standby and replication node type. (#11472) 2021-04-26 13:54:19 -04:00
Michael Golowka 4279bc8b34
Validate hostnames when using TLS in Cassandra (#11365) 2021-04-16 15:52:35 -06:00
Brian Kassouf 303c2aee7c
Run a more strict formatter over the code (#11312)
* Update tooling

* Run gofumpt

* go mod vendor
2021-04-08 09:43:39 -07:00
Vishal Nayak 586a5a73f0
Fix diff (#11057) 2021-03-08 13:20:01 -05:00
Vishal Nayak 3e55e79a3f
Autopilot: Server Stabilization, State and Dead Server Cleanup (#10856)
* k8s doc: update for 0.9.1 and 0.8.0 releases (#10825)

* k8s doc: update for 0.9.1 and 0.8.0 releases

* Update website/content/docs/platform/k8s/helm/configuration.mdx

Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>

* Autopilot initial commit

* Move autopilot related backend implementations to its own file

* Abstract promoter creation

* Add nil check for health

* Add server state oss no-ops

* Config ext stub for oss

* Make way for non-voters

* s/health/state

* s/ReadReplica/NonVoter

* Add synopsis and description

* Remove struct tags from AutopilotConfig

* Use var for config storage path

* Handle nin-config when reading

* Enable testing autopilot by using inmem cluster

* First passing test

* Only report the server as known if it is present in raft config

* Autopilot defaults to on for all existing and new clusters

* Add locking to some functions

* Persist initial config

* Clarify the command usage doc

* Add health metric for each node

* Fix audit logging issue

* Don't set DisablePerformanceStandby to true in test

* Use node id label for health metric

* Log updates to autopilot config

* Less aggressively consume config loading failures

* Return a mutable config

* Return early from known servers if raft config is unable to be pulled

* Update metrics name

* Reduce log level for potentially noisy log

* Add knob to disable autopilot

* Don't persist if default config is in use

* Autopilot: Dead server cleanup (#10857)

* Dead server cleanup

* Initialize channel in any case

* Fix a bunch of tests

* Fix panic

* Add follower locking in heartbeat tracker

* Add LastContactFailureThreshold to config

* Add log when marking node as dead

* Update follower state locking in heartbeat tracker

* Avoid follower states being nil

* Pull test to its own file

* Add execution status to state response

* Optionally enable autopilot in some tests

* Updates

* Added API function to fetch autopilot configuration

* Add test for default autopilot configuration

* Configuration tests

* Add State API test

* Update test

* Added TestClusterOptions.PhysicalFactoryConfig

* Update locking

* Adjust locking in heartbeat tracker

* s/last_contact_failure_threshold/left_server_last_contact_threshold

* Add disabling autopilot as a core config option

* Disable autopilot in some tests

* s/left_server_last_contact_threshold/dead_server_last_contact_threshold

* Set the lastheartbeat of followers to now when setting up active node

* Don't use config defaults from CLI command

* Remove config file support

* Remove HCL test as well

* Persist only supplied config; merge supplied config with default to operate

* Use pointer to structs for storing follower information

* Test update

* Retrieve non voter status from configbucket and set it up when a node comes up

* Manage desired suffrage

* Consider bucket being created already

* Move desired suffrage to its own entry

* s/DesiredSuffrageKey/LocalNodeConfigKey

* s/witnessSuffrage/recordSuffrage

* Fix test compilation

* Handle local node config post a snapshot install

* Commit to storage first; then record suffrage in fsm

* No need of local node config being nili case, post snapshot restore

* Reconcile autopilot config when a new leader takes over duty

* Grab fsm lock when recording suffrage

* s/Suffrage/DesiredSuffrage in FollowerState

* Instantiate autopilot only in leader

* Default to old ways in more scenarios

* Make API gracefully handle 404

* Address some feedback

* Make IsDead an atomic.Value

* Simplify follower hearbeat tracking

* Use uber.atomic

* Don't have multiple causes for having autopilot disabled

* Don't remove node from follower states if we fail to remove the dead server

* Autopilot server removals map (#11019)

* Don't remove node from follower states if we fail to remove the dead server

* Use map to track dead server removals

* Use lock and map

* Use delegate lock

* Adjust when to remove entry from map

* Only hold the lock while accessing map

* Fix race

* Don't set default min_quorum

* Fix test

* Ensure follower states is not nil before starting autopilot

* Fix race

Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
2021-03-03 13:59:50 -05:00
Brian Kassouf 1bc410783d OSS/ENT Drift 2021-03-01 10:51:04 -08:00
Nick Cabatoff c1ddfbb538
OSS parts of the new client controlled consistency feature (#10974) 2021-02-24 06:58:10 -05:00
Hridoy Roy 4a96126d5a
Revert "Vault Dependency Upgrades [VAULT-871] (#10903)" (#10939)
This reverts commit eb74ca61fc4dcb7038f39defb127d5d639ba0ca1.
2021-02-18 15:40:18 -05:00
Hridoy Roy a26d1300e8
Vault Dependency Upgrades [VAULT-871] (#10903)
* upgrade vault dependency set

* etcd and grpc issues:

* better for tests

* testing

* all upgrades for hashicorp deps

* kubernetes plugin upgrade seems to work

* kubernetes plugin upgrade seems to work

* etcd and a bunch of other stuff

* all vulnerable packages upgraded

* k8s is broken in linux env but not locally

* test fixes

* fix testing

* fix etcd and grpc

* fix etcd and grpc

* use master branch of go-testing-interface

* roll back etcd upgrade

* have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd

* rolling back in the replace directives

* a few more testing dependencies to clean up

* fix go mod vendor
2021-02-18 12:31:57 -08:00
Michael Golowka e494e8a141
Cassandra - Add username customization (#10906) 2021-02-16 12:39:24 -07:00
Michael Golowka 108d4c6a68
MySQL - Add username customization (#10834) 2021-02-11 14:08:32 -07:00
Nick Cabatoff 6a377a3df3
Add helper used in ent. (#10445) 2020-11-25 08:27:45 -05:00
Nick Cabatoff 0d6a929a4c
Same seal migration oss (#10224)
* Refactoring and test improvements.

* Support migrating from a given type of autoseal to that same type but with different parameters.
2020-10-23 14:16:04 -04:00
Nick Cabatoff 66274607b7
OSS changes for enterprise automated snapshots (#10160) 2020-10-16 14:57:11 -04:00
Brian Kassouf 84dbca38a1
Revert "Migrate internalshared out (#9727)" (#10141)
This reverts commit ee6391b691ac12ab6ca13c3912404f1d3a842bd6.
2020-10-13 16:38:21 -07:00
Jeff Mitchell e6881c8147
Migrate internalshared out (#9727)
* Migrate internalshared out

* fix merge issue

* fix merge issue

* go mod vendor

Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
2020-10-12 11:56:24 -07:00
Hridoy Roy d7a673321d
Retry Logic to Mssql Tests [VAULT-637] (#10039)
* added retry to mssql testing

* setting num retry to 3

* removed a comment and moved svc into loop

Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MacBook-Pro.local>
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MBP.hitronhub.home>
2020-09-28 09:03:23 -07:00
ncabatoff 3fb20ed529
Logging tweaks, mostly for tests (#10028)
Fix some places where raft wasn't hooking into the core logger as it should.
Revisited the code that was setting the log level to Error during cleanup: it's normal for there to be a bunch of errors then, which makes it harder to see what went wrong up to the point where the test was deemed to have failed.  So now, instead of setting log level to Error, we actually stop logging altogether.  This only applies if the test didn't pass in its own logger during cluster creation, but we should be moving away from that anyway.
2020-09-23 13:40:00 -04:00
ncabatoff 2872702995
Use a longer timeout for DeriveActiveCore in the hopes that giving more time will allow for raft leader election failure to recover. (#9988) 2020-09-18 11:07:35 -04:00