Commit graph

619 commits

Author SHA1 Message Date
Nick Cabatoff 4ee4374b3e
Use MAP_POPULATE for our bbolt mmaps (#13573)
* Use MAP_POPULATE for our bbolt mmaps, assuming the files fit in memory.  This should improve startup times when freelist sync is disabled.
2022-01-11 08:16:53 -05:00
Scott Miller 89f617a97c
Convert to Go 1.17 go:build directive (#13579) 2022-01-05 12:02:03 -06:00
Matt Mukerjee f574e75f11
fix consul tls error checking (#13135)
We don't check for errors in the consul storage TLS setup. We might fail here
because of a missing certificate, bad permissions, etc. If anything is wrong,
vault just ignores the issues and continues, resulting in a lot of confusion.

Instead, lets return an error to the caller if this fails.
2021-12-21 09:38:58 -08:00
Jim Kalafut 22c4ae5933
Rename master key to root key (#13324)
* See what it looks like to replace "master key" with "root key".  There are two places that would require more challenging code changes: the storage path `core/master`, and its contents (the JSON-serialized EncodedKeyringtructure.)

* Restore accidentally deleted line

* Add changelog

* Update root->recovery

* Fix test

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2021-12-06 17:12:20 -08:00
Eugene R f39f1ce8de
Aerospike backend update (#12165)
* upgrade aerospike-client-go to v5.2.0

* use strings.Contains to check an error

* add changelog file

* go mod tidy

* go mod tidy

* update the changelog

* revert .gitignore update

* go mod tidy
2021-11-29 11:09:12 -08:00
Nick Cabatoff 997a5ace91
Prevent raft transactions from containing overlarge keys. (#13286) 2021-11-26 08:38:39 -05:00
Nick Cabatoff f85908e1df
Return an error when trying to store a too-large key with Raft (#13282) 2021-11-25 14:07:03 -05:00
Josh Black d7c54b50e7
fix bolt 32 bit test (#13249) 2021-11-23 10:50:15 -08:00
Josh Black fe0dd6f867
Add InitialMmapSize to bolt options (#13178) 2021-11-22 20:16:57 -08:00
Brian Kassouf c705adc79c
Fix some linting errors (#12860) 2021-10-18 17:29:47 -07:00
Mayo 0bd0339c0b
cleanup unused code and fix t.Fatal usage in goroutine in testing (#11694) 2021-09-30 07:33:14 -04:00
Tero Saarni 944332d12d
Update Go client libraries for etcd (#11980)
* Update Go client libraries for etcd

* Added etcd server container to run etcd3 tests automatically.

* Removed etcd2 test case: it fails the backend tests but the failure is
  unrelated to the uplift.  The etcd2 backend implementation does not
  remove empty nested nodes when removing leaf (see comments in #11980).
2021-09-29 14:28:13 -04:00
akshya96 c643dc1d53
Add Custom metadata field to alias (#12502)
* adding changes

* removing q.Q

* removing empty lines

* testing

* checking tests

* fixing tests

* adding changes

* added requested changes

* added requested changes

* added policy templating changes and fixed tests

* adding proto changes

* making changes

* adding unit tests

* using suggested function
2021-09-17 11:03:47 -07:00
Jason O'Donnell 1cf3ff046e
plugin/snowflake: update gosnowflake to v1.6.1 (#12378)
* plugin/snowflake: update gosnowflake to v1.6.1

* changelog

* go mod tidy
2021-08-20 11:52:31 -04:00
Pratyoy Mukhopadhyay 5fda05adee
[VAULT-3226] Use os.rename on windows os (#12377)
* [VAULT-3226] Use os.rename on windows os

* [VAULT-3226] Add changelog
2021-08-19 16:05:53 -07:00
Nick Cabatoff 72499c3215
Check to make sure context isn't expired before doing a raft operation. (#12162) 2021-08-19 12:03:56 -04:00
Pratyoy Mukhopadhyay fa29e780f0
[NO-TICKET] Upgrade protoc-gen-go to 1.26, upgrade protoc to 3.17.3 (#12171)
* [NO-TICKET] Set protoc-gen-go to 1.23, upgrade protoc to 3.17.3

* [NO-TICKET] Upgrade version of protoc-gen-go to 1.26
2021-07-28 14:51:36 -07:00
Nick Cabatoff f7ecb978a6
Use a mode when opening the db file that won't result in excessive perms. (#12160) 2021-07-23 13:43:50 -04:00
Jeff Mitchell 33ff878946
Move awsutil over to the go-secure-stdlib version (#12128)
Unlike the other libraries that were migrated, there are no usages of
this lib in any of our plugins, and the only other known usage was in
go-kms-wrapping, which has been updated. Aliasing it like the other libs
would still keep the aws-sdk-go dep in the sdk module because of the
function signatures. So I've simply removed it entirely here.
2021-07-20 20:42:00 -04:00
Nick Cabatoff 1a32ab0628
Add support for testing Azure storage backend in CI using Azurite. (#12057)
Also hack the pagination test until the upstream azurite bugfix (https://github.com/Azure/Azurite/issues/910)
makes it into an Azurite release.
2021-07-16 10:33:53 -04:00
Jeff Mitchell f7147025dd
Migrate to sdk/internalshared libs in go-secure-stdlib (#12090)
* Swap sdk/helper libs to go-secure-stdlib

* Migrate to go-secure-stdlib reloadutil

* Migrate to go-secure-stdlib kv-builder

* Migrate to go-secure-stdlib gatedwriter
2021-07-15 20:17:31 -04:00
Nick Cabatoff a3ac49aa05
VAULT-2809: Tweak creation of vault.db file (#12034) 2021-07-09 14:45:50 -04:00
hghaf099 fedc31e905
Diagnose warns if HTTPS is not used for ha-storage-tls-consul [VAULT-2773] (#11967)
* Diagnose warns if HTTPS is not used for ha-storage-tls-consul

* Skipping TLS verification if https is not used in ha storage tls consul

* Adding diagnose skip message for consul service registration
2021-07-07 14:35:25 -04:00
Hridoy Roy e2614979f7
Diagnose Storage Panic Bugfixes (#11923)
* partial

* fix raft panics and ensure checks are skipped if storage isnt initialized

* cleanup directories

* newline

* typo in nil check

* another nil check
2021-06-24 09:56:38 -07:00
Nick Cabatoff ccae681628
Remove fragile link to docs from code. (#11928) 2021-06-23 15:43:44 -04:00
Brian Kassouf a794a6244f
raft: Set BatchApplyCh for more consistent batch sizes (#11907)
* raft: Set BatchApplyCh for more consistent batch sizes

* Add changelog file
2021-06-21 12:00:41 -07:00
Josh Black 8c069936e9
Add new boltdb options (#11895) 2021-06-21 11:35:40 -07:00
Hridoy Roy e38f991054
Diagnose checks for raft quorum status and file backend permissions (#11771)
* raft file and quorum checks

* raft checks

* backup

* raft file checks test

* address comments and add more raft and file and process checks

* syntax issues

* modularize functions to compile differently on different os

* compile raft checks everywhere

* more build tag issues

* raft-diagnose

* correct file permission checks

* upgrade tests and add a getConfigOffline test that currently does not work

* comment

* update file checks method signature on windows

* Update physical/raft/raft_test.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* raft tests

* add todo comment for windows root ownership

* voter count message

* raft checks test fixes

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2021-06-17 10:04:21 -07:00
Hridoy Roy e58c39033b
Add expiration warning to certificate checking for diagnose [VAULT-1018] (#11850)
* add expiration warning to certificate checking for diagnose

* Update serviceregistration/consul/consul_service_registration.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* review comments

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-06-15 09:53:29 -07:00
Brian Kassouf 0d9ea8a4b7
physical/raft: Add a function that gets the offline, stale configuration (#11821)
* Add a function that gets the offline, stale configuration

* Fix comment
2021-06-11 10:25:02 -07:00
Lars Lehtonen 5ac47a9265
physical: deprecate errwrap.Wrapf() (#11692) 2021-05-31 12:54:05 -04:00
Nick Cabatoff 60898d6e03
Pin aerospike container image to a known working tag. (#11677) 2021-05-20 13:05:26 -04:00
swayne275 335e4c3711
Introduce Logical Unrecoverable Error, Use it in Expiration Manager (#11477)
* build out zombie lease system

* add typo for CI

* undo test CI commit

* time equality test isn't working on CI, so let's see what this does...

* add unrecoverable proto error, make proto, go mod vendor

* zombify leases if unrecoverable error, tests

* test fix: somehow pointer in pointer rx is null after pointer rx called

* tweaks based on roy feedback

* improve zombie errors

* update which errors are unrecoverable

* combine zombie logic

* keep subset of zombie lease in memory
2021-05-03 17:56:06 -06:00
Scott Miller 85fbd45e1c
Create helpers which integrate with OpenTelemetry for diagnose collection (#11454)
* Create helpers which integrate with OpenTelemetry for diagnose collection

* Go mod vendor

* Comments

* Update vault/diagnose/helpers.go

Co-authored-by: swayne275 <swayne275@gmail.com>

* Add unit test/example

* tweak output

* More comments

* add spot check concept

* Get unit tests working on Result structs

* Fix unit test

* Get unit tests working, and make diagnose sessions local rather than global

* Comments

* Last comments

* No need for init

* :|

* Fix helpers_test

Co-authored-by: swayne275 <swayne275@gmail.com>
2021-04-29 13:32:41 -05:00
Hridoy Roy 36ae22c07a
Consul TLS Checks for Diagnose [draft] (#11467)
* consul tls checks

* fix some tests

* complete physical and service registration tls checks
2021-04-28 08:55:18 -07:00
Vishal Nayak 406abc19dc
Autopilot: Return leader info via delegate (#11247)
* Autopilot: Return leader info via delegate

* Pull in the new raft-autopilot lib dependencies

* update deps

* Add CL
2021-04-27 15:54:26 -04:00
Josh Black ec105f288f
Switch to shared raft-boltdb library and add metrics (#11269) 2021-04-26 16:01:26 -07:00
Michael Golowka 4279bc8b34
Validate hostnames when using TLS in Cassandra (#11365) 2021-04-16 15:52:35 -06:00
Brian Kassouf 303c2aee7c
Run a more strict formatter over the code (#11312)
* Update tooling

* Run gofumpt

* go mod vendor
2021-04-08 09:43:39 -07:00
Scott Miller 080c9ca6ba
Fix err shadowing (#11296) 2021-04-07 11:25:23 -05:00
Conrad Lara 3f51589be6
Potential data loss in DynamoDB backend (#10181)
fixes hashicorp/vault#5836

DynamoDB may when throttled return a 2xx response while not committing
all submitted items to the database.

Depending upon load all actions in a BatchWriteUpdate may be throttled
with ProvisionedThroughputExceededException in which case AWS SDK handles
the retry. If some messages were throttled but not all
ProvisionedThroughputExceededException is not returned to the SDK and it
is up to us to resubmit the request.

Using an exponential backoff as recommended in AWS SDK for times we possibly
get partially throttled repeatedly.
2021-04-07 09:44:42 -05:00
Hridoy Roy 049f2513e6
Initial Diagnose Command for TLS and Listener [VAULT-1896, VAULT-1899] (#11249)
* sanity checks for tls config in diagnose

* backup

* backup

* backup

* added necessary tests

* remove comment

* remove parallels causing test flakiness

* comments

* small fix

* separate out config hcl test case into new hcl file

* newline

* addressed comments

* addressed comments

* addressed comments

* addressed comments

* addressed comments

* reload funcs should be allowed to be nil
2021-04-06 16:40:43 -07:00
Nick Cabatoff fb60fd1e30
Use a 10s timeout instead of 3s to reduce test flakiness. The purpose of the timeout is to ensure that if something's broken the test doesn't run forever. However if things are slow, it could lead to the test failing. Also report on whether we got the lock so that if this fix doesn't solve the problem, we have more information to go on. (#11222) 2021-03-29 09:52:32 -04:00
Nick Cabatoff b3af58d758
Expose snapshot_interval tunable instead of setting it in prod code for the sake of a test. (#11160) 2021-03-19 15:41:42 -04:00
Brian Kassouf 28aba513f2
storage/raft: Ensure peers are informed of their correct suffrage when added with AutoPilot (#11155)
* storage/raft: Ensure peers are informed of their correct suffrage when added with AutoPilot

* Add test ensuring peer sets are equivalent
2021-03-19 11:53:50 -07:00
Nick Cabatoff 411495514c
Add a test for server stabilization (#11128) 2021-03-17 17:23:13 -04:00
Hridoy Roy 2213611fa4
fix for mysql flaky test (#11102) 2021-03-17 12:08:34 -07:00
Vishal Nayak fb2df6ca73
Fix autopilot fsm race (#11091)
* Fix autopilot fsm race

* No need to grab backend's lock
2021-03-11 13:14:11 -05:00
Vishal Nayak 9839e76192
Remove unneeded fields from state output (#11073) 2021-03-10 12:08:12 -05:00
Vishal Nayak e5b6ec4d05
Reset IsDead upon each heartbeat (#11049) 2021-03-05 19:50:36 -05:00