Commit graph

232 commits

Author SHA1 Message Date
Mike Jarmy 724af764bb
Test reusable storage (#8983)
* stub out reusable storage test

* implement reusable inmem test

* work on reusable raft test

* stub out simple raft test

* switch to reusable raft storage

* cleanup tests

* cleanup tests

* refactor tests

* verify raft configuration

* cleanup tests

* stub out reuseStorage

* use common base address across clusters

* attempt to reuse raft cluster

* tinker with test

* fix typo

* start debugging

* debug raft configuration

* add BaseClusterListenPort to TestCluster options

* use BaseClusterListenPort in test

* raft join works now

* misc cleanup of raft tests

* use configurable base port for raft test

* clean up raft tests

* add parallelized tests for all backends

* clean up reusable storage tests

* remove debugging code from startClusterListener()

* improve comments in testhelpers

* improve comments in teststorage

* improve comments and test logging

* fix typo in vault/testing

* fix typo in comments

* remove debugging code

* make number of cores parameterizable in test
2020-05-14 08:31:02 -04:00
ncabatoff 862beef59d
Factor out test logger creation so that it can be used when the caller wants to specify a logger explicitly, e.g. because they also want to use it when creating storage. (#8916) 2020-05-05 12:11:36 -04:00
ncabatoff a6809fdbaf
Write NewTestCluster logs to files, which get deleted if the test passes (#8911) 2020-05-01 12:26:26 -04:00
Jeff Mitchell 844b2c3a5d Bump API/SDK and adapt to move from SDK stuff 2020-02-15 14:58:05 -05:00
Vishal Nayak c87d34d1a4
Seal migration with Raft (#8103)
* Seal migration after unsealing

* Refactor migration fields migrationInformation in core

* Perform seal migration as part of postUnseal

* Remove the sleep logic

* Use proper seal in the unseal function

* Fix migration from Auto to Shamir

* Fix the recovery config missing issue

* Address the non-ha migration case

* Fix the multi cluster case

* Avoid re-running seal migration

* Run the post migration code in new leaders

* Fix the issue of wrong recovery being set

* Address review feedback

* Add more complete testing coverage for seal migrations.   (#8247)

* Add more complete testing coverage for seal migrations.  Also remove VAULT_ACC gate from some tests that just depend on docker, cleanup dangling recovery config in storage after migration, and fix a call in adjustCoreForSealMigration that seems broken.

* Fix the issue of wrong recovery key being set

* Adapt tests to work with multiple cores.

* Add missing line to disable raft join.

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* Fix all known issues

* Remove warning

* Review feedback.

* Revert my previous change that broke raft tests.  We'll need to come back and at least comment
this once we better understand why it's needed.

* Don't allow migration between same types for now

* Disable auto to auto tests for now since it uses migration between same types which is not allowed

* Update vault/core.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Add migration logs

* Address review comments

* Add the recovery config check back

* Skip a few steps if migration is already done

* Return from waitForLeadership if migration fails

Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-02-13 16:27:31 -05:00
Jeff Malnick 27ad920399
Add redshift database plugin (#8299)
* feat: add redshift database plugin

* build: update vendored libraries

* docs: add reference doc for redshift variant of the database secrets engine

* feat: set middlewear type name for better metrics naming (#8346)

Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
2020-02-13 09:42:30 -08:00
Jeff Mitchell 86327b8010 Bump api/sdk and fix imports 2020-02-13 10:41:16 -05:00
Jim Kalafut 2ee7b76469
Bundle MongoDB Atlas (#8309) 2020-02-07 14:09:39 -08:00
ncabatoff a0ac4bdd2a
Audit generate root requests and responses. (#8301) 2020-02-06 11:56:37 -05:00
ncabatoff 2c8b012d14
Changes needed so that benchmark-vault can run with Prometheus monitoring (#8295) 2020-02-05 13:45:16 -05:00
Brian Kassouf f32a86ee7a
Create network layer abstraction to allow in-memory cluster traffic (#8173) 2020-01-16 23:03:02 -08:00
Jeff Mitchell a0694943cc
Migrate built in auto seal to go-kms-wrapping (#8118) 2020-01-10 20:39:52 -05:00
ncabatoff 8543da27bd
TestSysRekey_Verification would fail sometimes when recovery=true (#7710)
because when unsealing it wouldn't wait for core 0 to come up and become
the active node. Much of our testing code assumes that core0 is the
active node.
2019-10-22 09:35:48 -04:00
ncabatoff 1c98152fa0
Shamir seals now come in two varieties: legacy and new-style. (#7694)
Shamir seals now come in two varieties: legacy and new-style. Legacy
Shamir is automatically converted to new-style when a rekey operation
is performed. All new Vault initializations using Shamir are new-style.

New-style Shamir writes an encrypted master key to storage, just like
AutoUnseal. The stored master key is encrypted using the shared key that
is split via Shamir's algorithm. Thus when unsealing, we take the key
fragments given, combine them into a Key-Encryption-Key, and use that
to decrypt the master key on disk. Then the master key is used to read
the keyring that decrypts the barrier.
2019-10-18 14:46:00 -04:00
Lexman c86fe212c0
oss changes for entropy augmentation feature (#7670)
* oss changes for entropy augmentation feature

* fix oss command/server/config tests

* update go.sum

* fix logical_system and http/ tests

* adds vendored files

* removes unused variable
2019-10-17 10:33:00 -07:00
Vishal Nayak 0d077d7945
Recovery Mode (#7559)
* Initial work

* rework

* s/dr/recovery

* Add sys/raw support to recovery mode (#7577)

* Factor the raw paths out so they can be run with a SystemBackend.

# Conflicts:
#	vault/logical_system.go

* Add handleLogicalRecovery which is like handleLogical but is only
sufficient for use with the sys-raw endpoint in recovery mode.  No
authentication is done yet.

* Integrate with recovery-mode.  We now handle unauthenticated sys/raw
requests, albeit on path v1/raw instead v1/sys/raw.

* Use sys/raw instead raw during recovery.

* Don't bother persisting the recovery token.  Authenticate sys/raw
requests with it.

* RecoveryMode: Support generate-root for autounseals (#7591)

* Recovery: Abstract config creation and log settings

* Recovery mode integration test. (#7600)

* Recovery: Touch up (#7607)

* Recovery: Touch up

* revert the raw backend creation changes

* Added recovery operation token prefix

* Move RawBackend to its own file

* Update API path and hit it using CLI flag on generate-root

* Fix a panic triggered when handling a request that yields a nil response. (#7618)

* Improve integ test to actually make changes while in recovery mode and
verify they're still there after coming back in regular mode.

* Refuse to allow a second recovery token to be generated.

* Resize raft cluster to size 1 and start as leader (#7626)

* RecoveryMode: Setup raft cluster post unseal (#7635)

* Setup raft cluster post unseal in recovery mode

* Remove marking as unsealed as its not needed

* Address review comments

* Accept only one seal config in recovery mode as there is no scope for migration
2019-10-15 00:55:31 -04:00
Calvin Leung Huang d8875b1991
sys/config: config state endpoint (#7424)
* sys/config: initial work on adding config state endpoint

* server/config: add tests, fix Sanitized method

* thread config through NewTestCluster's config to avoid panic on dev modes

* properly guard endpoint against request forwarding

* add http tests, guard against panics on nil RawConfig

* ensure non-nil rawConfig on NewTestCluster cores

* update non-forwarding logic

* fix imports; use no-forward handler

* add missing config test fixture; update gitignore

* return sanitized config as a map

* fix test, use deep.Equal to check for equality

* fix http test

* minor comment fix

* config: change Sanitized to return snake-cased keys, update tests

* core: hold rlock when reading config; add docstring

* update docstring
2019-10-08 10:57:15 -07:00
Michel Vocks a7a6dd55a5
Add config parameter to allow unauthenticated metrics access (#7550)
* Implement config parameter to allow unathenticated metricss access

* Add unit test for unauthenticated metrics access parameter

* go mod tidy
2019-10-04 09:29:51 +02:00
Calvin Leung Huang 0caff07201
testing: set MetricsHelper if it not provided through base config (#7480) 2019-09-16 17:50:51 -07:00
ncabatoff 366add2979
Generalization of the PhysicalFactory notion introduced by Raft (#7217)
Generalization of the PhysicalFactory notion introduced by Raft, so it can be used by other storage backends in tests.  These are the OSS changes needed for my rework of the ent integ tests and cluster helpers.
2019-08-23 15:51:25 -04:00
ncabatoff e1f8a82d81
Create test cores with an error injector. (#7243)
It's created with a 0% error rate, which means it's a no-op, but tests can opt-in to errors when
needed via core.underlyingPhysical.
2019-08-06 15:21:23 -04:00
Jeff Mitchell f0a9d9910f
Revert "Generalize and improve testcluster-building code (#7177)" (#7212)
This reverts commit d177fc7dd1aa7beda8b1c5bf5a8a4e85d725cca5.
2019-07-30 01:04:06 -04:00
ncabatoff 7512a71a4c
Generalize and improve testcluster-building code (#7177)
There are a few different things happening in this change.  First, some code that previously lived in enterprise has moved here: this includes some helper code for manipulating clusters and for building storage backends.  Second, the existing cluster-building code using inmem storage has been generalized to allow various storage backends. Third, added support for creating two-cluster DR setups. Finally, there are tweaks to handle edge cases that
result in intermittent failures, or to eliminate sleeps in favour of polling to detect state changes.

Also: generalize TestClusterOptions.PhysicalFactory so it can be used either
as a per-core factory (for raft) or a per-cluster factory (for other
storage backends.)
2019-07-26 16:42:51 -04:00
ncabatoff f8cefa9b18
When cleaning up a TestCluster, set each core's logger to level Error. (#7124)
This eliminates a bunch of noise at the end of a test's logs, making it
easier to diagnose test failures.

Add TestCluster.Logger.  This is intendend to be used when test helpers
want to log something. If TestClusterOptions.Logger is non-nil, it will
be used as the cluster logger, otherwise we create one based on test
name.  In the absence of CoreConfig.Logger, this logger is also used as
the parent of each core's log.  This makes it easy when running parallel
tests to identify which log message came from which test.
2019-07-23 15:17:37 -04:00
Mike Jarmy e0ce2195cc AWS upgrade role entries (#7025)
* upgrade aws roles

* test upgrade aws roles

* Initialize aws credential backend at mount time

* add a TODO

* create end-to-end test for builtin/credential/aws

* fix bug in initializer

* improve comments

* add Initialize() to logical.Backend

* use Initialize() in Core.enableCredentialInternal()

* use InitializeRequest to call Initialize()

* improve unit testing for framework.Backend

* call logical.Backend.Initialize() from all of the places that it needs to be called.

* implement backend.proto changes for logical.Backend.Initialize()

* persist current role storage version when upgrading aws roles

* format comments correctly

* improve comments

* use postUnseal funcs to initialize backends

* simplify test suite

* improve test suite

* simplify logic in aws role upgrade

* simplify aws credential initialization logic

* simplify logic in aws role upgrade

* use the core's activeContext for initialization

* refactor builtin/plugin/Backend

* use a goroutine to upgrade the aws roles

* misc improvements and cleanup

* do not run AWS role upgrade on DR Secondary

* always call logical.Backend.Initialize() when loading a plugin.

* improve comments

* on standbys and DR secondaries we do not want to run any kind of upgrade logic

* fix awsVersion struct

* clarify aws version upgrade

* make the upgrade logic for aws auth more explicit

* aws upgrade is now called from a switch

* fix fallthrough bug

* simplify logic

* simplify logic

* rename things

* introduce currentAwsVersion const to track aws version

* improve comments

* rearrange things once more

* conglomerate things into one function

* stub out aws auth initialize e2e test

* improve aws auth initialize e2e test

* finish aws auth initialize e2e test

* tinker with aws auth initialize e2e test

* tinker with aws auth initialize e2e test

* tinker with aws auth initialize e2e test

* fix typo in test suite

* simplify logic a tad

* rearrange assignment

* Fix a few lifecycle related issues in #7025 (#7075)

* Fix panic when plugin fails to load
2019-07-05 16:55:40 -07:00
Calvin Leung Huang 5428ab50ee audit: log invalid wrapping token request/response (#6541)
* audit: log invalid wrapping token request/response

* Update helper/consts/error.go

Co-Authored-By: calvn <cleung2010@gmail.com>

* update error comments

* Update vault/wrapping.go

Co-Authored-By: calvn <cleung2010@gmail.com>

* update comment

* move validateWrappingToken out of http and into logical

* minor refactor, add test cases

* comment rewording

* refactor validateWrappingToken to perform audit logging

* move ValidateWrappingToken back to wrappingVerificationFunc

* Fix tests

* Review feedback
2019-07-05 14:15:14 -07:00
Brian Kassouf 4d7d0d729a
storage/raft: When restoring a snapshot preseal first (#7011)
* storage/raft: When restoring a snapshot preseal first

* best-effort allow standbys to apply the restoreOp before sealing active node

* Don't cache the raft tls key

* Update physical/raft/raft.go

* Move pending raft peers to core

* Fix race on close bool

* Extend the leaderlease time for tests

* Update raft deps

* Fix audit hashing

* Fix race with auditing
2019-07-03 13:56:30 -07:00
ncabatoff d2beeefe79
Add support for hashing time.Time within slices (#6767)
Add support for hashing time.Time within slices, which unbreaks auditing of requests returning the request counters.  

Break Hash into struct-specific func like HashAuth, HashRequest. Move all the copying/hashing logic from FormatRequest/FormatResponse into the new Hash* funcs.  HashStructure now modifies in place instead of copying.

Instead of returning an error when trying to hash map keys of type time.Time, ignore them, i.e. pass them through unhashed.

Enable auditing on test clusters by default if the caller didn't specify any audit backends.  If they do, they're responsible for setting it up.
2019-07-02 18:18:40 -04:00
Jeff Mitchell edf0f3cbbd Use metrics helper from base in test cluster cores 2019-07-01 12:08:55 -04:00
Jeff Mitchell cf93f60bb1
Allow turning on client auth in test clusters (#6958) 2019-06-23 21:50:27 -04:00
Brian Kassouf 17b46e2979
Fix key upgrade and raft tests (#6949) 2019-06-21 11:38:21 -06:00
Brian Kassouf ed14061578
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
Becca Petrin f6703c3630 fix test 2019-06-18 14:04:08 -07:00
ncabatoff cc41e608dc
Add core.coreNumber field, used to differentiate multiple cores/clusters when running tests (#6855)
This is not used or exposed in prod.

Remove some test-specific code from the cluster-building helpers. The corresponding additions go on the ent side.
2019-06-10 14:07:16 -04:00
ncabatoff ad28263b69
Allow plugins to submit audit requests/responses via extended SystemView (#6777)
Move audit.LogInput to sdk/logical.  Allow the Data values in audited
logical.Request and Response to implement OptMarshaler, in which case
we delegate hashing/serializing responsibility to them.  Add new
ClientCertificateSerialNumber audit request field.

SystemView can now be cast to ExtendedSystemView to expose the Auditor
interface, which allows submitting requests and responses to the audit
broker.
2019-05-22 18:52:53 -04:00
Jeff Mitchell 9ebc57581d
Switch to go modules (#6585)
* Switch to go modules

* Make fmt
2019-04-13 03:44:06 -04:00
Jeff Mitchell 8bcb533a1b
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
Jeff Mitchell 3ec1e79b6a Fix test code breakage 2019-04-12 11:37:34 -04:00
Jeff Mitchell 4aa4a0665f Sync over 2019-04-11 11:12:37 -04:00
ncabatoff b08963779c Apply the bits from ent pr 852 that are relevant to OSS. (#6528) 2019-04-04 13:02:44 -04:00
Calvin Leung Huang 70108668a9
testing: allow logger and audit backends to be set in TestCoreWithSealAndUI (#6388) 2019-03-08 13:46:24 -08:00
ncabatoff cd747c9318
Add code for writing and reading request counters to storage. (#5918)
Increment a counter whenever a request is received. 
The in-memory counter is persisted to counters/requests/YYYY/MM.
When the month wraps around, we reset the in-memory counter to
zero.
Add an endpoint for querying the request counters across all time.
2019-03-05 14:55:07 -05:00
Jeff Mitchell a3a2a3cd04 A few more syncs 2019-03-04 13:53:15 -05:00
Jeff Mitchell e5ca13d0be
Don't add kv by default in non-dev scenarios (#6109)
It's retained for tests though since most rely on it.
2019-02-14 11:55:32 -08:00
Jeff Mitchell 9ef0680e7f
Fix leader info repopulation (#6167)
* Two things:

* Change how we populate and clear leader UUID. This fixes a case where
if a standby disconnects from an active node and reconnects, without the
active node restarting, the UUID doesn't change so triggers on a new
active node don't get run.

* Add a bunch of test helpers and minor updates to things.
2019-02-05 21:01:18 -05:00
Giacomo Tirabassi 0d3845c537 Influxdb secret engine built-in plugin (#5924)
* intial work for influxdb secret plugin

* fixed typo

* added comment

* added documentation

* added tests

* fixed tests

* added vendoring

* minor testing issue with hardcoded values

* minor fixes
2019-01-08 17:26:16 -08:00
Jim Kalafut d0e2badbae Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00
Brian Kassouf 737b7e6651
Fix plugin reload when in a namespace (#5937) 2018-12-11 17:21:23 -08:00
Brian Kassouf 33776b89c2
Wrap storage calls with encoding checks (#5819)
* Add encoding backend

* More work on encoding checks

* Update error message

* Update physical/encoding.go

* Disable key checks if configured
2018-11-19 13:13:16 -08:00
Vishal Nayak b4836575fb
Test for issue 5729 (#5750)
* Test for 5729

* Remove unneeded space

Co-Authored-By: vishalnayak <vishalnayak@users.noreply.github.com>
2018-11-13 11:16:10 -05:00
Becca Petrin 7bd22e6779
Run all builtins as plugins (#5536) 2018-11-06 17:21:24 -08:00
Jeff Mitchell 82992d6097
Seal migration (OSS) (#781) 2018-10-22 23:34:02 -07:00
Calvin Leung Huang 189b893b35
Add ability to provide env vars to plugins (#5359)
* Add ability to provide env vars to plugins

* Update docs

* Update docs with examples

* Refactor TestAddTestPlugin, remove TestAddTestPluginTempDir
2018-09-20 10:50:29 -07:00
Jeff Mitchell 919b968c27
The big one (#5346) 2018-09-17 23:03:00 -04:00
Becca Petrin 7a8c116fb1
undo make fmt (#5265) 2018-09-04 09:29:18 -07:00
Becca Petrin ed7639b0ec
run make fmt (#5261) 2018-09-04 09:12:59 -07:00
Calvin Leung Huang 9988ace85e gofmt files (#5233) 2018-08-31 09:15:40 -07:00
Jeff Mitchell 3b01b29056
Pass in an ErrorLog to http.Server (#5135)
Fixes #5108
2018-08-21 11:23:18 -04:00
Jeff Mitchell 4261618d10 Add request timeouts in normal request path and to expirations (#4971)
* Add request timeouts in normal request path and to expirations

* Add ability to adjust default max request duration

* Some test fixes

* Ensure tests have defaults set for max request duration

* Add context cancel checking to inmem/file

* Fix tests

* Fix tests

* Set default max request duration to basically infinity for this release for BC

* Address feedback
2018-07-24 14:50:49 -07:00
Jeff Mitchell 9687ccc8fa Tackle #4929 a different way (#4932)
* Tackle #4929 a different way

This turns c.sealed into an atomic, which allows us to call sealInternal
without a lock. By doing so we can better control lock grabbing when a
condition causing the standby loop to get out of active happens. This
encapsulates that logic into two distinct pieces (although they could
be combined into one), and makes lock guarding more understandable.

* Re-add context canceling to the non-HA version of sealInternal

* Return explicitly after stopCh triggered
2018-07-24 13:57:25 -07:00
Jeff Mitchell 8b0561aad8
Two small items: (#4934)
1) Disable MaxRetries in test cluster clients. We generally want to fail
as fast as possible in tests so adding unpredictable timing in doesn't
help things, especially if we're timing sensitive in the test.

2) EquivalentPolicies is supposed to return true if only one set
contains `default` and the other is empty, but if one set was nil
instead of simply a zero length slice it would always return false. This
means that renewing against, say, `userpass` when not actually
specifying any user policies would always fail.
2018-07-17 01:23:26 -04:00
Jeff Mitchell 4a3fe87a39
Allow max request size to be user-specified (#4824)
* Allow max request size to be user-specified

This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.

This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.

* Switch to ioutil.ReadAll()
2018-07-06 15:44:56 -04:00
Vishal Nayak b10c2a87fa Refactor and rewrite the test (#4796) 2018-06-19 16:59:03 -04:00
Vishal Nayak 0d8f424ab4
disallow token use if entity is invalid (#4791) 2018-06-19 12:57:19 -04:00
Jeff Mitchell 5207099042 Offline token revocation fix 2018-06-05 18:53:27 -04:00
Jeff Mitchell 6340add8c1 Finish non-recovery test 2018-05-20 02:42:15 -04:00
Becca Petrin abb621752f Clean up error string formatting (#4304) 2018-04-09 14:35:21 -04:00
Becca Petrin 03cf302e9a Move to "github.com/hashicorp/go-hclog" (#4227)
* logbridge with hclog and identical output

* Initial search & replace

This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.

* strip logbridge

* fix majority of tests

* update logxi aliases

* WIP fixing tests

* more test fixes

* Update test to hclog

* Fix format

* Rename hclog -> log

* WIP making hclog and logxi love each other

* update logger_test.go

* clean up merged comments

* Replace RawLogger interface with a Logger

* Add some logger names

* Replace Trace with Debug

* update builtin logical logging patterns

* Fix build errors

* More log updates

* update log approach in command and builtin

* More log updates

* update helper, http, and logical directories

* Update loggers

* Log updates

* Update logging

* Update logging

* Update logging

* Update logging

* update logging in physical

* prefixing and lowercase

* Update logging

* Move phyisical logging name to server command

* Fix som tests

* address jims feedback so far

* incorporate brians feedback so far

* strip comments

* move vault.go to logging package

* update Debug to Trace

* Update go-plugin deps

* Update logging based on review comments

* Updates from review

* Unvendor logxi

* Remove null_logger.go
2018-04-02 17:46:59 -07:00
Jeff Mitchell 26c891c223 Merge branch 'master-oss' into 0.10-beta 2018-03-28 10:14:43 -04:00
Jeff Mitchell 48cb30312a
Remove old workaround for a rollback error (#4206)
It can now cause problems in other situations
2018-03-27 16:34:06 -04:00
Brian Kassouf cc625e19ee
Add options to mount tune and mount endpoints in preparation for versioning (#4155)
* Add some requirements for versioned k/v

* Add a warning message when an upgrade is triggered

* Add path help values

* Make the kv header a const

* Add the uid to mount entry instead of options map

* Pass the backend aware uuid to the mounts and plugins

* Fix comment

* Add options to secret/auth enable and tune CLI commands (#4170)

* Switch mount/tune options to use TypeKVPairs (#4171)

* switching options to TypeKVPairs, adding bool parse for versioned flag

* flipping bool check

* Fix leases coming back from non-leased pluin kv store

* add a test for updating mount options

* Fix tests
2018-03-21 12:04:27 -07:00
Brian Kassouf 9dba3590ac
Add context to the NewSalt function (#4102) 2018-03-08 11:21:11 -08:00
Calvin Leung Huang e2fb199ce5
Non-HMAC audit values (#4033)
* Add non-hmac request keys

* Update comment

* Initial audit request keys implementation

* Add audit_non_hmac_response_keys

* Move where req.NonHMACKeys gets set

* Minor refactor

* Add params to auth tune endpoints

* Sync cache on loadCredentials

* Explicitly unset req.NonHMACKeys

* Do not error if entry is nil

* Add tests

* docs: Add params to api sections

* Refactor audit.Backend and Formatter interfaces, update audit broker methods

* Add audit_broker.go

* Fix method call params in audit backends

* Remove fields from logical.Request and logical.Response, pass keys via LogInput

* Use data.GetOk to allow unsetting existing values

* Remove debug lines

* Add test for unsetting values

* Address review feedback

* Initialize values in FormatRequest and FormatResponse using input values

* Update docs

* Use strutil.StrListContains

* Use strutil.StrListContains
2018-03-02 12:18:39 -05:00
Jeff Mitchell 4669f37c78 Add four cluster flag 2018-02-22 00:23:37 -05:00
Vishal Nayak bf66dc2841 Make mount view read only until after mount persist (#3910) 2018-02-09 14:04:25 -05:00
Vishal Nayak 150ad8405b
Remove logical.Initialize() method (#3848)
* Remove logical.Initialize() method

* More cleanup

* Fix test
2018-01-25 20:19:27 -05:00
Jeff Mitchell 33b68ebf3d Remove context from a few extraneous places 2018-01-19 03:44:06 -05:00
Brian Kassouf 2f19de0305 Add context to storage backends and wire it through a lot of places (#3817) 2018-01-19 01:44:44 -05:00
Jeff Mitchell 842a3a4a05 Remove RegisterLicense from logical.Backend
It's almost certainly the wrong signature and nothing uses it currently
anyways.
2018-01-18 13:44:29 -05:00
Tom Wilkie e06cb595fc Make the plugin catalog endpoint roundtrip so we can use terraform to manage them. (#3778) 2018-01-17 19:19:28 -05:00
Brian Kassouf 1c190d4bda
Pass context to backends (#3750)
* Start work on passing context to backends

* More work on passing context

* Unindent logical system

* Unindent token store

* Unindent passthrough

* Unindent cubbyhole

* Fix tests

* use requestContext in rollback and expiration managers
2018-01-08 10:31:38 -08:00
Chris Hoffman 164849f056
Add support for encrypted TLS key files (#3685) 2017-12-15 17:33:55 -05:00
Calvin Leung Huang 57bc19c169 Defer reader.Close that is used to determine sha256 2017-12-15 14:04:09 -05:00
Calvin Leung Huang 7c0b4f1333 Avoid unseal failure if plugin backends fail to setup during postUnseal (#3686) 2017-12-15 13:31:57 -05:00
Chris Hoffman 822ce95dc4
adding ability to override temp dir in dev cluster (#3673) 2017-12-11 18:02:35 -05:00
Jeff Mitchell 548629e8ef Port over some changes 2017-11-30 09:43:07 -05:00
Jeff Mitchell 0798ccdd7c Populate config error in three node test function 2017-11-10 17:44:25 -05:00
Jeff Mitchell e66c4b11d5 Add core numbers to output in dev three node 2017-11-10 16:21:46 -05:00
Jeff Mitchell 18f76d0909 Change test cluster to request client cert rather than verify if given 2017-11-09 15:55:23 -05:00
Jeff Mitchell dd551eb12f Put back original test cluster client redirect behavior 2017-11-08 10:15:56 -05:00
Jeff Mitchell a2db5671c5 Add logbridge for testing 2017-11-07 15:50:09 -05:00
Calvin Leung Huang d9eaacf5de
Barrier unseal using recovery keys (#3541)
* Barrier unseal using recovery keys

* Remove tests
2017-11-07 15:15:39 -05:00
Jeff Mitchell d229d7d5b0
Redo API locking (#3508)
* Redo the API client quite a bit to make the behavior of NewClient more
predictable and add locking to make it safer to use with Clone() and if
multiple goroutines for some reason decide to change things.

Along the way I discovered that currently, the x/net/http2 package is
broke with the built-in h2 support in released Go. For those using
DefaultConfig (the vast majority of cases) this will be a non-event.
Others can manually call http2.ConfigureTransport as needed. We should
keep an eye on commits on that repo and consider more updates before
release. Alternately we could go back revisions but miss out on bug
fixes; my theory is that this is not a purposeful break and I'll be
following up on this in the Go issue tracker.

In a few tests that don't use NewTestCluster, either for legacy or other
reasons, ensure that http2.ConfigureTransport is called.

* Use tls config cloning

* Don't http2.ConfigureServer anymore as current Go seems to work properly without requiring the http2 package

* Address feedback
2017-11-02 09:30:04 -05:00
Jeff Mitchell 47dae8ffc7 Sync 2017-10-23 14:59:37 -04:00
Jeff Mitchell d6a9a770c1 Sync some seal testing stuff 2017-10-23 13:42:04 -04:00
Vishal Nayak f7ed6732a5 Porting identity store (#3419)
* porting identity to OSS

* changes that glue things together

* add testing bits

* wrapped entity id

* fix mount error

* some more changes to core

* fix storagepacker tests

* fix some more tests

* fix mount tests

* fix http mount tests

* audit changes for identity

* remove upgrade structs on the oss side

* added go-memdb to vendor
2017-10-11 10:21:20 -07:00
Chris Hoffman 1029ad3b33 Rename "generic" secret backend to "kv" (#3292) 2017-09-15 09:02:29 -04:00
Chris Hoffman 9d73c81f38 Disable the sys/raw endpoint by default (#3329)
* disable raw endpoint by default

* adding docs

* config option raw -> raw_storage_endpoint

* docs updates

* adding listing on raw endpoint

* reworking tests for enabled raw endpoints

* root protecting base raw endpoint
2017-09-15 00:21:35 -04:00
Chris Hoffman 71952b7738 ExpirationManager restoration to load in the background (#3260) 2017-09-05 11:09:00 -04:00
Jeff Mitchell d7e933a072 Fix inverted skipinit logic 2017-09-04 18:39:57 -04:00