Commit graph

1833 commits

Author SHA1 Message Date
Brian Kassouf efe5671f36 make fmt 2019-02-20 12:12:21 -08:00
Brian Kassouf 26d8d318d7 Merge remote-tracking branch 'oss/master' into 1.1-beta 2019-02-19 12:17:15 -08:00
Brian Kassouf ab1a04f8e6
Port over some test fixes (#6261) 2019-02-19 12:03:02 -08:00
Martin 4c9e7f4478 typo in pathsToRadix doc (#6253) 2019-02-18 10:05:04 -08:00
Lexman 1ed2f2210b use deep.Equal instead of reflect.DeepEqual in some failing tests (#6249)
* use deep.Equal instead of reflect.DeepEqual in some failing tests

* changed test output a little bit
2019-02-15 14:00:14 -08:00
ncabatoff 4cf1e24a9e Fix TestSystemBackend_mount, TestSystemBackend_mounts. (#6247) 2019-02-15 11:14:45 -08:00
Calvin Leung Huang ac0b31b544 fix TestCore_Unmount 2019-02-15 11:13:26 -08:00
Calvin Leung Huang 35ecaa2b6b fix TestSystemBackend_InternalUIMounts 2019-02-15 10:46:38 -08:00
Jeff Mitchell 3dfa30acb4 Add ability to use path wildcard segments (#6164)
* Path globbing

* Add glob support at the beginning

* Ensure when evaluating an ACL that our path never has a leading slash. This already happens in the normal request path but not in tests; putting it here provides it for tests and extra safety in case the request path changes

* Simplify the algorithm, we don't really need to validate the prefix first as glob won't apply if it doesn't

* Add path segment wildcarding

* Disable path globbing for now

* Remove now-unneeded test

* Remove commented out globbing bits

* Remove more holdover glob bits

* Rename k var to something more clear
2019-02-14 18:31:43 -08:00
Brian Kassouf f5b5fbb392
Refactor the cluster listener (#6232)
* Port over OSS cluster port refactor components

* Start forwarding

* Cleanup a bit

* Fix copy error

* Return error from perf standby creation

* Add some more comments

* Fix copy/paste error
2019-02-14 18:14:56 -08:00
Martin c9918b93e8 Avoid panic at login when external group has a nil alias (#6230) 2019-02-14 12:57:20 -08:00
Martin 9044173d6e Prometheus support on v1/sys/metrics endpoint (#5308)
* initial commit for prometheus and sys/metrics support

* Throw an error if prometheusRetentionTime is 0,add prometheus in devmode

* return when format=prometheus is used and prom is disable

* parse prometheus_retention_time from string instead of int

* Initialize config.Telemetry if nil

* address PR issues

* add sys/metrics framework.Path in a factory

* Apply requiredMountTable entries's MountConfig to existing core table

* address pr comments

* enable prometheus sink by default

* Move Metric-related code in a separate metricsutil helper
2019-02-14 12:46:59 -08:00
Jeff Mitchell e5ca13d0be
Don't add kv by default in non-dev scenarios (#6109)
It's retained for tests though since most rely on it.
2019-02-14 11:55:32 -08:00
Jeff Mitchell 5fa9e48b21
Don't duplicate CORS headers (#6207)
Fixes #6182
2019-02-11 13:10:26 -05:00
Jeff Mitchell ebf57b15b4
Fixes a regression in forwarding from #6115 (#6191)
* Fixes a regression in forwarding from #6115

Although removing the authentication header is good defense in depth,
for forwarding mechanisms that use the raw request, we never add it
back. This caused perf standby tests to throw errors. Instead, once
we're past the point at which we would do any raw forwarding, but before
routing the request, remove the header.

To speed this up, a flag is set in the logical.Request to indicate where
the token is sourced from. That way we don't iterate through maps
unnecessarily.
2019-02-11 13:08:15 -05:00
Vishal Nayak b4ba344782
Merge entities during unseal only on the primary (#6075)
* Merge entities during unseal only on the primary

* Add another guard check

* Add perf standby to the check

* Make primary to not differ from case-insensitivity status w.r.t secondaries

* Ensure mutual exclusivity between loading and invalidations

* Both primary and secondaries won't persist during startup and invalidations

* Allow primary to persist when loading case sensitively

* Using core.perfStandby

* Add a tweak in core for testing

* Address review feedback

* update memdb but not storage in secondaries

* Wire all the things directly do mergeEntity

* Fix persist behavior

* Address review feedback
2019-02-08 16:32:06 -05:00
Jeff Mitchell 9ef0680e7f
Fix leader info repopulation (#6167)
* Two things:

* Change how we populate and clear leader UUID. This fixes a case where
if a standby disconnects from an active node and reconnects, without the
active node restarting, the UUID doesn't change so triggers on a new
active node don't get run.

* Add a bunch of test helpers and minor updates to things.
2019-02-05 21:01:18 -05:00
Jeff Mitchell 5f249d4005
Add allowed_response_headers (#6115) 2019-02-05 16:02:15 -05:00
Brian Kassouf aaca35be94
Updates to recovery keys (#6152) 2019-02-01 11:29:55 -08:00
Jeff Mitchell bbc1d53a5d Revert "Refactor common token fields and operations into a helper (#5953)"
This reverts commit 66c226c593bb1cd48cfd8364ac8510cb42b7d67a.
2019-02-01 11:23:40 -05:00
Jeff Mitchell 27c960d8df
Split SubView functionality into logical.StorageView (#6141)
This lets other parts of Vault that can't depend on the vault package
take advantage of the subview functionality.

This also allows getting rid of BarrierStorage and vault.Entry, two
totally redundant abstractions.
2019-01-31 09:25:18 -05:00
Jeff Mitchell 85a560abba
Refactor common token fields and operations into a helper (#5953) 2019-01-30 16:23:28 -05:00
Matthew Irish b777906fee
add entity lookup to the default policy (#6105)
* add entity lookup to the default policy

* only use id for lookup

* back in with name
2019-01-29 09:43:59 -06:00
Jeff Mitchell e781ea3ac4
First part of perf standby entity race fix (#6106) 2019-01-25 14:08:42 -05:00
Jeff Mitchell 1f57e3674a Move a common block up a level 2019-01-24 18:29:22 -05:00
Jeff Mitchell 6d22f3fc2e minor linting change 2019-01-23 17:19:06 -05:00
Seth Vargo 98ad431d6d Continuously attempt to unseal if sealed keys are supported (#6039)
* Add helper for checking if an error is a fatal error

The double-double negative was really confusing, and this pattern is used a few places in Vault. This negates the double negative, making the devx a bit easier to follow.

* Check return value of UnsealWithStoredKeys in sys/init

* Return proper error types when attempting unseal with stored key

Prior to this commit, "nil" could have meant unsupported auto-unseal, a transient error, or success. This updates the function to return the correct error type, signaling to the caller whether they should retry or fail.

* Continuously attempt to unseal if sealed keys are supported

This fixes a bug that occurs on bootstrapping an initial cluster. Given a collection of Vault nodes and an initialized storage backend, they will all go into standby waiting for initialization. After one node is initialized, the other nodes had no mechanism by which they "re-check" to see if unseal keys are present. This adds a goroutine to the server command which continually waits for unseal keys to exist. It exits in the following conditions:

- the node is unsealed
- the node does not support stored keys
- a fatal error occurs (as defined by Vault)
- the server is shutting down

In all other situations, the routine wakes up at the specified interval and attempts to unseal with the stored keys.
2019-01-23 16:34:34 -05:00
Jeff Mitchell c5d8391c38
Prefix path rename (#6089)
* Rename Prefix -> Path in internal struct

* Update test
2019-01-23 15:04:49 -05:00
Jeff Mitchell a11f2a3ba2
Rename glob -> prefix in ACL internals (#6086)
Really, it's a prefix
2019-01-23 13:55:40 -05:00
Jeff Mitchell 59bc9dd361 Add missing value to policy ShallowClone
Not related to a bug, just happened to notice it.
2019-01-23 13:20:04 -05:00
Jim Kalafut f097b8d934
Update existing alias metadata during authentication (#6068) 2019-01-23 08:26:50 -08:00
Stuart Moore 1e836c1f67 Typo fix in gcpckms.go (#6081) 2019-01-23 07:52:31 -05:00
Jeff Mitchell c7ac2e449a Sync up code 2019-01-22 17:44:13 -05:00
Jeff Mitchell 797c622567 Sync seal testing 2019-01-22 17:23:20 -05:00
Jeff Mitchell 9653f9e379 Sync logical_system 2019-01-22 17:21:53 -05:00
Becca Petrin 83e0c5e5e6
Check ec2 instance metadata for region (#6025) 2019-01-18 14:49:24 -08:00
Seth Vargo 018ec9cdb8 Upgrade to new Cloud KMS client libraries (#6051)
* Upgrade to new Cloud KMS client libraries

We recently released the new Cloud KMS client libraries which use GRPC
instead of HTTP. They are faster and look nicer (</opinion>), but more
importantly they drastically simplify a lot of the logic around client
creation, encryption, and decryption. In particular, we can drop all the
logic around looking up credentials and base64-encoding/decoding.

Tested on a brand new cluster (no pre-existing unseal keys) and against
a cluster with stored keys from a previous version of Vault to ensure no
regressions.

* Use the default scopes the client requests

The client already does the right thing here, so we don't need to
surface it, especially since we aren't allowing users to configure it.
2019-01-17 17:27:15 -05:00
Jeff Mitchell 440ef3b42e Fix bound cidrs propagation 2019-01-15 10:55:36 -05:00
Becca Petrin 015f641ada Return errs from parsing unknown plugins (#6038) 2019-01-15 10:51:55 -05:00
Jeff Mitchell 78b4ff570f Expose error so warning about not decoding accessors is more useful (#6034) 2019-01-14 09:55:49 -08:00
ncabatoff e78ca24d17 Instead of using the (nil) err, use the non-nil newCtErr for ctErr. (#5841) 2019-01-09 17:39:58 -08:00
Vishal Nayak 1119f47e13 Cubbyhole cleanup (#6006)
* fix cubbyhole deletion

* Fix error handling

* Move the cubbyhole tidy logic to token store and track the revocation count

* Move fetching of cubby keys before the tidy loop

* Fix context getting cancelled

* Test the cubbyhole cleanup logic

* Add progress counter for cubbyhole cleanup

* Minor polish

* Use map instead of slice for faster computation

* Add test for cubbyhole deletion

* Add a log statement for deletion

* Add SHA1 hashed tokens into the mix
2019-01-09 10:53:41 -08:00
Giacomo Tirabassi 0d3845c537 Influxdb secret engine built-in plugin (#5924)
* intial work for influxdb secret plugin

* fixed typo

* added comment

* added documentation

* added tests

* fixed tests

* added vendoring

* minor testing issue with hardcoded values

* minor fixes
2019-01-08 17:26:16 -08:00
Jim Kalafut d0e2badbae Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00
Seth Vargo c3f1043c24 Reduce required permissions for the GCPCKMS auto-unsealer (#5999)
This changes the behavior of the GCPCKMS auto-unsealer setup to attempt
encryption instead of a key lookup. Key lookups are a different API
method not covered by roles/cloudkms.cryptoKeyEncrypterDecrypter. This
means users must grant an extended scope to their service account
(granting the ability to read key data) which only seems to be used to
validate the existence of the key.

Worse, the only roles that include this permission are overly verbose
(e.g. roles/viewer which gives readonly access to everything in the
project and roles/cloudkms.admin which gives full control over all key
operations). This leaves the user stuck between choosing to create a
custom IAM role (which isn't fun) or grant overly broad permissions.

By changing to an encrypt call, we get better verification of the unseal
permissions and users can reduce scope to a single role.
2019-01-04 16:29:31 -05:00
Jeff Mitchell c8a029210b
Don't read AWS env vars (#5974)
* Don't read AWS env vars

Let AWS SDK env cred chain provider do it for us

Fixes #5965
2019-01-04 15:03:57 -05:00
Jim Kalafut 2547d7fb6a
Simplify base62.Random (#5982)
Also move existing base62 encode/decode operations to their only points
of use.
2018-12-20 07:40:01 -08:00
Brian Kassouf 75e25711a0
Default seal type to Shamir on older seal configs (#5956) 2018-12-13 16:44:56 -08:00
Jim Kalafut 5687892530
Add operationId field to OpenAPI output (#5876)
Fixes #5842
2018-12-12 13:59:23 -08:00
Brian Kassouf 737b7e6651
Fix plugin reload when in a namespace (#5937) 2018-12-11 17:21:23 -08:00
Jeff Mitchell 394ce75e33
Move some handle request functions up a level (#5889)
* Move some handle request functions up a level

Add clearing of token entry from request, fixing a test

* Update request_handling.go
2018-12-03 14:35:20 -05:00
Calvin Leung Huang e71017e5a9 Set request token entry within fetchACLTokenEntryAndEntity (#5880) 2018-12-03 11:57:53 -05:00
Calvin Leung Huang c2e87c20d8 Remove pt value from error output 2018-12-03 09:19:16 -05:00
Calvin Leung Huang be05907515 autoseal: fix error typos (#5877) 2018-11-29 16:07:08 -08:00
Calvin Leung Huang a510537778 Reset rekey progress once threshold has been met (#5743)
* Reset rekey progress once threshold has been met

* Reverting log message changes

* Add progress check on invalid rekey test

* Minor comment update
2018-11-19 17:03:07 -08:00
Calvin Leung Huang 907dd834ff Revert deprecated plugin var names (#5822)
* Revert field back to ListPluginsResponse.Names

* Revert field back to MountConfig.PluginName and APIMountConfig.PluginName
2018-11-19 15:23:48 -08:00
Calvin Leung Huang ad3e105012
Mount tune options (#5809)
* Refactor mount tune to support upsert options values and unset options.

* Do not allow unsetting options map

* add secret tune version regression test

* Only accept valid options version

* s/meVersion/optVersion/
2018-11-19 14:23:25 -08:00
Jeff Mitchell 127413461b
Remove token store paths with token/accessors in URLs (#5773) 2018-11-19 16:58:19 -05:00
Brian Kassouf 33776b89c2
Wrap storage calls with encoding checks (#5819)
* Add encoding backend

* More work on encoding checks

* Update error message

* Update physical/encoding.go

* Disable key checks if configured
2018-11-19 13:13:16 -08:00
Brian Kassouf c16f7485e7
perf-standby: Fix audit table upgrade on standbys (#5811) 2018-11-19 10:21:53 -08:00
Konstantinos Tsanaktsidis f75e3603ba Paper over GCS backend corruption issues (#5804)
We're having issues with leases in the GCS backend storage being
corrupted and failing MAC checking. When that happens, we need to know
the lease ID so we can address the corruption by hand and take
appropriate action.

This will hopefully prevent any instances of incomplete data being sent
to GSS
2018-11-16 08:07:06 -05:00
Vishal Nayak 43e3ff808a
Update group memberships when entity is deleted (#5786)
* Use common abstraction for entity deletion

* Update group memberships before deleting entity

* Added test

* Fix return statements

* Update comment

* Cleanup member entity IDs while loading groups

* Added test to ensure that upgrade happens properly

* Ensure that the group gets persisted if upgrade code modifies it
2018-11-15 20:07:45 -05:00
Calvin Leung Huang 227a664b06 Continue on plugin registration error in dev mode (#5791)
* Continue on plugin registration error in dev mode

* Continue only on unknown type error

* Continue only on unknown type error

* Print plugin registration error on exit

Co-Authored-By: calvn <cleung2010@gmail.com>
2018-11-15 16:55:24 -08:00
Calvin Leung Huang e99957aed9
Support registering plugin with name only (#5787)
* Support registering plugin with name only

* Make RegisterPlugin backwards compatible

* Add CLI backwards compat command to plugin info and deregister

* Add server-side deprecation warnings if old read/dereg API endpoints are called

* Address feedback
2018-11-15 14:33:11 -08:00
Vishal Nayak 56d6d929ce Fix sys/auth/path/tune to accept token_type (#5777) 2018-11-14 11:22:08 -08:00
vishalnayak c6faa3ee28 Add a comment to retain misspelling 2018-11-13 13:30:42 -05:00
vishalnayak a96641c86f Fix TestPolicy_ParseBadPath 2018-11-13 13:22:56 -05:00
Jeff Mitchell 9735bd7d69 Fix more awskms test brokenness 2018-11-13 13:01:40 -05:00
Jeff Mitchell e5aad14d79 Fix test where AWS wasn't being skipped properly 2018-11-13 12:45:30 -05:00
Vishal Nayak b4836575fb
Test for issue 5729 (#5750)
* Test for 5729

* Remove unneeded space

Co-Authored-By: vishalnayak <vishalnayak@users.noreply.github.com>
2018-11-13 11:16:10 -05:00
Calvin Leung Huang e4087474b6 Let ctx handle matching mount entry (#5765) 2018-11-12 20:02:02 -08:00
Chris Hoffman 3f5117e87d fix key version tracking (#5757) 2018-11-12 09:52:31 -08:00
Chris Hoffman 3d1320d997
Fixing AliCloud KMS seal encryption/decryption (#5756)
* fixing seal encryption/decryption

* Address feedback.

Co-Authored-By: chrishoffman <christopher.hoffman@gmail.com>
2018-11-12 10:57:02 -05:00
Becca Petrin 3b8d543189
AWS auto-unseal acceptance test (#5739) 2018-11-09 14:12:29 -08:00
Jeff Mitchell c01983cffd Fix two problems with entity alias updating (#5733)
Fix two problems with entity alias updating
2018-11-08 13:04:24 -05:00
Vishal Nayak 510726494a Fix panic when storage value is nil (#5724)
* Fix panic when storage value is nil

* Ensure the value is at least of expected length

* Format correction

* Address review feedback
2018-11-07 14:10:08 -08:00
Jeff Mitchell fa26beeaed fmt 2018-11-07 16:52:01 -05:00
Jeff Mitchell a742857edb
Fix some remount logic within namespaces (#5722) 2018-11-07 14:56:24 -05:00
Brian Kassouf 422b6a2274
Break plugins back out into two path functions (#5721) 2018-11-07 09:38:48 -08:00
Jeff Mitchell 8b6b344d86
Add default-service/default-batch to token store roles (#5711) 2018-11-07 09:45:09 -05:00
Becca Petrin 7bd22e6779
Run all builtins as plugins (#5536) 2018-11-06 17:21:24 -08:00
Vishal Nayak 0b5ea9917e
Disallow writing to empty paths (#5702) 2018-11-06 14:08:55 -05:00
Jim Kalafut 5806179144
Update sys path definitions for OpenAPI (#5687) 2018-11-06 10:09:06 -08:00
Calvin Leung Huang 46f37f3363
Add HSMAutoDeprecated for ent upgrade (#5704) 2018-11-06 09:42:03 -08:00
Jim Kalafut 8ac04495d3
Framework and API changes to support OpenAPI (#5546) 2018-11-05 12:24:39 -08:00
Jeff Mitchell 41649c1511 Clean up stored barrier keys after migration to shamir (#5671) 2018-11-05 14:06:39 -05:00
Vishal Nayak 332e32294a
Remove namespace.TestContext and namespace.TestNamespace (#5682) 2018-11-05 11:11:32 -05:00
Jim Kalafut b1bc2a6b2b
Fix a few vet warnings (#5674) 2018-11-02 13:21:44 -07:00
Chris Hoffman 237fa63908
matching config name to storage backend (#5670) 2018-11-02 11:15:07 -04:00
Chris Thunes 16f52969f4 Fix memory issue caused by append of group slice to itself. (#5611)
The slice returned by `collectGroupsReverseDFS` is an updated copy of
the slice given to it when called. Appending `pGroups` to `groups`
therefore led to expontential memory usage as the slice was repeatedly
appended to itself.

Fixes #5605
2018-10-29 10:38:34 -04:00
Chris Hoffman 8c88eb3e2a
Add -dev-auto-seal option (#5629)
* adding a -dev-auto-seal option

* adding logger to TestSeal
2018-10-29 09:30:24 -04:00
Jeff Mitchell f8ec4d59b8 Remove disableIndexing 2018-10-23 16:05:45 -04:00
Jeff Mitchell 8a274fba51 Add disable indexing to core object 2018-10-23 15:04:36 -04:00
Jeff Mitchell a979f49cd7 Add disable-indexing 2018-10-23 15:03:17 -04:00
Jeff Mitchell 82992d6097
Seal migration (OSS) (#781) 2018-10-22 23:34:02 -07:00
Jeff Mitchell 89f0efb6a1 fmt 2018-10-20 21:09:51 -04:00
Jeff Mitchell 9f6dd376e2 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-19 17:47:58 -04:00
Calvin Leung Huang a08ccbffa7
[Review Only] Autoseal OSS port (#757)
* Port awskms autoseal

* Rename files

* WIP autoseal

* Fix protobuf conflict

* Expose some structs to properly allow encrypting stored keys

* Update awskms with the latest changes

* Add KeyGuard implementation to abstract encryption/decryption of keys

* Fully decouple seal.Access implementations from sealwrap structs

* Add extra line to proto files, comment update

* Update seal_access_entry.go

* govendor sync

* Add endpoint info to configureAWSKMSSeal

* Update comment

* Refactor structs

* Update make proto

* Remove remove KeyGuard, move encrypt/decrypt to autoSeal

* Add rest of seals, update VerifyRecoveryKeys, add deps

* Fix some merge conflicts via govendor updates

* Rename SealWrapEntry to EncryptedBlobInfo

* Remove barrier type upgrade check in oss

* Add key to EncryptedBlobInfo proto

* Update barrierTypeUpgradeCheck signature
2018-10-19 14:43:57 -07:00
Vishal Nayak c677cd0790
Case insensitive identity names (#5404)
* case insensitive identity names

* TestIdentityStore_GroupHierarchyCases

* address review feedback

* Use errwrap.Contains instead of errwrap.ContainsType

* Warn about duplicate names all the time to help fix them

* Address review feedback
2018-10-19 12:47:26 -07:00
Chris Hoffman 09a4c8214f
safely clean up loaded map (#5558) 2018-10-19 15:21:42 -04:00
Jeff Mitchell 841c4fcdd1 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-19 09:25:17 -04:00
Vishal Nayak 6ab030511c
Remove lookup check during alias removal (#5524)
* Possible fix for 5348

* Fix compilation
2018-10-18 07:53:12 -07:00
Vishal Nayak 5818977dca
Deprecate SHA1 in token store (#770)
* Deprecate SHA1 in token store

* Fallback to SHA1 for user selected IDs

* Fix existing tests

* Added warning

* Address some review feedback and remove root token prefix

* Tests for service token prefixing

* Salting utility tests

* Adjust OTP length for root token generation

* Fix tests

* Address review feedback
2018-10-17 13:23:04 -07:00
Jeff Mitchell 224fbd4a88 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-16 10:08:03 -04:00
Jeff Mitchell 04e3f9b0f3
Add LastWAL in leader/health output (#5523) 2018-10-16 09:38:44 -04:00
Jeff Mitchell a64fc7d7cb
Batch tokens (#755) 2018-10-15 12:56:24 -04:00
Vivek Lakshmanan 2c55777606 Fix expiration handling to not leak goroutines (#5506)
* Fix expiration handling to not leak goroutines

* Apply feedback
2018-10-12 19:02:59 -07:00
Jim Kalafut 123e34f4a7
Don't copy HA lock file during migration (#5503) 2018-10-12 09:29:15 -07:00
Calvin Leung Huang b47e648ddf
Logger cleanup (#5480) 2018-10-09 09:43:17 -07:00
vishalnayak baad5a66fd Fix TestIdentityStore_GroupHierarchyCases 2018-10-05 05:46:09 -04:00
Vishal Nayak fbec18fef0
Added test for verifying member group id deletion (#5469) 2018-10-04 10:38:41 -07:00
Sebastian Plattner 782f8dedd2 Fix remove Group Member in Identity Group not working (#5466) 2018-10-04 09:27:29 -07:00
Jeff Mitchell ec2ab502fc make fmt 2018-10-02 14:30:10 -04:00
Calvin Leung Huang 37c0b83669
Add denylist check when filtering passthrough headers (#5436)
* Add denylist check when filtering passthrough headers

* Minor comment update
2018-10-01 12:20:31 -07:00
Martin 03fb39033f Add support for token passed Authorization Bearer header (#5397)
* Support Authorization Bearer as token header

* add requestAuth test

* remove spew debug output in test

* Add Authorization in CORS Allowed headers

* use const where applicable

* use less allocations in bearer token checking

* address PR comments on tests and apply last commit

* reorder error checking in a TestHandler_requestAuth
2018-10-01 10:33:21 -07:00
Vishal Nayak 8e66e474ca Ensure old group alias is removed when a new one is written (#5350) 2018-10-01 10:06:10 -07:00
Jeff Mitchell ef144c4c25 Send initialized information via sys/seal-status (#5424) 2018-09-27 14:03:37 -07:00
Joel Thompson 73112c49fb logical/aws: Harden WAL entry creation (#5202)
* logical/aws: Harden WAL entry creation

If AWS IAM user creation failed in any way, the WAL corresponding to the
IAM user would get left around and Vault would try to roll it back.
However, because the user never existed, the rollback failed. Thus, the
WAL would essentially get "stuck" and Vault would continually attempt to
roll it back, failing every time. A similar situation could arise if the
IAM user that Vault created got deleted out of band, or if Vault deleted
it but was unable to write the lease revocation back to storage (e.g., a
storage failure).

This attempts to harden it in two ways. One is by deleting the WAL log
entry if the IAM user creation fails. However, the WAL deletion could
still fail, and this wouldn't help where the user is deleted out of
band, so second, consider the user rolled back if the user just doesn't
exist, under certain circumstances.

Fixes #5190

* Fix segfault in expiration unit tests

TestExpiration_Tidy was passing in a leaseEntry that had a nil Secret,
which then caused a segfault as the changes to revokeEntry didn't check
whether Secret was nil; this is probably unlikely to occur in real life,
but good to be extra cautious.

* Fix potential segfault

Missed the else...

* Respond to PR feedback
2018-09-27 09:54:59 -05:00
Brian Kassouf f5d0541d5d
Fix Capabilities check when in a child namespace (#5406) 2018-09-26 15:10:36 -07:00
Brian Kassouf 8f212d702d
replication: Fix DR API checks when using a token (#5398) 2018-09-25 13:27:57 -07:00
Vishal Nayak 68a496dde4
Support operating on entities and groups by their names (#5355)
* Support operating on entities and groups by their names

* address review feedback
2018-09-25 12:28:28 -07:00
Martin 79ab601cdb use constant where x-vault-token was still hardcoded (#5392) 2018-09-25 09:34:40 -07:00
Calvin Leung Huang ed1e41ba5c
Short-circuit TestBackend_PluginMainEnv on plain test run (#5393) 2018-09-25 09:22:34 -07:00
Jeff Mitchell 33065a60db Fix compilation/protobuf 2018-09-22 17:58:39 -04:00
andrejvanderzee dc6ea9ecbb Fix for using ExplicitMaxTTL in auth method plugins. (#5379)
* Fix for using ExplicitMaxTTL in auth method plugins.

* Reverted pb.go files for readability of PR.

* Fixed indenting of comment.

* Reverted unintended change by go test.
2018-09-21 14:31:29 -07:00
Jim Kalafut 343c72dbe1
Detect and bypass cycles during token revocation (#5364)
Fixes #4803
2018-09-20 14:56:38 -07:00
Calvin Leung Huang 189b893b35
Add ability to provide env vars to plugins (#5359)
* Add ability to provide env vars to plugins

* Update docs

* Update docs with examples

* Refactor TestAddTestPlugin, remove TestAddTestPluginTempDir
2018-09-20 10:50:29 -07:00
Jeff Mitchell 919b968c27
The big one (#5346) 2018-09-17 23:03:00 -04:00
Jeff Mitchell f692c1e3a9 Revert "Detect and bypass cycles during token revocation (#5335)"
This reverts commit 00314eb4d1c5609a1935f653dc6f2fc83c0bfcc0.
2018-09-17 14:10:57 -04:00
Jim Kalafut 0ae6ec52b8
Detect and bypass cycles during token revocation (#5335)
Fixes #4803
2018-09-17 08:55:12 -07:00
Becca Petrin b2ff87c9c2
Poll for new creds in the AWS auth agent (#5300) 2018-09-12 13:30:57 -07:00
vishalnayak e421972efb Remove group alias mdmdb update outside of UpsertGroupInTxn 2018-09-06 12:19:00 -04:00
Martin d51f3a45f7 Fix group alias loading when identity memdb is initialized (#5289) 2018-09-06 09:17:44 -07:00
Jeff Mitchell 95bdbbe85e
Port fix over that ensures we use the right step-down context (#5290) 2018-09-06 12:03:26 -04:00
Jeff Mitchell c28ed23972
Allow most parts of Vault's logging to have its level changed on-the-fly (#5280)
* Allow most parts of Vault's logging to have its level changed on-the-fly

* Use a const for not set
2018-09-05 15:52:54 -04:00
Jeff Mitchell c9e2cd93e8
Move logic around a bit to avoid holding locks when not necessary (#5277)
Also, ensure we are error checking the rand call
2018-09-05 11:49:32 -04:00
Chris Hoffman e2ed8d3d61
Fixing capabilities check for templated policies (#5250)
* fixing capabilities check for templated policies

* remove unnecessary change

* formatting
2018-09-04 14:18:59 -04:00
Brian Shumate 45f1ca162f Log 'marked as sealed' at INFO instead (#5260) 2018-09-04 10:53:40 -07:00
Becca Petrin 7a8c116fb1
undo make fmt (#5265) 2018-09-04 09:29:18 -07:00
Becca Petrin ed7639b0ec
run make fmt (#5261) 2018-09-04 09:12:59 -07:00
Jeff Mitchell a67869de22 Fix typo 2018-09-01 12:15:02 -04:00
Calvin Leung Huang 9988ace85e gofmt files (#5233) 2018-08-31 09:15:40 -07:00
Chris Hoffman 716fb03ab7
perform policy templating on each path (#5229) 2018-08-30 18:45:11 -04:00
Jeff Mitchell d57dfc1875 Move things back 2018-08-29 19:13:10 -04:00
Jeff Mitchell c6f7312f6c Move physical types around 2018-08-29 19:05:33 -04:00
Brian Kassouf 346d87f1f8 Pass the ctx value to make the race detector happy (#5201) 2018-08-27 18:21:54 -07:00
Jeff Mitchell 4761209331 Fix build 2018-08-27 19:59:59 -04:00
Jeff Mitchell b44b25d816
Allow fallback to non /-suffixed path for list acling (#5197)
This works around a very, very common error where people write policies
to affect listing but forget the slash at the end. If there is no exact
rule with a slash at the end when doing a list, we look to see if there
is a rule without it, and if so, use those capabilities.

Fixes #mass-user-confusion
2018-08-27 16:44:07 -07:00
Brian Kassouf c0ba9e8ff7
Fix potential deadlock (#5189) 2018-08-27 10:01:33 -07:00
Brian Kassouf c603a8b811
Add performance standby status to status output (#5192)
* Add performance standby status to status output

* Update ha.go
2018-08-27 10:01:07 -07:00
Jeff Mitchell 7a723b510e
Properly persist alias metadata (#5188)
In addition, don't lie about what's actually being stored
2018-08-26 10:26:34 -07:00
Brian Kassouf b7e33f1d2e
Port some HA changes (#5186) 2018-08-25 14:41:55 -07:00
Jeff Mitchell c4ebf3deda Fix expiration test 2018-08-24 12:47:56 -04:00
Jeff Mitchell f5024770dc Allow comment key in policies 2018-08-24 09:42:47 -04:00
Jeff Mitchell 362a92945e Don't resetnamed 2018-08-23 15:04:18 -04:00
Jeff Mitchell ba0d029247
Restricts ACL templating to paths but allows failures (#5167)
When a templating failure happens, we now simply ignore that path,
rather than fail all access to all policies
2018-08-23 12:15:02 -04:00
Jeff Mitchell 50197d5bfd
Only write valid group alias memberships into leases (#5164) 2018-08-22 21:53:04 -04:00
Jeff Mitchell 4bf0b12bfa Migrate external tests in vault folder 2018-08-22 20:50:52 -04:00
Brian Kassouf 2a89c60c7b Update ha.go 2018-08-22 20:45:31 -04:00
Vishal Nayak ee07e6ae59
Checks for assigning root policy (#5152)
* Fail if other policies present along with root

* disallow entity/group to contain root policy
2018-08-21 16:12:23 -04:00
Jeff Mitchell 3b01b29056
Pass in an ErrorLog to http.Server (#5135)
Fixes #5108
2018-08-21 11:23:18 -04:00
Jeff Mitchell 21cd0dd71a Use strings.Contains for error possibly coming from storage
They may not well errwrap

Fixes #5046
2018-08-17 16:06:47 -04:00
Jeff Mitchell 71d92ef093 ACL Templating (#4994)
* Initial work on templating

* Add check for unbalanced closing in front

* Add missing templated assignment

* Add first cut of end-to-end test on templating.

* Make template errors be 403s and finish up testing

* Review feedback
2018-08-15 11:42:56 -07:00
Vishal Nayak 303b2f97ff
don't ignore errors (#5087) 2018-08-10 22:32:10 -04:00
Vishal Nayak 68d27c7c38
Adjust ACL error checking (#5086)
* Adjust acl error checking

* address review feedback

* unlock before checks
2018-08-10 21:05:10 -04:00
Jeff Mitchell 4798af88f9
Fix some cases where unauthorized calls could return 400 (#5083) 2018-08-10 08:59:58 -05:00
Jeff Mitchell 14dccd7744 Remove unused variable 2018-08-09 16:47:58 -04:00
Jeff Mitchell 2ed2e696a7
Merge Identity Entities if two claim the same alias (#5075)
* Merge Identity Entities if two claim the same alias

Past bugs/race conditions meant two entities could be created each
claiming the same alias. There are planned longer term fixes for this
(outside of the race condition being fixed in 0.10.4) that involve
changing the data model, but this is an immediate workaround that has
the same net effect: if two entities claim the same alias, assume they
were created due to this race condition and merge them.

In this situation, also automatically merge policies so we don't lose
e.g. RGPs.
2018-08-09 15:37:36 -05:00
Martin aa8662c6c0 Add mount_path and mount_type to group lookups (#5074) 2018-08-09 13:07:24 -05:00
Jeff Mitchell 7d90e6eae9 Fix typo 2018-08-08 14:38:11 -04:00
Brian Kassouf 3330a3954a
Rollback: Only grab statelock if we are running async (#5064) 2018-08-07 17:48:43 -07:00
Jim Kalafut 4bdbc820ed
Add PluginEnv to SystemView (#5028) 2018-08-03 09:32:17 -07:00
Chris Hoffman d8b1d19ed6
Plumbing request context through to expiration manager (#5021)
* plumbing request context to expiration manager

* moar context

* address feedback

* only using active context for revoke prefix

* using active context for revoke commands

* cancel tidy on active context

* address feedback
2018-08-01 21:39:39 -04:00
Brian Kassouf bb076f161d
core: Cancel context before taking state lock (#5020)
* core: Cancel context before taking state lock

* Create active context outside of postUnseal

* Attempt to drain requests before canceling context

* fix test
2018-08-01 12:07:37 -07:00
Chris Hoffman b229df9cd2
fixing rollback test 2018-07-31 22:45:38 -04:00
Brian Kassouf 1781ade3a1
core: Read lock when doing a rollback (#5016) 2018-07-30 19:55:43 -06:00
Chris Hoffman 51f8cd9668
do not grab statelock when requested not to (#5015) 2018-07-30 14:54:54 -04:00
Jeff Mitchell 0e659ca6be Simplify sealInternal 2018-07-24 23:26:28 -04:00
Chris Hoffman 1578c5b982 Add locking when adding aliases to existing entities (#4965) 2018-07-24 22:01:58 -04:00
Jeff Mitchell 4261618d10 Add request timeouts in normal request path and to expirations (#4971)
* Add request timeouts in normal request path and to expirations

* Add ability to adjust default max request duration

* Some test fixes

* Ensure tests have defaults set for max request duration

* Add context cancel checking to inmem/file

* Fix tests

* Fix tests

* Set default max request duration to basically infinity for this release for BC

* Address feedback
2018-07-24 14:50:49 -07:00
Jeff Mitchell 9687ccc8fa Tackle #4929 a different way (#4932)
* Tackle #4929 a different way

This turns c.sealed into an atomic, which allows us to call sealInternal
without a lock. By doing so we can better control lock grabbing when a
condition causing the standby loop to get out of active happens. This
encapsulates that logic into two distinct pieces (although they could
be combined into one), and makes lock guarding more understandable.

* Re-add context canceling to the non-HA version of sealInternal

* Return explicitly after stopCh triggered
2018-07-24 13:57:25 -07:00
Jeff Mitchell d144f2935e Two-pronged fix for renew policy checking (#4960)
1) In backends, ensure they are now using TokenPolicies
2) Don't reassign auth.Policies until after expmgr registration as we
don't need them at that point

Fixes #4829
2018-07-24 12:03:11 -07:00
Jeff Mitchell caa5661031
Pass identity metadata through to plugins (#4967)
It's not obvious why this should be secret, and if it were considered
secret, when and what anything would ever be allowed to access it.
Likely the right way to tie secret values to particular
entities/aliases/groups would be to use the upcoming templated ACL
feature.
2018-07-23 12:45:06 -04:00
Jeff Mitchell 1d99b7fd05
Properly watch quit context in expireID instead of locking first (#4970) 2018-07-20 17:00:09 -04:00
Jeff Mitchell 8b0561aad8
Two small items: (#4934)
1) Disable MaxRetries in test cluster clients. We generally want to fail
as fast as possible in tests so adding unpredictable timing in doesn't
help things, especially if we're timing sensitive in the test.

2) EquivalentPolicies is supposed to return true if only one set
contains `default` and the other is empty, but if one set was nil
instead of simply a zero length slice it would always return false. This
means that renewing against, say, `userpass` when not actually
specifying any user policies would always fail.
2018-07-17 01:23:26 -04:00
Jeff Mitchell 75547fcac3 Remove defer of atomic add to see if that fixes data race 2018-07-16 11:34:47 -04:00
Brian Kassouf 57d9c335d8
Don't shutdown if we lose leadership during lease restoration (#4924)
* Don't shutdown if we lose leadership during lease restoration

* Update comment
2018-07-13 11:30:08 -07:00
Seth Vargo cdfa124b04 Don't use activeContext for standby/leader cancelation (#4919)
Move cancelation to after cleanup

Fixes GH-4915
2018-07-13 11:04:23 -04:00
Jeff Mitchell 5c2237de79 Bind locally to opcount to try to make race detector happy 2018-07-13 10:33:42 -04:00
Jeff Mitchell 954f6c4ece
Add config flag to disable non-printable character check (#4917) 2018-07-12 16:29:36 -04:00
Calvin Leung Huang f801f4b808
Add description flag to secrets and auth tune subcommands (#4894)
* Add description flag to secrets and auth tune subcommands

* Allow empty description to be provided in secret and auth mount tune

* Use flagNameDescription
2018-07-12 11:15:50 -04:00
Jeff Mitchell cd51a769ca Fix tests 2018-07-12 10:18:50 -04:00
Jeff Mitchell 98bf463a65 Make single-lease revocation behave like expiration (#4883)
This change makes it so that if a lease is revoked through user action,
we set the expiration time to now and update pending, just as we do with
tokens. This allows the normal retry logic to apply in these cases as
well, instead of just erroring out immediately. The idea being that once
you tell Vault to revoke something it should keep doing its darndest to
actually make that happen.
2018-07-11 15:45:35 -04:00
Jeff Mitchell 8a9957fa4c
Add control group request endpoint to default policy (#4904) 2018-07-11 12:46:30 -04:00
Jeff Mitchell 28890ee198 Make proto
It appears the only thing that actually change is that the tag got
`proto3` values added.
2018-07-10 20:49:48 -04:00
Jeff Mitchell 3885365757 Fix race in exp tidy test 2018-07-06 17:01:02 -04:00
Jeff Mitchell 4a3fe87a39
Allow max request size to be user-specified (#4824)
* Allow max request size to be user-specified

This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.

This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.

* Switch to ioutil.ReadAll()
2018-07-06 15:44:56 -04:00
Jeff Mitchell 8f45bc69ba Fix tuning visibility in CLI (#4827)
The API elides the value if it's empty, but empty has meaning. This adds
"hidden" as an option which is fundamentally identical to the default.
2018-07-02 12:13:25 -04:00
Brian Kassouf 3a5d5cb872
Use run groups to manage HA go routines (#4671)
* Use rungroups to spinoff standby go routines

* Update comments
2018-06-29 16:23:52 -07:00
Brian Kassouf 4ede13c91f
HA: Bump the max send/recv size for the gRPC server (#4844) 2018-06-29 09:52:23 -07:00