Commit graph

2049 commits

Author SHA1 Message Date
Jim Kalafut 2721c3a629
Namespace support for identity tokens (#7045) 2019-07-02 20:15:43 -07:00
Jeff Mitchell a3fc497fec
Fix batch token test (#7047)
At the level of role config it doesn't mean anything to use
default-service or default-batch; that's for mount tuning. So disallow
it in tokenutil. This also fixes the fact that the switch statement
wasn't right.
2019-07-02 22:16:43 -04:00
ncabatoff d2beeefe79
Add support for hashing time.Time within slices (#6767)
Add support for hashing time.Time within slices, which unbreaks auditing of requests returning the request counters.  

Break Hash into struct-specific func like HashAuth, HashRequest. Move all the copying/hashing logic from FormatRequest/FormatResponse into the new Hash* funcs.  HashStructure now modifies in place instead of copying.

Instead of returning an error when trying to hash map keys of type time.Time, ignore them, i.e. pass them through unhashed.

Enable auditing on test clusters by default if the caller didn't specify any audit backends.  If they do, they're responsible for setting it up.
2019-07-02 18:18:40 -04:00
Lexman b74591f934
adds allowed_client_ids field to identity token named keys (#6993)
* adds allowed_roles field to identity token keys and updates tests

* removed a comment that was redundant

* allowed_roles uses role client_id s instead of role names

* renamed allowed_roles to allowed_clients

* renamed allowed_clients to allowed_clientIDs

* removes some warning messages and checks on keys when creating a role

* removes name field being set unneededly
2019-07-02 14:46:22 -07:00
Jeff Mitchell d7243f910a
Re-enable toggling renewable off for tokens (#7043)
Earlier in tokenutil's dev it seemed like there was no reason to allow
auth plugins to toggle renewability off. However, it turns out Centrify
makes use of this for sensible reasons. As a result, move the forcing-on
of renewability into tokenutil, but then allow overriding after
PopulateTokenAuth is called.
2019-07-02 10:23:46 -04:00
Jeff Mitchell edf0f3cbbd Use metrics helper from base in test cluster cores 2019-07-01 12:08:55 -04:00
Michel Vocks 2b5aca4300
Token identity support (#6267)
* Implemented token backend support for identity

* Fixed tests

* Refactored a few checks for the token entity overwrite. Fixed tests.

* Moved entity alias check up so that the entity and entity alias is only created when it has been specified in allowed_entity_aliases list

* go mod vendor

* Added glob pattern

* Optimized allowed entity alias check

* Added test for asterisk only

* Changed to glob pattern anywhere

* Changed response code in case of failure. Changed globbing pattern check. Added docs.

* Added missing token role get parameter. Added more samples

* Fixed failing tests

* Corrected some cosmetical review points

* Changed response code for invalid provided entity alias

* Fixed minor things

* Fixed failing test
2019-07-01 11:39:54 +02:00
Jeff Mitchell e18866498d Fix tests 2019-06-30 21:03:36 -04:00
Jeff Mitchell 5435645bb6
Fix upgrade logic with tokenutil (#7026)
If only a non-_token field is provided we don't want to clear out the
Token version of the params, we want to set both. Otherwise we can't
rely on using the Token version of the parameter when creating the Auth
struct.
2019-06-30 14:24:41 -04:00
Vishal Nayak 6dbd8c228f
Raft tests (#7008)
* Add join test

* Add configuration test

* Add remove peer test

* Test join with and without client certs
2019-06-28 14:08:53 -04:00
Jeff Mitchell 14303c475b Check ctx again after getting state lock when forwarding 2019-06-27 20:36:16 -04:00
Jeff Mitchell fe7bb0b630
Standardize how we format deprecated values in traditional path-help (#7007) 2019-06-27 14:52:52 -04:00
Brian Kassouf 62e14c280d
storage/raft: fix races in tests (#6996)
* storage/raft: fix races in tests

* Fix another test race
2019-06-27 10:00:03 -07:00
Jim Kalafut 15b68c35c3
Support EC and EdDSA in identity tokens (#6992) 2019-06-27 08:34:48 -07:00
dr-db 720db0ffbc Fix the doc string for IsFatalError (#7000) 2019-06-27 12:50:47 +02:00
Lexman 4b01a6b95c
adds tests for the identity/oidc backend (#6964)
* adds several tests

* use deep.Equal to validate responses

* use deep.Equal in a few other places
2019-06-25 12:55:20 -07:00
Jeff Mitchell b700389cdd Fix test 2019-06-23 23:05:51 -04:00
Jeff Mitchell cf93f60bb1
Allow turning on client auth in test clusters (#6958) 2019-06-23 21:50:27 -04:00
Brian Kassouf 17b46e2979
Fix key upgrade and raft tests (#6949) 2019-06-21 11:38:21 -06:00
Jim Kalafut 2bf5db4fe8 Add OIDC token generation to Identity (#6900)
* Add OIDC token generation to Identity

There are a few open TODOs and some remaining cleanup, but this is
functionally complete and ready for review.

(Tests will being added soon.)

* Simplified key update endpoint

* Cache the config

* Fix Issuer handling

* Suppose base64-encoded templates (#6919)

* Cache JWKS and switch to go-cache (#6918)

* Address review comments

* Add warning if neither Issue nor api_addr are set

* adds tests (#6937)

* adds help synopsis and descriptions to the framework path for the oid… (#6930)

* adds help synopsis and descriptions to the framework path for the oidc backend

* Update vault/identity_store_oidc.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* Add Now parameter to PopulateStringInput

* Addressing review comments

* Refactor template processing to improve mode-specific handling

* adds a test for the periodic func (#6943)

* adds a test for the periodic func

* removes commented out code

* adds a comment

* Add comments
2019-06-21 10:23:39 -07:00
Vishal Nayak 53035ce390
Raft CLI (#6893)
* raft cli

* Reuse the command's client

* Better response handling

* minor touchups
2019-06-20 21:32:00 -04:00
Jeff Mitchell 07dcdc8b79 Sync 2019-06-20 20:55:10 -04:00
Jeff Mitchell 7966231d88
Port some stuff (#6939)
* Port some fixes

* Sync some updates
2019-06-20 16:02:11 -04:00
Brian Kassouf ed14061578
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
Clint b55303eddb
Combined Database Backend: Static Accounts (#6834)
* Add priority queue to sdk

* fix issue of storing pointers and now copy

* update to use copy structure

* Remove file, put Item struct def. into other file

* add link

* clean up docs

* refactor internal data structure to hide heap method implementations. Other cleanup after feedback

* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods

* updates after feedback

* refactoring/renaming

* guard against pushing a nil item

* minor updates after feedback

* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC

* Initial Combined database backend implementation of static accounts and automatic rotation

* vendor updates

* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation

* add lock and setup of rotation queue

* vendor the queue

* rebase on new method signature of queue

* remove mongo tests for now

* update default role sql

* gofmt after rebase

* cleanup after rebasing to remove checks for ErrNotFound error

* rebase cdcr-priority-queue

* vendor dependencies with 'go mod vendor'

* website database docs for Static Role support

* document the rotate-role API endpoint

* postgres specific static role docs

* use constants for paths

* updates from review

* remove dead code

* combine and clarify error message for older plugins

* Update builtin/logical/database/backend.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* cleanups from feedback

* code and comment cleanups

* move db.RLock higher to protect db.GenerateCredentials call

* Return output with WALID if we failed to delete the WAL

* Update builtin/logical/database/path_creds_create.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* updates after running 'make fmt'

* update after running 'make proto'

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* update comment and remove and rearrange some dead code

* Update website/source/api/secret/databases/index.html.md

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* cleanups after review

* Update sdk/database/dbplugin/grpc_transport.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* code cleanup after feedback

* remove PasswordLastSet; it's not used

* document GenerateCredentials and SetCredentials

* Update builtin/logical/database/path_rotate_credentials.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* wrap pop and popbykey in backend methods to protect against nil cred rotation queue

* use strings.HasPrefix instead of direct equality check for path

* Forgot to commit this

* updates after feedback

* re-purpose an outdated test to now check that static and dynamic roles cannot share a name

* check for unique name across dynamic and static roles

* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials

* remove commented out code

* refactor to have loadstaticwals filter out wals for roles that no longer exist

* return error if nil input given

* add nil check for input into setStaticAccount

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* add constant for queue tick time in seconds, used for comparrison in updates

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* code cleanup after review

* remove misplaced code comment

* remove commented out code

* create a queue in the Factory method, even if it's never used

* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type

* document new method

* move rotation things into a specific file

* rename test file and consolidate some static account tests

* Update builtin/logical/database/path_roles.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update builtin/logical/database/rotation.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* update code comments, method names, and move more methods into rotation.go

* update comments to be capitalized

* remove the item from the queue before we try to destroy it

* findStaticWAL returns an error

* use lowercase keys when encoding WAL entries

* small cleanups

* remove vestigial static account check

* remove redundant DeleteWAL call in populate queue

* if we error on loading role, push back to queue with 10 second backoff

* poll in initqueue to make sure the backend is setup and can write/delete data

* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false

* add code comments on read-only loop

* code comment updates

* re-push if error returned from find static wal

* add locksutil and acquire locks when pop'ing from the queue

* grab exclusive locks for updating static roles

* Add SetCredentials and GenerateCredentials stubs to mockPlugin

* add a switch in initQueue to listen for cancelation

* remove guard on zero time, it should have no affect

* create a new context in Factory to pass on and use for closing the backend queue

* restore master copy of vendor dir
2019-06-19 14:45:39 -05:00
Jeff Mitchell 3f1c510bc9
Fix a deadlock if a panic happens during request handling (#6920)
* Fix a deadlock if a panic happens during request handling

During request handling, if a panic is created, deferred functions are
run but otherwise execution stops. #5889 changed some locks to
non-defers but had the side effect of causing the read lock to not be
released if the request panicked. This fixes that and addresses a few
other potential places where things could go wrong:

1) In sealInitCommon we always now defer a function that unlocks the
read lock if it hasn't been unlocked already
2) In StepDown we defer the RUnlock but we also had two error cases that
were calling it manually. These are unlikely to be hit but if they were
I believe would cause a panic.

* Add panic recovery test
2019-06-19 09:40:57 -04:00
Becca Petrin f6703c3630 fix test 2019-06-18 14:04:08 -07:00
Jeff Mitchell 8603573611 Update group alias handling to better protect against namespace differences
Also, allow canonical ID to be switched
2019-06-18 16:43:30 -04:00
Jeff Mitchell 8ae4149703 Update description field for some token store role values to be accurate 2019-06-18 11:33:56 -04:00
Jeff Mitchell d65a716b63 Add a force capability to delete in the policy store
This allows us to truly delete policies when we've either invalidated it
(which since they're singletons/default should only happen when we're
doing a namespace delete) or are doing a namespace delete on the local
node.
2019-06-18 10:25:57 -04:00
Jeff Mitchell 1e1e7c9321 Fix some tests 2019-06-17 17:02:34 -04:00
ncabatoff 006bab1e49
Add new license callback init step for logical backends. (#6887) 2019-06-17 14:11:35 -04:00
Jeff Mitchell 1ea0c0314a
Prevent entity alias creation when entity is in different NS than mount (#943) (#6886) 2019-06-14 12:53:00 -04:00
Jeff Mitchell 402ba1b0f0
Tokenhelper v2 (#6662)
This provides an sdk util for common token fields and parsing and plumbs it into token store roles.
2019-06-14 10:17:04 -04:00
Brian Kassouf 934b497101
Clear the Barrier AEAD cache on keyring reload (#6870)
* Clear the barrier's AEAD cache on keyring reload

* Update barrier_aes_gcm_test.go
2019-06-12 08:56:16 -07:00
Calvin Leung Huang 08e17cc111
core: add generic request forwarding bits to oss (#6866) 2019-06-11 13:13:03 -07:00
ncabatoff cc41e608dc
Add core.coreNumber field, used to differentiate multiple cores/clusters when running tests (#6855)
This is not used or exposed in prod.

Remove some test-specific code from the cluster-building helpers. The corresponding additions go on the ent side.
2019-06-10 14:07:16 -04:00
Jeff Mitchell c02abb969c
Attempt to grab read statelock in emit metrics to remove some raciness (#6829) 2019-06-05 12:26:29 -04:00
Michael Gaffney 055c5ff311
Fix data race in kvv2_upgrade_test (#6825) 2019-06-04 18:48:31 -04:00
Jeff Mitchell 5f7321dcc7 Fix a case where mounts could be duplicated (#6771)
When unmounting, the router entry would be tainted, preventing routing.
However, we would then unmount the router before clearing storage, so if
an error occurred the router would have forgotten the path. For auth
mounts this isn't a problem since they had a secondary check, but
regular mounts didn't (not sure why, but this is true back to at least
0.2.0). This meant you could then create a duplicate mount using the
same path which would then not conflict in the router until postUnseal.

This adds the extra check to regular mounts, and also moves the location
of the router unmount.

This also ensures that on the next router.Mount, tainted is set to the
mount entry's tainted status.

Fixes #6769
2019-06-04 10:33:36 -07:00
tonyd 0570966cb9 Allow logical backends access to the disabled state of an entity (#6791)
* Allow logical backends access to the disabled state of an entity via SystemView.EntityInfo().

* Add generated file in vendor directory.
2019-05-28 16:31:50 -05:00
ncabatoff ad28263b69
Allow plugins to submit audit requests/responses via extended SystemView (#6777)
Move audit.LogInput to sdk/logical.  Allow the Data values in audited
logical.Request and Response to implement OptMarshaler, in which case
we delegate hashing/serializing responsibility to them.  Add new
ClientCertificateSerialNumber audit request field.

SystemView can now be cast to ExtendedSystemView to expose the Auditor
interface, which allows submitting requests and responses to the audit
broker.
2019-05-22 18:52:53 -04:00
Jeff Mitchell ec3ea45858
Update grpc and protos (#6725)
gRPC updated to 1.20.1 which fixes a couple of important bugs.

Updates protos as well.
2019-05-13 12:09:30 -04:00
Vishal Nayak 550f2a52bd
Storage packer V1 updates (#6531)
* spv1 updates

* fix tests
2019-05-07 15:29:51 -04:00
Jeff Mitchell e8a9d47aca
Port over some SP v2 bits (#6516)
* Port over some SP v2 bits

Specifically:

* Add too-large handling to Physical (Consul only for now)
* Contextify some identity funcs
* Update SP protos

* Add size limiting to inmem storage
2019-05-01 13:47:41 -04:00
EdwinRobbins de885660e8 DynamoDB: Make Unlock key delete conditional on being old leader's (#6637) 2019-04-30 17:44:47 -07:00
Jim Kalafut eb8ea30ca6
List additional sys paths as unauthenticated (#6654)
These paths are handled directly in handler.go, but the list of special
paths here impacts the x-vault-unauthenticated field in generated
OpenAPI.

Fixes: #6651
2019-04-29 16:42:23 -07:00
Calvin Leung Huang 262608c2c0
core: honor non-HMAC keys in audit requests (#6653) 2019-04-29 13:14:26 -07:00
Mark Gritter ed722d8013
Merge branch 'master' into fix-6579 2019-04-29 11:52:24 -05:00
mgritter be2f69bc4a Check nil parameter value when processing an ACL. 2019-04-26 15:57:00 -07:00
Michel Vocks 23d50af9bb Fixed wrong imports in test after refactoring (#6639) 2019-04-25 09:08:03 -07:00
Brian Kassouf 04d0ddfdae
Add mount UUID to the secret and auth list API responses (#6633) 2019-04-24 12:27:43 -07:00
Chris Hoffman 25cd6da6a4
refactoring to unit test transit seal (#6605) 2019-04-23 15:13:56 -04:00
Jeff Mitchell a4285a487d Sync plugin reload file 2019-04-23 10:22:56 -04:00
ncabatoff 06574da57a
Merge multiple functions for creating consul containers into one. (#6612)
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
2019-04-22 12:26:10 -04:00
Jeff Mitchell 9a7eb54a68 Merge branch '1.1.2' into master-oss 2019-04-18 18:49:49 -04:00
Brian Kassouf 8afbb0ba46
Add a get handler function (#6603) 2019-04-17 14:21:56 -07:00
Brian Kassouf 43783a5dca
Move cluster logic out of vault package (#6601)
* Move cluster logic out of vault package

* Dedup heartbeat and fix tests

* Fix test
2019-04-17 13:50:31 -07:00
Jeff Mitchell c22075fa81 Add unit test for KV issue 31 (#6591) 2019-04-16 09:53:24 -07:00
Jeff Mitchell 213b9fd1cf Update to api 1.0.1 and sdk 0.1.8 2019-04-15 14:10:07 -04:00
Jeff Mitchell 9ebc57581d
Switch to go modules (#6585)
* Switch to go modules

* Make fmt
2019-04-13 03:44:06 -04:00
Jeff Mitchell 1b5155080b Update protobufs, sdk changes 2019-04-12 23:13:14 -04:00
Becca Petrin 6ded269700
Merge pull request #6268 from hashicorp/6234-aws-region
Add region to CLI for generating AWS login data
2019-04-12 16:15:38 -07:00
Jeff Mitchell 80c303ac83 Move ldaputil and tlsutil over to sdk 2019-04-12 18:26:54 -04:00
Jeff Mitchell 371db36ede Move useragent to sdk 2019-04-12 18:17:49 -04:00
Jeff Mitchell 8d6ce1ffb5 Move policyutil to sdk 2019-04-12 18:08:46 -04:00
Jeff Mitchell 8bcb533a1b
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
Becca Petrin f20772310b Merge branch 'opensource-master' into 6234-aws-region 2019-04-12 11:13:17 -07:00
Jeff Mitchell 3ec1e79b6a Fix test code breakage 2019-04-12 11:37:34 -04:00
ncabatoff 521e93c32b Save metrics to barrier without going through c.systemBarrierView (#884)
* Don't bother trying to save metrics when we don't have a barrier.  Use stateLock.

* Use c.barrier instead of c.systemBarrierView, thus we don't need locking
and don't need to worry about race with mount setup.

* Remove unneccessary lock.
2019-04-11 12:37:38 -04:00
Jeff Mitchell 9f148c499e Fix test break 2019-04-11 11:18:32 -04:00
Jeff Mitchell 4aa4a0665f Sync over 2019-04-11 11:12:37 -04:00
Jeff Mitchell aa6fafced9 Fix hasMountPath for segment wildcard mounts; introduce priority order (#6532)
* Add prioritization when multiple segment/glob rules can match.

* Disallow ambiguous "+*" in policy paths.
2019-04-10 17:46:17 -04:00
Vishal Nayak 0df2d14961
Properly invalidate group alias in a secondary (#6564) 2019-04-10 16:14:31 -04:00
Vishal Nayak 7ab753b569
Use NS context when loading entities and groups (#6563)
* Use NS context when loading entities and groups

* update context while group loading as well

* Address review feedback
2019-04-10 15:05:57 -04:00
Brian Kassouf f5be7c8cea
Fix perf standby elections when the new active node was also the previous active node (#6561) 2019-04-10 10:09:36 -07:00
Brian Kassouf 76969aa801
Fix deadlock on the mounts & auth locks (#6560)
* Fix deadlock on the mountsLock

* Fix deadlock on authLock too
2019-04-09 19:20:06 -07:00
Jeff Mitchell 12285f5ed0 Sync over 2019-04-08 14:57:14 -04:00
Jeff Mitchell 30209cdd35 Update token path with / at end to ensure it's the token store 2019-04-08 14:25:29 -04:00
Jeff Mitchell 1d982c234f
Fix some more test failures and recursive locking (#6549) 2019-04-08 13:40:54 -04:00
Jeff Mitchell 9f0a6edfcb
Remove some instances of potential recursive locking (#6548) 2019-04-08 12:45:28 -04:00
Vishal Nayak f4876b744c Fix handling member group IDs (#6527)
* Process member_group_ids only if supplied
2019-04-05 09:12:39 -04:00
Jeff Mitchell eec3f9afb2
Handle JWT checks with namespaced service tokens (#6536)
Some checks would fail because we considered a token with two dots a
JWT, but service tokens in namespaces also fit this bill.
2019-04-04 17:13:09 -04:00
ncabatoff b08963779c Apply the bits from ent pr 852 that are relevant to OSS. (#6528) 2019-04-04 13:02:44 -04:00
Calvin Leung Huang ee46fd4362
core: port over CG and perf standby handling bits (#6530) 2019-04-03 14:16:49 -07:00
Becca Petrin d6f489cca7 strip import whitespace 2019-04-03 13:43:54 -07:00
Becca Petrin ed03e8f04d add comment on region parsing 2019-04-03 11:58:20 -07:00
Becca Petrin 339cfcaaf8 merge master 2019-04-01 13:52:44 -07:00
Jim Kalafut 265e61b993
Add OpenAPI support for query parameters (#6490)
Also, fix handling of required properties in request body.
2019-03-28 14:40:56 -07:00
Brian Kassouf f50ff52a8b
Add metrics to seal methods (#6478) 2019-03-26 10:32:19 -07:00
Jeff Mitchell 775f547e4e Sync over 2019-03-25 14:18:43 -04:00
Jeff Mitchell 6797e21f54
Migrate from SermoDigital go Square JOSE (#6445) 2019-03-20 14:54:03 -04:00
T.K 453f1ac109 changed misspelled english words (#6432) 2019-03-19 09:32:45 -04:00
Brian Kassouf 2af9d9de98
Fix a locking issue in the Rollback manager (#6426)
* Fix a locking issue in the Rollback manager

* Update rollback.go

* Update rollback.go

* move state creation

* Update vault/rollback.go

Co-Authored-By: briankassouf <briankassouf@users.noreply.github.com>

* Simplify logic by canceling the lock grab

* Use context instead of a chan

* Update vault/rollback.go
2019-03-18 11:12:48 -07:00
Jeff Mitchell 9193792773 Sync over 2019-03-18 09:33:01 -04:00
Calvin Leung Huang 7a80fd0a81 add short sleep between revocation and lookup (#6425) 2019-03-15 18:24:53 -04:00
Michel Vocks f4b0bd484a
Added upstream changes from enterprise to OSS (#6419) 2019-03-15 14:25:05 +01:00
Calvin Leung Huang a71adbc2ce sys: do not show filtered mounts on internal/ui/mounts (#6412) 2019-03-14 11:43:44 -07:00
Calvin Leung Huang 70108668a9
testing: allow logger and audit backends to be set in TestCoreWithSealAndUI (#6388) 2019-03-08 13:46:24 -08:00
Chris Hoffman d9c54877ba Fixing transit seal tests (#6342)
* fixing dockertest to run on travis

* try a repo local directory

* precreate the directory

* strip extraneous comment

* check directory was created

* try to print container logs

* try writing out client logs

* one last try

* Attempt to fix test

* convert to insecure tls

* strip test-temp
2019-03-07 11:23:30 -08:00
Jeff Mitchell c4001cc40d This moves api_test package tests into vault/external_tests
This prevents projects importing Vault's API from then also importing
the rest of Vault.
2019-03-06 11:20:42 -05:00
ncabatoff cd747c9318
Add code for writing and reading request counters to storage. (#5918)
Increment a counter whenever a request is received. 
The in-memory counter is persisted to counters/requests/YYYY/MM.
When the month wraps around, we reset the in-memory counter to
zero.
Add an endpoint for querying the request counters across all time.
2019-03-05 14:55:07 -05:00
Becca Petrin 1909b20217 merge master 2019-03-05 09:39:53 -08:00
Jim Kalafut 69dbd2464d Fix failing TokenStore test 2019-03-04 17:44:00 -08:00
Becca Petrin a8ebfef8ef
fix typo 2019-03-04 16:42:48 -08:00
Calvin Leung Huang 1ae110e59a
core: handleLoginRequest should set resp.Auth.Orphan = true (#6335)
* core: handleLoginRequest should set resp.Auth.Orphan = true

* set auth.Orphan in RegisterAuth dynamically

* update related tests
2019-03-04 14:32:09 -08:00
Jeff Mitchell a83ed04730 Add ability to migrate autoseal to autoseal (#5930)
* Add ability to migrate autoseal to autoseal

This adds the ability to migrate from shamir to autoseal, autoseal to
shamir, or autoseal to autoseal, by allowing multiple seal stanzas. A
disabled stanza will be used as the config being migrated from; this can
also be used to provide an unwrap seal on ent over multiple unseals.

A new test is added to ensure that autoseal to autoseal works as
expected.

* Fix test

* Provide default shamir info if not given in config

* Linting feedback

* Remove context var that isn't used

* Don't run auto unseal watcher when in migration, and move SetCores to SetSealsForMigration func

* Slight logic cleanup

* Fix test build and fix bug

* Updates

* remove GetRecoveryKey function
2019-03-04 14:11:56 -08:00
Brian Kassouf ad3605e657
Revert "filtered-path endpoint (#6132)" (#6337)
This reverts commit dfdbb0bad975fab447f49766baaa5a6c956f8e3d.
2019-03-04 14:08:21 -08:00
ncabatoff 8814fe1ba5 filtered-path endpoint (#6132)
* First pass at filtered-path endpoint.  It seems to be working, but there are tests missing, and possibly some optimization to handle large key sets.

* Vendor go-cmp.

* Fix incomplete vendoring of go-cmp.

* Improve test coverage.  Fix bug whereby access to a subtree named X would expose existence of a the key named X at the same level.

* Add benchmarks, which showed that hasNonDenyCapability would be "expensive" to call for every member of a large folder.  Made a couple of minor tweaks so that now it can be done without allocations.

* Comment cleanup.

* Review requested changes: rename some funcs, use routeCommon instead of
querying storage directly.

* Keep the same endpoint for now, but move it from a LIST to a POST and allow multiple paths to be queried in one operation.

* Modify test to pass multiple paths in at once.

* Add endpoint to default policy.

* Move endpoint to /sys/access/filtered-path.
2019-03-04 11:04:29 -08:00
Jeff Mitchell a3a2a3cd04 A few more syncs 2019-03-04 13:53:15 -05:00
Michel Vocks ce832e402a Fixed ignored empty value set on token role update call (#6314)
* Fixed ignored empty value set on token role update call

* Made a pre-check a bit more elegant. Updated tests
2019-03-04 09:39:29 -08:00
Vishal Nayak d514ff573a
Set orphan status in the token creation response (#6320) 2019-03-01 18:55:58 -05:00
Brian Kassouf 7b910a093b
Handle ns lease and token renew/revoke via relative paths (#6236) (#6312)
* Handle ns lease and token renew/revoke via relative paths

* s/usin/using/

* add token and lease lookup paths; set ctx only on non-nil ns

Addtionally, use client token's ns for auth/token/lookup if no token is provided
2019-02-28 16:02:25 -08:00
Chris Hoffman 4f35c548fe Transit Autounseal (#5995)
* Adding Transit Autoseal

* adding tests

* adding more tests

* updating seal info

* send a value to test and set current key id

* updating message

* cleanup

* Adding tls config, addressing some feedback

* adding tls testing

* renaming config fields for tls
2019-02-28 13:13:56 -08:00
Becca Petrin 5829774e91
Support env vars for STS region (#6284) 2019-02-28 09:31:06 -08:00
vishalnayak 3f92836771 Port identity store cleanup code 2019-02-26 16:11:16 -05:00
Brian Kassouf efe5671f36 make fmt 2019-02-20 12:12:21 -08:00
Brian Kassouf 26d8d318d7 Merge remote-tracking branch 'oss/master' into 1.1-beta 2019-02-19 12:17:15 -08:00
Brian Kassouf ab1a04f8e6
Port over some test fixes (#6261) 2019-02-19 12:03:02 -08:00
Martin 4c9e7f4478 typo in pathsToRadix doc (#6253) 2019-02-18 10:05:04 -08:00
Lexman 1ed2f2210b use deep.Equal instead of reflect.DeepEqual in some failing tests (#6249)
* use deep.Equal instead of reflect.DeepEqual in some failing tests

* changed test output a little bit
2019-02-15 14:00:14 -08:00
ncabatoff 4cf1e24a9e Fix TestSystemBackend_mount, TestSystemBackend_mounts. (#6247) 2019-02-15 11:14:45 -08:00
Calvin Leung Huang ac0b31b544 fix TestCore_Unmount 2019-02-15 11:13:26 -08:00
Calvin Leung Huang 35ecaa2b6b fix TestSystemBackend_InternalUIMounts 2019-02-15 10:46:38 -08:00
Jeff Mitchell 3dfa30acb4 Add ability to use path wildcard segments (#6164)
* Path globbing

* Add glob support at the beginning

* Ensure when evaluating an ACL that our path never has a leading slash. This already happens in the normal request path but not in tests; putting it here provides it for tests and extra safety in case the request path changes

* Simplify the algorithm, we don't really need to validate the prefix first as glob won't apply if it doesn't

* Add path segment wildcarding

* Disable path globbing for now

* Remove now-unneeded test

* Remove commented out globbing bits

* Remove more holdover glob bits

* Rename k var to something more clear
2019-02-14 18:31:43 -08:00
Brian Kassouf f5b5fbb392
Refactor the cluster listener (#6232)
* Port over OSS cluster port refactor components

* Start forwarding

* Cleanup a bit

* Fix copy error

* Return error from perf standby creation

* Add some more comments

* Fix copy/paste error
2019-02-14 18:14:56 -08:00
Martin c9918b93e8 Avoid panic at login when external group has a nil alias (#6230) 2019-02-14 12:57:20 -08:00
Martin 9044173d6e Prometheus support on v1/sys/metrics endpoint (#5308)
* initial commit for prometheus and sys/metrics support

* Throw an error if prometheusRetentionTime is 0,add prometheus in devmode

* return when format=prometheus is used and prom is disable

* parse prometheus_retention_time from string instead of int

* Initialize config.Telemetry if nil

* address PR issues

* add sys/metrics framework.Path in a factory

* Apply requiredMountTable entries's MountConfig to existing core table

* address pr comments

* enable prometheus sink by default

* Move Metric-related code in a separate metricsutil helper
2019-02-14 12:46:59 -08:00
Jeff Mitchell e5ca13d0be
Don't add kv by default in non-dev scenarios (#6109)
It's retained for tests though since most rely on it.
2019-02-14 11:55:32 -08:00
Jeff Mitchell 5fa9e48b21
Don't duplicate CORS headers (#6207)
Fixes #6182
2019-02-11 13:10:26 -05:00
Jeff Mitchell ebf57b15b4
Fixes a regression in forwarding from #6115 (#6191)
* Fixes a regression in forwarding from #6115

Although removing the authentication header is good defense in depth,
for forwarding mechanisms that use the raw request, we never add it
back. This caused perf standby tests to throw errors. Instead, once
we're past the point at which we would do any raw forwarding, but before
routing the request, remove the header.

To speed this up, a flag is set in the logical.Request to indicate where
the token is sourced from. That way we don't iterate through maps
unnecessarily.
2019-02-11 13:08:15 -05:00
Vishal Nayak b4ba344782
Merge entities during unseal only on the primary (#6075)
* Merge entities during unseal only on the primary

* Add another guard check

* Add perf standby to the check

* Make primary to not differ from case-insensitivity status w.r.t secondaries

* Ensure mutual exclusivity between loading and invalidations

* Both primary and secondaries won't persist during startup and invalidations

* Allow primary to persist when loading case sensitively

* Using core.perfStandby

* Add a tweak in core for testing

* Address review feedback

* update memdb but not storage in secondaries

* Wire all the things directly do mergeEntity

* Fix persist behavior

* Address review feedback
2019-02-08 16:32:06 -05:00
Jeff Mitchell 9ef0680e7f
Fix leader info repopulation (#6167)
* Two things:

* Change how we populate and clear leader UUID. This fixes a case where
if a standby disconnects from an active node and reconnects, without the
active node restarting, the UUID doesn't change so triggers on a new
active node don't get run.

* Add a bunch of test helpers and minor updates to things.
2019-02-05 21:01:18 -05:00
Jeff Mitchell 5f249d4005
Add allowed_response_headers (#6115) 2019-02-05 16:02:15 -05:00
Brian Kassouf aaca35be94
Updates to recovery keys (#6152) 2019-02-01 11:29:55 -08:00
Jeff Mitchell bbc1d53a5d Revert "Refactor common token fields and operations into a helper (#5953)"
This reverts commit 66c226c593bb1cd48cfd8364ac8510cb42b7d67a.
2019-02-01 11:23:40 -05:00
Jeff Mitchell 27c960d8df
Split SubView functionality into logical.StorageView (#6141)
This lets other parts of Vault that can't depend on the vault package
take advantage of the subview functionality.

This also allows getting rid of BarrierStorage and vault.Entry, two
totally redundant abstractions.
2019-01-31 09:25:18 -05:00
Jeff Mitchell 85a560abba
Refactor common token fields and operations into a helper (#5953) 2019-01-30 16:23:28 -05:00
Matthew Irish b777906fee
add entity lookup to the default policy (#6105)
* add entity lookup to the default policy

* only use id for lookup

* back in with name
2019-01-29 09:43:59 -06:00
Jeff Mitchell e781ea3ac4
First part of perf standby entity race fix (#6106) 2019-01-25 14:08:42 -05:00
Jeff Mitchell 1f57e3674a Move a common block up a level 2019-01-24 18:29:22 -05:00
Jeff Mitchell 6d22f3fc2e minor linting change 2019-01-23 17:19:06 -05:00
Seth Vargo 98ad431d6d Continuously attempt to unseal if sealed keys are supported (#6039)
* Add helper for checking if an error is a fatal error

The double-double negative was really confusing, and this pattern is used a few places in Vault. This negates the double negative, making the devx a bit easier to follow.

* Check return value of UnsealWithStoredKeys in sys/init

* Return proper error types when attempting unseal with stored key

Prior to this commit, "nil" could have meant unsupported auto-unseal, a transient error, or success. This updates the function to return the correct error type, signaling to the caller whether they should retry or fail.

* Continuously attempt to unseal if sealed keys are supported

This fixes a bug that occurs on bootstrapping an initial cluster. Given a collection of Vault nodes and an initialized storage backend, they will all go into standby waiting for initialization. After one node is initialized, the other nodes had no mechanism by which they "re-check" to see if unseal keys are present. This adds a goroutine to the server command which continually waits for unseal keys to exist. It exits in the following conditions:

- the node is unsealed
- the node does not support stored keys
- a fatal error occurs (as defined by Vault)
- the server is shutting down

In all other situations, the routine wakes up at the specified interval and attempts to unseal with the stored keys.
2019-01-23 16:34:34 -05:00
Jeff Mitchell c5d8391c38
Prefix path rename (#6089)
* Rename Prefix -> Path in internal struct

* Update test
2019-01-23 15:04:49 -05:00
Jeff Mitchell a11f2a3ba2
Rename glob -> prefix in ACL internals (#6086)
Really, it's a prefix
2019-01-23 13:55:40 -05:00
Jeff Mitchell 59bc9dd361 Add missing value to policy ShallowClone
Not related to a bug, just happened to notice it.
2019-01-23 13:20:04 -05:00
Jim Kalafut f097b8d934
Update existing alias metadata during authentication (#6068) 2019-01-23 08:26:50 -08:00
Stuart Moore 1e836c1f67 Typo fix in gcpckms.go (#6081) 2019-01-23 07:52:31 -05:00
Jeff Mitchell c7ac2e449a Sync up code 2019-01-22 17:44:13 -05:00
Jeff Mitchell 797c622567 Sync seal testing 2019-01-22 17:23:20 -05:00
Jeff Mitchell 9653f9e379 Sync logical_system 2019-01-22 17:21:53 -05:00
Becca Petrin 83e0c5e5e6
Check ec2 instance metadata for region (#6025) 2019-01-18 14:49:24 -08:00
Seth Vargo 018ec9cdb8 Upgrade to new Cloud KMS client libraries (#6051)
* Upgrade to new Cloud KMS client libraries

We recently released the new Cloud KMS client libraries which use GRPC
instead of HTTP. They are faster and look nicer (</opinion>), but more
importantly they drastically simplify a lot of the logic around client
creation, encryption, and decryption. In particular, we can drop all the
logic around looking up credentials and base64-encoding/decoding.

Tested on a brand new cluster (no pre-existing unseal keys) and against
a cluster with stored keys from a previous version of Vault to ensure no
regressions.

* Use the default scopes the client requests

The client already does the right thing here, so we don't need to
surface it, especially since we aren't allowing users to configure it.
2019-01-17 17:27:15 -05:00
Jeff Mitchell 440ef3b42e Fix bound cidrs propagation 2019-01-15 10:55:36 -05:00
Becca Petrin 015f641ada Return errs from parsing unknown plugins (#6038) 2019-01-15 10:51:55 -05:00
Jeff Mitchell 78b4ff570f Expose error so warning about not decoding accessors is more useful (#6034) 2019-01-14 09:55:49 -08:00
ncabatoff e78ca24d17 Instead of using the (nil) err, use the non-nil newCtErr for ctErr. (#5841) 2019-01-09 17:39:58 -08:00
Vishal Nayak 1119f47e13 Cubbyhole cleanup (#6006)
* fix cubbyhole deletion

* Fix error handling

* Move the cubbyhole tidy logic to token store and track the revocation count

* Move fetching of cubby keys before the tidy loop

* Fix context getting cancelled

* Test the cubbyhole cleanup logic

* Add progress counter for cubbyhole cleanup

* Minor polish

* Use map instead of slice for faster computation

* Add test for cubbyhole deletion

* Add a log statement for deletion

* Add SHA1 hashed tokens into the mix
2019-01-09 10:53:41 -08:00
Giacomo Tirabassi 0d3845c537 Influxdb secret engine built-in plugin (#5924)
* intial work for influxdb secret plugin

* fixed typo

* added comment

* added documentation

* added tests

* fixed tests

* added vendoring

* minor testing issue with hardcoded values

* minor fixes
2019-01-08 17:26:16 -08:00
Jim Kalafut d0e2badbae Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00
Seth Vargo c3f1043c24 Reduce required permissions for the GCPCKMS auto-unsealer (#5999)
This changes the behavior of the GCPCKMS auto-unsealer setup to attempt
encryption instead of a key lookup. Key lookups are a different API
method not covered by roles/cloudkms.cryptoKeyEncrypterDecrypter. This
means users must grant an extended scope to their service account
(granting the ability to read key data) which only seems to be used to
validate the existence of the key.

Worse, the only roles that include this permission are overly verbose
(e.g. roles/viewer which gives readonly access to everything in the
project and roles/cloudkms.admin which gives full control over all key
operations). This leaves the user stuck between choosing to create a
custom IAM role (which isn't fun) or grant overly broad permissions.

By changing to an encrypt call, we get better verification of the unseal
permissions and users can reduce scope to a single role.
2019-01-04 16:29:31 -05:00
Jeff Mitchell c8a029210b
Don't read AWS env vars (#5974)
* Don't read AWS env vars

Let AWS SDK env cred chain provider do it for us

Fixes #5965
2019-01-04 15:03:57 -05:00
Jim Kalafut 2547d7fb6a
Simplify base62.Random (#5982)
Also move existing base62 encode/decode operations to their only points
of use.
2018-12-20 07:40:01 -08:00
Brian Kassouf 75e25711a0
Default seal type to Shamir on older seal configs (#5956) 2018-12-13 16:44:56 -08:00
Jim Kalafut 5687892530
Add operationId field to OpenAPI output (#5876)
Fixes #5842
2018-12-12 13:59:23 -08:00
Brian Kassouf 737b7e6651
Fix plugin reload when in a namespace (#5937) 2018-12-11 17:21:23 -08:00
Jeff Mitchell 394ce75e33
Move some handle request functions up a level (#5889)
* Move some handle request functions up a level

Add clearing of token entry from request, fixing a test

* Update request_handling.go
2018-12-03 14:35:20 -05:00
Calvin Leung Huang e71017e5a9 Set request token entry within fetchACLTokenEntryAndEntity (#5880) 2018-12-03 11:57:53 -05:00
Calvin Leung Huang c2e87c20d8 Remove pt value from error output 2018-12-03 09:19:16 -05:00
Calvin Leung Huang be05907515 autoseal: fix error typos (#5877) 2018-11-29 16:07:08 -08:00
Calvin Leung Huang a510537778 Reset rekey progress once threshold has been met (#5743)
* Reset rekey progress once threshold has been met

* Reverting log message changes

* Add progress check on invalid rekey test

* Minor comment update
2018-11-19 17:03:07 -08:00
Calvin Leung Huang 907dd834ff Revert deprecated plugin var names (#5822)
* Revert field back to ListPluginsResponse.Names

* Revert field back to MountConfig.PluginName and APIMountConfig.PluginName
2018-11-19 15:23:48 -08:00
Calvin Leung Huang ad3e105012
Mount tune options (#5809)
* Refactor mount tune to support upsert options values and unset options.

* Do not allow unsetting options map

* add secret tune version regression test

* Only accept valid options version

* s/meVersion/optVersion/
2018-11-19 14:23:25 -08:00
Jeff Mitchell 127413461b
Remove token store paths with token/accessors in URLs (#5773) 2018-11-19 16:58:19 -05:00
Brian Kassouf 33776b89c2
Wrap storage calls with encoding checks (#5819)
* Add encoding backend

* More work on encoding checks

* Update error message

* Update physical/encoding.go

* Disable key checks if configured
2018-11-19 13:13:16 -08:00
Brian Kassouf c16f7485e7
perf-standby: Fix audit table upgrade on standbys (#5811) 2018-11-19 10:21:53 -08:00
Konstantinos Tsanaktsidis f75e3603ba Paper over GCS backend corruption issues (#5804)
We're having issues with leases in the GCS backend storage being
corrupted and failing MAC checking. When that happens, we need to know
the lease ID so we can address the corruption by hand and take
appropriate action.

This will hopefully prevent any instances of incomplete data being sent
to GSS
2018-11-16 08:07:06 -05:00
Vishal Nayak 43e3ff808a
Update group memberships when entity is deleted (#5786)
* Use common abstraction for entity deletion

* Update group memberships before deleting entity

* Added test

* Fix return statements

* Update comment

* Cleanup member entity IDs while loading groups

* Added test to ensure that upgrade happens properly

* Ensure that the group gets persisted if upgrade code modifies it
2018-11-15 20:07:45 -05:00
Calvin Leung Huang 227a664b06 Continue on plugin registration error in dev mode (#5791)
* Continue on plugin registration error in dev mode

* Continue only on unknown type error

* Continue only on unknown type error

* Print plugin registration error on exit

Co-Authored-By: calvn <cleung2010@gmail.com>
2018-11-15 16:55:24 -08:00
Calvin Leung Huang e99957aed9
Support registering plugin with name only (#5787)
* Support registering plugin with name only

* Make RegisterPlugin backwards compatible

* Add CLI backwards compat command to plugin info and deregister

* Add server-side deprecation warnings if old read/dereg API endpoints are called

* Address feedback
2018-11-15 14:33:11 -08:00
Vishal Nayak 56d6d929ce Fix sys/auth/path/tune to accept token_type (#5777) 2018-11-14 11:22:08 -08:00
vishalnayak c6faa3ee28 Add a comment to retain misspelling 2018-11-13 13:30:42 -05:00
vishalnayak a96641c86f Fix TestPolicy_ParseBadPath 2018-11-13 13:22:56 -05:00
Jeff Mitchell 9735bd7d69 Fix more awskms test brokenness 2018-11-13 13:01:40 -05:00
Jeff Mitchell e5aad14d79 Fix test where AWS wasn't being skipped properly 2018-11-13 12:45:30 -05:00
Vishal Nayak b4836575fb
Test for issue 5729 (#5750)
* Test for 5729

* Remove unneeded space

Co-Authored-By: vishalnayak <vishalnayak@users.noreply.github.com>
2018-11-13 11:16:10 -05:00
Calvin Leung Huang e4087474b6 Let ctx handle matching mount entry (#5765) 2018-11-12 20:02:02 -08:00
Chris Hoffman 3f5117e87d fix key version tracking (#5757) 2018-11-12 09:52:31 -08:00
Chris Hoffman 3d1320d997
Fixing AliCloud KMS seal encryption/decryption (#5756)
* fixing seal encryption/decryption

* Address feedback.

Co-Authored-By: chrishoffman <christopher.hoffman@gmail.com>
2018-11-12 10:57:02 -05:00
Becca Petrin 3b8d543189
AWS auto-unseal acceptance test (#5739) 2018-11-09 14:12:29 -08:00
Jeff Mitchell c01983cffd Fix two problems with entity alias updating (#5733)
Fix two problems with entity alias updating
2018-11-08 13:04:24 -05:00
Vishal Nayak 510726494a Fix panic when storage value is nil (#5724)
* Fix panic when storage value is nil

* Ensure the value is at least of expected length

* Format correction

* Address review feedback
2018-11-07 14:10:08 -08:00
Jeff Mitchell fa26beeaed fmt 2018-11-07 16:52:01 -05:00
Jeff Mitchell a742857edb
Fix some remount logic within namespaces (#5722) 2018-11-07 14:56:24 -05:00
Brian Kassouf 422b6a2274
Break plugins back out into two path functions (#5721) 2018-11-07 09:38:48 -08:00
Jeff Mitchell 8b6b344d86
Add default-service/default-batch to token store roles (#5711) 2018-11-07 09:45:09 -05:00
Becca Petrin 7bd22e6779
Run all builtins as plugins (#5536) 2018-11-06 17:21:24 -08:00
Vishal Nayak 0b5ea9917e
Disallow writing to empty paths (#5702) 2018-11-06 14:08:55 -05:00
Jim Kalafut 5806179144
Update sys path definitions for OpenAPI (#5687) 2018-11-06 10:09:06 -08:00
Calvin Leung Huang 46f37f3363
Add HSMAutoDeprecated for ent upgrade (#5704) 2018-11-06 09:42:03 -08:00
Jim Kalafut 8ac04495d3
Framework and API changes to support OpenAPI (#5546) 2018-11-05 12:24:39 -08:00
Jeff Mitchell 41649c1511 Clean up stored barrier keys after migration to shamir (#5671) 2018-11-05 14:06:39 -05:00
Vishal Nayak 332e32294a
Remove namespace.TestContext and namespace.TestNamespace (#5682) 2018-11-05 11:11:32 -05:00
Jim Kalafut b1bc2a6b2b
Fix a few vet warnings (#5674) 2018-11-02 13:21:44 -07:00
Chris Hoffman 237fa63908
matching config name to storage backend (#5670) 2018-11-02 11:15:07 -04:00
Chris Thunes 16f52969f4 Fix memory issue caused by append of group slice to itself. (#5611)
The slice returned by `collectGroupsReverseDFS` is an updated copy of
the slice given to it when called. Appending `pGroups` to `groups`
therefore led to expontential memory usage as the slice was repeatedly
appended to itself.

Fixes #5605
2018-10-29 10:38:34 -04:00
Chris Hoffman 8c88eb3e2a
Add -dev-auto-seal option (#5629)
* adding a -dev-auto-seal option

* adding logger to TestSeal
2018-10-29 09:30:24 -04:00
Jeff Mitchell f8ec4d59b8 Remove disableIndexing 2018-10-23 16:05:45 -04:00
Jeff Mitchell 8a274fba51 Add disable indexing to core object 2018-10-23 15:04:36 -04:00
Jeff Mitchell a979f49cd7 Add disable-indexing 2018-10-23 15:03:17 -04:00
Jeff Mitchell 82992d6097
Seal migration (OSS) (#781) 2018-10-22 23:34:02 -07:00
Jeff Mitchell 89f0efb6a1 fmt 2018-10-20 21:09:51 -04:00
Jeff Mitchell 9f6dd376e2 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-19 17:47:58 -04:00
Calvin Leung Huang a08ccbffa7
[Review Only] Autoseal OSS port (#757)
* Port awskms autoseal

* Rename files

* WIP autoseal

* Fix protobuf conflict

* Expose some structs to properly allow encrypting stored keys

* Update awskms with the latest changes

* Add KeyGuard implementation to abstract encryption/decryption of keys

* Fully decouple seal.Access implementations from sealwrap structs

* Add extra line to proto files, comment update

* Update seal_access_entry.go

* govendor sync

* Add endpoint info to configureAWSKMSSeal

* Update comment

* Refactor structs

* Update make proto

* Remove remove KeyGuard, move encrypt/decrypt to autoSeal

* Add rest of seals, update VerifyRecoveryKeys, add deps

* Fix some merge conflicts via govendor updates

* Rename SealWrapEntry to EncryptedBlobInfo

* Remove barrier type upgrade check in oss

* Add key to EncryptedBlobInfo proto

* Update barrierTypeUpgradeCheck signature
2018-10-19 14:43:57 -07:00
Vishal Nayak c677cd0790
Case insensitive identity names (#5404)
* case insensitive identity names

* TestIdentityStore_GroupHierarchyCases

* address review feedback

* Use errwrap.Contains instead of errwrap.ContainsType

* Warn about duplicate names all the time to help fix them

* Address review feedback
2018-10-19 12:47:26 -07:00
Chris Hoffman 09a4c8214f
safely clean up loaded map (#5558) 2018-10-19 15:21:42 -04:00
Jeff Mitchell 841c4fcdd1 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-19 09:25:17 -04:00
Vishal Nayak 6ab030511c
Remove lookup check during alias removal (#5524)
* Possible fix for 5348

* Fix compilation
2018-10-18 07:53:12 -07:00
Vishal Nayak 5818977dca
Deprecate SHA1 in token store (#770)
* Deprecate SHA1 in token store

* Fallback to SHA1 for user selected IDs

* Fix existing tests

* Added warning

* Address some review feedback and remove root token prefix

* Tests for service token prefixing

* Salting utility tests

* Adjust OTP length for root token generation

* Fix tests

* Address review feedback
2018-10-17 13:23:04 -07:00
Jeff Mitchell 224fbd4a88 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-16 10:08:03 -04:00
Jeff Mitchell 04e3f9b0f3
Add LastWAL in leader/health output (#5523) 2018-10-16 09:38:44 -04:00
Jeff Mitchell a64fc7d7cb
Batch tokens (#755) 2018-10-15 12:56:24 -04:00
Vivek Lakshmanan 2c55777606 Fix expiration handling to not leak goroutines (#5506)
* Fix expiration handling to not leak goroutines

* Apply feedback
2018-10-12 19:02:59 -07:00
Jim Kalafut 123e34f4a7
Don't copy HA lock file during migration (#5503) 2018-10-12 09:29:15 -07:00
Calvin Leung Huang b47e648ddf
Logger cleanup (#5480) 2018-10-09 09:43:17 -07:00
vishalnayak baad5a66fd Fix TestIdentityStore_GroupHierarchyCases 2018-10-05 05:46:09 -04:00
Vishal Nayak fbec18fef0
Added test for verifying member group id deletion (#5469) 2018-10-04 10:38:41 -07:00
Sebastian Plattner 782f8dedd2 Fix remove Group Member in Identity Group not working (#5466) 2018-10-04 09:27:29 -07:00
Jeff Mitchell ec2ab502fc make fmt 2018-10-02 14:30:10 -04:00
Calvin Leung Huang 37c0b83669
Add denylist check when filtering passthrough headers (#5436)
* Add denylist check when filtering passthrough headers

* Minor comment update
2018-10-01 12:20:31 -07:00
Martin 03fb39033f Add support for token passed Authorization Bearer header (#5397)
* Support Authorization Bearer as token header

* add requestAuth test

* remove spew debug output in test

* Add Authorization in CORS Allowed headers

* use const where applicable

* use less allocations in bearer token checking

* address PR comments on tests and apply last commit

* reorder error checking in a TestHandler_requestAuth
2018-10-01 10:33:21 -07:00
Vishal Nayak 8e66e474ca Ensure old group alias is removed when a new one is written (#5350) 2018-10-01 10:06:10 -07:00
Jeff Mitchell ef144c4c25 Send initialized information via sys/seal-status (#5424) 2018-09-27 14:03:37 -07:00
Joel Thompson 73112c49fb logical/aws: Harden WAL entry creation (#5202)
* logical/aws: Harden WAL entry creation

If AWS IAM user creation failed in any way, the WAL corresponding to the
IAM user would get left around and Vault would try to roll it back.
However, because the user never existed, the rollback failed. Thus, the
WAL would essentially get "stuck" and Vault would continually attempt to
roll it back, failing every time. A similar situation could arise if the
IAM user that Vault created got deleted out of band, or if Vault deleted
it but was unable to write the lease revocation back to storage (e.g., a
storage failure).

This attempts to harden it in two ways. One is by deleting the WAL log
entry if the IAM user creation fails. However, the WAL deletion could
still fail, and this wouldn't help where the user is deleted out of
band, so second, consider the user rolled back if the user just doesn't
exist, under certain circumstances.

Fixes #5190

* Fix segfault in expiration unit tests

TestExpiration_Tidy was passing in a leaseEntry that had a nil Secret,
which then caused a segfault as the changes to revokeEntry didn't check
whether Secret was nil; this is probably unlikely to occur in real life,
but good to be extra cautious.

* Fix potential segfault

Missed the else...

* Respond to PR feedback
2018-09-27 09:54:59 -05:00
Brian Kassouf f5d0541d5d
Fix Capabilities check when in a child namespace (#5406) 2018-09-26 15:10:36 -07:00
Brian Kassouf 8f212d702d
replication: Fix DR API checks when using a token (#5398) 2018-09-25 13:27:57 -07:00
Vishal Nayak 68a496dde4
Support operating on entities and groups by their names (#5355)
* Support operating on entities and groups by their names

* address review feedback
2018-09-25 12:28:28 -07:00
Martin 79ab601cdb use constant where x-vault-token was still hardcoded (#5392) 2018-09-25 09:34:40 -07:00
Calvin Leung Huang ed1e41ba5c
Short-circuit TestBackend_PluginMainEnv on plain test run (#5393) 2018-09-25 09:22:34 -07:00
Jeff Mitchell 33065a60db Fix compilation/protobuf 2018-09-22 17:58:39 -04:00
andrejvanderzee dc6ea9ecbb Fix for using ExplicitMaxTTL in auth method plugins. (#5379)
* Fix for using ExplicitMaxTTL in auth method plugins.

* Reverted pb.go files for readability of PR.

* Fixed indenting of comment.

* Reverted unintended change by go test.
2018-09-21 14:31:29 -07:00
Jim Kalafut 343c72dbe1
Detect and bypass cycles during token revocation (#5364)
Fixes #4803
2018-09-20 14:56:38 -07:00
Calvin Leung Huang 189b893b35
Add ability to provide env vars to plugins (#5359)
* Add ability to provide env vars to plugins

* Update docs

* Update docs with examples

* Refactor TestAddTestPlugin, remove TestAddTestPluginTempDir
2018-09-20 10:50:29 -07:00
Jeff Mitchell 919b968c27
The big one (#5346) 2018-09-17 23:03:00 -04:00
Jeff Mitchell f692c1e3a9 Revert "Detect and bypass cycles during token revocation (#5335)"
This reverts commit 00314eb4d1c5609a1935f653dc6f2fc83c0bfcc0.
2018-09-17 14:10:57 -04:00
Jim Kalafut 0ae6ec52b8
Detect and bypass cycles during token revocation (#5335)
Fixes #4803
2018-09-17 08:55:12 -07:00
Becca Petrin b2ff87c9c2
Poll for new creds in the AWS auth agent (#5300) 2018-09-12 13:30:57 -07:00
vishalnayak e421972efb Remove group alias mdmdb update outside of UpsertGroupInTxn 2018-09-06 12:19:00 -04:00
Martin d51f3a45f7 Fix group alias loading when identity memdb is initialized (#5289) 2018-09-06 09:17:44 -07:00
Jeff Mitchell 95bdbbe85e
Port fix over that ensures we use the right step-down context (#5290) 2018-09-06 12:03:26 -04:00
Jeff Mitchell c28ed23972
Allow most parts of Vault's logging to have its level changed on-the-fly (#5280)
* Allow most parts of Vault's logging to have its level changed on-the-fly

* Use a const for not set
2018-09-05 15:52:54 -04:00
Jeff Mitchell c9e2cd93e8
Move logic around a bit to avoid holding locks when not necessary (#5277)
Also, ensure we are error checking the rand call
2018-09-05 11:49:32 -04:00
Chris Hoffman e2ed8d3d61
Fixing capabilities check for templated policies (#5250)
* fixing capabilities check for templated policies

* remove unnecessary change

* formatting
2018-09-04 14:18:59 -04:00
Brian Shumate 45f1ca162f Log 'marked as sealed' at INFO instead (#5260) 2018-09-04 10:53:40 -07:00
Becca Petrin 7a8c116fb1
undo make fmt (#5265) 2018-09-04 09:29:18 -07:00
Becca Petrin ed7639b0ec
run make fmt (#5261) 2018-09-04 09:12:59 -07:00
Jeff Mitchell a67869de22 Fix typo 2018-09-01 12:15:02 -04:00
Calvin Leung Huang 9988ace85e gofmt files (#5233) 2018-08-31 09:15:40 -07:00
Chris Hoffman 716fb03ab7
perform policy templating on each path (#5229) 2018-08-30 18:45:11 -04:00
Jeff Mitchell d57dfc1875 Move things back 2018-08-29 19:13:10 -04:00
Jeff Mitchell c6f7312f6c Move physical types around 2018-08-29 19:05:33 -04:00
Brian Kassouf 346d87f1f8 Pass the ctx value to make the race detector happy (#5201) 2018-08-27 18:21:54 -07:00
Jeff Mitchell 4761209331 Fix build 2018-08-27 19:59:59 -04:00
Jeff Mitchell b44b25d816
Allow fallback to non /-suffixed path for list acling (#5197)
This works around a very, very common error where people write policies
to affect listing but forget the slash at the end. If there is no exact
rule with a slash at the end when doing a list, we look to see if there
is a rule without it, and if so, use those capabilities.

Fixes #mass-user-confusion
2018-08-27 16:44:07 -07:00
Brian Kassouf c0ba9e8ff7
Fix potential deadlock (#5189) 2018-08-27 10:01:33 -07:00
Brian Kassouf c603a8b811
Add performance standby status to status output (#5192)
* Add performance standby status to status output

* Update ha.go
2018-08-27 10:01:07 -07:00
Jeff Mitchell 7a723b510e
Properly persist alias metadata (#5188)
In addition, don't lie about what's actually being stored
2018-08-26 10:26:34 -07:00
Brian Kassouf b7e33f1d2e
Port some HA changes (#5186) 2018-08-25 14:41:55 -07:00
Jeff Mitchell c4ebf3deda Fix expiration test 2018-08-24 12:47:56 -04:00
Jeff Mitchell f5024770dc Allow comment key in policies 2018-08-24 09:42:47 -04:00
Jeff Mitchell 362a92945e Don't resetnamed 2018-08-23 15:04:18 -04:00
Jeff Mitchell ba0d029247
Restricts ACL templating to paths but allows failures (#5167)
When a templating failure happens, we now simply ignore that path,
rather than fail all access to all policies
2018-08-23 12:15:02 -04:00
Jeff Mitchell 50197d5bfd
Only write valid group alias memberships into leases (#5164) 2018-08-22 21:53:04 -04:00
Jeff Mitchell 4bf0b12bfa Migrate external tests in vault folder 2018-08-22 20:50:52 -04:00
Brian Kassouf 2a89c60c7b Update ha.go 2018-08-22 20:45:31 -04:00
Vishal Nayak ee07e6ae59
Checks for assigning root policy (#5152)
* Fail if other policies present along with root

* disallow entity/group to contain root policy
2018-08-21 16:12:23 -04:00
Jeff Mitchell 3b01b29056
Pass in an ErrorLog to http.Server (#5135)
Fixes #5108
2018-08-21 11:23:18 -04:00
Jeff Mitchell 21cd0dd71a Use strings.Contains for error possibly coming from storage
They may not well errwrap

Fixes #5046
2018-08-17 16:06:47 -04:00
Jeff Mitchell 71d92ef093 ACL Templating (#4994)
* Initial work on templating

* Add check for unbalanced closing in front

* Add missing templated assignment

* Add first cut of end-to-end test on templating.

* Make template errors be 403s and finish up testing

* Review feedback
2018-08-15 11:42:56 -07:00
Vishal Nayak 303b2f97ff
don't ignore errors (#5087) 2018-08-10 22:32:10 -04:00
Vishal Nayak 68d27c7c38
Adjust ACL error checking (#5086)
* Adjust acl error checking

* address review feedback

* unlock before checks
2018-08-10 21:05:10 -04:00
Jeff Mitchell 4798af88f9
Fix some cases where unauthorized calls could return 400 (#5083) 2018-08-10 08:59:58 -05:00
Jeff Mitchell 14dccd7744 Remove unused variable 2018-08-09 16:47:58 -04:00
Jeff Mitchell 2ed2e696a7
Merge Identity Entities if two claim the same alias (#5075)
* Merge Identity Entities if two claim the same alias

Past bugs/race conditions meant two entities could be created each
claiming the same alias. There are planned longer term fixes for this
(outside of the race condition being fixed in 0.10.4) that involve
changing the data model, but this is an immediate workaround that has
the same net effect: if two entities claim the same alias, assume they
were created due to this race condition and merge them.

In this situation, also automatically merge policies so we don't lose
e.g. RGPs.
2018-08-09 15:37:36 -05:00
Martin aa8662c6c0 Add mount_path and mount_type to group lookups (#5074) 2018-08-09 13:07:24 -05:00
Jeff Mitchell 7d90e6eae9 Fix typo 2018-08-08 14:38:11 -04:00
Brian Kassouf 3330a3954a
Rollback: Only grab statelock if we are running async (#5064) 2018-08-07 17:48:43 -07:00
Jim Kalafut 4bdbc820ed
Add PluginEnv to SystemView (#5028) 2018-08-03 09:32:17 -07:00
Chris Hoffman d8b1d19ed6
Plumbing request context through to expiration manager (#5021)
* plumbing request context to expiration manager

* moar context

* address feedback

* only using active context for revoke prefix

* using active context for revoke commands

* cancel tidy on active context

* address feedback
2018-08-01 21:39:39 -04:00
Brian Kassouf bb076f161d
core: Cancel context before taking state lock (#5020)
* core: Cancel context before taking state lock

* Create active context outside of postUnseal

* Attempt to drain requests before canceling context

* fix test
2018-08-01 12:07:37 -07:00
Chris Hoffman b229df9cd2
fixing rollback test 2018-07-31 22:45:38 -04:00
Brian Kassouf 1781ade3a1
core: Read lock when doing a rollback (#5016) 2018-07-30 19:55:43 -06:00
Chris Hoffman 51f8cd9668
do not grab statelock when requested not to (#5015) 2018-07-30 14:54:54 -04:00
Jeff Mitchell 0e659ca6be Simplify sealInternal 2018-07-24 23:26:28 -04:00
Chris Hoffman 1578c5b982 Add locking when adding aliases to existing entities (#4965) 2018-07-24 22:01:58 -04:00
Jeff Mitchell 4261618d10 Add request timeouts in normal request path and to expirations (#4971)
* Add request timeouts in normal request path and to expirations

* Add ability to adjust default max request duration

* Some test fixes

* Ensure tests have defaults set for max request duration

* Add context cancel checking to inmem/file

* Fix tests

* Fix tests

* Set default max request duration to basically infinity for this release for BC

* Address feedback
2018-07-24 14:50:49 -07:00
Jeff Mitchell 9687ccc8fa Tackle #4929 a different way (#4932)
* Tackle #4929 a different way

This turns c.sealed into an atomic, which allows us to call sealInternal
without a lock. By doing so we can better control lock grabbing when a
condition causing the standby loop to get out of active happens. This
encapsulates that logic into two distinct pieces (although they could
be combined into one), and makes lock guarding more understandable.

* Re-add context canceling to the non-HA version of sealInternal

* Return explicitly after stopCh triggered
2018-07-24 13:57:25 -07:00
Jeff Mitchell d144f2935e Two-pronged fix for renew policy checking (#4960)
1) In backends, ensure they are now using TokenPolicies
2) Don't reassign auth.Policies until after expmgr registration as we
don't need them at that point

Fixes #4829
2018-07-24 12:03:11 -07:00
Jeff Mitchell caa5661031
Pass identity metadata through to plugins (#4967)
It's not obvious why this should be secret, and if it were considered
secret, when and what anything would ever be allowed to access it.
Likely the right way to tie secret values to particular
entities/aliases/groups would be to use the upcoming templated ACL
feature.
2018-07-23 12:45:06 -04:00
Jeff Mitchell 1d99b7fd05
Properly watch quit context in expireID instead of locking first (#4970) 2018-07-20 17:00:09 -04:00