Commit Graph

604 Commits

Author SHA1 Message Date
Scott Miller b13b27f37e
OSS side barrier encryption tracking and automatic rotation (#11007)
* Automatic barrier key rotation, OSS portion

* Fix build issues

* Vendored version

* Add missing encs field, not sure where this got lost.
2021-02-25 14:27:25 -06:00
Nick Cabatoff c1ddfbb538
OSS parts of the new client controlled consistency feature (#10974) 2021-02-24 06:58:10 -05:00
swayne275 6e1b183f79
Shutdown Test Cores when Tests Complete (#10912)
* Shutdown Test Cores when Tests Complete

* go mod vendor
2021-02-12 13:04:48 -07:00
Vishal Nayak 53cb1deb38
Revert "Read-replica instead of non-voter (#10875)" (#10890)
This reverts commit fc745670cf34821f5834357d9caebc3351dbc1e7.
2021-02-10 16:41:58 -05:00
Vishal Nayak a2394e7353
Read-replica instead of non-voter (#10875) 2021-02-10 09:58:18 -05:00
Mark Gritter fd55aa8378
Implement sys/seal-status and sys/leader in system backend (#10725)
* Implement sys/seal-status and sys/leader as normal API calls
(so that they can be used in namespaces.)
* Added changelog.
2021-01-20 14:04:24 -06:00
Nick Cabatoff 8cbc63d572
Add configuration to specify a TLS ServerName to use in the TLS handshake when performing a raft join. (#10698) 2021-01-19 17:54:28 -05:00
Gunjan 4900283ad5
Fix: handle max_request_size<=0 (#10072)
* Fix: handle max_request_size<=0

Signed-off-by: guacamole <gunjanwalecha@gmail.com>

* created test cases for listener

Signed-off-by: guacamole <gunjanwalecha@gmail.com>

* added test case for negative value of MaxRequestSize

Signed-off-by: guacamole <gunjanwalecha@gmail.com>

Co-authored-by: Hridoy Roy <roy@hashicorp.com>
2021-01-19 11:28:28 -08:00
Mark Gritter d076d95d37
Feature flags API (#10613)
* Added sys/internal/ui/feature-flags endpoint.
* Added documentation for new API endpoint.
* Added integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
2021-01-06 16:05:00 -06:00
Scott Miller 2eafeee15f
Avoid disclosing IP addresses in unauthenticated requests (#10579)
* removing extra information from the returned error, to avoid leaking it to unauthenticated requests

* removing extra information from the returned error, to avoid leaking it to unauthenticated requests

* Change the error message in a way that is retains the HTTP status code

Co-authored-by: bruj0 <ramakandra@gmail.com>
2020-12-22 10:30:03 -06:00
Scott G. Miller ffc375817e Revert "Disclosure of internal IP addresses from unauthenticated request (#10516)"
This reverts commit 9934f75f1daa21de3137c1cd16423a15d594709b.
2020-12-15 13:57:32 -06:00
Rodrigo D. L 1adb11c413
Disclosure of internal IP addresses from unauthenticated request (#10516)
* removing extra information from the returned error, to avoid leaking it to unauthenticated requests

* removing extra information from the returned error, to avoid leaking it to unauthenticated requests

Co-authored-by: Scott Miller <smiller@hashicorp.com>
2020-12-15 11:08:50 -06:00
Brian Kassouf 275ca323e8
core: Record the time a node became active (#10489)
* core: Record the time a node became active

* Update vault/core.go

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

* Add omitempty field

* Update vendor

* Added CL entry and fixed test

* Fix test

* Fix command package tests

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2020-12-11 16:50:19 -08:00
Nick Cabatoff 84d566db9e
Be consistent with how we report init status. (#10498)
Also make half-joined raft peers consider storage to be initialized, whether or not they're sealed.
2020-12-08 13:55:34 -05:00
Vishal Nayak 7912ac9713 Fix build 2020-10-26 18:14:01 -04:00
Jeff Mitchell 3b93a18ef2
Consolidate locking for sys/health (#9876)
* Consolidate locking for sys/health

This avoids a second state lock read-lock on every sys/health hit

* Address review feedback

Co-authored-by: Vishal Nayak <vishalnayakv@gmail.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-10-26 16:47:54 -04:00
Clint 95810d1360
Return logical.StatusBadRequest on requests with missing token (#8457)
* Add test for 400 status on missing token

* Return logical.StatusBadRequest on missing token

* remove commented out code

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-10-26 16:17:25 -04:00
Aleksandr Bezobchuk 95bbd8d920
Merge PR #10192: Auto-Join: Configurable Scheme & Port (and add k8s provider) 2020-10-23 16:13:09 -04:00
Nick Cabatoff 0d6a929a4c
Same seal migration oss (#10224)
* Refactoring and test improvements.

* Support migrating from a given type of autoseal to that same type but with different parameters.
2020-10-23 14:16:04 -04:00
Brian Kassouf 84dbca38a1
Revert "Migrate internalshared out (#9727)" (#10141)
This reverts commit ee6391b691ac12ab6ca13c3912404f1d3a842bd6.
2020-10-13 16:38:21 -07:00
Aleksandr Bezobchuk d37be9af6e
Merge PR #10095: Integrated Storage Cloud Auto-Join 2020-10-13 16:26:39 -04:00
Jeff Mitchell e6881c8147
Migrate internalshared out (#9727)
* Migrate internalshared out

* fix merge issue

* fix merge issue

* go mod vendor

Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
2020-10-12 11:56:24 -07:00
Ivan Lemeshev 8a2842b467
Fix panic in handleAuditNonLogical if vault is sealed (#9310) (#10103) 2020-10-07 08:30:36 -04:00
ncabatoff 30eba1eed1
Update retryablehttp to fix a data race (#9551) 2020-08-31 11:10:52 -04:00
Rodrigo D. L d0df8bfa21
adding new config flag disable_sentinel_trace (#9696) 2020-08-10 06:23:44 -04:00
Alexander Bezobchuk 1e262e5648
Merge PR #9581: Rate Limit Quota Headers 2020-07-29 15:15:05 -04:00
aphorise 7e2f80b8c4
Always provide version & storage type in status output (#9598) 2020-07-27 14:33:57 -04:00
Alexander Bezobchuk dfb28a8fcc
Merge PR #9390: http: revert resource quota changes 2020-07-07 00:05:28 -04:00
Vishal Nayak c6876fe00f
Resource Quotas: Rate Limiting (#9330) 2020-06-26 17:13:16 -04:00
Calvin Leung Huang c45bdca0b3
raft: add support for using backend for ha_storage (#9193)
* raft: initial work on raft ha storage support

* add note on join

* add todo note

* raft: add support for bootstrapping and joining existing nodes

* raft: gate bootstrap join by reading leader api address from storage

* raft: properly check for raft-only for certain conditionals

* raft: add bootstrap to api and cli

* raft: fix bootstrap cli command

* raft: add test for setting up new cluster with raft HA

* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends

* raft: add test for updating an existing cluster to use raft HA

* raft: remove debug log lines, clean up verifyRaftPeers

* raft: minor cleanup

* raft: minor cleanup

* Update physical/raft/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/ha.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/ha.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/logical_system_raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* address feedback comments

* address feedback comments

* raft: refactor tls keyring logic

* address feedback comments

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* address feedback comments

* testing: fix import ordering

* raft: rename var, cleanup comment line

* docs: remove ha_storage restriction note on raft

* docs: more raft HA interaction updates with migration and recovery mode

* docs: update the raft join command

* raft: update comments

* raft: add missing isRaftHAOnly check for clearing out state set earlier

* raft: update a few ha_storage config checks

* Update command/operator_raft_bootstrap.go

Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>

* raft: address feedback comments

* raft: fix panic when checking for config.HAStorage.Type

* Update vault/raft.go

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* Update website/pages/docs/commands/operator/raft.mdx

Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>

* raft: remove bootstrap cli command

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* Update vault/raft.go

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* raft: address review feedback

* raft: revert vendored sdk

* raft: don't send applied index and node ID info if we're HA-only

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
2020-06-23 12:04:13 -07:00
Josh Black 6e92c8cbd2
Add a new "vault monitor" command (#8477)
Add a new "vault monitor" command

Co-authored-by: ncabatoff <ncabatoff@hashicorp.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-21 13:07:50 -07:00
Brian Kassouf 1bb0bd489d
storage/raft: Add committed and applied indexes to the status output (#9011)
* storage/raft: Add committed and applied indexes to the status output

* Update api vendor

* changelog++

* Update http/sys_leader.go

Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>

Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
2020-05-18 16:07:27 -07:00
Jeff Mitchell 1d3d89e2aa
Create configutil and move some common config and setup functions there (#8362) 2020-05-14 09:19:27 -04:00
Calvin Leung Huang 0509985408
http: ensure return after writing response by respondError (#8796) 2020-04-21 15:30:36 -07:00
Graham Jenson c2c42ba2e9
[Fix] correct make command in ui page (#7976) 2020-03-24 14:15:27 -07:00
Vitaly Velikodny 3ae6568512
Add missed description field for GET /sys/auth/:path/tune endpoint (#8193)
* fix #7623: add missed description field for GET /sys/auth/:path/tune endpoint

* fix #7623: allow empty description

* fix #7623: update tests with description field
2020-02-15 10:32:47 -08:00
Jim Kalafut f4cafabb62
Support processing parameters sent as a URL-encoded form (#8325) 2020-02-12 14:20:22 -08:00
Michel Vocks 3a4d330f64
Return error when a standby node receives a metrics request (#8280)
* Return error when a standby node receives a metrics request

* fix test

* Add documentation note
2020-02-07 09:30:25 +01:00
ncabatoff a0ac4bdd2a
Audit generate root requests and responses. (#8301) 2020-02-06 11:56:37 -05:00
Jim Kalafut f17fc4e5c1
Run goimports (#8251) 2020-01-27 21:11:00 -08:00
Vishal Nayak 8891f2ba88 Raft retry join (#7856)
* Raft retry join

* update

* Make retry join work with shamir seal

* Return upon context completion

* Update vault/raft.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>

* Address some review comments

* send leader information slice as a parameter

* Make retry join work properly with Shamir case. This commit has a blocking issue

* Fix join goroutine exiting before the job is done

* Polishing changes

* Don't return after a successful join during unseal

* Added config parsing test

* Add test and fix bugs

* minor changes

* Address review comments

* Fix build error

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2020-01-13 17:02:16 -08:00
Jim Kalafut 59e526614d
Run go fmt (#7823) 2019-11-07 08:54:34 -08:00
Brian Kassouf 0bc14636b0 Fix build 2019-10-28 17:40:44 -07:00
Jeff Mitchell 698b0dd025 If standbyok/perfstandbyok are provided to sys/health, honor the values (#7749)
Don't just use the presence of it to indicate behavior.

Fixes #7323

Also, fixes a bug where if an error was returned along with a status
code, the status code was being ignored.
2019-10-28 16:55:20 -07:00
ncabatoff 20b8f8d7d0
Don't try to use req if we got a nonzero status, it'll be nil. (#7728) 2019-10-24 13:37:13 -04:00
ncabatoff 1c98152fa0
Shamir seals now come in two varieties: legacy and new-style. (#7694)
Shamir seals now come in two varieties: legacy and new-style. Legacy
Shamir is automatically converted to new-style when a rekey operation
is performed. All new Vault initializations using Shamir are new-style.

New-style Shamir writes an encrypted master key to storage, just like
AutoUnseal. The stored master key is encrypted using the shared key that
is split via Shamir's algorithm. Thus when unsealing, we take the key
fragments given, combine them into a Key-Encryption-Key, and use that
to decrypt the master key on disk. Then the master key is used to read
the keyring that decrypts the barrier.
2019-10-18 14:46:00 -04:00
Calvin Leung Huang f5b63cc815
logical: remove unneeded error check in handleLogicalInternal (#7691) 2019-10-17 12:33:29 -07:00
Lexman c86fe212c0
oss changes for entropy augmentation feature (#7670)
* oss changes for entropy augmentation feature

* fix oss command/server/config tests

* update go.sum

* fix logical_system and http/ tests

* adds vendored files

* removes unused variable
2019-10-17 10:33:00 -07:00
Vishal Nayak 5004899eb2 Re-add removed paths from the recovery PR 2019-10-15 11:42:51 -04:00
Vishal Nayak 0d077d7945
Recovery Mode (#7559)
* Initial work

* rework

* s/dr/recovery

* Add sys/raw support to recovery mode (#7577)

* Factor the raw paths out so they can be run with a SystemBackend.

# Conflicts:
#	vault/logical_system.go

* Add handleLogicalRecovery which is like handleLogical but is only
sufficient for use with the sys-raw endpoint in recovery mode.  No
authentication is done yet.

* Integrate with recovery-mode.  We now handle unauthenticated sys/raw
requests, albeit on path v1/raw instead v1/sys/raw.

* Use sys/raw instead raw during recovery.

* Don't bother persisting the recovery token.  Authenticate sys/raw
requests with it.

* RecoveryMode: Support generate-root for autounseals (#7591)

* Recovery: Abstract config creation and log settings

* Recovery mode integration test. (#7600)

* Recovery: Touch up (#7607)

* Recovery: Touch up

* revert the raw backend creation changes

* Added recovery operation token prefix

* Move RawBackend to its own file

* Update API path and hit it using CLI flag on generate-root

* Fix a panic triggered when handling a request that yields a nil response. (#7618)

* Improve integ test to actually make changes while in recovery mode and
verify they're still there after coming back in regular mode.

* Refuse to allow a second recovery token to be generated.

* Resize raft cluster to size 1 and start as leader (#7626)

* RecoveryMode: Setup raft cluster post unseal (#7635)

* Setup raft cluster post unseal in recovery mode

* Remove marking as unsealed as its not needed

* Address review comments

* Accept only one seal config in recovery mode as there is no scope for migration
2019-10-15 00:55:31 -04:00
Brian Kassouf 024c29c36a
OSS portions of raft non-voters (#7634)
* OSS portions of raft non-voters

* add file

* Update vault/raft.go

Co-Authored-By: Vishal Nayak <vishalnayak@users.noreply.github.com>
2019-10-11 11:56:59 -07:00
Calvin Leung Huang d8875b1991
sys/config: config state endpoint (#7424)
* sys/config: initial work on adding config state endpoint

* server/config: add tests, fix Sanitized method

* thread config through NewTestCluster's config to avoid panic on dev modes

* properly guard endpoint against request forwarding

* add http tests, guard against panics on nil RawConfig

* ensure non-nil rawConfig on NewTestCluster cores

* update non-forwarding logic

* fix imports; use no-forward handler

* add missing config test fixture; update gitignore

* return sanitized config as a map

* fix test, use deep.Equal to check for equality

* fix http test

* minor comment fix

* config: change Sanitized to return snake-cased keys, update tests

* core: hold rlock when reading config; add docstring

* update docstring
2019-10-08 10:57:15 -07:00
Michel Vocks a0c122926a
Unauth metrics: Fix missing parse form and error response (#7569)
* Unauth metrics: Fix missing parse form and error response

* Change metrics error response to text/plain content type always
2019-10-08 14:55:25 +02:00
Michel Vocks a7a6dd55a5
Add config parameter to allow unauthenticated metrics access (#7550)
* Implement config parameter to allow unathenticated metricss access

* Add unit test for unauthenticated metrics access parameter

* go mod tidy
2019-10-04 09:29:51 +02:00
Calvin Leung Huang 3f1c7c86a0
sys: add host-info endpoint (#7330)
* sys: add host-info endpoint, add client API method

* remove old commented handler

* add http tests, fix bugs

* query all partitions for disk usage

* fix Timestamp decoding

* add comments for clarification

* dont append a nil entry on disk usage query error

* remove HostInfo from the sdk api

We can use Logical().Read(...) to query this endpoint since the payload is contained with the data object. All warnings are preserved under Secret.Warnings.

* ensure that we're testing failure case against a standby node

* add and use TestWaitStandby to ensure core is on standby

* remove TestWaitStandby

* respond with local-only error

* move HostInfo into its own helper package

* fix imports; use new no-forward handler

* add cpu times to collection

* emit clearer multierrors/warnings by collection type

* add comments on HostInfo fields
2019-10-03 09:43:52 -07:00
Calvin Leung Huang f4fd84d52b
sys: add pprof endpoint (#7473)
* sys/pprof: add pprof routes to the system backend

* sys/pprof: add pprof paths to handler with local-only check

* fix trailing slash on pprof index endpoint

* use new no-forward handler on pprof

* go mod tidy

* add pprof external tests

* disallow streaming requests to exceed DefaultMaxRequestDuration

* add max request duration test
2019-09-19 13:44:37 -07:00
Matthew Irish 6e4cc02f4d
expose 'storage_type' on the sys/seal-status endpoint (#7486)
* expose 'storage_type' on the sys/seal-status endpoint

* add comments

* Update vault/core.go

Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
2019-09-18 14:07:18 -05:00
Calvin Leung Huang 801e15ffe3
http: add handleLogicalNoForward handler for local-only requests (#7482)
* http: add handleLogicalNoForward handler for local-only requests

* fix missing no-forward case; add docstring

* add docstring for handleLogicalInternal

* fix docstring reference

* fix comment on not-ok case
2019-09-16 17:50:28 -07:00
Calvin Leung Huang ec64b7c672
logical/request: store the entire http.Request object instead (#7412)
This allows logical operations (along with a non-nil response writer) to
process http handler funcs within the operation function while keeping
auth and audit checks that the logical request flow provides.
2019-09-06 12:40:15 -07:00
Brian Kassouf c2905773e4
Add download headers to snapshot take API (#7369)
* Add download headers to snapshot take API

* Add content type
2019-09-06 10:34:36 -07:00
Calvin Leung Huang bcb0109e26
http/raft: fix JoinRequest.LeaderCACert json tag (#7393) 2019-08-29 14:37:01 -07:00
ncabatoff 439ea99c83
Follow what documentation says we should do if we're a perf standby and perfstandbyok=true (#7241)
Follow what documentation says we should do if we're a perf standby and perfstandbyok=true, i.e. return 200 instead of 429.
2019-08-05 16:44:41 -04:00
Calvin Leung Huang e869893df3
logical: add support for passing data to delete (#7139)
* logical: add support for passing data to delete

* add back raft bit

* add back raft bit

* update error message

* fix command delete tests
2019-07-18 10:42:36 -07:00
Lexman 119854a865
adds Cache-Control header to oidc .well-known endpoints (#7108) 2019-07-15 11:04:45 -07:00
Calvin Leung Huang 5428ab50ee audit: log invalid wrapping token request/response (#6541)
* audit: log invalid wrapping token request/response

* Update helper/consts/error.go

Co-Authored-By: calvn <cleung2010@gmail.com>

* update error comments

* Update vault/wrapping.go

Co-Authored-By: calvn <cleung2010@gmail.com>

* update comment

* move validateWrappingToken out of http and into logical

* minor refactor, add test cases

* comment rewording

* refactor validateWrappingToken to perform audit logging

* move ValidateWrappingToken back to wrappingVerificationFunc

* Fix tests

* Review feedback
2019-07-05 14:15:14 -07:00
Vishal Nayak 13935f00af
Fix Join (#6974) 2019-06-24 18:49:28 -04:00
Vishal Nayak 06f30a7947
Move tls config creation to tlsutil (#6956)
* Move tls config creation to tlsutil

* Update sdk/helper/tlsutil/tlsutil.go

Co-Authored-By: Jim Kalafut <jim@kalafut.net>

* address review comments
2019-06-22 21:51:52 -04:00
Vishal Nayak 9296ca1f8c
raft join tls (#6932) 2019-06-21 17:41:07 -04:00
Brian Kassouf ed14061578
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
Brian Kassouf 04d0ddfdae
Add mount UUID to the secret and auth list API responses (#6633) 2019-04-24 12:27:43 -07:00
Jeff Mitchell 213b9fd1cf Update to api 1.0.1 and sdk 0.1.8 2019-04-15 14:10:07 -04:00
Jeff Mitchell 9ebc57581d
Switch to go modules (#6585)
* Switch to go modules

* Make fmt
2019-04-13 03:44:06 -04:00
Jeff Mitchell 8bcb533a1b
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
Jeff Mitchell 2fbf560fd2 Remove out-of-date comment 2019-04-08 16:49:10 -04:00
Jeff Mitchell a308bad657 Fix a potential panic due to ioutil.ReadAll not being safe with nil readers 2019-04-08 16:48:44 -04:00
Jeff Mitchell 991373c969 Fix tests 2019-04-08 12:44:12 -04:00
Jeff Mitchell 00295cd598
Save the original request body for forwarding (#6538)
* Save the original request body for forwarding

If we are forwarding a request after initial parsing the request body is
already consumed. As a result a forwarded call containing a request body
will have the body be nil. This saves the original request body for a
given request via a TeeReader and uses that in cases of forwarding past
body consumption.
2019-04-05 14:36:34 -04:00
Jeff Mitchell eec3f9afb2
Handle JWT checks with namespaced service tokens (#6536)
Some checks would fail because we considered a token with two dots a
JWT, but service tokens in namespaces also fit this bill.
2019-04-04 17:13:09 -04:00
T.K 453f1ac109 changed misspelled english words (#6432) 2019-03-19 09:32:45 -04:00
Vishal Nayak 59e8632d2d
Refactor respond error function (#6343) 2019-03-05 11:43:30 -05:00
Calvin Leung Huang 1ae110e59a
core: handleLoginRequest should set resp.Auth.Orphan = true (#6335)
* core: handleLoginRequest should set resp.Auth.Orphan = true

* set auth.Orphan in RegisterAuth dynamically

* update related tests
2019-03-04 14:32:09 -08:00
Brian Kassouf 7b910a093b
Handle ns lease and token renew/revoke via relative paths (#6236) (#6312)
* Handle ns lease and token renew/revoke via relative paths

* s/usin/using/

* add token and lease lookup paths; set ctx only on non-nil ns

Addtionally, use client token's ns for auth/token/lookup if no token is provided
2019-02-28 16:02:25 -08:00
Brian Kassouf efe5671f36 make fmt 2019-02-20 12:12:21 -08:00
Nick Cabatoff 3e871daec3 Fix TestSysMount. 2019-02-15 14:28:39 -05:00
ncabatoff 81c5a87d85 Fix TestSysRemount. (#6248) 2019-02-15 11:15:39 -08:00
ncabatoff 62113d614b Fix TestSysMounts, TestSysMounts_headerAuth. (#6246) 2019-02-15 11:15:02 -08:00
ncabatoff cab3e6815b Fix broken tests resulting from new mount config field passthrough_request_headers. (#6245) 2019-02-15 10:20:29 -08:00
Jeff Mitchell ebf57b15b4
Fixes a regression in forwarding from #6115 (#6191)
* Fixes a regression in forwarding from #6115

Although removing the authentication header is good defense in depth,
for forwarding mechanisms that use the raw request, we never add it
back. This caused perf standby tests to throw errors. Instead, once
we're past the point at which we would do any raw forwarding, but before
routing the request, remove the header.

To speed this up, a flag is set in the logical.Request to indicate where
the token is sourced from. That way we don't iterate through maps
unnecessarily.
2019-02-11 13:08:15 -05:00
Jeff Mitchell 5f249d4005
Add allowed_response_headers (#6115) 2019-02-05 16:02:15 -05:00
Jeff Mitchell e781ea3ac4
First part of perf standby entity race fix (#6106) 2019-01-25 14:08:42 -05:00
Seth Vargo 98ad431d6d Continuously attempt to unseal if sealed keys are supported (#6039)
* Add helper for checking if an error is a fatal error

The double-double negative was really confusing, and this pattern is used a few places in Vault. This negates the double negative, making the devx a bit easier to follow.

* Check return value of UnsealWithStoredKeys in sys/init

* Return proper error types when attempting unseal with stored key

Prior to this commit, "nil" could have meant unsupported auto-unseal, a transient error, or success. This updates the function to return the correct error type, signaling to the caller whether they should retry or fail.

* Continuously attempt to unseal if sealed keys are supported

This fixes a bug that occurs on bootstrapping an initial cluster. Given a collection of Vault nodes and an initialized storage backend, they will all go into standby waiting for initialization. After one node is initialized, the other nodes had no mechanism by which they "re-check" to see if unseal keys are present. This adds a goroutine to the server command which continually waits for unseal keys to exist. It exits in the following conditions:

- the node is unsealed
- the node does not support stored keys
- a fatal error occurs (as defined by Vault)
- the server is shutting down

In all other situations, the routine wakes up at the specified interval and attempts to unseal with the stored keys.
2019-01-23 16:34:34 -05:00
Jim Kalafut d0e2badbae Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00
Jim Kalafut 2547d7fb6a
Simplify base62.Random (#5982)
Also move existing base62 encode/decode operations to their only points
of use.
2018-12-20 07:40:01 -08:00
Brian Kassouf a32c37331d
Update handler_test.go 2018-11-19 15:25:41 -08:00
Calvin Leung Huang ad3e105012
Mount tune options (#5809)
* Refactor mount tune to support upsert options values and unset options.

* Do not allow unsetting options map

* add secret tune version regression test

* Only accept valid options version

* s/meVersion/optVersion/
2018-11-19 14:23:25 -08:00
Brian Kassouf 33776b89c2
Wrap storage calls with encoding checks (#5819)
* Add encoding backend

* More work on encoding checks

* Update error message

* Update physical/encoding.go

* Disable key checks if configured
2018-11-19 13:13:16 -08:00
Becca Petrin 7bd22e6779
Run all builtins as plugins (#5536) 2018-11-06 17:21:24 -08:00
Matthew Irish 5b1a889e00 serve robots.txt from the root when the UI is enabled (#5686) 2018-11-05 16:48:46 -05:00
Noelle Daley 3ca487be16 style ui not bundled page 2018-11-02 13:29:13 -07:00
Jeff Mitchell 82992d6097
Seal migration (OSS) (#781) 2018-10-22 23:34:02 -07:00
Jeff Mitchell d843e0b52c Merge branch 'master-oss' into 1.0-beta-oss 2018-10-18 10:28:14 -04:00
Vishal Nayak 5818977dca
Deprecate SHA1 in token store (#770)
* Deprecate SHA1 in token store

* Fallback to SHA1 for user selected IDs

* Fix existing tests

* Added warning

* Address some review feedback and remove root token prefix

* Tests for service token prefixing

* Salting utility tests

* Adjust OTP length for root token generation

* Fix tests

* Address review feedback
2018-10-17 13:23:04 -07:00
Matthew Irish 72b1d4270c
Merge branch 'master' into ui-handle-redirect 2018-10-17 08:59:24 -05:00
Jeff Mitchell 224fbd4a88 Merge branch 'master-oss' into 1.0-beta-oss 2018-10-16 10:08:03 -04:00
Jeff Mitchell 04e3f9b0f3
Add LastWAL in leader/health output (#5523) 2018-10-16 09:38:44 -04:00
Jeff Mitchell a64fc7d7cb
Batch tokens (#755) 2018-10-15 12:56:24 -04:00
Matthew Irish c81b2bdae6 adjust prefix 2018-10-15 11:45:16 -05:00
Matthew Irish d9f5c3d8cd redirect ui to ui/ 2018-10-15 11:23:46 -05:00
Jeff Mitchell d373b07f88
Fix root generation init not allowing empty input (#5495) 2018-10-10 11:54:12 -04:00
Calvin Leung Huang b47e648ddf
Logger cleanup (#5480) 2018-10-09 09:43:17 -07:00
Jeff Mitchell ec2ab502fc make fmt 2018-10-02 14:30:10 -04:00
Brian Kassouf 2ec54c3a0b
Fix seal status tests (#5443) 2018-10-01 18:09:20 -07:00
Martin 03fb39033f Add support for token passed Authorization Bearer header (#5397)
* Support Authorization Bearer as token header

* add requestAuth test

* remove spew debug output in test

* Add Authorization in CORS Allowed headers

* use const where applicable

* use less allocations in bearer token checking

* address PR comments on tests and apply last commit

* reorder error checking in a TestHandler_requestAuth
2018-10-01 10:33:21 -07:00
Jeff Mitchell ef144c4c25 Send initialized information via sys/seal-status (#5424) 2018-09-27 14:03:37 -07:00
Martin 79ab601cdb use constant where x-vault-token was still hardcoded (#5392) 2018-09-25 09:34:40 -07:00
Calvin Leung Huang 189b893b35
Add ability to provide env vars to plugins (#5359)
* Add ability to provide env vars to plugins

* Update docs

* Update docs with examples

* Refactor TestAddTestPlugin, remove TestAddTestPluginTempDir
2018-09-20 10:50:29 -07:00
Clint 5882156f53
Translate AWS Rate limiting errors to 502 errors (#5270)
* Initial implemntation of returning 529 for rate limits

- bump aws iam and sts packages to v1.14.31 to get mocking interface
- promote the iam and sts clients to the aws backend struct, for mocking in tests
- this also promotes some functions to methods on the Backend struct, so
  that we can use the injected client

Generating creds requires reading config/root for credentials to contact
IAM. Here we make pathConfigRoot a method on aws/backend so we can clear
the clients on successful update of config/root path. Adds a mutex to
safely clear the clients

* refactor locking and unlocking into methods on *backend

* refactor/simply the locking

* check client after grabbing lock
2018-09-18 15:26:06 -05:00
Jeff Mitchell 919b968c27
The big one (#5346) 2018-09-17 23:03:00 -04:00
Jeff Mitchell 7191bca71a
Re-add injecting into top routes (#5244) 2018-09-05 11:45:17 -04:00
Becca Petrin 7a8c116fb1
undo make fmt (#5265) 2018-09-04 09:29:18 -07:00
Becca Petrin ed7639b0ec
run make fmt (#5261) 2018-09-04 09:12:59 -07:00
Calvin Leung Huang 9988ace85e gofmt files (#5233) 2018-08-31 09:15:40 -07:00
Brian Kassouf 1ebba3a7fb
Fix http tests (#5195) 2018-08-27 15:13:41 -07:00
Brian Kassouf c603a8b811
Add performance standby status to status output (#5192)
* Add performance standby status to status output

* Update ha.go
2018-08-27 10:01:07 -07:00
Jeff Mitchell 3b01b29056
Pass in an ErrorLog to http.Server (#5135)
Fixes #5108
2018-08-21 11:23:18 -04:00
Jeff Mitchell 9b4e04917e Fix http tests 2018-08-14 17:09:55 -04:00
Jeff Mitchell f1d72abb39 Remove injection into top routes (#5101) 2018-08-14 15:29:22 -04:00
Jeff Mitchell 74175b29af
Add support for passing args via `vault read` (#5093)
We support this in the API as of 0.10.2 so read should support it too.

Trivially tested with some log info:

`core: data: data="map[string]interface {}{"zip":[]string{"zap", "zap2"}}"`
2018-08-13 22:00:26 -04:00
Jeff Mitchell 4261618d10 Add request timeouts in normal request path and to expirations (#4971)
* Add request timeouts in normal request path and to expirations

* Add ability to adjust default max request duration

* Some test fixes

* Ensure tests have defaults set for max request duration

* Add context cancel checking to inmem/file

* Fix tests

* Fix tests

* Set default max request duration to basically infinity for this release for BC

* Address feedback
2018-07-24 14:50:49 -07:00
Jeff Mitchell 9687ccc8fa Tackle #4929 a different way (#4932)
* Tackle #4929 a different way

This turns c.sealed into an atomic, which allows us to call sealInternal
without a lock. By doing so we can better control lock grabbing when a
condition causing the standby loop to get out of active happens. This
encapsulates that logic into two distinct pieces (although they could
be combined into one), and makes lock guarding more understandable.

* Re-add context canceling to the non-HA version of sealInternal

* Return explicitly after stopCh triggered
2018-07-24 13:57:25 -07:00
Jeff Mitchell 954f6c4ece
Add config flag to disable non-printable character check (#4917) 2018-07-12 16:29:36 -04:00
Jeff Mitchell cd51a769ca Fix tests 2018-07-12 10:18:50 -04:00
Jeff Mitchell 98bf463a65 Make single-lease revocation behave like expiration (#4883)
This change makes it so that if a lease is revoked through user action,
we set the expiration time to now and update pending, just as we do with
tokens. This allows the normal retry logic to apply in these cases as
well, instead of just erroring out immediately. The idea being that once
you tell Vault to revoke something it should keep doing its darndest to
actually make that happen.
2018-07-11 15:45:35 -04:00
Jeff Mitchell 49f14403f5
503 if node is not active (#4874)
If we get to respondStandby but we're actually not in an HA cluster, we
should instead indicate the correct status to the user. Although it
might be better to change any such behavior upstream, if any upstream
code manages this state we should still handle it correctly.

Fixes #4873
2018-07-06 17:01:19 -04:00
Jeff Mitchell 4a3fe87a39
Allow max request size to be user-specified (#4824)
* Allow max request size to be user-specified

This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.

This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.

* Switch to ioutil.ReadAll()
2018-07-06 15:44:56 -04:00
Jeff Mitchell 5d44c54947
Changes the way policies are reported in audit logs (#4747)
* This changes the way policies are reported in audit logs.

Previously, only policies tied to tokens would be reported. This could
make it difficult to perform after-the-fact analysis based on both the
initial response entry and further requests. Now, the full set of
applicable policies from both the token and any derived policies from
Identity are reported.

To keep things consistent, token authentications now also return the
full set of policies in api.Secret.Auth responses, so this both makes it
easier for users to understand their actual full set, and it matches
what the audit logs now report.
2018-06-14 09:49:33 -04:00
Jeff Mitchell 5b05bfe593 Add a basic transit bench test 2018-06-11 17:02:33 -04:00
Jeff Mitchell 256a6b2b33 Fix test build 2018-06-09 18:18:00 -04:00
Jeff Mitchell d3cbde6ae2 Fix nil pointer 2018-06-09 16:57:57 -04:00
Jeff Mitchell 8916f6b625
Some atomic cleanup (#4732)
Taking inspiration from
https://github.com/golang/go/issues/17604#issuecomment-256384471
suggests that taking the address of a stack variable for use in atomics
works (at least, the race detector doesn't complain) but is doing it
wrong.

The only other change is a change in Leader() detecting if HA is enabled
to fast-path out. This value never changes after NewCore, so we don't
need to grab the read lock to check it.
2018-06-09 15:35:22 -04:00
Jeff Mitchell 575a606db7
Move TokenEntry into logical. (#4729)
This allows the HTTP logicalAuth handler to cache the value in the
logical.Request, avoiding a lookup later when performing acl
checks/counting a use.
2018-06-08 17:24:27 -04:00
Brian Kassouf 80580b82f3
UI: Trim trailing slashes to avoid redirects (#4668) 2018-05-31 12:52:43 -07:00
Jim Kalafut 2528a261bf
Add gzip compression to UI static content responses (#4664) 2018-05-31 09:42:08 -07:00
Jeff Mitchell be2c8d0665 Fix tests 2018-05-29 17:02:52 -04:00
Jeff Mitchell b7005ad62d Fix missing verification nonce field 2018-05-29 16:13:08 -04:00
Jeff Mitchell c53717ba1c Fix panic and update some text 2018-05-29 13:13:47 -04:00
Jeff Mitchell 2d05e072b9 Fix a null pointer and update status threshold 2018-05-29 12:04:30 -04:00
Jeff Mitchell 14b65ff4db
Builds on top of #4600 to provide CLI support (#4605) 2018-05-28 00:39:53 -04:00
Jeff Mitchell 7e7163f826 Factor out a bunch of shared code 2018-05-21 17:46:32 -04:00
Jeff Mitchell 462afbd0b9 Address review feedback 2018-05-21 14:47:00 -04:00