Commit Graph

53 Commits

Author SHA1 Message Date
Brian Kassouf ed14061578
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
Brian Kassouf 43783a5dca
Move cluster logic out of vault package (#6601)
* Move cluster logic out of vault package

* Dedup heartbeat and fix tests

* Fix test
2019-04-17 13:50:31 -07:00
Jeff Mitchell 8bcb533a1b
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
ncabatoff b08963779c Apply the bits from ent pr 852 that are relevant to OSS. (#6528) 2019-04-04 13:02:44 -04:00
Brian Kassouf ab1a04f8e6
Port over some test fixes (#6261) 2019-02-19 12:03:02 -08:00
Brian Kassouf f5b5fbb392
Refactor the cluster listener (#6232)
* Port over OSS cluster port refactor components

* Start forwarding

* Cleanup a bit

* Fix copy error

* Return error from perf standby creation

* Add some more comments

* Fix copy/paste error
2019-02-14 18:14:56 -08:00
Jeff Mitchell 9ef0680e7f
Fix leader info repopulation (#6167)
* Two things:

* Change how we populate and clear leader UUID. This fixes a case where
if a standby disconnects from an active node and reconnects, without the
active node restarting, the UUID doesn't change so triggers on a new
active node don't get run.

* Add a bunch of test helpers and minor updates to things.
2019-02-05 21:01:18 -05:00
Jeff Mitchell 27c960d8df
Split SubView functionality into logical.StorageView (#6141)
This lets other parts of Vault that can't depend on the vault package
take advantage of the subview functionality.

This also allows getting rid of BarrierStorage and vault.Entry, two
totally redundant abstractions.
2019-01-31 09:25:18 -05:00
Jim Kalafut d0e2badbae Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00
Jeff Mitchell 919b968c27
The big one (#5346) 2018-09-17 23:03:00 -04:00
Becca Petrin 7a8c116fb1
undo make fmt (#5265) 2018-09-04 09:29:18 -07:00
Becca Petrin ed7639b0ec
run make fmt (#5261) 2018-09-04 09:12:59 -07:00
Calvin Leung Huang 9988ace85e gofmt files (#5233) 2018-08-31 09:15:40 -07:00
Vishal Nayak 28e3eb9e2c
Errwrap everywhere (#4252)
* package api

* package builtin/credential

* package builtin/logical

* package command

* package helper

* package http and logical

* package physical

* package shamir

* package vault

* package vault

* address feedback

* more fixes
2018-04-05 11:49:21 -04:00
Becca Petrin 03cf302e9a Move to "github.com/hashicorp/go-hclog" (#4227)
* logbridge with hclog and identical output

* Initial search & replace

This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.

* strip logbridge

* fix majority of tests

* update logxi aliases

* WIP fixing tests

* more test fixes

* Update test to hclog

* Fix format

* Rename hclog -> log

* WIP making hclog and logxi love each other

* update logger_test.go

* clean up merged comments

* Replace RawLogger interface with a Logger

* Add some logger names

* Replace Trace with Debug

* update builtin logical logging patterns

* Fix build errors

* More log updates

* update log approach in command and builtin

* More log updates

* update helper, http, and logical directories

* Update loggers

* Log updates

* Update logging

* Update logging

* Update logging

* Update logging

* update logging in physical

* prefixing and lowercase

* Update logging

* Move phyisical logging name to server command

* Fix som tests

* address jims feedback so far

* incorporate brians feedback so far

* strip comments

* move vault.go to logging package

* update Debug to Trace

* Update go-plugin deps

* Update logging based on review comments

* Updates from review

* Unvendor logxi

* Remove null_logger.go
2018-04-02 17:46:59 -07:00
Jeff Mitchell d4a431b298
Move local cluster parameters to atomic values to fix some potential data races (#4036) 2018-02-23 14:47:07 -05:00
Brian Kassouf cb08fb92d2 Port some replicated cluster changes from ent (#4037) 2018-02-23 14:01:15 -05:00
Jeff Mitchell 05723e1229 Revert cluster.go change 2018-02-04 20:42:45 -05:00
Jeff Mitchell f33563f667 Some vet fixes 2018-02-04 20:37:57 -05:00
Jeff Mitchell 69aead14f0 A bit more context plumbing 2018-01-19 04:11:59 -05:00
Brian Kassouf 2f19de0305 Add context to storage backends and wire it through a lot of places (#3817) 2018-01-19 01:44:44 -05:00
Jeff Mitchell 03cd40345f Properly format autogenerated clusteraddr 2017-10-25 14:43:05 -04:00
Jeff Mitchell cf1cf8eee4 If no clusterAddr is given but we have a single synthesized cluster addresses, automatically use it. (#3486) 2017-10-24 16:58:53 -04:00
Jeff Mitchell 3edb337a00 Add option to set cluster TLS cipher suites. (#3228)
* Add option to set cluster TLS cipher suites.

Fixes #3227
2017-08-30 16:28:23 -04:00
Jeff Mitchell 9d4801b1e8 Revert grpc back a version (they introduced a panic) and clean up a bunch of old request forwarding stuff 2017-05-24 10:38:48 -04:00
Jeff Mitchell 0d4e7fba69 Remove non-gRPC request forwarding 2017-05-24 09:34:59 -04:00
Jeff Mitchell bb05f2d8f8 Fix double-lock 2017-03-02 10:54:31 -05:00
Jeff Mitchell 36c84df326 Large update to request forwarding handling. (#2426) 2017-03-02 10:03:49 -05:00
Jeff Mitchell 90389323a2 Some more forwarding client cleanup 2017-03-01 20:59:20 -05:00
Jeff Mitchell b1c2a930fe Clean up request forwarding logic 2017-03-01 18:17:06 -05:00
Jeff Mitchell 494b4c844b More porting from rep (#2389)
* More porting from rep

* Address feedback
2017-02-16 20:13:19 -05:00
Jeff Mitchell 67410ab230 Make TLS 1.2 *explicitly* required for cluster communications 2017-01-31 13:30:25 -05:00
Jeff Mitchell 9e5d1eaac9 Port some updates 2017-01-06 15:42:18 -05:00
Jeff Mitchell 64fc18e523 When a JWT wrapping token is returned, audit the inner token both for
request and response. This makes it far easier to properly check
validity elsewhere in Vault because we simply replace the request client
token with the inner value.
2017-01-04 23:50:24 -05:00
Jeff Mitchell 3129187dc2 JWT wrapping tokens (#2172) 2017-01-04 16:44:03 -05:00
Vishal Nayak e3f56f375c Add 'no-store' response header from all the API outlets (#2183) 2016-12-15 17:53:07 -05:00
Jeff Mitchell e84a015487 Add extra logic around listener handling. (#2089) 2016-11-11 16:43:33 -05:00
Vishal Nayak e06aaf20e1 Remove unused WrapListenersForClustering (#2007) 2016-10-18 10:20:09 -04:00
Jeff Mitchell 58b32e5432 Convert to logxi 2016-08-21 18:13:37 -04:00
Jeff Mitchell 2bb8adcbde Cleanup and avoid unnecessary advertisement parsing in leader check 2016-08-19 14:49:11 -04:00
Jeff Mitchell b7acf5b5ab Rename proto service stuff and change log levels for some messages 2016-08-19 11:49:25 -04:00
Jeff Mitchell bdcfe05517 Clustering enhancements (#1747) 2016-08-19 11:03:53 -04:00
Jeff Mitchell 02d9702fbd Add local into handler path for forwarded requests 2016-08-16 11:46:37 -04:00
Jeff Mitchell 37320f8798 Request forwarding (#1721)
Add request forwarding.
2016-08-15 09:42:42 -04:00
vishalnayak 3895ea4c2b Address review feedback from @jefferai 2016-08-10 15:22:12 -04:00
vishalnayak 95f9c62523 Fix Cluster object being returned as nil when unsealed 2016-08-10 15:09:16 -04:00
Jeff Mitchell 4246ab1220 Change local cluster info path 2016-08-09 11:28:42 -04:00
Jeff Mitchell cc10fd7a7e Use config file cluster name after automatic gen 2016-08-09 11:03:50 -04:00
Jeff Mitchell a7ed50dbc8 coreClusterPath -> coreLocalClusterPath 2016-08-03 09:50:21 -04:00
vishalnayak 669bbdfa48 Address review feedback from @jefferai 2016-07-26 14:05:27 -04:00