Commit graph

489 commits

Author SHA1 Message Date
Brian Kassouf bdfa2c7828
Add additional raft chunk test (#7192)
* Add an end-to-end raft chunk test

* Apply suggestions from code review

Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
2019-07-29 14:11:46 -07:00
Brian Kassouf b83aaf7331
storage/raft: Support storage migration to raft storage (#7207)
* Support raft in the migration command

* Add comments
2019-07-29 13:05:43 -07:00
Brian Kassouf a77995cdb1 Ensure raft configuration properly lists the leader (#7188) 2019-07-25 08:41:14 -04:00
Jeff Mitchell 0425db59ab
Raft chunk snapshotting (#7185)
Support chunking, including snapshot handling
2019-07-24 20:44:13 -04:00
Brian Kassouf 965066161a
Revert "Fix the config output (#7113)" (#7184)
This reverts commit 2f7cfc9aae911c8860db37e556363fbfb1567075.
2019-07-24 10:23:30 -07:00
Michel Vocks 524d101008 Added s3 storage path parameter (#7157) 2019-07-24 12:48:26 -04:00
ncabatoff 35b944460d
TestCassandraBackend times out pretty often on the ent build. Bump (#7130)
its timeout from 5s to 15s in the hopes that helps.  The theory is that
since I haven't seen this on the OSS side, it's failing because the ent
side is heavier in terms of test load and thus the tests face more
resource contention.
2019-07-23 13:26:36 -04:00
Jeff Mitchell fd376b4bdf Use ChunkingConfigurationStore for raft 2019-07-23 10:59:21 -04:00
Vishal Nayak 0010d79498 Fix the config output (#7113) 2019-07-22 12:59:46 -04:00
Jeff Mitchell 3b22ab2486 Add chunking support to raft 2019-07-22 12:17:58 -04:00
Christian Muehlhaeuser e6febc5839 Fixed a bunch of typos (#7146) 2019-07-18 21:10:15 -04:00
Jeff Mitchell 7b8c0b58f1
Call goimports as well as gofmt when doing a make fmt (#7148)
Closes #7147
2019-07-18 21:04:56 -04:00
Jeff Mitchell d810758ca2
Rerun proto gen as some got gen'd with old proto version (#7090) 2019-07-09 01:02:20 +02:00
Brian Kassouf 4d7d0d729a
storage/raft: When restoring a snapshot preseal first (#7011)
* storage/raft: When restoring a snapshot preseal first

* best-effort allow standbys to apply the restoreOp before sealing active node

* Don't cache the raft tls key

* Update physical/raft/raft.go

* Move pending raft peers to core

* Fix race on close bool

* Extend the leaderlease time for tests

* Update raft deps

* Fix audit hashing

* Fix race with auditing
2019-07-03 13:56:30 -07:00
Garrett T 8fc4a63796 Set MaxIdleConns to reduce connection churn (postgresql physical) (#6967)
* Set MaxIdleConns to reduce connection churn (postgresql physical)

* Make new  "max_idle_connection" config option for physical postgresql

* Add docs for "max_idle_connections" for postgresql storage

* Add minimum version to docs for max_idle_connections
2019-07-02 15:03:56 -07:00
Vishal Nayak 4484de3ea6
Fix raft config response (#6975) 2019-06-27 17:39:52 -04:00
Brian Kassouf 62e14c280d
storage/raft: fix races in tests (#6996)
* storage/raft: fix races in tests

* Fix another test race
2019-06-27 10:00:03 -07:00
Brian Kassouf 5d0c68ca74
Fix 32-bit builds (#6948) 2019-06-21 09:52:02 -06:00
Vishal Nayak 53035ce390
Raft CLI (#6893)
* raft cli

* Reuse the command's client

* Better response handling

* minor touchups
2019-06-20 21:32:00 -04:00
Jeff Mitchell 07dcdc8b79 Sync 2019-06-20 20:55:10 -04:00
Brian Kassouf ed14061578
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
ncabatoff 2cf5009ff5
Fix deadlock in TestPostgreSQLBackend (#6884)
Make lock2's retryInterval smaller so it grabs the lock as soon as lock1's renewer fails to renew in time.  Fix the logic to test if lock1's leader channel gets closed: we don't need a goroutine, and
the logic was broken in that if we timed out we'd never write to the blocking channel we then try to read from.  Moreover the timeout was wrong.
2019-06-14 12:59:24 -04:00
Mark Gritter d4d1ae55a0
Be more permissive in what URLs to allow; added test for unix socket. (#6859) 2019-06-10 13:59:13 -05:00
Justin Weissig ef17e7cbaa Update comment spelling in postgresql.go (#6817)
Fixed minor comment: PostgresSQL/PostgreSQL
2019-06-04 02:29:36 -05:00
Jim Kalafut 8a0d423ed8
Fix gofmt (#6764) 2019-05-20 15:15:05 -07:00
ncabatoff 64adec23cd
Since the two branches of the UNION produce disjoint sets, do a UNION ALL (#6546)
This obviates the need for a sort/eliminate duplicates step, giving a performance boost in some key distributions with large datasets.
2019-05-10 13:26:01 -04:00
bjorndolk e8f10814b6 Added HA backend for postgres based on dynamodb model (#5731)
Add optional HA support for postgres backend if Postgres version >= 9.5.
2019-05-10 12:48:42 -04:00
Mark Gritter 5d4fc08fc4
Determine scheme from configured address. (#6689)
* Determine scheme from configured address.

* Added test; use parsed version of address for TLS config.
2019-05-08 11:48:31 -05:00
Scott Stevenson f145baa581 Set chunkSize when creating a new GCS backend (#6655)
Adds a small step to TestBackend to prevent regression.
2019-05-03 13:08:29 -07:00
Jeff Mitchell e8a9d47aca
Port over some SP v2 bits (#6516)
* Port over some SP v2 bits

Specifically:

* Add too-large handling to Physical (Consul only for now)
* Contextify some identity funcs
* Update SP protos

* Add size limiting to inmem storage
2019-05-01 13:47:41 -04:00
EdwinRobbins de885660e8 DynamoDB: Make Unlock key delete conditional on being old leader's (#6637) 2019-04-30 17:44:47 -07:00
Jim Kalafut fa2accd000
Exit DynamoDB tryToLock when stop channel is closed (#6640)
* Exit DynamoDB tryToLock when stop channel is closed

If the stop channel is closed (e.g. an error is returned which triggers
close(stop) in Lock), this loop will spin and use 100% CPU.

* Ensure ticker is stopped
2019-04-29 22:51:48 -07:00
Becca Petrin 96374f27c7
Merge pull request #6356 from kedarkale27/master
Update mssql.go
2019-04-23 09:06:14 -07:00
ncabatoff 06574da57a
Merge multiple functions for creating consul containers into one. (#6612)
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
2019-04-22 12:26:10 -04:00
Kedar Kale 435d19bc25 Update mssql.go
Issue with GO Lang not able to execute the USE database line. Fixed the issue by using the database where it is called
2019-04-17 13:41:13 +05:30
Jeff Mitchell 24b92d6e29 Move physical/file to sdk 2019-04-15 14:51:33 -04:00
Jeff Mitchell be78ad2366 Remove dependency on Consul lib dir 2019-04-13 01:35:48 -04:00
Jeff Mitchell 80c303ac83 Move ldaputil and tlsutil over to sdk 2019-04-12 18:26:54 -04:00
Jeff Mitchell 371db36ede Move useragent to sdk 2019-04-12 18:17:49 -04:00
Jeff Mitchell 8bcb533a1b
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
Maarten van der Hoef d98b477a7e Spelling fix (#6523) 2019-04-03 06:57:32 -07:00
Jim Kalafut a939613f78 Reset retry count after a successful lock check (#6522) 2019-04-02 17:08:49 -07:00
Mahmoud Abdelsalam 6479a73f3a Fix dynamodb HA lock race (#6512)
* Fix DynamoDB HA race issue

* Add test for race condition (which fails on the released DynamoDB code)
2019-04-02 16:19:21 -07:00
Kedar Kale 601d839c77 Update mssql_test.go
adding schema env variable
2019-04-02 15:12:30 +05:30
Naoki Ainoya f5345bc711 use option.WithCredentialsFile() instead of deprecated method (#6500) 2019-03-29 10:58:26 -04:00
Kedar Kale b2c51de7ab Update mssql_test.go
added a new line at the end of the code
2019-03-22 12:34:49 +05:30
Kedar Kale b076f25a8e Update mssql_test.go
updated the test case, schema is now written in double quotes
2019-03-22 12:30:44 +05:30
T.K 453f1ac109 changed misspelled english words (#6432) 2019-03-19 09:32:45 -04:00
Lokesh Jain fd994cfceb
Added unit test case 2019-03-13 09:40:09 +05:30
Kedar Kale f930705ed2 Update mssql.go
query will run on the database passed as the parameter instead of the master database
2019-03-06 15:45:41 +05:30