Commit graph

475 commits

Author SHA1 Message Date
Garrett T 8fc4a63796 Set MaxIdleConns to reduce connection churn (postgresql physical) (#6967)
* Set MaxIdleConns to reduce connection churn (postgresql physical)

* Make new  "max_idle_connection" config option for physical postgresql

* Add docs for "max_idle_connections" for postgresql storage

* Add minimum version to docs for max_idle_connections
2019-07-02 15:03:56 -07:00
Vishal Nayak 4484de3ea6
Fix raft config response (#6975) 2019-06-27 17:39:52 -04:00
Brian Kassouf 62e14c280d
storage/raft: fix races in tests (#6996)
* storage/raft: fix races in tests

* Fix another test race
2019-06-27 10:00:03 -07:00
Brian Kassouf 5d0c68ca74
Fix 32-bit builds (#6948) 2019-06-21 09:52:02 -06:00
Vishal Nayak 53035ce390
Raft CLI (#6893)
* raft cli

* Reuse the command's client

* Better response handling

* minor touchups
2019-06-20 21:32:00 -04:00
Jeff Mitchell 07dcdc8b79 Sync 2019-06-20 20:55:10 -04:00
Brian Kassouf ed14061578
Raft Storage Backend (#6888)
* Work on raft backend

* Add logstore locally

* Add encryptor and unsealable interfaces

* Add clustering support to raft

* Remove client and handler

* Bootstrap raft on init

* Cleanup raft logic a bit

* More raft work

* Work on TLS config

* More work on bootstrapping

* Fix build

* More work on bootstrapping

* More bootstrapping work

* fix build

* Remove consul dep

* Fix build

* merged oss/master into raft-storage

* Work on bootstrapping

* Get bootstrapping to work

* Clean up FMS and node-id

* Update local node ID logic

* Cleanup node-id change

* Work on snapshotting

* Raft: Add remove peer API (#906)

* Add remove peer API

* Add some comments

* Fix existing snapshotting (#909)

* Raft get peers API (#912)

* Read raft configuration

* address review feedback

* Use the Leadership Transfer API to step-down the active node (#918)

* Raft join and unseal using Shamir keys (#917)

* Raft join using shamir

* Store AEAD instead of master key

* Split the raft join process to answer the challenge after a successful unseal

* get the follower to standby state

* Make unseal work

* minor changes

* Some input checks

* reuse the shamir seal access instead of new default seal access

* refactor joinRaftSendAnswer function

* Synchronously send answer in auto-unseal case

* Address review feedback

* Raft snapshots (#910)

* Fix existing snapshotting

* implement the noop snapshotting

* Add comments and switch log libraries

* add some snapshot tests

* add snapshot test file

* add TODO

* More work on raft snapshotting

* progress on the ConfigStore strategy

* Don't use two buckets

* Update the snapshot store logic to hide the file logic

* Add more backend tests

* Cleanup code a bit

* [WIP] Raft recovery (#938)

* Add recovery functionality

* remove fmt.Printfs

* Fix a few fsm bugs

* Add max size value for raft backend (#942)

* Add max size value for raft backend

* Include physical.ErrValueTooLarge in the message

* Raft snapshot Take/Restore API  (#926)

* Inital work on raft snapshot APIs

* Always redirect snapshot install/download requests

* More work on the snapshot APIs

* Cleanup code a bit

* On restore handle special cases

* Use the seal to encrypt the sha sum file

* Add sealer mechanism and fix some bugs

* Call restore while state lock is held

* Send restore cb trigger through raft log

* Make error messages nicer

* Add test helpers

* Add snapshot test

* Add shamir unseal test

* Add more raft snapshot API tests

* Fix locking

* Change working to initalize

* Add underlying raw object to test cluster core

* Move leaderUUID to core

* Add raft TLS rotation logic (#950)

* Add TLS rotation logic

* Cleanup logic a bit

* Add/Remove from follower state on add/remove peer

* add comments

* Update more comments

* Update request_forwarding_service.proto

* Make sure we populate all nodes in the followerstate obj

* Update times

* Apply review feedback

* Add more raft config setting (#947)

* Add performance config setting

* Add more config options and fix tests

* Test Raft Recovery (#944)

* Test raft recovery

* Leave out a node during recovery

* remove unused struct

* Update physical/raft/snapshot_test.go

* Update physical/raft/snapshot_test.go

* fix vendoring

* Switch to new raft interface

* Remove unused files

* Switch a gogo -> proto instance

* Remove unneeded vault dep in go.sum

* Update helper/testhelpers/testhelpers.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* Update vault/cluster/cluster.go

* track active key within the keyring itself (#6915)

* track active key within the keyring itself

* lookup and store using the active key ID

* update docstring

* minor refactor

* Small text fixes (#6912)

* Update physical/raft/raft.go

Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>

* review feedback

* Move raft logical system into separate file

* Update help text a bit

* Enforce cluster addr is set and use it for raft bootstrapping

* Fix tests

* fix http test panic

* Pull in latest raft-snapshot library

* Add comment
2019-06-20 12:14:58 -07:00
ncabatoff 2cf5009ff5
Fix deadlock in TestPostgreSQLBackend (#6884)
Make lock2's retryInterval smaller so it grabs the lock as soon as lock1's renewer fails to renew in time.  Fix the logic to test if lock1's leader channel gets closed: we don't need a goroutine, and
the logic was broken in that if we timed out we'd never write to the blocking channel we then try to read from.  Moreover the timeout was wrong.
2019-06-14 12:59:24 -04:00
Mark Gritter d4d1ae55a0
Be more permissive in what URLs to allow; added test for unix socket. (#6859) 2019-06-10 13:59:13 -05:00
Justin Weissig ef17e7cbaa Update comment spelling in postgresql.go (#6817)
Fixed minor comment: PostgresSQL/PostgreSQL
2019-06-04 02:29:36 -05:00
Jim Kalafut 8a0d423ed8
Fix gofmt (#6764) 2019-05-20 15:15:05 -07:00
ncabatoff 64adec23cd
Since the two branches of the UNION produce disjoint sets, do a UNION ALL (#6546)
This obviates the need for a sort/eliminate duplicates step, giving a performance boost in some key distributions with large datasets.
2019-05-10 13:26:01 -04:00
bjorndolk e8f10814b6 Added HA backend for postgres based on dynamodb model (#5731)
Add optional HA support for postgres backend if Postgres version >= 9.5.
2019-05-10 12:48:42 -04:00
Mark Gritter 5d4fc08fc4
Determine scheme from configured address. (#6689)
* Determine scheme from configured address.

* Added test; use parsed version of address for TLS config.
2019-05-08 11:48:31 -05:00
Scott Stevenson f145baa581 Set chunkSize when creating a new GCS backend (#6655)
Adds a small step to TestBackend to prevent regression.
2019-05-03 13:08:29 -07:00
Jeff Mitchell e8a9d47aca
Port over some SP v2 bits (#6516)
* Port over some SP v2 bits

Specifically:

* Add too-large handling to Physical (Consul only for now)
* Contextify some identity funcs
* Update SP protos

* Add size limiting to inmem storage
2019-05-01 13:47:41 -04:00
EdwinRobbins de885660e8 DynamoDB: Make Unlock key delete conditional on being old leader's (#6637) 2019-04-30 17:44:47 -07:00
Jim Kalafut fa2accd000
Exit DynamoDB tryToLock when stop channel is closed (#6640)
* Exit DynamoDB tryToLock when stop channel is closed

If the stop channel is closed (e.g. an error is returned which triggers
close(stop) in Lock), this loop will spin and use 100% CPU.

* Ensure ticker is stopped
2019-04-29 22:51:48 -07:00
Becca Petrin 96374f27c7
Merge pull request #6356 from kedarkale27/master
Update mssql.go
2019-04-23 09:06:14 -07:00
ncabatoff 06574da57a
Merge multiple functions for creating consul containers into one. (#6612)
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
2019-04-22 12:26:10 -04:00
Kedar Kale 435d19bc25 Update mssql.go
Issue with GO Lang not able to execute the USE database line. Fixed the issue by using the database where it is called
2019-04-17 13:41:13 +05:30
Jeff Mitchell 24b92d6e29 Move physical/file to sdk 2019-04-15 14:51:33 -04:00
Jeff Mitchell be78ad2366 Remove dependency on Consul lib dir 2019-04-13 01:35:48 -04:00
Jeff Mitchell 80c303ac83 Move ldaputil and tlsutil over to sdk 2019-04-12 18:26:54 -04:00
Jeff Mitchell 371db36ede Move useragent to sdk 2019-04-12 18:17:49 -04:00
Jeff Mitchell 8bcb533a1b
Create sdk/ and api/ submodules (#6583) 2019-04-12 17:54:35 -04:00
Maarten van der Hoef d98b477a7e Spelling fix (#6523) 2019-04-03 06:57:32 -07:00
Jim Kalafut a939613f78 Reset retry count after a successful lock check (#6522) 2019-04-02 17:08:49 -07:00
Mahmoud Abdelsalam 6479a73f3a Fix dynamodb HA lock race (#6512)
* Fix DynamoDB HA race issue

* Add test for race condition (which fails on the released DynamoDB code)
2019-04-02 16:19:21 -07:00
Kedar Kale 601d839c77 Update mssql_test.go
adding schema env variable
2019-04-02 15:12:30 +05:30
Naoki Ainoya f5345bc711 use option.WithCredentialsFile() instead of deprecated method (#6500) 2019-03-29 10:58:26 -04:00
Kedar Kale b2c51de7ab Update mssql_test.go
added a new line at the end of the code
2019-03-22 12:34:49 +05:30
Kedar Kale b076f25a8e Update mssql_test.go
updated the test case, schema is now written in double quotes
2019-03-22 12:30:44 +05:30
T.K 453f1ac109 changed misspelled english words (#6432) 2019-03-19 09:32:45 -04:00
Lokesh Jain fd994cfceb
Added unit test case 2019-03-13 09:40:09 +05:30
Kedar Kale f930705ed2 Update mssql.go
query will run on the database passed as the parameter instead of the master database
2019-03-06 15:45:41 +05:30
Jeff Mitchell 718ae5a010 Minor syncs 2019-03-04 13:35:22 -05:00
Clint Shryock 7e1a4885e1
fix typo in code comment 2019-02-28 10:26:40 -06:00
Andrey Kuzmin b496fea4ad Etcd timeouts (#6285)
* Configurable lock and request etcd timeouts.

If etcd cluster placed on slow servers - request timeouts may be much greater, then hardcoded default values.
Also, in etcd setup, like above - may be need to greater lock timeout.

* Configurable lock and request etcd timeouts.

Docs.

* Use user friendly timeout syntax.

To allow specify more readable time values.
2019-02-27 18:34:29 -08:00
Jeff Mitchell 82a85aa8c8 Make fmt 2019-02-08 09:12:55 -05:00
Lexman c51de76dd1
adds port parameter (#6145) 2019-02-05 15:40:00 -08:00
Jeff Mitchell 5f249d4005
Add allowed_response_headers (#6115) 2019-02-05 16:02:15 -05:00
Jeff Mitchell bbc1d53a5d Revert "Refactor common token fields and operations into a helper (#5953)"
This reverts commit 66c226c593bb1cd48cfd8364ac8510cb42b7d67a.
2019-02-01 11:23:40 -05:00
Jeff Mitchell 85a560abba
Refactor common token fields and operations into a helper (#5953) 2019-01-30 16:23:28 -05:00
Gordon Shankman cd2f7bbde8 Adding support for SSE in the S3 storage backend. (#5996) 2019-01-26 16:48:08 -05:00
Jeff Mitchell 94e56d964f Fix build 2019-01-23 16:52:51 -05:00
Jeff Mitchell 3f1a7d4fdd
Update to latest etcd and use the new repository packages (#6087)
This will be necessary for go mod work

Additionally, the srv api has changed. This adapts to it.
2019-01-23 14:35:03 -05:00
Becca Petrin aac271ed7f swap the forked aliyun sdk for the original (#6024) 2019-01-23 11:24:51 -05:00
Jim Kalafut f5b19f9ce8
Fix physical/postgresql to return the full entry key (#6044)
Fixes #6041
2019-01-15 12:06:01 -08:00
Jim Kalafut d0e2badbae Run goimports across the repository (#6010)
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
2019-01-08 16:48:57 -08:00