luxolus/open-vault
2
0
Fork 0
Code Issues 1 Pull requests Releases Activity
10071 commits 1 branch 0 tags 214 MiB
Commit graph

444 commits

Author SHA1 Message Date
Jeff Mitchell 4f984569fa Plumb context through manta 2018-02-13 10:03:12 -05:00
Paul Stack 3c683dba92 Adding Manta Storage Backend (#3720) 
This PR adds a new Storage Backend for Triton's Object Storage - Manta

```
make testacc TEST=./physical/manta
==> Checking that code complies with gofmt requirements...
==> Checking that build is using go version >= 1.9.1...
go generate
VAULT_ACC=1 go test -tags='vault' ./physical/manta -v  -timeout 45m
=== RUN   TestMantaBackend
--- PASS: TestMantaBackend (61.18s)
PASS
ok  	github.com/hashicorp/vault/physical/manta	61.210s
```

Manta behaves differently to how S3 works - it has no such concepts of Buckets - it is merely a filesystem style object store

Therefore, we have chosen the approach of when writing a secret `foo` it will actually map (on disk) as foo/.vault_value

The reason for this is because if we write the secret `foo/bar` and then try and Delete a key using the name `foo` then Manta
will complain that the folder is not empty because `foo/bar` exists. Therefore, `foo/bar` is written as `foo/bar/.vault_value`

The value of the key is *always* written to a directory tree of the name and put in a `.vault_value` file.
 2018-02-12 18:22:41 -05:00
Jeff Mitchell ac382055d4
Validate Consul service name is RFC 1123 compliant (#3961) 2018-02-12 16:11:59 -05:00
Jeff Mitchell 844b7c395f Refactor fail logic in inmem 2018-02-12 11:25:48 -05:00
Jeff Mitchell 609648de4f Convert logical.InmemStorage to a wrapper around physical/inmem. 
The original reason for the split was physical's dependencies, but those
haven't been onerous for a long time. Meanwhile it's a totally separate
implementation so we could be getting faulty results from tests. Get rid
of it and use the unified physical/inmem.
 2018-02-12 11:16:16 -05:00
Jeff Mitchell 642b88c76a go vet fixes 2018-02-05 14:26:31 -05:00
Jeff Mitchell f33563f667 Some vet fixes 2018-02-04 20:37:57 -05:00
Xiang Li a120544b47 etcd: config etcd3 client's max response size (#3891) 2018-02-01 19:08:09 -05:00
Xiang Li 5fd85205cc etcd3: only create lock when lock is called (#3893) 2018-02-01 19:04:52 -05:00
Jeff Mitchell 60e2209532
Remove core restriction in cache and turn it into an active/standby restriction instead (#3849) 2018-01-25 22:21:51 -05:00
Brian Kassouf aa387bb4c2
Add compile tests to verify physical stores satisfy the correct interfaces (#3820) 2018-01-19 17:44:24 -08:00
Jeff Mitchell 395befc062 Update cache to satisfy Purge interface after context plumbing 2018-01-19 17:00:13 -05:00
Brian Kassouf 2f19de0305 Add context to storage backends and wire it through a lot of places (#3817) 2018-01-19 01:44:44 -05:00
Jon Davies 66e2593ef9 s3.go: Added options to use paths with S3 and the ability to disable SSL (#3730) 2018-01-03 12:11:00 -05:00
Antergone 312db6cc02 fix consul tls settings (#3719) 2017-12-19 14:24:21 -05:00
Antergone d68cc66954 check schema and table before create it (#3716) 2017-12-19 14:23:58 -05:00
Chris Hoffman f966d20225
Adding ability to cache core values, cache transaction improvements (#3640) 2017-12-06 12:25:17 -05:00
Jeff Mitchell 548629e8ef Port over some changes 2017-11-30 09:43:07 -05:00
Vlad Ungureanu 2ff547196f Remove unused recovery field in dynamodb backend (#3569) 2017-11-13 15:46:02 -05:00
Ben Higgins f78ab356d4 vault: recover from standby losing etcd lease (#3031) (#3511) 
This change makes these errors transient instead of permanent:

[ERROR] core: failed to acquire lock: error=etcdserver: requested lease not found

After this change, there can still be one of these errors when a
standby vault that lost its lease tries to become leader, but on the
next lock acquisition attempt a new session will be created. With this
new session, the standby will be able to become the leader.
 2017-11-03 13:38:16 -04:00
Jeff Mitchell 8a610e1e78 Move underscore tests to file from physical testing 2017-10-26 15:29:10 -04:00
Jeff Mitchell 28b0db38cc Revert couchdb changes 2017-10-26 15:27:20 -04:00
Jeff Mitchell 85500b5c3a Change prefix to a string that can be specified, rather than a bool 2017-10-26 15:26:28 -04:00
Jeff Mitchell 7e32ac15ec Add prefixing to couch to fix the error that was exposed 2017-10-26 15:26:28 -04:00
Jeff Mitchell 425b781fc8 Fix more tests 2017-10-26 15:26:28 -04:00
Jeff Mitchell 40a6635cd6 Fix testing 2017-10-26 15:26:28 -04:00
Jeff Mitchell e122853746 Add some more tests 2017-10-26 15:26:28 -04:00
Jeff Mitchell 3af3cf2b73 Allow underscores at the start of directories in file backend. 
Fixes #3476
 2017-10-26 15:26:28 -04:00
Jeff Mitchell 8e9317792d Fix some merge/update bugs 2017-10-23 16:49:46 -04:00
Jeff Mitchell c144f95be0 Sync over 2017-10-23 16:43:07 -04:00
Jeff Mitchell 1d852a7243 Use 0700 for directory permissions in file physical backend. 2017-10-12 14:24:34 -04:00
Ben Paxton 8ffc54cc1b Append trailing slash to folder listing in etcd3 backend (#3406) 2017-10-06 09:48:46 -04:00
Chris Hoffman 91338d7aa2 Adding latency injector option to -dev mode for storage operations (#3289) 2017-09-11 14:49:08 -04:00
Calvin Leung Huang c747caac2a Fix cassandra tests, explicitly set cluster port if provided (#3296) 
* Fix cassandra tests, explicitly set cluster port if provided

* Update cassandra.yml test-fixture

* Add port as part of the config option, fix tests

* Remove hostport splitting in cassandraConnectionProducer.createSession

* Include port in API docs
 2017-09-07 23:04:40 -04:00
Jeff Mitchell 44bf03e3b6 Fix compile after dep update 2017-09-05 18:18:34 -04:00
Jeff Mitchell 7585349e46 Use net.SplitHostPort on Consul address (#3268) 2017-08-31 12:31:34 -04:00
stephan stachurski e396d87bc5 add support to use application default credentials to gcs storage backend (#3257) 2017-08-30 15:42:02 -04:00
Christopher Pauley bd47ce523f update gcs backend tests- now strongly consistent (#3231) 2017-08-24 10:11:11 -04:00
Jeff Mitchell c864c0bad5 Return 500 if existence check fails, not 400 (#3173) 
Fixes #3162
 2017-08-15 16:44:16 -04:00
Chris Hoffman 6092538511 splitting cache into transactional and non-transactional cache structs (#3132) 2017-08-08 20:47:14 -04:00
Jeff Mitchell fdaaaadee2 Migrate physical backends into separate packages (#3106) 2017-08-03 13:24:27 -04:00
Jeff Mitchell 65d7face69 Merge branch 'master-oss' into issue-2241 2017-08-03 07:41:34 -04:00
Jeff Mitchell 4885b3e502 Use RemoteCredProvider instead of EC2RoleProvider (#2983) 2017-07-31 18:27:16 -04:00
Oliver Beattie 79058a3c95 Convert to dockertest.v3 2017-07-31 15:58:38 +01:00
Oliver Beattie 1cc1e7e615 Remove batching 2017-07-31 15:24:16 +01:00
Oliver Beattie 5046357e0f Fix Cassandra backend and tests 2017-07-31 15:24:16 +01:00
Filipe Varela a5a480551c Makes naming consistent w/ other storage backends (ie: etcd) 2017-07-31 15:18:07 +01:00
Filipe Varela 0177984e1b Fixes loading JSON pem bundles 2017-07-31 15:18:07 +01:00
Filipe Varela df388903e4 Fixes loading PEM bundles, JSON next 2017-07-31 15:18:06 +01:00
Filipe Varela b5144d833f Makes naming consistent with 'logical' 2017-07-31 15:18:05 +01:00
Filipe Varela cb08e543cb Use seconds for consistency with rest of project 2017-07-31 15:18:05 +01:00
Filipe Varela c6da462479 Adds support for TLS configuration 2017-07-31 15:18:04 +01:00
Filipe Varela 1c558c0c1d Adds support for authentication, protocol version and connection timeout 2017-07-31 15:18:04 +01:00
Filipe Varela 2abd4b6998 Make all operations share Session consistency setting 2017-07-31 15:18:03 +01:00
Oliver Beattie 2d04bfc447 Add dockertest support for Cassandra (it takes a while though ⏳) 2017-07-31 15:18:03 +01:00
Oliver Beattie 3919f38bd5 Add a (basic) Cassandra storage backend 2017-07-31 15:18:01 +01:00
Xiang Li d61a47a01c physical: format fixes (#3062) 2017-07-26 17:51:58 -04:00
Xiang Li 7c761b8414 physical: add default timeout for etcd3 requests (#3053) 2017-07-26 12:10:12 -04:00
Chris Hoffman 2aa02fb3f0 CockroachDB Physical Backend (#2713) 2017-07-23 08:54:33 -04:00
Jeff Mitchell 4387871bca Add max_parallel to mssql and postgresql (#3026) 
For storage backends, set max open connections to value of max_parallel.
 2017-07-17 13:04:49 -04:00
Lars Lehtonen 78edb1bc76 Fix swallowed error in physical package. (#2976) 2017-07-07 08:15:59 -04:00
Yann 27ca1c40c2 [physical][postgresql] concat→|| operator (#2945) 
Use `||` standard concatenation instead of the `concat` function in
order to use the `vault_kv_store` index on `parent_path`.
 2017-07-02 18:56:18 -04:00
Chris Hoffman c110f2188d Adding prefixed view of a physical backend (#2938) 2017-06-29 10:58:59 -04:00
Andri Mar Björgvinsson f0d103154e Better error messages using ListObjects than using HeadBucket. Might be a bigger request but messages are better than BadRequest, how this changes effect the messages are in the issue (#2892) 2017-06-20 01:16:41 +01:00
Raphael Randschau db4e1b4a99 CouchDB physical backend (#2880) 2017-06-17 11:22:10 -04:00
Jeff Mitchell 5d54aaf10a Fix azure test 2017-06-16 12:37:57 -04:00
Jeff Mitchell b6ea287ecb Change package in azure test 2017-06-16 12:18:16 -04:00
Jeff Mitchell f8f95524d0 Update Azure dep (#2881) 2017-06-16 12:06:09 -04:00
Dan Everton 32add0809e More efficient s3 paging (#2780) 2017-06-16 11:09:15 -04:00
Jeff Mitchell 3e7205c4c1 Add another nil guard to S3, follow on from #2785 2017-06-05 10:54:26 -04:00
Vishal Nayak c31b076360 Avoid panic in s3 list operation (#2785) 2017-06-05 10:53:20 -04:00
Mevan Samaratunga 731a7f187f fixed bug where the project name was not being read from configuration if it was provided via the "tenant" attribute. this was causing the swift client to crash with an EOF error. (#2803) 2017-06-05 10:48:39 -04:00
Eugene Bekker b55d972d24 Fixes #2789 (#2790) 2017-06-03 08:15:37 -04:00
Igor Katson 88118dce0f Add max_parallel parameter to MySQL backend. (#2760) 
* Add max_parallel parameter to MySQL backend.

This limits the number of concurrent connections, so that vault does not die
suddenly from "Too many connections".

This can happen when e.g. vault starts up, and tries to load all the
existing leases in parallel. At the time of writing this, the value
ExpirationRestoreWorkerCount in vault/helper/consts/const.go is set to
64, meaning that if there are enough leases in the vault's DB, it will
generate AT LEAST 64 concurrent connections to MySQL when loading the
data during start-up. On certain configurations, e.g. smaller AWS
RDS/Aurora instances, this will cause Vault to fail startup.

* Fix a typo in mysql storage readme
 2017-06-01 15:20:32 -07:00
Jeff Mitchell 9807f77bb8 Fix brokenness from Consul API updates 2017-05-24 11:10:59 -04:00
Michael Ansel 03dbe3f175 Ignore go-zookeeper lock children (#2724) 2017-05-22 13:23:28 -04:00
Paul Seiffert a8ec1466dc DynamoDB: Check for children more efficiently (#2722) 
* Check for children more efficiently

* Wrap comments to a width of 80
 2017-05-15 08:53:41 -07:00
Jeff Mitchell 26781471a6 Oops, fix tests again 2017-05-12 14:38:52 -04:00
Jeff Mitchell 680cc704d1 Fix tests 2017-05-12 14:12:53 -04:00
Jeff Mitchell 858deb9ca4 Don't allow parent references in file paths 2017-05-12 13:52:33 -04:00
Jeff Mitchell e98690d00c Ensure we aren't leaking any open FDs in the file backend if we hit certain error conditions 2017-05-09 09:24:43 -04:00
Chris Hoffman 847c86f788 Rename ParseDedupAndSortStrings to ParseDedupLowercaseAndSortStrings (#2614) 2017-04-19 10:39:07 -04:00
Jeff Mitchell 30af63c881 Fix azure test round 2 2017-04-17 14:52:52 -04:00
Jeff Mitchell 8cf0cd8cd2 Fix test for changed Azure 2017-04-17 13:18:34 -04:00
Jeff Mitchell e1e78b1409 Update to new Azure code after dep update (#2603) 2017-04-17 12:15:12 -04:00
Mevan Samaratunga 3b2c42f6dd Added "Domain" configuration parameter to Swift provider to enable V3 authentication (#2554) 2017-04-17 11:59:44 -04:00
Sebastian Haba 3322f637ac add mssql physical backend (#2546) 2017-04-06 09:33:49 -04:00
Jonathan Sokolowski a4ceaf0035 Etcd DNS discovery (#2521) 
* etcd: Add discovery_srv option
 2017-04-04 08:50:44 -07:00
VladV 1d4c901aeb Fix state change notification channels (#2548) 2017-03-31 09:01:55 -07:00
Vishal Nayak b9aa56c17e s3: use pooled transport for http client (#2481) 2017-03-29 10:27:27 -07:00
Marshall Brekka 1a73923a21 Etcd3: Write lock item with lease to ensure release on bad shutdown (#2526) 2017-03-28 11:08:41 -04:00
Dan Everton 4ef8ce1198 Add permitPool support to S3 (#2466) 2017-03-26 14:32:26 -04:00
Jeff Mitchell ff3c3db91b Have Consul's transaction handler use the permit pool 2017-03-09 12:59:42 -05:00
Jeff Mitchell 3d162b63cc Use locks in a slice rather than a map, which is faster and makes things cleaner (#2446) 2017-03-07 11:21:32 -05:00
Jeff Mitchell f5ffa229f4 Switch physical cache map index value to md5 from sha1 for all the performances 2017-03-06 13:11:14 -05:00
Jeff Mitchell 27399aeb7a Fix dynamo test that can double close a channel 2017-03-04 16:59:00 -05:00
Jeff Mitchell 111fbc5747 Make cache not actually cache values under core/ (#2439) 2017-03-03 16:04:31 -05:00
Marshall Brekka 184b47e20c Add a TTL to the dynamodb lock implementation. (#2141) 2017-02-27 14:30:34 -05:00
Jeff Mitchell 2cc0906b33 Fix breakage for HTTP2 support due to changes in wrapping introduced in 1.8 (#2412) 2017-02-27 12:49:35 -05:00
Jeff Mitchell 41ae5d14ce Add pseudo transactional test 2017-02-20 11:40:36 -05:00
Jeff Mitchell 4305900a64 Add faultPseudo for testing 2017-02-20 11:08:03 -05:00
Jeff Mitchell 3230f697bd Final rep porting (#2392) 2017-02-17 09:15:35 -05:00
Colin Arenz 99b01a3d82 Fix listing of deep paths in PostgreSQL backend (#2393) 
This change addresses an issue where deep paths would not be enumerated if parent paths did not contain a key.

Given the keys `shallow` and `deep` at the following paths...
```
secret/shallow
secret/path/deep
```

... a `LIST` request against `/v1/secret` would produce only one result, `shallow`.  With this change, the same list request will now list `shallow` and `path/`.
 2017-02-17 09:14:11 -05:00
Brian Kassouf 13ec9c5dbf Load leases into the expiration manager in parallel (#2370) 
* Add a benchmark for exiration.Restore

* Add benchmarks for consul Restore functions

* Add a parallel version of expiration.Restore

* remove debug code

* Up the MaxIdleConnsPerHost

* Add tests for etcd

* Return errors and ensure go routines are exited

* Refactor inmem benchmark

* Add s3 bench and refactor a bit

* Few tweaks

* Fix race with waitgroup.Add()

* Fix waitgroup race condition

* Move wait above the info log

* Add helper/consts package to store consts that are needed in cyclic packages

* Remove not used benchmarks
 2017-02-16 10:16:06 -08:00
Xiang Li 220930f539 etcdbackend: support version auto discovery (#2299) 2017-01-26 17:19:13 -05:00
Jeff Mitchell f856963706 Revert file backend base64ing, as we need to fix a pathological case for some keys 2017-01-25 12:27:18 -05:00
Brian Kassouf d6198b7e24 change consistency config value from a bool to a string (#2282) 2017-01-19 17:36:33 -05:00
Vishal Nayak bb1f28ce66 Merge pull request #2203 from hashicorp/file-backend-base64 
Base64 encode the file names in the 'file' physical backend
 2017-01-19 10:10:57 -05:00
Brian Kassouf 6aa097b727 Add require consistent flag to Consul Lock 2017-01-13 12:22:14 -08:00
Brian Kassouf ce6fa6b30e Add test for require_consistency option 2017-01-13 10:24:40 -08:00
Brian Kassouf fb19c81f62 add a option for strong consistancy for consul 2017-01-13 09:49:04 -08:00
vishalnayak 5aeb276018 correcting the error statement 2017-01-13 03:58:46 -05:00
vishalnayak 76a456cc97 file: correct the old entry check 2017-01-13 03:51:09 -05:00
vishalnayak 8b579d47a9 address review feedback 2017-01-13 03:39:33 -05:00
vishalnayak d2026364c7 physical/file: added test for base64 encoding the storage file names 2017-01-13 01:00:25 -05:00
vishalnayak cbccf9869d physical/file: Handle file duplication case while updating 2017-01-13 01:00:25 -05:00
vishalnayak 17652b486d physical/file: Fix the deletion flow 2017-01-13 01:00:25 -05:00
vishalnayak a952d324fe physical: file backend to have key base64 URL encoded 2017-01-13 01:00:25 -05:00
Adam Shannon 6fc53dc135 physical/zk: Ignore ErrNoNode when deleting znodes (#2256) 2017-01-11 09:42:30 -05:00
Jeff Mitchell 80dc5819d3 Use dockertest.v2 (#2247) 
New dockertest has a totally different API and will require some serious
refactoring. This will tide over until then by pinning the API version.
 2017-01-09 13:46:54 -05:00
Jeff Mitchell 9e5d1eaac9 Port some updates 2017-01-06 15:42:18 -05:00
Xiang Li 2faa3f5764 etcd3: remove wrong keys checking for prefix request (#2231) 2017-01-05 07:48:46 -05:00
Xiang Li 02070e0fc6 physical: add etcd3 backend (#2168) 2017-01-03 14:43:46 -05:00
Dan Everton e4a1f5a3bb Page results from S3. (#2224) 
S3 results require paging to ensure that all results are returned. This
PR changes the S3 physical backend to use the new ListObjectV2 method
and pages through all the results.

Fixes #2223.
 2017-01-03 11:15:48 -05:00
Jeff Mitchell f9c6fc2e6b Actually give the logger to inmem backend 2016-12-15 15:48:51 -05:00
Christopher Pauley f07a19c503 gcs physical backend (#2099) 2016-12-01 11:42:31 -08:00
Jeff Mitchell 736a4b111c Add some commenting to PermitPool 2016-11-28 18:34:58 -05:00
Kevin Goess 33bf26f320 check for failure on that mysql query (#2105) 2016-11-17 09:59:27 -05:00
Jeff Mitchell 9066f012a7 Fix cache default size and docs 2016-11-01 10:24:35 -04:00
Jeff Mitchell 9d5462ca04 Don't cache physical responses when thre was an error (#2040) 2016-10-28 12:55:56 -04:00
Paweł Rozlach 33b4683dfd Post-review fixes for file/zk recursive empty prefix delete 2016-10-05 08:08:00 -04:00
Pawel Rozlach 41ade15f73 Fix file backend so that it properly removes nested secrets. 
This patch makes file backend properly remove nested secrets, without leaving
empty directory artifacts, no matter how nested directories were.
 2016-10-04 21:56:12 +02:00
Pawel Rozlach 44b4704cfa Fix zookeeper backend so that properly deletes/lists secrets. 
This patch fixes two bugs in Zookeeper backends:
 * backend was determining if the node is a leaf or not basing on the number
   of the childer given node has. This is incorrect if you consider the fact
   that deleteing nested node can leave empty prefixes/dirs behind which have
   neither children nor data inside. The fix changes this situation by testing
   if the node has any data set - if not then it is not a leaf.
 * zookeeper does not delete nodes that do not have childern just like consul
   does and this leads to leaving empty nodes behind. In order to fix it, we
   scan the logical path of a secret being deleted for empty dirs/prefixes and
   remove them up until first non-empty one.
 2016-10-04 21:56:12 +02:00
Pawel Rozlach 68fc52958d Add tests for nested/prefixed secrets removal. 
Current tests were not checking if backends are properly removing
nested secrets. We follow here the behaviour of Consul backend, where
empty "directories/prefixes" are automatically removed by Consul itself.
 2016-10-04 21:55:33 +02:00
Jeff Mitchell 226ef5d78c Make HA in etcd off by default. (#1909) 
Fixes #1908

(Doesn't really "fix" it but someone from the community needs to step up
if they want to see this fixed.)
 2016-09-21 14:01:36 -04:00
James Pearson Hughes f598c78d98 DynamoDB: fix log typo (#1891) 2016-09-14 15:16:24 -04:00
Jeff Mitchell ffaaacd029 Have file backend remove empty dirs. (#1821) 
Add tests to check that prefixes are being properly removed (or at
least, not listed) from backends.
 2016-08-31 14:12:28 -04:00
Jeff Mitchell 2ce4397deb Plumb through the ability to set the storage read cache size. (#1784) 
Plumb through the ability to set the storage read cache size.

Fixes #1772
 2016-08-26 10:27:06 -04:00
Jeff Mitchell 58b32e5432 Convert to logxi 2016-08-21 18:13:37 -04:00
Jeff Mitchell 2860dcc60f gofmt 2016-08-19 16:48:32 -04:00
Jeff Mitchell 734e80ca56 Add permit pool to dynamodb 2016-08-15 19:45:06 -04:00
Jeff Mitchell dcba6129e3 Use dockertest for physical consul tests, and always run them 2016-08-15 16:20:32 -04:00
Jeff Mitchell 37320f8798 Request forwarding (#1721) 
Add request forwarding.
 2016-08-15 09:42:42 -04:00
Jeff Mitchell 9e204bd88c Add arbitrary string slice parsing. 
Like the KV function, this supports either separated strings or JSON
strings, base64-encoded or not.

Fixes #1619 in theory.
 2016-08-03 14:24:16 -04:00
Jeff Mitchell c025b292b5 Cleanup 2016-08-03 13:09:12 -04:00
vishalnayak 91e60a5824 Fixed the test after removing shutdown bool 2016-08-01 12:20:38 -04:00
Jeff Mitchell 6ffefb649d Close the shutdown channel instead of sending a value down 2016-08-01 11:58:45 -04:00
vishalnayak 05b8ce8348 Address review feedback 2016-08-01 11:15:25 -04:00
vishalnayak 21d155f4af Fix physical/consul test case 2016-08-01 10:55:47 -04:00
vishalnayak 5ed10f4074 Make the defer statement of waitgroup to execute last 2016-08-01 10:24:27 -04:00