Commit graph

422 commits

Author SHA1 Message Date
Martin Hrabovcin 6bfceb7f28 physical/zookeeper: Re-try to release lock in case of failure (#4569) 2018-05-17 15:52:50 -07:00
Matthew Surabian 4e0bc43bf8 DynamoDB Nested Values Bug (#4570)
* Add tests to ExerciseBackend to expose nested-values bug

* Update DynamoDB physical backend Delete and hasChildren logic to prevent overzealous cleanup of folders and values
2018-05-16 13:30:56 -04:00
Brian Kassouf 55997b6bf0
physical/cache: Add a list of prefixes to not cache (#4515)
* physical/cache: Add a list of prefixes to not cache

* Rename the pathmanager

* Move cache back to the beggining of postUnseal

* Fix comment
2018-05-10 10:29:26 -07:00
Matthew Surabian 01d63b8148 DynamoDB Deprecation Fixes (#4534)
* Use the AWS SDK's UnmarshalMap method for dynamodb backend, not the deprecated ConvertFromMap method

* Use the AWS SDK's MarshalMap method for dynamodb backend, not the deprecated ConvertToMap method

* Use the AWS SDK's session.NewSession method for dynamodb backend, not the deprecated session.New method

* Fix variable name awserr that colides with imported package in dynamodb backend
2018-05-10 08:25:51 -04:00
Jeff Mitchell c0ed57feae
Revert "proto changes (#4503)" (#4504)
This reverts commit 14594bd76e04ff09c442738800be5fdebc45512f.
2018-05-03 15:38:53 -04:00
Vishal Nayak 7549ea0d12
proto changes (#4503) 2018-05-03 15:23:14 -04:00
Becca Petrin d51acbde68
New proto version (#4501) 2018-05-03 10:19:39 -07:00
Calvin Leung Huang 7d214d2a3a
Purge opened connections on retries during tests (#4452) 2018-04-26 11:28:58 -04:00
Vishal Nayak 5fa9e4ca5c
phys/consul: Allow tuning of session ttl and lock wait time (#4352)
* phys/consul: allow tuning of session ttl and lock wait time

* use parseutil

* udpate docs
2018-04-18 13:09:55 -04:00
Becca Petrin abb621752f Clean up error string formatting (#4304) 2018-04-09 14:35:21 -04:00
Vishal Nayak 28e3eb9e2c
Errwrap everywhere (#4252)
* package api

* package builtin/credential

* package builtin/logical

* package command

* package helper

* package http and logical

* package physical

* package shamir

* package vault

* package vault

* address feedback

* more fixes
2018-04-05 11:49:21 -04:00
Becca Petrin 03cf302e9a Move to "github.com/hashicorp/go-hclog" (#4227)
* logbridge with hclog and identical output

* Initial search & replace

This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.

* strip logbridge

* fix majority of tests

* update logxi aliases

* WIP fixing tests

* more test fixes

* Update test to hclog

* Fix format

* Rename hclog -> log

* WIP making hclog and logxi love each other

* update logger_test.go

* clean up merged comments

* Replace RawLogger interface with a Logger

* Add some logger names

* Replace Trace with Debug

* update builtin logical logging patterns

* Fix build errors

* More log updates

* update log approach in command and builtin

* More log updates

* update helper, http, and logical directories

* Update loggers

* Log updates

* Update logging

* Update logging

* Update logging

* Update logging

* update logging in physical

* prefixing and lowercase

* Update logging

* Move phyisical logging name to server command

* Fix som tests

* address jims feedback so far

* incorporate brians feedback so far

* strip comments

* move vault.go to logging package

* update Debug to Trace

* Update go-plugin deps

* Update logging based on review comments

* Updates from review

* Unvendor logxi

* Remove null_logger.go
2018-04-02 17:46:59 -07:00
Jeff Mitchell 49ee1113aa Move close for s3 a bit earlier in case data is returned with 404 2018-03-30 13:03:33 -04:00
Jeff Mitchell 23e80d028f
Switch reading from S3 to io.Copy from io.ReadFull (#4225)
* Switch reading from S3 to io.Copy from io.ReadFull

If the Content-Length header wasn't being sent back, the current
behavior could panic. It's unclear when it will not be sent; it appears
to be CORS dependent. But this works around it by not trying to
preallocate a buffer of a specific size and instead just read until EOF.

In addition I noticed that Close wasn't being called.
https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#GetObjectOutput
specifies that Body is an io.ReadCloser so I added a call to Close.

Fixes #4222

* Add some extra efficiency
2018-03-30 12:42:48 -04:00
Seth Vargo b48a9878e7 Add HA support to the Google Cloud Storage backend (#4226) 2018-03-30 12:36:37 -04:00
Seth Vargo 56a48dc322 Fix broken build (#4212)
90e3ad2 broke the build. This fixes the syntax and ensures compilation.
I have no idea and make no claims that this is the correct behavior.
2018-03-28 15:47:40 -04:00
Nicholas Watkins 3b3af2f0a6 Add MySQL db max idle connections and connections lifetime assignment (#4211)
Allow the storage backend for MySQL to use a custom connection lifetime and max idle connection value if the parameter is specified in the config file of vault otherwise do not set in order to leave at default value.
2018-03-28 14:43:55 -04:00
Jim Kalafut d67ed1fdc2
Replace Batch operation in Cassandra Delete() (#4054)
This fixes failing Cassandra backend tests. It is also probably the
better route, since Batch operations aren't the preferred approach
unless necessary (https://docs.datastax.com/en/cql/3.3/cql/cql_using/useBatch.html).
2018-03-23 09:42:35 -07:00
Josh Soref 73b1fde82f Spelling (#4119) 2018-03-20 14:54:10 -04:00
Nicholas Watkins 475d5910e8 Allow configuration of dynamodb storage to specify the max retries of aws sdk (#4115) 2018-03-19 15:53:23 -04:00
Vishal Nayak a420d19bff
Remove limit on the couchdb listing (#4149) 2018-03-18 18:31:15 -04:00
Aleksandar a8304e5d4d Add the chunk_size optional parameter to gcs storage (#4060) 2018-03-05 08:32:48 -05:00
chris trott 78df6a630e Configurable Consul Service Address (#3971)
* Consul service address is blank

Setting an explicit service address eliminates the ability for Consul
to dynamically decide what it should be based on its translate_wan_addrs
setting.

translate_wan_addrs configures Consul to return its lan address to nodes
in its same datacenter but return its wan address to nodes in foreign
datacenters.

* service_address parameter for Consul storage backend

This parameter allows users to override the use of what Vault knows to
be its HA redirect address.

This option is particularly commpelling because if set to a blank
string, Consul will leverage the node configuration where the service is
registered which includes the `translate_wan_addrs` option. This option
conditionally associates nodes' lan or wan address based on where
requests originate.

* Add TestConsul_ServiceAddress

Ensures that the service_address configuration parameter is setting the
serviceAddress field of ConsulBackend instances properly.

If the "service_address" parameter is not set, the ConsulBackend
serviceAddress field must instantiate as nil to indicate that it can be
ignored.
2018-02-23 11:15:29 -05:00
Jeff Mitchell be53e38fe0 Fix test statement with formatting in fatal call 2018-02-20 00:26:41 -05:00
Vitya 66f4589195 Fix compatibility with some Postgres versions (#3986)
use server_version_num instead of parsing the text version. See: https://www.postgresql.org/docs/10/static/functions-info.html
2018-02-16 12:52:34 -05:00
Seth Vargo b5e4db975e Add useragent helper (#3991)
* Add useragent package

This helper provides a consistent user-agent header for Vault, taking into account different versions.

* Add user-agent headers to spanner and gcs
2018-02-15 18:30:31 -05:00
Seth Vargo cd930b1173 Add support for Google Cloud Spanner (#3977) 2018-02-14 20:31:20 -05:00
Jeff Mitchell a248a08102 Fix manta test 2018-02-14 10:39:52 -05:00
Jeff Mitchell 4f984569fa Plumb context through manta 2018-02-13 10:03:12 -05:00
Paul Stack 3c683dba92 Adding Manta Storage Backend (#3720)
This PR adds a new Storage Backend for Triton's Object Storage - Manta

```
make testacc TEST=./physical/manta
==> Checking that code complies with gofmt requirements...
==> Checking that build is using go version >= 1.9.1...
go generate
VAULT_ACC=1 go test -tags='vault' ./physical/manta -v  -timeout 45m
=== RUN   TestMantaBackend
--- PASS: TestMantaBackend (61.18s)
PASS
ok  	github.com/hashicorp/vault/physical/manta	61.210s
```

Manta behaves differently to how S3 works - it has no such concepts of Buckets - it is merely a filesystem style object store

Therefore, we have chosen the approach of when writing a secret `foo` it will actually map (on disk) as foo/.vault_value

The reason for this is because if we write the secret `foo/bar` and then try and Delete a key using the name `foo` then Manta
will complain that the folder is not empty because `foo/bar` exists. Therefore, `foo/bar` is written as `foo/bar/.vault_value`

The value of the key is *always* written to a directory tree of the name and put in a `.vault_value` file.
2018-02-12 18:22:41 -05:00
Jeff Mitchell ac382055d4
Validate Consul service name is RFC 1123 compliant (#3961) 2018-02-12 16:11:59 -05:00
Jeff Mitchell 844b7c395f Refactor fail logic in inmem 2018-02-12 11:25:48 -05:00
Jeff Mitchell 609648de4f Convert logical.InmemStorage to a wrapper around physical/inmem.
The original reason for the split was physical's dependencies, but those
haven't been onerous for a long time. Meanwhile it's a totally separate
implementation so we could be getting faulty results from tests. Get rid
of it and use the unified physical/inmem.
2018-02-12 11:16:16 -05:00
Jeff Mitchell 642b88c76a go vet fixes 2018-02-05 14:26:31 -05:00
Jeff Mitchell f33563f667 Some vet fixes 2018-02-04 20:37:57 -05:00
Xiang Li a120544b47 etcd: config etcd3 client's max response size (#3891) 2018-02-01 19:08:09 -05:00
Xiang Li 5fd85205cc etcd3: only create lock when lock is called (#3893) 2018-02-01 19:04:52 -05:00
Jeff Mitchell 60e2209532
Remove core restriction in cache and turn it into an active/standby restriction instead (#3849) 2018-01-25 22:21:51 -05:00
Brian Kassouf aa387bb4c2
Add compile tests to verify physical stores satisfy the correct interfaces (#3820) 2018-01-19 17:44:24 -08:00
Jeff Mitchell 395befc062 Update cache to satisfy Purge interface after context plumbing 2018-01-19 17:00:13 -05:00
Brian Kassouf 2f19de0305 Add context to storage backends and wire it through a lot of places (#3817) 2018-01-19 01:44:44 -05:00
Jon Davies 66e2593ef9 s3.go: Added options to use paths with S3 and the ability to disable SSL (#3730) 2018-01-03 12:11:00 -05:00
Antergone 312db6cc02 fix consul tls settings (#3719) 2017-12-19 14:24:21 -05:00
Antergone d68cc66954 check schema and table before create it (#3716) 2017-12-19 14:23:58 -05:00
Chris Hoffman f966d20225
Adding ability to cache core values, cache transaction improvements (#3640) 2017-12-06 12:25:17 -05:00
Jeff Mitchell 548629e8ef Port over some changes 2017-11-30 09:43:07 -05:00
Vlad Ungureanu 2ff547196f Remove unused recovery field in dynamodb backend (#3569) 2017-11-13 15:46:02 -05:00
Ben Higgins f78ab356d4 vault: recover from standby losing etcd lease (#3031) (#3511)
This change makes these errors transient instead of permanent:

[ERROR] core: failed to acquire lock: error=etcdserver: requested lease not found

After this change, there can still be one of these errors when a
standby vault that lost its lease tries to become leader, but on the
next lock acquisition attempt a new session will be created. With this
new session, the standby will be able to become the leader.
2017-11-03 13:38:16 -04:00
Jeff Mitchell 8a610e1e78 Move underscore tests to file from physical testing 2017-10-26 15:29:10 -04:00
Jeff Mitchell 28b0db38cc Revert couchdb changes 2017-10-26 15:27:20 -04:00
Jeff Mitchell 85500b5c3a Change prefix to a string that can be specified, rather than a bool 2017-10-26 15:26:28 -04:00
Jeff Mitchell 7e32ac15ec Add prefixing to couch to fix the error that was exposed 2017-10-26 15:26:28 -04:00
Jeff Mitchell 425b781fc8 Fix more tests 2017-10-26 15:26:28 -04:00
Jeff Mitchell 40a6635cd6 Fix testing 2017-10-26 15:26:28 -04:00
Jeff Mitchell e122853746 Add some more tests 2017-10-26 15:26:28 -04:00
Jeff Mitchell 3af3cf2b73 Allow underscores at the start of directories in file backend.
Fixes #3476
2017-10-26 15:26:28 -04:00
Jeff Mitchell 8e9317792d Fix some merge/update bugs 2017-10-23 16:49:46 -04:00
Jeff Mitchell c144f95be0 Sync over 2017-10-23 16:43:07 -04:00
Jeff Mitchell 1d852a7243 Use 0700 for directory permissions in file physical backend. 2017-10-12 14:24:34 -04:00
Ben Paxton 8ffc54cc1b Append trailing slash to folder listing in etcd3 backend (#3406) 2017-10-06 09:48:46 -04:00
Chris Hoffman 91338d7aa2 Adding latency injector option to -dev mode for storage operations (#3289) 2017-09-11 14:49:08 -04:00
Calvin Leung Huang c747caac2a Fix cassandra tests, explicitly set cluster port if provided (#3296)
* Fix cassandra tests, explicitly set cluster port if provided

* Update cassandra.yml test-fixture

* Add port as part of the config option, fix tests

* Remove hostport splitting in cassandraConnectionProducer.createSession

* Include port in API docs
2017-09-07 23:04:40 -04:00
Jeff Mitchell 44bf03e3b6 Fix compile after dep update 2017-09-05 18:18:34 -04:00
Jeff Mitchell 7585349e46 Use net.SplitHostPort on Consul address (#3268) 2017-08-31 12:31:34 -04:00
stephan stachurski e396d87bc5 add support to use application default credentials to gcs storage backend (#3257) 2017-08-30 15:42:02 -04:00
Christopher Pauley bd47ce523f update gcs backend tests- now strongly consistent (#3231) 2017-08-24 10:11:11 -04:00
Jeff Mitchell c864c0bad5 Return 500 if existence check fails, not 400 (#3173)
Fixes #3162
2017-08-15 16:44:16 -04:00
Chris Hoffman 6092538511 splitting cache into transactional and non-transactional cache structs (#3132) 2017-08-08 20:47:14 -04:00
Jeff Mitchell fdaaaadee2 Migrate physical backends into separate packages (#3106) 2017-08-03 13:24:27 -04:00
Jeff Mitchell 65d7face69 Merge branch 'master-oss' into issue-2241 2017-08-03 07:41:34 -04:00
Jeff Mitchell 4885b3e502 Use RemoteCredProvider instead of EC2RoleProvider (#2983) 2017-07-31 18:27:16 -04:00
Oliver Beattie 79058a3c95 Convert to dockertest.v3 2017-07-31 15:58:38 +01:00
Oliver Beattie 1cc1e7e615 Remove batching 2017-07-31 15:24:16 +01:00
Oliver Beattie 5046357e0f Fix Cassandra backend and tests 2017-07-31 15:24:16 +01:00
Filipe Varela a5a480551c Makes naming consistent w/ other storage backends (ie: etcd) 2017-07-31 15:18:07 +01:00
Filipe Varela 0177984e1b Fixes loading JSON pem bundles 2017-07-31 15:18:07 +01:00
Filipe Varela df388903e4 Fixes loading PEM bundles, JSON next 2017-07-31 15:18:06 +01:00
Filipe Varela b5144d833f Makes naming consistent with 'logical' 2017-07-31 15:18:05 +01:00
Filipe Varela cb08e543cb Use seconds for consistency with rest of project 2017-07-31 15:18:05 +01:00
Filipe Varela c6da462479 Adds support for TLS configuration 2017-07-31 15:18:04 +01:00
Filipe Varela 1c558c0c1d Adds support for authentication, protocol version and connection timeout 2017-07-31 15:18:04 +01:00
Filipe Varela 2abd4b6998 Make all operations share Session consistency setting 2017-07-31 15:18:03 +01:00
Oliver Beattie 2d04bfc447 Add dockertest support for Cassandra (it takes a while though ) 2017-07-31 15:18:03 +01:00
Oliver Beattie 3919f38bd5 Add a (basic) Cassandra storage backend 2017-07-31 15:18:01 +01:00
Xiang Li d61a47a01c physical: format fixes (#3062) 2017-07-26 17:51:58 -04:00
Xiang Li 7c761b8414 physical: add default timeout for etcd3 requests (#3053) 2017-07-26 12:10:12 -04:00
Chris Hoffman 2aa02fb3f0 CockroachDB Physical Backend (#2713) 2017-07-23 08:54:33 -04:00
Jeff Mitchell 4387871bca Add max_parallel to mssql and postgresql (#3026)
For storage backends, set max open connections to value of max_parallel.
2017-07-17 13:04:49 -04:00
Lars Lehtonen 78edb1bc76 Fix swallowed error in physical package. (#2976) 2017-07-07 08:15:59 -04:00
Yann 27ca1c40c2 [physical][postgresql] concat|| operator (#2945)
Use `||` standard concatenation instead of the `concat` function in
order to use the `vault_kv_store` index on `parent_path`.
2017-07-02 18:56:18 -04:00
Chris Hoffman c110f2188d Adding prefixed view of a physical backend (#2938) 2017-06-29 10:58:59 -04:00
Andri Mar Björgvinsson f0d103154e Better error messages using ListObjects than using HeadBucket. Might be a bigger request but messages are better than BadRequest, how this changes effect the messages are in the issue (#2892) 2017-06-20 01:16:41 +01:00
Raphael Randschau db4e1b4a99 CouchDB physical backend (#2880) 2017-06-17 11:22:10 -04:00
Jeff Mitchell 5d54aaf10a Fix azure test 2017-06-16 12:37:57 -04:00
Jeff Mitchell b6ea287ecb Change package in azure test 2017-06-16 12:18:16 -04:00
Jeff Mitchell f8f95524d0 Update Azure dep (#2881) 2017-06-16 12:06:09 -04:00
Dan Everton 32add0809e More efficient s3 paging (#2780) 2017-06-16 11:09:15 -04:00
Jeff Mitchell 3e7205c4c1 Add another nil guard to S3, follow on from #2785 2017-06-05 10:54:26 -04:00
Vishal Nayak c31b076360 Avoid panic in s3 list operation (#2785) 2017-06-05 10:53:20 -04:00
Mevan Samaratunga 731a7f187f fixed bug where the project name was not being read from configuration if it was provided via the "tenant" attribute. this was causing the swift client to crash with an EOF error. (#2803) 2017-06-05 10:48:39 -04:00