Commit graph

263 commits

Author SHA1 Message Date
Pawel Rozlach 41ade15f73 Fix file backend so that it properly removes nested secrets.
This patch makes file backend properly remove nested secrets, without leaving
empty directory artifacts, no matter how nested directories were.
2016-10-04 21:56:12 +02:00
Pawel Rozlach 44b4704cfa Fix zookeeper backend so that properly deletes/lists secrets.
This patch fixes two bugs in Zookeeper backends:
 * backend was determining if the node is a leaf or not basing on the number
   of the childer given node has. This is incorrect if you consider the fact
   that deleteing nested node can leave empty prefixes/dirs behind which have
   neither children nor data inside. The fix changes this situation by testing
   if the node has any data set - if not then it is not a leaf.
 * zookeeper does not delete nodes that do not have childern just like consul
   does and this leads to leaving empty nodes behind. In order to fix it, we
   scan the logical path of a secret being deleted for empty dirs/prefixes and
   remove them up until first non-empty one.
2016-10-04 21:56:12 +02:00
Pawel Rozlach 68fc52958d Add tests for nested/prefixed secrets removal.
Current tests were not checking if backends are properly removing
nested secrets. We follow here the behaviour of Consul backend, where
empty "directories/prefixes" are automatically removed by Consul itself.
2016-10-04 21:55:33 +02:00
Jeff Mitchell 226ef5d78c Make HA in etcd off by default. (#1909)
Fixes #1908

(Doesn't really "fix" it but someone from the community needs to step up
if they want to see this fixed.)
2016-09-21 14:01:36 -04:00
James Pearson Hughes f598c78d98 DynamoDB: fix log typo (#1891) 2016-09-14 15:16:24 -04:00
Jeff Mitchell ffaaacd029 Have file backend remove empty dirs. (#1821)
Add tests to check that prefixes are being properly removed (or at
least, not listed) from backends.
2016-08-31 14:12:28 -04:00
Jeff Mitchell 2ce4397deb Plumb through the ability to set the storage read cache size. (#1784)
Plumb through the ability to set the storage read cache size.

Fixes #1772
2016-08-26 10:27:06 -04:00
Jeff Mitchell 58b32e5432 Convert to logxi 2016-08-21 18:13:37 -04:00
Jeff Mitchell 2860dcc60f gofmt 2016-08-19 16:48:32 -04:00
Jeff Mitchell 734e80ca56 Add permit pool to dynamodb 2016-08-15 19:45:06 -04:00
Jeff Mitchell dcba6129e3 Use dockertest for physical consul tests, and always run them 2016-08-15 16:20:32 -04:00
Jeff Mitchell 37320f8798 Request forwarding (#1721)
Add request forwarding.
2016-08-15 09:42:42 -04:00
Jeff Mitchell 9e204bd88c Add arbitrary string slice parsing.
Like the KV function, this supports either separated strings or JSON
strings, base64-encoded or not.

Fixes #1619 in theory.
2016-08-03 14:24:16 -04:00
Jeff Mitchell c025b292b5 Cleanup 2016-08-03 13:09:12 -04:00
vishalnayak 91e60a5824 Fixed the test after removing shutdown bool 2016-08-01 12:20:38 -04:00
Jeff Mitchell 6ffefb649d Close the shutdown channel instead of sending a value down 2016-08-01 11:58:45 -04:00
vishalnayak 05b8ce8348 Address review feedback 2016-08-01 11:15:25 -04:00
vishalnayak 21d155f4af Fix physical/consul test case 2016-08-01 10:55:47 -04:00
vishalnayak 5ed10f4074 Make the defer statement of waitgroup to execute last 2016-08-01 10:24:27 -04:00
vishalnayak ea2e677f02 Sharing shutdown message with physical consul backend 2016-07-31 10:09:16 -04:00
vishalnayak a8b4fc0d3c Add waitgroup wait to allow physical consul to deregister checks 2016-07-30 13:17:29 -04:00
vishalnayak 32a72e84dc Added test for service tags 2016-07-22 09:04:42 -04:00
vishalnayak 58bd985551 Address review feedback from @jefferai 2016-07-22 08:44:16 -04:00
vishalnayak 9bfb518e3f Make debug statements specify that the logs are for physical/consul 2016-07-22 07:03:14 -04:00
vishalnayak 248889dd1b Fix broken test 2016-07-22 06:42:56 -04:00
vishalnayak 765d131b47 Added service-tags config option to provide additional tags to registered service 2016-07-22 04:41:48 -04:00
vishalnayak bd8ff10462 Address review feedback from @sean 2016-07-21 19:04:43 -04:00
vishalnayak 6872df833a Uniquify the check ID 2016-07-19 14:17:50 -04:00
vishalnayak f54dc7a31e Set QueryOptions while fetching service information from Consul catalog 2016-07-19 14:07:06 -04:00
vishalnayak 11e6fe0fb4 Fix tests 2016-07-19 07:58:47 -04:00
vishalnayak ac7ecbce5c Fixed re-registrations and health-check flatenning issue 2016-07-19 07:06:41 -04:00
Jeff Mitchell c47fc73bd1 Use parsebool 2016-07-18 13:49:05 -04:00
Jeff Mitchell a3ce0dcb0c Turn off DynamoDB HA by default.
The semantics are wonky and have caused issues from people not reading
docs. It can be enabled but by default is off.
2016-07-18 13:19:58 -04:00
vishalnayak c14235b206 Merge branch 'master-oss' into json-use-number
Conflicts:
	http/handler.go
	logical/framework/field_data.go
	logical/framework/wal.go
	vault/logical_passthrough.go
2016-07-15 19:21:55 -04:00
vishalnayak 407722a9b4 Added tls_min_version to consul storage backend 2016-07-12 20:10:54 -04:00
vishalnayak f34f0ef503 Make 'tls_min_version' configurable 2016-07-12 19:32:47 -04:00
vishalnayak 46d34130ac Set minimum TLS version in all tls.Config objects 2016-07-12 17:06:28 -04:00
vishalnayak ad7cb2c8f1 Added JSON Decode and Encode helpers.
Changed all the occurances of Unmarshal to use the helpers.
Fixed http/ package tests.
2016-07-06 12:25:40 -04:00
Tim Schindler b00535bcf5 fixed typo: explitely -> explicitly 2016-06-30 19:10:15 +00:00
Tim Schindler 23f08a139c os.GetEnv -> os.Getenv 2016-06-30 18:53:55 +00:00
Tim Schindler 6a1c142924 added ETCD_ADDR env var to etcd backend 2016-06-30 18:46:00 +00:00
Jeff Mitchell e7a1e95037 Merge pull request #1548 from quixoten/nopreparepsql
Remove prepared stmnts from pgsql physical backend
2016-06-28 16:38:13 +02:00
Jeff Mitchell b8c30aea18 Merge pull request #1502 from hashicorp/pr-1425
Staging area for me to fix up PR 1425
2016-06-08 12:31:31 -04:00
Jeff Mitchell 72a25d018c Add permit pool and cleanhttp support to Swift 2016-06-08 12:20:21 -04:00
Jeff Mitchell da6371ffc3 Merge remote-tracking branch 'origin/master' into pr-1425 2016-06-08 12:10:29 -04:00
Jeff Mitchell 3bf06b47e4 Add permitPool support to Azure 2016-06-08 12:01:43 -04:00
Bill Monkman de8477244e #1486 : Fixed sealed and leader checks for consul backend 2016-06-03 16:00:31 -07:00
Devin Christensen b0f50ecb6c Remove prepared stmnts from pgsql physical backend
Prepared statements prevent the use of connection multiplexing software
such as PGBouncer. Even when PGBouncer is configured for [session mode][1]
there's a possibility that a connection to PostgreSQL can be re-used by
different clients.  This leads to errors when clients use session based
features (like prepared statements).

This change removes prepared statements from the PostgreSQL physical
backend. This will allow vault to successfully work in infrastructures
that employ the use of PGBouncer or other connection multiplexing
software.

[1]: https://pgbouncer.github.io/config.html#poolmode
2016-05-26 17:07:21 -06:00
Stuart Glenn 1fea2799a5 Add backend for OpenStack Swift 2016-05-16 17:29:23 -05:00
Sean Chittenden 7a4b31ce51
Speling police 2016-05-15 09:58:36 -07:00
Sean Chittenden 99a5213f0b Merge pull request #1355 from hashicorp/f-vault-service
Vault/Consul Service refinement
2016-05-12 11:48:29 -07:00
Joe Hillenbrand 3b14f5043f Fix default etcd address
Should be `127.0.0.1`, not `128.0.0.1`
2016-05-10 12:50:11 -07:00
Jeff Mitchell 1b0df1d46f Cleanups, add shared provider, ability to specify http client, and port S3 physical backend over 2016-05-03 17:01:02 -04:00
Sean Chittenden 7e5dbf409e Be idiomatic. Use a switch instead of if/else 2016-04-29 11:35:33 -07:00
Sean Chittenden 614104717c Remove useless return statement 2016-04-28 13:16:17 -07:00
Sean Chittenden 93ba3a0b8a Refactor the Consul Backend to match evented demuxer 2016-04-28 11:05:18 -07:00
Sean Chittenden e129273e4f Debug log consul configuration parameters when set 2016-04-28 11:05:18 -07:00
Sean Chittenden 0b72906fc3 Change the interface of ServiceDiscovery
Instead of passing state, signal that the state has changed and provide a callback handler that can query Core.
2016-04-28 11:05:18 -07:00
Sean Chittenden cc64778628 Fix logger output
Pointed out by: ryanuber
2016-04-28 11:05:18 -07:00
Jeff Mitchell e5c31d66a2 Better handle nil responses in S3 backend, also a case where error wasn't checked 2016-04-26 08:11:56 -04:00
Sean Chittenden 557d8b8a24 Make use of logger interface inside of the Consul BE 2016-04-25 20:10:55 -07:00
Sean Chittenden aeea7628d6 Add a *log.Logger argument to physical.Factory
Logging in the backend is a good thing.  This is a noisy interface change but should be a functional noop.
2016-04-25 20:10:32 -07:00
Sean Chittenden 5fd5869bc5 Rewriting history before it gets away from me 2016-04-25 18:05:50 -07:00
Sean Chittenden 9b8095d7ea Change to the pre-0.6.4 Consul Check API
Consul is never going to pass in more than 1K of output.  This mitigates the pre-0.6.4 concern.
2016-04-25 18:01:13 -07:00
Sean Chittenden f5183fa506 Collapse UpdateAdvertiseAddr() into RunServiceDiscovery() 2016-04-25 18:01:13 -07:00
Sean Chittenden 5104c58c54 Update tests to chase sealed -> unsealed transition 2016-04-25 18:01:13 -07:00
Sean Chittenden 7fe0b2c6a1 Persistently retry to update service registration
If the local Consul agent is not available while attempting to step down from active or up to active, retry once a second.  Allow for concurrent changes to the state with a single registration updater.  Fix standby initialization.
2016-04-25 18:01:13 -07:00
Sean Chittenden 3449fa1bc3 Consistently skip Consul checks
Hide all Consul checks behind `CONSUL_HTTP_ADDR` env vs `CONSUL_ADDR` which is non-standard.
2016-04-25 18:01:13 -07:00
Sean Chittenden 60006f550f Various refactoring to clean up code organization
Brought to you by: Dept of 2nd thoughts before pushing enter on `git push`
2016-04-25 18:01:13 -07:00
Sean Chittenden 53f9cea87c Compare the correct values when validating check_timeout 2016-04-25 18:01:13 -07:00
Sean Chittenden 70ae7f73b4 Detect type conversion failure 2016-04-25 18:01:13 -07:00
Sean Chittenden ae66e65bcf Don't export the builtin backends 2016-04-25 18:01:13 -07:00
Sean Chittenden bd3335c1bd go fmt the PostgreSQL backend 2016-04-25 18:01:13 -07:00
Sean Chittenden 6b2c83564e Teach Vault how to register with Consul
Vault will now register itself with Consul.  The active node can be found using `active.vault.service.consul`.  All standby vaults are available via `standby.vault.service.consul`.  All unsealed vaults are considered healthy and available via `vault.service.consul`.  Change in status and registration is event driven and should happen at the speed of a write to Consul (~network RTT + ~1x fsync(2)).

Healthy/active:

```
curl -X GET 'http://127.0.0.1:8500/v1/health/service/vault?pretty' && echo;
[
    {
        "Node": {
            "Node": "vm1",
            "Address": "127.0.0.1",
            "TaggedAddresses": {
                "wan": "127.0.0.1"
            },
            "CreateIndex": 3,
            "ModifyIndex": 20
        },
        "Service": {
            "ID": "vault:127.0.0.1:8200",
            "Service": "vault",
            "Tags": [
                "active"
            ],
            "Address": "127.0.0.1",
            "Port": 8200,
            "EnableTagOverride": false,
            "CreateIndex": 17,
            "ModifyIndex": 20
        },
        "Checks": [
            {
                "Node": "vm1",
                "CheckID": "serfHealth",
                "Name": "Serf Health Status",
                "Status": "passing",
                "Notes": "",
                "Output": "Agent alive and reachable",
                "ServiceID": "",
                "ServiceName": "",
                "CreateIndex": 3,
                "ModifyIndex": 3
            },
            {
                "Node": "vm1",
                "CheckID": "vault-sealed-check",
                "Name": "Vault Sealed Status",
                "Status": "passing",
                "Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
                "Output": "",
                "ServiceID": "vault:127.0.0.1:8200",
                "ServiceName": "vault",
                "CreateIndex": 19,
                "ModifyIndex": 19
            }
        ]
    }
]
```

Healthy/standby:

```
[snip]
        "Service": {
            "ID": "vault:127.0.0.2:8200",
            "Service": "vault",
            "Tags": [
                "standby"
            ],
            "Address": "127.0.0.2",
            "Port": 8200,
            "EnableTagOverride": false,
            "CreateIndex": 17,
            "ModifyIndex": 20
        },
        "Checks": [
            {
                "Node": "vm2",
                "CheckID": "serfHealth",
                "Name": "Serf Health Status",
                "Status": "passing",
                "Notes": "",
                "Output": "Agent alive and reachable",
                "ServiceID": "",
                "ServiceName": "",
                "CreateIndex": 3,
                "ModifyIndex": 3
            },
            {
                "Node": "vm2",
                "CheckID": "vault-sealed-check",
                "Name": "Vault Sealed Status",
                "Status": "passing",
                "Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
                "Output": "",
                "ServiceID": "vault:127.0.0.2:8200",
                "ServiceName": "vault",
                "CreateIndex": 19,
                "ModifyIndex": 19
            }
        ]
    }
]
```

Sealed:

```
        "Checks": [
            {
                "Node": "vm2",
                "CheckID": "serfHealth",
                "Name": "Serf Health Status",
                "Status": "passing",
                "Notes": "",
                "Output": "Agent alive and reachable",
                "ServiceID": "",
                "ServiceName": "",
                "CreateIndex": 3,
                "ModifyIndex": 3
            },
            {
                "Node": "vm2",
                "CheckID": "vault-sealed-check",
                "Name": "Vault Sealed Status",
                "Status": "critical",
                "Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
                "Output": "Vault Sealed",
                "ServiceID": "vault:127.0.0.2:8200",
                "ServiceName": "vault",
                "CreateIndex": 19,
                "ModifyIndex": 38
            }
        ]
```
2016-04-25 18:01:13 -07:00
Sean Chittenden 230b59f34c Stub out service discovery functionality
Hook asynchronous notifications into Core to change the status of vault based on its active/standby, and sealed/unsealed status.
2016-04-25 18:00:54 -07:00
Jeff Mitchell f00beb4e32 Update azure backend for newer sdk 2016-04-26 00:08:07 +00:00
Jeff Mitchell a481bff2b1 Fix commenting S3 -> Azure 2016-04-25 19:53:07 +00:00
Tobias Haag 175e3cc354 added Azure backend support
updated Godeps
added website docs
updated vendor
2016-03-30 19:49:38 -07:00
Jeff Mitchell deed5cc121 Output original error on etcd sync failure.
Fixes #1141
2016-02-26 15:15:23 -05:00
Jeff Mitchell 50d3b68c8d Merge pull request #1078 from eyal-lupu/master
ZooKeeper Backend: Authnetication and Authorization support
2016-02-19 15:13:09 -05:00
Eyal Lupu a6e9820e8d typo in comment 2016-02-19 13:28:02 +00:00
Eyal Lupu 23303429c0 'Eagerly' parse ZK authentication and authorization to fast-fail bad configuration 2016-02-19 13:24:57 +00:00
Eyal Lupu c7fe99b1e9 1. gofmt
2. Change if expr syntax to be consist with the rest of Vault code
3. More details on error message
2016-02-19 12:19:01 +00:00
Jeff Mitchell 5edaf522a8 Use a pooled transport for the Consul physical backend and give it 4 idle connections 2016-02-17 16:53:30 -05:00
Eyal Lupu e9c7a02850 https://github.com/hashicorp/vault/issues/1058
Make sure locks are also using the same auth info as data
2016-02-15 15:29:08 +00:00
Eyal Lupu d4db2ea79c fixes to https://github.com/hashicorp/vault/issues/1058
Configuration now supports:
- auth_info
-znode_owner
2016-02-15 15:03:12 +00:00
Devin Christensen 4112809fb5 Make the PostgreSQL backend more performant 2016-01-29 13:47:10 -07:00
Jeff Mitchell 68dc0e2dd3 Merge pull request #945 from quixoten/postgres_physical
Add support for PostgreSQL as a physical backend
2016-01-29 10:35:38 -05:00
Devin Christensen 737df30939 Improve naming
Hopefully this naming scheme will be more straightforward.
2016-01-27 17:15:48 -07:00
Jeff Mitchell b7a49922a9 Update etcd sync option to be a string.
Ping #921
2016-01-27 17:15:52 -05:00
Jeff Mitchell b0bd06f5a4 Merge pull request #921 from faradayio/hosted-etcd-support
Load-balanced etcd support
2016-01-27 17:09:43 -05:00
Devin Christensen 9d776351a3 Merge 'upstream/master' into postgres_physical 2016-01-22 20:56:07 -07:00
Devin Christensen c226b0be7d Update naming and pull DDL for upsert back out 2016-01-22 17:15:10 -07:00
Devin Christensen 32b712ddb1 Move the upsert definition back into the code 2016-01-22 09:47:02 -07:00
Devin Christensen bfbdc72e03 Remove options for column configuration 2016-01-22 08:41:31 -07:00
Jeff Mitchell be1b4c8a46 Only allow listing on folders and enforce this. Also remove string sorting from Consul backend as it's not a requirement and other backends don't do it. 2016-01-22 10:07:32 -05:00
Devin Christensen 06641570c7 Remove DDL statements from the code 2016-01-20 18:52:49 -07:00
Devin Christensen bcc720be11 Remove superfluous comparison 2016-01-20 17:05:21 -07:00
Devin Christensen 65bd200fae Ensure rows.Close() is called in List 2016-01-20 17:02:23 -07:00
Devin Christensen 614f7b7157 Prefer TEXT over VARCHAR
From the PostgreSQL docs
(http://www.postgresql.org/docs/9.4/static/datatype-character.html):

 > Tip: There is no performance difference among these three types,
 > apart from increased storage space when using the blank-padded type,
 > and a few extra CPU cycles to check the length when storing into a
 > length-constrained column. While character(n) has performance
 > advantages in some other database systems, there is no such advantage
 > in PostgreSQL; in fact character(n) is usually the slowest of the
 > three because of its additional storage costs. In most situations
 > text or character varying should be used instead.
2016-01-20 16:56:46 -07:00