Commit graph

315 commits

Author SHA1 Message Date
Jeff Mitchell fba5f46e4d Fix compilation of test against new go-plugin 2018-02-06 13:33:29 -05:00
Jeff Mitchell f33563f667 Some vet fixes 2018-02-04 20:37:57 -05:00
Jeff Mitchell 676b302dbf
Add a sysview call to determine if a mount is local. (#3899)
This is useful for deciding when to run upgrade logic, e.g. if on a
performance secondary but local it's fine to run.
2018-02-02 18:17:12 -05:00
Brian Kassouf 1cee2a1415
plugins/gRPC: fix issues with reserved keywords in response data (#3881)
* plugins/gRPC: fix issues with reserved keywords in response data

* Add the path raw file for mock plugin

* Fix panic when special paths is nil

* Add tests for Listing and raw requests from plugins

* Add json.Number case when decoding the status

* Bump the version required for gRPC defaults

* Fix test for gRPC version check
2018-02-01 14:30:17 -08:00
Vishal Nayak 150ad8405b
Remove logical.Initialize() method (#3848)
* Remove logical.Initialize() method

* More cleanup

* Fix test
2018-01-25 20:19:27 -05:00
Calvin Leung Huang 385140ee6b
Version protocol switch (#3833)
* Use version to determine plugin protocol to use

* Remove field from ServeOpts

* Fix missing assignment, handle errors

* contraint -> constraint

* Inject the version string from the vault side

* Fix the version check

* Add grpc support check to database plugins

* Default to use grpc unless missing env var or fail on contraint check

* Add GRPCSupport test

* Add greater than test case

* Add go-version dep
2018-01-23 17:29:26 -05:00
Jeff Mitchell 8e8675053b Sync some bits over 2018-01-22 21:44:49 -05:00
Brian Kassouf b597e14f01
Update data values from byte arrays to strings in proto definition (#3829)
* Update data values from byte arrays to strings in proto definition

* Update comments
2018-01-22 17:56:34 -08:00
Brian Kassouf 2f19de0305 Add context to storage backends and wire it through a lot of places (#3817) 2018-01-19 01:44:44 -05:00
Brian Kassouf 7050c1ca41
gRPC Backend Plugins (#3808)
* Add grpc plugins

* Add grpc plugins

* Translate wrap info to/from proto

* Add nil checks

* Fix nil marshaling errors

* Provide logging through the go-plugin logger

* handle errors in the messages

* Update the TLS config so bidirectional connections work

* Add connectivity checks

* Restart plugin and add timeouts where context is not availible

* Add the response wrap data into the grpc system implementation

* Add leaseoptions to pb.Auth

* Add an error translator

* Add tests for translating the proto objects

* Fix rename of function

* Add tracing to plugins for easier debugging

* Handle plugin crashes with the go-plugin context

* Add test for grpcStorage

* Add tests for backend and system

* Bump go-plugin for GRPCBroker

* Remove RegisterLicense

* Add casing translations for new proto messages

* Use doneCtx in grpcClient

* Use doneCtx in grpcClient

* s/shutdown/shut down/
2018-01-18 13:49:20 -08:00
Jeff Mitchell 842a3a4a05 Remove RegisterLicense from logical.Backend
It's almost certainly the wrong signature and nothing uses it currently
anyways.
2018-01-18 13:44:29 -05:00
Vishal Nayak b826c56686
SHA2-256 salting for AppID (#3806)
* Use SHA2-256 hash with prefix to upgrade the paths

* test the SHA1 upgrade to SHA256

* Remove hash identifier and the delimiter; use 's' instead

* Added API test to verify the correctness of the fix

* Fix broken test

* remove unneeded test
2018-01-17 19:48:32 -05:00
Brian Kassouf 64da50c27c
Update plugin deps to include context changes (#3765)
* Update plugin deps to include context changes

* Fix tests
2018-01-08 12:26:13 -08:00
Brian Kassouf 1c190d4bda
Pass context to backends (#3750)
* Start work on passing context to backends

* More work on passing context

* Unindent logical system

* Unindent token store

* Unindent passthrough

* Unindent cubbyhole

* Fix tests

* use requestContext in rollback and expiration managers
2018-01-08 10:31:38 -08:00
Calvin Leung Huang 79cb82e133
Add logic for using Auth.Period when handling auth login/renew requests (#3677)
* Add logic for using Auth.Period when handling auth login/renew requests

* Set auth.TTL if not set in handleLoginRequest

* Always set auth.TTL = te.TTL on handleLoginRequest, check TTL and period against sys values on RenewToken

* Get sysView from le.Path, revert tests

* Add back auth.Policies

* Fix TokenStore tests, add resp warning when capping values

* Use switch for ttl/period check on RenewToken

* Move comments around
2017-12-15 13:30:05 -05:00
Brian Kassouf afe53eb862
Database gRPC plugins (#3666)
* Start work on context aware backends

* Start work on moving the database plugins to gRPC in order to pass context

* Add context to builtin database plugins

* use byte slice instead of string

* Context all the things

* Move proto messages to the dbplugin package

* Add a grpc mechanism for running backend plugins

* Serve the GRPC plugin

* Add backwards compatibility to the database plugins

* Remove backend plugin changes

* Remove backend plugin changes

* Cleanup the transport implementations

* If grpc connection is in an unexpected state restart the plugin

* Fix tests

* Fix tests

* Remove context from the request object, replace it with context.TODO

* Add a test to verify netRPC plugins still work

* Remove unused mapstructure call

* Code review fixes

* Code review fixes

* Code review fixes
2017-12-14 14:03:11 -08:00
Jeff Mitchell 2b78bc2a9b
Port over bits (#3575) 2017-11-13 15:31:32 -05:00
Jeff Mitchell 922ac56222
Seal wrap requesting in logical.Request wrapinfo (#3559) 2017-11-09 10:32:49 -05:00
Chris Hoffman fe52ce1115
Add TypeKVPairs field type (#3535) 2017-11-07 11:11:49 -05:00
Vishal Nayak 2b8d8f77d2
Handle 'not supplied' case for field type TypeNameString (#3546)
* Fix panic if value is not supplied for variables of TypeNameString

* Add tests for 'not supplied' case of all field types
2017-11-07 10:59:57 -05:00
Calvin Leung Huang 512b254820
Return role info for each role on pathRoleList (#3532)
* Return role info for each role on pathRoleList

* Change roles -> key_info, only return key_type

* Do not initialize result map in parseRole, refactor ListResponseWithInfo

* Add role list test
2017-11-03 17:12:03 -04:00
Vishal Nayak 7bae606662
External identity groups (#3447)
* external identity groups

* add local LDAP groups as well to group aliases

* add group aliases for okta credential backend

* Fix panic in tests

* fix build failure

* remove duplicated struct tag

* add test steps to test out removal of group member during renewals

* Add comment for having a prefix check in router

* fix tests

* s/parent_id/canonical_id

* s/parent/canonical in comments and errors
2017-11-02 16:05:48 -04:00
Jeff Mitchell d38a699c32 Make compile 2017-10-23 17:15:56 -04:00
Jeff Mitchell 8e9317792d Fix some merge/update bugs 2017-10-23 16:49:46 -04:00
Jeff Mitchell c144f95be0 Sync over 2017-10-23 16:43:07 -04:00
Seth Vargo 6fd9d11c79 Allow more complex errors from plugins (no interface change) (#3444)
* Allow more complex errors from plugins

This enables more complex types to be registered and returned from plugins.

*  Register common error types

This is a slightly less drastic change, which keeps the HTTPCodedError
as an interface.

* Remove replication error from list
2017-10-19 13:29:59 -07:00
Vishal Nayak f7ed6732a5 Porting identity store (#3419)
* porting identity to OSS

* changes that glue things together

* add testing bits

* wrapped entity id

* fix mount error

* some more changes to core

* fix storagepacker tests

* fix some more tests

* fix mount tests

* fix http mount tests

* audit changes for identity

* remove upgrade structs on the oss side

* added go-memdb to vendor
2017-10-11 10:21:20 -07:00
Seth Vargo 7c01dcd6b1 Fix tags on structure (#3442)
The comma breaks mapstructure for this actually.
2017-10-10 10:53:46 -04:00
Brian Kassouf f43d18fba5 Allow for response errors to be passed back from a plugin (#3412)
* Allow for response errors to be passed back from a plugin

* Fix issue with checking plugin.BasicError == nil
2017-10-06 14:20:57 -07:00
Vishal Nayak 3f7454939e plugin version bump (#3423) 2017-10-04 20:02:13 -04:00
Calvin Leung Huang 7df9ce2656 Fix plugin args output (#3424) 2017-10-04 20:00:53 -04:00
Vishal Nayak eb418f59a0 Rename persona to alias (#3420)
Merging this will break the build. I'll fix it post merge by updating the vendor libs.
2017-10-04 13:35:05 -04:00
Jeff Mitchell 7312c80e27 Add PersonaLookahead operation 2017-09-14 20:54:09 -04:00
Brian Kassouf a8d9426d9f Update locking components from DR replication changes (#3283)
* Update locking components from DR replication changes

* Fix plugin backend test

* Add a comment about needing the statelock:
2017-09-04 19:38:37 -04:00
Brian Kassouf ed6e818fea Plugin Version Update (#3275)
* Bump plugin version requirement

* Register time.Duration with gob
2017-09-01 00:01:14 -07:00
Calvin Leung Huang a581e96b78 Lazy-load plugin mounts (#3255)
* Lazy load plugins to avoid setup-unwrap cycle

* Remove commented blocks

* Refactor NewTestCluster, use single core cluster on basic plugin tests

* Set c.pluginDirectory in TestAddTestPlugin for setupPluginCatalog to work properly

* Add special path to mock plugin

* Move ensureCoresSealed to vault/testing.go

* Use same method for EnsureCoresSealed and Cleanup

* Bump ensureCoresSealed timeout to 60s

* Correctly handle nil opts on NewTestCluster

* Add metadata flag to APIClientMeta, use meta-enabled plugin when mounting to bootstrap

* Check metadata flag directly on the plugin process

* Plumb isMetadataMode down to PluginRunner

* Add NOOP shims when running in metadata mode

* Remove unused flag from the APIMetadata object

* Remove setupSecretPlugins and setupCredentialPlugins functions

* Move when we setup rollback manager to after the plugins are initialized

* Fix tests

* Fix merge issue

* start rollback manager after the credential setup

* Add guards against running certain client and server functions while in metadata mode

* Call initialize once a plugin is loaded on the fly

* Add more tests, update basic secret/auth plugin tests to trigger lazy loading

* Skip mount if plugin removed from catalog

* Fixup

* Remove commented line on LookupPlugin

* Fail on mount operation if plugin is re-added to catalog and mount is on existing path

* Check type and special paths on startBackend

* Fix merge conflicts

* Refactor PluginRunner run methods to use runCommon, fix TestSystemBackend_Plugin_auth
2017-09-01 01:02:03 -04:00
Calvin Leung Huang 86ea7e945d Add plugin auto-reload capability (#3171)
* Add automatic plugin reload

* Refactor builtin/backend

* Remove plugin reload at the core level

* Refactor plugin tests

* Add auto-reload test case

* Change backend to use sync.RWMutex, fix dangling test plugin processes

* Add a canary to plugin backends to avoid reloading many times (#3174)

* Call setupPluginCatalog before mount-related operations in postUnseal

* Don't create multiple system backends since core only holds a reference (#3176)

to one.
2017-08-15 22:10:32 -04:00
vishalnayak 8902240dfa Added persona to logical auth 2017-08-15 13:55:58 -04:00
Calvin Leung Huang b023d46cb8 Direct plugin logs through vault's logger (#3142)
* Direct plugin logs through vault's logger

* Pass in a logger in testConfig
2017-08-15 10:16:48 -04:00
Calvin Leung Huang 6853d80212 Add plugin backend reload capability (#3112)
* Add plugin reload capability on all mounts for a specific plugin type

* Comments cleanup

* Add per-mount plugin backend reload, add tests

* Fix typos

* Remove old comment

* Reuse existing storage view in reloadPluginCommon

* Correctly handle reloading auth plugin backends

* Update path to plugin/backend/reload

* Use multierrors on reloadMatchingPluginMounts, attempt to reload all mounts provided

* Use internal value as check to ensure plugin backend reload

* Remove connection state from request for plugins at the moment

* Minor cleanup

* Refactor tests
2017-08-08 00:18:59 -04:00
Calvin Leung Huang 0871e7eadb Update mock-plugin (#3107) 2017-08-04 11:43:36 -04:00
Jeff Mitchell fdaaaadee2 Migrate physical backends into separate packages (#3106) 2017-08-03 13:24:27 -04:00
Calvin Leung Huang db9d9e6415 Store original request path in WrapInfo (#3100)
* Store original request path in WrapInfo as CreationPath

* Add wrapping_token_creation_path to CLI output

* Add CreationPath to AuditResponseWrapInfo

* Fix tests

* Add and fix tests, update API docs with new sample responses
2017-08-02 18:28:58 -04:00
Jeff Mitchell 1d0c98257d Add more TypeDurationSecond conversions 2017-08-02 18:12:02 -04:00
Calvin Leung Huang 54dd6967d8 Default mock-plugin type to logical.TypeLogical on Factory func 2017-07-31 16:20:52 -04:00
Vishal Nayak f6b03e8b1b Adding logical/identity.go to OSS (#3054) 2017-07-27 11:56:32 -04:00
Jeff Mitchell 8519b3e8ed Make logical.InmemStorage standalone (#3066) 2017-07-26 17:59:14 -04:00
Calvin Leung Huang 40c1c93937 Fix gob register issue when using tls certs on plugins (#3060) 2017-07-26 13:44:07 -04:00
Calvin Leung Huang c00741d587 Do not send storage on HandleRequest and HandleExistenceCheck on plugins 2017-07-25 16:57:26 -04:00
Calvin Leung Huang bb54e9c131 Backend plugin system (#2874)
* Add backend plugin changes

* Fix totp backend plugin tests

* Fix logical/plugin InvalidateKey test

* Fix plugin catalog CRUD test, fix NoopBackend

* Clean up commented code block

* Fix system backend mount test

* Set plugin_name to omitempty, fix handleMountTable config parsing

* Clean up comments, keep shim connections alive until cleanup

* Include pluginClient, disallow LookupPlugin call from within a plugin

* Add wrapper around backendPluginClient for proper cleanup

* Add logger shim tests

* Add logger, storage, and system shim tests

* Use pointer receivers for system view shim

* Use plugin name if no path is provided on mount

* Enable plugins for auth backends

* Add backend type attribute, move builtin/plugin/package

* Fix merge conflict

* Fix missing plugin name in mount config

* Add integration tests on enabling auth backend plugins

* Remove dependency cycle on mock-plugin

* Add passthrough backend plugin, use logical.BackendType to determine lease generation

* Remove vault package dependency on passthrough package

* Add basic impl test for passthrough plugin

* Incorporate feedback; set b.backend after shims creation on backendPluginServer

* Fix totp plugin test

* Add plugin backends docs

* Fix tests

* Fix builtin/plugin tests

* Remove flatten from PluginRunner fields

* Move mock plugin to logical/plugin, remove totp and passthrough plugins

* Move pluginMap into newPluginClient

* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck

* Change shim logger's Fatal to no-op

* Change BackendType to uint32, match UX backend types

* Change framework.Backend Setup signature

* Add Setup func to logical.Backend interface

* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments

* Remove commented var in plugin package

* RegisterLicense on logical.Backend interface (#3017)

* Add RegisterLicense to logical.Backend interface

* Update RegisterLicense to use callback func on framework.Backend

* Refactor framework.Backend.RegisterLicense

* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs

* plugin: Revert BackendType to remove TypePassthrough and related references

* Fix typo in plugin backends docs
2017-07-20 13:28:40 -04:00