* Add request timeouts in normal request path and to expirations
* Add ability to adjust default max request duration
* Some test fixes
* Ensure tests have defaults set for max request duration
* Add context cancel checking to inmem/file
* Fix tests
* Fix tests
* Set default max request duration to basically infinity for this release for BC
* Address feedback
It's not obvious why this should be secret, and if it were considered
secret, when and what anything would ever be allowed to access it.
Likely the right way to tie secret values to particular
entities/aliases/groups would be to use the upcoming templated ACL
feature.
This change makes it so that if a lease is revoked through user action,
we set the expiration time to now and update pending, just as we do with
tokens. This allows the normal retry logic to apply in these cases as
well, instead of just erroring out immediately. The idea being that once
you tell Vault to revoke something it should keep doing its darndest to
actually make that happen.
* This changes the way policies are reported in audit logs.
Previously, only policies tied to tokens would be reported. This could
make it difficult to perform after-the-fact analysis based on both the
initial response entry and further requests. Now, the full set of
applicable policies from both the token and any derived policies from
Identity are reported.
To keep things consistent, token authentications now also return the
full set of policies in api.Secret.Auth responses, so this both makes it
easier for users to understand their actual full set, and it matches
what the audit logs now report.
Taking inspiration from
https://github.com/golang/go/issues/17604#issuecomment-256384471
suggests that taking the address of a stack variable for use in atomics
works (at least, the race detector doesn't complain) but is doing it
wrong.
The only other change is a change in Leader() detecting if HA is enabled
to fast-path out. This value never changes after NewCore, so we don't
need to grab the read lock to check it.
* Add entity information request to system view
* fixing a few comments
* sharing types between plugin and logical
* sharing types between plugin and logical
* fixing output directory for proto
* removing extra replacement
* adding mount type lookup
* empty entities return nil instead of error
* adding some comments
This takes place in two parts, since working on this exposed an issue
with response wrapping when there is a raw body set. The changes are (in
diff order):
* A CurrentWrappingLookupFunc has been added to return the current
value. This is necessary for the lookahead call since we don't want the
lookahead call to be wrapped.
* Support for unwrapping < 0.6.2 tokens via the API/CLI has been
removed, because we now have backends returning 404s with data and can't
rely on the 404 trick. These can still be read manually via
cubbyhole/response.
* KV preflight version request now ensures that its calls is not
wrapped, and restores any given function after.
* When responding with a raw body, instead of always base64-decoding a
string value and erroring on failure, on failure we assume that it
simply wasn't a base64-encoded value and use it as is.
* A test that fails on master and works now that ensures that raw body
responses that are wrapped and then unwrapped return the expected
values.
* A flag for response data that indicates to the wrapping handling that
the data contained therein is already JSON decoded (more later).
* RespondWithStatusCode now defaults to a string so that the value is
HMAC'd during audit. The function always JSON encodes the body, so
before now it was always returning []byte which would skip HMACing. We
don't know what's in the data, so this is a "better safe than sorry"
issue. If different behavior is needed, backends can always manually
populate the data instead of relying on the helper function.
* We now check unwrapped data after unwrapping to see if there were raw
flags. If so, we try to detect whether the value can be unbase64'd. The
reason is that if it can it was probably originally a []byte and
shouldn't be audit HMAC'd; if not, it was probably originally a string
and should be. In either case, we then set the value as the raw body and
hit the flag indicating that it's already been JSON decoded so not to
try again before auditing. Doing it this way ensures the right typing.
* There is now a check to see if the data coming from unwrapping is
already JSON decoded and if so the decoding is skipped before setting
the audit response.
* Allow returning list information and other data in 404s.
On read it'll output data and/or warnings on a 404 if they exist. On
list, the same behavior; the actual 'vault list' command doesn't change
behavior though in terms of output unless there are no actual keys (so
it doesn't just magically show other data).
This corrects some assumptions in response_util and wrapping.go; it also
corrects a few places in the latter where it could leak a (useless)
token in some error cases.
* Use same 404 logic in delete/put too
* Add the same secret parsing logic to the KV request functions
* govet cleanup in token store
* adding general ttl handling to login requests
* consolidating TTL calculation to system view
* deprecate LeaseExtend
* deprecate LeaseExtend
* set the increment to the correct value
* move calculateTTL out of SystemView
* remove unused value
* add back clearing of lease id
* implement core ttl in some backends
* removing increment and issue time from lease options
* adding ttl tests, fixing some compile issue
* adding ttl tests
* fixing some explicit max TTL logic
* fixing up some tests
* removing unneeded test
* off by one errors...
* adding back some logic for bc
* adding period to return on renewal
* tweaking max ttl capping slightly
* use the appropriate precision for ttl calculation
* deprecate proto fields instead of delete
* addressing feedback
* moving TTL handling for backends to core
* mongo is a secret backend not auth
* adding estimated ttl for backends that also manage the expiration time
* set the estimate values before calling the renew request
* moving calculate TTL to framework, revert removal of increment and issue time from logical
* minor edits
* addressing feedback
* address more feedback
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
* Add some requirements for versioned k/v
* Add a warning message when an upgrade is triggered
* Add path help values
* Make the kv header a const
* Add the uid to mount entry instead of options map
* Pass the backend aware uuid to the mounts and plugins
* Fix comment
* Add options to secret/auth enable and tune CLI commands (#4170)
* Switch mount/tune options to use TypeKVPairs (#4171)
* switching options to TypeKVPairs, adding bool parse for versioned flag
* flipping bool check
* Fix leases coming back from non-leased pluin kv store
* add a test for updating mount options
* Fix tests
* Add a method for returning a 404 with data
* Pass the full resp object through to respond raw
* Add comment
* Refactor so it works across plugin gRPC
* Handle some review comments
* Pass request object instead of request ID
The original reason for the split was physical's dependencies, but those
haven't been onerous for a long time. Meanwhile it's a totally separate
implementation so we could be getting faulty results from tests. Get rid
of it and use the unified physical/inmem.
* plugins/gRPC: fix issues with reserved keywords in response data
* Add the path raw file for mock plugin
* Fix panic when special paths is nil
* Add tests for Listing and raw requests from plugins
* Add json.Number case when decoding the status
* Bump the version required for gRPC defaults
* Fix test for gRPC version check
* Use version to determine plugin protocol to use
* Remove field from ServeOpts
* Fix missing assignment, handle errors
* contraint -> constraint
* Inject the version string from the vault side
* Fix the version check
* Add grpc support check to database plugins
* Default to use grpc unless missing env var or fail on contraint check
* Add GRPCSupport test
* Add greater than test case
* Add go-version dep
* Add grpc plugins
* Add grpc plugins
* Translate wrap info to/from proto
* Add nil checks
* Fix nil marshaling errors
* Provide logging through the go-plugin logger
* handle errors in the messages
* Update the TLS config so bidirectional connections work
* Add connectivity checks
* Restart plugin and add timeouts where context is not availible
* Add the response wrap data into the grpc system implementation
* Add leaseoptions to pb.Auth
* Add an error translator
* Add tests for translating the proto objects
* Fix rename of function
* Add tracing to plugins for easier debugging
* Handle plugin crashes with the go-plugin context
* Add test for grpcStorage
* Add tests for backend and system
* Bump go-plugin for GRPCBroker
* Remove RegisterLicense
* Add casing translations for new proto messages
* Use doneCtx in grpcClient
* Use doneCtx in grpcClient
* s/shutdown/shut down/
* Use SHA2-256 hash with prefix to upgrade the paths
* test the SHA1 upgrade to SHA256
* Remove hash identifier and the delimiter; use 's' instead
* Added API test to verify the correctness of the fix
* Fix broken test
* remove unneeded test
* Start work on passing context to backends
* More work on passing context
* Unindent logical system
* Unindent token store
* Unindent passthrough
* Unindent cubbyhole
* Fix tests
* use requestContext in rollback and expiration managers
* Add logic for using Auth.Period when handling auth login/renew requests
* Set auth.TTL if not set in handleLoginRequest
* Always set auth.TTL = te.TTL on handleLoginRequest, check TTL and period against sys values on RenewToken
* Get sysView from le.Path, revert tests
* Add back auth.Policies
* Fix TokenStore tests, add resp warning when capping values
* Use switch for ttl/period check on RenewToken
* Move comments around
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Return role info for each role on pathRoleList
* Change roles -> key_info, only return key_type
* Do not initialize result map in parseRole, refactor ListResponseWithInfo
* Add role list test
* external identity groups
* add local LDAP groups as well to group aliases
* add group aliases for okta credential backend
* Fix panic in tests
* fix build failure
* remove duplicated struct tag
* add test steps to test out removal of group member during renewals
* Add comment for having a prefix check in router
* fix tests
* s/parent_id/canonical_id
* s/parent/canonical in comments and errors
* Allow more complex errors from plugins
This enables more complex types to be registered and returned from plugins.
* Register common error types
This is a slightly less drastic change, which keeps the HTTPCodedError
as an interface.
* Remove replication error from list
* porting identity to OSS
* changes that glue things together
* add testing bits
* wrapped entity id
* fix mount error
* some more changes to core
* fix storagepacker tests
* fix some more tests
* fix mount tests
* fix http mount tests
* audit changes for identity
* remove upgrade structs on the oss side
* added go-memdb to vendor
* Lazy load plugins to avoid setup-unwrap cycle
* Remove commented blocks
* Refactor NewTestCluster, use single core cluster on basic plugin tests
* Set c.pluginDirectory in TestAddTestPlugin for setupPluginCatalog to work properly
* Add special path to mock plugin
* Move ensureCoresSealed to vault/testing.go
* Use same method for EnsureCoresSealed and Cleanup
* Bump ensureCoresSealed timeout to 60s
* Correctly handle nil opts on NewTestCluster
* Add metadata flag to APIClientMeta, use meta-enabled plugin when mounting to bootstrap
* Check metadata flag directly on the plugin process
* Plumb isMetadataMode down to PluginRunner
* Add NOOP shims when running in metadata mode
* Remove unused flag from the APIMetadata object
* Remove setupSecretPlugins and setupCredentialPlugins functions
* Move when we setup rollback manager to after the plugins are initialized
* Fix tests
* Fix merge issue
* start rollback manager after the credential setup
* Add guards against running certain client and server functions while in metadata mode
* Call initialize once a plugin is loaded on the fly
* Add more tests, update basic secret/auth plugin tests to trigger lazy loading
* Skip mount if plugin removed from catalog
* Fixup
* Remove commented line on LookupPlugin
* Fail on mount operation if plugin is re-added to catalog and mount is on existing path
* Check type and special paths on startBackend
* Fix merge conflicts
* Refactor PluginRunner run methods to use runCommon, fix TestSystemBackend_Plugin_auth
* Add automatic plugin reload
* Refactor builtin/backend
* Remove plugin reload at the core level
* Refactor plugin tests
* Add auto-reload test case
* Change backend to use sync.RWMutex, fix dangling test plugin processes
* Add a canary to plugin backends to avoid reloading many times (#3174)
* Call setupPluginCatalog before mount-related operations in postUnseal
* Don't create multiple system backends since core only holds a reference (#3176)
to one.
* Add plugin reload capability on all mounts for a specific plugin type
* Comments cleanup
* Add per-mount plugin backend reload, add tests
* Fix typos
* Remove old comment
* Reuse existing storage view in reloadPluginCommon
* Correctly handle reloading auth plugin backends
* Update path to plugin/backend/reload
* Use multierrors on reloadMatchingPluginMounts, attempt to reload all mounts provided
* Use internal value as check to ensure plugin backend reload
* Remove connection state from request for plugins at the moment
* Minor cleanup
* Refactor tests
* Store original request path in WrapInfo as CreationPath
* Add wrapping_token_creation_path to CLI output
* Add CreationPath to AuditResponseWrapInfo
* Fix tests
* Add and fix tests, update API docs with new sample responses
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
Jeff Mitchell