* Database updates
* Add create/update distinction for connection config
* Add create/update distinction for role config
* Add db name and revocation statements to leases to give revocation a
shot at working if the role has been deleted
Fixes#3544Fixes#4782
* Add create/update info to docs
* Add an idle timeout for the server
Because tidy operations can be long-running, this also changes all tidy
operations to behave the same operationally (kick off the process, get a
warning back, log errors to server log) and makes them all run in a
goroutine.
This could mean a sort of hard stop if Vault gets sealed because the
function won't have the read lock. This should generally be okay
(running tidy again should pick back up where it left off), but future
work could use cleanup funcs to trigger the functions to stop.
* Fix up tidy test
* Add deadline to cluster connections and an idle timeout to the cluster server, plus add readheader/read timeout to api server
This massively simplifies transit locking behavior by pushing some
locking down to the Policy level, and embedding either a local or global
lock in the Policy depending on whether caching is enabled or not.
* Update PKI to natively use time.Duration
Among other things this now means PKI will output durations in seconds
like other backends, instead of as Go strings.
* Add a warning when refusing to blow away an existing root instead of just returning success
* Fix another issue found while debugging this...
The reason it wasn't caught on tests in the first place is that the ttl
and max ttl were only being compared if in addition to a provided csr, a
role was also provided. This was because the check was in the role !=
nil block instead of outside of it. This has been fixed, which made the
problem occur in all sign-verbatim cases and the changes in this PR have
now verified the fix.
* add require_cn to pki roles
* add policy_identifiers and basic_constraints_valid_for_non_ca to pki role form
* add new fields to the PKI docs
* add add_basic_constraints field
* prevent returning password in reads of connection config info
* fixing a test
* masking password in connection url on reads
* addressing feedback
* removing extra check
* govet cleanup in token store
* adding general ttl handling to login requests
* consolidating TTL calculation to system view
* deprecate LeaseExtend
* deprecate LeaseExtend
* set the increment to the correct value
* move calculateTTL out of SystemView
* remove unused value
* add back clearing of lease id
* implement core ttl in some backends
* removing increment and issue time from lease options
* adding ttl tests, fixing some compile issue
* adding ttl tests
* fixing some explicit max TTL logic
* fixing up some tests
* removing unneeded test
* off by one errors...
* adding back some logic for bc
* adding period to return on renewal
* tweaking max ttl capping slightly
* use the appropriate precision for ttl calculation
* deprecate proto fields instead of delete
* addressing feedback
* moving TTL handling for backends to core
* mongo is a secret backend not auth
* adding estimated ttl for backends that also manage the expiration time
* set the estimate values before calling the renew request
* moving calculate TTL to framework, revert removal of increment and issue time from logical
* minor edits
* addressing feedback
* address more feedback
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
* redoing connection handling
* a little more cleanup
* empty implementation of rotation
* updating rotate signature
* signature update
* updating interfaces again :(
* changing back to interface
* adding templated url support and rotation for postgres
* adding correct username
* return updates
* updating statements to be a list
* adding error sanitizing middleware
* fixing log sanitizier
* adding postgres rotate test
* removing conf from rotate
* adding rotate command
* adding mysql rotate
* finishing up the endpoint in the db backend for rotate
* no more structs, just store raw config
* fixing tests
* adding db instance lock
* adding support for statement list in cassandra
* wip redoing interface to support BC
* adding falllback for Initialize implementation
* adding backwards compat for statements
* fix tests
* fix more tests
* fixing up tests, switching to new fields in statements
* fixing more tests
* adding mssql and mysql
* wrapping all the things in middleware, implementing templating for mongodb
* wrapping all db servers with error santizer
* fixing test
* store the name with the db instance
* adding rotate to cassandra
* adding compatibility translation to both server and plugin
* reordering a few things
* store the name with the db instance
* reordering
* adding a few more tests
* switch secret values from slice to map
* addressing some feedback
* reinstate execute plugin after resetting connection
* set database connection to closed
* switching secret values func to map[string]interface for potential future uses
* addressing feedback
* Max role's max_ttl parameter a TypeDurationString like ttl
* Don't clamp values at write time in favor of evaluating at issue time,
as is the current best practice
* Lots of general cleanup of logic to fix missing cases
* Accept temp creds in AWS secret backend acceptance tests
The AWS secret backend acceptance tests implicitly accepted long-lived
AWS credentials (i.e., AWS IAM user and/or root credentials) in two
ways:
1. It expected credentials to be passed in via the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables. By not accepting
AWS_SESSION_TOKEN or AWS_SECURITY_TOKEN, temporary credentials could
not be passed in. (This also forced all credentials to be passed in
via environment variables, which is a bit ugly).
2. The AWS sts:GetFederationToken call is only allowed from long-term
credentials. This is called by the Vault code which the acceptance
tests exercise.
1 is solved by deleting explicit references to credentials, which allows
the SDK to do one of the things it does best -- find credentials via the
default chain.
2 is a little more complicated. Rather than pass in whatever creds the
acceptance test was run under to the backend, the acceptance test now
creates a new IAM user and gets an access key from it, then passes the
IAM user's creds back to the backend so that it can call
sts:GetFederationToken (and then tries to clean up afterwards).
* Fix Travis build failure
The Travis build was failing because the user creation was happening
regardless of whether it was running in acceptance test mode or not.
This moves the user creation into the acceptance test precheck, which
requires lazily evaluating the credentials when configuring the backend
in the STS accetpance test, and so moving that to a PreFlight closure.
* Reduce blind sleeps in AWS secret backend acceptance tests
This removes a blind "sleep 10 seconds and then attempt to reuse the
credential" codepath and instead just keeps attemtping to reuse the
credential for 10 seconds and fails if there aren't any successful uses
after 10 seconds. This adds a few seconds speedup of acceptance test
runs from my experiments.
* added a flag to make common name optional if desired
* Cover one more case where cn can be empty
* remove skipping when empty; instead check for emptiness before calling validateNames
* Add verification before adding to DNS names to also fix#3918
* Use version to determine plugin protocol to use
* Remove field from ServeOpts
* Fix missing assignment, handle errors
* contraint -> constraint
* Inject the version string from the vault side
* Fix the version check
* Add grpc support check to database plugins
* Default to use grpc unless missing env var or fail on contraint check
* Add GRPCSupport test
* Add greater than test case
* Add go-version dep
* Add grpc plugins
* Add grpc plugins
* Translate wrap info to/from proto
* Add nil checks
* Fix nil marshaling errors
* Provide logging through the go-plugin logger
* handle errors in the messages
* Update the TLS config so bidirectional connections work
* Add connectivity checks
* Restart plugin and add timeouts where context is not availible
* Add the response wrap data into the grpc system implementation
* Add leaseoptions to pb.Auth
* Add an error translator
* Add tests for translating the proto objects
* Fix rename of function
* Add tracing to plugins for easier debugging
* Handle plugin crashes with the go-plugin context
* Add test for grpcStorage
* Add tests for backend and system
* Bump go-plugin for GRPCBroker
* Remove RegisterLicense
* Add casing translations for new proto messages
* Use doneCtx in grpcClient
* Use doneCtx in grpcClient
* s/shutdown/shut down/
* Start work on passing context to backends
* More work on passing context
* Unindent logical system
* Unindent token store
* Unindent passthrough
* Unindent cubbyhole
* Fix tests
* use requestContext in rollback and expiration managers
* oss/master:
Defer reader.Close that is used to determine sha256
changelog++
Avoid unseal failure if plugin backends fail to setup during postUnseal (#3686)
Add logic for using Auth.Period when handling auth login/renew requests (#3677)
plugins/database: use context with plugins that use database/sql package (#3691)
changelog++
Fix plaintext backup in transit (#3692)
Database gRPC plugins (#3666)
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Return role info for each role on pathRoleList
* Change roles -> key_info, only return key_type
* Do not initialize result map in parseRole, refactor ListResponseWithInfo
* Add role list test
* encrypt/decrypt/sign/verify RSA
* update path-help and doc
* Fix the bug which was breaking convergent encryption
* support both 2048 and 4096
* update doc to contain both 2048 and 4096
* Add test for encrypt, decrypt and rotate on RSA keys
* Support exporting RSA keys
* Add sign and verify test steps
* Remove 'RSA' from PEM header
* use the default salt length
* Add 'RSA' to PEM header since openssl is expecting that
* export rsa keys as signing-key as well
* Comment the reasoning behind the PEM headers
* remove comment
* update comment
* Parameterize hashing for RSA signing and verification
* Added test steps to check hash algo choice for RSA sign/verify
* fix test by using 'prehashed'
* Fix using wrong public key in sign-self-issued
* Change behavior of TTL in sign-intermediate
This allows signing CA certs with an expiration past the signer's
NotAfter.
It also change sign-self-issued to replace the Issuer, since it's
potentially RFC legal but stacks won't validate it.
Ref: https://groups.google.com/d/msg/vault-tool/giP69-n2o20/FfhRpW1vAQAJ
* Add pki/root/sign-self-issued.
This is useful for root CA rolling, and is also suitably dangerous.
Along the way I noticed we weren't setting the authority key IDs
anywhere, so I addressed that.
* Add tests
gRPC doesn't have a handler for recovering from a panic like a normal
HTTP request so a panic will actually kill Vault's listener. This
basically copies the net/http logic for managing this.
The SSH-specific logic is removed here as the underlying issue is caused
by the request forwarding mechanism.
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
A change in copystructure has caused some panics due to the custom copy
function. I'm more nervous about production panics than I am about
keeping some bad code wiping out some existing warnings, so remove the
custom copy function and just allow direct setting of Warnings.
* Initialized basic outline of TOTP backend using Postgresql backend as template
* Updated TOTP backend.go's structure and help string
* Updated TOTP path_roles.go's structure and help strings
* Updated TOTP path_role_create.go's structure and help strings
* Fixed typo in path_roles.go
* Fixed errors in path_role_create.go and path_roles.go
* Added TOTP secret backend information to cli commands
* Fixed build errors in path_roles.go and path_role_create.go
* Changed field values of period and digits from uint to int, added uint conversion of period when generating passwords
* Initialized TOTP test file based on structure of postgresql test file
* Added enforcement of input values
* Added otp library to vendor folder
* Added test steps and cleaned up errors
* Modified read credential test step, not working yet
* Use of vendored package not allowed - Test error
* Removed vendor files for TOTP library
* Revert "Removed vendor files for TOTP library"
This reverts commit fcd030994bc1741dbf490f3995944e091b11da61.
* Hopefully fixed vendor folder issue with TOTP Library
* Added additional tests for TOTP backend
* Cleaned up comments in TOTP backend_test.go
* Added default values of period, algorithm and digits to field schema
* Changed account_name and issuer fields to optional
* Removed MD5 as a hash algorithm option
* Implemented requested pull request changes
* Added ability to validate TOTP codes
* Added ability to have a key generated
* Added skew, qr size and key size parameters
* Reset vendor.json prior to merge
* Readded otp and barcode libraries to vendor.json
* Modified help strings for path_role_create.go
* Fixed test issue in testAccStepReadRole
* Cleaned up error formatting, variable names and path names. Also added some additional documentation
* Moveed barcode and url output to key creation function and did some additional cleanup based on requested changes
* Added ability to pass in TOTP urls
* Added additional tests for TOTP server functions
* Removed unused QRSize, URL and Generate members of keyEntry struct
* Removed unnecessary urlstring variable from pathKeyCreate
* Added website documentation for TOTP secret backend
* Added errors if generate is true and url or key is passed, removed logger from backend, and revised parameter documentation.
* Updated website documentation and added QR example
* Added exported variable and ability to disable QR generation, cleaned up error reporting, changed default skew value, updated documentation and added additional tests
* Updated API documentation to inlude to exported variable and qr size option
* Cleaned up return statements in path_code, added error handling while validating codes and clarified documentation for generate parameters in path_keys
Jeff Mitchell