* Port over some SP v2 bits
Specifically:
* Add too-large handling to Physical (Consul only for now)
* Contextify some identity funcs
* Update SP protos
* Add size limiting to inmem storage
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
* Listener refactoring and file system permissions
* added listenerutil and move some common code there
* Added test for verifying socket file permissions
* Change default port of agent to 8200
* address review feedback
* Address review feedback
* Read socket options from listener config
* Implemented a warning when tls_cipher_suites includes only cipher suites which are not supprted by the HTTP/2 spec
* Added test for cipher suites
* Added hard fail on startup when all defined cipher suites are blacklisted. Added warning when some ciphers are blacklisted.
* Replaced hard failure with warning. Removed bad cipher util function and replaced it by external library.
* Added missing dependency. Fixed renaming of package name.
* Port over OSS cluster port refactor components
* Start forwarding
* Cleanup a bit
* Fix copy error
* Return error from perf standby creation
* Add some more comments
* Fix copy/paste error
* initial commit for prometheus and sys/metrics support
* Throw an error if prometheusRetentionTime is 0,add prometheus in devmode
* return when format=prometheus is used and prom is disable
* parse prometheus_retention_time from string instead of int
* Initialize config.Telemetry if nil
* address PR issues
* add sys/metrics framework.Path in a factory
* Apply requiredMountTable entries's MountConfig to existing core table
* address pr comments
* enable prometheus sink by default
* Move Metric-related code in a separate metricsutil helper
* Two things:
* Change how we populate and clear leader UUID. This fixes a case where
if a standby disconnects from an active node and reconnects, without the
active node restarting, the UUID doesn't change so triggers on a new
active node don't get run.
* Add a bunch of test helpers and minor updates to things.
* Make useCache explicit everywhere in lock manager
This also clears up a case where we could insert into the cache when it
wasn't active
* Address feedback
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
* Strip empty strings from database revocation stmts
It's technically valid to give empty strings as statements to run on
most databases. However, in the case of revocation statements, it's not
only generally inadvisable but can lead to lack of revocations when you
expect them. This strips empty strings from the array of revocation
statements.
It also makes two other changes:
* Return statements on read as empty but valid arrays rather than nulls,
so that typing information is inferred (this is more in line with the
rest of Vault these days)
* Changes field data for TypeStringSlice and TypeCommaStringSlice such
that a client-supplied value of `""` doesn't turn into `[]string{""}`
but rather `[]string{}`.
The latter and the explicit revocation statement changes are related,
and defense in depth.
* Remove DEL characters from password input
iTerm password manager sends \x03\0x7f before sending a password
from its password manager to make sure the password is not being
echoed to the screen. Unfortunately, vault login does not handle
the Space DEL sequence, causing the login to fail when using the
password manager. This patch uses a simple method to delete the
sequence if present anywhere in the string, although it is strictly
only needed at the start of input.
* Simplify iTerm handling to only remove iTerm prefix
The logic now only removes the two byte prefix sent in by iTerm
instead of trying to remove all deletes in the string.
This has been tested to work with the iTerm password manager.
As a small correction, the byte sequence is \x20\x7f. The
earlier commit message incorrectly stated it was \x03\x7f.
* Support registering plugin with name only
* Make RegisterPlugin backwards compatible
* Add CLI backwards compat command to plugin info and deregister
* Add server-side deprecation warnings if old read/dereg API endpoints are called
* Address feedback
* Add test file for testing path_restore in Transit backend. Fails because 'force' is not implemented yet
* initial implementation of 'force', to force restore of existing transit key atomically
* Initial implemntation of returning 529 for rate limits
- bump aws iam and sts packages to v1.14.31 to get mocking interface
- promote the iam and sts clients to the aws backend struct, for mocking in tests
- this also promotes some functions to methods on the Backend struct, so
that we can use the injected client
Generating creds requires reading config/root for credentials to contact
IAM. Here we make pathConfigRoot a method on aws/backend so we can clear
the clients on successful update of config/root path. Adds a mutex to
safely clear the clients
* refactor locking and unlocking into methods on *backend
* refactor/simply the locking
* check client after grabbing lock
* Disallow adding CA's serial to revocation list
* Allow disabling revocation list generation. This returns an empty (but
signed) list, but does not affect tracking of revocations so turning it
back on will populate the list properly.
* Nomad: updating max token length to 256
* Initial support for supporting custom max token name length for Nomad
* simplify/correct tests
* document nomad max_token_name_length
* removed support for max token length env var. Rename field for clarity
* cleanups after removing env var support
* move RandomWithPrefix to testhelpers
* fix spelling
* Remove default 256 value. Use zero as a sentinel value and ignore it
* update docs
* Initial work on templating
* Add check for unbalanced closing in front
* Add missing templated assignment
* Add first cut of end-to-end test on templating.
* Make template errors be 403s and finish up testing
* Review feedback
* Merge Identity Entities if two claim the same alias
Past bugs/race conditions meant two entities could be created each
claiming the same alias. There are planned longer term fixes for this
(outside of the race condition being fixed in 0.10.4) that involve
changing the data model, but this is an immediate workaround that has
the same net effect: if two entities claim the same alias, assume they
were created due to this race condition and merge them.
In this situation, also automatically merge policies so we don't lose
e.g. RGPs.
* Add request timeouts in normal request path and to expirations
* Add ability to adjust default max request duration
* Some test fixes
* Ensure tests have defaults set for max request duration
* Add context cancel checking to inmem/file
* Fix tests
* Fix tests
* Set default max request duration to basically infinity for this release for BC
* Address feedback
1) Disable MaxRetries in test cluster clients. We generally want to fail
as fast as possible in tests so adding unpredictable timing in doesn't
help things, especially if we're timing sensitive in the test.
2) EquivalentPolicies is supposed to return true if only one set
contains `default` and the other is empty, but if one set was nil
instead of simply a zero length slice it would always return false. This
means that renewing against, say, `userpass` when not actually
specifying any user policies would always fail.
* Allow max request size to be user-specified
This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.
This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.
* Switch to ioutil.ReadAll()
* Add an idle timeout for the server
Because tidy operations can be long-running, this also changes all tidy
operations to behave the same operationally (kick off the process, get a
warning back, log errors to server log) and makes them all run in a
goroutine.
This could mean a sort of hard stop if Vault gets sealed because the
function won't have the read lock. This should generally be okay
(running tidy again should pick back up where it left off), but future
work could use cleanup funcs to trigger the functions to stop.
* Fix up tidy test
* Add deadline to cluster connections and an idle timeout to the cluster server, plus add readheader/read timeout to api server
This massively simplifies transit locking behavior by pushing some
locking down to the Policy level, and embedding either a local or global
lock in the Policy depending on whether caching is enabled or not.
* Update kv command to use a preflight check
* Make the existing ui endpoint return the allowed mounts
* Add kv subcommand tests
* Enable `-field` in `vault kv get/put` (#4426)
* Enable `-field` in `vault kv get/put`
Fixes#4424
* Unify nil value handling
* Use preflight helper
* Update vkv plugin
* Add all the mount info when authenticated
* Add fix the error message on put
* add metadata test
* No need to sort the capabilities
* Remove the kv client header
* kv patch command (#4432)
* Fix test
* Fix tests
* Use permission denied instead of entity disabled
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
* Add more cli subcommands
* Add metadata commands
* Add more subcommands
* Update cli
* Move archive commands to delete
* Add helpers for making http calls to the kv backend
* rename cli header
* Format the various maps from kv
* Add list command
* Update help text
* Add a command to enable versioning on a backend
* Rename enable-versions command
* Some review feedback
* Fix listing of top level keys
* Fix issue when metadata is nil
* Add test for lising top level keys
* Fix some typos
* Add a note about deleting all versions
* Add some requirements for versioned k/v
* Add a warning message when an upgrade is triggered
* Add path help values
* Make the kv header a const
* Add the uid to mount entry instead of options map
* Pass the backend aware uuid to the mounts and plugins
* Fix comment
* Add options to secret/auth enable and tune CLI commands (#4170)
* Switch mount/tune options to use TypeKVPairs (#4171)
* switching options to TypeKVPairs, adding bool parse for versioned flag
* flipping bool check
* Fix leases coming back from non-leased pluin kv store
* add a test for updating mount options
* Fix tests
* helper/keysutil: Add a policy encrypted path storage
* Add vendored deps
* Fix spelling and paths that start with a /
* Add a key version template to change configure the ciphertext prefix
* Use big.Int for base58 instead of external lib
* Update go requirment to 1.10
* Add a version prefix cache
* Move logic to helper function
* Cache the template parts
* Add a storage prefix to policy
* Add an error if the policy passed in is nil
* Pull in the go1.10 version of the math/big package until we can update
* Add useragent package
This helper provides a consistent user-agent header for Vault, taking into account different versions.
* Add user-agent headers to spanner and gcs
* plugins/gRPC: fix issues with reserved keywords in response data
* Add the path raw file for mock plugin
* Fix panic when special paths is nil
* Add tests for Listing and raw requests from plugins
* Add json.Number case when decoding the status
* Bump the version required for gRPC defaults
* Fix test for gRPC version check
* Use version to determine plugin protocol to use
* Remove field from ServeOpts
* Fix missing assignment, handle errors
* contraint -> constraint
* Inject the version string from the vault side
* Fix the version check
* Add grpc support check to database plugins
* Default to use grpc unless missing env var or fail on contraint check
* Add GRPCSupport test
* Add greater than test case
* Add go-version dep
* Add grpc plugins
* Add grpc plugins
* Translate wrap info to/from proto
* Add nil checks
* Fix nil marshaling errors
* Provide logging through the go-plugin logger
* handle errors in the messages
* Update the TLS config so bidirectional connections work
* Add connectivity checks
* Restart plugin and add timeouts where context is not availible
* Add the response wrap data into the grpc system implementation
* Add leaseoptions to pb.Auth
* Add an error translator
* Add tests for translating the proto objects
* Fix rename of function
* Add tracing to plugins for easier debugging
* Handle plugin crashes with the go-plugin context
* Add test for grpcStorage
* Add tests for backend and system
* Bump go-plugin for GRPCBroker
* Remove RegisterLicense
* Add casing translations for new proto messages
* Use doneCtx in grpcClient
* Use doneCtx in grpcClient
* s/shutdown/shut down/
* Use SHA2-256 hash with prefix to upgrade the paths
* test the SHA1 upgrade to SHA256
* Remove hash identifier and the delimiter; use 's' instead
* Added API test to verify the correctness of the fix
* Fix broken test
* remove unneeded test
* Start work on passing context to backends
* More work on passing context
* Unindent logical system
* Unindent token store
* Unindent passthrough
* Unindent cubbyhole
* Fix tests
* use requestContext in rollback and expiration managers
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Mention api_addr on VaultPluginTLSProvider logs, update docs
* Clarify message and mention automatic api_address detection
* Change error message to use api_addr
* Change error messages to use api_addr
* encrypt/decrypt/sign/verify RSA
* update path-help and doc
* Fix the bug which was breaking convergent encryption
* support both 2048 and 4096
* update doc to contain both 2048 and 4096
* Add test for encrypt, decrypt and rotate on RSA keys
* Support exporting RSA keys
* Add sign and verify test steps
* Remove 'RSA' from PEM header
* use the default salt length
* Add 'RSA' to PEM header since openssl is expecting that
* export rsa keys as signing-key as well
* Comment the reasoning behind the PEM headers
* remove comment
* update comment
* Parameterize hashing for RSA signing and verification
* Added test steps to check hash algo choice for RSA sign/verify
* fix test by using 'prehashed'
* external identity groups
* add local LDAP groups as well to group aliases
* add group aliases for okta credential backend
* Fix panic in tests
* fix build failure
* remove duplicated struct tag
* add test steps to test out removal of group member during renewals
* Add comment for having a prefix check in router
* fix tests
* s/parent_id/canonical_id
* s/parent/canonical in comments and errors
This removes all references I could find to:
- credential provider
- authentication backend
- authentication provider
- auth provider
- auth backend
in favor of the unified:
- auth method
* porting identity to OSS
* changes that glue things together
* add testing bits
* wrapped entity id
* fix mount error
* some more changes to core
* fix storagepacker tests
* fix some more tests
* fix mount tests
* fix http mount tests
* audit changes for identity
* remove upgrade structs on the oss side
* added go-memdb to vendor
* Lazy load plugins to avoid setup-unwrap cycle
* Remove commented blocks
* Refactor NewTestCluster, use single core cluster on basic plugin tests
* Set c.pluginDirectory in TestAddTestPlugin for setupPluginCatalog to work properly
* Add special path to mock plugin
* Move ensureCoresSealed to vault/testing.go
* Use same method for EnsureCoresSealed and Cleanup
* Bump ensureCoresSealed timeout to 60s
* Correctly handle nil opts on NewTestCluster
* Add metadata flag to APIClientMeta, use meta-enabled plugin when mounting to bootstrap
* Check metadata flag directly on the plugin process
* Plumb isMetadataMode down to PluginRunner
* Add NOOP shims when running in metadata mode
* Remove unused flag from the APIMetadata object
* Remove setupSecretPlugins and setupCredentialPlugins functions
* Move when we setup rollback manager to after the plugins are initialized
* Fix tests
* Fix merge issue
* start rollback manager after the credential setup
* Add guards against running certain client and server functions while in metadata mode
* Call initialize once a plugin is loaded on the fly
* Add more tests, update basic secret/auth plugin tests to trigger lazy loading
* Skip mount if plugin removed from catalog
* Fixup
* Remove commented line on LookupPlugin
* Fail on mount operation if plugin is re-added to catalog and mount is on existing path
* Check type and special paths on startBackend
* Fix merge conflicts
* Refactor PluginRunner run methods to use runCommon, fix TestSystemBackend_Plugin_auth
* Fix the mysql legacy username length
* Remove boolean parameter
* Add a MySQL 5.6 container to test the legacy MySQL plugin against
* Add database plugins to the make file
* Fix credsutil test
* Store original request path in WrapInfo as CreationPath
* Add wrapping_token_creation_path to CLI output
* Add CreationPath to AuditResponseWrapInfo
* Fix tests
* Add and fix tests, update API docs with new sample responses
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
* Added HANA dynamic secret backend
* Added acceptance tests for HANA secret backend
* Add HANA backend as a logical backend to server
* Added documentation to HANA secret backend
* Added vendored libraries
* Go fmt
* Migrate hana credential creation to plugin
* Removed deprecated hana logical backend
* Migrated documentation for HANA database plugin
* Updated HANA DB plugin to use role name in credential generation
* Update HANA plugin tests
* If env vars are not configured, tests will skip rather than succeed
* Fixed some improperly named string variables
* Removed unused import
* Import SAP hdb driver
* Add a benchmark for exiration.Restore
* Add benchmarks for consul Restore functions
* Add a parallel version of expiration.Restore
* remove debug code
* Up the MaxIdleConnsPerHost
* Add tests for etcd
* Return errors and ensure go routines are exited
* Refactor inmem benchmark
* Add s3 bench and refactor a bit
* Few tweaks
* Fix race with waitgroup.Add()
* Fix waitgroup race condition
* Move wait above the info log
* Add helper/consts package to store consts that are needed in cyclic packages
* Remove not used benchmarks
Becca Petrin