* Merge Identity Entities if two claim the same alias
Past bugs/race conditions meant two entities could be created each
claiming the same alias. There are planned longer term fixes for this
(outside of the race condition being fixed in 0.10.4) that involve
changing the data model, but this is an immediate workaround that has
the same net effect: if two entities claim the same alias, assume they
were created due to this race condition and merge them.
In this situation, also automatically merge policies so we don't lose
e.g. RGPs.
* Add request timeouts in normal request path and to expirations
* Add ability to adjust default max request duration
* Some test fixes
* Ensure tests have defaults set for max request duration
* Add context cancel checking to inmem/file
* Fix tests
* Fix tests
* Set default max request duration to basically infinity for this release for BC
* Address feedback
1) Disable MaxRetries in test cluster clients. We generally want to fail
as fast as possible in tests so adding unpredictable timing in doesn't
help things, especially if we're timing sensitive in the test.
2) EquivalentPolicies is supposed to return true if only one set
contains `default` and the other is empty, but if one set was nil
instead of simply a zero length slice it would always return false. This
means that renewing against, say, `userpass` when not actually
specifying any user policies would always fail.
* Allow max request size to be user-specified
This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.
This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.
* Switch to ioutil.ReadAll()
* Add an idle timeout for the server
Because tidy operations can be long-running, this also changes all tidy
operations to behave the same operationally (kick off the process, get a
warning back, log errors to server log) and makes them all run in a
goroutine.
This could mean a sort of hard stop if Vault gets sealed because the
function won't have the read lock. This should generally be okay
(running tidy again should pick back up where it left off), but future
work could use cleanup funcs to trigger the functions to stop.
* Fix up tidy test
* Add deadline to cluster connections and an idle timeout to the cluster server, plus add readheader/read timeout to api server
This massively simplifies transit locking behavior by pushing some
locking down to the Policy level, and embedding either a local or global
lock in the Policy depending on whether caching is enabled or not.
* Update kv command to use a preflight check
* Make the existing ui endpoint return the allowed mounts
* Add kv subcommand tests
* Enable `-field` in `vault kv get/put` (#4426)
* Enable `-field` in `vault kv get/put`
Fixes#4424
* Unify nil value handling
* Use preflight helper
* Update vkv plugin
* Add all the mount info when authenticated
* Add fix the error message on put
* add metadata test
* No need to sort the capabilities
* Remove the kv client header
* kv patch command (#4432)
* Fix test
* Fix tests
* Use permission denied instead of entity disabled
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
* Add more cli subcommands
* Add metadata commands
* Add more subcommands
* Update cli
* Move archive commands to delete
* Add helpers for making http calls to the kv backend
* rename cli header
* Format the various maps from kv
* Add list command
* Update help text
* Add a command to enable versioning on a backend
* Rename enable-versions command
* Some review feedback
* Fix listing of top level keys
* Fix issue when metadata is nil
* Add test for lising top level keys
* Fix some typos
* Add a note about deleting all versions
* Add some requirements for versioned k/v
* Add a warning message when an upgrade is triggered
* Add path help values
* Make the kv header a const
* Add the uid to mount entry instead of options map
* Pass the backend aware uuid to the mounts and plugins
* Fix comment
* Add options to secret/auth enable and tune CLI commands (#4170)
* Switch mount/tune options to use TypeKVPairs (#4171)
* switching options to TypeKVPairs, adding bool parse for versioned flag
* flipping bool check
* Fix leases coming back from non-leased pluin kv store
* add a test for updating mount options
* Fix tests
* helper/keysutil: Add a policy encrypted path storage
* Add vendored deps
* Fix spelling and paths that start with a /
* Add a key version template to change configure the ciphertext prefix
* Use big.Int for base58 instead of external lib
* Update go requirment to 1.10
* Add a version prefix cache
* Move logic to helper function
* Cache the template parts
* Add a storage prefix to policy
* Add an error if the policy passed in is nil
* Pull in the go1.10 version of the math/big package until we can update
* Add useragent package
This helper provides a consistent user-agent header for Vault, taking into account different versions.
* Add user-agent headers to spanner and gcs
* plugins/gRPC: fix issues with reserved keywords in response data
* Add the path raw file for mock plugin
* Fix panic when special paths is nil
* Add tests for Listing and raw requests from plugins
* Add json.Number case when decoding the status
* Bump the version required for gRPC defaults
* Fix test for gRPC version check
* Use version to determine plugin protocol to use
* Remove field from ServeOpts
* Fix missing assignment, handle errors
* contraint -> constraint
* Inject the version string from the vault side
* Fix the version check
* Add grpc support check to database plugins
* Default to use grpc unless missing env var or fail on contraint check
* Add GRPCSupport test
* Add greater than test case
* Add go-version dep
* Add grpc plugins
* Add grpc plugins
* Translate wrap info to/from proto
* Add nil checks
* Fix nil marshaling errors
* Provide logging through the go-plugin logger
* handle errors in the messages
* Update the TLS config so bidirectional connections work
* Add connectivity checks
* Restart plugin and add timeouts where context is not availible
* Add the response wrap data into the grpc system implementation
* Add leaseoptions to pb.Auth
* Add an error translator
* Add tests for translating the proto objects
* Fix rename of function
* Add tracing to plugins for easier debugging
* Handle plugin crashes with the go-plugin context
* Add test for grpcStorage
* Add tests for backend and system
* Bump go-plugin for GRPCBroker
* Remove RegisterLicense
* Add casing translations for new proto messages
* Use doneCtx in grpcClient
* Use doneCtx in grpcClient
* s/shutdown/shut down/
* Use SHA2-256 hash with prefix to upgrade the paths
* test the SHA1 upgrade to SHA256
* Remove hash identifier and the delimiter; use 's' instead
* Added API test to verify the correctness of the fix
* Fix broken test
* remove unneeded test
* Start work on passing context to backends
* More work on passing context
* Unindent logical system
* Unindent token store
* Unindent passthrough
* Unindent cubbyhole
* Fix tests
* use requestContext in rollback and expiration managers
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Mention api_addr on VaultPluginTLSProvider logs, update docs
* Clarify message and mention automatic api_address detection
* Change error message to use api_addr
* Change error messages to use api_addr
* encrypt/decrypt/sign/verify RSA
* update path-help and doc
* Fix the bug which was breaking convergent encryption
* support both 2048 and 4096
* update doc to contain both 2048 and 4096
* Add test for encrypt, decrypt and rotate on RSA keys
* Support exporting RSA keys
* Add sign and verify test steps
* Remove 'RSA' from PEM header
* use the default salt length
* Add 'RSA' to PEM header since openssl is expecting that
* export rsa keys as signing-key as well
* Comment the reasoning behind the PEM headers
* remove comment
* update comment
* Parameterize hashing for RSA signing and verification
* Added test steps to check hash algo choice for RSA sign/verify
* fix test by using 'prehashed'
* external identity groups
* add local LDAP groups as well to group aliases
* add group aliases for okta credential backend
* Fix panic in tests
* fix build failure
* remove duplicated struct tag
* add test steps to test out removal of group member during renewals
* Add comment for having a prefix check in router
* fix tests
* s/parent_id/canonical_id
* s/parent/canonical in comments and errors
This removes all references I could find to:
- credential provider
- authentication backend
- authentication provider
- auth provider
- auth backend
in favor of the unified:
- auth method
* porting identity to OSS
* changes that glue things together
* add testing bits
* wrapped entity id
* fix mount error
* some more changes to core
* fix storagepacker tests
* fix some more tests
* fix mount tests
* fix http mount tests
* audit changes for identity
* remove upgrade structs on the oss side
* added go-memdb to vendor
* Lazy load plugins to avoid setup-unwrap cycle
* Remove commented blocks
* Refactor NewTestCluster, use single core cluster on basic plugin tests
* Set c.pluginDirectory in TestAddTestPlugin for setupPluginCatalog to work properly
* Add special path to mock plugin
* Move ensureCoresSealed to vault/testing.go
* Use same method for EnsureCoresSealed and Cleanup
* Bump ensureCoresSealed timeout to 60s
* Correctly handle nil opts on NewTestCluster
* Add metadata flag to APIClientMeta, use meta-enabled plugin when mounting to bootstrap
* Check metadata flag directly on the plugin process
* Plumb isMetadataMode down to PluginRunner
* Add NOOP shims when running in metadata mode
* Remove unused flag from the APIMetadata object
* Remove setupSecretPlugins and setupCredentialPlugins functions
* Move when we setup rollback manager to after the plugins are initialized
* Fix tests
* Fix merge issue
* start rollback manager after the credential setup
* Add guards against running certain client and server functions while in metadata mode
* Call initialize once a plugin is loaded on the fly
* Add more tests, update basic secret/auth plugin tests to trigger lazy loading
* Skip mount if plugin removed from catalog
* Fixup
* Remove commented line on LookupPlugin
* Fail on mount operation if plugin is re-added to catalog and mount is on existing path
* Check type and special paths on startBackend
* Fix merge conflicts
* Refactor PluginRunner run methods to use runCommon, fix TestSystemBackend_Plugin_auth
* Fix the mysql legacy username length
* Remove boolean parameter
* Add a MySQL 5.6 container to test the legacy MySQL plugin against
* Add database plugins to the make file
* Fix credsutil test
* Store original request path in WrapInfo as CreationPath
* Add wrapping_token_creation_path to CLI output
* Add CreationPath to AuditResponseWrapInfo
* Fix tests
* Add and fix tests, update API docs with new sample responses
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
* Added HANA dynamic secret backend
* Added acceptance tests for HANA secret backend
* Add HANA backend as a logical backend to server
* Added documentation to HANA secret backend
* Added vendored libraries
* Go fmt
* Migrate hana credential creation to plugin
* Removed deprecated hana logical backend
* Migrated documentation for HANA database plugin
* Updated HANA DB plugin to use role name in credential generation
* Update HANA plugin tests
* If env vars are not configured, tests will skip rather than succeed
* Fixed some improperly named string variables
* Removed unused import
* Import SAP hdb driver
* Add a benchmark for exiration.Restore
* Add benchmarks for consul Restore functions
* Add a parallel version of expiration.Restore
* remove debug code
* Up the MaxIdleConnsPerHost
* Add tests for etcd
* Return errors and ensure go routines are exited
* Refactor inmem benchmark
* Add s3 bench and refactor a bit
* Few tweaks
* Fix race with waitgroup.Add()
* Fix waitgroup race condition
* Move wait above the info log
* Add helper/consts package to store consts that are needed in cyclic packages
* Remove not used benchmarks
Use the stored information to validate the source address and credential issue time.
Correct the logic used to verify BoundCIDRList on the role.
Reverify the subset requirements between secret ID and role during credential issue time.
NetBSD doesn't have the right symbols defined in Go for mlockall support. The OS supports it just fine, but the definitions aren't present in Go. If someone wanted to they could add support XOR the values from `sys/mman.h` for `MCL_CURRENT | MCL_FUTURE` which is almost certainly `0x01 | 0x02` but we're not going to do that in code due to the maintenance of a one-off just for NetBSD. PR's welcome.
The `syscall` package has been frozen in favor of `x/sys`. As a result, all of the BSDs are supported and do have `mlockall(2)` support in current versions of Go.
This introduces a function that compares two string policy sets while
ignoring the presence of "default" (since it's added by core, not the
backend), and ensuring that ordering and/or duplication are not failure
conditions.
Fixes#1256
Jeff Mitchell