This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Conditionally overwrite TLS parameters in MySQL DSN
Overwrite MySQL TLS configuration in MySQL DSN only if have `tls_ca` or `tls_certificate_key` set
Current logic always overwrites it
* Add test for MySQL DSN with a valid TLS parameter in query string
* raft: initial work on raft ha storage support
* add note on join
* add todo note
* raft: add support for bootstrapping and joining existing nodes
* raft: gate bootstrap join by reading leader api address from storage
* raft: properly check for raft-only for certain conditionals
* raft: add bootstrap to api and cli
* raft: fix bootstrap cli command
* raft: add test for setting up new cluster with raft HA
* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends
* raft: add test for updating an existing cluster to use raft HA
* raft: remove debug log lines, clean up verifyRaftPeers
* raft: minor cleanup
* raft: minor cleanup
* Update physical/raft/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/logical_system_raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* address feedback comments
* address feedback comments
* raft: refactor tls keyring logic
* address feedback comments
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* address feedback comments
* testing: fix import ordering
* raft: rename var, cleanup comment line
* docs: remove ha_storage restriction note on raft
* docs: more raft HA interaction updates with migration and recovery mode
* docs: update the raft join command
* raft: update comments
* raft: add missing isRaftHAOnly check for clearing out state set earlier
* raft: update a few ha_storage config checks
* Update command/operator_raft_bootstrap.go
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* raft: address feedback comments
* raft: fix panic when checking for config.HAStorage.Type
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update website/pages/docs/commands/operator/raft.mdx
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* raft: remove bootstrap cli command
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft: address review feedback
* raft: revert vendored sdk
* raft: don't send applied index and node ID info if we're HA-only
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* Extract certificate helpers for use in non-mongodb packages
* Created mTLS/X509 test for MySQL secrets engine.
* Ensure mysql username and passwords aren't url encoded
* Skip mTLS test for circleCI
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
* Don't use string formatting to prepare queries.
We should, when possible, use the built-in params and ? format when
preparing and executing a query. This is done to prevent SQL Injection
attacks.
* Revert some changes due to failing tests, update mssql go driver
* Add docker container startup for some MSSQL tests
* Remove acceptance test flagging, add more SQL injection protection
* Refactor MSSQL prepareTestContainer to a test helper
Also, remove all ? references and convert them to @p*
* Enable root user credential rotation in MongoDB
This takes its logic from the SetCredentials function with some changes
(ex: it's generating a password rather than taking one as a parameter).
This will error if the username isn't specified in the config. Since
Mongo defaults to unauthorized, this seemed like an easy check to make
to prevent strange behaviors when it tries to rotate the "" user.
* fix: rotate root credentials for database plugins using WAL
* test: adds a test for WAL rollback logic
* fix: progress on wal rollback
* docs: updates some comments
* docs: updates some comments
* test: adds additional test coverage for WAL rollback
* chore: remove unneeded log
* style: error handling, imports, signature line wraps
* fix: always close db plugin connection
* add test reproducing issue
* add code fixing issue
* check for END in unquoted string frags
* move delimiters inside parens
* begin checking with stmt
* PR feedback
* fix comment
* add tests with templates
* update test name
* remove unnecessary backslashes from test
* Mark deprecated plugins as deprecated
* Add redaction capability to database plugins
* Add x509 client auth
* Update vendored files
* Add integration test for x509 client auth
* Remove redaction logic pending further discussion
* Update vendored files
* Minor updates from code review
* Updated docs with x509 client auth
* Roles are required
* Disable x509 test because it doesn't work in CircleCI
* Add timeouts for container lifetime
* Fix typos
* Update Oracle DB secrets docs to show support for Static Roles
* Add warning about username case sensitivity
* Remove warning about casing
* Fix typo
Co-Authored-By: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
* database/influx: fix panic when trying to revoke user
Guard against other nil responses
* return an error if response is nil, which is unlikely but best safe than sorry
* refactor a deeply nested statement into a function
* Switch mongodb driver to mongo-driver
* Tidy mod
* Make writeConcern private
* Implement review feedback
* Add retry functionality
* Added backoff time
* go mod vendor
* Fix failing test
* goimport
Currently whenever we start a new C* session in the database plugin, we
run `LIST ALL` to determine whether we are a superuser, or otherwise
have permissions on roles. This is a fairly sensible way of checking
this, except it can be really slow when you have a lot of roles (C*
isn't so good at listing things). It's also really intensive to C* and
leads to a lot of data transfer. We've seen timeout issues when doing
this query, and can of course raise the timeout, but we'd probably
prefer to be able to switch it off.
Currently in the C* database plugin, connection validation errors, as
well as a parsing error, can lead us to return an error and never use an
open gocql session, which may in fact have many open connections. These
connections stay open forever. If you end up in an error loop due to,
for example, a problem with permissions, you will eventually exhaust
file descriptors on the machine.
We simply need to close the session if we aren't going to use it.
* Implement SetCredentials for MongoDB, adding support for static accounts
* rework SetCredentials to split from CreateUser, and to parse the url for database
* Add integration test for mongodb static account rotation
* check the length of the password results to avoid out-of-bounds
* remove unused method
* use the pre-existing test helper for this. Add parse method to helper
* remove unused command
* temp support for mysql+static accounts
* remove create/update database user for static accounts
* update tests after create/delete removed
* small cleanups
* update postgresql setcredentials test
* temp support for mysql+static accounts
* Add Static Account support to MySQL
* add note that MySQL supports static roles
* remove code comment
* tidy up tests
* Update plugins/database/mysql/mysql_test.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* Update plugins/database/mysql/mysql.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* update what password we test
* refactor CreateUser and SetCredentials to use a common helper
* add close statements for statements in loops
* remove some redundant checks in the mysql test
* use root rotation statements as default for static accounts
* missed a file save
* remove create/update database user for static accounts
* update tests after create/delete removed
* small cleanups
* update postgresql setcredentials test
* Add priority queue to sdk
* fix issue of storing pointers and now copy
* update to use copy structure
* Remove file, put Item struct def. into other file
* add link
* clean up docs
* refactor internal data structure to hide heap method implementations. Other cleanup after feedback
* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods
* updates after feedback
* refactoring/renaming
* guard against pushing a nil item
* minor updates after feedback
* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC
* Initial Combined database backend implementation of static accounts and automatic rotation
* vendor updates
* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation
* add lock and setup of rotation queue
* vendor the queue
* rebase on new method signature of queue
* remove mongo tests for now
* update default role sql
* gofmt after rebase
* cleanup after rebasing to remove checks for ErrNotFound error
* rebase cdcr-priority-queue
* vendor dependencies with 'go mod vendor'
* website database docs for Static Role support
* document the rotate-role API endpoint
* postgres specific static role docs
* use constants for paths
* updates from review
* remove dead code
* combine and clarify error message for older plugins
* Update builtin/logical/database/backend.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups from feedback
* code and comment cleanups
* move db.RLock higher to protect db.GenerateCredentials call
* Return output with WALID if we failed to delete the WAL
* Update builtin/logical/database/path_creds_create.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* updates after running 'make fmt'
* update after running 'make proto'
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update comment and remove and rearrange some dead code
* Update website/source/api/secret/databases/index.html.md
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups after review
* Update sdk/database/dbplugin/grpc_transport.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* code cleanup after feedback
* remove PasswordLastSet; it's not used
* document GenerateCredentials and SetCredentials
* Update builtin/logical/database/path_rotate_credentials.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* wrap pop and popbykey in backend methods to protect against nil cred rotation queue
* use strings.HasPrefix instead of direct equality check for path
* Forgot to commit this
* updates after feedback
* re-purpose an outdated test to now check that static and dynamic roles cannot share a name
* check for unique name across dynamic and static roles
* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials
* remove commented out code
* refactor to have loadstaticwals filter out wals for roles that no longer exist
* return error if nil input given
* add nil check for input into setStaticAccount
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* add constant for queue tick time in seconds, used for comparrison in updates
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* code cleanup after review
* remove misplaced code comment
* remove commented out code
* create a queue in the Factory method, even if it's never used
* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type
* document new method
* move rotation things into a specific file
* rename test file and consolidate some static account tests
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update code comments, method names, and move more methods into rotation.go
* update comments to be capitalized
* remove the item from the queue before we try to destroy it
* findStaticWAL returns an error
* use lowercase keys when encoding WAL entries
* small cleanups
* remove vestigial static account check
* remove redundant DeleteWAL call in populate queue
* if we error on loading role, push back to queue with 10 second backoff
* poll in initqueue to make sure the backend is setup and can write/delete data
* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false
* add code comments on read-only loop
* code comment updates
* re-push if error returned from find static wal
* add locksutil and acquire locks when pop'ing from the queue
* grab exclusive locks for updating static roles
* Add SetCredentials and GenerateCredentials stubs to mockPlugin
* add a switch in initQueue to listen for cancelation
* remove guard on zero time, it should have no affect
* create a new context in Factory to pass on and use for closing the backend queue
* restore master copy of vendor dir
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
When Vault is concurrently creating and revoking leases for MongoDB
users as part of the database secrets engine, and then loses connection
to MongoDB, it can panic. This occurrs because the RevokeUser path does
_not_ lock the mutex, but the CreateUser path does. Both threads of
execution can concurently decide to call c.session.Close() in
mongodb/connection_producer.go:119, and then mgo panics when the second
close attempt occurs.
1) In backends, ensure they are now using TokenPolicies
2) Don't reassign auth.Policies until after expmgr registration as we
don't need them at that point
Fixes#4829
* redoing connection handling
* a little more cleanup
* empty implementation of rotation
* updating rotate signature
* signature update
* updating interfaces again :(
* changing back to interface
* adding templated url support and rotation for postgres
* adding correct username
* return updates
* updating statements to be a list
* adding error sanitizing middleware
* fixing log sanitizier
* adding postgres rotate test
* removing conf from rotate
* adding rotate command
* adding mysql rotate
* finishing up the endpoint in the db backend for rotate
* no more structs, just store raw config
* fixing tests
* adding db instance lock
* adding support for statement list in cassandra
* wip redoing interface to support BC
* adding falllback for Initialize implementation
* adding backwards compat for statements
* fix tests
* fix more tests
* fixing up tests, switching to new fields in statements
* fixing more tests
* adding mssql and mysql
* wrapping all the things in middleware, implementing templating for mongodb
* wrapping all db servers with error santizer
* fixing test
* store the name with the db instance
* adding rotate to cassandra
* adding compatibility translation to both server and plugin
* reordering a few things
* store the name with the db instance
* reordering
* adding a few more tests
* switch secret values from slice to map
* addressing some feedback
* reinstate execute plugin after resetting connection
* set database connection to closed
* switching secret values func to map[string]interface for potential future uses
* addressing feedback
This was in the deprecated backend where it fixed a similar issue a long
time ago but for some reason didn't make it over. Additionally the
function wasn't being locked properly.
Hopefully fixes#2973
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Initial work on write concern support, set for the lifetime of the session
* Add base64 encoded value support, include docs and tests
* Handle error from json.Unmarshal, fix test and docs
* Remove writeConcern struct, move JSON unmarshal to Initialize
* Return error on empty mapping of write_concern into mgo.Safe struct
* database/mysql: Allow the creation statement to use commands that are not yet supported by the prepare statement protocol
* Remove unnecessary else block
* Fix cassandra tests, explicitly set cluster port if provided
* Update cassandra.yml test-fixture
* Add port as part of the config option, fix tests
* Remove hostport splitting in cassandraConnectionProducer.createSession
* Include port in API docs
Michael Golowka