2019-04-15 15:36:10 +00:00
|
|
|
package connutil
|
|
|
|
|
|
|
|
import (
|
|
|
|
"context"
|
|
|
|
"database/sql"
|
|
|
|
"fmt"
|
2019-07-08 23:02:54 +00:00
|
|
|
"net/url"
|
2019-04-15 15:36:10 +00:00
|
|
|
"strings"
|
|
|
|
"sync"
|
|
|
|
"time"
|
|
|
|
|
|
|
|
"github.com/hashicorp/errwrap"
|
Combined Database Backend: Static Accounts (#6834)
* Add priority queue to sdk
* fix issue of storing pointers and now copy
* update to use copy structure
* Remove file, put Item struct def. into other file
* add link
* clean up docs
* refactor internal data structure to hide heap method implementations. Other cleanup after feedback
* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods
* updates after feedback
* refactoring/renaming
* guard against pushing a nil item
* minor updates after feedback
* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC
* Initial Combined database backend implementation of static accounts and automatic rotation
* vendor updates
* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation
* add lock and setup of rotation queue
* vendor the queue
* rebase on new method signature of queue
* remove mongo tests for now
* update default role sql
* gofmt after rebase
* cleanup after rebasing to remove checks for ErrNotFound error
* rebase cdcr-priority-queue
* vendor dependencies with 'go mod vendor'
* website database docs for Static Role support
* document the rotate-role API endpoint
* postgres specific static role docs
* use constants for paths
* updates from review
* remove dead code
* combine and clarify error message for older plugins
* Update builtin/logical/database/backend.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups from feedback
* code and comment cleanups
* move db.RLock higher to protect db.GenerateCredentials call
* Return output with WALID if we failed to delete the WAL
* Update builtin/logical/database/path_creds_create.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* updates after running 'make fmt'
* update after running 'make proto'
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update comment and remove and rearrange some dead code
* Update website/source/api/secret/databases/index.html.md
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups after review
* Update sdk/database/dbplugin/grpc_transport.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* code cleanup after feedback
* remove PasswordLastSet; it's not used
* document GenerateCredentials and SetCredentials
* Update builtin/logical/database/path_rotate_credentials.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* wrap pop and popbykey in backend methods to protect against nil cred rotation queue
* use strings.HasPrefix instead of direct equality check for path
* Forgot to commit this
* updates after feedback
* re-purpose an outdated test to now check that static and dynamic roles cannot share a name
* check for unique name across dynamic and static roles
* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials
* remove commented out code
* refactor to have loadstaticwals filter out wals for roles that no longer exist
* return error if nil input given
* add nil check for input into setStaticAccount
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* add constant for queue tick time in seconds, used for comparrison in updates
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* code cleanup after review
* remove misplaced code comment
* remove commented out code
* create a queue in the Factory method, even if it's never used
* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type
* document new method
* move rotation things into a specific file
* rename test file and consolidate some static account tests
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update code comments, method names, and move more methods into rotation.go
* update comments to be capitalized
* remove the item from the queue before we try to destroy it
* findStaticWAL returns an error
* use lowercase keys when encoding WAL entries
* small cleanups
* remove vestigial static account check
* remove redundant DeleteWAL call in populate queue
* if we error on loading role, push back to queue with 10 second backoff
* poll in initqueue to make sure the backend is setup and can write/delete data
* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false
* add code comments on read-only loop
* code comment updates
* re-push if error returned from find static wal
* add locksutil and acquire locks when pop'ing from the queue
* grab exclusive locks for updating static roles
* Add SetCredentials and GenerateCredentials stubs to mockPlugin
* add a switch in initQueue to listen for cancelation
* remove guard on zero time, it should have no affect
* create a new context in Factory to pass on and use for closing the backend queue
* restore master copy of vendor dir
2019-06-19 19:45:39 +00:00
|
|
|
"github.com/hashicorp/vault/sdk/database/dbplugin"
|
2019-04-15 16:14:20 +00:00
|
|
|
"github.com/hashicorp/vault/sdk/database/helper/dbutil"
|
2019-04-15 15:36:10 +00:00
|
|
|
"github.com/hashicorp/vault/sdk/helper/parseutil"
|
|
|
|
"github.com/mitchellh/mapstructure"
|
|
|
|
)
|
|
|
|
|
|
|
|
var _ ConnectionProducer = &SQLConnectionProducer{}
|
|
|
|
|
|
|
|
// SQLConnectionProducer implements ConnectionProducer and provides a generic producer for most sql databases
|
|
|
|
type SQLConnectionProducer struct {
|
|
|
|
ConnectionURL string `json:"connection_url" mapstructure:"connection_url" structs:"connection_url"`
|
|
|
|
MaxOpenConnections int `json:"max_open_connections" mapstructure:"max_open_connections" structs:"max_open_connections"`
|
|
|
|
MaxIdleConnections int `json:"max_idle_connections" mapstructure:"max_idle_connections" structs:"max_idle_connections"`
|
|
|
|
MaxConnectionLifetimeRaw interface{} `json:"max_connection_lifetime" mapstructure:"max_connection_lifetime" structs:"max_connection_lifetime"`
|
|
|
|
Username string `json:"username" mapstructure:"username" structs:"username"`
|
|
|
|
Password string `json:"password" mapstructure:"password" structs:"password"`
|
|
|
|
|
|
|
|
Type string
|
|
|
|
RawConfig map[string]interface{}
|
|
|
|
maxConnectionLifetime time.Duration
|
|
|
|
Initialized bool
|
|
|
|
db *sql.DB
|
|
|
|
sync.Mutex
|
|
|
|
}
|
|
|
|
|
|
|
|
func (c *SQLConnectionProducer) Initialize(ctx context.Context, conf map[string]interface{}, verifyConnection bool) error {
|
|
|
|
_, err := c.Init(ctx, conf, verifyConnection)
|
|
|
|
return err
|
|
|
|
}
|
|
|
|
|
|
|
|
func (c *SQLConnectionProducer) Init(ctx context.Context, conf map[string]interface{}, verifyConnection bool) (map[string]interface{}, error) {
|
|
|
|
c.Lock()
|
|
|
|
defer c.Unlock()
|
|
|
|
|
|
|
|
c.RawConfig = conf
|
|
|
|
|
|
|
|
err := mapstructure.WeakDecode(conf, &c)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
if len(c.ConnectionURL) == 0 {
|
|
|
|
return nil, fmt.Errorf("connection_url cannot be empty")
|
|
|
|
}
|
|
|
|
|
2019-07-08 23:02:54 +00:00
|
|
|
// QueryHelper doesn't do any SQL escaping, but if it starts to do so
|
|
|
|
// then maybe we won't be able to use it to do URL substitution any more.
|
2019-04-15 15:36:10 +00:00
|
|
|
c.ConnectionURL = dbutil.QueryHelper(c.ConnectionURL, map[string]string{
|
2019-07-08 23:02:54 +00:00
|
|
|
"username": url.PathEscape(c.Username),
|
|
|
|
"password": url.PathEscape(c.Password),
|
2019-04-15 15:36:10 +00:00
|
|
|
})
|
|
|
|
|
|
|
|
if c.MaxOpenConnections == 0 {
|
2019-07-18 20:16:22 +00:00
|
|
|
c.MaxOpenConnections = 4
|
2019-04-15 15:36:10 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if c.MaxIdleConnections == 0 {
|
|
|
|
c.MaxIdleConnections = c.MaxOpenConnections
|
|
|
|
}
|
|
|
|
if c.MaxIdleConnections > c.MaxOpenConnections {
|
|
|
|
c.MaxIdleConnections = c.MaxOpenConnections
|
|
|
|
}
|
|
|
|
if c.MaxConnectionLifetimeRaw == nil {
|
|
|
|
c.MaxConnectionLifetimeRaw = "0s"
|
|
|
|
}
|
|
|
|
|
|
|
|
c.maxConnectionLifetime, err = parseutil.ParseDurationSecond(c.MaxConnectionLifetimeRaw)
|
|
|
|
if err != nil {
|
|
|
|
return nil, errwrap.Wrapf("invalid max_connection_lifetime: {{err}}", err)
|
|
|
|
}
|
|
|
|
|
|
|
|
// Set initialized to true at this point since all fields are set,
|
|
|
|
// and the connection can be established at a later time.
|
|
|
|
c.Initialized = true
|
|
|
|
|
|
|
|
if verifyConnection {
|
|
|
|
if _, err := c.Connection(ctx); err != nil {
|
|
|
|
return nil, errwrap.Wrapf("error verifying connection: {{err}}", err)
|
|
|
|
}
|
|
|
|
|
|
|
|
if err := c.db.PingContext(ctx); err != nil {
|
|
|
|
return nil, errwrap.Wrapf("error verifying connection: {{err}}", err)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return c.RawConfig, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
func (c *SQLConnectionProducer) Connection(ctx context.Context) (interface{}, error) {
|
|
|
|
if !c.Initialized {
|
|
|
|
return nil, ErrNotInitialized
|
|
|
|
}
|
|
|
|
|
|
|
|
// If we already have a DB, test it and return
|
|
|
|
if c.db != nil {
|
|
|
|
if err := c.db.PingContext(ctx); err == nil {
|
|
|
|
return c.db, nil
|
|
|
|
}
|
|
|
|
// If the ping was unsuccessful, close it and ignore errors as we'll be
|
|
|
|
// reestablishing anyways
|
|
|
|
c.db.Close()
|
|
|
|
}
|
|
|
|
|
|
|
|
// For mssql backend, switch to sqlserver instead
|
|
|
|
dbType := c.Type
|
|
|
|
if c.Type == "mssql" {
|
|
|
|
dbType = "sqlserver"
|
|
|
|
}
|
|
|
|
|
|
|
|
// Otherwise, attempt to make connection
|
|
|
|
conn := c.ConnectionURL
|
|
|
|
|
|
|
|
// Ensure timezone is set to UTC for all the connections
|
|
|
|
if strings.HasPrefix(conn, "postgres://") || strings.HasPrefix(conn, "postgresql://") {
|
|
|
|
if strings.Contains(conn, "?") {
|
|
|
|
conn += "&timezone=utc"
|
|
|
|
} else {
|
|
|
|
conn += "?timezone=utc"
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
var err error
|
|
|
|
c.db, err = sql.Open(dbType, conn)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
// Set some connection pool settings. We don't need much of this,
|
|
|
|
// since the request rate shouldn't be high.
|
|
|
|
c.db.SetMaxOpenConns(c.MaxOpenConnections)
|
|
|
|
c.db.SetMaxIdleConns(c.MaxIdleConnections)
|
|
|
|
c.db.SetConnMaxLifetime(c.maxConnectionLifetime)
|
|
|
|
|
|
|
|
return c.db, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
func (c *SQLConnectionProducer) SecretValues() map[string]interface{} {
|
|
|
|
return map[string]interface{}{
|
|
|
|
c.Password: "[password]",
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Close attempts to close the connection
|
|
|
|
func (c *SQLConnectionProducer) Close() error {
|
|
|
|
// Grab the write lock
|
|
|
|
c.Lock()
|
|
|
|
defer c.Unlock()
|
|
|
|
|
|
|
|
if c.db != nil {
|
|
|
|
c.db.Close()
|
|
|
|
}
|
|
|
|
|
|
|
|
c.db = nil
|
|
|
|
|
|
|
|
return nil
|
|
|
|
}
|
Combined Database Backend: Static Accounts (#6834)
* Add priority queue to sdk
* fix issue of storing pointers and now copy
* update to use copy structure
* Remove file, put Item struct def. into other file
* add link
* clean up docs
* refactor internal data structure to hide heap method implementations. Other cleanup after feedback
* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods
* updates after feedback
* refactoring/renaming
* guard against pushing a nil item
* minor updates after feedback
* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC
* Initial Combined database backend implementation of static accounts and automatic rotation
* vendor updates
* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation
* add lock and setup of rotation queue
* vendor the queue
* rebase on new method signature of queue
* remove mongo tests for now
* update default role sql
* gofmt after rebase
* cleanup after rebasing to remove checks for ErrNotFound error
* rebase cdcr-priority-queue
* vendor dependencies with 'go mod vendor'
* website database docs for Static Role support
* document the rotate-role API endpoint
* postgres specific static role docs
* use constants for paths
* updates from review
* remove dead code
* combine and clarify error message for older plugins
* Update builtin/logical/database/backend.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups from feedback
* code and comment cleanups
* move db.RLock higher to protect db.GenerateCredentials call
* Return output with WALID if we failed to delete the WAL
* Update builtin/logical/database/path_creds_create.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* updates after running 'make fmt'
* update after running 'make proto'
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update comment and remove and rearrange some dead code
* Update website/source/api/secret/databases/index.html.md
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups after review
* Update sdk/database/dbplugin/grpc_transport.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* code cleanup after feedback
* remove PasswordLastSet; it's not used
* document GenerateCredentials and SetCredentials
* Update builtin/logical/database/path_rotate_credentials.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* wrap pop and popbykey in backend methods to protect against nil cred rotation queue
* use strings.HasPrefix instead of direct equality check for path
* Forgot to commit this
* updates after feedback
* re-purpose an outdated test to now check that static and dynamic roles cannot share a name
* check for unique name across dynamic and static roles
* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials
* remove commented out code
* refactor to have loadstaticwals filter out wals for roles that no longer exist
* return error if nil input given
* add nil check for input into setStaticAccount
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* add constant for queue tick time in seconds, used for comparrison in updates
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* code cleanup after review
* remove misplaced code comment
* remove commented out code
* create a queue in the Factory method, even if it's never used
* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type
* document new method
* move rotation things into a specific file
* rename test file and consolidate some static account tests
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update code comments, method names, and move more methods into rotation.go
* update comments to be capitalized
* remove the item from the queue before we try to destroy it
* findStaticWAL returns an error
* use lowercase keys when encoding WAL entries
* small cleanups
* remove vestigial static account check
* remove redundant DeleteWAL call in populate queue
* if we error on loading role, push back to queue with 10 second backoff
* poll in initqueue to make sure the backend is setup and can write/delete data
* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false
* add code comments on read-only loop
* code comment updates
* re-push if error returned from find static wal
* add locksutil and acquire locks when pop'ing from the queue
* grab exclusive locks for updating static roles
* Add SetCredentials and GenerateCredentials stubs to mockPlugin
* add a switch in initQueue to listen for cancelation
* remove guard on zero time, it should have no affect
* create a new context in Factory to pass on and use for closing the backend queue
* restore master copy of vendor dir
2019-06-19 19:45:39 +00:00
|
|
|
|
|
|
|
// SetCredentials uses provided information to set/create a user in the
|
|
|
|
// database. Unlike CreateUser, this method requires a username be provided and
|
|
|
|
// uses the name given, instead of generating a name. This is used for creating
|
|
|
|
// and setting the password of static accounts, as well as rolling back
|
|
|
|
// passwords in the database in the event an updated database fails to save in
|
|
|
|
// Vault's storage.
|
|
|
|
func (c *SQLConnectionProducer) SetCredentials(ctx context.Context, statements dbplugin.Statements, staticUser dbplugin.StaticUserConfig) (username, password string, err error) {
|
|
|
|
return "", "", dbutil.Unimplemented()
|
|
|
|
}
|