open-vault/sdk/database/helper/connutil/sql.go

179 lines
5.3 KiB
Go
Raw Normal View History

2019-04-15 15:36:10 +00:00
package connutil
import (
"context"
"database/sql"
"fmt"
"net/url"
2019-04-15 15:36:10 +00:00
"strings"
"sync"
"time"
"github.com/hashicorp/errwrap"
Combined Database Backend: Static Accounts (#6834) * Add priority queue to sdk * fix issue of storing pointers and now copy * update to use copy structure * Remove file, put Item struct def. into other file * add link * clean up docs * refactor internal data structure to hide heap method implementations. Other cleanup after feedback * rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods * updates after feedback * refactoring/renaming * guard against pushing a nil item * minor updates after feedback * Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC * Initial Combined database backend implementation of static accounts and automatic rotation * vendor updates * initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation * add lock and setup of rotation queue * vendor the queue * rebase on new method signature of queue * remove mongo tests for now * update default role sql * gofmt after rebase * cleanup after rebasing to remove checks for ErrNotFound error * rebase cdcr-priority-queue * vendor dependencies with 'go mod vendor' * website database docs for Static Role support * document the rotate-role API endpoint * postgres specific static role docs * use constants for paths * updates from review * remove dead code * combine and clarify error message for older plugins * Update builtin/logical/database/backend.go Co-Authored-By: Jim Kalafut <jim@kalafut.net> * cleanups from feedback * code and comment cleanups * move db.RLock higher to protect db.GenerateCredentials call * Return output with WALID if we failed to delete the WAL * Update builtin/logical/database/path_creds_create.go Co-Authored-By: Jim Kalafut <jim@kalafut.net> * updates after running 'make fmt' * update after running 'make proto' * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * update comment and remove and rearrange some dead code * Update website/source/api/secret/databases/index.html.md Co-Authored-By: Jim Kalafut <jim@kalafut.net> * cleanups after review * Update sdk/database/dbplugin/grpc_transport.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * code cleanup after feedback * remove PasswordLastSet; it's not used * document GenerateCredentials and SetCredentials * Update builtin/logical/database/path_rotate_credentials.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * wrap pop and popbykey in backend methods to protect against nil cred rotation queue * use strings.HasPrefix instead of direct equality check for path * Forgot to commit this * updates after feedback * re-purpose an outdated test to now check that static and dynamic roles cannot share a name * check for unique name across dynamic and static roles * refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials * remove commented out code * refactor to have loadstaticwals filter out wals for roles that no longer exist * return error if nil input given * add nil check for input into setStaticAccount * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * add constant for queue tick time in seconds, used for comparrison in updates * Update builtin/logical/database/path_roles.go Co-Authored-By: Jim Kalafut <jim@kalafut.net> * code cleanup after review * remove misplaced code comment * remove commented out code * create a queue in the Factory method, even if it's never used * update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type * document new method * move rotation things into a specific file * rename test file and consolidate some static account tests * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * update code comments, method names, and move more methods into rotation.go * update comments to be capitalized * remove the item from the queue before we try to destroy it * findStaticWAL returns an error * use lowercase keys when encoding WAL entries * small cleanups * remove vestigial static account check * remove redundant DeleteWAL call in populate queue * if we error on loading role, push back to queue with 10 second backoff * poll in initqueue to make sure the backend is setup and can write/delete data * add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false * add code comments on read-only loop * code comment updates * re-push if error returned from find static wal * add locksutil and acquire locks when pop'ing from the queue * grab exclusive locks for updating static roles * Add SetCredentials and GenerateCredentials stubs to mockPlugin * add a switch in initQueue to listen for cancelation * remove guard on zero time, it should have no affect * create a new context in Factory to pass on and use for closing the backend queue * restore master copy of vendor dir
2019-06-19 19:45:39 +00:00
"github.com/hashicorp/vault/sdk/database/dbplugin"
2019-04-15 16:14:20 +00:00
"github.com/hashicorp/vault/sdk/database/helper/dbutil"
2019-04-15 15:36:10 +00:00
"github.com/hashicorp/vault/sdk/helper/parseutil"
"github.com/mitchellh/mapstructure"
)
var _ ConnectionProducer = &SQLConnectionProducer{}
// SQLConnectionProducer implements ConnectionProducer and provides a generic producer for most sql databases
type SQLConnectionProducer struct {
ConnectionURL string `json:"connection_url" mapstructure:"connection_url" structs:"connection_url"`
MaxOpenConnections int `json:"max_open_connections" mapstructure:"max_open_connections" structs:"max_open_connections"`
MaxIdleConnections int `json:"max_idle_connections" mapstructure:"max_idle_connections" structs:"max_idle_connections"`
MaxConnectionLifetimeRaw interface{} `json:"max_connection_lifetime" mapstructure:"max_connection_lifetime" structs:"max_connection_lifetime"`
Username string `json:"username" mapstructure:"username" structs:"username"`
Password string `json:"password" mapstructure:"password" structs:"password"`
Type string
RawConfig map[string]interface{}
maxConnectionLifetime time.Duration
Initialized bool
db *sql.DB
sync.Mutex
}
func (c *SQLConnectionProducer) Initialize(ctx context.Context, conf map[string]interface{}, verifyConnection bool) error {
_, err := c.Init(ctx, conf, verifyConnection)
return err
}
func (c *SQLConnectionProducer) Init(ctx context.Context, conf map[string]interface{}, verifyConnection bool) (map[string]interface{}, error) {
c.Lock()
defer c.Unlock()
c.RawConfig = conf
err := mapstructure.WeakDecode(conf, &c)
if err != nil {
return nil, err
}
if len(c.ConnectionURL) == 0 {
return nil, fmt.Errorf("connection_url cannot be empty")
}
// QueryHelper doesn't do any SQL escaping, but if it starts to do so
// then maybe we won't be able to use it to do URL substitution any more.
2019-04-15 15:36:10 +00:00
c.ConnectionURL = dbutil.QueryHelper(c.ConnectionURL, map[string]string{
"username": url.PathEscape(c.Username),
"password": url.PathEscape(c.Password),
2019-04-15 15:36:10 +00:00
})
if c.MaxOpenConnections == 0 {
c.MaxOpenConnections = 4
2019-04-15 15:36:10 +00:00
}
if c.MaxIdleConnections == 0 {
c.MaxIdleConnections = c.MaxOpenConnections
}
if c.MaxIdleConnections > c.MaxOpenConnections {
c.MaxIdleConnections = c.MaxOpenConnections
}
if c.MaxConnectionLifetimeRaw == nil {
c.MaxConnectionLifetimeRaw = "0s"
}
c.maxConnectionLifetime, err = parseutil.ParseDurationSecond(c.MaxConnectionLifetimeRaw)
if err != nil {
return nil, errwrap.Wrapf("invalid max_connection_lifetime: {{err}}", err)
}
// Set initialized to true at this point since all fields are set,
// and the connection can be established at a later time.
c.Initialized = true
if verifyConnection {
if _, err := c.Connection(ctx); err != nil {
return nil, errwrap.Wrapf("error verifying connection: {{err}}", err)
}
if err := c.db.PingContext(ctx); err != nil {
return nil, errwrap.Wrapf("error verifying connection: {{err}}", err)
}
}
return c.RawConfig, nil
}
func (c *SQLConnectionProducer) Connection(ctx context.Context) (interface{}, error) {
if !c.Initialized {
return nil, ErrNotInitialized
}
// If we already have a DB, test it and return
if c.db != nil {
if err := c.db.PingContext(ctx); err == nil {
return c.db, nil
}
// If the ping was unsuccessful, close it and ignore errors as we'll be
// reestablishing anyways
c.db.Close()
}
// For mssql backend, switch to sqlserver instead
dbType := c.Type
if c.Type == "mssql" {
dbType = "sqlserver"
}
// Otherwise, attempt to make connection
conn := c.ConnectionURL
// Ensure timezone is set to UTC for all the connections
if strings.HasPrefix(conn, "postgres://") || strings.HasPrefix(conn, "postgresql://") {
if strings.Contains(conn, "?") {
conn += "&timezone=utc"
} else {
conn += "?timezone=utc"
}
}
var err error
c.db, err = sql.Open(dbType, conn)
if err != nil {
return nil, err
}
// Set some connection pool settings. We don't need much of this,
// since the request rate shouldn't be high.
c.db.SetMaxOpenConns(c.MaxOpenConnections)
c.db.SetMaxIdleConns(c.MaxIdleConnections)
c.db.SetConnMaxLifetime(c.maxConnectionLifetime)
return c.db, nil
}
func (c *SQLConnectionProducer) SecretValues() map[string]interface{} {
return map[string]interface{}{
c.Password: "[password]",
}
}
// Close attempts to close the connection
func (c *SQLConnectionProducer) Close() error {
// Grab the write lock
c.Lock()
defer c.Unlock()
if c.db != nil {
c.db.Close()
}
c.db = nil
return nil
}
Combined Database Backend: Static Accounts (#6834) * Add priority queue to sdk * fix issue of storing pointers and now copy * update to use copy structure * Remove file, put Item struct def. into other file * add link * clean up docs * refactor internal data structure to hide heap method implementations. Other cleanup after feedback * rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods * updates after feedback * refactoring/renaming * guard against pushing a nil item * minor updates after feedback * Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC * Initial Combined database backend implementation of static accounts and automatic rotation * vendor updates * initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation * add lock and setup of rotation queue * vendor the queue * rebase on new method signature of queue * remove mongo tests for now * update default role sql * gofmt after rebase * cleanup after rebasing to remove checks for ErrNotFound error * rebase cdcr-priority-queue * vendor dependencies with 'go mod vendor' * website database docs for Static Role support * document the rotate-role API endpoint * postgres specific static role docs * use constants for paths * updates from review * remove dead code * combine and clarify error message for older plugins * Update builtin/logical/database/backend.go Co-Authored-By: Jim Kalafut <jim@kalafut.net> * cleanups from feedback * code and comment cleanups * move db.RLock higher to protect db.GenerateCredentials call * Return output with WALID if we failed to delete the WAL * Update builtin/logical/database/path_creds_create.go Co-Authored-By: Jim Kalafut <jim@kalafut.net> * updates after running 'make fmt' * update after running 'make proto' * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * update comment and remove and rearrange some dead code * Update website/source/api/secret/databases/index.html.md Co-Authored-By: Jim Kalafut <jim@kalafut.net> * cleanups after review * Update sdk/database/dbplugin/grpc_transport.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * code cleanup after feedback * remove PasswordLastSet; it's not used * document GenerateCredentials and SetCredentials * Update builtin/logical/database/path_rotate_credentials.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * wrap pop and popbykey in backend methods to protect against nil cred rotation queue * use strings.HasPrefix instead of direct equality check for path * Forgot to commit this * updates after feedback * re-purpose an outdated test to now check that static and dynamic roles cannot share a name * check for unique name across dynamic and static roles * refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials * remove commented out code * refactor to have loadstaticwals filter out wals for roles that no longer exist * return error if nil input given * add nil check for input into setStaticAccount * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * add constant for queue tick time in seconds, used for comparrison in updates * Update builtin/logical/database/path_roles.go Co-Authored-By: Jim Kalafut <jim@kalafut.net> * code cleanup after review * remove misplaced code comment * remove commented out code * create a queue in the Factory method, even if it's never used * update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type * document new method * move rotation things into a specific file * rename test file and consolidate some static account tests * Update builtin/logical/database/path_roles.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * Update builtin/logical/database/rotation.go Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com> * update code comments, method names, and move more methods into rotation.go * update comments to be capitalized * remove the item from the queue before we try to destroy it * findStaticWAL returns an error * use lowercase keys when encoding WAL entries * small cleanups * remove vestigial static account check * remove redundant DeleteWAL call in populate queue * if we error on loading role, push back to queue with 10 second backoff * poll in initqueue to make sure the backend is setup and can write/delete data * add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false * add code comments on read-only loop * code comment updates * re-push if error returned from find static wal * add locksutil and acquire locks when pop'ing from the queue * grab exclusive locks for updating static roles * Add SetCredentials and GenerateCredentials stubs to mockPlugin * add a switch in initQueue to listen for cancelation * remove guard on zero time, it should have no affect * create a new context in Factory to pass on and use for closing the backend queue * restore master copy of vendor dir
2019-06-19 19:45:39 +00:00
// SetCredentials uses provided information to set/create a user in the
// database. Unlike CreateUser, this method requires a username be provided and
// uses the name given, instead of generating a name. This is used for creating
// and setting the password of static accounts, as well as rolling back
// passwords in the database in the event an updated database fails to save in
// Vault's storage.
func (c *SQLConnectionProducer) SetCredentials(ctx context.Context, statements dbplugin.Statements, staticUser dbplugin.StaticUserConfig) (username, password string, err error) {
return "", "", dbutil.Unimplemented()
}