Decrypting an ciphertext where its corresponding value equals empty, the payload property "plaintext" is missing in the response object. This fixes the problem by adding a new, distinct struct for decrypt batch response items where "omitempty" is not set.
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Lower the interval for rotation during tests, to make it more likely
that our five second grace period is sufficient.
* Rewrite to make the rotateCredentials ticker a configurable value.
* 'go mod vendor' for SDK changes.
* secrets/ssh: allow algorithm_signer to use the key's default algo
* add test for ed25519 key signing
* test: add role upgrade test case
* test: rename and add more test cases
* test: clean up tests cases, fix broken test case on expected error
* test: fix broken test case on expected error
* Bring over PSIRT-37 changes from ENT
* Add additional allowed headers
* Already had this one
* Change to string slice comma separated parsing
* Add allowed_sts_header_values to read output
* Only validate AWS related request headers
* one per line
* Import ordering
* Update test
* Add X-Amz-Credential
* Reorder imports
* pki: use revocationInfo.RevocationTimeUTC when revoking certs with tidy_revoked_certs set to true
* update comment
* tidy: use same time snapshot for OR comparison
We've noticed some leftover processes from vault plugins on our boxes. Some
of them were even left over from previous instances of the service and reparented
to init. This could cause issues if too many of them accumulate.
When running with TRACE logging the logs showed that there was an error return by
the call to Setup() the plugin. Looking through the code it looks like we do not
call Cleanup() in that case.
Co-authored-by: Sorin Dumitru <sorindumitru@users.noreply.github.com>
* Remove two tests from TestTransit_SignVerify_P256. These verify that
you can't verify something signed using JWS marshaling when claiming
it's ASN1, and vice versa. The problem is that we rely on unmarshaling
failing, and it doesn't always. Both encodings use base64, one with
padding, one without, so depending on the data sometimes unmarshaling
will work when we expect it to fail. It would be nice to preserve
these tests if they could be made reliable, but I didn't see an easy way,
and I don't think they add enough value to warrant greater effort.
* Restore the tests I removed, and improve the verify function to broaden the notion of errExpected.
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* add first stepwise test env, Docker, with example transit test
* update transit stepwise test
* add other tests that use stepwise
* cleanup test, make names different than just 'transit'
* return the stderr if compile fails with error
* minor cleanups
* minor cleanups
* go mod vendor
* cleanups
* remove some extra code, and un-export some fields/methods
* update vendor
* remove reference to vault.CoreConfig, which really wasn't used anyway
* update with go mod vendor
* restore Precheck method to test cases
* clean up some networking things; create networks with UUID, clean up during teardown
* vendor stepwise
* Update sdk/testing/stepwise/environments/docker/environment.go
haha thanks :D
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/environments/docker/environment.go
Great catch, thanks
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* fix redundant name
* update error message in test
* Update builtin/credential/userpass/stepwise_test.go
More explicit error checking and responding
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/logical/aws/stepwise_test.go
`test` -> `testFunc`
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/logical/transit/stepwise_test.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* fix typos
* update error messages to provide clarity
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* update error handling / collection in Teardown
* panic if GenerateUUID returns an error
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/credential/userpass/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update builtin/logical/aws/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update builtin/logical/transit/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* import ordering
* standardize on dc from rc for cluster
* lowercase name
* CreateAPIClient -> NewAPIClient
* testWait -> ensure
* go mod cleanup
* cleanups
* move fields and method around
* make start and dockerclusternode private; use better random serial number
* use better random for SerialNumber
* add a timeout to the context used for terminating the docker container
* Use a constant for the Docker client version
* rearrange import statements
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Resolve merge conflicts and updates from running a test
* move testing/_test.go over to legacy
* updates
* Add core of plugin test framework Stepwise (#9166)
* adding stepwise testing, but there are protocol buff error :/
* move file and update sdk/go.mo
* update/sync modules
* update from other branch
* update sdk/go.mod
* some cleanups after feedback
* remove enviornments from this PR
* update vendor
* change from running go mod tidy
* change from go mod tidy
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* change panic to error
* Update sdk/testing/stepwise/helpers.go
return `nil` and not `err` at the end
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Defer close() on successful Open of a file
* document the re-creation of steps
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* remove unused BarrierKeys()
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* updates from feedback
* fix return with bad arguments
* Rename things:
- StepOperation -> Operation
- StepwiseEnvironment -> Environment
- StepCheckFunc -> AssertionFunc
- step.Check -> step.Assert
* document the environment interface methods
* rename EnvironmentOptions to MountOptions
* rename Name to RegistryName
* remove ExpectError because it's redundant
* minor doc update
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* add checkShouldRun function
* remove redundant return
* remove vestigial PreCheck function
* add tt.Helper() to makeRequest
* minor code formatting and document 1-based index for log output of Steps
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* minor updates
* update sdk
* use local reference for api, vault dep
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* cleanup some defer functions
* call fatal if environment setup fails, and don't call teardown
* defer re-setting client token in makeRequest
* Move legacy logicaltest back to testhelpers
* update mods and test files with go mod tidy
* go mod vendor
* remove relative replace directives
* restore old logical test location
* move declaration to main stepwise file
* remove index var and use i+1
* add testing for write, delete paths of makeRequest
* update stepwise core testing to do request counting
* remove unused methods
* Update sdk/testing/stepwise/stepwise.go
remove dead line
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update sdk/testing/stepwise/stepwise.go
fix capitalization in code comment
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* update code comments for SkipTeardown to clarify its use
* update stepwise
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
Allows vault roles to be associated with IAM groups in the AWS
secrets engine, since IAM groups are a recommended way to manage
IAM user policies. IAM users generated against a vault role will
be added to the IAM Groups. For a credential type of
`assumed_role` or `federation_token`, the policies sent to the
corresponding AWS call (sts:AssumeRole or sts:GetFederation) will
be the policies from each group in `iam_groups` combined with the
`policy_document` and `policy_arns` parameters.
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
* Don't use string formatting to prepare queries.
We should, when possible, use the built-in params and ? format when
preparing and executing a query. This is done to prevent SQL Injection
attacks.
* Revert some changes due to failing tests, update mssql go driver
* Add docker container startup for some MSSQL tests
* Remove acceptance test flagging, add more SQL injection protection
* Refactor MSSQL prepareTestContainer to a test helper
Also, remove all ? references and convert them to @p*
* helpful errors: print not only CN but also exactly what we are comparing
* helpful errors: return different errors for non-existent and unknown keys
* helpful errors: print error about encrypted key instead of "private key not found"
* add aliasmetadata sdk helper and add to aws auth
* split into ec2_metadata and iam_metadata fields
* fix tests
* strip pointer
* add test of default metadata
* more test <3
* switch from interface to custom marshallers
* add tests for marshalling
* store nil when selected fields are default
* separate loop into pieces
* separate acc test into multiple
* Update builtin/credential/aws/path_login.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* changes from feedback
* update aws test
* refactor to also populate auth metadata
* update how jsonification is tested
* only add populated metadata values
* add auth_type to ec2 logins
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* fix: rotate root credentials for database plugins using WAL
* test: adds a test for WAL rollback logic
* fix: progress on wal rollback
* docs: updates some comments
* docs: updates some comments
* test: adds additional test coverage for WAL rollback
* chore: remove unneeded log
* style: error handling, imports, signature line wraps
* fix: always close db plugin connection
Makes "ldap operation failed" error messages a little more useful. Also
makes the errors unique so it's easier to debug where an error is coming
from when one occurs.
Revamp race test to make it aware of when tidies start/stop, and to generate dangling accessors that need cleaning up. Also run for longer but with slightly less concurrency to ensure the writes and the cleanup actually overlap.
* RSA3072 implementation in transit secrets engine
* moved new KeyType at the end of the list
So already stored keys still work properly
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* adding support for TLS 1.3 for TCP listeners
* removed test as CI uses go 1.12
* removed Cassandra support, added deprecation notice
* re-added TestTCPListener_tls13
AWS client object caches are by region. Some AWS API calls don't care
what region's client they use, but the existing getAnyRegionForAwsPartition
scheme was returning a random region, which in turn triggered maintaining many
more client objects than are necessary (e.g. 18 regions in the main AWS
partition). This can be an issue for heavy STS users bumping up against
STS rate limits, since 18 sets of creds are being cached and renewed per
STS role.
Upgrade to new official Okta sdk lib. Since it requires an API token, use old unofficial okta lib for no-apitoken case.
Update test to use newer field names. Remove obsolete test invalidated by #4798. Properly handle case where an error was expected and didn't occur.
* Switch mongodb driver to mongo-driver
* Tidy mod
* Make writeConcern private
* Implement review feedback
* Add retry functionality
* Added backoff time
* go mod vendor
* Fix failing test
* goimport
* fix aws region tests
* strip logger
* return an error, restore tests to master
* fix extra line at import
* revert changes in spacing and comments
* Update sdk/helper/awsutil/region.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* strip explicit nil value
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* Add aws metadata to identity alias
This allows for writing identity token templates that include these attributes
(And including these attributes in path templates)
* Add alias metadata asserstion to IAM login check
If a CSR contains a SAN of type otherName, encoded in UTF-8, and the signing role specifies use_csr_sans, the otherName SAN will be included in the signed cert's SAN extension.
Allow single star in allowed_other_sans to match any OtherName. Update documentation to clarify globbing behaviour.
* enable auth method item configuration in go code
* properly parse and list generated items
* make sure we only set name on attrs if a label comes from openAPI
* correctly construct paths object for method index route
* set sensitive property on password for userpass
* remove debugger statements
* pass method model to list route template to use paths on model for tabs
* update tab generation in generated item list, undo enabling userpass users
* enable openapi generated itams for certs and userpass, update ldap to no longer have action on list endpoint
* add editType to DisplayAttributes, pull tokenutil fields into field group
* show sensitive message for sensitive fields displayed in fieldGroupShow component
* grab sensitive and editType fields from displayAttrs in openapi-to-attrs util
* make sure we don't ask for paths for secret backends since that isn't setup yet
* fix styling of sensitive text for fieldGroupShow component
* update openapi-to-attrs util test to no longer include label by default, change debugger to console.err in path-help, remove dynamic ui auth methods from tab count test
* properly log errors to the console
* capitalize This value is sensitive...
* get rid of extra padding on bottom of fieldgroupshow
* make auth methods clickable and use new confirm ux
* Update sdk/framework/path.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* Update sdk/framework/path.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* add whitespace
* return intErr instead of err
* uncomment out helpUrl because we need it
* remove extra box class
* use const instead of let
* remove extra conditional since we already split the pathName later on
* ensure we request the correct url when listing generated items
* use const
* link to list and show pages
* remove dead code
* show nested item name instead of id
* add comments
* show tooltip for text-file inputs
* fix storybook
* remove extra filter
* add TODOs
* add comments
* comment out unused variables but leave them in function signature
* only link to auth methods that can be fully managed in the ui
* clean up comments
* only render tooltip if there is helpText
* rename id authMethodPath
* remove optionsForQuery since we don't need it
* add indentation
* standardize ConfirmMessage and show model name instead of id when editing
* standardize ConfirmMessage and show model name instead of id when editing
* add comments
* post to the correct updateUrl so we can edit users and groups
* use pop instead of slice
* add TODO for finding a better way to store ids
* ensure ids are handled the same way on list and show pages; fix editing and deleting
* add comment about difference between list and show urls
* use model.id instead of name since we do not need it
* remove dead code
* ensure list pages have page headers
* standardize using authMethodPath instead of method and remove dead code
* i love indentation
* remove more dead code
* use new Confirm
* show correct flash message when deleting an item
* update flash message for creating and updating
* use plus icon for creating group/user instead of an arrow
* secrets/aws: Support permissions boundaries on iam_user creds
This allows configuring Vault to attach a permissions boundary policy to
IAM users that it creates, configured on a per-Vault-role basis.
* Fix indentation of policy in docs
Use spaces instead of tabs
I probably left this cleanup commented out as part of debugging test
errors in #6789 and forgot to uncomment it, so actually cleaning up the
test user.
* secret/aws: Pass policy ARNs to AssumedRole and FederationToken roles
AWS now allows you to pass policy ARNs as well as, and in addition to,
policy documents for AssumeRole and GetFederationToken (see
https://aws.amazon.com/about-aws/whats-new/2019/05/session-permissions/).
Vault already collects policy ARNs for iam_user credential types; now it
will allow policy ARNs for assumed_role and federation_token credential
types and plumb them through to the appropriate AWS calls.
This brings along a minor breaking change. Vault roles of the
federation_token credential type are now required to have either a
policy_document or a policy_arns specified. This was implicit
previously; a missing policy_document would result in a validation error
from the AWS SDK when retrieving credentials. However, it would still
allow creating a role that didn't have a policy_document specified and
then later specifying it, after which retrieving the AWS credentials
would work. Similar workflows in which the Vault role didn't have a
policy_document specified for some period of time, such as deleting the
policy_document and then later adding it back, would also have worked
previously but will now be broken.
The reason for this breaking change is because a credential_type of
federation_token without either a policy_document or policy_arns
specified will return credentials that have equivalent permissions to
the credentials the Vault server itself is using. This is quite
dangerous (e.g., it could allow Vault clients access to retrieve
credentials that could modify Vault's underlying storage) and so should
be discouraged. This scenario is still possible when passing in an
appropriate policy_document or policy_arns parameter, but clients should
be explicitly aware of what they are doing and opt in to it by passing
in the appropriate role parameters.
* Error out on dangerous federation token retrieval
The AWS secrets role code now disallows creation of a dangerous role
configuration; however, pre-existing roles could have existed that would
trigger this now-dangerous code path, so also adding a check for this
configuration at credential retrieval time.
* Run makefmt
* Fix tests
* Fix comments/docs
This refactors role validation for the AWS secrets engine to be in a
separate method. Previously, all validation was interspersed with the
parsing of parameters when creating/updating a role, which led to a high
degree of complexity. Now, all validation is centralized which makes it
easier to understand and also easier to test (and so a number of test
cases have been added).
* Add maximum amount of random characters requested at any given time
* Readability changes
* Removing sys/tools/random from the default policy
* Setting the maxBytes value as const
* Declaring maxBytes in the package to use it everywhere
* Using maxBytes in the error message
* upgrade aws roles
* test upgrade aws roles
* Initialize aws credential backend at mount time
* add a TODO
* create end-to-end test for builtin/credential/aws
* fix bug in initializer
* improve comments
* add Initialize() to logical.Backend
* use Initialize() in Core.enableCredentialInternal()
* use InitializeRequest to call Initialize()
* improve unit testing for framework.Backend
* call logical.Backend.Initialize() from all of the places that it needs to be called.
* implement backend.proto changes for logical.Backend.Initialize()
* persist current role storage version when upgrading aws roles
* format comments correctly
* improve comments
* use postUnseal funcs to initialize backends
* simplify test suite
* improve test suite
* simplify logic in aws role upgrade
* simplify aws credential initialization logic
* simplify logic in aws role upgrade
* use the core's activeContext for initialization
* refactor builtin/plugin/Backend
* use a goroutine to upgrade the aws roles
* misc improvements and cleanup
* do not run AWS role upgrade on DR Secondary
* always call logical.Backend.Initialize() when loading a plugin.
* improve comments
* on standbys and DR secondaries we do not want to run any kind of upgrade logic
* fix awsVersion struct
* clarify aws version upgrade
* make the upgrade logic for aws auth more explicit
* aws upgrade is now called from a switch
* fix fallthrough bug
* simplify logic
* simplify logic
* rename things
* introduce currentAwsVersion const to track aws version
* improve comments
* rearrange things once more
* conglomerate things into one function
* stub out aws auth initialize e2e test
* improve aws auth initialize e2e test
* finish aws auth initialize e2e test
* tinker with aws auth initialize e2e test
* tinker with aws auth initialize e2e test
* tinker with aws auth initialize e2e test
* fix typo in test suite
* simplify logic a tad
* rearrange assignment
* Fix a few lifecycle related issues in #7025 (#7075)
* Fix panic when plugin fails to load
* Implement SetCredentials for MongoDB, adding support for static accounts
* rework SetCredentials to split from CreateUser, and to parse the url for database
* Add integration test for mongodb static account rotation
* check the length of the password results to avoid out-of-bounds
* remove unused method
* use the pre-existing test helper for this. Add parse method to helper
* remove unused command
This was inspired by #7022 but has the advantage of avoiding
double-locking and needing to perform lock upgrades while also
simplifying the logic and being faster.
Original, #7022, this:
goos: linux
goarch: amd64
pkg: github.com/hashicorp/vault/builtin/audit/file
BenchmarkAuditFile_request-4 30000 60734 ns/op
PASS
ok github.com/hashicorp/vault/builtin/audit/file 2.428s
goos: linux
goarch: amd64
pkg: github.com/hashicorp/vault/builtin/audit/file
BenchmarkAuditFile_request-4 50000 34772 ns/op
PASS
ok github.com/hashicorp/vault/builtin/audit/file 2.086s
goos: linux
goarch: amd64
pkg: github.com/hashicorp/vault/builtin/audit/file
BenchmarkAuditFile_request-4 50000 25302 ns/op
PASS
ok github.com/hashicorp/vault/builtin/audit/file 1.542s
Fixes#7014Closes#7022
If only a non-_token field is provided we don't want to clear out the
Token version of the params, we want to set both. Otherwise we can't
rely on using the Token version of the parameter when creating the Auth
struct.
* remove create/update database user for static accounts
* update tests after create/delete removed
* small cleanups
* update postgresql setcredentials test
* add kmip engine
* adjust where kmip engine is mounted and sketch out routes
* add secret mount path service to share params to engines
* move list-controller and list-route mixins to core addon and adjust imports
* properly link kmip secrets from the secrets list page
* tweak routes and add list controllers
* stub out some models and adapters
* fix mixin exports
* move a bunch of components into the core addon
* use new empty yield in list-view in the namespace template
* scopes list using list-view and list-item components
* simplify and flatten routes, templates for all of the list pages
* role show route and template and scope create template
* add ember-router-helpers
* add more packages to the dependencies of the core addon
* add field-group-show component for listing fields from a model
* move more components to the shared addon
* make configure and configuration routes work and save a generated model
* save and list scopes
* role create, list, read
* list credentials properly
* move allowed attributes to field group
* show allowed operations on role details page
* add kmip logo to mount secrets engine list page
* add role edit page
* show all model attributes on role show page
* enable role edit
* fix newFields error by creating open api role model on the role list route
* only show selected fields on role edit page
* do not send scope and backend attrs to api
* move path-or-array to core addon
* move string-list component to core addon
* remove extra top border when there is only one field group
* add icons for all of the list pages
* update kmip config model so defaultValue doesn't error
* generate credentials
* credential create and show
* only show kmip when feature is enabled
* fix saving of TTL fields generated from Open API
* move masked-input and list-pagination components to core addon
* add param on edit form to allow for calling onSave after render happens
* polish credential show page and redirect there after generating credentials
* add externalLink for kmip engine
* add kmip-breadcrumb component
* use kmip-breadcrumb component
* add linkPrefix param to linked-block component to allow for routing programmatically inside an engine
* redirect to the right place when enabling kmip
* fix linting
* review feedback
* update signature for path-help usage
* fix ttl field expansion test
* remove role filed from role form, fix generate redirect
* remove field-group-show because it's in the core addon
* remove bottom rule from show pages
* fix Max TTL displayAttrs for ssh role
* update edit-form to take fields or attrs
* fix linting
* remove listenAddrs and set default val on ttl if a val is passed in
* Use a role cache to avoid separate locking paths
Due to the various locked/nonlocked paths we had a case where we weren't
always checking for secondary status before trying to upgrade. This
broadly simplifies things by using a cache to store the current role
values (avoiding a lot of storage hits) and updating the cache on any
write, delete, or invalidation.
* Add priority queue to sdk
* fix issue of storing pointers and now copy
* update to use copy structure
* Remove file, put Item struct def. into other file
* add link
* clean up docs
* refactor internal data structure to hide heap method implementations. Other cleanup after feedback
* rename PushItem and PopItem to just Push/Pop, after encapsulating the heap methods
* updates after feedback
* refactoring/renaming
* guard against pushing a nil item
* minor updates after feedback
* Add SetCredentials, GenerateCredentials gRPC methods to combined database backend gPRC
* Initial Combined database backend implementation of static accounts and automatic rotation
* vendor updates
* initial implementation of static accounts with Combined database backend, starting with PostgreSQL implementation
* add lock and setup of rotation queue
* vendor the queue
* rebase on new method signature of queue
* remove mongo tests for now
* update default role sql
* gofmt after rebase
* cleanup after rebasing to remove checks for ErrNotFound error
* rebase cdcr-priority-queue
* vendor dependencies with 'go mod vendor'
* website database docs for Static Role support
* document the rotate-role API endpoint
* postgres specific static role docs
* use constants for paths
* updates from review
* remove dead code
* combine and clarify error message for older plugins
* Update builtin/logical/database/backend.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups from feedback
* code and comment cleanups
* move db.RLock higher to protect db.GenerateCredentials call
* Return output with WALID if we failed to delete the WAL
* Update builtin/logical/database/path_creds_create.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* updates after running 'make fmt'
* update after running 'make proto'
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update comment and remove and rearrange some dead code
* Update website/source/api/secret/databases/index.html.md
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* cleanups after review
* Update sdk/database/dbplugin/grpc_transport.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* code cleanup after feedback
* remove PasswordLastSet; it's not used
* document GenerateCredentials and SetCredentials
* Update builtin/logical/database/path_rotate_credentials.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* wrap pop and popbykey in backend methods to protect against nil cred rotation queue
* use strings.HasPrefix instead of direct equality check for path
* Forgot to commit this
* updates after feedback
* re-purpose an outdated test to now check that static and dynamic roles cannot share a name
* check for unique name across dynamic and static roles
* refactor loadStaticWALs to return a map of name/setCredentialsWAL struct to consolidate where we're calling set credentials
* remove commented out code
* refactor to have loadstaticwals filter out wals for roles that no longer exist
* return error if nil input given
* add nil check for input into setStaticAccount
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* add constant for queue tick time in seconds, used for comparrison in updates
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Jim Kalafut <jim@kalafut.net>
* code cleanup after review
* remove misplaced code comment
* remove commented out code
* create a queue in the Factory method, even if it's never used
* update path_roles to use a common set of fields, with specific overrides for dynamic/static roles by type
* document new method
* move rotation things into a specific file
* rename test file and consolidate some static account tests
* Update builtin/logical/database/path_roles.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update builtin/logical/database/rotation.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* update code comments, method names, and move more methods into rotation.go
* update comments to be capitalized
* remove the item from the queue before we try to destroy it
* findStaticWAL returns an error
* use lowercase keys when encoding WAL entries
* small cleanups
* remove vestigial static account check
* remove redundant DeleteWAL call in populate queue
* if we error on loading role, push back to queue with 10 second backoff
* poll in initqueue to make sure the backend is setup and can write/delete data
* add revoke_user_on_delete flag to allow users to opt-in to revoking the static database user on delete of the Vault role. Default false
* add code comments on read-only loop
* code comment updates
* re-push if error returned from find static wal
* add locksutil and acquire locks when pop'ing from the queue
* grab exclusive locks for updating static roles
* Add SetCredentials and GenerateCredentials stubs to mockPlugin
* add a switch in initQueue to listen for cancelation
* remove guard on zero time, it should have no affect
* create a new context in Factory to pass on and use for closing the backend queue
* restore master copy of vendor dir
* transit cache is an Interface implemented by wrapped versions of syncmap and golang-lru
* transit cache is an Interface implemented by wrapped versions of syncmap and golang-lru
* changed some import paths to point to sdk
* Apply suggestions from code review
Co-Authored-By: Lexman42 <Lexman42@users.noreply.github.com>
* updates docs with information on transit/cache-config endpoint
* updates vendored files
* fixes policy tests to actually use a cache where expected and renames the struct and storage path used for cache configurations to be more generic
* updates document links
* fixed a typo in a documentation link
* changes cache_size to just size for the cache-config endpoint
Move audit.LogInput to sdk/logical. Allow the Data values in audited
logical.Request and Response to implement OptMarshaler, in which case
we delegate hashing/serializing responsibility to them. Add new
ClientCertificateSerialNumber audit request field.
SystemView can now be cast to ExtendedSystemView to expose the Auditor
interface, which allows submitting requests and responses to the audit
broker.
Break dataBundle into two pieces: inputBundle, which contains data that
is specific to the pki backend, and creationBundle, which is a more
generic bundle of validated inputs given to certificate creation/signing routines.
Move functions that only take creationBundle to certutil and make them public.
Merge both functions for creating mongodb containers into one.
Add retries to docker container cleanups.
Require $VAULT_ACC be set to enable AWS tests.
* add explanation to the database interface
* wordsmithing
* wrap comments, add comments for Type and Close methods
* will be stored, not with be stored
* update comment on the Type method
Append call in form of `append(s)` has no effect,
it just returns `s`. Sometimes such invocation is a sign
of a programming error, so it's better to remove these.
Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
* Fix SSH zero address OTP delete
Fixed bug where SSH OTP roles could not be deleted if a zero-address role
previously existed, and there currently exist no zero-address roles.
Fixes#6382
* Eliminate zeroAddressRoles remove function
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
* Strip empty strings from database revocation stmts
It's technically valid to give empty strings as statements to run on
most databases. However, in the case of revocation statements, it's not
only generally inadvisable but can lead to lack of revocations when you
expect them. This strips empty strings from the array of revocation
statements.
It also makes two other changes:
* Return statements on read as empty but valid arrays rather than nulls,
so that typing information is inferred (this is more in line with the
rest of Vault these days)
* Changes field data for TypeStringSlice and TypeCommaStringSlice such
that a client-supplied value of `""` doesn't turn into `[]string{""}`
but rather `[]string{}`.
The latter and the explicit revocation statement changes are related,
and defense in depth.
This PR fix#5778.
Easy test case to reproduce the problem:
https://play.golang.org/p/CAMdrOHT7C1
Since `certStr` is empty string during first iteration `strings.Join()`
will merge empty line with first CA cert.
Extra `strings.TrimSpace` call will remove that empty line, before
certificate will be return.
* Adding support for Consul 1.4 ACL system
* Working tests
* Fixed logic gate
* Fixed logical gate that evaluate empty policy or empty list of policy names
* Ensure tests are run against appropiate Consul versions
* Running tests against official container with a 1.4.0-rc1 tag
* policies can never be nil (as even if it is empty will be an empty array)
* addressing feedback, refactoring tests
* removing cast
* converting old lease field to ttl, adding max ttl
* cleanup
* adding missing test
* testing wrong version
* adding support for local tokens
* addressing feedback
* Change ordering of user lookup vs. password hashing
This fixes a very minor information leak where someone could brute force
the existence of a username. It's not perfect as the underlying storage
plays a part but bcrypt's slowness puts that much more in the noise.
* Added role-option max_sts_ttl to cap TTL for AWS STS credentials.
* Allow for setting max_sts_ttl to 0 after it has been set already.
* Fixed message in error response for default_sts_ttl > max_sts_ttl.
This endpoint eventually goes through generateCreationBundle where we
already have the right checks.
Also add expiration to returned value to match output when using root
generation.
Fixes#5549
* Set allowed OIDs to any value when generaing a CA.
Also, allow utf-8 in addition to utf8 as the OID type specifier, and
allow `*` to specify any OID of a supported type.
* Update PKI docs
* Add ability to configure the NotBefore property of certificates in role api
* Update index.html.md
* converting field to time.Duration
* setting default back to 30s
* renaming the parameter not_before_duration to differentiate between the NotBefore datetime on the cert
* Update description
* Allow specifying role-default TTLs in AWS secret engine
* Add an acceptance test
* Add docs for AWS secret role-default TTLs
* Rename default_ttl to default_sts_ttl
* Return default_ttl as int64 instead of time.Duration
* Fix broken tests
The merge of #5383 broke the tests due to some changes in the test style
that didn't actually cause a git merge conflict. This updates the tests
to the new style.
Specifying the `allowed_organiztaional_units` parameter to a cert auth
backend role will require client certificates to contain at least one of
a list of one or more "organizational units" (OU).
Example use cases:
Certificates are issued to entities in an organization arrangement by
organizational unit (OU). The OU may be a department, team, or any other logical
grouping of resources with similar roles. The entities within the OU
should be granted the same policies.
```
$ vault write auth/cert/certs/ou-engineering \
certificate=@ca.pem \
policies=engineering \
allowed_organiztaional_units=engineering
$ vault write auth/cert/certs/ou-engineering \
certificate=@ca.pem \
policies=engineering \
allowed_organiztaional_units=engineering,support
```
* logical/aws: Harden WAL entry creation
If AWS IAM user creation failed in any way, the WAL corresponding to the
IAM user would get left around and Vault would try to roll it back.
However, because the user never existed, the rollback failed. Thus, the
WAL would essentially get "stuck" and Vault would continually attempt to
roll it back, failing every time. A similar situation could arise if the
IAM user that Vault created got deleted out of band, or if Vault deleted
it but was unable to write the lease revocation back to storage (e.g., a
storage failure).
This attempts to harden it in two ways. One is by deleting the WAL log
entry if the IAM user creation fails. However, the WAL deletion could
still fail, and this wouldn't help where the user is deleted out of
band, so second, consider the user rolled back if the user just doesn't
exist, under certain circumstances.
Fixes#5190
* Fix segfault in expiration unit tests
TestExpiration_Tidy was passing in a leaseEntry that had a nil Secret,
which then caused a segfault as the changes to revokeEntry didn't check
whether Secret was nil; this is probably unlikely to occur in real life,
but good to be extra cautious.
* Fix potential segfault
Missed the else...
* Respond to PR feedback
* auth/aws: Make identity alias configurable
This is inspired by #4178, though not quite exactly what is requested
there. Rather than just use RoleSessionName as the Identity alias, the
full ARN is uses as the Alias. This mitigates against concerns that an
AWS role with an insufficiently secured trust policy could allow an
attacker to generate arbitrary RoleSessionNames in AssumeRole calls to
impersonate anybody in the Identity store that had an alias set up.
By using the full ARN, the owner of the identity store has to explicitly
trust specific AWS roles in specific AWS accounts to generate an
appropriate RoleSessionName to map back to an identity.
Fixes#4178
* Respond to PR feedback
* Remove CreateOperation
Response to PR feedback
* Add AWS Secret Engine Root Credential Rotation
This allows the AWS Secret Engine to rotate its credentials used to
access AWS. This will only work when the AWS Secret Engine has been
provided explicit IAM credentials via the config/root endpoint, and
further, when the IAM credentials provided are the only access key on
the IAM user associated wtih the access key (because AWS allows a
maximum of 2 access keys per user).
Fixes#4385
* Add test for AWS root credential rotation
Also fix a typo in the root credential rotation code
* Add docs for AWS root rotation
* Add locks around reading and writing config/root
And wire the backend up in a bunch of places so the config can get the
lock
* Respond to PR feedback
* Fix casing in error messages
* Fix merge errors
* Fix locking bugs
* Parallelize a couple AWS acceptance tests
Starting an effort to paralleize AWS secret engine acceptance tests.
Currently they take over a minute to run, and this parallelizes the two
that explicitly call a 10-second sleep, reulting in a 10-second speedup
in test time.
* Parameterize IAM user name
Probably not needed, but future-proofing the code
* Make remainder of tests parallel
AWS_ACCOUNT_ID environment variable is no longer being used; global
mutable state is a recipe for disaster when trying to run things in
parallel, and parallelizing the tests exposed a race condition in which
they were depending on the AWS_ACCOUNT_ID environment variable to be set
before they were run.
AWS_DEFAULT_REGION is still left as an environment variable because it
is required by AWS SDKs, but its configuration is now protected by a
sync.Once to ensure it only ever gets called a single time.
* Replace generateUnique*Name with testhelpers method
* Add test file for testing path_restore in Transit backend. Fails because 'force' is not implemented yet
* initial implementation of 'force', to force restore of existing transit key atomically
* Initial implemntation of returning 529 for rate limits
- bump aws iam and sts packages to v1.14.31 to get mocking interface
- promote the iam and sts clients to the aws backend struct, for mocking in tests
- this also promotes some functions to methods on the Backend struct, so
that we can use the injected client
Generating creds requires reading config/root for credentials to contact
IAM. Here we make pathConfigRoot a method on aws/backend so we can clear
the clients on successful update of config/root path. Adds a mutex to
safely clear the clients
* refactor locking and unlocking into methods on *backend
* refactor/simply the locking
* check client after grabbing lock
Update AWS Auth backend to use TypeHeader for iam request headers
- Remove parseIamRequestHeaders function and test, no longer needed with new TypeHeader
- Update AWS auth login docs
This will cause them to be removed even if they have not expired yet,
whereas before it would simply leave them in the store until they were
expired, but remove from revocation info.
* Disallow adding CA's serial to revocation list
* Allow disabling revocation list generation. This returns an empty (but
signed) list, but does not affect tracking of revocations so turning it
back on will populate the list properly.
* Nomad: updating max token length to 256
* Initial support for supporting custom max token name length for Nomad
* simplify/correct tests
* document nomad max_token_name_length
* removed support for max token length env var. Rename field for clarity
* cleanups after removing env var support
* move RandomWithPrefix to testhelpers
* fix spelling
* Remove default 256 value. Use zero as a sentinel value and ignore it
* update docs
* Make AWS credential types more explicit
The AWS secret engine had a lot of confusing overloading with role
paramemters and how they mapped to each of the three credential types
supported. This now adds parameters to remove the overloading while
maintaining backwards compatibility.
With the change, it also becomes easier to add other feature requests.
Attaching multiple managed policies to IAM users and adding a policy
document to STS AssumedRole credentials is now also supported.
Fixes#4229Fixes#3751Fixes#2817
* Add missing write action to STS endpoint
* Allow unsetting policy_document with empty string
This allows unsetting the policy_document by passing in an empty string.
Previously, it would fail because the empty string isn't a valid JSON
document.
* Respond to some PR feedback
* Refactor and simplify role reading/upgrading
This gets rid of the duplicated role upgrade code between both role
reading and role writing by handling the upgrade all in the role
reading.
* Eliminate duplicated AWS secret test code
The testAccStepReadUser and testAccStepReadSTS were virtually identical,
so they are consolidated into a single method with the path passed in.
* Switch to use AWS ARN parser
The backwards compatibility logic was preventing updates to role
statements from taking effect. This change removes persistence of
deprecated statement fields.
1) In backends, ensure they are now using TokenPolicies
2) Don't reassign auth.Policies until after expmgr registration as we
don't need them at that point
Fixes#4829
