* feat: DB plugin multiplexing (#13734)
* WIP: start from main and get a plugin runner from core
* move MultiplexedClient map to plugin catalog
- call sys.NewPluginClient from PluginFactory
- updates to getPluginClient
- thread through isMetadataMode
* use go-plugin ClientProtocol interface
- call sys.NewPluginClient from dbplugin.NewPluginClient
* move PluginSets to dbplugin package
- export dbplugin HandshakeConfig
- small refactor of PluginCatalog.getPluginClient
* add removeMultiplexedClient; clean up on Close()
- call client.Kill from plugin catalog
- set rpcClient when muxed client exists
* add ID to dbplugin.DatabasePluginClient struct
* only create one plugin process per plugin type
* update NewPluginClient to return connection ID to sdk
- wrap grpc.ClientConn so we can inject the ID into context
- get ID from context on grpc server
* add v6 multiplexing protocol version
* WIP: backwards compat for db plugins
* Ensure locking on plugin catalog access
- Create public GetPluginClient method for plugin catalog
- rename postgres db plugin
* use the New constructor for db plugins
* grpc server: use write lock for Close and rlock for CRUD
* cleanup MultiplexedClients on Close
* remove TODO
* fix multiplexing regression with grpc server connection
* cleanup grpc server instances on close
* embed ClientProtocol in Multiplexer interface
* use PluginClientConfig arg to make NewPluginClient plugin type agnostic
* create a new plugin process for non-muxed plugins
* feat: plugin multiplexing: handle plugin client cleanup (#13896)
* use closure for plugin client cleanup
* log and return errors; add comments
* move rpcClient wrapping to core for ID injection
* refactor core plugin client and sdk
* remove unused ID method
* refactor and only wrap clientConn on multiplexed plugins
* rename structs and do not export types
* Slight refactor of system view interface
* Revert "Slight refactor of system view interface"
This reverts commit 73d420e5cd2f0415e000c5a9284ea72a58016dd6.
* Revert "Revert "Slight refactor of system view interface""
This reverts commit f75527008a1db06d04a23e04c3059674be8adb5f.
* only provide pluginRunner arg to the internal newPluginClient method
* embed ClientProtocol in pluginClient and name logger
* Add back MLock support
* remove enableMlock arg from setupPluginCatalog
* rename plugin util interface to PluginClient
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
* feature: multiplexing: fix unit tests (#14007)
* fix grpc_server tests and add coverage
* update run_config tests
* add happy path test case for grpc_server ID from context
* update test helpers
* feat: multiplexing: handle v5 plugin compiled with new sdk
* add mux supported flag and increase test coverage
* set multiplexingSupport field in plugin server
* remove multiplexingSupport field in sdk
* revert postgres to non-multiplexed
* add comments on grpc server fields
* use pointer receiver on grpc server methods
* add changelog
* use pointer for grpcserver instance
* Use a gRPC server to determine if a plugin should be multiplexed
* Apply suggestions from code review
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* add lock to removePluginClient
* add multiplexingSupport field to externalPlugin struct
* do not send nil to grpc MultiplexingSupport
* check err before logging
* handle locking scenario for cleanupFunc
* allow ServeConfigMultiplex to dispense v5 plugin
* reposition structs, add err check and comments
* add comment on locking for cleanupExternalPlugin
Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Ensure we can issue against generated SSH CA keys
This adds a test to ensure that we can issue leaf SSH certificates using
the newly generated SSH CA keys. Presently this fails because the
ed25519 key private is stored using PKIX's PKCS8 PrivateKey object
format rather than using OpenSSH's desired private key format:
> path_config_ca_test.go:211: bad case 12: err: failed to parse stored CA private key: ssh: invalid openssh private key format, resp: <nil>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add dependency on edkey for OpenSSH ed25519 keys
As mentioned in various terraform-provider-tls discussions, OpenSSH
doesn't understand the standard OpenSSL/PKIX ed25519 key structure (as
generated by PKCS8 marshalling). Instead, we need to place it into the
OpenSSH RFC 8709 format. As mentioned in this dependency's README,
support in golang.org/x/crypto/ssh is presently lacking for this.
When the associated CL is merged, we should be able to remove this dep
and rely on the (extended) standard library, however, no review progress
appears to have been made since the CL was opened by the author.
See also: https://go-review.googlesource.com/c/crypto/+/218620/
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add generation support for other SSH CA key types
This adds two new arguments to config/ca, mirroring the values of PKI
secrets engine but tailored towards SSH mounts. Key types are specified
as x/crypto/ssh KeyAlgo identifiers (e.g., ssh-rsa or ssh-ed25519)
and respect current defaults (ssh-rsa/4096). Key bits defaults to 0,
which for ssh-rsa then takes a value of 4096.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation on key_type, key_bits for ssh/config/ca
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow all other_sans in sign-intermediate and sign-verbatim
/sign-verbatim and /sign-intermediate are more dangerous endpoints in
that they (usually) do not have an associated role. In this case, a
permissive role is constructed during execution of these tests. However,
the AllowedOtherSANs field was missing from this, prohibiting its use
when issuing certificates.
Resolves: #13157
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add duration/count metrics to PKI issue and revoke flows
* docs, changelog
* tidy
* last tidy
* remove err
* Update callsites
* Simple returns
* Handle the fact that test cases don't have namespaces
* Add mount point to the request
* fmt
* Handle empty mount point, and add it to unit tests
* improvement
* Turns out sign-verbatim is tricky, it can take a role but doesn't have to
* Get around the field schema problem
* Use application/pem-certificate-chain for PEMs
As mentioned in #10948, it appears we're incorrectly using the
`application/pkix-cert` media type for PEM blobs, when
`application/x-pem-file` is more appropriate. Per RFC 5280 Section
4.2.1.13, `application/pkix-crl` is only appropriate when the CRL is in
DER form. Likewise, Section 4.2.2.1 states that `application/pkix-cert`
is only applicable when a single DER certificate is used.
Per recommendation in RFC 8555 ("ACME"), Section 7.4.2 and 9.1, we use
the newer `application/pem-certificate-chain` media type for
certificates. However, this is not applicable for CRLs, so we use fall
back to `application/x-pem-file` for these. Notably, no official IETF
source is present for the latter. On the OpenSSL PKI tutorial
(https://pki-tutorial.readthedocs.io/en/latest/mime.html), this type is
cited as coming from S/MIME's predecessor, PEM, but neither of the main
PEM RFCs (RFC 934, 1421, 1422, 1423, or 1424) mention this type.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Include full chain in /cert/ca_chain response
This allows callers to get the full chain (including issuing
certificates) from a call to /cert/ca_chain. Previously, most endpoints
(including during issuance) do not include the root authority, requiring
an explicit call to /cert/ca to fetch. This allows full chains to be
constructed without without needing multiple calls to the API.
Resolves: #13489
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test case for full CA issuance
We test three main scenarios:
1. A root-only CA's `/cert/ca_chain`'s `.data.ca_chain` field should
contain only the root,
2. An intermediate CA (with root provide) should contain both the root
and the intermediate.
3. An external (e.g., `/config/ca`-provided) CA with both root and
intermediate should contain both certs.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation for new ca_chain field
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add note about where to find the entire chain
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Support Y10K value in notAfter field when signing non-CA certificates
* Add changelog entry for 13736
* Add test for using not_after parameter for non-CA certificates that are being signed
* Fix CA value for test for not_after value when signing non-CA certs
* Address formatting
* Add changelog file
* Revert changelog entry commit f28b54e7b5ad21144c8a2da942d766e64a332caf
* Fix a transit deadlock caused by indefinite lock holding in key autorotation.
* Move down manual policy locking in transit autorotation to avoid NPE.
* Wrap conditional transit key autorotation in a function to allow for cleaner policy lock management.
* Remove a dnagling continue statement from transit key autorotation.
* Fix upndomain bug causing alias name to change
* Fix nil map
* Add changelog
* revert
* Update changelog
* Add test for alias metadata name
* Fix code comment
* Add auto_rotate_interval field to transit key creation path.
* Add auto_rotate_interval field to transit key config update path.
* Implement transit automatic key rotation on an hourly interval.
* Fixes transit key autorotation key listing typo.
* Add unit tests for transit key autorotation.
* Add unit tests for transit key creation with autorotation interval.
* Add unit tests for transit key config update with autorotation interval.
* Document new auto_rotate_interval fields in key creation and key config update endpoints.
* Add changelog for transit key autorotation.
* Wrap individual transit key autorotation in a policy lock.
* Add a safeguard to transit key autorotation to ensure only one execution happens simultaneously.
* auth/cert: Add certificate extensions as metadata
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
* Add changelog for #13348
Signed-off-by: Peter Verraedt <peter.verraedt@kuleuven.be>
* CLI changes for new mount tune config parameter allowed_managed_keys
* Correct allowed_managed_keys description in auth and secrets
* Documentation update for secrets and removed changes for auth
* Add changelog and remove documentation changes for auth
* removed changelog
* Correct the field description
* Replace - with : when listing certificate serials
* Add allowed_uri_sans_template
Enables identity templating for the allowed_uri_sans field in PKI cert roles.
Implemented as suggested in #8509
* changelog++
* Update docs with URI SAN templating
* github auth: use org id to verify creds
* add check for required org param; add test case
* update UTs
* add nil check for org
* add changelog
* fix typo in ut
* set org ID if it is unset; add more ut coverage
* add optional organization_id
* move client instantiation
* refactor parse URL; add UT for setting org ID
* fix comment in UT
* add nil check
* don't update org name on change; return warning
* refactor verifyCredentials
* error when unable to fetch org ID on config write; add warnings
* fix bug in log message
* update UT and small refactor
* update comments and log msg
* use getter for org ID
* Allow universal default for key_bits
This allows the key_bits field to take a universal default value, 0,
which, depending on key_type, gets adjusted appropriately into a
specific default value (rsa->2048, ec->256, ignored under ed25519).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Handle universal default key size in certutil
Also move RSA < 2048 error message into certutil directly, instead of in
ca_util/path_roles.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing RSA key sizes to pki/backend_test.go
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to returning updated values
When determining the default, don't pass in pointer types, but instead
return the newly updated value.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Re-add fix for ed25519 from #13254
Ed25519 internally specifies a hash length; by changing the default from
256 to 0, we fail validation in ValidateSignatureLength(...) unless we
specify the key algorithm.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
- The test was attempting to test the convergent encryption behaviour
with several key types but the common function never used the passed
in key type. So we ran the test with the default aes256-gcm96 only.
* Warn user supplying nonce values in FIPS mode for transit encryption requests
- Send back a warning within the response if an end-user supplies nonce
values that we use within the various transit encrypt apis.
- We do not send a warning if an end-user supplies a nonce value but we
don't use it.
- Affected api methods are encrypt, rewrap and datakey
- The warning is only sent when we are operating in FIPS mode.
* Adding support for SHA3 in the transit backend.
* Adds SHA-3 tests for transit sign/verify path. Adds SHA-3 tests for logical system tools path hash functionality. Updates documentation to include SHA-3 algorithms in system tools path hashing.
* Adds changelog entry.
Co-authored-by: robison jacka <robison@packetized.io>
Update the AWS auth backend acceptance tests to account for the new
`iam_tags` field that comes back on responses.
* marked only tests requiring creds as acceptance
Renamed TestBackend_* to TestAcceptanceBackend_* if the test requires
AWS credentials. Otherwise left the name as TestBackend_* and set
`AcceptanceTest: false`.
* ensure generated names aren't too long
IAM roles and users have a 64 character limit, and adding Acceptance
to the test names was putting some over the length limit, so modified
generateUniqueName() to take a max length parameter and added
functions for each type of name generation (user, role, group).
* return pkcs8 format for ed25519 curve
convertRespToPKCS8 does not recognize the ed25519 key. Changes
to recognize ed25519 key and return its PKCS8 format
* Restrict ECDSA signatures with NIST P-Curve hashes
When using an ECDSA signature with a NIST P-Curve, we should follow
recommendations from BIS (Section 4.2) and Mozilla's root store policy
(section 5.1.2) to ensure that arbitrary selection of signature_bits
does not exceed what the curve is capable of signing.
Related: #11245
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to certutil.ValidateKeyTypeSignatureLength(...)
Replaces previous calls to certutil.ValidateKeyTypeLength(...) and
certutil.ValidateSignatureLength(...) with a single call, allowing for
curve<->hash validation.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch to autodetection of signature_bits
This enables detection of whether the caller manually specified a value
for signature_bits or not; when not manually specified, we can provision
a value that complies with new NIST P-Curve policy.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Select hash function length automatically
Due to our change in behavior (to default to -1 as the value to
signature_bits to allow for automatic hash selection), switch
ValidateKeyTypeSignatureLength(...) to accept a pointer to hashBits and
provision it with valid default values.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Prevent invalid Curve size lookups
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Switch from -1 to 0 as default SignatureBits
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
VAULT-444: Add PKI tidy-status endpoint.
Add metrics so that the PKI tidy status can be monitored using telemetry as well.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Document allow_different_signature_algorithm param
* Flip the semantics of different key types for sign self issued
* More language tweaks
* Fix the field definition description
* Rework differenttype test for the new flag
* typo
* Native Login method, userpass and approle interfaces to implement it
* Add AWS auth interface for Login, unexported struct fields for now
* Add Kubernetes client login
* Add changelog
* Add a test for approle client login
* Return errors from LoginOptions, use limited reader for secret ID
* Fix auth comment length
* Return actual type not interface, check for client token in tests
* Require specification of secret ID location using SecretID struct as AppRole arg
* Allow password from env, file, or plaintext
* Add flexibility in how to fetch k8s service token, but still with default
* Avoid passing strings that need to be validated by just having different login options
* Try a couple real tests with approle and userpass login
* Fix method name in comment
* Add context to Login methods, remove comments about certain sources being inherently insecure
* Perform read of secret ID at login time
* Read password from file at login time
* Pass context in integ tests
* Read env var values in at login time, add extra tests
* Update api version
* Revert "Update api version"
This reverts commit 1ef3949497dcf878c47e0e5ffcbc8cac1c3c1679.
* Update api version in all go.mod files
* Modernize SSH key lengths
No default change was made in this commit; note that the code already
enforced a default of 2048 bits. ssh-keygen and Go's RSA key generation
allows for key sizes including 3072, 4096, 8192; update the values of
SSH key generation to match PKI's allowed RSA key sizes (from
certutil.ValidateKeyTypeLength(...)). We still allow the legacy SSH key
size of 1024; in the near future we should likely remove it.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Ensure minimum of 2048-bit PKI RSA keys
While the stated path is a false-positive, verifying all paths is
non-trivial. We largely validate API call lengths using
certutil.ValidateKeyTypeLength(...), but ensuring no other path calls
certutil.generatePrivateKey(...) --- directly or indirectly --- is
non-trivial. Thus enforcing a minimum in this method sounds like a sane
compromise.
Resolves: https://github.com/hashicorp/vault/security/code-scanning/55
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Let allowed_users template mix templated and non-templated parts (#10388)
* Add documentation
* Change test function names
* Add documentation
* Add changelog entry
* Forbid ssh key signing with specified extensions when role allowed_extensions is not set
- This is a behaviour change on how we process the allowed_extensions role
parameter when it does not contain a value. The previous handling allowed
a client to override and specify any extension they requested.
- We now require a role to explicitly set this behaviour by setting the parameter
to a '*' value which matches the behaviour of other keys such as allowed_users
within the role.
- No migration of existing roles is provided either, so operators if they truly
want this behaviour will need to update existing roles appropriately.
* update azure instructions
Update instructions in regards to azure AD Authentication and OIDC
* Initial pass of ed25519
* Fix typos on marshal function
* test wip
* typo
* fix tests
* missef changelog
* fix mismatch between signature and algo
* added test coverage for ed25519
* remove pkcs1 since does not exist for ed25519
* add ed25519 support to getsigner
* pull request feedback
Signed-off-by: Anner J. Bonilla <abonilla@hoyosintegrity.com>
* typo on key
Signed-off-by: Anner J. Bonilla <abonilla@hoyosintegrity.com>
* cast mistake
Signed-off-by: Anner J. Bonilla <abonilla@hoyosintegrity.com>
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* core: set namespace from the sysview's mount entry on GeneratePasswordFromPolicy
* test: update TestDynamicSystemView to be ns-aware, update tests
* add changelog entry
* port of ldap fix for early cred rotation
* some more porting
* another couple lines to port
* final commits before report
* remove deadlock
* needs testing
* updates
* Sync with OpenLDAP PR
* Update the update error handling for items not found in the queue
* WIP unit tests
* Need to configure DB mount correctly, with db type mockv5
* Need to find a way to inject errors into that mock db
* throw error on role creation failure
* do not swallow error on role creation
* comment out wip tests and add in a test for disallowed role
* Use newly generated password in WAL
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* return err on popFromRotationQueueByKey error; cleanup on setStaticAccount
* test: fix TestPlugin_lifecycle
* Uncomment and fix unit tests
* Use mock database plugin to inject errors
* Tidy test code to rely less on code internals where possible
* Some stronger test assertions
* Undo logging updates
* Add changelog
* Remove ticker and background threads from WAL tests
* Keep pre-existing API behaviour of allowing update static role to act as a create
* Switch test back to update operation
* Revert my revert, and fix some test bugs
* Fix TestBackend_StaticRole_LockRegression
* clean up defer on TestPlugin_lifecycle
* unwrap reqs on cleanup
* setStaticAccount: don't hold a write lock
* TestStoredWALsCorrectlyProcessed: set replication state to unknown
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
* WIP: Unset the certificate's SignatureAlgorithm to allown cross-signing of different key types
* Allow signing self issued certs with a different public key algorithm
* Remove cruft
* Remove stale import
* changelog
* eliminate errwrap
* Add a test to cover the lack of opt-in flag
* Better comment
Co-authored-by: catsby <clint@ctshryock.com>
* Enforce Minimum cache size for transit backend
* enfore minimum cache size and log a warning during backend construction
* Update documentation for transit backend cache configuration
* Added changelog
* Addressed review feedback and added unit test
* Modify code in pathCacheConfigWrite to make use of the updated cache size
* Updated code to refresh cache size on transit backend without restart
* Update code to acquire read and write locks appropriately
This change adds the ability to set the signature algorithm of the
CAs that Vault generates and any certificates it signs. This is a
potentially useful stepping stone for a SHA3 transition down the line.
Summary:
* Adds the field "signature_bits" to CA and Sign endpoints
* Adds support for SHA256, SHA384 and SHA512 signatures on EC and RSA
keytypes.
* Fix pkcs7 parsing in some cases
brings in https://github.com/mozilla-services/pkcs7/pull/61 from upstream
In some cases but not all, aws includes a certificate in the pkcs7 response,
and currently vault fails to parse those certificates:
```
URL: PUT https://vault.example.com/v1/auth/aws/login
Code: 500. Errors
* failed to parse the BER encoded PKCS#7 signature: ber2der: Invalid BER format
```
This fixes logins on those instances. Note we could not readily ascertain why
some instances have those certificates and others don't.
* Add changelog entry
* Correct missed line
* Send x-forwarded-for in Okta Push Factor request
Why:
In order for Okta to properly report the location of the authentication
attempt, the X-Forwarded-For header must be included in the request to
Okta (if it exists).
This change addresses the need by:
* Duplicating the value of X-Forwarded-For if it's passed through to the
auth backend
* Add changelog entry for 12320
* update genUsername to cap STS usernames at 64 chars
* add changelog
* refactor tests into t.Run block
* patch: remove warningExpected bool and include expected string
* patch: revert sts to cap at 32 chars and add assume_role case in genUsername
* update changelog
* update genUsername to return error if username generated exceeds length limits
* update changelog
* add conditional default username template to provide custom STS usernames
* update changelog
* include test for failing STS length case
* update comments for more clarity
* Work around rabbitmq regression with UserInfo.Tags in rabbitmq 3.9: use v3.8 docker image in tests.
* Also pin cassandra docker image version to 3.11 (4.00 was making tests fail)
Unlike the other libraries that were migrated, there are no usages of
this lib in any of our plugins, and the only other known usage was in
go-kms-wrapping, which has been updated. Aliasing it like the other libs
would still keep the aws-sdk-go dep in the sdk module because of the
function signatures. So I've simply removed it entirely here.
* add ability to customize IAM usernames based on templates
* add changelog
* remove unnecessary logs
* patch: add test for readConfig
* patch: add default STS Template
* patch: remove unnecessary if cases
* patch: add regex checks in username test
* patch: update genUsername to return an error instead of warnings
* patch: separate tests for default and custom templates
* patch: return truncate warning from genUsername and trigger a 400 response on errors
* patch: truncate midString to 42 chars in default template
* docs: add new username_template field to aws docs
* fix: cap token TTL at login time based on default lease TTL
* add changelog file
* patch: update warning messages to not include 'at login'
* patch: remove default lease capping and test
* update changelog
* patch: revert warning message
* initializing resp variable with aa *logical.Response before using it to add warning for default-service or default-batch token type. Also adding guard around code that sets resp to a new logical.Response further on in the function.
* adding changelog entry
* renaming changelog file to match PR number
* add username customization for rabbitmq
* add changelog for rabbitmq
* Update builtin/logical/rabbitmq/path_config_connection.go
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* updating API docs
* moved to changelog folder
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* approle: convert Callbacks to Operations
The usage of oldstyle "Callbacks" is causing the `cannot write to readonly
storage` error message when `login` is attempted against a performance standby.
Use the newstyle "Operations" and additionally set the Forward
parameters to forward the request to the Active vault node.
* add changelog
* do not forward for alias lookahead operation
* remove forward fields and remove changelog
- Because this request is an UpdateOperation, it should have automatically been
routed to the primary/active by the router before it reaches the backend.
- changelog should not be needed as this change is only a refactor with
no user-facing behavior changes.
* feat(aws): add ability to provide a sessionName to sts credentials
Co-authored-by: Brad Vernon <bvernon@nvidia.com>
Co-authored-by: Jim Kalafut <jim@kalafut.net>
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Add support for templated values in SSH CA DefaultExtensions.
* Reworking the logic per feedback, adding basic test.
* Adding test, so we cover both default extension templating & ignoring default when user-provided extensions are present.
* Fixed up an unintentional extension handling defect, added test to cover the case.
* Refactor Default Extension tests into `enabled` and `disabled`.
* Replace deprecated terms in AWS Auth
This PR is part of an effort to remove non-inclusive language throughout
Vault. The AWS Auth backend uses the "whitelist" and "blacklist" term
extensively, and these are the focus of the PR:
* Add new API endpoints that use the preferred terminology, while
deprecating the old endpoints. These endpoints offer identical
functionality and are basically aliases. This is the only functional
change in the PR except for terms in error messages.
* Replace "whitelist" -> "access list", "blacklist" -> "deny list" in
variable names, comments, etc.
Note that storage locations were *not* changed at this time, as that is
a more complex process involving versioning that we may tackle in a future
revision. We have reduced the occurrences of non-inclusive language,
however.
Reviewers should be sure to "Ignore Whitespace" in diffs, especially for
the tests, which were basically indented one level as part of looping
over the tests with both the old and new names.
* Added support for iam_tags for AWS secret roles
This change allows iam_users generated by the secrets engine
to add custom tags in the form of key-value pairs to users
that are created.
* upgrade vault dependency set
* etcd and grpc issues:
* better for tests
* testing
* all upgrades for hashicorp deps
* kubernetes plugin upgrade seems to work
* kubernetes plugin upgrade seems to work
* etcd and a bunch of other stuff
* all vulnerable packages upgraded
* k8s is broken in linux env but not locally
* test fixes
* fix testing
* fix etcd and grpc
* fix etcd and grpc
* use master branch of go-testing-interface
* roll back etcd upgrade
* have to fix grpc since other vendors pull in grpc 1.35.0 but we cant due to etcd
* rolling back in the replace directives
* a few more testing dependencies to clean up
* fix go mod vendor
* Send a test message before committing a new audit device.
Also, lower timeout on connection attempts in socket device.
* added changelog
* go mod vendor (picked up some unrelated changes.)
* Skip audit device check in integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
* Update go version to 1.15.3
* Fix OU ordering for go1.15.x testing
* Fix CI version
* Update docker image
* Fix test
* packagespec upgrade -version 0.1.8
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
* Improve errors for aws login with an unbound ARN
* Factor hasWildcardBind into its own function
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
This change allows people who are using templated policies to use the
role_name in their templates through {{
identity.entity.aliases.approle.metadata.role_name }}.
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
This is part 1 of 4 for renaming the `newdbplugin` package. This copies the existing package to the new location but keeps the current one in place so we can migrate the existing references over more easily.
* Support salt in DeriveKey
* Revert "Support salt in DeriveKey"
This reverts commit b295ae42673308a2d66d66b53527c6f9aba92ac9.
* Refactor out key derivation, symmetric encryption, and symmetric decryption into generic functions
* comments
* comments
* go mod vendor
* bump both go.mods
* This one too
* bump
* bump
* bump
* Make the lesser used params of symmetric ops a struct
* go fmt
* Call GetKey instead of DeriveKey
* Address feedback
* Wrong rv
* Rename calls
* Assign the nonce field
* trivial change
* Check nonce len instead
* go mod vendor
Adds debug and warn logging around AWS credential chain generation,
specifically to help users debugging auto-unseal problems on AWS, by
logging which role is being used in the case of a webidentity token.
Adds a deferred call to flush the log output as well, to ensure logs
are output in the event of an initialization failure.
* Use us-gov-west-1 for global APIs in aws-us-gov
Certain partition-global AWS services, like IAM, seem to require
specific regions. In the regular 'aws' partition, this is us-east-1. In
the 'aws-us-gov' partition, this is us-gov-west-1. Providing
us-gov-east-1 returns an error from AWS:
SignatureDoesNotMatch: Credential should be scoped to a valid region, not 'us-gov-east-1'.
This resolves a problem where AWS authentication could randomly fail
depending on the value cached by Vault at startup.
Decrypting an ciphertext where its corresponding value equals empty, the payload property "plaintext" is missing in the response object. This fixes the problem by adding a new, distinct struct for decrypt batch response items where "omitempty" is not set.
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Lower the interval for rotation during tests, to make it more likely
that our five second grace period is sufficient.
* Rewrite to make the rotateCredentials ticker a configurable value.
* 'go mod vendor' for SDK changes.
* secrets/ssh: allow algorithm_signer to use the key's default algo
* add test for ed25519 key signing
* test: add role upgrade test case
* test: rename and add more test cases
* test: clean up tests cases, fix broken test case on expected error
* test: fix broken test case on expected error
* Bring over PSIRT-37 changes from ENT
* Add additional allowed headers
* Already had this one
* Change to string slice comma separated parsing
* Add allowed_sts_header_values to read output
* Only validate AWS related request headers
* one per line
* Import ordering
* Update test
* Add X-Amz-Credential
* Reorder imports
* pki: use revocationInfo.RevocationTimeUTC when revoking certs with tidy_revoked_certs set to true
* update comment
* tidy: use same time snapshot for OR comparison
We've noticed some leftover processes from vault plugins on our boxes. Some
of them were even left over from previous instances of the service and reparented
to init. This could cause issues if too many of them accumulate.
When running with TRACE logging the logs showed that there was an error return by
the call to Setup() the plugin. Looking through the code it looks like we do not
call Cleanup() in that case.
Co-authored-by: Sorin Dumitru <sorindumitru@users.noreply.github.com>
* Remove two tests from TestTransit_SignVerify_P256. These verify that
you can't verify something signed using JWS marshaling when claiming
it's ASN1, and vice versa. The problem is that we rely on unmarshaling
failing, and it doesn't always. Both encodings use base64, one with
padding, one without, so depending on the data sometimes unmarshaling
will work when we expect it to fail. It would be nice to preserve
these tests if they could be made reliable, but I didn't see an easy way,
and I don't think they add enough value to warrant greater effort.
* Restore the tests I removed, and improve the verify function to broaden the notion of errExpected.
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* add first stepwise test env, Docker, with example transit test
* update transit stepwise test
* add other tests that use stepwise
* cleanup test, make names different than just 'transit'
* return the stderr if compile fails with error
* minor cleanups
* minor cleanups
* go mod vendor
* cleanups
* remove some extra code, and un-export some fields/methods
* update vendor
* remove reference to vault.CoreConfig, which really wasn't used anyway
* update with go mod vendor
* restore Precheck method to test cases
* clean up some networking things; create networks with UUID, clean up during teardown
* vendor stepwise
* Update sdk/testing/stepwise/environments/docker/environment.go
haha thanks :D
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/environments/docker/environment.go
Great catch, thanks
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* fix redundant name
* update error message in test
* Update builtin/credential/userpass/stepwise_test.go
More explicit error checking and responding
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/logical/aws/stepwise_test.go
`test` -> `testFunc`
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/logical/transit/stepwise_test.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* fix typos
* update error messages to provide clarity
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* update error handling / collection in Teardown
* panic if GenerateUUID returns an error
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update builtin/credential/userpass/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update builtin/logical/aws/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update builtin/logical/transit/stepwise_test.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/environments/docker/environment.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* import ordering
* standardize on dc from rc for cluster
* lowercase name
* CreateAPIClient -> NewAPIClient
* testWait -> ensure
* go mod cleanup
* cleanups
* move fields and method around
* make start and dockerclusternode private; use better random serial number
* use better random for SerialNumber
* add a timeout to the context used for terminating the docker container
* Use a constant for the Docker client version
* rearrange import statements
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Resolve merge conflicts and updates from running a test
* move testing/_test.go over to legacy
* updates
* Add core of plugin test framework Stepwise (#9166)
* adding stepwise testing, but there are protocol buff error :/
* move file and update sdk/go.mo
* update/sync modules
* update from other branch
* update sdk/go.mod
* some cleanups after feedback
* remove enviornments from this PR
* update vendor
* change from running go mod tidy
* change from go mod tidy
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Update sdk/testing/stepwise/helpers.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* change panic to error
* Update sdk/testing/stepwise/helpers.go
return `nil` and not `err` at the end
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* Defer close() on successful Open of a file
* document the re-creation of steps
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* remove unused BarrierKeys()
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* updates from feedback
* fix return with bad arguments
* Rename things:
- StepOperation -> Operation
- StepwiseEnvironment -> Environment
- StepCheckFunc -> AssertionFunc
- step.Check -> step.Assert
* document the environment interface methods
* rename EnvironmentOptions to MountOptions
* rename Name to RegistryName
* remove ExpectError because it's redundant
* minor doc update
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* add checkShouldRun function
* remove redundant return
* remove vestigial PreCheck function
* add tt.Helper() to makeRequest
* minor code formatting and document 1-based index for log output of Steps
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
* minor updates
* update sdk
* use local reference for api, vault dep
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk/testing/stepwise/stepwise.go
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* cleanup some defer functions
* call fatal if environment setup fails, and don't call teardown
* defer re-setting client token in makeRequest
* Move legacy logicaltest back to testhelpers
* update mods and test files with go mod tidy
* go mod vendor
* remove relative replace directives
* restore old logical test location
* move declaration to main stepwise file
* remove index var and use i+1
* add testing for write, delete paths of makeRequest
* update stepwise core testing to do request counting
* remove unused methods
* Update sdk/testing/stepwise/stepwise.go
remove dead line
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update sdk/testing/stepwise/stepwise.go
fix capitalization in code comment
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* update code comments for SkipTeardown to clarify its use
* update stepwise
Co-authored-by: Michael Golowka <72365+pcman312@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* This package is new for 1.5 so this is not a breaking change.
* This is being moved because this code was originally intended to be used
within plugins, however the design of password policies has changed such
that this is no longer needed. Thus, this code doesn't need to be in the
public SDK.
Allows vault roles to be associated with IAM groups in the AWS
secrets engine, since IAM groups are a recommended way to manage
IAM user policies. IAM users generated against a vault role will
be added to the IAM Groups. For a credential type of
`assumed_role` or `federation_token`, the policies sent to the
corresponding AWS call (sts:AssumeRole or sts:GetFederation) will
be the policies from each group in `iam_groups` combined with the
`policy_document` and `policy_arns` parameters.
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* Refactor PG container creation.
* Rework rotation tests to use shorter sleeps.
* Refactor rotation tests.
* Add a static role rotation test for MongoDB Atlas.
* Don't use string formatting to prepare queries.
We should, when possible, use the built-in params and ? format when
preparing and executing a query. This is done to prevent SQL Injection
attacks.
* Revert some changes due to failing tests, update mssql go driver
* Add docker container startup for some MSSQL tests
* Remove acceptance test flagging, add more SQL injection protection
* Refactor MSSQL prepareTestContainer to a test helper
Also, remove all ? references and convert them to @p*
* helpful errors: print not only CN but also exactly what we are comparing
* helpful errors: return different errors for non-existent and unknown keys
* helpful errors: print error about encrypted key instead of "private key not found"
* add aliasmetadata sdk helper and add to aws auth
* split into ec2_metadata and iam_metadata fields
* fix tests
* strip pointer
* add test of default metadata
* more test <3
* switch from interface to custom marshallers
* add tests for marshalling
* store nil when selected fields are default
* separate loop into pieces
* separate acc test into multiple
* Update builtin/credential/aws/path_login.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* changes from feedback
* update aws test
* refactor to also populate auth metadata
* update how jsonification is tested
* only add populated metadata values
* add auth_type to ec2 logins
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* fix: rotate root credentials for database plugins using WAL
* test: adds a test for WAL rollback logic
* fix: progress on wal rollback
* docs: updates some comments
* docs: updates some comments
* test: adds additional test coverage for WAL rollback
* chore: remove unneeded log
* style: error handling, imports, signature line wraps
* fix: always close db plugin connection
Makes "ldap operation failed" error messages a little more useful. Also
makes the errors unique so it's easier to debug where an error is coming
from when one occurs.
Revamp race test to make it aware of when tidies start/stop, and to generate dangling accessors that need cleaning up. Also run for longer but with slightly less concurrency to ensure the writes and the cleanup actually overlap.
* RSA3072 implementation in transit secrets engine
* moved new KeyType at the end of the list
So already stored keys still work properly
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* adding support for TLS 1.3 for TCP listeners
* removed test as CI uses go 1.12
* removed Cassandra support, added deprecation notice
* re-added TestTCPListener_tls13
AWS client object caches are by region. Some AWS API calls don't care
what region's client they use, but the existing getAnyRegionForAwsPartition
scheme was returning a random region, which in turn triggered maintaining many
more client objects than are necessary (e.g. 18 regions in the main AWS
partition). This can be an issue for heavy STS users bumping up against
STS rate limits, since 18 sets of creds are being cached and renewed per
STS role.
Upgrade to new official Okta sdk lib. Since it requires an API token, use old unofficial okta lib for no-apitoken case.
Update test to use newer field names. Remove obsolete test invalidated by #4798. Properly handle case where an error was expected and didn't occur.
* Switch mongodb driver to mongo-driver
* Tidy mod
* Make writeConcern private
* Implement review feedback
* Add retry functionality
* Added backoff time
* go mod vendor
* Fix failing test
* goimport
* fix aws region tests
* strip logger
* return an error, restore tests to master
* fix extra line at import
* revert changes in spacing and comments
* Update sdk/helper/awsutil/region.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* strip explicit nil value
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* Add aws metadata to identity alias
This allows for writing identity token templates that include these attributes
(And including these attributes in path templates)
* Add alias metadata asserstion to IAM login check
If a CSR contains a SAN of type otherName, encoded in UTF-8, and the signing role specifies use_csr_sans, the otherName SAN will be included in the signed cert's SAN extension.
Allow single star in allowed_other_sans to match any OtherName. Update documentation to clarify globbing behaviour.
* enable auth method item configuration in go code
* properly parse and list generated items
* make sure we only set name on attrs if a label comes from openAPI
* correctly construct paths object for method index route
* set sensitive property on password for userpass
* remove debugger statements
* pass method model to list route template to use paths on model for tabs
* update tab generation in generated item list, undo enabling userpass users
* enable openapi generated itams for certs and userpass, update ldap to no longer have action on list endpoint
* add editType to DisplayAttributes, pull tokenutil fields into field group
* show sensitive message for sensitive fields displayed in fieldGroupShow component
* grab sensitive and editType fields from displayAttrs in openapi-to-attrs util
* make sure we don't ask for paths for secret backends since that isn't setup yet
* fix styling of sensitive text for fieldGroupShow component
* update openapi-to-attrs util test to no longer include label by default, change debugger to console.err in path-help, remove dynamic ui auth methods from tab count test
* properly log errors to the console
* capitalize This value is sensitive...
* get rid of extra padding on bottom of fieldgroupshow
* make auth methods clickable and use new confirm ux
* Update sdk/framework/path.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* Update sdk/framework/path.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* add whitespace
* return intErr instead of err
* uncomment out helpUrl because we need it
* remove extra box class
* use const instead of let
* remove extra conditional since we already split the pathName later on
* ensure we request the correct url when listing generated items
* use const
* link to list and show pages
* remove dead code
* show nested item name instead of id
* add comments
* show tooltip for text-file inputs
* fix storybook
* remove extra filter
* add TODOs
* add comments
* comment out unused variables but leave them in function signature
* only link to auth methods that can be fully managed in the ui
* clean up comments
* only render tooltip if there is helpText
* rename id authMethodPath
* remove optionsForQuery since we don't need it
* add indentation
* standardize ConfirmMessage and show model name instead of id when editing
* standardize ConfirmMessage and show model name instead of id when editing
* add comments
* post to the correct updateUrl so we can edit users and groups
* use pop instead of slice
* add TODO for finding a better way to store ids
* ensure ids are handled the same way on list and show pages; fix editing and deleting
* add comment about difference between list and show urls
* use model.id instead of name since we do not need it
* remove dead code
* ensure list pages have page headers
* standardize using authMethodPath instead of method and remove dead code
* i love indentation
* remove more dead code
* use new Confirm
* show correct flash message when deleting an item
* update flash message for creating and updating
* use plus icon for creating group/user instead of an arrow
* secrets/aws: Support permissions boundaries on iam_user creds
This allows configuring Vault to attach a permissions boundary policy to
IAM users that it creates, configured on a per-Vault-role basis.
* Fix indentation of policy in docs
Use spaces instead of tabs
I probably left this cleanup commented out as part of debugging test
errors in #6789 and forgot to uncomment it, so actually cleaning up the
test user.
* secret/aws: Pass policy ARNs to AssumedRole and FederationToken roles
AWS now allows you to pass policy ARNs as well as, and in addition to,
policy documents for AssumeRole and GetFederationToken (see
https://aws.amazon.com/about-aws/whats-new/2019/05/session-permissions/).
Vault already collects policy ARNs for iam_user credential types; now it
will allow policy ARNs for assumed_role and federation_token credential
types and plumb them through to the appropriate AWS calls.
This brings along a minor breaking change. Vault roles of the
federation_token credential type are now required to have either a
policy_document or a policy_arns specified. This was implicit
previously; a missing policy_document would result in a validation error
from the AWS SDK when retrieving credentials. However, it would still
allow creating a role that didn't have a policy_document specified and
then later specifying it, after which retrieving the AWS credentials
would work. Similar workflows in which the Vault role didn't have a
policy_document specified for some period of time, such as deleting the
policy_document and then later adding it back, would also have worked
previously but will now be broken.
The reason for this breaking change is because a credential_type of
federation_token without either a policy_document or policy_arns
specified will return credentials that have equivalent permissions to
the credentials the Vault server itself is using. This is quite
dangerous (e.g., it could allow Vault clients access to retrieve
credentials that could modify Vault's underlying storage) and so should
be discouraged. This scenario is still possible when passing in an
appropriate policy_document or policy_arns parameter, but clients should
be explicitly aware of what they are doing and opt in to it by passing
in the appropriate role parameters.
* Error out on dangerous federation token retrieval
The AWS secrets role code now disallows creation of a dangerous role
configuration; however, pre-existing roles could have existed that would
trigger this now-dangerous code path, so also adding a check for this
configuration at credential retrieval time.
* Run makefmt
* Fix tests
* Fix comments/docs
This refactors role validation for the AWS secrets engine to be in a
separate method. Previously, all validation was interspersed with the
parsing of parameters when creating/updating a role, which led to a high
degree of complexity. Now, all validation is centralized which makes it
easier to understand and also easier to test (and so a number of test
cases have been added).
* Add maximum amount of random characters requested at any given time
* Readability changes
* Removing sys/tools/random from the default policy
* Setting the maxBytes value as const
* Declaring maxBytes in the package to use it everywhere
* Using maxBytes in the error message
