* The verify-sign command in it's cleanest existing form.
* Working state
* Updates to proper verification syntax
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
* make fmt
* Base functionality.
* make fmt; changelog
* pki issue command.
* Make fmt. Changelog.
* Error Handling Is Almost A Tutorial
* Issue and ReIssue are Almost the Same Command
* Make Fmt + Changelog.
* Make some of the tests go.
* make fmt
* Merge fix (take 2)
* Fix existing support, add support for use_pss, max_path_length, not_after, permitted_dns_domains and skid
* Good Test which Fails
* Test-correction.
* Fix update to key_type key_bits; allow "," in OU or similar
* More specific includeCNinSANs
* Add tests around trying to use_pss on an ec key.
* GoDoc Test Paragraph thing.
---------
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
* fix: upgrade vault-plugin-secrets-mongodbatlas to v0.9.1
* add changelog
* Update changelog/19111.txt
Co-authored-by: Max Coulombe <109547106+maxcoulombe@users.noreply.github.com>
* use correct plugin type in changelog
---------
Co-authored-by: Max Coulombe <109547106+maxcoulombe@users.noreply.github.com>
* Use UTC for leaf exceeding CA's notAfter
When generating a leaf which exceeds the CA's validity period, Vault's
error message was confusing as the leaf would use the server's time
zone, but the CA's notAfter date would use UTC. This could cause
user confusion as the leaf's expiry might look before the latter, due
to using different time zones. E.g.:
> cannot satisfy request, as TTL would result in notAfter
> 2023-03-06T16:41:09.757694-08:00 that is beyond the expiration of
> the CA certificate at 2023-03-07T00:29:52Z
Consistently use UTC for this instead.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix race accessing b.crls within cert auth
- Discovered by CircleCI the pathLogin, pathLoginRenew paths access
and reloads the b.crls member variable without a lock.
- Also discovered that pathLoginResolveRole never populated an empty
b.crls before usage within b.verifyCredentials
* Add cl
* Misc cleanup
- Introduce a login path wrapper instead of repeating in all the
various login methods the crl reloading
- Cleanup updatedConfig, never returned an error and nothing looked at
the error returned
- Make the test within TestCRLFetch a little less timing sensitive as
I was able to trigger a failure due to my machine taking more than
150ms to load the new CRL
* Revert "Don't execute the seal recovery tests on ENT. (#18841)"
This reverts commit 990d3bacc203c229d0f6729929d7562e678a1ac2.
* Revert "Add the ability to unseal using recovery keys via an explicit seal option. (#18683)"
This reverts commit 2ffe49aab0fc1a527c5182637c8fa3ac39b08d45.
* Apply URL encoding/unencoding to OCSP Get requests
- Missed this during development and sadly the unit tests were written
at a level that did not expose this issue originally, there are
certain combinations of issuer cert + serial that lead to base64
data containing a '/' which will lead to the OCSP handler not getting
the full parameter.
- Do as the spec says, this should be treated as url-encoded data.
* Add cl
* Add higher level PKI OCSP GET/POST tests
* Rename PKI ocsp files to path_ocsp to follow naming conventions
* make fmt
* Add ability to clean up host keys for dynamic keys
This adds a new endpoint, tidy/dynamic-keys that removes any stale host
keys still present on the mount. This does not clean up any pending
dynamic key leases and will not remove these keys from systems with
authorized hosts entries created by Vault.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Regexp metacharacter `.` should be escaped when used literally
The paths including `/.well-known/` in the Vault API could currently
technically be invoked with any random character in place of the dot.
* Replace implementation of OpenAPI path translator with regexp AST-based one
* Add changelog
* Typo fix from PR review - thanks!
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Add comment based on review feedback
* Change style of error handling as suggested in code review
* Make a further tweak to the handling of the error case
* Add more tests, testing cases which fail with the previous implementation
* Resolve issue with a test, and improve comment
---------
Co-authored-by: Anton Averchenkov <84287187+averche@users.noreply.github.com>
* Remove dynamic keys from SSH Secrets Engine
This removes the functionality of Vault creating keys and adding them to
the authorized keys file on hosts.
This functionality has been deprecated since Vault version 0.7.2.
The preferred alternative is to use the SSH CA method, which also allows
key generation but places limits on TTL and doesn't require Vault reach
out to provision each key on the specified host, making it much more
secure.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove dynamic ssh references from documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove dynamic key secret type entirely
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clarify changelog language
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add removal notice to the website
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Read total cert counts with atomic.LoadUint32(...)
When generating the tidy status, we read the values of two backend
atomics, b.certCount and b.revokedCertCount, without using the atomic
load operation. This resulted in a data race when the status was read
at the same time as an on-going tidy operation:
WARNING: DATA RACE
Write at 0x00c00c77680c by goroutine 90522:
sync/atomic.AddInt32()
/usr/local/go/src/runtime/race_amd64.s:281 +0xb
sync/atomic.AddUint32()
<autogenerated>:1 +0x1a
github.com/hashicorp/vault/builtin/logical/pki.(*backend).tidyStatusIncRevokedCertCount()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:1236 +0x107
github.com/hashicorp/vault/builtin/logical/pki.(*backend).doTidyRevocationStore()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:525 +0x1404
github.com/hashicorp/vault/builtin/logical/pki.(*backend).startTidyOperation.func1.1()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:290 +0x1a4
github.com/hashicorp/vault/builtin/logical/pki.(*backend).startTidyOperation.func1()
/home/circleci/go/src/github.com/hashicorp/vault/builtin/logical/pki/path_tidy.go:342 +0x278
Previous read at 0x00c00c77680c by goroutine 90528:
reflect.Value.Uint()
/usr/local/go/src/reflect/value.go:2584 +0x195
encoding/json.uintEncoder()
/usr/local/go/src/encoding/json/encode.go:562 +0x45
encoding/json.ptrEncoder.encode()
/usr/local/go/src/encoding/json/encode.go:944 +0x3c2
encoding/json.ptrEncoder.encode-fm()
<autogenerated>:1 +0x90
encoding/json.(*encodeState).reflectValue()
/usr/local/go/src/encoding/json/encode.go:359 +0x88
encoding/json.interfaceEncoder()
/usr/local/go/src/encoding/json/encode.go:715 +0x17b
encoding/json.mapEncoder.encode()
/usr/local/go/src/encoding/json/encode.go:813 +0x854
... more stack trace pointing into JSON encoding and http
handler...
In particular, because the tidy status was directly reading the uint
value without resorting to the atomic side, the JSON serialization could
race with a later atomic update.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Also use atomic load in tests
Because no tidy operation is running here, it should be safe to read the
pointed value directly, but use the safer atomic.Load for consistency.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* wip
* Transit byok cli
* It works!
* changelog
* document return codes
* Update command/transit_import_key.go
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* make fmt
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* The verify-sign command in it's cleanest existing form.
* Working state
* Updates to proper verification syntax
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
* make fmt
* Git CI caught some stuff.
* Base functionality.
* make fmt; changelog
* pki issue command.
* Make fmt. Changelog.
* Error Handling Is Almost A Tutorial
* What I thought empty issuers response fix would be.
* Some tests
* PR-review updates.
* make fmt.
* Fix null response data for listing empty issuers causing a crash.
* Update command/pki_list_children_command.go
Fix double specifier
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Add test for pki_list_children.
* Fix tests.
* Update descriptions for correctness based on PR reviews.
* make fmt.
* Updates based on PR feedback.
* Allow multiple arguements (space separated)
* Remove bad merge-thing.
* White-space hell fix change.
* Tests, and return information for issue ca
* Fix make fmt error introduced here: https://github.com/hashicorp/vault/pull/18876
* Update command/pki_issue_intermediate.go
Puncutation.
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove smart quotes for standard quotes.
* More information as part of the help text.
* Better help text.
* Add missing "/" into error message.
---------
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Move cert auth backend setup into initialize
In further review with new understanding after #18244, loading
configuration and CRLs within the backend's initialize function is the
ideal approach: Factory construction is strictly serial, resulting in
backend initialization blocking until config and CRLs are loaded.
By using an InitializeFunc(...), we delay loading until after all
backends are constructed (either right on startup in 1.12+, else during
the initial PeriodicFunc(...) invocation on 1.11 and earlier).
We also invoke initialize automatically on test Factory construction.
Resolves: #17847
Co-authored-by: valli_0x <personallune@mail.ru>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
---------
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: valli_0x <personallune@mail.ru>
* Base functionality.
* make fmt; changelog
* What I thought empty issuers response fix would be.
* Fix null response data for listing empty issuers causing a crash.
* Update command/pki_list_children_command.go
Fix double specifier
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Add test for pki_list_children.
* Fix tests.
* Update descriptions for correctness based on PR reviews.
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* WIP/initial routing-ish
* refactor date dropdown to reuse in modal and allowe current month selection
* swap linter disable line
* refactor date-dropdown to return object
* refactor calendar widget, add tests
* change calendar start and end args to getters
* refactor dashboard to use date objects instead of array of year, month
* remove dashboard files for easier to follow git diff
* comment out dashboard tab until route name updated
* delete current tab and route
* fix undefined banner time
* cleanup version history serializer and upgrade data
* first pass of updating tests
* add changelog
* update client count util test
* validate end time is after start time
* update comment
* add current month to calendar widget
* add comments for code changes to make following API update
* Removed a modified file from pull request
* address comments/cleanup
* update variables to const
* update test const
* rename history -> dashboard, fix tests
* fix timestamps for attribution chart
* update release note
* refactor using backend start and end time params
* add test for adapter formatting time params
* fix tests
* cleanup adapter comment and query params
* change back history file name for diff
* rename file using cli
* revert filenames
* rename files via git cli
* revert route file name
* last cli rename
* refactor mirage
* hold off on running total changes
* update params in test
* refactor to remove conditional assertions
* finish tests
* fix firefox tooltip
* remove current-when
* refactor version history
* add timezone/UTC note
* final cleanup!!!!
* fix test
* fix client count date tests
* fix date-dropdown test
* clear datedropdown completely
* update date selectors to accommodate new year (#18586)
* Revert "hold off on running total changes"
This reverts commit 8dc79a626d549df83bc47e290392a556c670f98f.
* remove assumed 0 values
* update average helper to only calculate for array of objects
* remove passing in bar chart data, map in running totals component instead
* cleanup usage stat component
* clear ss filters for new queries
* update csv export, add explanation to modal
* update test copy
* consistently return null if no upgrade during activity (instead of empty array)
* update description, add clarifying comments
* update tes
* add more clarifying comments
* fix historic single month chart
* remove old test tag
* Update ui/app/components/clients/dashboard.js
* The verify-sign command in it's cleanest existing form.
* Working state
* Updates to proper verification syntax
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
* make fmt
* Git CI caught some stuff.
* Some tests
* PR-review updates.
* make fmt.
Co-authored-by: 'Alex Scheel' <alex.scheel@hashicorp.com>
* The fields.
* UserID set, add to certificate
* Changelog.
* Fix test (set default).
* Add UserID constant to certutil, revert extension changes
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add user_ids as field for leaf signing
Presumably, this isn't necessary for CAs, given that CAs probably don't
have a user ID corresponding to them.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Support setting multiple user_ids in Subject
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow any User ID with sign-verbatim
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for User IDs in PKI
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add docs about user_ids, allowed_user_ids
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Parallel migration (#18815)
* flagParallel sanity check
* Attempt to use ErrGroups
* Updated docs
* Allow 'start' and 'max-parallel' together
* parallel flag renamed to max-parallel
* tests for start + parallel
* Removed permit pool
* Updated docs to make it clearer that a high setting might not be honored based on storage backend setting
* System dependent max int size
* Default max-parallel 1 => 10
* Test folder/paths updated
Co-authored-by: Tomasz Pawelczak <10206601+gites@users.noreply.github.com>
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
* wip
* wip
* Got it 'working', but not happy about cleanliness yet
* Switch to a dedicated defaultSeal with recovery keys
This is simpler than trying to hijack SealAccess as before. Instead, if the operator
has requested recovery unseal mode (via a flag in the seal stanza), we new up a shamir
seal with the recovery unseal key path instead of the auto seal. Then everything proceeds
as if you had a shamir seal to begin with.
* Handle recovery rekeying
* changelog
* Revert go.mod redirect
* revert multi-blob info
* Dumb nil unmarshal target
* More comments
* Update vault/seal.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* pr feedback
* Fix recovery rekey, which needs to fetch root keys and restore them under the new recovery split
* Better comment on recovery seal during adjustSealMigration
* Make it possible to migrate from an auto-seal in recovery mode to shamir
* Fix sealMigrated to account for a recovery seal
* comments
* Update changelog/18683.txt
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Address PR feedback
* Refactor duplicated migration code into helpers, using UnsealRecoveryKey/RecoveryKey where appropriate
* Don't shortcut the reast of seal migration
* get rid of redundant transit server cleanup
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* named MFA method configurations
* fix a test
* CL
* fix an issue with same config name different ID and add a test
* feedback
* feedback on test
* consistent use of passcode for all MFA methods (#18611)
* make use of passcode factor consistent for all MFA types
* improved type for MFA factors
* add method name to login CLI
* minor refactoring
* only accept MFA method name with its namespace path in the login request MFA header
* fix a bug
* fixing an ErrorOrNil return value
* more informative error message
* Apply suggestions from code review
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* feedback
* test refactor a bit
* adding godoc for a test
* feedback
* remove sanitize method name
* guard a possbile nil ref
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* added audit-hash operations
* more audit paths
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* added audit fields
* add changelog file
* dynamic fields should be nil
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* start to add test helper
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
* add tests for /sys/audit openapi paths
Signed-off-by: Daniel Huckins <dhuckins@users.noreply.github.com>
Co-authored-by: Anton Averchenkov <anton.averchenkov@hashicorp.com>
* Ember Engine for Kubernetes Secrets Engine (#17881)
* adds in-repo ember engine for kubernetes secrets engine
* updates kubernetes engine class name
* Kubernetes route plumbing (#17895)
* kubernetes route plumbing
* adds kubernetes role index route with redirect to details
* adds kubernetes as mountable and supported secrets engine (#17891)
* adds models, adapters and serializers for kubernetes secrets engine (#18010)
* adds mirage factories and handlers for kubernetes (#17943)
* Kubernetes Secrets Engine Configuration (#18093)
* moves RadioCard component to core addon
* adds kubernetes configuration view
* fixes tests using RadioCard after label for and input id changes
* adds confirm modal when editing kubernetes config
* addresses review comments
* Kubernetes Configuration View (#18147)
* removes configuration edit and index routes
* adds kubernetes configuration view
* Kubernetes Roles List (#18211)
* removes configuration edit and index routes
* adds kubernetes configuration view
* adds kubernetes secrets engine roles list view
* updates role details disabled state to explicitly check for false
* VAULT-9863 Kubernetes Overview Page (#18232)
* Add overview page view
* Add overview page tests
* Address feedback to update tests and minor changes
* Use template built in helper for conditionally showing num roles
* Set up roleOptions in constructor
* Set up models in tests and fix minor bug
* Kubernetes Secrets Engine Create/Edit Views (#18271)
* moves kv-object-editor to core addon
* moves json-editor to core addon
* adds kubernetes secrets engine create/edit views
* updates kubernetes/role adapter test
* addresses feedback
* fixes issue with overview route showing 404 page (#18303)
* Kubernetes Role Details View (#18294)
* moves format-duration helper to core addon
* adds kubernetes secrets engine role details view
* adds tests for role details page component
* adds capabilities checks for toolbar actions
* fixes list link for secrets in an ember engine (#18313)
* Manual Testing: Bug Fixes and Improvements (#18333)
* updates overview, configuration and roles components to pass args for individual model properties
* bug fixes and improvements
* adds top level index route to redirect to overview
* VAULT-9877 Kubernetes Credential Generate/View Pages (#18270)
* Add credentials route with create and view components
* Update mirage response for creds and add ajax post call for creds in adapter
* Move credentials create and view into one component
* Add test classes
* Remove files and update backend property name
* Code cleanup and add tests
* Put test helper in helper function
* Add one more test!
* Add code optimizations
* Fix model in route and add form
* Add onSubmit to form and preventDefault
* Fix tests
* Update mock data for test to be strong rather than record
* adds acceptance tests for kubernetes secrets engine roles (#18360)
* VAULT-11862 Kubernetes acceptance tests (#18431)
* VAULT-12185 overview acceptance tests
* VAULT-12298 credentials acceptance tests
* VAULT-12186 configuration acceptance tests
* VAULT-12127 Refactor breadcrumbs to use breadcrumb component (#18489)
* VAULT-12127 Refactor breadcrumbs to use Page::Breadcrumbs component
* Fix failing tests by adding breadcrumbs properties
* VAULT-12166 add jsdocs to kubernetes secrets engine pages (#18509)
* fixes incorrect merge conflict resolution
* updates kubernetes check env vars endpoint (#18588)
* hides kubernetes ca cert field if not defined in configuration view
* fixes loading substate handling issue (#18592)
* adds changelog entry
Co-authored-by: Kianna <30884335+kiannaquach@users.noreply.github.com>
* OpenAPI `generic_mount_paths` follow-up
An incremental improvement within larger context discussed in #18560.
* Following the revert in #18617, re-introduce the change from
`{mountPath}` to `{<path-of-mount>_mount_path}`; this is needed, as
otherwise paths from multiple plugins would clash - e.g. almost every
auth method would provide a conflicting definition for
`auth/{mountPath}/login`, and the last one written into the map would
win.
* Move the half of the functionality that was in `sdk/framework/` to
`vault/logical_system.go` with the rest; this is needed, as
`sdk/framework/` gets compiled in to externally built plugins, and
therefore there may be version skew between it and the Vault main
code. Implementing the `generic_mount_paths` feature entirely on one
side of this boundary frees us from problems caused by this.
* Update the special exception that recognizes `system` and `identity`
as singleton mounts to also include the other two singleton mounts,
`cubbyhole` and `auth/token`.
* Include a comment that documents to restricted circumstances in which
the `generic_mount_paths` option makes sense to use:
// Note that for this to actually be useful, you have to be using it with
// a Vault instance in which you have mounted one of each secrets engine
// and auth method of types you are interested in, at paths which identify
// their type, and for the KV secrets engine you will probably want to
// mount separate kv-v1 and kv-v2 mounts to include the documentation for
// each of those APIs.
* Fix tests
Also remove comment "// TODO update after kv repo update" which was
added 4 years ago in #5687 - the implied update has not happened.
* Add changelog
* Update 18663.txt
Removing the timeout logic from raw-response functions and adding documentation comments. The following functions are affected:
- `ReadRaw`
- `ReadRawWithContext` (newly added)
- `ReadRawWithData`
- `ReadRawWithDataWithContext`
The previous logic of using `ctx, _ = c.c.withConfiguredTimeout(ctx)` could cause a potential [context leak](https://pkg.go.dev/context):
> Failing to call the CancelFunc leaks the child and its children until the parent is canceled or the timer fires. The go vet tool checks that CancelFuncs are used on all control-flow paths.
Cancelling the context would have caused more issues since the context would be cancelled before the request body is closed.
Resolves: #18658
This PR modifies every test in `builtin/credentials/approle/path_role_test.go` with new validation checks to ensure that approle/path_role successful responses align with the declared response schema.
It also introduces a test helper in `sdk/helper/testhelpers`:
```go
func FindResponseSchema(t *testing.T, ...)
```
This test helper will be useful for all plugins that require similar response schema validation in tests.
### Background
This PR is part of the ongoing work to add structured responses in Vault OpenAPI (VLT-234)
This pull request adds 3 functions (and corresponding tests):
`testhelpers/response_validation.go`:
- `ValidateResponse`
- `ValidateResponseData`
field_data.go:
- `ValidateStrict` (has the "strict" validation logic)
The functions are primarily meant to be used in tests to ensure that the responses are consistent with the defined response schema. An example of how the functions can be used in tests can be found in #18636.
### Background
This PR is part of the ongoing work to add structured responses in Vault OpenAPI (VLT-234)
This PR relates to a feature request logged through HashiCorp commercial
support.
Vault lacks pagination in its APIs. As a result, certain list operations
can return **very** large responses. The user's chosen audit sinks may
experience difficulty consuming audit records that swell to tens of
megabytes of JSON.
In our case, one of the systems consuming audit log data could not cope,
and failed.
The responses of list operations are typically not very interesting, as
they are mostly lists of keys, or, even when they include a "key_info"
field, are not returning confidential information. They become even less
interesting once HMAC-ed by the audit system.
Some example Vault "list" operations that are prone to becoming very
large in an active Vault installation are:
auth/token/accessors/
identity/entity/id/
identity/entity-alias/id/
pki/certs/
In response, I've coded a new option that can be applied to audit
backends, `elide_list_responses`. When enabled, response data is elided
from audit logs, only when the operation type is "list".
For added safety, the elision only applies to the "keys" and "key_info"
fields within the response data - these are conventionally the only
fields present in a list response - see logical.ListResponse, and
logical.ListResponseWithInfo. However, other fields are technically
possible if a plugin author writes unusual code, and these will be
preserved in the audit log even with this option enabled.
The elision replaces the values of the "keys" and "key_info" fields with
an integer count of the number of entries. This allows even the elided
audit logs to still be useful for answering questions like "Was any data
returned?" or "How many records were listed?".
* add core state lockd eadlock detection config option v2
* add changelog
* split out NewTestCluster function to maintain build flag
* replace long func with constant
* remove line
* rename file, and move where detect deadlock flag is set
* Allow tidy to backup legacy CA bundles
With the new tidy_move_legacy_ca_bundle option, we'll use tidy to move
the legacy CA bundle from /config/ca_bundle to /config/ca_bundle.bak.
This does two things:
1. Removes ca_bundle from the hot-path of initialization after initial
migration has completed. Because this entry is seal wrapped, this
may result in performance improvements.
2. Allows recovery of this value in the event of some other failure
with migration.
Notably, this cannot occur during migration in the unlikely (and largely
unsupported) case that the operator immediately downgrades to Vault
<1.11.x. Thus, we reuse issuer_safety_buffer; while potentially long,
tidy can always be run manually with a shorter buffer (and only this
flag) to manually move the bundle if necessary.
In the event of needing to recover or undo this operation, it is
sufficient to use sys/raw to read the backed up value and subsequently
write it to its old path (/config/ca_bundle).
The new entry remains seal wrapped, but otherwise isn't used within the
code and so has better performance characteristics.
Performing a fat deletion (DELETE /root) will again remove the backup
like the old legacy bundle, preserving its wipe characteristics.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation about new tidy parameter
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for migration scenarios
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Clean up time comparisons
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update command/agent.go
* Attempt to only reload log level and certs
* Mimicked 'server' test for cert reload in 'agent'
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
Left out the `c.config` tweak that meant changes to lots of lines of code within the `Run` function of Agent command. :)
* Correctly distinguish empty issuer names
When using client.Logical().JSONMergePatch(...) with an empty issuer
name, patch incorrectly reports:
> issuer name contained invalid characters
In this case, both the error in getIssuerName(...) is incorrect and
patch should allow setting an empty issuer name explicitly.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* expand NodeStatusReporter with new fields
* only call IsRaftVoter if using raft storage
* add changelog entry
* fix listeners
* return LogLevel as enum
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* bump github.com/hashicorp/vault/vault/hcp_link/proto
* go mod tidy
* go get link proto @vault-11829-meta-get-cluster-status
* add HA status
* add HAEnabled method
* add raft config
* allocate HA nodes based on actual count
* add raft autopilot status
* add raft quorum warnings
* add ClusterID method
* add StorageType
* add ClusterID
* update github.com/hashicorp/vault/vault/hcp_link/proto
* add changelog entry
* fix raft config panic
* remove "Warning" quorum message prefix
* add error wrapping
* add Core.HAStateWithLock method
* reduce quorum warnings to single string
* fix HCP_API_HOST test env var check
* Revert "fix HCP_API_HOST test env var check"
This reverts commit 97c73c4798b77b84aea84f341f2c63c4d657914d.
Too many newlines are stripped, which is responsible for the `FEATURES:`
heading in the current in-progress 1.13.0 changelog entry being
erroneously appended to the end of the last bullet point of the previous
`CHANGES:` section.
* code changes for unlock
* add test
* adding sys help
* adding sys help
* updating unlock user function
* edit test
* add changelog
* syshelp
* adding open api response definition
* removing response fields
* change path name
* Add issuer reference info on JSON endpoint
This endpoint is unauthenticated and shouldn't contain sensitive
information. However, listing the issuers (LIST /issuers) already
returns both the issuer ID and the issuer name (if any) so this
information is safe to return here.
When fetching /pki/issuer/default/json, it would be nice to know exactly
which issuer ID and name it corresponds to, without having to fetch the
authenticated endpoint as well.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow mounting external plugins with same name/type as deprecated builtins
* Add some go tests for deprecation status handling
* Move timestamp storage to post-unseal
* Add upgrade-aware deprecation shutdown and tests
* Return the partial success code override for all batch error types
* changelog
* docs
* Lost the actual override logic. :)
* And don't hardcode 400
* gate on success
* Initial worker pool
* Run postUnsealFuncs in parallel
* Use the old logic for P=1
* changelog
* Use a CPU count relative worker pool
* Update vault/core.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Done must be called once per postUnsealFunc
* Defer is overkill
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Rename path_config -> path_keys_config
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add config/keys to disable upserting
Transit would allow anyone with Create permissions on the encryption
endpoint to automatically create new encryption keys. This becomes hard
to reason about for operators, especially if typos are subtly
introduced (e.g., my-key vs my_key) -- there is no way to merge these
two keys afterwards.
Add the ability to globally disable upserting, so that if the
applications using Transit do not need the capability, it can be
globally disallowed even under permissive policies.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation on disabling upsert
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Update website/content/api-docs/secret/transit.mdx
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
* Update website/content/api-docs/secret/transit.mdx
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: tjperry07 <tjperry07@users.noreply.github.com>
* add Link config, init, and capabilities
* add node status proto
* bump protoc version to 3.21.9
* make proto
* adding link tests
* remove wrapped link
* add changelog entry
* update changelog entry
Move version out of SDK. For now it's a copy rather than move: the part not addressed by this change is sdk/helper/useragent.String, which we'll want to remove in favour of PluginString. That will have to wait until we've removed uses of useragent.String from all builtins.
* adding oss file changes
* check disabled and read values from config
* isUserLocked, getUserLockout Configurations, check user lock before login and return error
* remove stale entry from storage during read
* added failed login process workflow
* success workflow updated
* user lockouts external tests
* changing update to support delete
* provide access to alias look ahead function
* adding path alias lookahead
* adding tests
* added changelog
* added comments
* adding changes from ent branch
* adding lock to UpdateUserFailedLoginInfo
* fix return default bug
This PR modifies the path schema of `approle/path_role.go`, switching the old `Callbacks` to the equivalent `Operations` objects with a list of response fields for the 200 responses. This will allow us to generate a response structures in openapi.json. This PR is split out from #18055 along with #18192.
### Example
For `GET "/auth/approle/role/{role_name}/bind-secret-id"` path, it will update the response as follows:
```diff
"responses": {
"200": {
"description": "OK",
++ "content": {
++ "application/json": {
++ "schema": {
++ "$ref": "#/components/schemas/ApproleRoleBindSecretIdResponse"
++ }
++ }
}
}
}
```
And will add the actual response structure:
```diff
++ "ApproleRoleBindSecretIdResponse": {
++ "type": "object",
++ "properties": {
++ "bind_secret_id": {
++ "type": "boolean",
++ "description": "Impose secret_id to be presented when logging in using this role. Defaults to 'true'."
++ }
++ }
++ },
```
* VAULT-11510 Vault Agent can start listeners without caching
* VAULT-11510 fix order of imports
* VAULT-11510 changelog
* VAULT-11510 typo and better switch
* VAULT-11510 update name
* VAULT-11510 New api_proxy stanza to configure API proxy
* VAULT-11510 First pass at API Proxy docs
* VAULT-11510 nav data
* VAULT-11510 typo
* VAULT-11510 docs update
* Respond with data to all writes in PKI engine
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow templating of cluster-local AIA URIs
This adds a new configuration path, /config/cluster, which retains
cluster-local configuration. By extending /config/urls and its issuer
counterpart to include an enable_templating parameter, we can allow
operators to correctly identify the particular cluster a cert was
issued on, and tie its AIA information to this (cluster, issuer) pair
dynamically.
Notably, this does not solve all usage issues around AIA URIs: the CRL
and OCSP responder remain local, meaning that some merge capability is
required prior to passing it to other systems if they use CRL files and
must validate requests with certs from any arbitrary PR cluster.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation about templated AIAs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* AIA URIs -> AIA URLs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* issuer.AIAURIs might be nil
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow non-nil response to config/urls
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Always validate URLs on config update
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Ensure URLs lack templating parameters
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Review feedback
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Skip plugin startup for missing plugins
* Skip secrets startup for missing plugins
* Add changelog for bugfix
* Make plugin handling on unseal version-aware
* Update plugin lazy-load logic/comments for readability
* Add register/mount/deregister/seal/unseal go test
* Consolidate lazy mount logic to prevent inconsistencies
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
And return an error instead of panicking.
This situation can occur if a plugin attempts to access the system
view during setup when Vault is checking the plugin metadata.
Fixes#17878.
* Address a nil panic when writing an empty POST request to the ocsp handler
- Seems when no JSON body is sent with a POST request Vault will not
populate the HTTPRequest member variable which caused the nil panic
- vault write -force pki/ocsp
- Add a check for it and the Body member variable to be nil before use.
* Add cl
* Add test that fails due to audit log panic
* Rebuild VersionedPlugin as map of primitive types before adding to response
* Changelog
* Fix casting in external plugin tests
* ui: mfa: use proper request id generation
Fixes: 712cc9ee, ca14c191
Signed-off-by: Konstantin Demin <rockdrilla@gmail.com>
Signed-off-by: Konstantin Demin <rockdrilla@gmail.com>
* Work to unify log-file for agent/server and add rotation
* Updates to rotation code, tried to centralise the log config setup
* logging + tests
* Move LogFile to ShareConfig in test
* Docs
* Fix typos
* Return http 400 when wrong unseal key is supplied
* Add changelog
* Add test cases and change one more return case to http 400
The new case is triggered when key length is within valid range
[16, 32], but it has uneven bytes, causing crypto/aes to return
invalid key size.
* remove expected in unit tests
* include error in the new error reason
* add multikey and autoseal test cases
* return invalid key for few more code paths
* Removes _builtin_ versions from mount storage where it already exists
* Stops new builtin versions being put into storage on mount creation/tuning
* Stops the plugin catalog from returning a builtin plugin that has been overridden, so it more accurately reflects the plugins that are available to actually run
* fixes: #17113: Make MFA setup wizard call token based generate endpoint
* add changelog for #17932
* cleanup: bring back .gitkeep
* re trigger ci jobs
* New PKI API to generate and sign a CRL based on input data
- Add a new PKI API that allows an end-user to feed in all the
information required to generate and sign a CRL by a given issuer.
- This is pretty powerful API allowing an escape hatch for 3rd parties
to craft customized CRLs with extensions based on their individual
needs
* Add api-docs and error if reserved extension is provided as input
* Fix copy/paste error in Object Identifier constants
* Return nil on errors instead of partially filled slices
* Add cl
* wip
* Add cached OCSP client support to Cert Auth
* ->pointer
* Code cleanup
* Fix unit tests
* Use an LRU cache, and only persist up to 1000 of the most recently used values to stay under the storage entry limit
* Fix caching, add fail open mode parameter to cert auth roles
* reduce logging
* Add the retry client and GET then POST logic
* Drop persisted cache, make cache size configurable, allow for parallel testing of multiple servers
* dead code
* Update builtin/credential/cert/path_certs.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Hook invalidate to reinit the ocsp cache size
* locking
* Conditionally init the ocsp client
* Remove cache size config from cert configs, it's a backend global
* Add field
* Remove strangely complex validity logic
* Address more feedback
* Rework error returning logic
* More edge cases
* MORE edge cases
* Add a test matrix with a builtin responder
* changelog
* Use an atomic for configUpdated
* Actually use ocsp_enabled, and bind to a random port for testing
* Update builtin/credential/cert/path_login.go
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Refactor unit tests
* Add status to cache
* Make some functions private
* Rename for testing, and attribute
* Up to date gofumpt
* remove hash from key, and disable the vault dependent unit test
* Comment out TestMultiOCSP
* imports
* more imports
* Address semgrep results
* Attempt to pass some sort of logging to test_responder
* fix overzealous search&replace
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
* refactor ss+modal to accept multiple models
* create policy form
* cleanup and fix test
* add tabs to policy modal form
* add search select with modal to entity form
* update group form;
* allow modal to fit-content
* add changelog
* add check for policy create ability
* add id so tests pass
* filter out root option
* fix test
* add cleanup method
* add ACL policy link
* cleanup from comments
* refactor sending action to parent
* refactor, data down actions up!
* cleanup comments
* form field refactor
* add ternary to options
* update tests
* Remodel component structure for clearer logic
Includes fixing the wizard
* address comments
* cleanup args
* refactor inline oidc assignment form
* add line break
* cleanup comments
* fix tests
* add policy template to ss+modal test
* cleanup =true from test
* final cleanup!!!!!!
* actual final cleanup
* fix typo, please be done
Co-authored-by: Chelsea Shaw <82459713+hashishaw@users.noreply.github.com>
* Add crl list capabilities to cert auth
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add docs on cert auth CRL listing
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for cert auth listing
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Stub out initial health check command
This command will be used to generate health check results for the PKI
engine.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Start common health check implementation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add common health check utilities
These utilities will collect helpers not specific to PKI health checks,
such as formatting longer durations more legibly.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add PKI health check common utils
Many health checks will need issuer and/or CRL information in order to
execute. We've centrally located these helpers to avoid particular
health checks from needing to reimplement them each time.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Adding ca_validity_period health check
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Begin using health-checks in PKI command
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Allow parsing raw requests afterwards
This shifts the last of the logic difference between Read(...) and
ReadRaw(...) to a new helper, allowing ReadRaw(...) requests to be
parsed into the same response structure afterwards as Read(...); this
allows API callers to fetch the raw secret and inspect the raw response
object in case something went wrong (error code &c) -- and when the
request succeeds, they can still get the api.Secret out.
This will be used with the PKI health check functionality, making both
LIST and READ operations use ReadRaw, and optionally parsing the secret
afterwards.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add crl_validity_period health check
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for PKI health check
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix bug in raw reading with contexts
When reading raw objects, don't manually call the context cancellation:
this causes timeouts and/or EOF errors when attempting to read or parse
the response body. See message in client.RawRequestWithContext(...) for
more information.
This was causing the test suite to randomly fail, due to the context
cancelling. The test suite's client usually had a default timeout,
whereas the CLI didn't, and thus didn't exhibit the same issue.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Fix typo in permissions message
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move %v->%w for errs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
This commit adds some logic to handle the case where a mount entry has a
non-builtin RunningVersion. This ensures that we only report deprecation
status for builtins.
* Started work on adding log-file support to Agent
* Allow log file to be picked up and appended
* Use NewLogFile everywhere
* Tried to pull out the config aggregation from Agent.Run
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Expose ssh algorithm_signer in web interface (#10114)
* Adds allowed values for algorithm_signer to ssh plugin API
* Adds algorithm_signer as field in UI
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
The current behaviour is to only add mount paths into the generated `opeanpi.json` spec if a `generic_mount_paths` flag is added to the request. This means that we would have to maintain two different `openapi.json` files, which is not ideal. The new solution in this PR is to add `{mount_path}` into every path with a default value specified:
```diff
-- "/auth/token/accessors/": {
++ "/auth/{mount_path}/accessors/": {
"parameters": [
{
"name": "mount_path",
"description": "....",
"in": "path",
"schema": {
"type": "string",
++ "default": "token"
}
}
],
```
Additionally, fixed the logic to generate the `operationId` (used to generate method names in the code generated from OpenAPI spec). It had a bug where the ID had `mountPath` in it. The new ID will look like this:
```diff
-- "operationId": "listAuthMountpathAccessors",
++ "operationId": "listTokenAccessors",
```
* Add automatic tidy of expired issuers
To aid PKI users like Consul, which periodically rotate intermediates,
and provided a little more consistency with older versions of Vault
which would silently (and dangerously!) replace the configured CA on
root/intermediate generation, we introduce an automatic tidy of expired
issuers.
This includes a longer safety buffer (1 year) and logging of the
relevant issuer information prior to deletion (certificate contents, key
ID, and issuer ID/name) to allow admins to recover this value if
desired, or perform further cleanup of keys.
From my PoV, removal of the issuer is thus a relatively safe operation
compared to keys (which I do not feel comfortable removing) as they can
always be re-imported if desired. Additionally, this is an opt-in tidy
operation, not enabled by default. Lastly, most major performance
penalties comes with lots of issuers within the mount, not as much
large numbers of keys (as only new issuer creation/import operations are
affected, unlike LIST /issuers which is a public, unauthenticated
endpoint).
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for tidy
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add docs on tidy of issuers
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Restructure logging
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add missing fields to expected tidy output
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correctly preserve other issuer config params
When setting a new default issuer, our helper function would overwrite
other parameters in the issuer configuration entry. However, up until
now, there were none.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new parameter to allow default to follow new
This parameter will allow operators to have the default issuer
automatically update when a new root is generated or a single issuer
with a key (potentially with others lacking key) is imported.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Storage migration tests fail on new members
These internal members shouldn't be tested by the storage migration
code, and so should be elided from the test results.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Follow new issuer on root generation, import
This updates the two places where issuers can be created (outside of
legacy CA bundle migration which already sets the default) to follow
newly created issuers when the config is set.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add test for new default-following behavior
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add new API to PKI to list revoked certificates
- A new API that will return the list of serial numbers of
revoked certificates on the local cluster.
* Add cl
* PR feedback
* Ensure correct write ordering in rebuildIssuersChains
When troubleshooting a recent migration failure from 1.10->1.11, it was
noted that some PKI mounts had bad chain construction despite having
valid, chaining issuers. Due to the cluster's leadership trashing
between nodes, the migration logic was re-executed several times,
partially succeeding each time. While the legacy CA bundle migration
logic was written with this in mind, one shortcoming in the chain
building code lead us to truncate the ca_chain: by sorting the list of
issuers after including non-written issuers (with random IDs), these
issuers would occasionally be persisted prior to storage _prior_ to
existing CAs with modified chains.
The migration code carefully imported the active issuer prior to its
parents. However, due to this bug, there was a chance that, if write to
the pending parent succeeded but updating the active issuer didn't, the
active issuer's ca_chain field would only contain the self-reference and
not the parent's reference as well. Ultimately, a workaround of setting
and subsequently unsetting a manual chain would force a chain
regeneration.
In this patch, we simply fix the write ordering: because we need to
ensure a stable chain sorting, we leave the sort location in the same
place, but delay writing the provided referenceCert to the last
position. This is because the reference is meant to be the user-facing
action: without transactional write capabilities, other chains may
succeed, but if the last user-facing action fails, the user will
hopefully retry the action. This will also correct migration, by
ensuring the subsequent issuer import will be attempted again,
triggering another chain build and only persisting this issuer when
all other issuers have also been updated.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remigrate ca_chains to fix any missing issuers
In the previous commit, we identified an issue that would occur on
legacy issuer migration to the new storage format. This is easy enough
to detect for any given mount (by an operator), but automating scanning
and remediating all PKI mounts in large deployments might be difficult.
Write a new storage migration version to regenerate all chains on
upgrade, once.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add issue to PKI considerations documentation
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Correct %v -> %w in chain building errs
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* moves service worker message event listener from addon to raft-storage-overview component
* adds changelog entry
* adds raft-storage-overview test for downloading snapshot via service worker
By adding the link flags `-s -w` we can reduce the Vault binary size
from 204 MB to 167 MB (about 18% reduction in size).
This removes the DWARF section of the binary.
i.e., before:
```
$ objdump --section-headers vault-debug
vault-debug: file format mach-o arm64
Sections:
Idx Name Size VMA Type
0 __text 03a00340 0000000100001000 TEXT
1 __symbol_stub1 00000618 0000000103a01340 TEXT
2 __rodata 00c18088 0000000103a01960 DATA
3 __rodata 015aee18 000000010461c000 DATA
4 __typelink 0004616c 0000000105bcae20 DATA
5 __itablink 0000eb68 0000000105c10fa0 DATA
6 __gosymtab 00000000 0000000105c1fb08 DATA
7 __gopclntab 02a5b8e0 0000000105c1fb20 DATA
8 __go_buildinfo 00008c10 000000010867c000 DATA
9 __nl_symbol_ptr 00000410 0000000108684c10 DATA
10 __noptrdata 000fed00 0000000108685020 DATA
11 __data 0004e1f0 0000000108783d20 DATA
12 __bss 00052520 00000001087d1f20 BSS
13 __noptrbss 000151b0 0000000108824440 BSS
14 __zdebug_abbrev 00000129 000000010883c000 DATA, DEBUG
15 __zdebug_line 00651374 000000010883c129 DATA, DEBUG
16 __zdebug_frame 001e1de9 0000000108e8d49d DATA, DEBUG
17 __debug_gdb_scri 00000043 000000010906f286 DATA, DEBUG
18 __zdebug_info 00de2c09 000000010906f2c9 DATA, DEBUG
19 __zdebug_loc 00a619ea 0000000109e51ed2 DATA, DEBUG
20 __zdebug_ranges 001e94a6 000000010a8b38bc DATA, DEBUG
```
And after:
```
$ objdump --section-headers vault-no-debug
vault-no-debug: file format mach-o arm64
Sections:
Idx Name Size VMA Type
0 __text 03a00340 0000000100001000 TEXT
1 __symbol_stub1 00000618 0000000103a01340 TEXT
2 __rodata 00c18088 0000000103a01960 DATA
3 __rodata 015aee18 000000010461c000 DATA
4 __typelink 0004616c 0000000105bcae20 DATA
5 __itablink 0000eb68 0000000105c10fa0 DATA
6 __gosymtab 00000000 0000000105c1fb08 DATA
7 __gopclntab 02a5b8e0 0000000105c1fb20 DATA
8 __go_buildinfo 00008c20 000000010867c000 DATA
9 __nl_symbol_ptr 00000410 0000000108684c20 DATA
10 __noptrdata 000fed00 0000000108685040 DATA
11 __data 0004e1f0 0000000108783d40 DATA
12 __bss 00052520 00000001087d1f40 BSS
13 __noptrbss 000151b0 0000000108824460 BSS
```
The only side effect I have been able to find is that it is no longer
possible to use [delve](https://github.com/go-delve/delve) to run the
Vault binary.
Note, however, that running delve and other debuggers requires access
to the full source code, which isn't provided for the Enterprise, HSM,
etc. binaries, so it isn't possible to debug those anyway outside of
people who have the full source.
* panic traces
* `vault debug`
* error messages
* Despite what the documentation says, these flags do *not* delete the
function symbol table (so it is not the same as having a `strip`ped
binary).
It contains mappings between the compiled binary and functions,
paramters, and variables in the source code.
Using `llvm-dwarfdump`, it looks like:
```
0x011a6d85: DW_TAG_subprogram
DW_AT_name ("github.com/hashicorp/vault/api.(*replicationStateStore).recordState")
DW_AT_low_pc (0x0000000000a99300)
DW_AT_high_pc (0x0000000000a99419)
DW_AT_frame_base (DW_OP_call_frame_cfa)
DW_AT_decl_file ("/home/swenson/vault/api/client.go")
DW_AT_external (0x01)
0x011a6de1: DW_TAG_formal_parameter
DW_AT_name ("w")
DW_AT_variable_parameter (0x00)
DW_AT_decl_line (1735)
DW_AT_type (0x00000000001e834a "github.com/hashicorp/vault/api.replicationStateStore *")
DW_AT_location (0x009e832a:
[0x0000000000a99300, 0x0000000000a9933a): DW_OP_reg0 RAX
[0x0000000000a9933a, 0x0000000000a99419): DW_OP_call_frame_cfa)
0x011a6def: DW_TAG_formal_parameter
DW_AT_name ("resp")
DW_AT_variable_parameter (0x00)
DW_AT_decl_line (1735)
DW_AT_type (0x00000000001e82a2 "github.com/hashicorp/vault/api.Response *")
DW_AT_location (0x009e8370:
[0x0000000000a99300, 0x0000000000a9933a): DW_OP_reg3 RBX
[0x0000000000a9933a, 0x0000000000a99419): DW_OP_fbreg +8)
0x011a6e00: DW_TAG_variable
DW_AT_name ("newState")
DW_AT_decl_line (1738)
DW_AT_type (0x0000000000119f32 "string")
DW_AT_location (0x009e83b7:
[0x0000000000a99385, 0x0000000000a99385): DW_OP_reg0 RAX, DW_OP_piece 0x8, DW_OP_piece 0x8
[0x0000000000a99385, 0x0000000000a993a4): DW_OP_reg0 RAX, DW_OP_piece 0x8, DW_OP_reg3 RBX, DW_OP_piece 0x8
[0x0000000000a993a4, 0x0000000000a993a7): DW_OP_piece 0x8, DW_OP_reg3 RBX, DW_OP_piece 0x8)
```
This says that the particular binary section is the function
`github.com/hashicorp/vault/api.(*replicationStateStore).recordState`,
from the file `/home/swenson/vault/api/client.go`, containing
the `w` parameter on line 1735 mapped to certain registers and memory,
the `resp` paramter on line 1735 mapped to certain reigsters and memory,
and the `newState` variable on line 1738, mapped to certain registers,
and memory.
It's really only useful for a debugger.
Anyone running the code in a debugger will need full access the source
code anyway, so presumably they will be able to run `make dev` and build
the version with the DWARF sections intact, and then run their debugger.
* Return revocation info within existing certs/<serial> api
- The api already returned both the certificate and a revocation_time
field populated. Update the api to return revocation_time_rfc3339
as we do elsewhere and also the issuer id if it was revoked.
- This will allow callers to associate a revoked cert with an issuer
* Add cl
* PR feedback (docs update)
* Expose raw request from client.Logical()
Not all Vault API endpoints return well-formatted JSON objects.
Sometimes, in the case of the PKI secrets engine, they're not even
printable (/pki/ca returns a binary (DER-encoded) certificate). While
this endpoint isn't authenticated, in general the API caller would
either need to use Client.RawRequestWithContext(...) directly (which
the docs advise against), or setup their own net/http client and
re-create much of Client and/or Client.Logical.
Instead, exposing the raw Request (via the new ReadRawWithData(...))
allows callers to directly consume these non-JSON endpoints like they
would nearly any other endpoint.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add raw formatter for direct []byte data
As mentioned in the previous commit, some API endpoints return non-JSON
data. We get as far as fetching this data (via client.Logical().Read),
but parsing it as an api.Secret fails (as in this case, it is non-JSON).
Given that we intend to update `vault read` to support such endpoints,
we'll need a "raw" formatter that accepts []byte-encoded data and simply
writes it to the UI.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support for reading raw API endpoints
Some endpoints, such as `pki/ca` and `pki/ca/pem` return non-JSON
objects. When calling `vault read` on these endpoints, an error
is returned because they cannot be parsed as api.Secret instances:
> Error reading pki/ca/pem: invalid character '-' in numeric literal
Indeed, we go to all the trouble of (successfully) fetching this value,
only to be unable to Unmarshal into a Secrets value. Instead, add
support for a new -format=raw option, allowing these endpoints to be
consumed by callers of `vault read` directly.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog entry
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Remove panic
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add regression test for default CRL expiry
Also fixes a bug w.r.t. upgrading older entries and missing the Delta
Rebuild Interval field, setting it to the default.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog for earlier PR
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add support for PKCSv1_5_NoOID signatures
This assumes a pre-hashed input has been provided to Vault, but we do
not write the hash's OID into the signature stream. This allows us to
generate the alternative PKCSv1_5_NoOID signature type rather than the
existing PKCSv1_5_DERnull signature type we presently use.
These are specified in RFC 3447 Section 9.2.
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add changelog
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Exclude new none type from PSS based tests
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Add tests for PKCS#1v1.5 signatures
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* adds error handling to auth-jwt component for missing roles and fixes bug where role wasn't being retained when using alternate oidc mount path at login
* fixes jwt login bug from auth mount tabs and adds test
* updates okta-number-challenge success value to arg in template
* adds changelog entry
* fixes issues logging in manually with jwt
* reverts mistaken change
* Store login MFA secret with tokenhelper
* Clean up and refactor tokenhelper paths
* Refactor totp test code for re-use
* Add login MFA command tests
* Use longer sleep times and sha512 for totp test
* Add changelog
Chelsea Shaw