Commit graph

2763 commits

Author SHA1 Message Date
Nick Cabatoff 559754d580
Break grabLockOrStop into two pieces to facilitate investigating deadlocks (#17187)
Break grabLockOrStop into two pieces to facilitate investigating deadlocks.  Without this change, the "grab" goroutine looks the same regardless of who was calling grabLockOrStop, so there's no way to identify one of the deadlock parties.
2022-09-20 11:03:16 -04:00
Nick Cabatoff d8101f82ee
Handle when pluginCatalog.Get returns (nil,nil) during cred backend creation (#17204) 2022-09-20 08:57:08 -04:00
Nick Cabatoff c7f4d79684
We don't need to test LifetimeWatcher's behaviour with database leases specifically. (#17208) 2022-09-20 08:23:51 -04:00
Tom Proctor f5655ae857
Plugins: Consistently use plugin_version (#17171)
* Delete Sha field, rename RunningSha -> RunningSha256
* Rename version -> plugin_version
2022-09-20 12:35:50 +01:00
Steven Clark dae2ef535b
Update protos to match update of protobuf go library (#17215) 2022-09-19 16:45:44 -04:00
Steven Clark 05a5928b8d
Update missing go-kms-wrapping v2 dep and address some ENT->OSS drift (#17178)
* Update missing go-kms-wrapping v2 dep and address some ENT->OSS drift

* Bump go-kms-wrapping/wrappers/gcpckms/v2 to v2.0.1
2022-09-19 10:23:40 -04:00
Mike Palmiotto 2bb11d2d4c
semgrep: Add replication-has-state and fix findings (#17179) 2022-09-19 08:15:27 -04:00
Nick Cabatoff b7c5dbd713
Reduce time taken to run the vault test package (#17157)
Factored out some plugin related tests into their own test package, and added a bunch of parallelism.  Moved some non-plugin tests that were in logical_system_integ_test into another file (keeping them in vault package) just for cohesion.
2022-09-16 09:53:16 -04:00
Christopher Swenson b136a7ecd8
Add plugin version to GRPC interface (#17088)
Add plugin version to GRPC interface

Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems.

I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion:

If a plugin has a PluginVersion() method implemented, then that is used
If not, and the plugin is built into the Vault binary, then the go.mod version is used
Otherwise, the it will be the empty string.
My apologies for the length of this PR.

* Placeholder backend should be external

We use a placeholder backend (previously a framework.Backend) before a
GRPC plugin is lazy-loaded. This makes us later think the plugin is a
builtin plugin.

So we added a `placeholderBackend` type that overrides the
`IsExternal()` method so that later we know that the plugin is external,
and don't give it a default builtin version.
2022-09-15 16:37:59 -07:00
Josh Black 1e6401a8eb
make proto (#17120) 2022-09-13 16:06:11 -04:00
Josh Black 6d94dd991d
merkle sync undo logs (#17103) 2022-09-13 10:03:19 -07:00
georgethebeatle f9439a9c41
Make key completion work for both kv-v1 and kv-v2 (#16553)
Co-authored-by: Kieron Browne <kbrowne@vmware.com>
Co-authored-by: Georgi Sabev <georgethebeatle@gmail.com>
Co-authored-by: Danail Branekov <danailster@gmail.com>
2022-09-13 12:11:00 -04:00
Hamid Ghaf 77ec84cfb1
updating hcp link structs, and fix diagnose (#17097) 2022-09-12 11:10:01 -04:00
Max Coulombe 6b2f4e5354
+ added redis elasticache as a built-in plugin (#17075)
* added redis elasticache as a built-in plugin
2022-09-09 16:16:30 -04:00
Mike Palmiotto 9849af8663
Add deprecation status to plugin api and cli (#17077)
* api: Add deprecation status to plugin endpoints

* cli: Add -detailed flag to `plugin list`

* docs: Update plugin list/info docs
2022-09-09 16:03:07 -04:00
Hamid Ghaf 102f5f6832
node status as a module to be importable by HCP cloud (#17089) 2022-09-09 14:51:05 -04:00
Milena Zlaticanin 0977bd1ddc
Import Redis OSS database plugin into Vault (#17070)
* Import Redis OSS database plugin into Vault

* update the total number of db plugins

* small nit for testing

* adding changelog
2022-09-09 13:42:25 -05:00
Tom Proctor 65adf42d48
Support running versioned plugins from the catalog (#17015) 2022-09-09 18:14:26 +01:00
Tom Proctor aa50e42fca
Support version selection for database plugins (#16982)
* Support version selection for database plugins
* Don't consider unversioned plugins for version selection algorithm
* Added version to 'plugin not found' error
* Add PluginFactoryVersion function to avoid changing sdk/ API
2022-09-09 17:32:28 +01:00
Nick Cabatoff 3075c5bd65
Do not attempt to write a new TLS keyring at startup if raft is already setup (#17079) 2022-09-09 12:19:57 -04:00
Max Coulombe f9b5d1a563
Multiplexing opt out flag (#16972)
* added mplexing opt-out flag
2022-09-08 11:32:46 -04:00
Mike Palmiotto 403fdd77be
api: Add deprecation warnings to secrets/auth POST endpoints (#17058)
* api: Add deprecation warnings to endpoints
* Add changelog
2022-09-08 09:15:10 -04:00
Josh Black d8e0a13aae
update gofumpt to 0.3.1 and reformat the repo (#17055)
* update gofumpt to 0.3.1 and reformat the repo

* output the version of the formatter we're using
2022-09-07 17:31:20 -07:00
Nick Cabatoff 7842b861b3
Make some activity log tests less flaky (#17028)
* OSS parts of ent #3157.  Some activity log tests were flaky because background workers could race with them; now we overload DisableTimers to stop some of them from running, and add some channels we can use to wait for others to complete before we start testing.

* Add CL
2022-09-07 09:06:15 -04:00
Mike Palmiotto bf744e3bde
Handle deprecated builtins (#17005)
* core: Handle deprecated mounts on enable and unseal
* changelog: Deprecation Status handling
* core: Add Pending Removal override var
* core: Add some documentation for Pending Removal override
2022-09-06 15:49:35 -04:00
Nick Cabatoff 5db952eada
autopilot: assume nodes we haven't received heartbeats from are running the same version as we are (#17019)
OSS parts of ent PR #3172: assume nodes we haven't received heartbeats from are running the same version as we are.  Failing to provide a version/upgrade_version will result in Autopilot (on ent) demoting those unversioned nodes to non-voters until we receive a heartbeat from them.
2022-09-06 14:49:04 -04:00
Hamid Ghaf a034ebfd27
HCP link integration (#16939)
* HCP link integration

* update configure-git.yml

* more OSS stuff

* removing internal repos

* adding a nil check

* removing config test to be included in ENT only

* updating hcp-sdk-go to v0.22.0

* remove Hostname and AuthURL link config params

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-09-06 14:11:04 -04:00
John-Michael Faircloth e6f5ffda4f
plugin: update debug logging (#16953) 2022-09-06 18:56:10 +02:00
Austin Gebauer 1ea50db6c6
identity/oidc: adds claims_supported to discovery document (#16992)
* identity/oidc: adds claims_supported to discovery document

* adds changelog
2022-09-02 09:19:25 -07:00
davidadeleon 24e346bf53
auth/token: Fix ignored parameter warnings for valid parameters on token create (#16938)
* Add fields to schema for relevant paths

* add changelog
2022-09-01 08:32:40 -04:00
Mike Palmiotto 2c16be25e3
Add deprecation status to auth/secrets list (#16849)
* auth: Add Deprecation Status to auth list -detailed
* secrets: Add Deprecation Status to secrets list -detailed
* Add changelog entry for deprecation status list
2022-08-31 16:11:14 -04:00
Christopher Swenson 09ad6ab72c
Update mount table and CLI with plugin version for auth (#16856) 2022-08-31 19:23:05 +01:00
Hridoy Roy 8e7fec59ef
Load SSCT Generation Counter Upon DR Promotion [OSS] (#16956)
* port ssct bugfix to load epoch from storage

* changelog

* update changelog to be user-facing

* change 2 to two
2022-08-31 11:05:21 -07:00
Max Bowsher ff23bbf1e4
Fix incorrect MaxNsIdLength (#16827)
Namespace IDs are 5 characters, not 4.
2022-08-31 08:56:45 -04:00
Alexander Scheel a5fafd8163
Add ability to perform automatic tidy operations (#16900)
* Add ability to perform automatic tidy operations

This enables the PKI secrets engine to allow tidy to be started
periodically by the engine itself, avoiding the need for interaction.
This operation is disabled by default (to avoid load on clusters which
don't need tidy to be run) but can be enabled.

In particular, a default tidy configuration is written (via
/config/auto-tidy) which mirrors the options passed to /tidy. Two
additional parameters, enabled and interval, are accepted, allowing
auto-tidy to be enabled or disabled and controlling the interval
(between successful tidy runs) to attempt auto-tidy.

Notably, a manual execution of tidy will delay additional auto-tidy
operations. Status is reported via the existing /tidy-status endpoint.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation on auto-tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests for auto-tidy

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Prevent race during parallel testing

We modified the RollbackManager's execution window to allow more
faithful testing of the periodicFunc. However, the TestAutoRebuild and
the new TestAutoTidy would then race against each other for modifying
the period and creating their clusters (before resetting to the old
value).

This changeset adds a lock around this, preventing the races.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Use tidyStatusLock to gate lastTidy time

This prevents a data race between the periodic func and the execution of
the running tidy.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add read lock around tidyStatus gauges

When reading from tidyStatus for computing gauges, since the underlying
values aren't atomics, we really should be gating these with a read lock
around the status access.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-08-30 15:45:54 -04:00
Violet Hynes 001e060e54
VAULT-6433 do not return nil resp if ns is nil (#16937)
* VAULT-6433 do not return nil resp if ns is nil

* VAULT-6433 typo
2022-08-30 13:53:41 -04:00
John-Michael Faircloth b6c05fae33
feature: secrets/auth plugin multiplexing (#14946)
* enable registering backend muxed plugins in plugin catalog

* set the sysview on the pluginconfig to allow enabling secrets/auth plugins

* store backend instances in map

* store single implementations in the instances map

cleanup instance map and ensure we don't deadlock

* fix system backend unit tests

move GetMultiplexIDFromContext to pluginutil package

fix pluginutil test

fix dbplugin ut

* return error(s) if we can't get the plugin client

update comments

* refactor/move GetMultiplexIDFromContext test

* add changelog

* remove unnecessary field on pluginClient

* add unit tests to PluginCatalog for secrets/auth plugins

* fix comment

* return pluginClient from TestRunTestPlugin

* add multiplexed backend test

* honor metadatamode value in newbackend pluginconfig

* check that connection exists on cleanup

* add automtls to secrets/auth plugins

* don't remove apiclientmeta parsing

* use formatting directive for fmt.Errorf

* fix ut: remove tls provider func

* remove tlsproviderfunc from backend plugin tests

* use env var to prevent test plugin from running as a unit test

* WIP: remove lazy loading

* move non lazy loaded backend to new package

* use version wrapper for backend plugin factory

* remove backendVersionWrapper type

* implement getBackendPluginType for plugin catalog

* handle backend plugin v4 registration

* add plugin automtls env guard

* modify plugin factory to determine the backend to use

* remove old pluginsets from v5 and log pid in plugin catalog

* add reload mechanism via context

* readd v3 and v4 to pluginset

* call cleanup from reload if non-muxed

* move v5 backend code to new package

* use context reload for for ErrPluginShutdown case

* add wrapper on v5 backend

* fix run config UTs

* fix unit tests

- use v4/v5 mapping for plugin versions
- fix test build err
- add reload method on fakePluginClient
- add multiplexed cases for integration tests

* remove comment and update AutoMTLS field in test

* remove comment

* remove errwrap and unused context

* only support metadatamode false for v5 backend plugins

* update plugin catalog errors

* use const for env variables

* rename locks and remove unused

* remove unneeded nil check

* improvements based on staticcheck recommendations

* use const for single implementation string

* use const for context key

* use info default log level

* move pid to pluginClient struct

* remove v3 and v4 from multiplexed plugin set

* return from reload when non-multiplexed

* update automtls env string

* combine getBackend and getBrokeredClient

* update comments for plugin reload, Backend return val and log

* revert Backend return type

* allow non-muxed plugins to serve v5

* move v5 code to existing sdk plugin package

* do next export sdk fields now that we have removed extra plugin pkg

* set TLSProvider in ServeMultiplex for backwards compat

* use bool to flag multiplexing support on grpc backend server

* revert userpass main.go

* refactor plugin sdk

- update comments
- make use of multiplexing boolean and single implementation ID const

* update comment and use multierr

* attempt v4 if dispense fails on getPluginTypeForUnknown

* update comments on sdk plugin backend
2022-08-29 21:42:26 -05:00
Violet Hynes 8c9c1d2b2a
VAULT-6433: Add namespace path to MFA read/list endpoints (#16911)
* VAULT-6433 Add namespace_path to MFA endpoints

* VAULT-6433 add changelog

* VAULT-6433 Return error in case of error

* VAULT-6433 Make logic a bit more concise
2022-08-29 09:11:25 -04:00
Nick Cabatoff df61151034
Wait for standby to have a working grpc connection before we try to use it (#16905)
Also teach WaitForStandbyNode to do a better job waiting for standbys to be healthy.
2022-08-26 12:50:10 -04:00
Tom Proctor 4edf768f17
Version-aware plugin catalog (#16688)
Adds support for using semantic version information when registering
and managing plugins. New `detailed` field in the response data for listing
plugins and new `version` field in the response data for reading a
single plugin.
2022-08-25 21:31:42 +01:00
Mike Palmiotto 6a438fd087
Vault 7133/registry status (#16846)
* plugins: Add Deprecation Status to builtinRegistry

* changelog: Deprecation Status method
2022-08-23 16:34:30 -04:00
Jason O'Donnell 1200020fdc
identity/entity-alias: fix bug where alias metadata was shared if alias had same name (#16838) 2022-08-23 15:39:45 -04:00
Scott Miller 3bd38fd5dc
OSS portion of wrapper-v2 (#16811)
* OSS portion of wrapper-v2

* Prefetch barrier type to avoid encountering an error in the simple BarrierType() getter

* Rename the OveriddenType to WrapperType and use it for the barrier type prefetch

* Fix unit test
2022-08-23 15:37:16 -04:00
Alexander Scheel cacb23bda6
Enable periodic, automatic rebuilding of CRLs (#16762)
* Allow automatic rebuilding of CRLs

When enabled, periodic rebuilding of CRLs will improve PKI mounts in two
way:

 1. Reduced load during periods of high (new) revocations, as the CRL
    isn't rebuilt after each revocation but instead on a fixed schedule.
 2. Ensuring the CRL is never stale as long as the cluster remains up,
    by checking for next CRL expiry and regenerating CRLs before that
    happens. This may increase cluster load when operators have large
    CRLs that they'd prefer to let go stale, rather than regenerating
    fresh copies.

In particular, we set a grace period before expiration of CRLs where,
when the periodic function triggers (about once a minute), we check
upcoming CRL expirations and check if we need to rebuild the CRLs.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add changelog entry

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add documentation on periodic rebuilding

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Allow modification of rollback period for testing

When testing backends that use the periodic func, and specifically,
testing the behavior of that periodic func, waiting for the usual 1m
interval can lead to excessively long test execution. By switching to a
shorter period--strictly for testing--we can make these tests execute
faster.

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Add tests for auto-rebuilding of CRLs

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Remove non-updating getConfig variant

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Avoid double reload of config

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-08-23 13:27:15 -04:00
Violet Hynes 0b3d7fdf10
VAULT-7698 Fix ignored parameter warnings for endpoint arbitrary data options (#16794)
* VAULT-7698 Fix warnings for endpoint arbitrary data options

* VAULT-7698 Add changelog
2022-08-23 08:51:23 -04:00
Hridoy Roy 7fb60a7617
oss port of vault-7225-bugfix (#16745) 2022-08-16 16:38:11 -07:00
Hridoy Roy 91ef527a1f
re-add namespace attribution to current month (#16473)
* re-add namespace attribution to current month

* delete cl

* parity with ent branch
2022-08-16 16:27:20 -07:00
Hridoy Roy 54444b9d2e
handle ssct errors in one other place (#16497) 2022-08-15 14:16:32 -07:00
Anton Averchenkov 6d45a421ff
Add a sentinel error for missing KV secrets (#16699) 2022-08-12 19:29:42 -04:00
John-Michael Faircloth bb58775bb5
secrets/auth: fix bug with aliased backends (#16673)
* secrets/auth: fix bug with aliased backends

* add changelog

* update changelog to include affected backends
2022-08-10 20:02:05 -05:00
John-Michael Faircloth dd4fb82b5d
unit test: remove postgres dependecy from testing.go (#16675) 2022-08-10 17:01:24 -05:00
Violet Hynes 4850a3ff0e
VAULT-6818 - Restrict ability to merge entities with mount-accessor-conflicting aliases unless one is explicitly chosen to be kept (#16539)
* VAULT-6818 delete unmerged entity aliases instead of orphaning them

* VAULT-6818 Prevent merge with clashing aliases, allow for resolution of clashing entity aliases

* VAULT-6818 Small updates

* VAULT-6818 Restrict to only one clash merge at once

* VAULT-6818 changelog

* VAULT-6818 use strutil package instead of slices

* VAULT-6818 Update variable names for clarity

* VAULT-6818 Update test

* VAULT-6818 update error message

* VAULT-6818 Use helper method

* VAULT-6818 validate entityIds

* VAULT-6818 group imports better

* VAULT-6818 use change instead of bug

* VAULT-6818 use multierror instead of custom struct

* VAULT-6818 Use multierror properly

* VAULT-6818 Small refactor based on feedback
2022-08-10 09:10:02 -04:00
Austin Gebauer ed143c5678
identity/oidc: reorder authorization endpoint validation for invalid redirect uris (#16601)
* identity/oidc: reorder authorization endpoint validation for invalid redirect uris

* adds changelog

* use provider.allowedClientID
2022-08-08 09:02:18 -07:00
Austin Gebauer 59831a8d5c
identity/oidc: adds client_secret_post token endpoint authentication method (#16598)
* identity/oidc: adds client_secret_post token endpoint authentication method

* fix test

* adds changelog
2022-08-08 08:41:09 -07:00
Austin Gebauer 67339b71e8
identity/oidc: fixes validation of the request and request_uri parameters (#16600)
* identity/oidc: add request_parameter_supported to discovery document

* adds changelog
2022-08-05 11:55:15 -07:00
Austin Gebauer a2bc8cfb96
identity/oidc: change the state parameter to optional (#16599)
* identity/oidc: change the state parameter to optional

* adds changelog

* update docs
2022-08-05 11:37:24 -07:00
Austin Gebauer e2d3846a25
identity/oidc: adds detailed listing capability for clients and providers (#16567)
* identity/oidc: adds detailed listing capability for clients and providers

* change approach to use ListResponseWithInfo

* adds changelog
2022-08-04 10:10:28 -07:00
Violet Hynes ac582c86cd
VAULT-7432 Fix flaky expiration behaviour (#16586) 2022-08-04 10:47:14 -04:00
Mike Palmiotto cd1157a905
Vault 7338/fix retry join (#16550)
* storage/raft: Fix cluster init with retry_join

Commit 8db66f4853abce3f432adcf1724b1f237b275415 introduced an error
wherein a join() would return nil (no error) with no information on its
channel if a joining node had been initialized. This was not handled
properly by the caller and resulted in a canceled `retry_join`.

Fix this by handling the `nil` channel respone by treating it as an
error and allowing the existing mechanics to work as intended.

* storage/raft: Improve retry_join go test

* storage/raft: Make VerifyRaftPeers pollable

* storage/raft: Add changelog entry for retry_join fix

* storage/raft: Add description to VerifyRaftPeers
2022-08-03 20:44:57 -05:00
Mike Palmiotto 42900b554b
storage/raft: Make raftInfo atomic (#16565)
* storage/raft: Make raftInfo atomic

This fixes some racy behavior discovered in parallel testing. Change the
core struct member to an atomic and update references throughout.
2022-08-03 18:40:49 -04:00
Eng Zer Jun 61262ad98e
refactor: replace strings.Replace with strings.ReplaceAll (#15392)
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-08-03 15:22:48 -04:00
swayne275 4632a26a09
Use %q for quoted strings where appropriate (#15216)
* change '%s' to %q where single vs double quotes shouldn't matter

* replace double quotes with %q in logs and errors
2022-08-03 12:32:45 -06:00
Violet Hynes 6f89461d76
VAULT-7065 move lease-quota updating to process restore (#16422)
* VAULT-7065 move lease-quota updating to process restore

* VAULT-7065 Avoid off-by-one issue that can happen during Restore

* VAULT-7065 Add comment to explain why we don't count creates in restore mode
2022-08-02 09:43:54 -04:00
John-Michael Faircloth 528373de6c
oidc provider: use identity store's storage view in test (#16520) 2022-08-01 09:37:23 -06:00
Austin Gebauer b3f138679c
identity/oidc: allow filtering the list providers response by an allowed_client_id (#16181)
* identity/oidc: allow filtering the list providers response by an allowed_client_id

* adds changelog

* adds api documentation

* use identity store view in list provider test
2022-07-28 09:47:53 -07:00
Chris Capurso 013e1d12b1
move custom metadata validation logic to its own package (#16464)
* move custom metadata validation logic to its own package

* add comments

* add custom metadata Validate unit tests
2022-07-28 10:40:38 -04:00
Nick Cabatoff 488858e919
Clone created entities that were inserted into memdb... (#16487)
* Clone created entities that were inserted into memdb to prevent possibility of data race.
2022-07-28 09:43:24 -04:00
Nick Cabatoff b9181077fd
Fix a panic at cleanup time in an expiration restore lease benchmark. (#16485) 2022-07-28 05:54:03 -07:00
Hridoy Roy 3429d7dfc5
Current Month Activity Estimate, Probabilistic Tests, And Bugfixes (#16447)
* bugfixes and probabilistic hll tests

* changelog

* changelog fix

* remove activity log test and keep in ent

* update cl
2022-07-26 13:00:27 -07:00
Anton Averchenkov 166c618589
Fix linter issues in policy.go & acl.go (#16366) 2022-07-22 14:13:14 -04:00
Violet Hynes 8163271ee2
VAULT-7046 Allow trailing globbing at the end of a path suffix quota (#16386)
* VAULT-7046 OSS changes for trailing glob quotas

* VAULT-7046 allow glob of 'a*' to match 'a'

* VAULT-7046 Add changelog

* VAULT-7046 fix minor typo
2022-07-21 15:31:23 -04:00
Pratyoy Mukhopadhyay 77ca499c6e
oss changes (#16407) 2022-07-21 10:53:42 -07:00
Austin Gebauer 7df39640e0
Update gopsutil to v3 to fix MacOS deprecation warnings (#16321)
* Update gopsutil to v3

* Adds v2 field names in host-info response to allow eventual deprecation in favor of v3 field names

* Map v3 to v2 field names to keep host-info api compat

* copy gopsutil license into source
2022-07-20 16:37:10 -07:00
Brian Kassouf d6bb62a0ab
Increase the allowed concurrent gRPC streams (#16327)
* Increase the allowed concurrent gRPC streams

* Add a env override for the max streams setting

* Add changelog

* go fmt

* fix builds on 32bit systems
2022-07-20 15:26:52 -04:00
John-Michael Faircloth a5349bd1ef
Revert "AutoMTLS for secrets/auth plugins (#15671)" (#16377)
This reverts commit 39bcd5c71529f5f4eb61aae68b17d06d192ea55f.
2022-07-20 10:36:23 -05:00
John-Michael Faircloth 7e170e7d87
AutoMTLS for secrets/auth plugins (#15671)
* use automtls for v5 secrets/auth plugins

* add automtls env guard

* start backend without metadata mode

* use PluginClientConfig for backend's NewPluginClient param

refactor

* - fix pluginutil test
- do not expect plugin to be unloaded in UT
- fix pluginutil tests --need new env var
- use require in UT
- fix lazy load test

* add changelog

* prioritize automtls; improve comments

* user multierror; refactor pluginSet for v4 unit test

* add test cases for v4 and v5 plugin versions

* remove unnecessary call to AutoMTLSSupported

* update comment on pluginSets

* use runconfig directly in sdk newpluginclient

* use automtls without metadatamode for v5 backend plugin registration

* use multierror for plugin runconfig calls

* remove some unnecessary code
2022-07-18 16:25:18 -05:00
Mike Palmiotto 439e35f50f
Vault 6773/raft rejoin nonvoter (#16324)
* raft: Ensure init before setting suffrage

As reported in https://hashicorp.atlassian.net/browse/VAULT-6773:

	The /sys/storage/raft/join endpoint is intended to be unauthenticated. We rely
	on the seal to manage trust.

	It’s possible to use multiple join requests to switch nodes from voter to
	non-voter. The screenshot shows a 3 node cluster where vault_2 is the leader,
	and vault_3 and vault_4 are followers with non-voters set to false.  sent two
	requests to the raft join endpoint to have vault_3 and vault_4 join the cluster
	with non_voters:true.

This commit fixes the issue by delaying the call to SetDesiredSuffrage until after
the initialization check, preventing unauthenticated mangling of voter status.

Tested locally using
https://github.com/hashicorp/vault-tools/blob/main/users/ncabatoff/cluster/raft.sh
and the reproducer outlined in VAULT-6773.

* raft: Return join err on failure

This is necessary to correctly distinguish errors returned from the Join
workflow. Previously, errors were being masked as timeouts.

* raft: Default autopilot parameters in teststorage

Change some defaults so we don't have to pass in parameters or set them
in the originating tests. These storage types are only used in two
places:

1) Raft HA testing
2) Seal migration testing

Both consumers have been tested and pass with this change.

* changelog: Unauthn voter status change bugfix
2022-07-18 14:37:12 -04:00
Hridoy Roy 573e01af1d
Throw SSCT Error In Some Cases (#16270)
* throw the ssct error if it signifies that the server needs to take an action

* use errors.Is instead of checking string comparison
2022-07-11 11:12:02 -07:00
Chris Capurso 068a413311
remove GetCoreConfigInternal from logger API tests (#16263) 2022-07-08 19:23:18 +02:00
Violet Hynes 0c80ee5cf5
VAULT-6614 Enable role based quotas for lease-count quotas (OSS) (#16157)
* VAULT-6613 add DetermineRoleFromLoginRequest function to Core

* Fix body handling

* Role resolution for rate limit quotas

* VAULT-6613 update precedence test

* Add changelog

* VAULT-6614 start of changes for roles in LCQs

* Expiration changes for leases

* Add role information to RequestAuth

* VAULT-6614 Test updates

* VAULT-6614 Add expiration test with roles

* VAULT-6614 fix comment

* VAULT-6614 Protobuf on OSS

* VAULT-6614 Add rlock to determine role code

* VAULT-6614 Try lock instead of rlock

* VAULT-6614 back to rlock while I think about this more

* VAULT-6614 Additional safety for nil dereference

* VAULT-6614 Use %q over %s

* VAULT-6614 Add overloading to plugin backends

* VAULT-6614 RLocks instead

* VAULT-6614 Fix return for backend factory
2022-07-05 13:02:00 -04:00
AnPucel 3215cdbd32
Dynamic parameter for mountpaths in OpenApi Spec generation(#15835)
"generic_mount_paths" query parameter for OpenApiSpec generation
2022-06-30 07:43:04 -07:00
Hridoy Roy 2f14d60a4b
Port: Use Stored Hll to Compute New Clients For Current Month (#16184)
* port hll storage changes

* changelog
2022-06-29 10:51:23 -07:00
Hamid Ghaf fa754c7fa5
Replicate member_entity_ids and policies in identity/group across nodes identically (#16088)
* Replicate values of group member_entity_ids and policies across nodes identically

* Adding CL

* fixing tests
2022-06-28 19:54:24 -04:00
Hridoy Roy 895e422c4c
move unused GetCoreConfigInternal to common file to prevent ent panics (#16165) 2022-06-27 14:41:56 -07:00
Hridoy Roy b3959534c2
activity log refactoring port (#16162)
* activity log refactoring port

* changelog
2022-06-27 13:33:45 -07:00
akshya96 42b13448f9
ActivityLog Implement HyperLogLog Store Functionality During Precomputation (#16146)
* adding hll for each month

* add changelog

* removing influxdb

* removing influxdb

* removing influxdb

* changing switch to if-else for semgrep
2022-06-27 09:38:32 -07:00
Chris Capurso 9501d44ed5
Add endpoints to provide ability to modify logging verbosity (#16111)
* add func to set level for specific logger

* add endpoints to modify log level

* initialize base logger with IndependentLevels

* test to ensure other loggers remain unchanged

* add DELETE loggers endpoints to revert back to config

* add API docs page

* add changelog entry

* remove extraneous line

* add log level field to Core struct

* add godoc for getLogLevel

* add some loggers to c.allLoggers
2022-06-27 11:39:53 -04:00
Violet Hynes 5d21fa1e8f
VAULT-6613 Missed a part of OSS for rate limit role quotas (#16132)
* VAULT-6613 add DetermineRoleFromLoginRequest function to Core

* Fix body handling

* Role resolution for rate limit quotas

* VAULT-6613 update precedence test

* Add changelog

* Handle body error

* VAULT-6613 Return early if error with json parsing

* VAULT-6613 add to teardown function
2022-06-24 09:45:53 -04:00
Violet Hynes d57fea2cd1
VAULT-6613 Add role support for rate limit quotas (OSS Changes) (#16115)
* VAULT-6613 add DetermineRoleFromLoginRequest function to Core

* Fix body handling

* Role resolution for rate limit quotas

* VAULT-6613 update precedence test

* Add changelog

* Handle body error

* VAULT-6613 Return early if error with json parsing
2022-06-24 08:58:02 -04:00
Josh Black 2ee2b6ed7c
Return a 403 for a bad SSCT instead of 500 (#16112) 2022-06-23 13:01:20 -07:00
Jason O'Donnell f957573108
Fix bug where id not existing in multiplexing map causes panic (#16094)
* multiplexing: guard against connection panic

* changelog

* Update vault/plugin_catalog.go

Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>

Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
2022-06-22 14:29:25 -04:00
Violet Hynes 56ed9ca8ae
Start of implementation of the plumbing for role resolution logic on auth mounts (#16049)
* VAULT-6612 Initial scaffolding for role determination

* VAULT-6612 Simplify code

* Fix fmt error that somehow happened

* VAULT-6612 Refactor resolve role response
2022-06-21 09:31:36 -04:00
Nick Cabatoff 27ea9a0e41
Set NumCores=1 since we only return a single client anyway. (#16039) 2022-06-17 11:28:25 -04:00
Violet Hynes 96371ed0a5
Small refactor of mount entry manipulation (#16024) 2022-06-17 08:52:43 -04:00
Violet Hynes abed5cf6e7
(OSS) Path Suffix Support for Rate Limit Quotas (#15989)
* Support for rate limit path suffix quotas

* Support for rate limit path suffix quotas

* Precedence test for support for rate limit path suffix quotas

* Update clone method

* Fix mount determination

* Add changelog

* use constant for mounts

* Fix read endpoint, and remount/disable mount

* update godocs for queryquota
2022-06-16 13:23:02 -04:00
akshya96 7e313e29fd
Activity Log Filtering Limit Parameter (#16000)
* adding changes from ent branch

* adding fmt changes

* adding changelog
2022-06-15 15:41:31 -07:00
swayne275 54262d2f4e
clarify lazy revoke prefix if not sync (#15967)
* clarify lazy revoke prefix if not sync

* comment improvement
2022-06-15 12:03:56 -06:00
Josh Black d2ed39a04e
Correct drift between ENT and OSS (#15966) 2022-06-14 17:53:19 -07:00
Chris Capurso 94c5936e27
return bad request instead of server error for identity group cycle detection (#15912)
* return bad request for identity group cycle detection

* add changelog entry

* use change release note instead of improvement

* fix err reference

* fix TestIdentityStore_GroupHierarchyCases
2022-06-10 10:15:31 -04:00
Hridoy Roy 934989809b
Limit SSCT WAL Check on Perf Standbys to Raft Backends Only (#15879)
* ensure that ssct wal check only occurs for non-raft storage on perf standbys

* changelog
2022-06-08 13:58:22 -07:00
VAL 98ed9c7ab9
Add PutMetadata and PatchMetadata KV helpers (#15755)
* Add PutMetadata and PatchMetadata KV helpers

* Refactor tests, stop needlessly converting zero values to nil

* Merge fix for delete-version-after zero value bug

* Cast nils to zero value on Put

* Use 2 different structs for put/patch metadata input, clarity fixes

* Use local constants and simplify stringification
2022-06-08 09:29:22 -07:00
Alexander Scheel b3ad79fb70
Fix listing_visibility value documentation (#15833)
* Match listing_visibility in system/auth with system/mounts

See also: #15209

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

* Fix path-help for listing_visibility

Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-06-06 16:44:46 -04:00
Josh Black 9c48c62d6e
Use the incoming request version to populate follower state (#15806) 2022-06-06 08:44:24 -07:00
Violet Hynes d62b140b7c
VAULT-6371 Fix issue with lease quotas on read requests that generate leases (#15735)
* VAULT-6371 Fix issue with lease quotas on non-auth mounts

* VAULT-6371 Add changelog

* VAULT-6371 Amend changelog given new understanding
2022-06-03 15:45:21 -04:00
VAL 3cfafe619b
KV helpers for DeleteMetadata, Undelete, Destroy, and Rollback (#15637)
* KV helpers for DeleteMetadata, Undelete, Destroy, and Rollback

* Allow rollback when no secret data on latest version, and update error messages
2022-06-03 10:42:43 -07:00
Hridoy Roy e64d7df041
refactor some code in modifyResponseMonths and ensure that the last mo… (#15767)
* refactr some code in modifyResponseMonths and ensure that the last month comparison with end is comparing end of month with end of month

* calibrate end of month apropriately and fix parens issue for lastmonth
2022-06-03 10:34:54 -07:00
Hridoy Roy 671aaf1fe0
iterate through all available logs for precomputation and query gets (#15768) 2022-06-03 09:53:53 -07:00
Hridoy Roy a5f70d7fe0
fix off by one error in activity log nil padding for month data (#15731) 2022-06-01 11:09:06 -07:00
VAL ed7c1d4800
Add Patch KV helper (#15587)
* Add Read methods for KVClient

* KV write helper

* Add changelog

* Add Delete method

* Use extractVersionMetadata inside extractDataAndVersionMetadata

* Return nil, nil for v1 writes

* Add test for extracting version metadata

* Split kv client into v1 and v2-specific clients

* Add ability to set options on Put

* Add test for KV helpers

* Add custom metadata to top level and allow for getting versions as sorted slice

* Update tests

* Separate KV v1 and v2 into different files

* Add test for GetVersionsAsList, rename Metadata key to VersionMetadata for clarity

* Move structs and godoc comments to more appropriate files

* Add more tests for extract methods

* Rework custom metadata helper to be more consistent with other helpers

* Remove KVSecret from custom metadata test now that we don't append to it as part of helper method

* Add Patch KV helper

* Add godoc comment and use WithOption ourselves in other KVOption functions

* Clean up options-handling and resp parsing logic; add more tests

* Add constants and more patch tests
2022-06-01 07:50:56 -07:00
Hamid Ghaf bf087f9d0d
prevent deleting MFA method through an invalid path (#15482)
* prevent deleting MFA method through an invalid path

* Adding CL
2022-05-31 14:22:04 -04:00
Nick Cabatoff 69c5e8c946
Avoid deadlocking on stateLock in emitMetrics (#15693)
When stopCh is closed we should stop trying to get the lock.
2022-05-31 12:15:39 -04:00
Violet Hynes 4aac96238c
VAULT-6131 OpenAPI schema now includes /auth/token endpoints when explicit permission has been granted (#15552)
* VAULT-6131 OpenAPI schema now includes /auth/token endpoints when explicit permission has been granted

* VAULT-6131 add changelog

* VAULT-6131 Update changelog and fix related bug
2022-05-31 11:25:27 -04:00
Nick Cabatoff ea099fdffd
Like #15682 but this time fix the correct test, i.e. TestRateLimitQuota_Allow (#15684) 2022-05-30 15:08:01 -04:00
Nick Cabatoff 8c2b69f961
Spawn fewer goroutines to make TestRateLimitQuota_Allow less vulnerable to scheduler. (#15682) 2022-05-30 13:33:41 -04:00
Chris Capurso cdb73ab265
use provided namespace for wrapping lookup cubbyhole request (#15583)
* use provided namespace for wrapping lookup cubbyhole request

* add changelog entry
2022-05-26 15:17:29 -04:00
Peter Wilson b7fc4645f3
Only add distinct policies to identity group (#15638)
* Only add distinct policies to identity group
2022-05-26 13:52:19 +01:00
John-Michael Faircloth fc04699f57
Fix plugin reload mounts (#15579)
* fix plugin reload mounts

* do not require sys/ prefix

* update plugin reload docs with examples

* fix unit test credential read path

* update docs to reflect correct cli usage

* allow sys/auth/foo or auth/foo

* append trailing slash if it doesn't exist in request

* add changelog

* use correct changelog number
2022-05-25 13:37:42 -05:00
VAL 64448b62a4
KV helper methods for api package (#15305)
* Add Read methods for KVClient

* KV write helper

* Add changelog

* Add Delete method

* Use extractVersionMetadata inside extractDataAndVersionMetadata

* Return nil, nil for v1 writes

* Add test for extracting version metadata

* Split kv client into v1 and v2-specific clients

* Add ability to set options on Put

* Add test for KV helpers

* Add custom metadata to top level and allow for getting versions as sorted slice

* Update tests

* Separate KV v1 and v2 into different files

* Add test for GetVersionsAsList, rename Metadata key to VersionMetadata for clarity

* Move structs and godoc comments to more appropriate files

* Add more tests for extract methods

* Rework custom metadata helper to be more consistent with other helpers

* Remove KVSecret from custom metadata test now that we don't append to it as part of helper method

* Return early for readability and make test value name less confusing
2022-05-25 11:17:13 -07:00
Brian Kassouf df8ae055be
Add an API for exporting activity log data (#15586)
* Add an API for exporting activity log data

* Add changelog entry

* Switch to error logs
2022-05-24 17:00:46 -07:00
davidadeleon 0026788d4b
api/monitor: Adding log format to monitor command and debug (#15536)
* Correct handling of "unspecified" log level

* Setting log-format default on monitor path

* Create changelog file

* Update website/content/api-docs/system/monitor.mdx

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-05-24 13:10:53 -04:00
Josh Black ebbb828b80
Update autopilot update interval (#15558) 2022-05-20 15:56:24 -07:00
Josh Black 416504d8c3
Add autopilot automated upgrades and redundancy zones (#15521) 2022-05-20 16:49:11 -04:00
Violet Hynes 6d4497bcbf
VAULT-4306 Ensure /raft/bootstrap/challenge call ignores erroneous namespaces set (#15519)
* VAULT-4306 Ensure /raft/bootstrap/challenge call ignores erroneous namespaces set

* VAULT-4306 Add changelog

* VAULT-4306 Update changelog/15519.txt

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>

Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2022-05-19 16:27:51 -04:00
Nick Cabatoff bc9f69af2e
Forward autopilot state reqs, avoid self-dialing (#15493)
Make sure that autopilot is disabled when we step down from active node state.  Forward autopilot state requests to the active node.  Avoid self-dialing due to stale advertisement.
2022-05-18 14:50:18 -04:00
Pratyoy Mukhopadhyay 62c09bc2be
oss changes (#15487)
* oss changes

* add changelog
2022-05-18 09:16:13 -07:00
Hamid Ghaf 66c6de50a7
Username format login mfa (#15363)
* change username_template to username_format for login MFA

* fixing a test

* Update website/content/docs/auth/login-mfa/faq.mdx

Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>

Co-authored-by: Calvin Leung Huang <1883212+calvn@users.noreply.github.com>
2022-05-17 16:31:50 -04:00
Hamid Ghaf 77be41c83d
possibly forward cached MFA auth response to leader (#15469)
* possibly forward cached MFA auth response to leader

* adding CL
2022-05-17 16:30:36 -04:00
Hridoy Roy 679ccc81a5
Query and Precompute Non-Contiguous Segments in the Activity Log (#15352)
* query and precompute non-contiguous segments in the activity log

* changelog

* newline formatting

* make fmt

* report listener and storage types as found keys

* report listener and storage types as found keys

* Update vault/activity_log_test.go

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>

* review comments

* merge conflict

* merge conflict

* merge conflict

* fix unchecked merge conflict

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-05-17 12:17:32 -07:00
Chris Hoffman 24e8b73c73
Updating Okta MFA to use official SDK (#15355)
* updating MFA to use official Okta SDK

* add changelog

* Update vault/login_mfa.go

Co-authored-by: swayne275 <swayne@hashicorp.com>

* cleanup query param building

* skip if not user factor

* updating struct tags to be more explicit

* fixing incorrect merge

* worrying that URL construction may change in the future, reimplementing GetFactorTransactionStatus

* adding some safety around url building

Co-authored-by: swayne275 <swayne@hashicorp.com>
2022-05-17 15:14:26 -04:00
Hamid Ghaf 364f8789cd
Globally scoped MFA method Get/List endpoints (#15248)
* Globally scoped MFA method Get/List endpoints

* Adding CL

* minor changes

* removing unwanted information from an error msg
2022-05-17 14:54:16 -04:00
akshya96 4e9e9b7eda
Vault-6037 making filesystem permissions check opt-in (#15452)
* adding env var changes

* adding changelog

* adding strcov.ParseBool
2022-05-17 11:34:31 -07:00
Brian Kassouf b5472aadf3
Add list of granting policies audit logs (#15457)
* Add list of granting policies audit logs

* Add changelog
2022-05-16 16:23:08 -07:00
Chris Hoffman b14dc0d95d Remove duplicate policies when creating/updating identity groups (#15055)
* Remove duplicate policies for identity groups

* adding changelog

* test cleanup
2022-05-16 17:20:48 -04:00
Hridoy Roy 90538739bd
append nil months to query get to cover all requested months (OSS) (#15420)
* fill out nil response months in activity log query handle response based on requested month data

* changelog

* reverse month ordering for nil end months

* typo caught in ent test
2022-05-16 13:01:28 -07:00
Chris Hoffman 63b31a13f9
Disabling client side rate limiting in Okta login MFA client (#15369)
* disabling client side rate limiting for MFA client

* add changelog
2022-05-12 15:55:33 -04:00
John-Michael Faircloth 45efa37c4a
unit test: fix oidc periodicfunc flaky test (#15320)
* unit test: fix oidc periodicfunc flaky test

* update cycle 1 for two test cases
2022-05-09 13:43:23 -05:00
Hamid Ghaf 2ee602cfdd
removing prem/pro references as part of removing some build targets (#15278) 2022-05-06 09:09:42 -04:00
Hamid Ghaf 0fc7a363cc
loading MFA configs upont restart (#15261)
* loading MFA configs upont restart

* Adding CL

* feedback

* Update vault/core.go

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-05-05 18:53:57 -04:00
Christopher Swenson 0affe226ad
Update deps for consul-template 0.29.0 (#15293)
This requires bumping https://github.com/mitchellh/go-testing-interface.
For this new version, we have to create a wrapper to convert
the stdlib `testing.TB` interface to the
`mitchellh/go-testing-interface` `T` interface, since it uses
`Parallel()` now, which is not supported by `testing.TB`. This had to be
added to a new package, `benchhelpers`, to avoid a circular dependency
in `testhelpers`.

We also have to *unbump* https://github.com/armon/go-metrics since
updating it breaks our usage of
https://github.com/google/go-metrics-stackdriver

I verified that the new `pkiCert` template function works with agent
injection using annotations like:

```yaml
vault.hashicorp.com/agent-inject-secret-sample.crt: "pki/issue/example-dot-com"
vault.hashicorp.com/agent-inject-template-sample.crt: |
  {{ pkiCert "pki/issue/example-dot-com" "common_name=foo.example.com" "ttl=1h" }}
```
2022-05-05 10:30:40 -07:00
Vinayak 988468a67b
Skip metric increment during existence check (#12763)
* Skip metric increment during existence check

Signed-off-by: Vinayak Kadam <kadamvinayak03@gmail.com>

* Adding changelog.txt

Signed-off-by: Vinayak Kadam <kadamvinayak03@gmail.com>

* Updated changelog text

Signed-off-by: Vinayak Kadam <kadamvinayak03@gmail.com>
2022-05-05 10:22:19 -07:00
Hridoy Roy 9a5b46436e
change ordering of activity log month data to sort by ascending order… (#15259)
* change ordering of activity log month data to sort by ascending order of timestamp

* changelog

* changelog
2022-05-03 13:39:29 -07:00
Scott Miller bef350c916
Allow callers to choose the entropy source for the random endpoints. (#15213)
* Allow callers to choose the entropy source for the random endpoints

* Put source in the URL for sys as well

* changelog

* docs

* Fix unit tests, and add coverage

* refactor to use a single common implementation

* Update documentation

* one more tweak

* more cleanup

* Readd lost test expected code

* fmt
2022-05-02 14:42:07 -05:00
Steven Clark 6d9888e09b
Allow looking up mount entries by their backend UUIDs (#15217) 2022-04-29 16:15:29 -04:00
VAL a06c8a139f
Add enterprise sudo paths to api.SudoPaths map (#15219)
* Add enterprise sudo paths to api.SudoPaths map

* add comment to denote ent-only sudo paths

* go fmt

Co-authored-by: Chris Capurso <1036769+ccapurso@users.noreply.github.com>
2022-04-29 10:09:25 -04:00
Nick Cabatoff c5928c1d15
Raft: use a larger initial heartbeat/election timeout (#15042) 2022-04-29 08:32:16 -04:00
VAL 0ef529b710
Global flag that outputs minimum policy HCL required for an operation (#14899)
* WIP: output policy

* Outputs example policy HCL for given request

* Simplify conditional

* Add PATCH capability

* Use OpenAPI spec and regex patterns to determine if path is sudo

* Add test for isSudoPath

* Add changelog

* Fix broken CLI tests

* Add output-policy to client cloning code

* Smaller fixes from PR comments

* Clone client instead of saving and restoring custom values

* Fix test

* Address comments

* Don't unset output-policy flag on KV requests otherwise the preflight request will fail and not populate LastOutputPolicyError

* Print errors saved in buffer from preflight KV requests

* Unescape characters in request URL

* Rename methods and properties to improve readability

* Put KV-specificness at front of KV-specific error

* Simplify logic by doing more direct returns of strings and errors

* Use precompiled regexes and move OpenAPI call to tests

* Remove commented out code

* Remove legacy MFA paths

* Remove unnecessary use of client

* Move sudo paths map to plugin helper

* Remove unused error return

* Add explanatory comment

* Remove need to pass in address

* Make {name} regex less greedy

* Use method and path instead of info from retryablerequest

* Add test for IsSudoPaths, use more idiomatic naming

* Use precompiled regexes and move OpenAPI call to tests (#15170)

* Use precompiled regexes and move OpenAPI call to tests

* Remove commented out code

* Remove legacy MFA paths

* Remove unnecessary use of client

* Move sudo paths map to plugin helper

* Remove unused error return

* Add explanatory comment

* Remove need to pass in address

* Make {name} regex less greedy

* Use method and path instead of info from retryablerequest

* Add test for IsSudoPaths, use more idiomatic naming

* Make stderr writing more obvious, fix nil pointer deref
2022-04-27 16:35:18 -07:00
Rémi Lapeyre 089b6ea970
Remove dead code in setupCredentials() (#15194)
This should have been removed as part of f09e39ea42 but somehow got
forgotten.
2022-04-27 10:47:04 -04:00
Josh Black a4593e8913
When tainting a route during setup, pre-calculate the namespace specific path (#15067) 2022-04-26 09:13:45 -07:00
Chris Capurso cc531c793d
fix raft tls key rotation panic when rotation time in past (#15156)
* fix raft tls key rotation panic when rotation time in past

* add changelog entry

* push out next raft TLS rotation time in case close to elapsing

* consolidate tls key rotation duration calculation

* reduce raft getNextRotationTime padding to 10 seconds

* move tls rotation ticker reset to where its duration is calculated
2022-04-25 21:48:34 -04:00
Nick Cabatoff 7e64e105a0
Clone identity objects to prevent races. (#15123) 2022-04-22 13:04:34 -04:00
Chris Capurso e69f89c279
Add build date (#14957)
* add BuildDate to version base

* populate BuildDate with ldflags

* include BuildDate in FullVersionNumber

* add BuildDate to seal-status and associated status cmd

* extend core/versions entries to include BuildDate

* include BuildDate in version-history API and CLI

* fix version history tests

* fix sys status tests

* fix TestStatusFormat

* remove extraneous LD_FLAGS from build.sh

* add BuildDate to build.bat

* fix TestSysUnseal_Reset

* attempt to add build-date to release builds

* add branch to github build workflow

* add get-build-date to build-* job needs

* fix release build command vars

* add missing quote in release build command

* Revert "add branch to github build workflow"

This reverts commit b835699ecb7c2c632757fa5fe64b3d5f60d2a886.

* add changelog entry
2022-04-19 14:28:08 -04:00
Vinny Mannello 6116903a37
[Vault-5248] MFA support for api login helpers (#14900)
* Add MFA support to login helpers
2022-04-15 11:13:15 -07:00
John-Michael Faircloth b9acac050a
fix regression on Core.pluginDirectory when backend is builtin (#15027)
* fix dev-plugin-dir when backend is builtin

* use builtinRegistry.Contains

* revert aa76337

* use correct plugin type for logical backend after revert

* fix factory func default setting after revert

* add ut coverage for builtin plugin with plugin directory set

* add coverage for secrets plugin type

* use totp in tests to avoid test import cycle in ssh package

* use nomad in tests to avoid test import cycle

* remove secrets mount tests due to unavoidable test import cycle
2022-04-14 15:54:23 -05:00
Hamid Ghaf a1d73ddfec
VAULT-5422: Add rate limit for TOTP passcode attempts (#14864)
* VAULT-5422: Add rate limit for TOTP passcode attempts

* fixing the docs

* CL

* feedback

* Additional info in doc

* rate limit is done per entity per methodID

* refactoring a test

* rate limit OSS work for policy MFA

* adding max_validation_attempts to TOTP config

* feedback

* checking for non-nil reference
2022-04-14 13:48:24 -04:00
Hamid Ghaf a12271af46
forwarding requests subjected to Login MFA to the active node (#15009)
* forwarding requests subjected to Login MFA to the active node

* CL, and making fmt happy
2022-04-13 10:11:53 -04:00
Anton Averchenkov 23a3f950e4
Revert the WithContext changes to vault tests (#14947) 2022-04-07 15:12:58 -04:00
Austin Gebauer f9ce8d7615
Adds Vault version prerelease and metadata to logical.PluginEnvironment (#14851) 2022-04-04 22:31:01 -07:00
akshya96 796003ddda
Vault 3992 ToB Config and Plugins Permissions (#14817)
* updating changes from ent PR

* adding changelog

* fixing err

* fixing semgrep error
2022-04-04 09:45:41 -07:00
hghaf099 aafb5d6427
VAULT-4240 time.After() in a select statement can lead to memory leak (#14814)
* VAULT-4240 time.After() in a select statement can lead to memory leak

* CL
2022-04-01 10:17:11 -04:00
Steven Clark 77feaad6fe
Address incorrect table metric value for local mounts (#14755)
* Address incorrect table metric value for local mounts

 - Reported within issue #14750 as a panic, it was identified that
   we were using the wrong value for local mounts within the table metrics.

* Add changelog
2022-03-30 13:06:49 -04:00
Chris Capurso 2354435262
treat logical.ErrRelativePath as 400 instead of 500 (#14328)
* treat logical.ErrRelativePath as 400 instead of 500

* add changelog entry

* return UserError for logical.ErrRelativePath
2022-03-30 09:08:02 -04:00
VAL f9372145dd
Warn on upper case in policy name (#14670)
* Warn on upper case in policy name

* Rename name variable to be less confusing

* Use more general solution for other string issues

* Clarify changelog

* Remove unnecessary check

* Don't throw CLI warning until after past errors

* Add before and after names with quotes to show spacing changes
2022-03-24 13:29:11 -07:00
Anton Averchenkov 1222375d1a
Add context-aware functions to vault/api (#14388) 2022-03-23 17:47:43 -04:00
Nick Cabatoff 34af6bab1e
Add a check for missing entity during local alias invalidation. (#14622) 2022-03-21 15:09:31 -04:00
John-Michael Faircloth 3c2659b09d
oidc provider: add test case for clients sharing keys (#14555) 2022-03-18 16:21:21 -05:00
Hridoy Roy 7e0abe3c7e
Add Semgrep Rules to OSS (#14513)
* add semgrep yml

* add semgrep ci job

* remove replication semgrep rule in oss

* fix makefile

* add semgrep to ci

* upwind triple if in ui.go semgrep refactoring
2022-03-18 11:14:03 -07:00
Josh Black 92b97b19ed
sys/mfa/validate should forward (#14566) 2022-03-18 10:14:17 -07:00
Jason O'Donnell 219df7087c
identity/token: fix duplicate keys in well-known (#14543)
* identity/token: fix duplicate kids in well-known

* Remove unused check

* changelog

* use map-based approach to dedup key IDs

* improve changelog description

* move jwks closer to usage; specify capacity

Co-authored-by: Austin Gebauer <agebauer@hashicorp.com>
2022-03-16 18:48:10 -07:00
Lars Lehtonen 50a9dedbcb
vault/external_tests/raft: fix dropped test error (#14519) 2022-03-16 09:32:57 -06:00
Nick Cabatoff 9e18350cf4
Handle the empty mount accessor case. (#14507) 2022-03-16 09:28:05 -04:00
Hridoy Roy 1558387af4
port of semgrep fixes oss (#14488) 2022-03-15 13:17:55 -07:00
Pratyoy Mukhopadhyay d222981cec
Fixes from mount move testing (#14492)
* Add validation, fix docs

* add changelog

* fmt fix

* Update vault/logical_system.go

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>

* Update vault/logical_system.go

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>

* Update vault/logical_system_test.go

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>

* Update vault/logical_system_test.go

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>

Co-authored-by: Josh Black <raskchanky@users.noreply.github.com>
2022-03-15 11:11:23 -07:00
hghaf099 361646ab26
add MFA validation support to vault login command (#14425)
* add MFA validation support to vault login command

* correctly report new totp code availability period
2022-03-14 15:54:41 -04:00
Vinny Mannello 2290ca5e83
[VAULT-5003] Use net/http client in Sys().RaftSnapshotRestore (#14269)
Use net/http client when body could be too big for retryablehttp client
2022-03-14 10:13:33 -07:00
Anton Averchenkov c425078008
Change OpenAPI code generator to extract request objects (#14217) 2022-03-11 19:00:26 -05:00
Austin Gebauer d016b67915
identity/oidc: prevent key rotation on performance secondary clusters (#14426) 2022-03-09 15:41:02 -08:00
hghaf099 b358bd6ffa
remove mount accessor from MFA config (#14406)
* remove mount accessor from MFA config

* Update login_mfa_duo_test.go

* DUO test with entity templating

* using identitytpl.PopulateString to perform templating

* minor refactoring

* fixing fmt failures in CI

* change username format to username template

* fixing username_template example
2022-03-09 09:14:30 -08:00
Lars Lehtonen b9a6ec67c9
vault: fix dropped test errors (#14402) 2022-03-08 12:32:27 -07:00
John-Michael Faircloth baafd9ff38
plugin multiplexing: add catalog test coverage (#14398)
* plugin client and plugin catalog tests

* add v5 plugin cases and more checks

* improve err msg

* refactor tests; fix test err msg
2022-03-08 10:33:24 -06:00
Josh Black 1f8d282c2b
fix version check (#14395) 2022-03-07 15:42:06 -08:00
Josh Black 5c43bf4864
Only create new batch tokens if we're on at least 1.10.0 (#14370) 2022-03-04 14:16:51 -08:00
Hridoy Roy d8155aa7c4
SSCT Optimizations (OSS) (#14323)
* update ci.hcl to remove 1.6.x and add in 1.10.x

* SSCT OSS PR review comments and optimizations

* check errors in populate token entry calls
2022-03-01 12:24:45 -08:00
Scott Miller ba49365ecb
Fix autoseal health check race by passing metrics sink in CoreConfig (#14196) 2022-03-01 09:00:39 -05:00
Pratyoy Mukhopadhyay 69c22b8078
Fix raft paralle retry bug (#14303) 2022-02-28 10:38:34 -08:00
Josh Black c9d5734924
Add login enforcement listing (#14284) 2022-02-25 13:48:28 -08:00
Steven Clark 6f8e3c618b
Address slow CI causing failures in TestRateLimitQuota_Allow_WithBlock (#14042)
* Address slow CI causing failures in TestRateLimitQuota_Allow_WithBlock

 - An attempt to fix CI runs that are extremely slow and the for loop
   runs across two BlockIntervals within the rate limit window of operation.
 - Increasing BlockInterval was looked at but the normal test times would
   be increased due to us also validating that we are releasing clients post
   BlockInterval.

* Address TestRateLimitQuota_Allow_WithBlock slowness issue (take 2)

 - Increase the overall Interval value within the blocking test so that we
   should always be able to request at least 17 requests within the interval
   value.
 - Tested by changing the time.Sleep within the for loop to 20 from 2
   and could see that within the response, coming back from the rlq.allow,
   that we were no longer being rate limited by going over the 1 second
   interval value per host.

* Limit the number of active go routines in TestRateLimitQuota_Allow_WithBlock
2022-02-25 16:05:44 -05:00
hghaf099 671cdbcadb
interactive CLI for mfa login (#14131)
* Login MFA

* ENT OSS segragation (#14088)

* Delete method id if not used in an MFA enforcement config (#14063)

* Delete an MFA methodID only if it is not used by an MFA enforcement config

* Fixing a bug: mfa/validate is an unauthenticated path, and goes through the handleLoginRequest path

* adding use_passcode field to DUO config (#14059)

* add changelog

* preventing replay attack on MFA passcodes (#14056)

* preventing replay attack on MFA passcodes

* using %w instead of %s for error

* Improve CLI command for login mfa (#14106)

CLI prints a warning message indicating the login request needs to get validated

* adding the validity period of a passcode to error messages (#14115)

* interactive CLI for mfa login

* minor fixes

* bail if no input was inserted

* change label name

* interactive CLI when single methodID is returned from login request

* minor fix

* adding changelog

* addressing feedback

* a user with a terminal should be able to choose between interactive and non-interactive.  A user without a terminal should not be able to use the interactive mode.

Co-authored-by: Josh Black <raskchanky@gmail.com>
2022-02-24 15:16:15 -05:00
Hridoy Roy 3438f5dbae
Fix ent diff check 2022/02/23 (#14237)
* achieve parity with ent in core.go

* add VAULT_DISABLE_LOCAL_AUTH_MOUNT_ENTITIES

* parity in build.yml with ent but without adding the +ent

* pass base version to ldflags

Co-authored-by: Kyle Penfound <kpenfound11@gmail.com>
2022-02-24 11:57:40 -08:00
Nick Cabatoff 0421b6895f
Remove use of compressionutil from activitylog segments. We can leave it in for the precomputed queries since they're JSON, not protobuf. (#14239) 2022-02-23 17:33:24 -05:00
Vishal Nayak 35d3d4226d
Minor fixes in response (#14230) 2022-02-23 13:17:53 -05:00
Nick Cabatoff 2551a3e8ce
Ensure that fewer goroutines survive after a test completes (#14197)
* Various changes to try to ensure that fewer goroutines survive after a test completes:
* add Core.ShutdownWait that doesn't return until shutdown is done
* create the usedCodes cache on seal and nil it out on pre-seal so that the finalizer kills the janitor goroutine
* stop seal health checks on seal rather than wait for them to discover the active context is done
* make sure all lease-loading goroutines are done before returning from restore
* make uniquePoliciesGc discover closed quitCh immediately instead of only when the ticker fires
* make sure all loading goroutines are done before returning from loadEntities, loadCachedEntitiesOfLocalAliases
2022-02-23 10:33:52 -05:00
mickael-hc 6b753fb2f5
Replace dgrijava/jwt-go with golang-jwt/jwt (#14167) 2022-02-22 15:12:15 -05:00
Austin Gebauer 462a924722
identity/oidc: Adds default provider, key, and allow_all assignment (#14119) 2022-02-22 08:33:19 -08:00
Victor Rodriguez f7f6f937e4
Invalidate the ManagedKeyRegistry cache when Vault config is updated. (#14179)
Add the hook needed to implement this functionality in vault-enterprise.
2022-02-21 09:55:44 -05:00
Pratyoy Mukhopadhyay c8cca2cab5
add missed test (#14168) 2022-02-18 14:01:43 -08:00
Vishal Nayak e29cc5989f
s/path/mount_path (#14164) 2022-02-18 13:44:43 -05:00
Chris Capurso 9e2f6f3b4f
wrap kv reqs with kvRequestWithRetry in external tests (#14163) 2022-02-18 13:24:46 -05:00
Vishal Nayak 6898e038ec
OSS Port: Activity log breakdowns (#14160)
* Activity log breakdowns

* s/path/namespace_path

* Remove ent specific tests
2022-02-18 13:01:28 -05:00
Chris Capurso c7ba8b1f25
Add HTTP-level tests for KV subkeys endpoint (#14079)
* go get plugin-secrets-kv@v0.11.0; go mod tidy

* add HTTP-level tests for KV subkeys endpoint

* check status in TestKV_Subkeys_CurrentVersion

* some test cleanup
2022-02-18 11:54:36 -05:00
Pratyoy Mukhopadhyay 475b55b460
Vault 4632 auth remount oss (#14141)
* Update plugin-portal.mdx (#13229)

Add a Vault plugin to allow authentication via SSH certificates and public keys

* oss changes

Co-authored-by: Wim <wim@42.be>
2022-02-18 08:04:21 -08:00