* a few tests to the operator diagnose stub command
* a few tests to the operator diagnose stub command
* a few tests to the operator diagnose stub command
* empty commit to fix circle ci permissions issue
* empty commit to fix circle ci permissions issue
Remove template_retry config section. Add new vault.retry section which only has num_retries field; if num_retries is 0 or absent, default it to 12 for backwards compat with pre-1.7 template retrying. Setting num_retries=-1 disables retries.
Configured retries are used for both templating and api proxy, though if template requests go through proxy (currently requires persistence enabled) we'll only configure retries for the latter to avoid duplicate retrying. Though there is some duplicate retrying already because whenever the template server does a retry when not going through the proxy, the Vault client it uses allows for 2 behind-the-scenes retries for some 400/500 http error codes.
* snapshot
* basic test
* update command and add documentation
* update help text
* typo
* add changelog for lease lookup command
* run go mod vendor
* remove tabs from help output
The existing code would retain the previous backoff value even after the
system had recovered. This PR fixes that issue and improves the
structure of the backoff code.
Adds the option of a write-through cache, backed by boltdb
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* k8s doc: update for 0.9.1 and 0.8.0 releases (#10825)
* k8s doc: update for 0.9.1 and 0.8.0 releases
* Update website/content/docs/platform/k8s/helm/configuration.mdx
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
* Autopilot initial commit
* Move autopilot related backend implementations to its own file
* Abstract promoter creation
* Add nil check for health
* Add server state oss no-ops
* Config ext stub for oss
* Make way for non-voters
* s/health/state
* s/ReadReplica/NonVoter
* Add synopsis and description
* Remove struct tags from AutopilotConfig
* Use var for config storage path
* Handle nin-config when reading
* Enable testing autopilot by using inmem cluster
* First passing test
* Only report the server as known if it is present in raft config
* Autopilot defaults to on for all existing and new clusters
* Add locking to some functions
* Persist initial config
* Clarify the command usage doc
* Add health metric for each node
* Fix audit logging issue
* Don't set DisablePerformanceStandby to true in test
* Use node id label for health metric
* Log updates to autopilot config
* Less aggressively consume config loading failures
* Return a mutable config
* Return early from known servers if raft config is unable to be pulled
* Update metrics name
* Reduce log level for potentially noisy log
* Add knob to disable autopilot
* Don't persist if default config is in use
* Autopilot: Dead server cleanup (#10857)
* Dead server cleanup
* Initialize channel in any case
* Fix a bunch of tests
* Fix panic
* Add follower locking in heartbeat tracker
* Add LastContactFailureThreshold to config
* Add log when marking node as dead
* Update follower state locking in heartbeat tracker
* Avoid follower states being nil
* Pull test to its own file
* Add execution status to state response
* Optionally enable autopilot in some tests
* Updates
* Added API function to fetch autopilot configuration
* Add test for default autopilot configuration
* Configuration tests
* Add State API test
* Update test
* Added TestClusterOptions.PhysicalFactoryConfig
* Update locking
* Adjust locking in heartbeat tracker
* s/last_contact_failure_threshold/left_server_last_contact_threshold
* Add disabling autopilot as a core config option
* Disable autopilot in some tests
* s/left_server_last_contact_threshold/dead_server_last_contact_threshold
* Set the lastheartbeat of followers to now when setting up active node
* Don't use config defaults from CLI command
* Remove config file support
* Remove HCL test as well
* Persist only supplied config; merge supplied config with default to operate
* Use pointer to structs for storing follower information
* Test update
* Retrieve non voter status from configbucket and set it up when a node comes up
* Manage desired suffrage
* Consider bucket being created already
* Move desired suffrage to its own entry
* s/DesiredSuffrageKey/LocalNodeConfigKey
* s/witnessSuffrage/recordSuffrage
* Fix test compilation
* Handle local node config post a snapshot install
* Commit to storage first; then record suffrage in fsm
* No need of local node config being nili case, post snapshot restore
* Reconcile autopilot config when a new leader takes over duty
* Grab fsm lock when recording suffrage
* s/Suffrage/DesiredSuffrage in FollowerState
* Instantiate autopilot only in leader
* Default to old ways in more scenarios
* Make API gracefully handle 404
* Address some feedback
* Make IsDead an atomic.Value
* Simplify follower hearbeat tracking
* Use uber.atomic
* Don't have multiple causes for having autopilot disabled
* Don't remove node from follower states if we fail to remove the dead server
* Autopilot server removals map (#11019)
* Don't remove node from follower states if we fail to remove the dead server
* Use map to track dead server removals
* Use lock and map
* Use delegate lock
* Adjust when to remove entry from map
* Only hold the lock while accessing map
* Fix race
* Don't set default min_quorum
* Fix test
* Ensure follower states is not nil before starting autopilot
* Fix race
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
* Stub "operator diagnose" command.
* Parse configuration files.
* Refactor storage setup to call from diagnose.
* Add the ability to run Diagnose as a prequel to server start.
* agent: do not grap idLock writelock until caching entry
* agent: inflight cache using sync.Map
* agent: implement an inflight caching mechanism
* agent/lease: add lock for inflight cache to prevent simultaneous Set calls
* agent/lease: lock on a per-ID basis so unique requests can be processed independently
* agent/lease: add some concurrency tests
* test: use lease_id for uniqueness
* agent: remove env flags, add comments around locks
* agent: clean up test comment
* agent: clean up test comment
* agent: remove commented debug code
* agent/lease: word-smithing
* Update command/agent/cache/lease_cache.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* agent/lease: return the context error if the Done ch got closed
* agent/lease: fix data race in concurrency tests
* agent/lease: mockDelayProxier: return ctx.Err() if context got canceled
* agent/lease: remove unused inflightCacheLock
* agent/lease: test: bump context timeout to 3s
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* feat(agent): add retry configuration for vault agent
* feat(agent): add test fixtures for retry
* fix(retry): move retry stanza to top level as template_retry
* fix(retry): add retry config to ServerConfig struct
* fix(retry): point config parser to parse template_retry instead of retry
* remove netlify config (#10711)
* Fix build (#10749)
* Move the declaration to a OSS build tag file to not have it collide w… (#10750)
* Move the declaration to a OSS build tag file to not have it collide with ent declarations
* Add comment
* Remove comment to trigger ci
* Unconditionally use the root namespace when calling sys/seal-status. (#10742)
* feat(agent): add retry configuration for vault agent
* feat(agent): add test fixtures for retry
* fix(retry): move retry stanza to top level as template_retry
* fix(retry): add retry config to ServerConfig struct
* fix(retry): point config parser to parse template_retry instead of retry
Co-authored-by: Hridoy Roy <roy@hashicorp.com>
Co-authored-by: Jeff Escalante <jescalan@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
* Adding snowflake as a bundled database secrets plugin
* Add snowflake-database-plugin to expected bundled plugins
* Add snowflake plugin name to the mockBuiltinRegistry
* Send a test message before committing a new audit device.
Also, lower timeout on connection attempts in socket device.
* added changelog
* go mod vendor (picked up some unrelated changes.)
* Skip audit device check in integration test.
Co-authored-by: swayne275 <swayne@hashicorp.com>
* core: Record the time a node became active
* Update vault/core.go
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Add omitempty field
* Update vendor
* Added CL entry and fixed test
* Fix test
* Fix command package tests
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* Fix KV list command with whitespaces
* Fix kv list whitespace
* Fix list whitespace
* Fix failing test
Co-authored-by: swayne275 <swayne@hashicorp.com>
* Update go version to 1.15.3
* Fix OU ordering for go1.15.x testing
* Fix CI version
* Update docker image
* Fix test
* packagespec upgrade -version 0.1.8
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
This also temporarily disables couchbase, elasticsearch, and
mongodbatlas because the `Serve` function needs to change signatures
and those plugins are vendored in from external repos, causing problems
when building.
* agent: return a non-zero exit code on error
* agent/template: always return on template server error, add case for error_on_missing_key
* agent: fix tests by updating Run params to use an errCh
* agent/template: add permission denied test case, clean up test var
* agent: use unbuffered errCh, emit fatal errors directly to the UI output
* agent: use oklog's run.Group to schedule subsystem runners (#9761)
* agent: use oklog's run.Group to schedule subsystem runners
* agent: clean up unused DoneCh, clean up agent's main Run func
* agent/template: use ts.stopped.CAS to atomically swap value
* fix tests
* fix tests
* agent/template: add timeout on TestRunServer
* agent: output error via logs and return a generic error on non-zero exit
* fix TestAgent_ExitAfterAuth
* agent/template: do not restart ct runner on new incoming token if exit_after_auth is set to true
* agent: drain ah.OutputCh after sink exits to avoid blocking on the channel
* use context.WithTimeout, expand comments around ordering of defer cancel()
Adds debug and warn logging around AWS credential chain generation,
specifically to help users debugging auto-unseal problems on AWS, by
logging which role is being used in the case of a webidentity token.
Adds a deferred call to flush the log output as well, to ensure logs
are output in the event of an initialization failure.
* normalize format output for vault status
* interim commit
* interim commit
* make formatting idiomatic
* clean up comments
* added formatting test
* updated comments in format test to match godocs
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MBP.hitronhub.home>
Co-authored-by: HridoyRoy <hridoyroy@Hridoys-MacBook-Pro.local>
* strip redundant field type declarations
* root credential rotation for aws creds plugin
* Change location of mocks awsutil and update methods that no longer exist
* Update website/pages/docs/auth/aws.mdx
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Update sdk version to get the awsutil mock file
* Re-vendor modules to pass CI
* Use write lock for the entirety of AWS root cred rotation
* Update docs for AWS root cred rotation for clarity
Co-authored-by: Becca Petrin <beccapetrin@gmail.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* Register a log sink that delays the printing of the big dev warning until logs have settled down
* Since this is always an intercept logger, just be explicit about the type
* changelog++
* Add date/time argument type.
* Add an argument to select which time formats are valid.
* Increase minimum date for epoch timestamps to avoid ambiguity.
* TOB-018 remediation
* Make key derivation an optional config flag, off by default, for backwards compatibility
* Fix unit tests
* Address some feedback
* Set config on unit test
* Fix another test failure
* One more conf fail
* Switch one of the test cases to not use a derive dkey
* wip
* comments
Hexadecimal integers will be converted to decimal, which is unfortunate but shouldn't have any negative effects other than perhaps confusion in the `vault debug` output.
* Carefully move changes from the plugin-cluster-reload branch into this clean branch off master.
* Don't test this at this level, adequately covered in the api level tests
* Change PR link
* go.mod
* Vendoring
* Vendor api/sys_plugins.go
* Revert "Some of the OSS changes were clobbered when merging with quotas out of, master (#9343)"
This reverts commit 8719a9b7c4d6ca7afb2e0a85e7c570cc17081f41.
* Revert "OSS side of Global Plugin Reload (#9340)"
This reverts commit f98afb998ae50346849050e882b6be50807983ad.
* raft: initial work on raft ha storage support
* add note on join
* add todo note
* raft: add support for bootstrapping and joining existing nodes
* raft: gate bootstrap join by reading leader api address from storage
* raft: properly check for raft-only for certain conditionals
* raft: add bootstrap to api and cli
* raft: fix bootstrap cli command
* raft: add test for setting up new cluster with raft HA
* raft: extend TestRaft_HA_NewCluster to include inmem and consul backends
* raft: add test for updating an existing cluster to use raft HA
* raft: remove debug log lines, clean up verifyRaftPeers
* raft: minor cleanup
* raft: minor cleanup
* Update physical/raft/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/ha.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/logical_system_raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* address feedback comments
* address feedback comments
* raft: refactor tls keyring logic
* address feedback comments
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* address feedback comments
* testing: fix import ordering
* raft: rename var, cleanup comment line
* docs: remove ha_storage restriction note on raft
* docs: more raft HA interaction updates with migration and recovery mode
* docs: update the raft join command
* raft: update comments
* raft: add missing isRaftHAOnly check for clearing out state set earlier
* raft: update a few ha_storage config checks
* Update command/operator_raft_bootstrap.go
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* raft: address feedback comments
* raft: fix panic when checking for config.HAStorage.Type
* Update vault/raft.go
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* Update website/pages/docs/commands/operator/raft.mdx
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
* raft: remove bootstrap cli command
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Update vault/raft.go
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* raft: address review feedback
* raft: revert vendored sdk
* raft: don't send applied index and node ID info if we're HA-only
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
Co-authored-by: Alexander Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* Add new Telemetry config options
Add cluster_name, maximum_gauge_cardinality, and usage_gauge_period
configuration options to the config stanza.
Update unit tests.
Document.
Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
* enable seal wrap in all seal migration tests
* move adjustForSealMigration to vault package
* fix adjustForSealMigration
* begin working on new seal migration test
* create shamir seal migration test
* refactor testhelpers
* add VerifyRaftConfiguration to testhelpers
* stub out TestTransit
* Revert "refactor testhelpers"
This reverts commit 39593defd0d4c6fd79aedfd37df6298391abb9db.
* get shamir test working again
* stub out transit join
* work on transit join
* Revert "move resuable storage test to avoid creating import cycle"
This reverts commit b3ff2317381a5af12a53117f87d1c6fbb093af6b.
* remove debug code
* initTransit now works with raft join
* runTransit works with inmem
* work on runTransit with raft
* runTransit works with raft
* get rid of dis-used test
* cleanup tests
* TestSealMigration_TransitToShamir_Pre14
* TestSealMigration_ShamirToTransit_Pre14
* split for pre-1.4 testing
* add simple tests for transit and shamir
* fix typo in test suite
* debug wrapper type
* test debug
* test-debug
* refactor core migration
* Revert "refactor core migration"
This reverts commit a776452d32a9dca7a51e3df4a76b9234d8c0c7ce.
* begin refactor of adjustForSealMigration
* fix bug in adjustForSealMigration
* clean up tests
* clean up core refactoring
* fix bug in shamir->transit migration
* remove unnecessary lock from setSealsForMigration()
* rename sealmigration test package
* use ephemeral ports below 30000
* simplify use of numTestCores
For situations where you want the Vault agent to handle one or more templates but do not require the acquired credentials elsewhere.
Modify the logic in SyncServer so that if there are no sinks, ignore any new credentials. Since SyncServer is responsible for shutting down the agent, make sure it still properly shuts down in this new situation.
Solves #7988
Fix JSON encoding adding newlines.
This manifested itself when encoding config values, which all map to
strings. An extra new line would get added by json.Encode, which caused
other things to break with confusing error messagges. Switching to
json.Marshal seems to solve the problem.
* storage/raft: Add committed and applied indexes to the status output
* Update api vendor
* changelog++
* Update http/sys_leader.go
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
* serivceregistration: refactor service registration logic to run later
* move state check to the internal func
* sr/kubernetes: update setInitialStateInternal godoc
* sr/kubernetes: remove return in setInitialState
* core/test: fix mockServiceRegistration
* address review feedback
* raft: check for nil on concrete type in SetupCluster
* raft: move check to its own func
* raft: func cleanup
* raft: disallow disable_clustering = true when raft storage is used
* docs: update disable_clustering to mention new behavior
* rename some seal migration tests
* add comments to seal migration tests
* add comments to seal migration tests
* rename sealhelper
* creat TestSealMigration_ShamirToTestSeal
* fix typo in test suite
* create TestSealMigration_TransitToTestSeal
* adding support for TLS 1.3 for TCP listeners
* removed test as CI uses go 1.12
* removed Cassandra support, added deprecation notice
* re-added TestTCPListener_tls13
* provide vault server flag to exit on core shutdown
* Update command/server.go
Co-Authored-By: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* rename UseAutoAuthForce to ForceAutoAuth, because I think it reads better
* Document 'ForceAuthAuthToken' option for Agent Cache
* Update website/pages/docs/agent/caching/index.mdx
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* Add additional tests around use_auto_auth=force and add documentation
* remove note, it's no longer correct
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* Guard against using Raft as a seperate HA Storage
* Document that Raft cannot be used as a seperate ha_storage backend at this time
* remove duplicate imports from updating with master
* Seal migration after unsealing
* Refactor migration fields migrationInformation in core
* Perform seal migration as part of postUnseal
* Remove the sleep logic
* Use proper seal in the unseal function
* Fix migration from Auto to Shamir
* Fix the recovery config missing issue
* Address the non-ha migration case
* Fix the multi cluster case
* Avoid re-running seal migration
* Run the post migration code in new leaders
* Fix the issue of wrong recovery being set
* Address review feedback
* Add more complete testing coverage for seal migrations. (#8247)
* Add more complete testing coverage for seal migrations. Also remove VAULT_ACC gate from some tests that just depend on docker, cleanup dangling recovery config in storage after migration, and fix a call in adjustCoreForSealMigration that seems broken.
* Fix the issue of wrong recovery key being set
* Adapt tests to work with multiple cores.
* Add missing line to disable raft join.
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
* Fix all known issues
* Remove warning
* Review feedback.
* Revert my previous change that broke raft tests. We'll need to come back and at least comment
this once we better understand why it's needed.
* Don't allow migration between same types for now
* Disable auto to auto tests for now since it uses migration between same types which is not allowed
* Update vault/core.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Add migration logs
* Address review comments
* Add the recovery config check back
* Skip a few steps if migration is already done
* Return from waitForLeadership if migration fails
Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Added flag to disable X-Vault-Token header proxy if client passes the token
* Reveresed the flag value to better match the name intent
* Introduced UseAutoAuthTokenRaw for Cache to support triplicate value of true/false/force
Co-authored-by: Clint <catsby@users.noreply.github.com>
* use observer pattern for service discovery
* update perf standby method
* fix test
* revert usersTags to being called serviceTags
* use previous consul code
* vault isnt a performance standby before starting
* log err
* changes from feedback
* add Run method to interface
* changes from feedback
* fix core test
* update example
* Use Shamir as KeK when migrating from auto-seal to shamir
* Use the correct number of shares/threshold for the migrated seal.
* Fix log message
* Add WaitForActiveNode to test
* Make test fail
* Minor updates
* Test with more shares and a threshold
* Add seal/unseal step to the test
* Update the logic that prepares seal migration (#8187)
* Update the logic that preps seal migration
* Add test and update recovery logic
Co-authored-by: ncabatoff <nick.cabatoff@gmail.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Raft retry join
* update
* Make retry join work with shamir seal
* Return upon context completion
* Update vault/raft.go
Co-Authored-By: Brian Kassouf <briankassouf@users.noreply.github.com>
* Address some review comments
* send leader information slice as a parameter
* Make retry join work properly with Shamir case. This commit has a blocking issue
* Fix join goroutine exiting before the job is done
* Polishing changes
* Don't return after a successful join during unseal
* Added config parsing test
* Add test and fix bugs
* minor changes
* Address review comments
* Fix build error
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* move ServiceDiscovery into methods
* add ServiceDiscoveryFactory
* add serviceDiscovery field to vault.Core
* refactor ConsulServiceDiscovery into separate struct
* cleanup
* revert accidental change to go.mod
* cleanup
* get rid of un-needed struct tags in vault.CoreConfig
* add service_discovery parser
* add ServiceDiscovery to config
* cleanup
* cleanup
* add test for ConfigServiceDiscovery to Core
* unit testing for config service_discovery stanza
* cleanup
* get rid of un-needed redirect_addr stuff in service_discovery stanza
* improve test suite
* cleanup
* clean up test a bit
* create docs for service_discovery
* check if service_discovery is configured, but storage does not support HA
* tinker with test
* tinker with test
* tweak docs
* move ServiceDiscovery into its own package
* tweak a variable name
* fix comment
* rename service_discovery to service_registration
* tweak service_registration config
* Revert "tweak service_registration config"
This reverts commit 5509920a8ab4c5a216468f262fc07c98121dce35.
* simplify naming
* refactor into ./serviceregistration/consul
* add regression test for #7883
* Add logic to count render events and match them to expected
* remove the WAIT label and make some changes to remove the break statements
* change the 'randomness' of the templateContents test helper method
* Vault Agent Template: parse templates (#7540)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* Update command/agent/config/config.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* return the decode error instead of swallowing it
* Update command/agent/config/config_test.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* go mod tidy
* change error checking style
* Add agent template doc
* TemplateServer: render secrets with Consul Template (#7621)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* add template package
* WIP: add runner
* fix panic, actually copy templates, etc
* rework how the config.Vault is created and enable reading from the environment
* this was supposed to be a part of the prior commit
* move/add methods to testhelpers for converting some values to pointers
* use new methods in testhelpers
* add an unblock channel to block agent until a template has been rendered
* add note
* unblock if there are no templates
* cleanups
* go mod tidy
* remove dead code
* simple test to starT
* add simple, empty templates test
* Update package doc, error logs, and add missing close() on channel
* update code comment to be clear what I'm referring to
* have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only
* Update command/agent.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* update with test
* Add README and doc.go to the command/agent directory (#7503)
* Add README and doc.go to the command/agent directory
* Add link to website
* address feedback for agent.go
* updated with feedback from Calvin
* Rework template.Server to export the unblock channel, and remove it from the NewServer function
* apply feedback from Nick
* fix/restructure rendering test
* Add pointerutil package for converting types to their pointers
* Remove pointer helper methods; use sdk/helper/pointerutil instead
* update newRunnerConfig to use pointerutil and empty strings
* only wait for unblock if template server is initialized
* update test structure
* some test cleanup
* follow up tests
* remove debugging, fix issue in replacing runner config
* need to handle first render/token
* Simplify the blocking logic to support exit after auth
* fix channel name
* expand TestAgent_Template to include multiple scenarios
* cleanup
* test cleanups after feedback
* Sync up Agent and API's renewers.
This introduces a new type, LifetimeWatcher, which can handle both
renewable and non-renewable secrets, modeled after the version in Agent.
It allows the user to select behavior, with the new style being the
default when calling Start(), and old style if using the legacy Renew()
call.
No tests have been modified (except for reflect issues) and no other
code has been modified to make sure the changes are backwards
compatible.
Once this is accepted I'll pull the Agent version out.
* Move compat flags to NewRenewer
* Port agent to shared lifetime watcher lib
* Vault Agent Template: parse templates (#7540)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* Update command/agent/config/config.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* return the decode error instead of swallowing it
* Update command/agent/config/config_test.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* go mod tidy
* change error checking style
* Add agent template doc
* TemplateServer: render secrets with Consul Template (#7621)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* add template package
* WIP: add runner
* fix panic, actually copy templates, etc
* rework how the config.Vault is created and enable reading from the environment
* this was supposed to be a part of the prior commit
* move/add methods to testhelpers for converting some values to pointers
* use new methods in testhelpers
* add an unblock channel to block agent until a template has been rendered
* add note
* unblock if there are no templates
* cleanups
* go mod tidy
* remove dead code
* simple test to starT
* add simple, empty templates test
* Update package doc, error logs, and add missing close() on channel
* update code comment to be clear what I'm referring to
* have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only
* Update command/agent.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* update with test
* Add README and doc.go to the command/agent directory (#7503)
* Add README and doc.go to the command/agent directory
* Add link to website
* address feedback for agent.go
* updated with feedback from Calvin
* Rework template.Server to export the unblock channel, and remove it from the NewServer function
* apply feedback from Nick
* fix/restructure rendering test
* Add pointerutil package for converting types to their pointers
* Remove pointer helper methods; use sdk/helper/pointerutil instead
* update newRunnerConfig to use pointerutil and empty strings
* only wait for unblock if template server is initialized
* drain the token channel in this test
* conditionally send on channel
Shamir seals now come in two varieties: legacy and new-style. Legacy
Shamir is automatically converted to new-style when a rekey operation
is performed. All new Vault initializations using Shamir are new-style.
New-style Shamir writes an encrypted master key to storage, just like
AutoUnseal. The stored master key is encrypted using the shared key that
is split via Shamir's algorithm. Thus when unsealing, we take the key
fragments given, combine them into a Key-Encryption-Key, and use that
to decrypt the master key on disk. Then the master key is used to read
the keyring that decrypts the barrier.
* cli: initial work on debug; server-status target
* debug: add metrics capture target (#7376)
* check against DR secondary
* debug: add compression
* refactor check into preflight func
* debug: set short test time on tests, fix exit code bug
* debug: use temp dir for output on tests
* debug: use mholt/archiver for compression
* first pass on adding pprof
* use logger for output
* refactor polling target capture logic
* debug: poll and collect replication status
* debug: poll and collect host-info; rename output files and collection refactor
* fix comments
* add archive test; fix bugs found
* rename flag name to singular target
* add target output test; scaffold other tests cases
* debug/test: add pprof and index file tests
* debug/test: add min timing check tests
* debug: fix index gen race and collection goroutine race
* debug: extend archive tests, handle race between program exit and polling goroutines
* update docstring
* debug: correctly add to pollingWg
* debug: add config target support
* debug: don't wait on interrupt shutdown; add file exists unit tests
* move pprof bits into its goroutine
* debug: skip empty metrics and some pprof file creation if permission denied, add matching unit test
* address comments and feedback
* Vault debug using run.Group (#7658)
* debug: switch to use oklog/run.Group
* debug: use context to cancel requests and interrupt rungroups.
* debug: trigger the first interval properly
* debug: metrics collection should use metrics interval
* debug: add missing continue on metrics error
* debug: remove the use of buffered chan to trigger first interval
* debug: don't shadow BaseCommand's client, properly block on interval capture failures
* debug: actually use c.cachedClient everywhere
* go mod vendor
* debug: run all pprof in goroutines; bump pprof timings in tests to reduce flakiness
* debug: update help text
* Initial work
* rework
* s/dr/recovery
* Add sys/raw support to recovery mode (#7577)
* Factor the raw paths out so they can be run with a SystemBackend.
# Conflicts:
# vault/logical_system.go
* Add handleLogicalRecovery which is like handleLogical but is only
sufficient for use with the sys-raw endpoint in recovery mode. No
authentication is done yet.
* Integrate with recovery-mode. We now handle unauthenticated sys/raw
requests, albeit on path v1/raw instead v1/sys/raw.
* Use sys/raw instead raw during recovery.
* Don't bother persisting the recovery token. Authenticate sys/raw
requests with it.
* RecoveryMode: Support generate-root for autounseals (#7591)
* Recovery: Abstract config creation and log settings
* Recovery mode integration test. (#7600)
* Recovery: Touch up (#7607)
* Recovery: Touch up
* revert the raw backend creation changes
* Added recovery operation token prefix
* Move RawBackend to its own file
* Update API path and hit it using CLI flag on generate-root
* Fix a panic triggered when handling a request that yields a nil response. (#7618)
* Improve integ test to actually make changes while in recovery mode and
verify they're still there after coming back in regular mode.
* Refuse to allow a second recovery token to be generated.
* Resize raft cluster to size 1 and start as leader (#7626)
* RecoveryMode: Setup raft cluster post unseal (#7635)
* Setup raft cluster post unseal in recovery mode
* Remove marking as unsealed as its not needed
* Address review comments
* Accept only one seal config in recovery mode as there is no scope for migration
* implement SSRF protection header
* add test for SSRF protection header
* cleanup
* refactor
* implement SSRF header on a per-listener basis
* cleanup
* cleanup
* creat unit test for agent SSRF
* improve unit test for agent SSRF
* add VaultRequest SSRF header to CLI
* fix unit test
* cleanup
* improve test suite
* simplify check for Vault-Request header
* add constant for Vault-Request header
* improve test suite
* change 'config' to 'agentConfig'
* Revert "change 'config' to 'agentConfig'"
This reverts commit 14ee72d21fff8027966ee3c89dd3ac41d849206f.
* do not remove header from request
* change header name to X-Vault-Request
* simplify http.Handler logic
* cleanup
* simplify http.Handler logic
* use stdlib errors package
* command/server: fix TestLoadConfigFile_json2 test, fix hcl tags
Fixes test to call the equality check, and add missing values to the expected object. Fixes hcl tags in the Telemetry structs.
* fix PrometheusRetentionTime tag
When starting a vault dev server the token helper is invoked to store
the dev root token.
This option gives the user the ability to not store the token.
Storing the token can be undesirable in certain circumstances
(e.g. running local tests) as the user's existing vault token is
clobbered without warning.
Fixes#1861
* Read config before creating logger when booting vault server
* Allow for specifying log output in JSON format in a config file, via a 'log_level' flag
* Create parser for log format flag
* Allow for specifying log format in a config file, via a 'log_format' flag. Also, get rid of 'log_json' flag.
* Add 'log-format' command line flag
* Update documentation to include description of log_format setting
* Tweak comment for VAULT_LOG_FORMAT environment variable
* add test for ParseEnvLogFormat()
* clarify how log format is set
* fix typos in documentation
* Implemented token backend support for identity
* Fixed tests
* Refactored a few checks for the token entity overwrite. Fixed tests.
* Moved entity alias check up so that the entity and entity alias is only created when it has been specified in allowed_entity_aliases list
* go mod vendor
* Added glob pattern
* Optimized allowed entity alias check
* Added test for asterisk only
* Changed to glob pattern anywhere
* Changed response code in case of failure. Changed globbing pattern check. Added docs.
* Added missing token role get parameter. Added more samples
* Fixed failing tests
* Corrected some cosmetical review points
* Changed response code for invalid provided entity alias
* Fixed minor things
* Fixed failing test
* Work on raft backend
* Add logstore locally
* Add encryptor and unsealable interfaces
* Add clustering support to raft
* Remove client and handler
* Bootstrap raft on init
* Cleanup raft logic a bit
* More raft work
* Work on TLS config
* More work on bootstrapping
* Fix build
* More work on bootstrapping
* More bootstrapping work
* fix build
* Remove consul dep
* Fix build
* merged oss/master into raft-storage
* Work on bootstrapping
* Get bootstrapping to work
* Clean up FMS and node-id
* Update local node ID logic
* Cleanup node-id change
* Work on snapshotting
* Raft: Add remove peer API (#906)
* Add remove peer API
* Add some comments
* Fix existing snapshotting (#909)
* Raft get peers API (#912)
* Read raft configuration
* address review feedback
* Use the Leadership Transfer API to step-down the active node (#918)
* Raft join and unseal using Shamir keys (#917)
* Raft join using shamir
* Store AEAD instead of master key
* Split the raft join process to answer the challenge after a successful unseal
* get the follower to standby state
* Make unseal work
* minor changes
* Some input checks
* reuse the shamir seal access instead of new default seal access
* refactor joinRaftSendAnswer function
* Synchronously send answer in auto-unseal case
* Address review feedback
* Raft snapshots (#910)
* Fix existing snapshotting
* implement the noop snapshotting
* Add comments and switch log libraries
* add some snapshot tests
* add snapshot test file
* add TODO
* More work on raft snapshotting
* progress on the ConfigStore strategy
* Don't use two buckets
* Update the snapshot store logic to hide the file logic
* Add more backend tests
* Cleanup code a bit
* [WIP] Raft recovery (#938)
* Add recovery functionality
* remove fmt.Printfs
* Fix a few fsm bugs
* Add max size value for raft backend (#942)
* Add max size value for raft backend
* Include physical.ErrValueTooLarge in the message
* Raft snapshot Take/Restore API (#926)
* Inital work on raft snapshot APIs
* Always redirect snapshot install/download requests
* More work on the snapshot APIs
* Cleanup code a bit
* On restore handle special cases
* Use the seal to encrypt the sha sum file
* Add sealer mechanism and fix some bugs
* Call restore while state lock is held
* Send restore cb trigger through raft log
* Make error messages nicer
* Add test helpers
* Add snapshot test
* Add shamir unseal test
* Add more raft snapshot API tests
* Fix locking
* Change working to initalize
* Add underlying raw object to test cluster core
* Move leaderUUID to core
* Add raft TLS rotation logic (#950)
* Add TLS rotation logic
* Cleanup logic a bit
* Add/Remove from follower state on add/remove peer
* add comments
* Update more comments
* Update request_forwarding_service.proto
* Make sure we populate all nodes in the followerstate obj
* Update times
* Apply review feedback
* Add more raft config setting (#947)
* Add performance config setting
* Add more config options and fix tests
* Test Raft Recovery (#944)
* Test raft recovery
* Leave out a node during recovery
* remove unused struct
* Update physical/raft/snapshot_test.go
* Update physical/raft/snapshot_test.go
* fix vendoring
* Switch to new raft interface
* Remove unused files
* Switch a gogo -> proto instance
* Remove unneeded vault dep in go.sum
* Update helper/testhelpers/testhelpers.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* Update vault/cluster/cluster.go
* track active key within the keyring itself (#6915)
* track active key within the keyring itself
* lookup and store using the active key ID
* update docstring
* minor refactor
* Small text fixes (#6912)
* Update physical/raft/raft.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* review feedback
* Move raft logical system into separate file
* Update help text a bit
* Enforce cluster addr is set and use it for raft bootstrapping
* Fix tests
* fix http test panic
* Pull in latest raft-snapshot library
* Add comment
* Audit listing with format json returns json, not a string
Fixes#6775
* list, kv list and namespace list with format json returns json, not a string
* Changed audit list return code to 2 which aligns with other list commands return codes
* http timeout fields are configurable
* move return statement for server config tests outside of range loop
* adds documentation for configurable listener http_* values
* fixed some formatting for the docs markdown
* adding auto auth for cert auth methods
* Adding Docs for Cert Auto-auth method
* fixes errors in docs
* wrong documentation title
* repariting a few typos in the docs and being very clear about where the certificates should be configured
* clarifying the name parameter documentation
* Fixes gofmt issues in command/agent.go
* Fix typo in docs
* Add nil check to Config for cert auto-auth and add test with and without a specified name
* Simplify Run(): the function that was being sent over a channel doesn't
need to close over anything except latestToken, and we don't need to
create a new one each iteration. Instead just pass the relevant items,
namely the token and sink to work on.
* Disallow the following config combinations:
1. auto_auth.method.wrap_ttl > 0 and multiple file sinks
2. auto_auth.method.wrap_ttl > 0 and single file sink with wrap_ttl > 0
3. auto_auth.method.wrap_ttl > 0 and cache.use_auto_auth_token = true
* Expose errors that occur when APIProxy is forwarding request to Vault.
* Fix merge issues.
If you were migrating to Shamir but didn't specify a Shamir block
migration would fail. Being explicit is nice but it's also not really
obvious since you don't need the block normally.
* Since we want to use the Agent listener for #6384, move listener config
from top-level 'cache' block to new top-level 'listeners' block.
* Make agent config allow cache and listener blocks without auto-auth
configured.
Append call in form of `append(s)` has no effect,
it just returns `s`. Sometimes such invocation is a sign
of a programming error, so it's better to remove these.
Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
* Listener refactoring and file system permissions
* added listenerutil and move some common code there
* Added test for verifying socket file permissions
* Change default port of agent to 8200
* address review feedback
* Address review feedback
* Read socket options from listener config
* Added warning when init command uses auto unseal and -key-shares and -key-threshold was set
* Reworked
* Update command/operator_init.go
Co-Authored-By: michelvocks <michelvocks@gmail.com>
* agent/caching: proxy redirect and non-json responses; update tests
* agent/caching: do not wrap error responses as internal errors, simply proxy them back as-is
* minor refactoring of APIProxy.Send logic
* add test case to ensure error response is not wrapped
* Add ability to migrate autoseal to autoseal
This adds the ability to migrate from shamir to autoseal, autoseal to
shamir, or autoseal to autoseal, by allowing multiple seal stanzas. A
disabled stanza will be used as the config being migrated from; this can
also be used to provide an unwrap seal on ent over multiple unseals.
A new test is added to ensure that autoseal to autoseal works as
expected.
* Fix test
* Provide default shamir info if not given in config
* Linting feedback
* Remove context var that isn't used
* Don't run auto unseal watcher when in migration, and move SetCores to SetSealsForMigration func
* Slight logic cleanup
* Fix test build and fix bug
* Updates
* remove GetRecoveryKey function
* Implemented a warning when tls_cipher_suites includes only cipher suites which are not supprted by the HTTP/2 spec
* Added test for cipher suites
* Added hard fail on startup when all defined cipher suites are blacklisted. Added warning when some ciphers are blacklisted.
* Replaced hard failure with warning. Removed bad cipher util function and replaced it by external library.
* Added missing dependency. Fixed renaming of package name.
* Adding Transit Autoseal
* adding tests
* adding more tests
* updating seal info
* send a value to test and set current key id
* updating message
* cleanup
* Adding tls config, addressing some feedback
* adding tls testing
* renaming config fields for tls
* agent/caching: enable caching of leases generated by autoauth token
* add test for auth/token/create path
* update error message log
* Some minor updates
* add sleep timer for renewal logic to process
* Properly compute auto-auth token
* Use inmem sink to track auto-auth token
* update debug statement
* Only add inmem sink if using auto-auth token is allowed
* vault-agent-cache: squashed 250+ commits
* Add proper token revocation validations to the tests
* Add more test cases
* Avoid leaking by not closing request/response bodies; add comments
* Fix revoke orphan use case; update tests
* Add CLI test for making request over unix socket
* agent/cache: remove namespace-related tests
* Strip-off the auto-auth token from the lookup response
* Output listener details along with configuration
* Add scheme to API address output
* leasecache: use IndexNameLease for prefix lease revocations
* Make CLI accept the fully qualified unix address
* export VAULT_AGENT_ADDR=unix://path/to/socket
* unix:/ to unix://
* initial commit for prometheus and sys/metrics support
* Throw an error if prometheusRetentionTime is 0,add prometheus in devmode
* return when format=prometheus is used and prom is disable
* parse prometheus_retention_time from string instead of int
* Initialize config.Telemetry if nil
* address PR issues
* add sys/metrics framework.Path in a factory
* Apply requiredMountTable entries's MountConfig to existing core table
* address pr comments
* enable prometheus sink by default
* Move Metric-related code in a separate metricsutil helper
* Add helper for checking if an error is a fatal error
The double-double negative was really confusing, and this pattern is used a few places in Vault. This negates the double negative, making the devx a bit easier to follow.
* Check return value of UnsealWithStoredKeys in sys/init
* Return proper error types when attempting unseal with stored key
Prior to this commit, "nil" could have meant unsupported auto-unseal, a transient error, or success. This updates the function to return the correct error type, signaling to the caller whether they should retry or fail.
* Continuously attempt to unseal if sealed keys are supported
This fixes a bug that occurs on bootstrapping an initial cluster. Given a collection of Vault nodes and an initialized storage backend, they will all go into standby waiting for initialization. After one node is initialized, the other nodes had no mechanism by which they "re-check" to see if unseal keys are present. This adds a goroutine to the server command which continually waits for unseal keys to exist. It exits in the following conditions:
- the node is unsealed
- the node does not support stored keys
- a fatal error occurs (as defined by Vault)
- the server is shutting down
In all other situations, the routine wakes up at the specified interval and attempts to unseal with the stored keys.
This changes (*Config).Merge to merge all fields of a Config.
Previously, when merging Configs, some configuration fields were
ignored and completely lost, including APIAddr, ClusterAddr, and
a couple boolean fields. This only occurs when using multiple config
files and does not affect single config files (even when loading from
a directory -- Merge is only called after a second file is loaded).
- Fix APIAddr not being merged.
- Fix ClusterAddr not being merged.
- Fix DisablePrintableCheck not being merged.
- Fix DisableClustering not being merged. The DisableClusteringRaw
value is also preserved so that it can be used in overrides for
storage fields.
- Use merged top-level config as storage field overrides.
- Update config dir test fixtures to set some fields missed by
(*Config).Merge previously.
The result will still pass gofmtcheck and won't trigger additional
changes if someone isn't using goimports, but it will avoid the
piecemeal imports changes we've been seeing.
* re-add plugin name for backwards compat
* add plugin name to table for backwards compat
* conditionally set the plugin name
* Update command/auth_list.go
Co-Authored-By: tyrannosaurus-becks <beccapetrin@posteo.net>
* Update command/secrets_list.go
Co-Authored-By: tyrannosaurus-becks <beccapetrin@posteo.net>
* update auth and secrets list commands
* add comments regarding deprecation
* Add support for custom JWT path in Agent: kubernetes auth
- add support for "token_path" configuration
- add a reader for mocking in tests
* add documentation for token_path
* Refactor mount tune to support upsert options values and unset options.
* Do not allow unsetting options map
* add secret tune version regression test
* Only accept valid options version
* s/meVersion/optVersion/
* Continue on plugin registration error in dev mode
* Continue only on unknown type error
* Continue only on unknown type error
* Print plugin registration error on exit
Co-Authored-By: calvn <cleung2010@gmail.com>
* Support registering plugin with name only
* Make RegisterPlugin backwards compatible
* Add CLI backwards compat command to plugin info and deregister
* Add server-side deprecation warnings if old read/dereg API endpoints are called
* Address feedback
Running 'vault auth' with no parameters was panicking:
panic: assignment to entry in nil map
github.com/hashicorp/vault/command/login.go:255 +0xdee
Now it will show help.
The addition of CheckMigration to the server startup process means
that physical backends in this test need to be able to respond to Get() without error.
Hridoy Roy