Uses a bufconn listener between consul-template and vault-agent when
caching is enabled and either templates or a listener is defined. This
means no listeners need to be defined in vault-agent for just
templating. Always routes consul-template through the vault-agent
cache (instead of only when persistent cache is enabled).
Uses a local transportDialer interface in config.Cache{}.
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Ben Ash <32777270+benashz@users.noreply.github.com>
* Customizing HTTP headers in the config file
* Add changelog, fix bad imports
* fixing some bugs
* fixing interaction of custom headers and /ui
* Defining a member in core to set custom response headers
* missing additional file
* Some refactoring
* Adding automated tests for the feature
* Changing some error messages based on some recommendations
* Incorporating custom response headers struct into the request context
* removing some unused references
* fixing a test
* changing some error messages, removing a default header value from /ui
* fixing a test
* wrapping ResponseWriter to set the custom headers
* adding a new test
* some cleanup
* removing some extra lines
* Addressing comments
* fixing some agent tests
* skipping custom headers from agent listener config,
removing two of the default headers as they cause issues with Vault in UI mode
Adding X-Content-Type-Options to the ui default headers
Let Content-Type be set as before
* Removing default custom headers, and renaming some function varibles
* some refacotring
* Refactoring and addressing comments
* removing a function and fixing comments
* agent: restart template runner on retry for unlimited retries
* template: log error message early
* template: delegate retries back to template if param is set to true
* agent: add and use the new template config stanza
* agent: fix panic, fix existing tests
* changelog: add changelog entry
* agent: add tests for exit_on_retry_failure
* agent: properly check on agent exit cases, add separate tests for missing key vs missing secrets
* agent: add note on difference between missing key vs missing secret
* docs: add docs for template_config
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
* docs: fix exit_on_retry_failure, fix Functionality section
* docs: update interaction title
* template: add internal note on behavior for persist case
* docs: update agent, template, and template-config docs
* docs: update agent docs on retry stanza
* Apply suggestions from code review
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
* Update changelog/11775.txt
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* agent/test: rename expectExit to expectExitFromError
* agent/test: add check on early exits on the happy path
* Update website/content/docs/agent/template-config.mdx
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: Jim Kalafut <jkalafut@hashicorp.com>
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
* Actually call config.Validate in diagnose
* Wire configuration checks into diagnose and fix resulting bugs.
* go mod vendor
* Merge to vendorless version
* Remove sentinel section to allow diagnose_ok to pass
* Fix unit tests
* Expose unknown fields and duplicate sections as diagnose warnings
* section counts not needed, already handled
* Address PR feedback
* Prune more of the new fields before tests call deep.Equals
* Update go.mod
Remove template_retry config section. Add new vault.retry section which only has num_retries field; if num_retries is 0 or absent, default it to 12 for backwards compat with pre-1.7 template retrying. Setting num_retries=-1 disables retries.
Configured retries are used for both templating and api proxy, though if template requests go through proxy (currently requires persistence enabled) we'll only configure retries for the latter to avoid duplicate retrying. Though there is some duplicate retrying already because whenever the template server does a retry when not going through the proxy, the Vault client it uses allows for 2 behind-the-scenes retries for some 400/500 http error codes.
Adds the option of a write-through cache, backed by boltdb
Co-authored-by: Theron Voran <tvoran@users.noreply.github.com>
Co-authored-by: Jason O'Donnell <2160810+jasonodonnell@users.noreply.github.com>
Co-authored-by: Calvin Leung Huang <cleung2010@gmail.com>
* feat(agent): add retry configuration for vault agent
* feat(agent): add test fixtures for retry
* fix(retry): move retry stanza to top level as template_retry
* fix(retry): add retry config to ServerConfig struct
* fix(retry): point config parser to parse template_retry instead of retry
* remove netlify config (#10711)
* Fix build (#10749)
* Move the declaration to a OSS build tag file to not have it collide w… (#10750)
* Move the declaration to a OSS build tag file to not have it collide with ent declarations
* Add comment
* Remove comment to trigger ci
* Unconditionally use the root namespace when calling sys/seal-status. (#10742)
* feat(agent): add retry configuration for vault agent
* feat(agent): add test fixtures for retry
* fix(retry): move retry stanza to top level as template_retry
* fix(retry): add retry config to ServerConfig struct
* fix(retry): point config parser to parse template_retry instead of retry
Co-authored-by: Hridoy Roy <roy@hashicorp.com>
Co-authored-by: Jeff Escalante <jescalan@users.noreply.github.com>
Co-authored-by: Vishal Nayak <vishalnayak@users.noreply.github.com>
Co-authored-by: Mark Gritter <mgritter@hashicorp.com>
* TOB-018 remediation
* Make key derivation an optional config flag, off by default, for backwards compatibility
* Fix unit tests
* Address some feedback
* Set config on unit test
* Fix another test failure
* One more conf fail
* Switch one of the test cases to not use a derive dkey
* wip
* comments
For situations where you want the Vault agent to handle one or more templates but do not require the acquired credentials elsewhere.
Modify the logic in SyncServer so that if there are no sinks, ignore any new credentials. Since SyncServer is responsible for shutting down the agent, make sure it still properly shuts down in this new situation.
Solves #7988
* rename UseAutoAuthForce to ForceAutoAuth, because I think it reads better
* Document 'ForceAuthAuthToken' option for Agent Cache
* Update website/pages/docs/agent/caching/index.mdx
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* Add additional tests around use_auto_auth=force and add documentation
* remove note, it's no longer correct
Co-authored-by: Jim Kalafut <jim@kalafut.net>
* Added flag to disable X-Vault-Token header proxy if client passes the token
* Reveresed the flag value to better match the name intent
* Introduced UseAutoAuthTokenRaw for Cache to support triplicate value of true/false/force
Co-authored-by: Clint <catsby@users.noreply.github.com>
* Vault Agent Template: parse templates (#7540)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* Update command/agent/config/config.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* return the decode error instead of swallowing it
* Update command/agent/config/config_test.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* go mod tidy
* change error checking style
* Add agent template doc
* TemplateServer: render secrets with Consul Template (#7621)
* add template config parsing, but it's wrong b/c it's not using mapstructure
* parsing consul templates in agent config
* add additional test to configuration parsing, to cover basics
* another test fixture, rework simple test into table
* refactor into table test
* rename test
* remove flattenKeys and add other test fixture
* add template package
* WIP: add runner
* fix panic, actually copy templates, etc
* rework how the config.Vault is created and enable reading from the environment
* this was supposed to be a part of the prior commit
* move/add methods to testhelpers for converting some values to pointers
* use new methods in testhelpers
* add an unblock channel to block agent until a template has been rendered
* add note
* unblock if there are no templates
* cleanups
* go mod tidy
* remove dead code
* simple test to starT
* add simple, empty templates test
* Update package doc, error logs, and add missing close() on channel
* update code comment to be clear what I'm referring to
* have template.NewServer return a (<- chan) type, even though it's a normal chan, as a better practice to enforce reading only
* Update command/agent.go
Co-Authored-By: Jim Kalafut <jkalafut@hashicorp.com>
* update with test
* Add README and doc.go to the command/agent directory (#7503)
* Add README and doc.go to the command/agent directory
* Add link to website
* address feedback for agent.go
* updated with feedback from Calvin
* Rework template.Server to export the unblock channel, and remove it from the NewServer function
* apply feedback from Nick
* fix/restructure rendering test
* Add pointerutil package for converting types to their pointers
* Remove pointer helper methods; use sdk/helper/pointerutil instead
* update newRunnerConfig to use pointerutil and empty strings
* only wait for unblock if template server is initialized
* drain the token channel in this test
* conditionally send on channel
* implement SSRF protection header
* add test for SSRF protection header
* cleanup
* refactor
* implement SSRF header on a per-listener basis
* cleanup
* cleanup
* creat unit test for agent SSRF
* improve unit test for agent SSRF
* add VaultRequest SSRF header to CLI
* fix unit test
* cleanup
* improve test suite
* simplify check for Vault-Request header
* add constant for Vault-Request header
* improve test suite
* change 'config' to 'agentConfig'
* Revert "change 'config' to 'agentConfig'"
This reverts commit 14ee72d21fff8027966ee3c89dd3ac41d849206f.
* do not remove header from request
* change header name to X-Vault-Request
* simplify http.Handler logic
* cleanup
* simplify http.Handler logic
* use stdlib errors package
* Simplify Run(): the function that was being sent over a channel doesn't
need to close over anything except latestToken, and we don't need to
create a new one each iteration. Instead just pass the relevant items,
namely the token and sink to work on.
* Disallow the following config combinations:
1. auto_auth.method.wrap_ttl > 0 and multiple file sinks
2. auto_auth.method.wrap_ttl > 0 and single file sink with wrap_ttl > 0
3. auto_auth.method.wrap_ttl > 0 and cache.use_auto_auth_token = true
* Expose errors that occur when APIProxy is forwarding request to Vault.
* Fix merge issues.
* Since we want to use the Agent listener for #6384, move listener config
from top-level 'cache' block to new top-level 'listeners' block.
* Make agent config allow cache and listener blocks without auto-auth
configured.
* Listener refactoring and file system permissions
* added listenerutil and move some common code there
* Added test for verifying socket file permissions
* Change default port of agent to 8200
* address review feedback
* Address review feedback
* Read socket options from listener config
* vault-agent-cache: squashed 250+ commits
* Add proper token revocation validations to the tests
* Add more test cases
* Avoid leaking by not closing request/response bodies; add comments
* Fix revoke orphan use case; update tests
* Add CLI test for making request over unix socket
* agent/cache: remove namespace-related tests
* Strip-off the auto-auth token from the lookup response
* Output listener details along with configuration
* Add scheme to API address output
* leasecache: use IndexNameLease for prefix lease revocations
* Make CLI accept the fully qualified unix address
* export VAULT_AGENT_ADDR=unix://path/to/socket
* unix:/ to unix://
This allows it to authenticate once, then exit once all sinks have
reported success. Useful for things like an init container vs. a
sidecard container.
Also adds command-level testing of it.
Jason O'Donnell