* Allow max request size to be user-specified
This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.
This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.
* Switch to ioutil.ReadAll()
In current Vault server EncodedToken will always be populated regardless
of type (root, DR), so prioritize that, and properly refer to it as
Encoded Token instead of Root Token.
Additionally refer to the nonce as the Operation nonce instead of the
Root generation operation nonce since it's used for both strategies.
* Add an idle timeout for the server
Because tidy operations can be long-running, this also changes all tidy
operations to behave the same operationally (kick off the process, get a
warning back, log errors to server log) and makes them all run in a
goroutine.
This could mean a sort of hard stop if Vault gets sealed because the
function won't have the read lock. This should generally be okay
(running tidy again should pick back up where it left off), but future
work could use cleanup funcs to trigger the functions to stop.
* Fix up tidy test
* Add deadline to cluster connections and an idle timeout to the cluster server, plus add readheader/read timeout to api server
* Add `kv rollback`
Like `kv patch` this is more of a helper than anything else; it provides
a single command to fetch the current version (for CAS), read the
version you want to roll back to, and set it as the new version (using
CAS for safety).
This can be used when errors are happening early on to avoid them being
swallowed by logGate.
This also does a bit of cleanup of format env var checking --
helper/logging internally looks for this so it was totally unnecessary
since moving to hclog.
* Allow vault ssh to accept ssh commands in any ssh compatible format
Previously vault ssh required ssh commands to be in the format
`username@hostname <flags> command`. While this works just fine for human
users this breaks a lot of automation workflows and is not compatible
with the options that the ssh client supports.
Motivation
We currently run ansible which uses vault ssh to connect to hosts.
Ansible generates ssh commands with the format `ssh <flags> -o User=username hostname
command`. While this is a valid ssh command it currently breaks with
vault because vault expects the format to be `username@hostname`. To work
around this we currently use a wrapper script to parse the correct username being set
by ansible and translate this into a vault ssh compatible `username@hostname` format
Changes
* You can now specify arguments in any order that ssh client allows. All
arguments are passed directly to the ssh command and the format isn't
modified in any way.
* The username and port are parsed from the specified ssh command. It
will accept all of the options supported by the ssh command and also
will properly prefer `-p` and `user@` if both options are specified.
* The ssh port is only added from the vault credentials if it hasn't
been specified on the command line
* This changes the way policies are reported in audit logs.
Previously, only policies tied to tokens would be reported. This could
make it difficult to perform after-the-fact analysis based on both the
initial response entry and further requests. Now, the full set of
applicable policies from both the token and any derived policies from
Identity are reported.
To keep things consistent, token authentications now also return the
full set of policies in api.Secret.Auth responses, so this both makes it
easier for users to understand their actual full set, and it matches
what the audit logs now report.
* Fix writing to KVv2 root via `kv put`
The check that adds the API path wasn't taking into account the root,
e.g. if it's mounted at `kv`, `kv` and `kv/` would end up creating an
extra copy of the mount path in front, leading to paths like
`kv/data/kv`.
* Output warnings if they come back and fix a panic in metadata_get
* Also add to metadata put/delete
Don't set a default value for the UserKnownHostsFile flag.
Only append `-o UserKnownHostsFile` to the ssh command if it
has been specified by the user or vault ssh has set it based on another
flag (such as flagHostKeyMountPoint)
Fixes https://github.com/hashicorp/vault/issues/4672
This is implementing the same fix that was added for the CA mode for vault
ssh in https://github.com/hashicorp/vault/pull/3922
Using the IP address caused `Host` entries in the ssh_config to not
match anymore meaning you would need to hardcode all of your IP
addresses in your ssh config instead of using DNS to connect to hosts
This takes place in two parts, since working on this exposed an issue
with response wrapping when there is a raw body set. The changes are (in
diff order):
* A CurrentWrappingLookupFunc has been added to return the current
value. This is necessary for the lookahead call since we don't want the
lookahead call to be wrapped.
* Support for unwrapping < 0.6.2 tokens via the API/CLI has been
removed, because we now have backends returning 404s with data and can't
rely on the 404 trick. These can still be read manually via
cubbyhole/response.
* KV preflight version request now ensures that its calls is not
wrapped, and restores any given function after.
* When responding with a raw body, instead of always base64-decoding a
string value and erroring on failure, on failure we assume that it
simply wasn't a base64-encoded value and use it as is.
* A test that fails on master and works now that ensures that raw body
responses that are wrapped and then unwrapped return the expected
values.
* A flag for response data that indicates to the wrapping handling that
the data contained therein is already JSON decoded (more later).
* RespondWithStatusCode now defaults to a string so that the value is
HMAC'd during audit. The function always JSON encodes the body, so
before now it was always returning []byte which would skip HMACing. We
don't know what's in the data, so this is a "better safe than sorry"
issue. If different behavior is needed, backends can always manually
populate the data instead of relying on the helper function.
* We now check unwrapped data after unwrapping to see if there were raw
flags. If so, we try to detect whether the value can be unbase64'd. The
reason is that if it can it was probably originally a []byte and
shouldn't be audit HMAC'd; if not, it was probably originally a string
and should be. In either case, we then set the value as the raw body and
hit the flag indicating that it's already been JSON decoded so not to
try again before auditing. Doing it this way ensures the right typing.
* There is now a check to see if the data coming from unwrapping is
already JSON decoded and if so the decoding is skipped before setting
the audit response.
* Update kv command to use a preflight check
* Make the existing ui endpoint return the allowed mounts
* Add kv subcommand tests
* Enable `-field` in `vault kv get/put` (#4426)
* Enable `-field` in `vault kv get/put`
Fixes#4424
* Unify nil value handling
* Use preflight helper
* Update vkv plugin
* Add all the mount info when authenticated
* Add fix the error message on put
* add metadata test
* No need to sort the capabilities
* Remove the kv client header
* kv patch command (#4432)
* Fix test
* Fix tests
* Use permission denied instead of entity disabled
This always specifies a color UI, but explicitly marks the output as
noncolorable if we don't want color. This allows getting rid of our
hacky Output function in favor of cli's normal functions.
* Allow returning list information and other data in 404s.
On read it'll output data and/or warnings on a 404 if they exist. On
list, the same behavior; the actual 'vault list' command doesn't change
behavior though in terms of output unless there are no actual keys (so
it doesn't just magically show other data).
This corrects some assumptions in response_util and wrapping.go; it also
corrects a few places in the latter where it could leak a (useless)
token in some error cases.
* Use same 404 logic in delete/put too
* Add the same secret parsing logic to the KV request functions
* adding UI handlers and UI header configuration
* forcing specific static headers
* properly getting UI config value from config/environment
* fixing formatting in stub UI text
* use http.Header
* case-insensitive X-Vault header check
* fixing var name
* wrap both stubbed and real UI in header handler
* adding test for >1 keys
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
Some commands didn't setup a local test server since they didn't need
it. Other commands didn't setup a local test server because Seth forgot.
Long story short, I kept seeing weird requests to my Vault server when I
ran tests, and that should never happen. This ensures all test requests
will go to a test Vault instance.
Benchmarks show this adds 0.4s to the command test suite.
* Add more cli subcommands
* Add metadata commands
* Add more subcommands
* Update cli
* Move archive commands to delete
* Add helpers for making http calls to the kv backend
* rename cli header
* Format the various maps from kv
* Add list command
* Update help text
* Add a command to enable versioning on a backend
* Rename enable-versions command
* Some review feedback
* Fix listing of top level keys
* Fix issue when metadata is nil
* Add test for lising top level keys
* Fix some typos
* Add a note about deleting all versions
* Add some requirements for versioned k/v
* Add a warning message when an upgrade is triggered
* Add path help values
* Make the kv header a const
* Add the uid to mount entry instead of options map
* Pass the backend aware uuid to the mounts and plugins
* Fix comment
* Add options to secret/auth enable and tune CLI commands (#4170)
* Switch mount/tune options to use TypeKVPairs (#4171)
* switching options to TypeKVPairs, adding bool parse for versioned flag
* flipping bool check
* Fix leases coming back from non-leased pluin kv store
* add a test for updating mount options
* Fix tests
* Add audit hmac values to AuthConfigInput and AuthConfigOutput, fix docs
* docs: Add ttl params to auth enable endpoint
* Rewording of go string to simply string
* Add audit hmac keys as CLI flags on auth/secrets enable
* Fix copypasta mistake
* WIP on auth-list endpoint
* Rename variable to be singular, add CLI flag, show value in auth and secrets list
* Add audit hmac keys to auth and secrets list
* Only set config values if they exist
* Fix http sys/auth tests
* More auth plugin_name test fixes
* Rename tag internal_ui_show_mount to _ui_show_mount
* Add tests
* Make endpoint unauthed
* Rename field to listing_visibility
* Add listing-visibility to cli tune commands
* Use ListingVisiblityType
* Fix type conversion
* Do not actually change token's value on testHttpGet
* Remove unused ListingVisibilityAuth, use const in pathInternalUIMountsRead
* Add audit hmac values to AuthConfigInput and AuthConfigOutput, fix docs
* docs: Add ttl params to auth enable endpoint
* Rewording of go string to simply string
* Add audit hmac keys as CLI flags on auth/secrets enable
* Fix copypasta mistake
* Add audit hmac keys to auth and secrets list
* Only set config values if they exist
* Fix http sys/auth tests
* More auth plugin_name test fixes
* Pass API values into MountEntry's config when creating auth/secrets mount
* Update usage wording
```
:~# vault status
Key Value
--- -----
Seal Type shamir
Sealed false
Total Shares 8
Threshold 2
Version 0.9.5
Cluster Name vault-cluster-8c85f1aa
Cluster ID aaaaaaaa-1111-2222-3333-444444444444
HA Enabled true
HA Cluster https://10.0.0.1:8201
HA Mode standby
Active Node Address: https://10.0.0.1:8200
```
`Active Node Address:` - is the only one with a colon at the end.
This PR fix that output style issue.
