This always specifies a color UI, but explicitly marks the output as
noncolorable if we don't want color. This allows getting rid of our
hacky Output function in favor of cli's normal functions.
* Allow returning list information and other data in 404s.
On read it'll output data and/or warnings on a 404 if they exist. On
list, the same behavior; the actual 'vault list' command doesn't change
behavior though in terms of output unless there are no actual keys (so
it doesn't just magically show other data).
This corrects some assumptions in response_util and wrapping.go; it also
corrects a few places in the latter where it could leak a (useless)
token in some error cases.
* Use same 404 logic in delete/put too
* Add the same secret parsing logic to the KV request functions
* adding UI handlers and UI header configuration
* forcing specific static headers
* properly getting UI config value from config/environment
* fixing formatting in stub UI text
* use http.Header
* case-insensitive X-Vault header check
* fixing var name
* wrap both stubbed and real UI in header handler
* adding test for >1 keys
* logbridge with hclog and identical output
* Initial search & replace
This compiles, but there is a fair amount of TODO
and commented out code, especially around the
plugin logclient/logserver code.
* strip logbridge
* fix majority of tests
* update logxi aliases
* WIP fixing tests
* more test fixes
* Update test to hclog
* Fix format
* Rename hclog -> log
* WIP making hclog and logxi love each other
* update logger_test.go
* clean up merged comments
* Replace RawLogger interface with a Logger
* Add some logger names
* Replace Trace with Debug
* update builtin logical logging patterns
* Fix build errors
* More log updates
* update log approach in command and builtin
* More log updates
* update helper, http, and logical directories
* Update loggers
* Log updates
* Update logging
* Update logging
* Update logging
* Update logging
* update logging in physical
* prefixing and lowercase
* Update logging
* Move phyisical logging name to server command
* Fix som tests
* address jims feedback so far
* incorporate brians feedback so far
* strip comments
* move vault.go to logging package
* update Debug to Trace
* Update go-plugin deps
* Update logging based on review comments
* Updates from review
* Unvendor logxi
* Remove null_logger.go
Some commands didn't setup a local test server since they didn't need
it. Other commands didn't setup a local test server because Seth forgot.
Long story short, I kept seeing weird requests to my Vault server when I
ran tests, and that should never happen. This ensures all test requests
will go to a test Vault instance.
Benchmarks show this adds 0.4s to the command test suite.
* Add more cli subcommands
* Add metadata commands
* Add more subcommands
* Update cli
* Move archive commands to delete
* Add helpers for making http calls to the kv backend
* rename cli header
* Format the various maps from kv
* Add list command
* Update help text
* Add a command to enable versioning on a backend
* Rename enable-versions command
* Some review feedback
* Fix listing of top level keys
* Fix issue when metadata is nil
* Add test for lising top level keys
* Fix some typos
* Add a note about deleting all versions
* Add some requirements for versioned k/v
* Add a warning message when an upgrade is triggered
* Add path help values
* Make the kv header a const
* Add the uid to mount entry instead of options map
* Pass the backend aware uuid to the mounts and plugins
* Fix comment
* Add options to secret/auth enable and tune CLI commands (#4170)
* Switch mount/tune options to use TypeKVPairs (#4171)
* switching options to TypeKVPairs, adding bool parse for versioned flag
* flipping bool check
* Fix leases coming back from non-leased pluin kv store
* add a test for updating mount options
* Fix tests
* Add audit hmac values to AuthConfigInput and AuthConfigOutput, fix docs
* docs: Add ttl params to auth enable endpoint
* Rewording of go string to simply string
* Add audit hmac keys as CLI flags on auth/secrets enable
* Fix copypasta mistake
* WIP on auth-list endpoint
* Rename variable to be singular, add CLI flag, show value in auth and secrets list
* Add audit hmac keys to auth and secrets list
* Only set config values if they exist
* Fix http sys/auth tests
* More auth plugin_name test fixes
* Rename tag internal_ui_show_mount to _ui_show_mount
* Add tests
* Make endpoint unauthed
* Rename field to listing_visibility
* Add listing-visibility to cli tune commands
* Use ListingVisiblityType
* Fix type conversion
* Do not actually change token's value on testHttpGet
* Remove unused ListingVisibilityAuth, use const in pathInternalUIMountsRead
* Add audit hmac values to AuthConfigInput and AuthConfigOutput, fix docs
* docs: Add ttl params to auth enable endpoint
* Rewording of go string to simply string
* Add audit hmac keys as CLI flags on auth/secrets enable
* Fix copypasta mistake
* Add audit hmac keys to auth and secrets list
* Only set config values if they exist
* Fix http sys/auth tests
* More auth plugin_name test fixes
* Pass API values into MountEntry's config when creating auth/secrets mount
* Update usage wording
```
:~# vault status
Key Value
--- -----
Seal Type shamir
Sealed false
Total Shares 8
Threshold 2
Version 0.9.5
Cluster Name vault-cluster-8c85f1aa
Cluster ID aaaaaaaa-1111-2222-3333-444444444444
HA Enabled true
HA Cluster https://10.0.0.1:8201
HA Mode standby
Active Node Address: https://10.0.0.1:8200
```
`Active Node Address:` - is the only one with a colon at the end.
This PR fix that output style issue.
* Create a new command/config subpackage.
This PR extracts the functions associated with loading and parsing
configs, and the DefaultTokenHelper, into a command/config subpackage,
just like TokenHelpers are in the command/token subpackage. The goal is
to allow other clients (in this case, the Vault and Nomad Terraform
providers, but in theory any client that wants to lean on Vault's
default behaviour) to reuse this logic and not drift from Vault, without
vendoring the entirety of Vault.
To retain backwards compatibility, I didn't remove any functions from
the command package; I just copied them into the command/config package,
and update the functions in the command package to call through to the
config package.
* Allow formatted data when using -field and -format together.
As a special case, allows "data" to be passed in to get the entire data
struct output.
* If data exists in the output map use that instead when special casing
This PR adds a new Storage Backend for Triton's Object Storage - Manta
```
make testacc TEST=./physical/manta
==> Checking that code complies with gofmt requirements...
==> Checking that build is using go version >= 1.9.1...
go generate
VAULT_ACC=1 go test -tags='vault' ./physical/manta -v -timeout 45m
=== RUN TestMantaBackend
--- PASS: TestMantaBackend (61.18s)
PASS
ok github.com/hashicorp/vault/physical/manta 61.210s
```
Manta behaves differently to how S3 works - it has no such concepts of Buckets - it is merely a filesystem style object store
Therefore, we have chosen the approach of when writing a secret `foo` it will actually map (on disk) as foo/.vault_value
The reason for this is because if we write the secret `foo/bar` and then try and Delete a key using the name `foo` then Manta
will complain that the folder is not empty because `foo/bar` exists. Therefore, `foo/bar` is written as `foo/bar/.vault_value`
The value of the key is *always* written to a directory tree of the name and put in a `.vault_value` file.
* Use Colored UI if stdout is a tty
* Add format options to operator unseal
* Add format test on operator unseal
* Add -no-color output flag, and use BasicUi if no-color flag is provided
* Move seal status formatting logic to OutputSealStatus
* Apply no-color to warnings from DeprecatedCommands as well
* Add OutputWithFormat to support arbitrary data, add format option to auth list
* Add ability to output arbitrary list data on TableFormatter
* Clear up switch logic on format
* Add format option for list-related commands
* Add format option to rest of commands that returns a client API response
* Remove initOutputYAML and initOutputJSON, and use OutputWithFormat instead
* Remove outputAsYAML and outputAsJSON, and use OutputWithFormat instead
* Remove -no-color flag, use env var exclusively to toggle colored output
* Fix compile
* Remove -no-color flag in main.go
* Add missing FlagSetOutputFormat
* Fix generate-root/decode test
* Migrate init functions to main.go
* Add no-color flag back as hidden
* Handle non-supported data types for TableFormatter.OutputList
* Pull formatting much further up to remove the need to use c.flagFormat (#3950)
* Pull formatting much further up to remove the need to use c.flagFormat
Also remove OutputWithFormat as the logic can cause issues.
* Use const for env var
* Minor updates
* Remove unnecessary check
* Fix SSH output and some tests
* Fix tests
* Make race detector not run on generate root since it kills Travis these days
* Update docs
* Update docs
* Address review feedback
* Handle --format as well as -format
* Use SHA2-256 hash with prefix to upgrade the paths
* test the SHA1 upgrade to SHA256
* Remove hash identifier and the delimiter; use 's' instead
* Added API test to verify the correctness of the fix
* Fix broken test
* remove unneeded test
* external identity groups
* add local LDAP groups as well to group aliases
* add group aliases for okta credential backend
* Fix panic in tests
* fix build failure
* remove duplicated struct tag
* add test steps to test out removal of group member during renewals
* Add comment for having a prefix check in router
* fix tests
* s/parent_id/canonical_id
* s/parent/canonical in comments and errors
* Redo the API client quite a bit to make the behavior of NewClient more
predictable and add locking to make it safer to use with Clone() and if
multiple goroutines for some reason decide to change things.
Along the way I discovered that currently, the x/net/http2 package is
broke with the built-in h2 support in released Go. For those using
DefaultConfig (the vast majority of cases) this will be a non-event.
Others can manually call http2.ConfigureTransport as needed. We should
keep an eye on commits on that repo and consider more updates before
release. Alternately we could go back revisions but miss out on bug
fixes; my theory is that this is not a purposeful break and I'll be
following up on this in the Go issue tracker.
In a few tests that don't use NewTestCluster, either for legacy or other
reasons, ensure that http2.ConfigureTransport is called.
* Use tls config cloning
* Don't http2.ConfigureServer anymore as current Go seems to work properly without requiring the http2 package
* Address feedback
This removes all references I could find to:
- credential provider
- authentication backend
- authentication provider
- auth provider
- auth backend
in favor of the unified:
- auth method
If the value is being "piped", we don't print colors or the newline character at the end. If it's not, we still give users pretty when selecting a raw field/value.
* disable raw endpoint by default
* adding docs
* config option raw -> raw_storage_endpoint
* docs updates
* adding listing on raw endpoint
* reworking tests for enabled raw endpoints
* root protecting base raw endpoint
* Add basic autocompletion
* Add autocomplete to some common commands
* Autocomplete the generate-root flags
* Add information about autocomplete to the docs
Previously we lowercased names on ingress but not on lookup or delete
which could cause unexpected results. Now, just unilaterally lowercase
policy names on write and delete. On get, to avoid the performance hit
of always lowercasing when not necessary since it's in the critical
path, we have a minor optimization -- we check the LRU first before
normalizing. For tokens, because they're already normalized when adding
policies during creation, this should always work; it might just be
slower for API calls.
Fixes#3187
1. The current implementation of the SSH command is heavily tied to the
assumptions of OTP/dynamic key types. The SSH CA backend is
fundamentally a different approach to login and authentication. As a
result, there was some restructuring of existing methods to share more
code and state.
2. Each authentication method (ca, otp, dynamic) are now fully-contained
in their own handle* function.
3. -mode and -role are going to be required for SSH CA, and I don't
think the magical UX (and overhead) of guessing them is a good UX. It's
confusing as to which role and how Vault guesses. We can reduce 66% of
the API calls and add more declaration to the CLI by making -mode and
-role required. This commit adds warnings for that deprecation, but
these values are both required for CA type authentication.
4. The principal and extensions are currently fixed, and I personally
believe that's good enough for the first pass at this. Until we
understand what configuration options users will want, I think we should
ship with all the local extensions enabled. Users who don't want that
can generate the key themselves directly (current behavior) or submit
PRs to make the map of extensions customizable.
5. Host key checking for the CA backend is not currently implemented.
It's not strictly required at setup, so I need to think about whether it
belongs here.
This is not ready for merge, but it's ready for early review.
From the sshpass manpage:
> The -p option should be considered the least secure of all of sshpass's options. All system users can see the password in the command line with a simple "ps" command. Sshpass makes a minimal attempt to hide the password, but such attempts are doomed to create race conditions without actually solving the problem. Users of sshpass are encouraged to use one of the other password passing techniques, which are all more secure.
This PR changes the sshpass behavior to execute a subprocess with the
SSHPASS envvar (which is generally regarded as more secure) than using
the -p option.
* Store original request path in WrapInfo as CreationPath
* Add wrapping_token_creation_path to CLI output
* Add CreationPath to AuditResponseWrapInfo
* Fix tests
* Add and fix tests, update API docs with new sample responses
* Add backend plugin changes
* Fix totp backend plugin tests
* Fix logical/plugin InvalidateKey test
* Fix plugin catalog CRUD test, fix NoopBackend
* Clean up commented code block
* Fix system backend mount test
* Set plugin_name to omitempty, fix handleMountTable config parsing
* Clean up comments, keep shim connections alive until cleanup
* Include pluginClient, disallow LookupPlugin call from within a plugin
* Add wrapper around backendPluginClient for proper cleanup
* Add logger shim tests
* Add logger, storage, and system shim tests
* Use pointer receivers for system view shim
* Use plugin name if no path is provided on mount
* Enable plugins for auth backends
* Add backend type attribute, move builtin/plugin/package
* Fix merge conflict
* Fix missing plugin name in mount config
* Add integration tests on enabling auth backend plugins
* Remove dependency cycle on mock-plugin
* Add passthrough backend plugin, use logical.BackendType to determine lease generation
* Remove vault package dependency on passthrough package
* Add basic impl test for passthrough plugin
* Incorporate feedback; set b.backend after shims creation on backendPluginServer
* Fix totp plugin test
* Add plugin backends docs
* Fix tests
* Fix builtin/plugin tests
* Remove flatten from PluginRunner fields
* Move mock plugin to logical/plugin, remove totp and passthrough plugins
* Move pluginMap into newPluginClient
* Do not create storage RPC connection on HandleRequest and HandleExistenceCheck
* Change shim logger's Fatal to no-op
* Change BackendType to uint32, match UX backend types
* Change framework.Backend Setup signature
* Add Setup func to logical.Backend interface
* Move OptionallyEnableMlock call into plugin.Serve, update docs and comments
* Remove commented var in plugin package
* RegisterLicense on logical.Backend interface (#3017)
* Add RegisterLicense to logical.Backend interface
* Update RegisterLicense to use callback func on framework.Backend
* Refactor framework.Backend.RegisterLicense
* plugin: Prevent plugin.SystemViewClient.ResponseWrapData from getting JWTs
* plugin: Revert BackendType to remove TypePassthrough and related references
* Fix typo in plugin backends docs
* exclude /sys/leases/renew from registering with expiration manager
* adding sys/leases/renew to return full secret object, adding tests to catch renew errors
* Normalize "X arguments expected" messages
* Use "Vault" when referring to the product and "vault" when referring to an instance of the product
* Various minor tweaks to improve readability and/or provide clarity
Adds HUP support for audit log files to close and reopen. This makes it
much easier to deal with normal log rotation methods.
As part of testing this I noticed that HUP and other items that come out
of command/server.go are going to stderr, which is where our normal log
lines go. This isn't so much problematic with our normal output but as
we officially move to supporting other formats this can cause
interleaving issues, so I moved those to stdout instead.
* Provide base64 keys in addition to hex encoded.
Accept these at unseal/rekey time.
Also fix a bug where backup would not be honored when doing a rekey with
no operation currently ongoing.
This makes it easier to understand the expected lifetime without a
lookup call that uses the single use left on the token.
This also adds a couple of safety checks and for JSON uses int, rather
than int64, for the TTL for the wrapped token.
Vault will now register itself with Consul. The active node can be found using `active.vault.service.consul`. All standby vaults are available via `standby.vault.service.consul`. All unsealed vaults are considered healthy and available via `vault.service.consul`. Change in status and registration is event driven and should happen at the speed of a write to Consul (~network RTT + ~1x fsync(2)).
Healthy/active:
```
curl -X GET 'http://127.0.0.1:8500/v1/health/service/vault?pretty' && echo;
[
{
"Node": {
"Node": "vm1",
"Address": "127.0.0.1",
"TaggedAddresses": {
"wan": "127.0.0.1"
},
"CreateIndex": 3,
"ModifyIndex": 20
},
"Service": {
"ID": "vault:127.0.0.1:8200",
"Service": "vault",
"Tags": [
"active"
],
"Address": "127.0.0.1",
"Port": 8200,
"EnableTagOverride": false,
"CreateIndex": 17,
"ModifyIndex": 20
},
"Checks": [
{
"Node": "vm1",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"CreateIndex": 3,
"ModifyIndex": 3
},
{
"Node": "vm1",
"CheckID": "vault-sealed-check",
"Name": "Vault Sealed Status",
"Status": "passing",
"Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
"Output": "",
"ServiceID": "vault:127.0.0.1:8200",
"ServiceName": "vault",
"CreateIndex": 19,
"ModifyIndex": 19
}
]
}
]
```
Healthy/standby:
```
[snip]
"Service": {
"ID": "vault:127.0.0.2:8200",
"Service": "vault",
"Tags": [
"standby"
],
"Address": "127.0.0.2",
"Port": 8200,
"EnableTagOverride": false,
"CreateIndex": 17,
"ModifyIndex": 20
},
"Checks": [
{
"Node": "vm2",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"CreateIndex": 3,
"ModifyIndex": 3
},
{
"Node": "vm2",
"CheckID": "vault-sealed-check",
"Name": "Vault Sealed Status",
"Status": "passing",
"Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
"Output": "",
"ServiceID": "vault:127.0.0.2:8200",
"ServiceName": "vault",
"CreateIndex": 19,
"ModifyIndex": 19
}
]
}
]
```
Sealed:
```
"Checks": [
{
"Node": "vm2",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"CreateIndex": 3,
"ModifyIndex": 3
},
{
"Node": "vm2",
"CheckID": "vault-sealed-check",
"Name": "Vault Sealed Status",
"Status": "critical",
"Notes": "Vault service is healthy when Vault is in an unsealed status and can become an active Vault server",
"Output": "Vault Sealed",
"ServiceID": "vault:127.0.0.2:8200",
"ServiceName": "vault",
"CreateIndex": 19,
"ModifyIndex": 38
}
]
```
No signal handler was setup to receive SIGINT. I didn't investigate to
see if signal(2) mask was setup (ala `SIG_IGN`) or if sigprocmask(2) is
being used, but in either case, the correct behavior is to capture and
treat SIGINT the same as SIGTERM. At some point in the future these two
signals may affect the running process differently, but we will clarify
that difference in the future.
One thing that has been a point of confusion for users is Vault's
response when deleting a key that does not actually exist in the system.
For example, consider:
$ vault delete secret/foo
Success! Deleted 'secret/foo'
This message is misleading if the secret does not exist, especially if
the same command is run twice in a row.
Obviously the reason for this is clear - returning an error if a secret
does not exist would reveal the existence of a secret (the same reason
everything on S3 is a 403 or why GitHub repos 404 instead of 403 if you
do not have permission to view them).
I think we can make the UX a little bit better by adding just a few
words to the output:
$ vault delete secret/foo
Success! Deleted 'secret/foo' if it existed
This makes it clear that the operation was only performed if the secret
existed, but it does not reveal any more information.
This came up twice, in two different training courses. The UX is a
little confusing here on the CLI. Users are used to running:
$ vault auth abcd-1234...
So when they auth using a method, the output leads them to believe the
need to "re-auth" as the generated token:
$ vault auth -method=userpass username=foo password=bar
Successfully authenticated!
token: defg-5678...
A number of users then run:
$ vault auth defg-5678
I've added some helpful text to hint this is not required if the method
is not "token".