* Add new Database v5 interface with gRPC client & server
This is primarily for making password policies available to the DB engine, however since there are a number of other problems with the current interface this is getting an overhaul to a more gRPC request/response approach for easier future compatibility.
This is the first in a series of PRs to add support for password policies in the combined database engine
* add packagespec build system
- The majority of changes in this commit are files generated
by packagespec (everything in the packages-oss.lock directory).
* add .yamllint
* update to packagespec@fd54aea4
* ci: bust packagespec cache
- Change to packagespec results in package IDs that can use
git tag refs, not just commit refs.
* update to packagepsec@5fc121d0
- This busts all caches, because of a change to the way we
no longer traverse from tag refs to commit refs, due to
the potential confusion this can cause.
- See fd54aea482
for the original change to packagespec necessitating this.
* update to packagespec@5e6c87b6
- This completes the change to allowing git tag refs
to be used for source IDs, begun in f130b940a8fbe3e9398225b08ea1d63420bef7b6
* update to packagespec@4d3c9e8b
- This busts cache, needed to apply previous change.
* remove RELEASE_BUILD_WORKFLOW_NAME
* update packagespec, add watch-ci target
* fix package names (do not refer to EDITION)
* remove EDITION input from packages-oss.yml
* bump package cache, update packagespec
* update packagespec, add 'aliases' target
* update packagespec; less output noise
* ci: give release bundle file a better name
- When performing a release build, this will include the build ID
as part of the name, making it easier to distinguish from other
builds.
* ci: create aliases tarball artifact
* ci: cache package metadata files
* ci: add metadata artifact
* ci: bust circleci package cache
* Revert "ci: bust circleci package cache"
This reverts commit 1320d182613466f0999d63f5742db12ac17f8e92.
* ci: remove aliases artifact
* ci: use buildID not workflowName to id artifacts
* packages: add BUNDLE_NAME metadata
* do not cache package metadata with binaries
* ci: bump package cache
* ci: debugging
* ci: fix package cache; update packagespec
* ci: update packagespec to 10e7beb2
* ci: write package metadata and aliases
* ci: switch to .zip artifacts
* switch package bundle back to tar.gz (from zip)
- Because of the way zip works, the zip archive was over 2GB rather than under 750MB as with tar.gz.
* bump packagespec, adds list-staged-builds
* update packagespec
* add publish stub + general tidy up
* bump packagespec
* bump packagespec; add make publish-config
* Makefile: tidy up packagespec targets
* pass PRODUCT_REPO_ROOT to packagespec
* bump go to 1.14.6
* packages-oss.yml: use more explicit base image
* bump packagespec to b899b7c1
* bump packagespec to f040ce8f
* packages-oss.yml: pin base image to digest
- This digest is pointed to by debian:buster-20200720
- Using a specific digest ensures that builds use the same
base image in all contexts
* add release-repo; bump packagespec
* remove BUILD_TAGS and -tags flag
* bump packagespec to e444f742
* bump to go1.14.7
* ci: bump to go1.14.7
* UI update headless chrome flags to fix ember/chrome crash in test
* UI update references from test-oss to test:oss
* UI removed disable-gpu flag from headless Chrome
* UI and also remove the comment
* be more specific about node version, and specify a yarn version
* update ember, ember-cli, ember-data, ember-data-model-fragments
* use router handlers to access transition information
* fix shadowing of component helper
* update ivy-codemirror, ember-cli-inject-live-reload
* remove custom router service
* don't use transition.queryParams
* update ember-cli-deprecation-workflow
* refactor kv v1 to use 'path' instead of 'id' on creation
* fix auth-jwt-test and toolbar-link-test
* update ember composable helpers
* remove Ember.copy from test file
* no more deprecations in the workflow
* fix more secret tests
* fix remaining failed tests
* move select component to core because it's used by ttl-picker
* generate new model class for each test instead of reusing an existing one
* fix selectors on kmip tests
* refactor how control groups construct urls from the new transition objects
* add router service override back in, and have it be evented so that we can trigger router events on it
* move stories and markdown files to core if the component lives in core
* update ember-cli, ember-cli-babel, ember-auto-import
* update base64js, date-fns, deepmerge, codemirror, broccoli-asset-rev
* update linting rules
* fix test selectors
* update ember-api-actions, ember-concurrency, ember-load-initializers, escape-string-regexp, normalize.css, prettier-eslint-cli, jsdoc-to-markdown
* remove test-results dir
* update base64js, ember-cli-clipboard, ember-cli-sass, ember-cli-string-helpers, ember-cli-template-lint, ember-cli-uglify, ember-link-action
* fix linting
* run yarn install without restoring from cache
* refactor how tests are run and handle the vault server subprocess
* update makefile for new test task names
* update circle config to use the new yarn task
* fix writing the seal keys when starting the dev server
* remove optional deps from the lockfile
* don't ignore-optional on yarn install
* remove errant console.log
* update ember-basic-dropdown-hover, jsonlint, yargs-parser
* update ember-cli-flash
* add back optionalDeps
* update @babel/core@7.5.5, ember-basic-dropdown@1.1.3, eslint-plugin-ember@6.8.2
* update storybook to the latest release
* add a babel config with targets so that the ember babel plugin works properly
* update ember-resolver, move ember-cli-storybook to devDependencies
* revert normalize.css upgrade
* silence fetchadapter warning for now
* exclude 3rd party array helper now that ember includes one
* fix switch and entity lookup styling
* only add -root suffix if it's not in versions mode
* make sure drop always has an array on the aws role form
* fix labels like we did with the backport
* update eslintignore
* update the yarn version in the docker build file
* update eslint ignore
* ci: update Makefile
- Copied from c5c5149583/copythis.circleci/Makefile
* Makefile: make ci- targets phony and more explicit
* ci: add pre-flight checks
- Right now just validates ci config is up to date.
* ci: add --skip-update-check to circleci
* ci: remove cruft from Makefile
* ci: fix missing source dir error message
* ci: consolidate circleci path handling
* Work on raft backend
* Add logstore locally
* Add encryptor and unsealable interfaces
* Add clustering support to raft
* Remove client and handler
* Bootstrap raft on init
* Cleanup raft logic a bit
* More raft work
* Work on TLS config
* More work on bootstrapping
* Fix build
* More work on bootstrapping
* More bootstrapping work
* fix build
* Remove consul dep
* Fix build
* merged oss/master into raft-storage
* Work on bootstrapping
* Get bootstrapping to work
* Clean up FMS and node-id
* Update local node ID logic
* Cleanup node-id change
* Work on snapshotting
* Raft: Add remove peer API (#906)
* Add remove peer API
* Add some comments
* Fix existing snapshotting (#909)
* Raft get peers API (#912)
* Read raft configuration
* address review feedback
* Use the Leadership Transfer API to step-down the active node (#918)
* Raft join and unseal using Shamir keys (#917)
* Raft join using shamir
* Store AEAD instead of master key
* Split the raft join process to answer the challenge after a successful unseal
* get the follower to standby state
* Make unseal work
* minor changes
* Some input checks
* reuse the shamir seal access instead of new default seal access
* refactor joinRaftSendAnswer function
* Synchronously send answer in auto-unseal case
* Address review feedback
* Raft snapshots (#910)
* Fix existing snapshotting
* implement the noop snapshotting
* Add comments and switch log libraries
* add some snapshot tests
* add snapshot test file
* add TODO
* More work on raft snapshotting
* progress on the ConfigStore strategy
* Don't use two buckets
* Update the snapshot store logic to hide the file logic
* Add more backend tests
* Cleanup code a bit
* [WIP] Raft recovery (#938)
* Add recovery functionality
* remove fmt.Printfs
* Fix a few fsm bugs
* Add max size value for raft backend (#942)
* Add max size value for raft backend
* Include physical.ErrValueTooLarge in the message
* Raft snapshot Take/Restore API (#926)
* Inital work on raft snapshot APIs
* Always redirect snapshot install/download requests
* More work on the snapshot APIs
* Cleanup code a bit
* On restore handle special cases
* Use the seal to encrypt the sha sum file
* Add sealer mechanism and fix some bugs
* Call restore while state lock is held
* Send restore cb trigger through raft log
* Make error messages nicer
* Add test helpers
* Add snapshot test
* Add shamir unseal test
* Add more raft snapshot API tests
* Fix locking
* Change working to initalize
* Add underlying raw object to test cluster core
* Move leaderUUID to core
* Add raft TLS rotation logic (#950)
* Add TLS rotation logic
* Cleanup logic a bit
* Add/Remove from follower state on add/remove peer
* add comments
* Update more comments
* Update request_forwarding_service.proto
* Make sure we populate all nodes in the followerstate obj
* Update times
* Apply review feedback
* Add more raft config setting (#947)
* Add performance config setting
* Add more config options and fix tests
* Test Raft Recovery (#944)
* Test raft recovery
* Leave out a node during recovery
* remove unused struct
* Update physical/raft/snapshot_test.go
* Update physical/raft/snapshot_test.go
* fix vendoring
* Switch to new raft interface
* Remove unused files
* Switch a gogo -> proto instance
* Remove unneeded vault dep in go.sum
* Update helper/testhelpers/testhelpers.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* Update vault/cluster/cluster.go
* track active key within the keyring itself (#6915)
* track active key within the keyring itself
* lookup and store using the active key ID
* update docstring
* minor refactor
* Small text fixes (#6912)
* Update physical/raft/raft.go
Co-Authored-By: Calvin Leung Huang <cleung2010@gmail.com>
* review feedback
* Move raft logical system into separate file
* Update help text a bit
* Enforce cluster addr is set and use it for raft bootstrapping
* Fix tests
* fix http test panic
* Pull in latest raft-snapshot library
* Add comment
* ci: break config into separate files
* Untangle githooks
* githooks: fix whitespace
* .hooks/pre-commit: add ui -> lint-staged check
- We no longer require dependency on husky with this change.
* ui: remove husky dependency and config
- The previous commit obviates the need for it.
- We will now have to manage these hooks by hand, but this removes
the conflict between husky-installed hooks and those in the .hooks dir.
* ui: update yarn.lock with husky removed
* .hooks/pre-commit: always use subshell + docs
- Always use subshell means we consistently exit from the
same place which feels less complex.
- Docs are necessary for horrible bash like this I think...
* Makefile: remove old husky githooks
- Husky has installed a handler for every single git hook.
- This causes warnings on every git operation.
- Eventually we can remove this, but better not to confuse
people with these messages for now.
* ci: fix go build tags
* Makefile: improve compatibility of rm call
- Looks like the xargs in Travis does something different to the one
on my mac, this more verbose call should be safe everywhere.
* ci: fix make target names
* ci: fix test-ui invocation
* Makefile: simplify husky hook cleanup
* ci: more focussed readme
* ci: make ember-ci-test (browserstack) VLTES-28
* Update .circleci/config.yml
Co-Authored-By: Josh Freda <jfreda@users.noreply.github.com>
* Update .circleci/config.yml
Co-Authored-By: Josh Freda <jfreda@users.noreply.github.com>
* ci: rename test-ember -> test-ui-browserstack
* Makefile: ember-ci-test -> test-ui-browserstack
- Use the same name for this test everywhere to reduce cognitive load.
* browserstack: exit non-zero on failure to connect
* .travis.yml: ember-ci-test -> test-ui-browserstack
* browserstack: add vault bin to the path
* Makefile: browserstack: fail early w/clear msgs
This might save someone time later. The same checks could be applied
elsewhere too trivially.
* add browserstack
* check for data before removing root token
* fix root prefix and select by attributes for ie11
* use objectAt for ie11
* use blobs instead of files for ie11
* manually round cirucmference for ie11
* skip csp test on ie11
* skip tests in ie11
* include polyfill for CI
* remove on exit hooks
* update which browserstack tests are run
* remove ie check since we are not running these tests in ie
* remove ie check since we are not running these tests in ie
The static-assets target has a dependency on *either* ember-dist or
ember-dist-dev, so these targets must not execute in parallel. Since
this is an either/or dependency, it cannot be expressed as a regular
dependency unless the targets are refactored somehow.
Fixes: 7a312d7c37bb ("Add Makefile/Dockerfile UI bits")
* add make targets to build the ui without minifying the source
* add polyfill for string.prototype.endswith and move to npm version of string.prototype.startswith
* replace the current history state with a / on the list secrets endpoint
* add test for list with id
* Add entity information request to system view
* fixing a few comments
* sharing types between plugin and logical
* sharing types between plugin and logical
* fixing output directory for proto
* removing extra replacement
* adding mount type lookup
* empty entities return nil instead of error
* adding some comments
* Add grpc plugins
* Add grpc plugins
* Translate wrap info to/from proto
* Add nil checks
* Fix nil marshaling errors
* Provide logging through the go-plugin logger
* handle errors in the messages
* Update the TLS config so bidirectional connections work
* Add connectivity checks
* Restart plugin and add timeouts where context is not availible
* Add the response wrap data into the grpc system implementation
* Add leaseoptions to pb.Auth
* Add an error translator
* Add tests for translating the proto objects
* Fix rename of function
* Add tracing to plugins for easier debugging
* Handle plugin crashes with the go-plugin context
* Add test for grpcStorage
* Add tests for backend and system
* Bump go-plugin for GRPCBroker
* Remove RegisterLicense
* Add casing translations for new proto messages
* Use doneCtx in grpcClient
* Use doneCtx in grpcClient
* s/shutdown/shut down/
* Start work on context aware backends
* Start work on moving the database plugins to gRPC in order to pass context
* Add context to builtin database plugins
* use byte slice instead of string
* Context all the things
* Move proto messages to the dbplugin package
* Add a grpc mechanism for running backend plugins
* Serve the GRPC plugin
* Add backwards compatibility to the database plugins
* Remove backend plugin changes
* Remove backend plugin changes
* Cleanup the transport implementations
* If grpc connection is in an unexpected state restart the plugin
* Fix tests
* Fix tests
* Remove context from the request object, replace it with context.TODO
* Add a test to verify netRPC plugins still work
* Remove unused mapstructure call
* Code review fixes
* Code review fixes
* Code review fixes
* Explicitly check go version in build
Several GH issues have been opened by people trying to use an older
version of Go to build Vault (e.g., #3307 is the most recent). This adds
an explicit check to the build to hopefully make it more clear to users
in the future.
* Also add checking for go patch version
* Up minimum go version
And fix a comment
* Bump travis to go1.9.1
* porting identity to OSS
* changes that glue things together
* add testing bits
* wrapped entity id
* fix mount error
* some more changes to core
* fix storagepacker tests
* fix some more tests
* fix mount tests
* fix http mount tests
* audit changes for identity
* remove upgrade structs on the oss side
* added go-memdb to vendor
* Fix the mysql legacy username length
* Remove boolean parameter
* Add a MySQL 5.6 container to test the legacy MySQL plugin against
* Add database plugins to the make file
* Fix credsutil test
Mark Gritter