Commit graph

188 commits

Author SHA1 Message Date
hc-github-team-secure-vault-core f52a686b91
[QT-506] Use enos scenario samples for testing (#22641) (#22933)
Replace our prior implementation of Enos test groups with the new Enos
sampling feature. With this feature we're able to describe which
scenarios and variant combinations are valid for a given artifact and
allow enos to create a valid sample field (a matrix of all compatible
scenarios) and take an observation (select some to run) for us. This
ensures that every valid scenario and variant combination will
now be a candidate for testing in the pipeline. See QT-504[0] for further
details on the Enos sampling capabilities.

Our prior implementation only tested the amd64 and arm64 zip artifacts,
as well as the Docker container. We now include the following new artifacts
in the test matrix:
* CE Amd64 Debian package
* CE Amd64 RPM package
* CE Arm64 Debian package
* CE Arm64 RPM package

Each artifact includes a sample definition for both pre-merge/post-merge
(build) and release testing.

Changes:
* Remove the hand crafted `enos-run-matrices` ci matrix targets and replace
  them with per-artifact samples.
* Use enos sampling to generate different sample groups on all pull
  requests.
* Update the enos scenario matrices to handle HSM and FIPS packages.
* Simplify enos scenarios by using shared globals instead of
  cargo-culted locals.

Note: This will require coordination with vault-enterprise to ensure a
smooth migration to the new system. Integrating new scenarios or
modifying existing scenarios/variants should be much smoother after this
initial migration.

[0] https://github.com/hashicorp/enos/pull/102

Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-09-08 13:31:09 -06:00
Sarah Thompson 2ae56bd4ac
cherrypick of a9a4b0b9ff (#22813) 2023-09-06 18:24:39 +01:00
hc-github-team-secure-vault-core b1c7b966d2
backport of commit 1e491e16d4a25001423434ca950823fbeb9ac1d9 (#22187)
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
2023-08-17 14:49:23 -04:00
hc-github-team-secure-vault-core 83b95d3efe
backport of commit 1a46088afb0d5e442273350c6793d1216b6be4d1 (#21985)
Co-authored-by: Ryan Cragun <me@ryan.ec>
2023-07-20 22:39:29 +00:00
hc-github-team-secure-vault-core 45af65da81
Backport of Limit number of tests in CI comment into release/1.14.x (#21971)
* backport of commit dc104898f700447f7764919445c7559baeb7e987 (#21853)

* fix multiline

* shellcheck, and success message for builds

* add full path

* cat the summary

* fix and faster

* fix if condition

* base64 in a separate step

* echo

* check against empty string

* add echo

* only use matrix ids

* only id

* echo matrix

* remove wrapping array

* tojson

* try echo again

* use jq to get packages

* don't quote

* only run binary tests once

* only run binary tests once

* test what's wrong with the binary

* separate file

* use matrix file

* failed test

* update comment on success

* correct variable name

* bae64 fix

* output to file

* use multiline

* fix

* fix formatting

* fix newline

* fix whitespace

* correct body, remove comma

* small fixes

* shellcheck

* another shellcheck fix

* fix deprecation checker

* only run comments for prs

* Update .github/workflows/test-go.yml

Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>

* Update .github/workflows/test-go.yml

Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>

* fixes

---------

Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>

* backport of commit 3b00dde1ba4d479fbd67b1d0767e421e495d8cce (#21936)

* limit test comments

* remove unecessary tee

* fix go test condition

* fix

* fail test

* remove ailways entirely

* fix columns

* make a bunch of tests fail

* separate line

* include Failures:

* remove test fails

* fix whitespace

* backport of commit 245430215c00d80a38283020fca114bade022e0f (#21973)

* only add binary tests if they exist

* shellcheck

---------

Co-authored-by: miagilepner <mia.epner@hashicorp.com>
Co-authored-by: Mike Palmiotto <mike.palmiotto@hashicorp.com>
2023-07-20 15:07:20 +02:00
hc-github-team-secure-vault-core 6655be126a
backport of commit 31a71f6db405e3665b8cbd951aa1d1f31d343dc8 (#21729)
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-07-14 17:30:14 +00:00
hc-github-team-secure-vault-core 5f6c3f4155
backport of commit b0fb3b14206c63c01041fe3f561b147a3d41de74 (#21720)
Co-authored-by: Alexander Scheel <alex.scheel@hashicorp.com>
2023-07-10 17:07:30 +00:00
hc-github-team-secure-vault-core d1210427d1
backport of commit 8c18f24b9da475c13f7908e609c5d4be24c773e6 (#21611) (#21615)
* combine into one checker

* combine and simplify ci checks

* add to test package list

* remove testing test

* only run deprecations check

* only run deprecations check

* remove unneeded repo check

* fix bash options

Co-authored-by: miagilepner <mia.epner@hashicorp.com>
2023-07-10 17:05:20 +02:00
Jordan Reimer c84d267c61
Sidebar Navigation (#19296)
* Add Helios Design System Components (#19278)

* adds hds dependency

* updates reset import path

* sets minifyCSS advanced option to false

* Remove node-sass (#19376)

* removes node-sass and fixes sass compilation

* fixes active tab li class

* Sidebar Navigation Components (#19446)

* links ember-shared-components addon and imports styles

* adds sidebar frame and nav components

* updates HcNav component name to HcAppFrame and adds sidebar UserMenu component

* adds tests for sidebar components

* fixes tests

* updates user menu styling

* fixes typos in nav cluster component

* changes padding value in sidebar stylesheet to use variable

* Replace and remove old nav components with new ones (#19447)

* links ember-shared-components addon and imports styles

* adds sidebar frame and nav components

* updates activeCluster on auth service and adds activeSession prop for sidebar visibility

* replaces old nav components with new ones in templates

* fixes sidebar visibility issue and updates user menu label class

* removes NavHeader usage

* adds clients index route to redirect to dashboard

* removes unused HcAppFrame footer block and reduces page header top margin

* Nav component cleanup (#19681)

* removes nav-header components

* removes navbar styling

* removes status-menu component and styles

* removes cluster and auth info components

* removes menu-sidebar component and styling

* fixes tests

* Console Panel Updates (#19741)

* updates console panel styling

* adds test for opening and closing the console panel

* updates console panel background color to use hds token

* adds right margin to console panel input

* updates link-status banner styling

* updates hc nav components to new API

* Namespace Picker Updates (#19753)

* updates namespace-picker

* updates namespace picker menu styling

* adds bottom margin to env banner

* updates class order on namespace picker link

* restores manage namespaces refresh icon

* removes manage namespaces nav icon

* removes home link component (#20027)

* Auth and Error View Updates (#19749)

* adds vault logo to auth page

* updates top level error template

* updates loading substate handling and moves policies link from access to cluster nav (#20033)

* moves console panel to bottom of viewport (#20183)

* HDS Sidebar Nav Components (#20197)

* updates nav components to hds

* upgrades project yarn version to 3.5

* fixes issues in app frame component

* updates sidenav actions to use icon button component

* Sidebar navigation acceptance tests (#20270)

* adds sidebar navigation acceptance tests and fixes other test failures

* console panel styling tweaks

* bumps addon version

* remove and ignore yarn install-state file

* fixes auth service and console tests

* moves classes from deleted files after bulma merge

* fixes sass syntax errors blocking build

* cleans up dart sass deprecation warnings

* adds changelog entry

* hides namespace picker when sidebar nav panel is minimized

* style tweaks

* fixes sidebar nav tests

* bumps hds addon to latest version and removes style override

* updates modify-passthrough-response helper

* updates sidebar nav tests

* mfa-setup test fix attempt

* fixes cluster mfa setup test

* remove deprecated yarn ignore-optional flag from makefile

* removes another instance of yarn ignore-optional and updates ui readme

* removes unsupported yarn verbose flag from ci-helper

* hides nav headings when user does not have access to any sub links

* removes unused optional deps and moves lint-staged to dev deps

* updates has-permission helper and permissions service tests

* fixes issue with console panel not filling container width
2023-05-02 19:36:15 -06:00
miagilepner b6d768d36f
Fix testonly flag in Makefile (#20457) 2023-05-02 14:46:13 +02:00
Hamid Ghaf 0cd138c0eb
use tab in Makefile instead of spaces (#20305) 2023-04-24 04:51:37 -07:00
miagilepner 1ea85c56d7
VAULT-14734: activity log write endpoint (#20019)
* add noop endpoint with testonly build flag

* add tests for endpoint

* cleanup

* fix test name

* add changelog

* pr fixes
2023-04-12 18:26:26 +02:00
akshya96 c3b1c3188c
Github Action to check deprecations in PR (#19666)
* deprecation check

* adding script

* add execute permission to script

* revert changes

* adding the script back

* added working script for local and GHA

* give execute permissions

* updating revgrep

* adding changes to script, tools

* run go mod tidy

* removing default ref

* make bootstrap

* adding to makefile
2023-03-27 22:50:58 -07:00
Marc Boudreau 27103f9908
Remove CircleCI Configuration (#19751)
* modify pre-commit hook to not look for circleci

* remove circleci configuration
2023-03-24 14:24:06 -04:00
Kuba Wieczorek 89fa73c9f7
Fix a typo in the Makefile (#19663) 2023-03-21 20:59:40 +00:00
Kuba Wieczorek 641f42f767
Add a GHA job running Go tests with race detection enabled to the CI … (#19561)
* Add a GHA job running Go tests with race detection enabled to the CI workflow

* Incorporate logic from test-go-race into the test-go testing matrix

* Make test-go testing matrix job names more meaningful

* Fix some a bug in script's logic

* Experiment: bump wait time in the failing TestLoginMFASinglePhase test to see if that makes a difference

* Lower the wait time in TestLoginMFASinglePhase

* Change the wait time in TestLoginMFASinglePhase to 15

* Add more detail to test-go testing matrix job names

* Test whether we already have access to larger runners

* Run Go tests with enabled data race detection from a separate job than the standard suite of tests

* Tweak runner sizes for OSS

* Try rebalancing test buckets

* Change instance type for larger ENT runners

* Undo rebalancing of test buckets as it changed nothing

* Change instance type for larger OSS runners

* Change the way we generate names for matrix jobs

* Consolidate the Go build tags variables, update them to use comma as a separator and fix the if statement in test-go

* Fix a typo
2023-03-21 14:13:40 -04:00
Hamid Ghaf 27bb03bbc0
adding copyright header (#19555)
* adding copyright header

* fix fmt and a test
2023-03-15 09:00:52 -07:00
Marc Boudreau f286ee5b3c
Fix failing TestHCPLinkConnected Test (#19474)
* replace use of os.Unsetenv in test with t.Setenv and remove t.Parallel from test that rely on env being modified.

* experiment with using fromJSON function

* revert previous experiment

* including double quotes in the output value for the string ubuntu-latest

* use go run to launch gofumpt
2023-03-09 13:46:54 -05:00
miagilepner c49d180bc8
VAULT-13169 Require go docs for all new test functions (#18971)
* example for checking go doc tests

* add analyzer test and action

* get metadata step

* install revgrep

* fix for ci

* add revgrep to go.mod

* clarify how analysistest works
2023-02-07 10:41:04 +01:00
Hamid Ghaf edbf093290
Allow Token Create Requests To Be Replicated (#18689)
* Allow Token Create Requests To Be Replicated

* adding a test

* revert a test
2023-01-24 14:00:27 -05:00
Ryan Cragun bd5d738ad7
[QT-436] Pseudo random artifact test scenarios (#18056)
Introducing a new approach to testing Vault artifacts before merge
and after merge/notorization/signing. Rather than run a few static
scenarios across the artifacts, we now have the ability to run a
pseudo random sample of scenarios across many different build artifacts.

We've added 20 possible scenarios for the AMD64 and ARM64 binary
bundles, which we've broken into five test groups. On any given push to
a pull request branch, we will now choose a random test group and
execute its corresponding scenarios against the resulting build
artifacts. This gives us greater test coverage but lets us split the
verification across many different pull requests.

The post-merge release testing pipeline behaves in a similar fashion,
however, the artifacts that we use for testing have been notarized and
signed prior to testing. We've also reduce the number of groups so that
we run more scenarios after merge to a release branch.

We intend to take what we've learned building this in Github Actions and
roll it into an easier to use feature that is native to Enos. Until then,
we'll have to manually add scenarios to each matrix file and manually
number the test group. It's important to note that Github requires every
matrix to include at least one vector, so every artifact that is being
tested must include a single scenario in order for all workflows to pass
and thus satisfy branch merge requirements.

* Add support for different artifact types to enos-run
* Add support for different runner type to enos-run
* Add arm64 scenarios to build matrix
* Expand build matrices to include different variants
* Update Consul versions in Enos scenarios and matrices
* Refactor enos-run environment
* Add minimum version filtering support to enos-run. This allows us to
  automatically exclude scenarios that require a more recent version of
  Vault
* Add maximum version filtering support to enos-run. This allows us to
  automatically exclude scenarios that require an older version of
  Vault
* Fix Node 12 deprecation warnings
* Rename enos-verify-stable to enos-release-testing-oss
* Convert artifactory matrix into enos-release-testing-oss matrices
* Add all Vault editions to Enos scenario matrices
* Fix verify version with complex Vault edition metadata
* Rename the crt-builder to ci-helper
* Add more version helpers to ci-helper and Makefile
* Update CODEOWNERS for quality team
* Add support for filtering matrices by group and version constraints
* Add support for pseudo random test scenario execution

Signed-off-by: Ryan Cragun <me@ryan.ec>
2022-12-12 13:46:04 -07:00
Chris Capurso d4d3e47296
remove deleted link proto paths (#18296) 2022-12-09 20:00:44 +00:00
Chris Capurso c81485d5cd
copy over link_control and meta to link proto module (#18285) 2022-12-09 11:57:35 -05:00
Chris Capurso 4dc5155c5f
Link OSS (#18228)
* add Link config, init, and capabilities

* add node status proto

* bump protoc version to 3.21.9

* make proto

* adding link tests

* remove wrapped link

* add changelog entry

* update changelog entry
2022-12-08 15:02:18 -05:00
Ryan Cragun 4c4798417f
[QT-358] Unify CRT and local builder workflows (#17766)
Here we make the following major changes:

* Centralize CRT builder logic into a script utility so that we can share the
  logic for building artifacts in CI or locally.
* Simplify the build workflow by calling a reusable workflow many times
  instead of repeating the contents.
* Create a workflow that validates whether or not the build workflow and all
  child workflows have succeeded to allow for merge protection.

Motivation

* We need branch requirements for the build workflow and all subsequent
  integration tests (QT-353)
* We need to ensure that the Enos local builder works (QT-558)
* Debugging build failures can be difficult because one has to hand craft the
  steps to recreate the build
* Merge conflicts between Vault OSS and Vault ENT build workflows are quite
  painful. As the build workflow must be the same file and name we'll reduce
  what is contained in each that is unique. Implementations of building
  will be unique per edition so we don't have to worry about conflict
  resolution.
* Since we're going to be touching the build workflow to do the first two
  items we might as well try and improve those other issues at the same time
  to reduce the overhead of backports and conflicts.

Considerations

* Build logic for Vault OSS and Vault ENT differs
* The Enos local builder was duplicating a lot of what we did in the CRT build
  workflow
* Version and other artifact metadata has been an issue before. Debugging it
  has been tedious and error prone.
* The build workflow is full of brittle copy and paste that is hard to
  understand, especially for all of the release editions in Vault Enterprise
* Branch check requirements for workflows are incredibly painful to use for
  workflows that are dynamic or change often. The required workflows have to be
  configured in Github settings by administrators. They would also prevent us
  from having simple docs PRs since required integration workflows always have
  to run to satisfy branch requirements.
* Doormat credentials requirements that are coming will require us to modify
  which event types trigger workflows. This changes those ahead of time since
  we're doing so much to build workflow. The only noticeable impact will be
  that the build workflow no longer runs on pushes to non-main or release
  branches. In order to test other branches it requires a workflow_dispatch
  from the Actions tab or a pull request.

Solutions

* Centralize the logic that determines build metadata and creates releasable
  Vault artifacts. Instead of cargo-culting logic multiple times in the build
  workflow and the Enos local modules, we now have a crt-builder script which
  determines build metadata and also handles building the UI, Vault, and the
  package bundle. There are make targets for all of the available sub-commands.
  Now what we use in the pipeline is the same thing as the local builder, and
  it can be executed locally by developers. The crt-builder script works in OSS
  and Enterprise so we will never have to deal with them being divergent or with
  special casing things in the build workflow.
* Refactor the bulk of the Vault building into a reusable workflow that we can
  call multiple times. This allows us to define Vault builds in a much simpler
  manner and makes resolving merge conflicts much easier.
* Rather than trying to maintain a list and manually configure the branch check
  requirements for build, we'll trigger a single workflow that uses the github
  event system to determine if the build workflow (all of the sub-workflows
  included) have passed. We'll then create branch restrictions on that single
  workflow down the line.

Signed-off-by: Ryan Cragun me@ryan.ec
2022-11-11 13:14:43 -07:00
Tom Proctor 63203751ff
Fix .go-version path in Makefile (#17798) 2022-11-03 14:04:53 +00:00
Christopher Swenson 19b3b8a7c6
Proposal: Remove debug symbols from build (#17678)
By adding the link flags `-s -w` we can reduce the Vault binary size
from 204 MB to 167 MB (about 18% reduction in size).

This removes the DWARF section of the binary.

i.e., before:

```
$ objdump --section-headers vault-debug

vault-debug:	file format mach-o arm64

Sections:
Idx Name             Size     VMA              Type
  0 __text           03a00340 0000000100001000 TEXT
  1 __symbol_stub1   00000618 0000000103a01340 TEXT
  2 __rodata         00c18088 0000000103a01960 DATA
  3 __rodata         015aee18 000000010461c000 DATA
  4 __typelink       0004616c 0000000105bcae20 DATA
  5 __itablink       0000eb68 0000000105c10fa0 DATA
  6 __gosymtab       00000000 0000000105c1fb08 DATA
  7 __gopclntab      02a5b8e0 0000000105c1fb20 DATA
  8 __go_buildinfo   00008c10 000000010867c000 DATA
  9 __nl_symbol_ptr  00000410 0000000108684c10 DATA
 10 __noptrdata      000fed00 0000000108685020 DATA
 11 __data           0004e1f0 0000000108783d20 DATA
 12 __bss            00052520 00000001087d1f20 BSS
 13 __noptrbss       000151b0 0000000108824440 BSS
 14 __zdebug_abbrev  00000129 000000010883c000 DATA, DEBUG
 15 __zdebug_line    00651374 000000010883c129 DATA, DEBUG
 16 __zdebug_frame   001e1de9 0000000108e8d49d DATA, DEBUG
 17 __debug_gdb_scri 00000043 000000010906f286 DATA, DEBUG
 18 __zdebug_info    00de2c09 000000010906f2c9 DATA, DEBUG
 19 __zdebug_loc     00a619ea 0000000109e51ed2 DATA, DEBUG
 20 __zdebug_ranges  001e94a6 000000010a8b38bc DATA, DEBUG
```

And after:

```
$ objdump --section-headers vault-no-debug

vault-no-debug:	file format mach-o arm64

Sections:
Idx Name            Size     VMA              Type
  0 __text          03a00340 0000000100001000 TEXT
  1 __symbol_stub1  00000618 0000000103a01340 TEXT
  2 __rodata        00c18088 0000000103a01960 DATA
  3 __rodata        015aee18 000000010461c000 DATA
  4 __typelink      0004616c 0000000105bcae20 DATA
  5 __itablink      0000eb68 0000000105c10fa0 DATA
  6 __gosymtab      00000000 0000000105c1fb08 DATA
  7 __gopclntab     02a5b8e0 0000000105c1fb20 DATA
  8 __go_buildinfo  00008c20 000000010867c000 DATA
  9 __nl_symbol_ptr 00000410 0000000108684c20 DATA
 10 __noptrdata     000fed00 0000000108685040 DATA
 11 __data          0004e1f0 0000000108783d40 DATA
 12 __bss           00052520 00000001087d1f40 BSS
 13 __noptrbss      000151b0 0000000108824460 BSS
```

The only side effect I have been able to find is that it is no longer
possible to use [delve](https://github.com/go-delve/delve) to run the
Vault binary.

Note, however, that running delve and other debuggers requires access
to the full source code, which isn't provided for the Enterprise, HSM,
etc. binaries, so it isn't possible to debug those anyway outside of
people who have the full source.

* panic traces
* `vault debug`
* error messages
* Despite what the documentation says, these flags do *not* delete the
function symbol table (so it is not the same as having a `strip`ped
binary).

It contains mappings between the compiled binary and functions,
paramters, and variables in the source code.

Using `llvm-dwarfdump`, it looks like:

```
0x011a6d85:   DW_TAG_subprogram
                DW_AT_name	("github.com/hashicorp/vault/api.(*replicationStateStore).recordState")
                DW_AT_low_pc	(0x0000000000a99300)
                DW_AT_high_pc	(0x0000000000a99419)
                DW_AT_frame_base	(DW_OP_call_frame_cfa)
                DW_AT_decl_file	("/home/swenson/vault/api/client.go")
                DW_AT_external	(0x01)

0x011a6de1:     DW_TAG_formal_parameter
                  DW_AT_name	("w")
                  DW_AT_variable_parameter	(0x00)
                  DW_AT_decl_line	(1735)
                  DW_AT_type	(0x00000000001e834a "github.com/hashicorp/vault/api.replicationStateStore *")
                  DW_AT_location	(0x009e832a:
                     [0x0000000000a99300, 0x0000000000a9933a): DW_OP_reg0 RAX
                     [0x0000000000a9933a, 0x0000000000a99419): DW_OP_call_frame_cfa)

0x011a6def:     DW_TAG_formal_parameter
                  DW_AT_name	("resp")
                  DW_AT_variable_parameter	(0x00)
                  DW_AT_decl_line	(1735)
                  DW_AT_type	(0x00000000001e82a2 "github.com/hashicorp/vault/api.Response *")
                  DW_AT_location	(0x009e8370:
                     [0x0000000000a99300, 0x0000000000a9933a): DW_OP_reg3 RBX
                     [0x0000000000a9933a, 0x0000000000a99419): DW_OP_fbreg +8)

0x011a6e00:     DW_TAG_variable
                  DW_AT_name	("newState")
                  DW_AT_decl_line	(1738)
                  DW_AT_type	(0x0000000000119f32 "string")
                  DW_AT_location	(0x009e83b7:
                     [0x0000000000a99385, 0x0000000000a99385): DW_OP_reg0 RAX, DW_OP_piece 0x8, DW_OP_piece 0x8
                     [0x0000000000a99385, 0x0000000000a993a4): DW_OP_reg0 RAX, DW_OP_piece 0x8, DW_OP_reg3 RBX, DW_OP_piece 0x8
                     [0x0000000000a993a4, 0x0000000000a993a7): DW_OP_piece 0x8, DW_OP_reg3 RBX, DW_OP_piece 0x8)
```

This says that the particular binary section is the function
`github.com/hashicorp/vault/api.(*replicationStateStore).recordState`,
from the file `/home/swenson/vault/api/client.go`, containing
the `w` parameter on line 1735 mapped to certain registers and memory,
the `resp` paramter on line 1735 mapped to certain reigsters and memory,
and the `newState` variable on line 1738, mapped to certain registers,
and memory.

It's really only useful for a debugger.

Anyone running the code in a debugger will need full access the source
code anyway, so presumably they will be able to run `make dev` and build
the version with the DWARF sections intact, and then run their debugger.
2022-11-02 10:47:13 -07:00
Nick Cabatoff f1c92200b2
Reduce number of places where go version is set (#17762)
Reduce go version references to two: .go_version and @executors.yml.
2022-11-01 15:37:13 -04:00
akshya96 1e189016e2
update protoc version to 3.21.7 oss (#17499)
* update protoc to 3.21.7

* adding changelog
2022-10-26 16:49:44 -07:00
Mike Palmiotto 9dc8744885
Update main to go 1.19.2 (#17437)
* Update go version to 1.19.2

This commit updates the default version of go to 1.19.2. This update
includes minor security fixes for archive/tar, net/http/httputil, and
regexp packages.

For more information on the release, see: https://go.dev/doc/devel/release#go1.19.2

* Update Docker versions in CI to 20.10.17

After updating Vault to go version 1.19.2, there were several SIGABRTs
in the vault tests. These were related to a missing `pthread_create`
syscall in Docker. Since CI was using a much older version of Docker,
the fix was to bump it to latest-1 (20.10.17).

While we're at it, add a note in the developer docs encouraging the use
of the latest Docker version.
2022-10-07 14:24:14 -04:00
Christopher Swenson b136a7ecd8
Add plugin version to GRPC interface (#17088)
Add plugin version to GRPC interface

Added a version interface in the sdk/logical so that it can be shared between all plugin types, and then wired it up to RunningVersion in the mounts, auth list, and database systems.

I've tested that this works with auth, database, and secrets plugin types, with the following logic to populate RunningVersion:

If a plugin has a PluginVersion() method implemented, then that is used
If not, and the plugin is built into the Vault binary, then the go.mod version is used
Otherwise, the it will be the empty string.
My apologies for the length of this PR.

* Placeholder backend should be external

We use a placeholder backend (previously a framework.Backend) before a
GRPC plugin is lazy-loaded. This makes us later think the plugin is a
builtin plugin.

So we added a `placeholderBackend` type that overrides the
`IsExternal()` method so that later we know that the plugin is external,
and don't give it a default builtin version.
2022-09-15 16:37:59 -07:00
Josh Black 14c8008181
Enforce a minimum version for protoc (#17122) 2022-09-13 19:46:35 -07:00
Hamid Ghaf 102f5f6832
node status as a module to be importable by HCP cloud (#17089) 2022-09-09 14:51:05 -04:00
akshya96 ab1e264f0b
updating go to 1.19.1 (#17053) 2022-09-07 16:13:33 -07:00
Hamid Ghaf c5df41e750
update to GO 1.18.5 (#16840)
* update to GO 1.18.5

* reverting a change

* feedback
2022-08-23 19:05:43 -04:00
Christopher Swenson 1068a1b528
Update Go to 1.18 (#16522)
Update Go to 1.18

From 1.17.12

1.18.5 was just released, but not all packages have been updated, so I
went with 1.18.4

Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
2022-08-02 10:36:12 -07:00
Christopher Swenson 81b702b918
Remove gox in favor of go build. (#16353)
Remove gox in favor of go build.

`gox` hasn't had a release to update it in many years, so is missing
support for many modern systems, like `darwin/arm64`.

In any case, we only use it for dev builds, where we don't even use
the ability of it to build for multiple platforms. Release builds use
`go build` now.

So, this switches to `go build` everywhere.

I pulled this down and tested it in Windows as well. (Side note: I
couldn't get `gox` to work in Windows, so couldn't build before this
change.)
2022-07-20 10:44:41 -07:00
Chris Capurso 3581811289
Update go to version 1.17.12 (#16336)
* update to go 1.17.12

* update changelog entry

* update readme
2022-07-18 16:28:47 -04:00
claire bontempo 28ada99c48
UI/VAULT-3645/Remove browserstack (#15997)
* remove browserstack
2022-06-15 14:50:44 -07:00
Steven Clark a97da32b4b
Update Go to 1.17.11 (#15818)
* Update Go to 1.17.11

 See https://go.dev/doc/devel/release#go1.17.minor for release notes
2022-06-06 13:18:24 -04:00
Chris Capurso e69f89c279
Add build date (#14957)
* add BuildDate to version base

* populate BuildDate with ldflags

* include BuildDate in FullVersionNumber

* add BuildDate to seal-status and associated status cmd

* extend core/versions entries to include BuildDate

* include BuildDate in version-history API and CLI

* fix version history tests

* fix sys status tests

* fix TestStatusFormat

* remove extraneous LD_FLAGS from build.sh

* add BuildDate to build.bat

* fix TestSysUnseal_Reset

* attempt to add build-date to release builds

* add branch to github build workflow

* add get-build-date to build-* job needs

* fix release build command vars

* add missing quote in release build command

* Revert "add branch to github build workflow"

This reverts commit b835699ecb7c2c632757fa5fe64b3d5f60d2a886.

* add changelog entry
2022-04-19 14:28:08 -04:00
Alexander Scheel 4a58bc8892
Update to Go 1.17.9 (#15043)
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
2022-04-14 14:41:05 -04:00
Hridoy Roy 7e0abe3c7e
Add Semgrep Rules to OSS (#14513)
* add semgrep yml

* add semgrep ci job

* remove replication semgrep rule in oss

* fix makefile

* add semgrep to ci

* upwind triple if in ui.go semgrep refactoring
2022-03-18 11:14:03 -07:00
Alvin Huang 40e24f3688
rename Dockerfile build-arg VERSION to PRODUCT_VERSION (#14369) 2022-03-10 12:59:30 +00:00
Mike Baum e03a8b43d5
[QTI-188] Add test-ember-enos Makefile target, update enos-test-ember script to accept additional paramters (#14403) 2022-03-08 10:58:28 -05:00
Nick Cabatoff 6a452bc3e6
Update to Go 1.17.7 (#14232) 2022-02-23 15:08:08 -05:00
Hridoy Roy ebf8e5308a
SSCT Tokens Feature [OSS] (#14109)
* port SSCT OSS

* port header hmac key to ent and generate token proto without make command

* remove extra nil check in request handling

* add changelog

* add comment to router.go

* change test var to use length constants

* remove local index is 0 check and extra defer which can be removed after use of ExternalID
2022-02-17 11:43:07 -08:00
John-Michael Faircloth 1cf74e1179
feature: multiplexing support for database plugins (#14033)
* feat: DB plugin multiplexing (#13734)

* WIP: start from main and get a plugin runner from core

* move MultiplexedClient map to plugin catalog
- call sys.NewPluginClient from PluginFactory
- updates to getPluginClient
- thread through isMetadataMode

* use go-plugin ClientProtocol interface
- call sys.NewPluginClient from dbplugin.NewPluginClient

* move PluginSets to dbplugin package
- export dbplugin HandshakeConfig
- small refactor of PluginCatalog.getPluginClient

* add removeMultiplexedClient; clean up on Close()
- call client.Kill from plugin catalog
- set rpcClient when muxed client exists

* add ID to dbplugin.DatabasePluginClient struct

* only create one plugin process per plugin type

* update NewPluginClient to return connection ID to sdk
- wrap grpc.ClientConn so we can inject the ID into context
- get ID from context on grpc server

* add v6 multiplexing  protocol version

* WIP: backwards compat for db plugins

* Ensure locking on plugin catalog access

- Create public GetPluginClient method for plugin catalog
- rename postgres db plugin

* use the New constructor for db plugins

* grpc server: use write lock for Close and rlock for CRUD

* cleanup MultiplexedClients on Close

* remove TODO

* fix multiplexing regression with grpc server connection

* cleanup grpc server instances on close

* embed ClientProtocol in Multiplexer interface

* use PluginClientConfig arg to make NewPluginClient plugin type agnostic

* create a new plugin process for non-muxed plugins

* feat: plugin multiplexing: handle plugin client cleanup (#13896)

* use closure for plugin client cleanup

* log and return errors; add comments

* move rpcClient wrapping to core for ID injection

* refactor core plugin client and sdk

* remove unused ID method

* refactor and only wrap clientConn on multiplexed plugins

* rename structs and do not export types

* Slight refactor of system view interface

* Revert "Slight refactor of system view interface"

This reverts commit 73d420e5cd2f0415e000c5a9284ea72a58016dd6.

* Revert "Revert "Slight refactor of system view interface""

This reverts commit f75527008a1db06d04a23e04c3059674be8adb5f.

* only provide pluginRunner arg to the internal newPluginClient method

* embed ClientProtocol in pluginClient and name logger

* Add back MLock support

* remove enableMlock arg from setupPluginCatalog

* rename plugin util interface to PluginClient

Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>

* feature: multiplexing: fix unit tests (#14007)

* fix grpc_server tests and add coverage

* update run_config tests

* add happy path test case for grpc_server ID from context

* update test helpers

* feat: multiplexing: handle v5 plugin compiled with new sdk

* add mux supported flag and increase test coverage

* set multiplexingSupport field in plugin server

* remove multiplexingSupport field in sdk

* revert postgres to non-multiplexed

* add comments on grpc server fields

* use pointer receiver on grpc server methods

* add changelog

* use pointer for grpcserver instance

* Use a gRPC server to determine if a plugin should be multiplexed

* Apply suggestions from code review

Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>

* add lock to removePluginClient

* add multiplexingSupport field to externalPlugin struct

* do not send nil to grpc MultiplexingSupport

* check err before logging

* handle locking scenario for cleanupFunc

* allow ServeConfigMultiplex to dispense v5 plugin

* reposition structs, add err check and comments

* add comment on locking for cleanupExternalPlugin

Co-authored-by: Brian Kassouf <bkassouf@hashicorp.com>
Co-authored-by: Brian Kassouf <briankassouf@users.noreply.github.com>
2022-02-17 08:50:33 -06:00
Michele Degges ce163e36bb
Fix for main: Add vault revision to --version cmd (#13428) 2021-12-15 11:38:20 -08:00
Meggie c0b962ecfa
Main go version bump (#13408)
* Go 1.17.2 -> 1.17.5
* Switching to cimg
2021-12-14 11:11:13 -05:00