ci: break config into separate files (#6849)

* ci: break config into separate files

* Untangle githooks

* githooks: fix whitespace

* .hooks/pre-commit: add ui -> lint-staged check

- We no longer require dependency on husky with this change.

* ui: remove husky dependency and config

- The previous commit obviates the need for it.
- We will now have to manage these hooks by hand, but this removes
  the conflict between husky-installed hooks and those in the .hooks dir.

* ui: update yarn.lock with husky removed

* .hooks/pre-commit: always use subshell + docs

- Always use subshell means we consistently exit from the
  same place which feels less complex.
- Docs are necessary for horrible bash like this I think...

* Makefile: remove old husky githooks

- Husky has installed a handler for every single git hook.
- This causes warnings on every git operation.
- Eventually we can remove this, but better not to confuse
  people with these messages for now.

* ci: fix go build tags

* Makefile: improve compatibility of rm call

- Looks like the xargs in Travis does something different to the one
  on my mac, this more verbose call should be safe everywhere.

* ci: fix make target names

* ci: fix test-ui invocation

* Makefile: simplify husky hook cleanup

* ci: more focussed readme
This commit is contained in:
Sam Salisbury 2019-06-11 15:55:53 +01:00 committed by Matthew Irish
parent fcf1b9c54e
commit e2e5e16ff2
18 changed files with 992 additions and 249 deletions

1
.circleci/.gitattributes vendored Normal file
View File

@ -0,0 +1 @@
config.yml linguist-generated

1
.circleci/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
.tmp/

47
.circleci/Makefile Normal file
View File

@ -0,0 +1,47 @@
.PHONY: default
default: ci-config
.PHONY: check-circleci-installed
check-circleci-installed:
@command -v circleci > /dev/null 2>&1 || { \
echo "Please install circleci-cli, see https://circleci.com/docs/2.0/local-cli/#installation"; \
exit 1; }
.PHONY: ci-config
# ci-config is just an alias for config.yml for now
ci-config: config.yml
CONFIG_SOURCE_DIR := config/
CONFIG_SOURCE := $(shell find config/) Makefile
OUT := config.yml
TMP := .tmp/config.yml.tmp
CONFIG_21 := .tmp/config.2.1.tmp
# Ensure the .tmp dir exists.
$(shell [ -d .tmp ] || mkdir .tmp)
define GEN_CONFIG
@circleci config pack $(CONFIG_SOURCE_DIR) > $(CONFIG_21)
@echo "### Generated by 'make ci-config' do not manually edit this file." > $@
@circleci config process $(CONFIG_21) >> $@
endef
$(OUT): $(CONFIG_SOURCE) check-circleci-installed
$(GEN_CONFIG)
@echo "$@ updated"
$(TMP): $(CONFIG_SOURCE) check-circleci-installed
$(GEN_CONFIG)
.PHONY: config-up-to-date
config-up-to-date: $(TMP) # Note this must not depend on $(OUT)!
@if diff config.yml $<; then \
echo "Generated $(OUT) is up to date!"; \
else \
echo "Generated $(OUT) is out of date, run make ci-config to update."; \
exit 1; \
fi
.PHONY: ci-verify
ci-verify: config-up-to-date
@circleci config validate config.yml

117
.circleci/README.md Normal file
View File

@ -0,0 +1,117 @@
# CircleCI config
This directory contains both the source code (under `./config/`)
and the generated single-file `config.yml`
which defines the CircleCI workflows for this project.
The Makefile in this directory generates the `./config.yml`
in CircleCI 2.0 syntax,
from the tree rooted at `./config/`,
which contains files in CircleCI 2.1 syntax.
CircleCI supports [generating a single config file from many],
using the `$ circleci config pack` command.
It also supports [expanding 2.1 syntax to 2.0 syntax]
using the `$ circleci config process` command.
[generating a single config file from many]: https://circleci.com/docs/2.0/local-cli/#packing-a-config
[expanding 2.1 syntax to 2.0 syntax]: https://circleci.com/docs/2.0/local-cli/#processing-a-config
## Prerequisites
You will need the [CircleCI CLI tool] installed and working,
at least version `0.1.5607`.
```
$ circleci version
0.1.5607+f705856
```
NOTE: It is recommended to [download this tool directly from GitHub Releases].
Do not install it using Homebrew, as this version cannot be easily updated.
It is also not recommended to pipe curl to bash (which CircleCI recommend) for security reasons!
[CircleCI CLI tool]: https://circleci.com/docs/2.0/local-cli/
[download this tool directly from GitHub Releases]: https://github.com/CircleCI-Public/circleci-cli/releases
## How to make changes
Before making changes, be sure to understand the layout
of the `./config/` file tree, as well as circleci 2.1 syntax.
See the [Syntax and layout] section below.
To update the config, you should edit, add or remove files
in the `./config/` directory,
and then run `make ci-config`.
If that's successful,
you should then commit every `*.yml` file in the tree rooted in this directory.
That is: you should commit both the source under `./config/`
and the generated file `./config.yml` at the same time, in the same commit.
Do not edit the `./config.yml` file directly, as you will lose your changes
next time `make ci-config` is run.
[Syntax and layout]: #syntax-and-layout
### Verifying `./config.yml`
To check whether or not the current `./config.yml` is up to date with the source,
and whether it is valid, run `$ make ci-verify`.
Note that `$ make ci-verify` should be run in CI,
as well as by a local git commit hook,
to ensure we never commit files that are invalid or out of date.
#### Example shell session
```sh
$ make ci-config
config.yml updated
$ git add -A . # The -A makes sure to include deletions/renames etc.
$ git commit -m "ci: blah blah blah"
Changes detected in .circleci/, running 'make -C .circleci ci-verify'
--> Generated config.yml is up to date!
--> Config file at config.yml is valid.
```
### Syntax and layout
It is important to understand the layout of the config directory.
Read the documentation on [packing a config] for a full understanding
of how multiple YAML files are merged by the circleci CLI tool.
[packing a config]: https://circleci.com/docs/2.0/local-cli/#packing-a-config
Here is an example file tree (with comments added afterwards):
```sh
$ tree .
.
├── Makefile
├── README.md # This file.
├── config # The source code for config.yml is rooted here.
│   ├── @config.yml # Files beginning with @ are treated specially by `circleci config pack`
│   ├── commands # Subdirectories of config become top-level keys.
│   │   └── go_test.yml # Filenames (minus .yml) become top-level keys under their parent (in this case "commands").
│ │ # The contents of go_test.yml therefore are placed at: .commands.go_test:
│   └── jobs # jobs also becomes a top-level key under config...
│   ├── build-go-dev.yml # ...and likewise filenames become keys under their parent.
│   ├── go-mod-download.yml
│   ├── install-ui-dependencies.yml
│   ├── test-go-race.yml
│   ├── test-go.yml
│   └── test-ui.yml
└── config.yml # The generated file in 2.0 syntax.
```
About those `@` files... Preceding a filename with `@`
indicates to `$ circleci config pack` that the contents of this YAML file
should be at the top-level, rather than underneath a key named after their filename.
This naming convention is unfortunate as it breaks autocompletion in bash,
but there we go.
### Why not just use YAML references?
YAML references only work within a single file,
this is because `circleci config pack` is not a text-level packer,
but rather stitches together the structures defined in each YAML
file according to certain rules.
Therefore it must parse each file separately,
and YAML references are handled by the parser.

417
.circleci/config.yml generated
View File

@ -1,70 +1,58 @@
### Generated by 'make ci-config' do not manually edit this file.
version: 2
references:
images:
go: &GOLANG_IMAGE golang:1.12.4-stretch # Pin Go to patch version (ex: 1.2.3)
node: &NODE_IMAGE node:10-stretch # Pin Node.js to major version (ex: 10)
environment: &ENVIRONMENT
CIRCLECI_CLI_VERSION: 0.1.5546 # Pin CircleCI CLI to patch version (ex: 1.2.3)
GO_VERSION: 1.12.4 # Pin Go to patch version (ex: 1.2.3)
GOTESTSUM_VERSION: 0.3.3 # Pin gotestsum to patch version (ex: 1.2.3)
cache:
go-sum: &GO_SUM_CACHE_KEY go-sum-v1-{{ checksum "go.sum" }}
yarn-lock: &YARN_LOCK_CACHE_KEY yarn-lock-v1-{{ checksum "ui/yarn.lock" }}
jobs:
install-ui-dependencies:
docker:
- image: *NODE_IMAGE
- image: node:10-stretch
working_directory: /src
steps:
- checkout
- restore_cache:
key: *YARN_LOCK_CACHE_KEY
key: yarn-lock-v1-{{ checksum "ui/yarn.lock" }}
- run:
name: Install UI dependencies
command: |
set -eux -o pipefail
cd ui
yarn install --ignore-optional
npm rebuild node-sass
name: Install UI dependencies
- save_cache:
key: *YARN_LOCK_CACHE_KEY
key: yarn-lock-v1-{{ checksum "ui/yarn.lock" }}
paths:
- ui/node_modules
go-mod-download:
docker:
- image: *GOLANG_IMAGE
- image: golang:1.12.4-stretch
working_directory: /src
steps:
- add_ssh_keys:
fingerprints:
- c6:96:98:82:dc:04:6c:39:dd:ac:83:05:e3:15:1c:98
- checkout
- restore_cache:
key: *GO_SUM_CACHE_KEY
key: go-sum-v1-{{ checksum "go.sum" }}
- run:
name: Download Go modules
command: go mod download
name: Download Go modules
- run:
name: Verify checksums of Go modules
command: go mod verify
name: Verify checksums of Go modules
- save_cache:
key: *GO_SUM_CACHE_KEY
key: go-sum-v1-{{ checksum "go.sum" }}
paths:
- /go/pkg/mod
build-go-dev:
docker:
- image: *GOLANG_IMAGE
- image: golang:1.12.4-stretch
working_directory: /src
steps:
- checkout
- restore_cache:
key: *GO_SUM_CACHE_KEY
key: go-sum-v1-{{ checksum "go.sum" }}
- attach_workspace:
at: .
- run:
name: Build dev binary
command: |
set -eux -o pipefail
@ -74,24 +62,23 @@ jobs:
# Build dev binary
make bootstrap dev
name: Build dev binary
- persist_to_workspace:
root: .
paths:
- bin
root: .
test-ui:
docker:
- image: *NODE_IMAGE
- image: node:10-stretch
working_directory: /src
resource_class: medium+
steps:
- checkout
- restore_cache:
key: *YARN_LOCK_CACHE_KEY
key: yarn-lock-v1-{{ checksum "ui/yarn.lock" }}
- attach_workspace:
at: .
- run:
name: Test UI
command: |
set -eux -o pipefail
@ -106,57 +93,33 @@ jobs:
rm -rf /var/lib/apt/lists/* /var/cache/apt/*
# Add ./bin to the PATH so vault binary can be run by Ember tests
export PATH="${PWD}"/bin:${PATH}
export PATH="${PWD}/bin:${PATH}"
# Run Ember tests
cd ui
mkdir -p test-results/qunit
yarn run test-oss
name: Test UI
- store_artifacts:
path: ui/test-results
- store_test_results:
path: ui/test-results
test-ui-browserstack:
docker:
- image: *NODE_IMAGE
steps:
- checkout
- restore_cache:
key: *YARN_LOCK_CACHE_KEY
- attach_workspace:
at: .
- run:
name: Run BrowserStack tests
command: |
set -eux -o pipefail
# Add ./bin to the PATH so vault binary can be run by Ember tests
export PATH="${PWD}"/bin:${PATH}
make test-ui-browserstack
test-go:
machine: true
environment:
<<: *ENVIRONMENT
GO_TAGS:
parallelism: 2
working_directory: ~/src
parallelism: 2
steps:
- checkout
- run:
name: Allow circleci user to restore Go modules cache
command: |
set -eux -o pipefail
sudo mkdir /go
sudo chown -R circleci:circleci /go
name: Allow circleci user to restore Go modules cache
- restore_cache:
key: *GO_SUM_CACHE_KEY
key: go-sum-v1-{{ checksum "go.sum" }}
- run:
name: Run Go tests
no_output_timeout: 20m
command: |
set -eux -o pipefail
@ -179,6 +142,7 @@ jobs:
package_names=$(go list \
-tags "${GO_TAGS}" \
./... \
| grep -v /integ \
| grep -v /vendor/ \
| sort \
| circleci tests split --split-by=timings --timings-type=classname)
@ -199,15 +163,91 @@ jobs:
-tags "${GO_TAGS}" \
-timeout=40m \
-parallel=20 \
\
${package_names}
name: Run Go tests
no_output_timeout: 20m
- store_artifacts:
path: test-results
- store_test_results:
path: test-results
environment:
- CIRCLECI_CLI_VERSION: 0.1.5546
- GO_TAGS: null
- GO_VERSION: 1.12.4
- GOTESTSUM_VERSION: 0.3.3
test-go-race:
machine: true
working_directory: ~/src
steps:
- checkout
- run:
command: |
set -eux -o pipefail
sudo mkdir /go
sudo chown -R circleci:circleci /go
name: Allow circleci user to restore Go modules cache
- restore_cache:
key: go-sum-v1-{{ checksum "go.sum" }}
- run:
command: |
set -eux -o pipefail
# Install Go
curl -sSLO "https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz"
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf "go${GO_VERSION}.linux-amd64.tar.gz"
rm -f "go${GO_VERSION}.linux-amd64.tar.gz"
export GOPATH=/go
export PATH="${PATH}:${GOPATH}/bin:/usr/local/go/bin"
# Install CircleCI CLI
curl -sSL \
"https://github.com/CircleCI-Public/circleci-cli/releases/download/v${CIRCLECI_CLI_VERSION}/circleci-cli_${CIRCLECI_CLI_VERSION}_linux_amd64.tar.gz" \
| sudo tar --overwrite -xz \
-C /usr/local/bin \
"circleci-cli_${CIRCLECI_CLI_VERSION}_linux_amd64/circleci"
# Split Go tests by prior test times
package_names=$(go list \
-tags "${GO_TAGS}" \
./... \
| grep -v /integ \
| grep -v /vendor/ \
| sort \
| circleci tests split --split-by=timings --timings-type=classname)
# Install gotestsum
curl -sSL "https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz" \
| sudo tar --overwrite -xz -C /usr/local/bin gotestsum
# Run tests
make prep
mkdir -p test-results/go-test
CGO_ENABLED= \
VAULT_ADDR= \
VAULT_TOKEN= \
VAULT_DEV_ROOT_TOKEN_ID= \
VAULT_ACC= \
gotestsum --format=short-verbose --junitfile test-results/go-test/results.xml -- \
-tags "${GO_TAGS}" \
-timeout=40m \
-parallel=20 \
-race \
${package_names}
name: Run Go tests
no_output_timeout: 20m
- store_artifacts:
path: test-results
- store_test_results:
path: test-results
environment:
- CIRCLECI_CLI_VERSION: 0.1.5546
- GO_TAGS: null
- GO_VERSION: 1.12.4
- GOTESTSUM_VERSION: 0.3.3
workflows:
version: 2
ci:
jobs:
- install-ui-dependencies
@ -222,7 +262,250 @@ workflows:
- test-go:
requires:
- build-go-dev
- test-ui-browserstack:
- test-go-race:
requires:
- install-ui-dependencies
- build-go-dev
version: 2
# Original config.yml file:
# commands:
# go_test:
# description: run go tests
# parameters:
# extra_flags:
# default: \"\"
# type: string
# steps:
# - run:
# command: |
# set -eux -o pipefail
#
# # Install Go
# curl -sSLO \"https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz\"
# sudo rm -rf /usr/local/go
# sudo tar -C /usr/local -xzf \"go${GO_VERSION}.linux-amd64.tar.gz\"
# rm -f \"go${GO_VERSION}.linux-amd64.tar.gz\"
# export GOPATH=/go
# export PATH=\"${PATH}:${GOPATH}/bin:/usr/local/go/bin\"
#
# # Install CircleCI CLI
# curl -sSL \\
# \"https://github.com/CircleCI-Public/circleci-cli/releases/download/v${CIRCLECI_CLI_VERSION}/circleci-cli_${CIRCLECI_CLI_VERSION}_linux_amd64.tar.gz\" \\
# | sudo tar --overwrite -xz \\
# -C /usr/local/bin \\
# \"circleci-cli_${CIRCLECI_CLI_VERSION}_linux_amd64/circleci\"
#
# # Split Go tests by prior test times
# package_names=$(go list \\
# -tags \"${GO_TAGS}\" \\
# ./... \\
# | grep -v /integ \\
# | grep -v /vendor/ \\
# | sort \\
# | circleci tests split --split-by=timings --timings-type=classname)
#
# # Install gotestsum
# curl -sSL \"https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz\" \\
# | sudo tar --overwrite -xz -C /usr/local/bin gotestsum
#
# # Run tests
# make prep
# mkdir -p test-results/go-test
# CGO_ENABLED= \\
# VAULT_ADDR= \\
# VAULT_TOKEN= \\
# VAULT_DEV_ROOT_TOKEN_ID= \\
# VAULT_ACC= \\
# gotestsum --format=short-verbose --junitfile test-results/go-test/results.xml -- \\
# -tags \"${GO_TAGS}\" \\
# -timeout=40m \\
# -parallel=20 \\
# << parameters.extra_flags >> \\
# ${package_names}
# name: Run Go tests
# no_output_timeout: 20m
# restore_go_cache:
# steps:
# - restore_cache:
# key: go-sum-v1-{{ checksum \"go.sum\" }}
# restore_yarn_cache:
# steps:
# - restore_cache:
# key: yarn-lock-v1-{{ checksum \"ui/yarn.lock\" }}
# save_go_cache:
# steps:
# - save_cache:
# key: go-sum-v1-{{ checksum \"go.sum\" }}
# paths:
# - /go/pkg/mod
# save_yarn_cache:
# steps:
# - save_cache:
# key: yarn-lock-v1-{{ checksum \"ui/yarn.lock\" }}
# paths:
# - ui/node_modules
# executors:
# go:
# docker:
# - image: golang:1.12.4-stretch
# working_directory: /src
# go-machine:
# environment:
# CIRCLECI_CLI_VERSION: 0.1.5546
# GO_TAGS: null
# GO_VERSION: 1.12.4
# GOTESTSUM_VERSION: 0.3.3
# machine: true
# working_directory: ~/src
# node:
# docker:
# - image: node:10-stretch
# working_directory: /src
# jobs:
# build-go-dev:
# executor: go
# steps:
# - checkout
# - restore_go_cache
# - attach_workspace:
# at: .
# - run:
# command: |
# set -eux -o pipefail
#
# # Move dev UI assets to expected location
# rm -rf ./pkg
# mkdir ./pkg
#
# # Build dev binary
# make bootstrap dev
# name: Build dev binary
# - persist_to_workspace:
# paths:
# - bin
# root: .
# go-mod-download:
# executor: go
# steps:
# - add_ssh_keys:
# fingerprints:
# - c6:96:98:82:dc:04:6c:39:dd:ac:83:05:e3:15:1c:98
# - checkout
# - restore_go_cache
# - run:
# command: go mod download
# name: Download Go modules
# - run:
# command: go mod verify
# name: Verify checksums of Go modules
# - save_go_cache
# install-ui-dependencies:
# executor: node
# steps:
# - checkout
# - restore_yarn_cache
# - run:
# command: |
# set -eux -o pipefail
#
# cd ui
# yarn install --ignore-optional
# npm rebuild node-sass
# name: Install UI dependencies
# - save_yarn_cache
# test-go:
# executor: go-machine
# parallelism: 2
# steps:
# - checkout
# - run:
# command: |
# set -eux -o pipefail
#
# sudo mkdir /go
# sudo chown -R circleci:circleci /go
# name: Allow circleci user to restore Go modules cache
# - restore_go_cache
# - go_test
# - store_artifacts:
# path: test-results
# - store_test_results:
# path: test-results
# test-go-race:
# executor: go-machine
# steps:
# - checkout
# - run:
# command: |
# set -eux -o pipefail
#
# sudo mkdir /go
# sudo chown -R circleci:circleci /go
# name: Allow circleci user to restore Go modules cache
# - restore_go_cache
# - go_test:
# extra_flags: -race
# - store_artifacts:
# path: test-results
# - store_test_results:
# path: test-results
# test-ui:
# executor: node
# resource_class: medium+
# steps:
# - checkout
# - restore_yarn_cache
# - attach_workspace:
# at: .
# - run:
# command: |
# set -eux -o pipefail
#
# # Install Chrome
# wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub \\
# | apt-key add -
# echo \"deb http://dl.google.com/linux/chrome/deb/ stable main\" \\
# | tee /etc/apt/sources.list.d/google-chrome.list
# apt-get update
# apt-get -y install google-chrome-stable
# rm /etc/apt/sources.list.d/google-chrome.list
# rm -rf /var/lib/apt/lists/* /var/cache/apt/*
#
# # Add ./bin to the PATH so vault binary can be run by Ember tests
# export PATH=\"${PWD}/bin:${PATH}\"
#
# # Run Ember tests
# cd ui
# mkdir -p test-results/qunit
# yarn run test-oss
# name: Test UI
# - store_artifacts:
# path: ui/test-results
# - store_test_results:
# path: ui/test-results
# references:
# cache:
# go-sum: go-sum-v1-{{ checksum \"go.sum\" }}
# yarn-lock: yarn-lock-v1-{{ checksum \"ui/yarn.lock\" }}
# images:
# go: golang:1.12.4-stretch
# node: node:10-stretch
# version: 2.1
# workflows:
# ci:
# jobs:
# - install-ui-dependencies
# - go-mod-download
# - build-go-dev:
# requires:
# - go-mod-download
# - test-ui:
# requires:
# - install-ui-dependencies
# - build-go-dev
# - test-go:
# requires:
# - build-go-dev
# - test-go-race:
# requires:
# - build-go-dev

View File

@ -0,0 +1,53 @@
---
version: 2.1
references:
images:
go: &GOLANG_IMAGE golang:1.12.4-stretch # Pin Go to patch version (ex: 1.2.3)
node: &NODE_IMAGE node:10-stretch # Pin Node.js to major version (ex: 10)
cache:
go-sum: &GO_SUM_CACHE_KEY go-sum-v1-{{ checksum "go.sum" }}
yarn-lock: &YARN_LOCK_CACHE_KEY yarn-lock-v1-{{ checksum "ui/yarn.lock" }}
# more commands defined in commands/
commands:
restore_yarn_cache:
steps:
- restore_cache:
key: *YARN_LOCK_CACHE_KEY
save_yarn_cache:
steps:
- save_cache:
key: *YARN_LOCK_CACHE_KEY
paths:
- ui/node_modules
restore_go_cache:
steps:
- restore_cache:
key: *GO_SUM_CACHE_KEY
save_go_cache:
steps:
- save_cache:
key: *GO_SUM_CACHE_KEY
paths:
- /go/pkg/mod
executors:
go:
docker:
- image: *GOLANG_IMAGE
working_directory: /src
go-machine:
machine: true
environment:
CIRCLECI_CLI_VERSION: 0.1.5546 # Pin CircleCI CLI to patch version (ex: 1.2.3)
GO_VERSION: 1.12.4 # Pin Go to patch version (ex: 1.2.3)
GOTESTSUM_VERSION: 0.3.3 # Pin gotestsum to patch version (ex: 1.2.3)
GO_TAGS:
working_directory: ~/src
node:
docker:
- image: *NODE_IMAGE
working_directory: /src

View File

@ -0,0 +1,55 @@
description: run go tests
parameters:
extra_flags:
type: string
default: ""
steps:
- run:
name: Run Go tests
no_output_timeout: 20m
command: |
set -eux -o pipefail
# Install Go
curl -sSLO "https://dl.google.com/go/go${GO_VERSION}.linux-amd64.tar.gz"
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf "go${GO_VERSION}.linux-amd64.tar.gz"
rm -f "go${GO_VERSION}.linux-amd64.tar.gz"
export GOPATH=/go
export PATH="${PATH}:${GOPATH}/bin:/usr/local/go/bin"
# Install CircleCI CLI
curl -sSL \
"https://github.com/CircleCI-Public/circleci-cli/releases/download/v${CIRCLECI_CLI_VERSION}/circleci-cli_${CIRCLECI_CLI_VERSION}_linux_amd64.tar.gz" \
| sudo tar --overwrite -xz \
-C /usr/local/bin \
"circleci-cli_${CIRCLECI_CLI_VERSION}_linux_amd64/circleci"
# Split Go tests by prior test times
package_names=$(go list \
-tags "${GO_TAGS}" \
./... \
| grep -v /integ \
| grep -v /vendor/ \
| sort \
| circleci tests split --split-by=timings --timings-type=classname)
# Install gotestsum
curl -sSL "https://github.com/gotestyourself/gotestsum/releases/download/v${GOTESTSUM_VERSION}/gotestsum_${GOTESTSUM_VERSION}_linux_amd64.tar.gz" \
| sudo tar --overwrite -xz -C /usr/local/bin gotestsum
# Run tests
make prep
mkdir -p test-results/go-test
CGO_ENABLED= \
VAULT_ADDR= \
VAULT_TOKEN= \
VAULT_DEV_ROOT_TOKEN_ID= \
VAULT_ACC= \
gotestsum --format=short-verbose --junitfile test-results/go-test/results.xml -- \
-tags "${GO_TAGS}" \
-timeout=40m \
-parallel=20 \
<< parameters.extra_flags >> \
${package_names}

View File

@ -0,0 +1,21 @@
executor: go
steps:
- checkout
- restore_go_cache
- attach_workspace:
at: .
- run:
name: Build dev binary
command: |
set -eux -o pipefail
# Move dev UI assets to expected location
rm -rf ./pkg
mkdir ./pkg
# Build dev binary
make bootstrap dev
- persist_to_workspace:
root: .
paths:
- bin

View File

@ -0,0 +1,15 @@
executor: go
steps:
- add_ssh_keys:
fingerprints:
# "CircleCI SSH Checkout" SSH key associated with hashicorp-ci GitHub user
- "c6:96:98:82:dc:04:6c:39:dd:ac:83:05:e3:15:1c:98"
- checkout
- restore_go_cache
- run:
name: Download Go modules
command: go mod download
- run:
name: Verify checksums of Go modules
command: go mod verify
- save_go_cache

View File

@ -0,0 +1,13 @@
executor: node
steps:
- checkout
- restore_yarn_cache
- run:
name: Install UI dependencies
command: |
set -eux -o pipefail
cd ui
yarn install --ignore-optional
npm rebuild node-sass
- save_yarn_cache

View File

@ -0,0 +1,17 @@
executor: go-machine
steps:
- checkout
- run:
name: Allow circleci user to restore Go modules cache
command: |
set -eux -o pipefail
sudo mkdir /go
sudo chown -R circleci:circleci /go
- restore_go_cache
- go_test:
extra_flags: "-race"
- store_artifacts:
path: test-results
- store_test_results:
path: test-results

View File

@ -0,0 +1,17 @@
executor: go-machine
parallelism: 2
steps:
- checkout
- run:
name: Allow circleci user to restore Go modules cache
command: |
set -eux -o pipefail
sudo mkdir /go
sudo chown -R circleci:circleci /go
- restore_go_cache
- go_test
- store_artifacts:
path: test-results
- store_test_results:
path: test-results

View File

@ -0,0 +1,33 @@
executor: node
resource_class: medium+
steps:
- checkout
- restore_yarn_cache
- attach_workspace:
at: .
- run:
name: Test UI
command: |
set -eux -o pipefail
# Install Chrome
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub \
| apt-key add -
echo "deb http://dl.google.com/linux/chrome/deb/ stable main" \
| tee /etc/apt/sources.list.d/google-chrome.list
apt-get update
apt-get -y install google-chrome-stable
rm /etc/apt/sources.list.d/google-chrome.list
rm -rf /var/lib/apt/lists/* /var/cache/apt/*
# Add ./bin to the PATH so vault binary can be run by Ember tests
export PATH="${PWD}/bin:${PATH}"
# Run Ember tests
cd ui
mkdir -p test-results/qunit
yarn run test-oss
- store_artifacts:
path: ui/test-results
- store_test_results:
path: ui/test-results

View File

@ -0,0 +1,16 @@
jobs:
- install-ui-dependencies
- go-mod-download
- build-go-dev:
requires:
- go-mod-download
- test-ui:
requires:
- install-ui-dependencies
- build-go-dev
- test-go:
requires:
- build-go-dev
- test-go-race:
requires:
- build-go-dev

104
.hooks/pre-commit Executable file
View File

@ -0,0 +1,104 @@
#!/usr/bin/env bash
# READ THIS BEFORE MAKING CHANGES:
#
# If you want to add a new pre-commit check, here are the rules:
#
# 1. Create a bash function for your check (see e.g. ui_lint below).
# NOTE: Each function will be called in a sub-shell so you can freely
# change directory without worrying about interference.
# 2. Add the name of the function to the CHECKS variable.
# 3. If no changes relevant to your new check are staged, then
# do not output anything at all - this would be annoying noise.
# In this case, call 'return 0' from your check function to return
# early without blocking the commit.
# 4. If any non-trivial check-specific thing has to be invoked,
# then output '==> [check description]' as the first line of
# output. Each sub-check should output '--> [subcheck description]'
# after it has run, indicating success or failure.
# 5. Call 'block [reason]' to block the commit. This ensures the last
# line of output calls out that the commit was blocked - which may not
# be obvious from random error messages generated in 4.
#
# At the moment, there are no automated tests for this hook, so please run it
# locally to check you have not broken anything - breaking this will interfere
# with other peoples' workflows significantly, so be sure, check everything twice.
set -euo pipefail
# Call block to block the commit with a message.
block() {
echo "$@"
echo "Commit blocked - see errors above."
exit 1
}
# Add all check functions to this space separated list.
# They are executed in this order (see end of file).
CHECKS="ui_lint circleci_verify"
# Run ui linter if changes in that dir detected.
ui_lint() {
local DIR=ui LINTER=node_modules/.bin/lint-staged
# Silently succeed if no changes staged for $DIR
if git diff --name-only --cached --exit-code -- $DIR/; then
return 0
fi
# Silently succeed if the linter has not been installed.
# We assume that if you're doing UI dev, you will have installed the linter
# by running yarn.
if [ ! -x $DIR/$LINTER ]; then
return 0
fi
echo "==> Changes detected in $DIR/: Running linter..."
# Run the linter from the UI dir.
cd $DIR
$LINTER || block "UI lint failed"
}
# Check .circleci/config.yml is up to date and valid, and that all changes are
# included together in this commit.
circleci_verify() {
# Change to the root dir of the repo.
cd "$(git rev-parse --show-toplevel)"
# Fail early if we accidentally used '.yaml' instead of '.yml'
if ! git diff --name-only --cached --exit-code -- '.circleci/***.yaml'; then
# This is just for consistency, as I keep making this mistake - Sam.
block "ERROR: File(s) with .yaml extension detected. Please rename them .yml instead."
fi
# Succeed early if no changes to yml files in .circleci/ are currently staged.
# make ci-verify is slow so we really don't want to run it unnecessarily.
if git diff --name-only --cached --exit-code -- '.circleci/***.yml'; then
return 0
fi
# Make sure to add no explicit output before this line, as it would just be noise
# for those making non-circleci changes.
echo "==> Verifying config changes in .circleci/"
echo "--> OK: All files are .yml not .yaml"
# Ensure commit includes _all_ files in .circleci/
# So not only are the files up to date, but we are also committing them in one go.
if ! git diff --name-only --exit-code -- '.circleci/***.yml'; then
echo "ERROR: Some .yml diffs in .circleci/ are staged, others not."
block "Please commit the entire .circleci/ directory together, or omit it altogether."
fi
echo "--> OK: All .yml files in .circleci are staged."
if ! make -C .circleci ci-verify; then
block "ERROR: make ci-verify failed"
fi
echo "--> OK: make ci-verify succeeded."
}
for CHECK in $CHECKS; do
# Force each check into a subshell to avoid crosstalk.
( $CHECK ) || exit $?
done

View File

@ -102,8 +102,15 @@ vet:
prep: fmtcheck
@sh -c "'$(CURDIR)/scripts/goversioncheck.sh' '$(GO_VERSION_MIN)'"
@go generate $(go list ./... | grep -v /vendor/)
@# Remove old (now broken) husky git hooks.
@[ ! -d .git/hooks ] || grep -l '^# husky$$' .git/hooks/* | xargs rm -f
@if [ -d .git/hooks ]; then cp .hooks/* .git/hooks/; fi
ci-config:
@$(MAKE) -C .circleci
ci-verify:
@$(MAKE) -C .circleci ci-verify
# bootstrap the build by downloading additional tools
bootstrap:
@for tool in $(EXTERNAL_TOOLS) ; do \

View File

@ -139,7 +139,6 @@
"@storybook/ember": "^5.0.5",
"@storybook/ember-cli-storybook": "meirish/ember-cli-storybook#6bd58326d8c21e986d390b541ae5e49089d61b93",
"babel-loader": "^8.0.5",
"husky": "^1.1.3",
"jsdoc-to-markdown": "^4.0.1",
"lint-staged": "^8.0.4"
},
@ -154,10 +153,5 @@
"lib/replication"
]
},
"husky": {
"hooks": {
"pre-commit": "lint-staged"
}
},
"dependencies": {}
}

View File

@ -5479,11 +5479,6 @@ ci-info@^1.3.0, ci-info@^1.4.0:
resolved "https://registry.yarnpkg.com/ci-info/-/ci-info-1.5.0.tgz#38327c69e98dab18487744b84e5d6e841a09a1a7"
integrity sha512-Bx/xWOzip4whERIvC97aIHjWCa8FxEn0ezng0oVn4kma6p+90Fbs3bTcJw6ZL0da2EPHydxsXJPZxNUv5oWb1Q==
ci-info@^1.5.0:
version "1.6.0"
resolved "https://registry.yarnpkg.com/ci-info/-/ci-info-1.6.0.tgz#2ca20dbb9ceb32d4524a683303313f0304b1e497"
integrity sha512-vsGdkwSCDpWmP80ncATX7iea5DWQemg1UgCW5J8tqjU3lYw4FBYuj89J0CTVomA7BEfvSZd84GmHko+MxFQU2A==
cidr-regex@^2.0.8:
version "2.0.9"
resolved "https://registry.yarnpkg.com/cidr-regex/-/cidr-regex-2.0.9.tgz#9c17bb2b18e15af07f7d0c3b994b961d687ed1c9"
@ -6194,7 +6189,7 @@ cosmiconfig@^5.0.1:
js-yaml "^3.9.0"
parse-json "^4.0.0"
cosmiconfig@^5.0.2, cosmiconfig@^5.0.6:
cosmiconfig@^5.0.2:
version "5.0.7"
resolved "https://registry.yarnpkg.com/cosmiconfig/-/cosmiconfig-5.0.7.tgz#39826b292ee0d78eda137dfa3173bd1c21a43b04"
integrity sha512-PcLqxTKiDmNT6pSpy4N6KtuPwb53W+2tzNvwOZw0WH9N6O0vLIBq0x8aj8Oj75ere4YcGi48bDFCL+3fRJdlNA==
@ -8578,19 +8573,6 @@ execa@^0.7.0:
signal-exit "^3.0.0"
strip-eof "^1.0.0"
execa@^0.9.0:
version "0.9.0"
resolved "https://registry.yarnpkg.com/execa/-/execa-0.9.0.tgz#adb7ce62cf985071f60580deb4a88b9e34712d01"
integrity sha512-BbUMBiX4hqiHZUA5+JujIjNb6TyAlp2D5KLheMjMluwOuzcnylDL4AxZYLLn1n2AGB49eSWwyKvvEQoRpnAtmA==
dependencies:
cross-spawn "^5.0.1"
get-stream "^3.0.0"
is-stream "^1.1.0"
npm-run-path "^2.0.0"
p-finally "^1.0.0"
signal-exit "^3.0.0"
strip-eof "^1.0.0"
execa@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/execa/-/execa-1.0.0.tgz#c6236a5bb4df6d6f15e88e7f017798216749ddd8"
@ -10242,22 +10224,6 @@ humanize-ms@^1.2.1:
dependencies:
ms "^2.0.0"
husky@^1.1.3:
version "1.1.3"
resolved "https://registry.yarnpkg.com/husky/-/husky-1.1.3.tgz#3ccfdb4d7332896bf7cd0e618c6fb8be09d9de4b"
integrity sha512-6uc48B/A2Mqi65yeg37d/TPcTb0bZ1GTkMYOM0nXLOPuPaTRhXCeee80/noOrbavWd12x72Tusja7GJ5rzvV6g==
dependencies:
cosmiconfig "^5.0.6"
execa "^0.9.0"
find-up "^3.0.0"
get-stdin "^6.0.0"
is-ci "^1.2.1"
pkg-dir "^3.0.0"
please-upgrade-node "^3.1.1"
read-pkg "^4.0.1"
run-node "^1.0.0"
slash "^2.0.0"
iconv-lite@0.4.19:
version "0.4.19"
resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.19.tgz#f7468f60135f5e5dad3399c0a81be9a1603a082b"
@ -10676,13 +10642,6 @@ is-ci@^1.0.10:
dependencies:
ci-info "^1.3.0"
is-ci@^1.2.1:
version "1.2.1"
resolved "https://registry.yarnpkg.com/is-ci/-/is-ci-1.2.1.tgz#e3779c8ee17fccf428488f6e281187f2e632841c"
integrity sha512-s6tfsaQaQi3JNciBH6shVqEDvhGut0SUXr31ag8Pd8BBbVVlcGfWhpPmEOoM6RJ5TFhbypvf5yyRw/VXW1IiWg==
dependencies:
ci-info "^1.5.0"
is-cidr@^2.0.6:
version "2.0.6"
resolved "https://registry.yarnpkg.com/is-cidr/-/is-cidr-2.0.6.tgz#4b01c9693d8e18399dacd18a4f3d60ea5871ac60"
@ -14388,7 +14347,7 @@ pkg-up@2.0.0, pkg-up@^2.0.0:
dependencies:
find-up "^2.1.0"
please-upgrade-node@^3.0.2, please-upgrade-node@^3.1.1:
please-upgrade-node@^3.0.2:
version "3.1.1"
resolved "https://registry.yarnpkg.com/please-upgrade-node/-/please-upgrade-node-3.1.1.tgz#ed320051dfcc5024fae696712c8288993595e8ac"
integrity sha512-KY1uHnQ2NlQHqIJQpnh/i54rKkuxCEBx+voJIS/Mvb+L2iYd2NMotwduhKTMjfC1uKoX3VXOxLjIYG66dfJTVQ==
@ -15344,7 +15303,7 @@ read-pkg@^3.0.0:
normalize-package-data "^2.3.2"
path-type "^3.0.0"
read-pkg@^4.0.0, read-pkg@^4.0.1:
read-pkg@^4.0.0:
version "4.0.1"
resolved "https://registry.yarnpkg.com/read-pkg/-/read-pkg-4.0.1.tgz#963625378f3e1c4d48c85872b5a6ec7d5d093237"
integrity sha1-ljYlN48+HE1IyFhytabsfV0JMjc=
@ -16031,11 +15990,6 @@ run-async@^2.2.0:
dependencies:
is-promise "^2.1.0"
run-node@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/run-node/-/run-node-1.0.0.tgz#46b50b946a2aa2d4947ae1d886e9856fd9cabe5e"
integrity sha512-kc120TBlQ3mih1LSzdAJXo4xn/GWS2ec0l3S+syHDXP9uRr0JAT8Qd3mdMuyjqCzeZktgP3try92cEgf9Nks8A==
run-queue@^1.0.0, run-queue@^1.0.3:
version "1.0.3"
resolved "https://registry.yarnpkg.com/run-queue/-/run-queue-1.0.3.tgz#e848396f057d223f24386924618e25694161ec47"
@ -16511,11 +16465,6 @@ slash@^1.0.0:
resolved "https://registry.yarnpkg.com/slash/-/slash-1.0.0.tgz#c41f2f6c39fc16d1cd17ad4b5d896114ae470d55"
integrity sha1-xB8vbDn8FtHNF61LXYlhFK5HDVU=
slash@^2.0.0:
version "2.0.0"
resolved "https://registry.yarnpkg.com/slash/-/slash-2.0.0.tgz#de552851a1759df3a8f206535442f5ec4ddeab44"
integrity sha512-ZYKh3Wh2z1PpEXWr0MpSBZ0V6mZHAQfYevttO11c51CaWjGTaadiKZ+wVt1PbMlDV5qhMFslpZCemhwOK7C89A==
slice-ansi@0.0.4:
version "0.0.4"
resolved "https://registry.yarnpkg.com/slice-ansi/-/slice-ansi-0.0.4.tgz#edbf8903f66f7ce2f8eafd6ceed65e264c831b35"