open-consul/.circleci/config.yml

1182 lines
38 KiB
YAML
Raw Normal View History

2019-03-08 20:33:03 +00:00
---
version: 2.1
2019-01-28 16:18:30 +00:00
2021-01-14 04:30:00 +00:00
parameters:
commit:
type: string
default: ""
description: "Commit to run load tests against"
trigger-load-test:
type: boolean
default: false
description: "Boolean whether to run the load test workflow"
2019-03-08 20:33:03 +00:00
references:
images:
# When updating the Go version, remember to also update the versions in the
# workflows section for go-test-lib jobs.
2021-10-20 18:37:29 +00:00
go: &GOLANG_IMAGE docker.mirror.hashicorp.services/circleci/golang:1.17.2
2021-05-18 15:30:19 +00:00
ember: &EMBER_IMAGE docker.mirror.hashicorp.services/circleci/node:14-browsers
2019-03-08 20:33:03 +00:00
paths:
test-results: &TEST_RESULTS_DIR /tmp/test-results
2019-01-28 16:18:30 +00:00
2019-03-08 20:33:03 +00:00
cache:
ui: Adds initial CRUD for partitions (#11188) * Add `is` and `test` helpers in a similar vein to `can` Adds 2 new helpers in a similar vein to ember-cans can: - `is` allows you to use vocab/phrases such as (is "something model") which calls isSomething() on the models ability. - `test` allows you to use vocab/phrases such as (test "is something model") or (test "can something model")which calls isSomething() / canSomething() on the models ability. Mostly using the is helper and the can helper. It's basically the is/can helper combined. * Adds TextInput component + related modifiers/helpers/machines/services (#11189) Adds a few new components/modifiers/helpers to aid building forms. - state-chart helper, used in lieu of a more generic approach for requiring our statecharts. - A few modifications to our existing disabled modifier. - A new 'validation' modifier, a super small form validation approach built to make use of state charts (optionally). Eventually we should be able to replace our current validation approach (ember-changeset-validations + extra deps) with this. - A new TextInput component, which is the first of our new components specifically to make it easy to build forms with validations. This is still a WIP, I left some comments in pointing out where this one would be progressed, but as we don't need the planned functionality yet, I left it where it was. All of this will be fleshed out more at a later date. Documentation is included for all of ^ * ui: Adds initial CRUD for partitions (#11190) Adds basic CRUD support for partitions. Engineering-wise probably the biggest takeaway here is that we needed to write very little javascript code to add this entire feature, and the little javascript we did need to write was very straightforwards. Everything is pretty much just HTML. Another note to make is that both ember-changeset and ember-data (model layer things) are now completely abstracted away from the view layer of the application. New components: - Consul::Partition::Form - Consul::Partition::List - Consul::Partition::Notifications - Consul::Partition::SearchBar - Consul::Partition::Selector See additional documentation here for more details New Route templates: - index.hbs partition listing/searching/filtering - edit.hbs partition editing and creation Additionally: There is some additional debug work here for better observability and to prevent any errors regarding our href-to usage when a dc is not available in our documentation site. Our softDelete functionality has been DRYed out a little to be used across two repos. isLinkable was removed from our ListCollection component for lists like upstream and service listing, and instead use our new is helper from within the ListCollection, meaning we've added a few more lighterweight templateOnly components. * ui: Exclude all debug-like files from the build (#11211) This PR adds **/*-debug.* to our test/prod excluded files (realised I needed to add test-support.js also so added that here as its more or less the same thing). Conditionally juggling ES6 static imports (specifically debug ones) for this was also getting a little hairy, so I moved it all to use the same approach as our conditional routes. All in all it brings the vendor build back down to ~430kb gzipped.
2021-10-08 15:29:30 +00:00
yarn: &YARN_CACHE_KEY consul-ui-v5-{{ checksum "ui/yarn.lock" }}
2019-03-08 20:33:03 +00:00
rubygem: &RUBYGEM_CACHE_KEY static-site-gems-v1-{{ checksum "Gemfile.lock" }}
environment: &ENVIRONMENT
TEST_RESULTS_DIR: *TEST_RESULTS_DIR
EMAIL: noreply@hashicorp.com
GIT_AUTHOR_NAME: circleci-consul
GIT_COMMITTER_NAME: circleci-consul
S3_ARTIFACT_BUCKET: consul-dev-artifacts-v2
BASH_ENV: .circleci/bash_env.sh
VAULT_BINARY_VERSION: 1.2.2
2019-01-28 16:18:30 +00:00
steps:
install-gotestsum: &install-gotestsum
name: install gotestsum
environment:
GOTESTSUM_RELEASE: 1.6.4
command: |
url=https://github.com/gotestyourself/gotestsum/releases/download
curl -sSL "${url}/v${GOTESTSUM_RELEASE}/gotestsum_${GOTESTSUM_RELEASE}_linux_amd64.tar.gz" | \
sudo tar -xz --overwrite -C /usr/local/bin gotestsum
get-aws-cli: &get-aws-cli
run:
name: download and install AWS CLI
command: |
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
echo -e "${AWS_CLI_GPG_KEY}" | gpg --import
curl -o awscliv2.sig https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip.sig
gpg --verify awscliv2.sig awscliv2.zip
unzip awscliv2.zip
sudo ./aws/install
# This step MUST be at the end of any set of steps due to the 'when' condition
notify-slack-failure: &notify-slack-failure
name: notify-slack-failure
when: on_fail
command: |
if [[ $CIRCLE_BRANCH == "main" ]]; then
CIRCLE_ENDPOINT="https://app.circleci.com/pipelines/github/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?branch=${CIRCLE_BRANCH}"
GITHUB_ENDPOINT="https://github.com/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/commit/${CIRCLE_SHA1}"
COMMIT_MESSAGE=$(git log -1 --pretty=%B | head -n1)
SHORT_REF=$(git rev-parse --short "${CIRCLE_SHA1}")
curl -X POST -H 'Content-type: application/json' \
--data \
"{ \
\"attachments\": [ \
{ \
\"fallback\": \"CircleCI job failed!\", \
\"text\": \"❌ Failed: \`${CIRCLE_USERNAME}\`'s <${CIRCLE_BUILD_URL}|${CIRCLE_STAGE}> job failed for commit <${GITHUB_ENDPOINT}|${SHORT_REF}> on \`${CIRCLE_BRANCH}\`!\n\n- <${COMMIT_MESSAGE}\", \
\"footer\": \"${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}\", \
\"ts\": \"$(date +%s)\", \
\"color\": \"danger\" \
} \
] \
}" "${FEED_CONSUL_GH_URL}"
else
echo "Not posting slack failure notifications for non-main branch"
fi
commands:
assume-role:
description: "Assume role to an ARN"
parameters:
access-key:
type: env_var_name
default: AWS_ACCESS_KEY_ID
secret-key:
type: env_var_name
default: AWS_SECRET_ACCESS_KEY
role-arn:
type: env_var_name
default: ROLE_ARN
steps:
- run: |
export AWS_ACCESS_KEY_ID="${<< parameters.access-key >>}"
export AWS_SECRET_ACCESS_KEY="${<< parameters.secret-key >>}"
export ROLE_ARN="${<< parameters.role-arn >>}"
# assume role has duration of 15 min (the minimum allowed)
CREDENTIALS="$(aws sts assume-role --duration-seconds 900 --role-arn ${ROLE_ARN} --role-session-name build-${CIRCLE_SHA1} | jq '.Credentials')"
echo "export AWS_ACCESS_KEY_ID=$(echo $CREDENTIALS | jq -r '.AccessKeyId')" >> $BASH_ENV
echo "export AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIALS | jq -r '.SecretAccessKey')" >> $BASH_ENV
echo "export AWS_SESSION_TOKEN=$(echo $CREDENTIALS | jq -r '.SessionToken')" >> $BASH_ENV
jobs:
# lint consul tests
lint-consul-retry:
docker:
- image: *GOLANG_IMAGE
steps:
- checkout
- run: go get -u github.com/hashicorp/lint-consul-retry && lint-consul-retry
- run: *notify-slack-failure
lint:
description: "Run golangci-lint"
parameters:
go-arch:
type: string
default: ""
docker:
2019-05-17 15:42:56 +00:00
- image: *GOLANG_IMAGE
environment:
GOTAGS: "" # No tags for OSS but there are for enterprise
GOARCH: "<<parameters.go-arch>>"
steps:
2019-05-17 15:42:56 +00:00
- checkout
- run: go env
2019-05-17 15:42:56 +00:00
- run:
name: Install golangci-lint
2019-05-17 15:42:56 +00:00
command: |
download=https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh
wget -O- -q $download | sh -x -s -- -d -b /go/bin/ v1.40.1
- run: go mod download
2019-05-17 15:42:56 +00:00
- run:
name: lint
command: &lintcmd |
golangci-lint run --build-tags="$GOTAGS" -v --concurrency 2
- run:
name: lint api
working_directory: api
command: *lintcmd
- run:
name: lint sdk
working_directory: sdk
command: *lintcmd
- run: *notify-slack-failure
2019-05-17 15:42:56 +00:00
check-go-mod:
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
steps:
- checkout
- run: go mod tidy
- run: |
2021-01-22 16:36:35 +00:00
if [[ -n $(git status -s) ]]; then
echo "Git directory has changes"
git status -s
exit 1
fi
- run: *notify-slack-failure
check-generated-protobuf:
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
steps:
- checkout
- run:
name: Install protobuf
command: |
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.12.3/protoc-3.12.3-linux-x86_64.zip
sudo unzip -d /usr/local protoc-*.zip
sudo chmod +x /usr/local/bin/protoc
rm protoc-*.zip
- run:
name: Install gogo/protobuf
command: |
gogo_version=$(go list -m github.com/gogo/protobuf | awk '{print $2}')
mkdir -p .gotools; cd .gotools; go mod init consul-tools
go get -v github.com/hashicorp/protoc-gen-go-binary
go get -v github.com/gogo/protobuf/protoc-gen-gofast@${gogo_version}
- run:
command: make --always-make proto
- run: |
if ! git diff --exit-code; then
echo "Generated code was not updated correctly"
exit 1
fi
2019-05-17 15:42:56 +00:00
go-test:
docker:
- image: *GOLANG_IMAGE
parallelism: 4
2019-05-17 15:42:56 +00:00
environment:
<<: *ENVIRONMENT
GOTAGS: "" # No tags for OSS but there are for enterprise
# GOMAXPROCS defaults to number of cores on underlying hardware, set
# explicitly to avoid OOM issues https://support.circleci.com/hc/en-us/articles/360034684273-common-GoLang-memory-issues
GOMAXPROCS: 4
# The medium resource class (default) boxes are 2 vCPUs, 4GB RAM
# https://circleci.com/docs/2.0/configuration-reference/#docker-executor
# but we can run a little over that limit.
2019-05-17 15:42:56 +00:00
steps:
- checkout
- attach_workspace:
at: /go/bin
- run: sudo apt-get update --allow-releaseinfo-change-suite --allow-releaseinfo-change-version && sudo apt-get install -y rsyslog
2019-05-17 15:42:56 +00:00
- run: sudo service rsyslog start
- run: *install-gotestsum
- run: go mod download
- run:
name: go test
command: |
2020-05-02 20:42:39 +00:00
mkdir -p $TEST_RESULTS_DIR /tmp/jsonfile
PACKAGE_NAMES=$(go list -tags "$GOTAGS" ./... | circleci tests split --split-by=timings --timings-type=classname)
echo "Running $(echo $PACKAGE_NAMES | wc -w) packages"
echo $PACKAGE_NAMES
gotestsum \
--format=short-verbose \
2020-05-02 20:42:39 +00:00
--jsonfile /tmp/jsonfile/go-test-${CIRCLE_NODE_INDEX}.log \
--debug \
--rerun-fails=3 \
--rerun-fails-max-failures=40 \
--rerun-fails-report=/tmp/gotestsum-rerun-fails \
--packages="$PACKAGE_NAMES" \
--junitfile $TEST_RESULTS_DIR/gotestsum-report.xml -- \
-tags="$GOTAGS" -p 2 \
-cover -coverprofile=coverage.txt
2019-05-17 15:42:56 +00:00
- store_test_results:
path: *TEST_RESULTS_DIR
2019-05-17 15:42:56 +00:00
- store_artifacts:
path: *TEST_RESULTS_DIR
2020-05-02 20:42:39 +00:00
- store_artifacts:
path: /tmp/jsonfile
- run: &rerun-fails-report
name: "Re-run fails report"
command: |
.circleci/scripts/rerun-fails-report.sh /tmp/gotestsum-rerun-fails
- run: *notify-slack-failure
2019-05-17 15:42:56 +00:00
go-test-race:
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
GOTAGS: "" # No tags for OSS but there are for enterprise
# GOMAXPROCS defaults to number of cores on underlying hardware, set
# explicitly to avoid OOM issues https://support.circleci.com/hc/en-us/articles/360034684273-common-GoLang-memory-issues
GOMAXPROCS: 4
# The medium resource class (default) boxes are 2 vCPUs, 4GB RAM
# https://circleci.com/docs/2.0/configuration-reference/#docker-executor
# but we can run a little over that limit.
steps:
- checkout
- run: *install-gotestsum
- run: go mod download
- run:
name: go test -race
command: |
mkdir -p $TEST_RESULTS_DIR /tmp/jsonfile
pkgs="$(go list ./... | \
grep -E -v '^github.com/hashicorp/consul/agent(/consul|/local|/routine-leak-checker)?$' | \
grep -E -v '^github.com/hashicorp/consul/command/')"
gotestsum \
--jsonfile /tmp/jsonfile/go-test-race.log \
--junitfile $TEST_RESULTS_DIR/gotestsum-report.xml -- \
-tags="$GOTAGS" -p 2 \
-race -gcflags=all=-d=checkptr=0 \
$pkgs
- store_test_results:
path: *TEST_RESULTS_DIR
- store_artifacts:
path: *TEST_RESULTS_DIR
- store_artifacts:
path: /tmp/jsonfile
- run: *notify-slack-failure
# go-test-32bit is to catch problems where 64-bit ints must be 64-bit aligned
# to use them with sync/atomic. See https://golang.org/pkg/sync/atomic/#pkg-note-BUG.
# Running tests with GOARCH=386 seems to be the best way to detect this
# problem. Only runs tests that are -short to limit the time we spend checking
# for these bugs.
go-test-32bit:
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
GOTAGS: "" # No tags for OSS but there are for enterprise
steps:
- checkout
- run: *install-gotestsum
- run: go mod download
- run:
name: go test 32-bit
environment:
GOARCH: 386
command: |
mkdir -p $TEST_RESULTS_DIR /tmp/jsonfile
go env
PACKAGE_NAMES=$(go list -tags "$GOTAGS" ./...)
gotestsum \
--jsonfile /tmp/jsonfile/go-test-32bit.log \
--rerun-fails=3 \
--rerun-fails-max-failures=40 \
--rerun-fails-report=/tmp/gotestsum-rerun-fails \
--packages="$PACKAGE_NAMES" \
--junitfile $TEST_RESULTS_DIR/gotestsum-report.xml -- \
-tags="$GOTAGS" -p 2 \
-short
- store_test_results:
path: *TEST_RESULTS_DIR
- store_artifacts:
path: *TEST_RESULTS_DIR
- run: *notify-slack-failure
go-test-lib:
description: "test a library against a specific Go version"
parameters:
go-version:
type: string
path:
type: string
2019-05-17 15:42:56 +00:00
docker:
- image: "docker.mirror.hashicorp.services/circleci/golang:<<parameters.go-version>>"
environment:
<<: *ENVIRONMENT
GOTAGS: "" # No tags for OSS but there are for enterprise
2019-05-17 15:42:56 +00:00
steps:
- checkout
- attach_workspace:
at: /go/bin
- run: *install-gotestsum
- run:
working_directory: <<parameters.path>>
command: go mod download
2019-05-17 15:42:56 +00:00
- run:
working_directory: <<parameters.path>>
name: go test
2019-05-17 15:42:56 +00:00
command: |
2020-05-02 20:42:39 +00:00
mkdir -p $TEST_RESULTS_DIR /tmp/jsonfile
gotestsum \
--format=short-verbose \
--jsonfile /tmp/jsonfile/go-test-<<parameters.path>>.log \
--junitfile $TEST_RESULTS_DIR/gotestsum-report.xml -- \
-tags="$GOTAGS" -cover -coverprofile=coverage.txt \
./...
2019-05-17 15:42:56 +00:00
- store_test_results:
path: *TEST_RESULTS_DIR
2019-05-17 15:42:56 +00:00
- store_artifacts:
path: *TEST_RESULTS_DIR
2020-05-02 20:42:39 +00:00
- store_artifacts:
path: /tmp/jsonfile
- run: *notify-slack-failure
2019-03-20 19:24:17 +00:00
# build all distros
build-distros: &build-distros
2019-03-08 20:33:03 +00:00
docker:
- image: *GOLANG_IMAGE
2019-03-20 19:24:17 +00:00
environment: &build-env
<<: *ENVIRONMENT
2019-01-28 16:18:30 +00:00
steps:
- checkout
2019-03-20 19:24:17 +00:00
- run: ./build-support/scripts/build-local.sh
2019-01-28 16:18:30 +00:00
# save dev build to CircleCI
- store_artifacts:
2019-03-20 19:24:17 +00:00
path: ./pkg/bin
- run: *notify-slack-failure
2019-03-20 19:24:17 +00:00
2019-04-02 16:24:48 +00:00
# build all 386 architecture supported OS binaries
2019-03-20 19:24:17 +00:00
build-386:
<<: *build-distros
environment:
<<: *build-env
XC_OS: "freebsd linux windows"
2019-03-20 19:24:17 +00:00
XC_ARCH: "386"
2019-04-02 16:24:48 +00:00
# build all amd64 architecture supported OS binaries
2019-03-20 19:24:17 +00:00
build-amd64:
<<: *build-distros
environment:
<<: *build-env
XC_OS: "darwin freebsd linux solaris windows"
XC_ARCH: "amd64"
2019-04-02 16:24:48 +00:00
# build all arm/arm64 architecture supported OS binaries
build-arm:
docker:
- image: *GOLANG_IMAGE
2019-03-20 19:24:17 +00:00
environment:
<<: *ENVIRONMENT
CGO_ENABLED: 1
GOOS: linux
steps:
- checkout
- run: sudo apt-get update --allow-releaseinfo-change-suite --allow-releaseinfo-change-version && sudo apt-get install -y gcc-arm-linux-gnueabi gcc-arm-linux-gnueabihf gcc-aarch64-linux-gnu
- run:
environment:
GOARM: 5
CC: arm-linux-gnueabi-gcc
GOARCH: arm
command: go build -o ./pkg/bin/linux_armel/consul -ldflags="-linkmode=external ${GOLDFLAGS}"
- run:
environment:
GOARM: 6
CC: arm-linux-gnueabihf-gcc
GOARCH: arm
command: go build -o ./pkg/bin/linux_armhf/consul -ldflags="-linkmode=external ${GOLDFLAGS}"
- run:
environment:
CC: aarch64-linux-gnu-gcc
GOARCH: arm64
command: go build -o ./pkg/bin/linux_aarch64/consul -ldflags="-linkmode=external ${GOLDFLAGS}"
- store_artifacts:
path: ./pkg/bin
- run: *notify-slack-failure
2019-03-20 19:24:17 +00:00
2019-04-02 16:24:48 +00:00
# create a development build
2019-03-20 19:24:17 +00:00
dev-build:
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
2019-03-20 19:24:17 +00:00
steps:
- checkout
- attach_workspace: # this normally runs as the first job and has nothing to attach; only used in main branch after rebuilding UI
at: .
2019-05-17 15:42:56 +00:00
- run:
command: make dev
2019-01-28 16:18:30 +00:00
# save dev build to pass to downstream jobs
- persist_to_workspace:
root: /go/bin
paths:
- consul
- run: *notify-slack-failure
2019-01-28 16:18:30 +00:00
2019-04-02 16:24:48 +00:00
# upload development build to s3
dev-upload-s3:
docker:
- image: *GOLANG_IMAGE
2019-04-02 16:24:48 +00:00
environment:
<<: *ENVIRONMENT
steps:
2020-09-04 14:42:18 +00:00
- checkout
- *get-aws-cli
- assume-role:
access-key: AWS_ACCESS_KEY_ID_S3_UPLOAD
secret-key: AWS_SECRET_ACCESS_KEY_S3_UPLOAD
role-arn: ROLE_ARN_S3_UPLOAD
2019-04-02 16:24:48 +00:00
# get consul binary
- attach_workspace:
at: bin/
- run:
name: package binary
2021-01-14 04:27:17 +00:00
command: zip -j consul.zip bin/consul
2019-04-02 16:24:48 +00:00
- run:
name: Upload to s3
command: |
if [ -n "${S3_ARTIFACT_PATH}" ]; then
aws s3 cp \
--metadata "CIRCLECI=${CIRCLECI},CIRCLE_BUILD_URL=${CIRCLE_BUILD_URL},CIRCLE_BRANCH=${CIRCLE_BRANCH}" \
2021-01-14 04:27:17 +00:00
"consul.zip" "s3://${S3_ARTIFACT_BUCKET}/${S3_ARTIFACT_PATH}/${CIRCLE_SHA1}.zip" --acl public-read
else
echo "CircleCI - S3_ARTIFACT_PATH was not set"
exit 1
fi
- run: *notify-slack-failure
2019-04-02 16:24:48 +00:00
# upload dev docker image
dev-upload-docker:
docker:
- image: *GOLANG_IMAGE # use a circleci image so the attach_workspace step works (has ca-certs installed)
environment:
<<: *ENVIRONMENT
steps:
- checkout
# get consul binary
- attach_workspace:
at: bin/
2020-02-12 18:56:16 +00:00
- setup_remote_docker
- run: make ci.dev-docker
- run: *notify-slack-failure
2019-01-28 16:18:30 +00:00
# Nomad 0.8 builds on go0.10
# Run integration tests on nomad/v0.8.7
nomad-integration-0_8:
2019-03-08 20:33:03 +00:00
docker:
- image: docker.mirror.hashicorp.services/circleci/golang:1.10
2019-03-08 20:33:03 +00:00
environment:
<<: *ENVIRONMENT
NOMAD_WORKING_DIR: &NOMAD_WORKING_DIR /go/src/github.com/hashicorp/nomad
NOMAD_VERSION: v0.8.7
steps: &NOMAD_INTEGRATION_TEST_STEPS
- run: git clone https://github.com/hashicorp/nomad.git --branch ${NOMAD_VERSION} ${NOMAD_WORKING_DIR}
# get consul binary
- attach_workspace:
at: /go/bin
# make dev build of nomad
- run:
command: make pkg/linux_amd64/nomad
2019-03-08 20:33:03 +00:00
working_directory: *NOMAD_WORKING_DIR
- run: *install-gotestsum
2019-03-08 20:33:03 +00:00
# run integration tests
- run:
name: go test
command: |
mkdir -p $TEST_RESULTS_DIR
gotestsum \
--format=short-verbose \
--junitfile $TEST_RESULTS_DIR/results.xml -- \
./command/agent/consul -run TestConsul
2019-03-08 20:33:03 +00:00
working_directory: *NOMAD_WORKING_DIR
# store test results for CircleCI
- store_test_results:
path: *TEST_RESULTS_DIR
- store_artifacts:
path: *TEST_RESULTS_DIR
- run: *notify-slack-failure
2019-01-28 16:18:30 +00:00
# run integration tests on nomad/main
nomad-integration-main:
2019-03-08 20:33:03 +00:00
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
NOMAD_WORKING_DIR: /go/src/github.com/hashicorp/nomad
NOMAD_VERSION: main
2019-03-08 20:33:03 +00:00
steps: *NOMAD_INTEGRATION_TEST_STEPS
2019-01-28 16:18:30 +00:00
2020-04-13 23:41:41 +00:00
build-website-docker-image:
docker:
- image: docker.mirror.hashicorp.services/circleci/buildpack-deps
2020-04-13 23:41:41 +00:00
shell: /usr/bin/env bash -euo pipefail -c
steps:
- checkout
2020-04-13 23:41:41 +00:00
- setup_remote_docker
- run:
name: Build Docker Image if Necessary
command: |
# Ignore job if running an enterprise build
IMAGE_TAG=$(cat website/Dockerfile website/package-lock.json | sha256sum | awk '{print $1;}')
echo "Using $IMAGE_TAG"
if [ "$CIRCLE_REPOSITORY_URL" != "git@github.com:hashicorp/consul.git" ]; then
echo "Not Consul OSS Repo, not building website docker image"
elif curl https://hub.docker.com/v2/repositories/hashicorp/consul-website/tags/$IMAGE_TAG -fsL > /dev/null; then
echo "Dependencies have not changed, not building a new website docker image."
else
cd website/
docker build -t hashicorp/consul-website:$IMAGE_TAG .
docker tag hashicorp/consul-website:$IMAGE_TAG hashicorp/consul-website:latest
docker login -u $WEBSITE_DOCKER_USER -p $WEBSITE_DOCKER_PASS
docker push hashicorp/consul-website
fi
- run: *notify-slack-failure
2020-06-29 16:14:43 +00:00
algolia-index:
docker:
- image: docker.mirror.hashicorp.services/node:12
2020-06-29 16:14:43 +00:00
steps:
- checkout
- run:
name: Push content to Algolia Index
command: |
if [ "$CIRCLE_REPOSITORY_URL" != "git@github.com:hashicorp/consul.git" ]; then
echo "Not Consul OSS Repo, not indexing Algolia"
exit 0
fi
cd website/
npm install
node scripts/index_search_content.js
- run: *notify-slack-failure
2020-06-29 16:14:43 +00:00
2019-01-04 03:57:40 +00:00
# build frontend yarn cache
2018-12-20 18:51:48 +00:00
frontend-cache:
docker:
2019-03-08 20:33:03 +00:00
- image: *EMBER_IMAGE
2018-12-20 18:51:48 +00:00
steps:
- checkout
2019-03-08 20:33:03 +00:00
2018-12-20 18:51:48 +00:00
# cache yarn deps
- restore_cache:
2019-03-08 20:33:03 +00:00
key: *YARN_CACHE_KEY
2018-12-20 18:51:48 +00:00
- run:
name: install yarn packages
command: cd ui && make deps
2019-03-08 20:33:03 +00:00
2018-12-20 18:51:48 +00:00
- save_cache:
2019-03-08 20:33:03 +00:00
key: *YARN_CACHE_KEY
2018-12-20 18:51:48 +00:00
paths:
- ui/node_modules
- ui/packages/consul-ui/node_modules
- run: *notify-slack-failure
2019-03-08 20:33:03 +00:00
2018-12-20 18:51:48 +00:00
# build ember so frontend tests run faster
ember-build-oss: &ember-build-oss
2018-12-20 18:51:48 +00:00
docker:
2019-03-08 20:33:03 +00:00
- image: *EMBER_IMAGE
environment:
JOBS: 2 # limit parallelism for broccoli-babel-transpiler
CONSUL_NSPACES_ENABLED: 0
2018-12-20 18:51:48 +00:00
steps:
- checkout
- restore_cache:
2019-03-08 20:33:03 +00:00
key: *YARN_CACHE_KEY
- run: cd ui/packages/consul-ui && make build-ci
2018-12-20 18:51:48 +00:00
# saves the build to a workspace to be passed to a downstream job
- persist_to_workspace:
root: ui
2018-12-20 18:51:48 +00:00
paths:
- packages/consul-ui/dist
- run: *notify-slack-failure
2019-01-04 03:57:40 +00:00
# build ember so frontend tests run faster
ember-build-ent:
<<: *ember-build-oss
environment:
JOBS: 2 # limit parallelism for broccoli-babel-transpiler
CONSUL_NSPACES_ENABLED: 1
# rebuild UI for packaging
ember-build-prod:
docker:
- image: *EMBER_IMAGE
environment:
JOBS: 2 # limit parallelism for broccoli-babel-transpiler
steps:
- checkout
- restore_cache:
key: *YARN_CACHE_KEY
- run: cd ui && make
# saves the build to a workspace to be passed to a downstream job
- persist_to_workspace:
root: ui
paths:
- packages/consul-ui/dist
- run: *notify-slack-failure
# build static-assets file
build-static-assets:
docker:
- image: *GOLANG_IMAGE
steps:
- checkout
- attach_workspace:
at: ./pkg
- run: mv pkg/packages/consul-ui/dist pkg/web_ui # 'make static-assets' looks for the 'pkg/web_ui' path
- run: make tools
- run: make static-assets
- persist_to_workspace:
root: .
paths:
- ./agent/uiserver/bindata_assetfs.go
- run: *notify-slack-failure
# commits static assets to git
publish-static-assets:
docker:
- image: *GOLANG_IMAGE
steps:
- checkout
- add_ssh_keys: # needs a key to push updated static asset commit back to github
fingerprints:
2021-04-22 14:08:55 +00:00
- "fc:55:84:15:0a:1d:c8:e9:06:d0:e8:9c:7b:a9:b7:31"
- attach_workspace:
at: .
- run:
name: commit agent/uiserver/bindata_assetfs.go if there are UI changes
command: |
# check if there are any changes in ui/
# if there are, we commit the ui static asset file
# HEAD^! is shorthand for HEAD^..HEAD (parent of HEAD and HEAD)
if ! git diff --quiet --exit-code HEAD^! ui/; then
2021-04-22 14:08:55 +00:00
git config --local user.email "github-team-consul-core@hashicorp.com"
git config --local user.name "hc-github-team-consul-core"
short_sha=$(git rev-parse --short HEAD)
git add agent/uiserver/bindata_assetfs.go
git commit -m "auto-updated agent/uiserver/bindata_assetfs.go from commit ${short_sha}"
git push origin main
else
echo "no UI changes so no static assets to publish"
fi
- run: *notify-slack-failure
2020-10-06 19:58:12 +00:00
# run node tests
node-tests:
docker:
- image: *EMBER_IMAGE
steps:
- checkout
- restore_cache:
key: *YARN_CACHE_KEY
- attach_workspace:
at: ui
2020-10-06 19:58:12 +00:00
- run:
working_directory: ui/packages/consul-ui
2020-10-06 19:58:12 +00:00
command: make test-node
- run: *notify-slack-failure
# run yarn workspace wide checks/tests
workspace-tests:
docker:
- image: *EMBER_IMAGE
steps:
- checkout
- restore_cache:
key: *YARN_CACHE_KEY
- attach_workspace:
at: ui
- run:
working_directory: ui
command: make test-workspace
- run: *notify-slack-failure
2020-10-06 19:58:12 +00:00
2019-01-04 03:57:40 +00:00
# run ember frontend tests
ember-test-oss:
2018-12-20 18:51:48 +00:00
docker:
2019-03-08 20:33:03 +00:00
- image: *EMBER_IMAGE
2018-12-20 18:51:48 +00:00
environment:
EMBER_TEST_REPORT: test-results/report-oss.xml #outputs test report for CircleCI test summary
2020-02-10 13:41:02 +00:00
EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam
CONSUL_NSPACES_ENABLED: 0
parallelism: 4
steps:
- checkout
- restore_cache:
key: *YARN_CACHE_KEY
- attach_workspace:
at: ui
- run:
working_directory: ui/packages/consul-ui
command: node_modules/.bin/ember exam --split=$CIRCLE_NODE_TOTAL --partition=`expr $CIRCLE_NODE_INDEX + 1` --path dist --silent -r xunit
- store_test_results:
path: ui/packages/consul-ui/test-results
- run: *notify-slack-failure
# run ember frontend tests
ember-test-ent:
docker:
- image: *EMBER_IMAGE
environment:
EMBER_TEST_REPORT: test-results/report-ent.xml #outputs test report for CircleCI test summary
2020-02-10 13:41:02 +00:00
EMBER_TEST_PARALLEL: true #enables test parallelization with ember-exam
CONSUL_NSPACES_ENABLED: 1
parallelism: 4
2018-12-20 18:51:48 +00:00
steps:
- checkout
- restore_cache:
2019-03-08 20:33:03 +00:00
key: *YARN_CACHE_KEY
2018-12-20 18:51:48 +00:00
- attach_workspace:
at: ui
2018-12-20 18:51:48 +00:00
- run:
working_directory: ui/packages/consul-ui
command: node_modules/.bin/ember exam --split=$CIRCLE_NODE_TOTAL --partition=`expr $CIRCLE_NODE_INDEX + 1` --path dist --silent -r xunit
2018-12-20 18:51:48 +00:00
- store_test_results:
path: ui/packages/consul-ui/test-results
- run: *notify-slack-failure
# run ember frontend unit tests to produce coverage report
ember-coverage:
docker:
- image: *EMBER_IMAGE
steps:
- checkout
- restore_cache:
key: *YARN_CACHE_KEY
- attach_workspace:
at: ui
- run:
working_directory: ui/packages/consul-ui
command: make test-coverage-ci
- run: *notify-slack-failure
2018-12-20 18:51:48 +00:00
envoy-integration-test-1_17_4: &ENVOY_TESTS
docker:
# We only really need bash and docker-compose which is installed on all
# Circle images but pick Go since we have to pick one of them.
- image: *GOLANG_IMAGE
parallelism: 2
environment:
ENVOY_VERSION: "1.17.4"
steps: &ENVOY_INTEGRATION_TEST_STEPS
- checkout
# Get go binary from workspace
- attach_workspace:
at: .
2020-02-12 18:56:16 +00:00
- setup_remote_docker
# Build the consul-dev image from the already built binary
- run: docker build -t consul-dev -f ./build-support/docker/Consul-Dev.dockerfile .
- run:
name: Envoy Integration Tests
command: |
subtests=$(ls -d test/integration/connect/envoy/*/ | xargs -n 1 basename | circleci tests split)
echo "Running $(echo $subtests | wc -w) subtests"
echo "$subtests"
subtests_pipe_sepr=$(echo "$subtests" | xargs | sed 's/ /|/g')
mkdir -p /tmp/test-results/
gotestsum -- -timeout=30m -tags integration ./test/integration/connect/envoy -run="TestEnvoy/($subtests_pipe_sepr)"
environment:
GOTESTSUM_JUNITFILE: /tmp/test-results/results.xml
GOTESTSUM_FORMAT: standard-verbose
COMPOSE_INTERACTIVE_NO_CLI: 1
# tput complains if this isn't set to something.
TERM: ansi
- store_artifacts:
path: ./test/integration/connect/envoy/workdir/logs
destination: container-logs
- store_test_results:
path: *TEST_RESULTS_DIR
- store_artifacts:
path: *TEST_RESULTS_DIR
- run: *notify-slack-failure
envoy-integration-test-1_18_4:
<<: *ENVOY_TESTS
environment:
ENVOY_VERSION: "1.18.4"
2021-09-21 22:39:36 +00:00
envoy-integration-test-1_19_1:
<<: *ENVOY_TESTS
environment:
ENVOY_VERSION: "1.19.1"
envoy-integration-test-1_20_0:
<<: *ENVOY_TESTS
environment:
ENVOY_VERSION: "1.20.0"
# run integration tests for the connect ca providers
test-connect-ca-providers:
docker:
- image: *GOLANG_IMAGE
environment:
<<: *ENVIRONMENT
steps:
- run:
name: Install vault
command: |
wget -q -O /tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_BINARY_VERSION}/vault_${VAULT_BINARY_VERSION}_linux_amd64.zip
sudo unzip -d /usr/local/bin /tmp/vault.zip
rm -rf /tmp/vault*
- checkout
- run: go mod download
- run:
name: go test
command: |
mkdir -p $TEST_RESULTS_DIR
make test-connect-ca-providers
- store_test_results:
path: *TEST_RESULTS_DIR
- run: *notify-slack-failure
# only runs on main: checks latest commit to see if the PR associated has a backport/* or docs* label to cherry-pick
cherry-picker:
docker:
- image: docker.mirror.hashicorp.services/alpine:3.12
steps:
- run: apk add --no-cache --no-progress git bash curl ncurses jq openssh-client
- checkout
- add_ssh_keys: # needs a key to push cherry-picked commits back to github
fingerprints:
2021-04-22 14:08:55 +00:00
- "fc:55:84:15:0a:1d:c8:e9:06:d0:e8:9c:7b:a9:b7:31"
- run: .circleci/scripts/cherry-picker.sh
- run: *notify-slack-failure
trigger-oss-merge:
docker:
- image: docker.mirror.hashicorp.services/alpine:3.12
steps:
- run: apk add --no-cache --no-progress curl jq
- run:
name: trigger oss merge
command: |
curl -s -X POST \
--header "Circle-Token: ${CIRCLECI_API_TOKEN}" \
--header "Content-Type: application/json" \
-d '{"build_parameters": {"CIRCLE_JOB": "oss-merge"}}' \
"https://circleci.com/api/v1.1/project/github/hashicorp/consul-enterprise/tree/${CIRCLE_BRANCH}" | jq -r '.build_url'
- run: *notify-slack-failure
2021-01-14 04:30:00 +00:00
# Run load tests against a commit
load-test:
docker:
- image: hashicorp/terraform:latest
environment:
AWS_DEFAULT_REGION: us-east-2
BUCKET: consul-ci-load-tests
BASH_ENV: /etc/profile
shell: /bin/sh -leo pipefail
2021-01-14 04:30:00 +00:00
steps:
- checkout
- run: apk add jq curl bash
- run:
name: export load-test credentials
command: |
echo "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID_LOAD_TEST" >> $BASH_ENV
echo "export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY_LOAD_TEST" >> $BASH_ENV
- run:
name: export role arn
command: |
echo "export TF_VAR_role_arn=$ROLE_ARN_LOAD_TEST" >> $BASH_ENV
- run:
name: setup TF_VARs
command: |
# if pipeline.parameters.commit="" it was not triggered/set through the API
# so we use the latest commit from _this_ branch. This is the case for nightly tests.
if [ "<< pipeline.parameters.commit >>" = "" ]; then
LOCAL_COMMIT_SHA=$(git rev-parse HEAD)
else
LOCAL_COMMIT_SHA="<< pipeline.parameters.commit >>"
fi
echo "export LOCAL_COMMIT_SHA=${LOCAL_COMMIT_SHA}" >> $BASH_ENV
short_ref=$(git rev-parse --short ${LOCAL_COMMIT_SHA})
2021-01-14 04:30:00 +00:00
echo "export TF_VAR_ami_owners=$LOAD_TEST_AMI_OWNERS" >> $BASH_ENV
echo "export TF_VAR_vpc_name=$short_ref" >> $BASH_ENV
echo "export TF_VAR_cluster_name=$short_ref" >> $BASH_ENV
echo "export TF_VAR_consul_download_url=https://${S3_ARTIFACT_BUCKET}.s3.${AWS_DEFAULT_REGION}.amazonaws.com/${S3_ARTIFACT_PATH}/${LOCAL_COMMIT_SHA}.zip" >> $BASH_ENV
2021-01-14 04:30:00 +00:00
- run:
name: wait for dev build from test-integrations workflow
command: |
echo "curl-ing https://${S3_ARTIFACT_BUCKET}.s3.${AWS_DEFAULT_REGION}.amazonaws.com/${S3_ARTIFACT_PATH}/${LOCAL_COMMIT_SHA}.zip"
2021-01-14 04:30:00 +00:00
until [ $SECONDS -ge 300 ] && exit 1; do
curl -o /dev/null --fail --silent "https://${S3_ARTIFACT_BUCKET}.s3.${AWS_DEFAULT_REGION}.amazonaws.com/${S3_ARTIFACT_PATH}/${LOCAL_COMMIT_SHA}.zip" && exit
2021-01-14 04:30:00 +00:00
echo -n "."
sleep 2
done
- run:
working_directory: .circleci/terraform/load-test
name: terraform init
command: |
echo "commit is ${LOCAL_COMMIT_SHA}"
2021-01-14 04:30:00 +00:00
terraform init \
-backend-config="bucket=${BUCKET}" \
-backend-config="key=${LOCAL_COMMIT_SHA}" \
2021-01-14 04:30:00 +00:00
-backend-config="region=${AWS_DEFAULT_REGION}" \
-backend-config="role_arn=${ROLE_ARN_LOAD_TEST}"
- run:
working_directory: .circleci/terraform/load-test
name: run terraform apply
command: |
terraform apply -auto-approve
- run:
working_directory: .circleci/terraform/load-test
when: always
name: terraform destroy
command: |
terraform destroy -auto-approve
- run: *notify-slack-failure
2021-01-14 04:30:00 +00:00
# The noop job is a used as a very fast job in the verify-ci workflow because every workflow
# requires at least one job. It does nothing.
noop:
docker:
- image: docker.mirror.hashicorp.services/alpine:latest
steps:
- run: "echo ok"
workflows:
version: 2
# verify-ci is a no-op workflow that must run on every PR. It is used in a
# branch protection rule to detect when CI workflows are not running.
verify-ci:
jobs: [noop]
go-tests:
2021-01-14 04:30:00 +00:00
unless: << pipeline.parameters.trigger-load-test >>
2019-03-20 19:24:17 +00:00
jobs:
- check-go-mod: &filter-ignore-non-go-branches
filters:
branches:
ignore:
- stable-website
- /^docs\/.*/
- /^ui\/.*/
- check-generated-protobuf: *filter-ignore-non-go-branches
- lint-consul-retry: *filter-ignore-non-go-branches
- lint: *filter-ignore-non-go-branches
- lint:
name: "lint-32bit"
go-arch: "386"
<<: *filter-ignore-non-go-branches
- test-connect-ca-providers: *filter-ignore-non-go-branches
- dev-build: *filter-ignore-non-go-branches
- go-test:
requires: [dev-build]
- go-test-lib:
name: "go-test-api go1.16"
path: api
go-version: "1.16"
requires: [dev-build]
- go-test-lib:
2021-10-20 18:37:29 +00:00
name: "go-test-api go1.17"
path: api
2021-10-20 18:37:29 +00:00
go-version: "1.17"
requires: [dev-build]
- go-test-lib:
name: "go-test-sdk go1.16"
path: sdk
go-version: "1.16"
<<: *filter-ignore-non-go-branches
- go-test-lib:
2021-10-20 18:37:29 +00:00
name: "go-test-sdk go1.17"
path: sdk
2021-10-20 18:37:29 +00:00
go-version: "1.17"
<<: *filter-ignore-non-go-branches
- go-test-race: *filter-ignore-non-go-branches
- go-test-32bit: *filter-ignore-non-go-branches
build-distros:
2021-01-14 04:30:00 +00:00
unless: << pipeline.parameters.trigger-load-test >>
jobs:
- check-go-mod: *filter-ignore-non-go-branches
- build-386: &require-check-go-mod
2019-10-08 19:55:36 +00:00
requires:
- check-go-mod
- build-amd64: *require-check-go-mod
- build-arm: *require-check-go-mod
# every commit on main will have a rebuilt UI
- frontend-cache:
filters:
branches:
only:
- main
- ember-build-prod:
requires:
- frontend-cache
- build-static-assets:
requires:
- ember-build-prod
- publish-static-assets:
filters:
branches:
only:
- main
- /release\/\d+\.\d+\.x$/
requires:
- build-static-assets
- dev-build:
requires:
- build-static-assets
- dev-upload-s3:
requires:
- dev-build
- dev-upload-docker:
requires:
- dev-build
context: consul-ci
test-integrations:
2021-01-14 04:30:00 +00:00
unless: << pipeline.parameters.trigger-load-test >>
jobs:
- dev-build: *filter-ignore-non-go-branches
- dev-upload-s3: &dev-upload
2019-04-02 16:24:48 +00:00
requires:
- dev-build
filters:
branches:
ignore:
- /^pull\/.*$/ # only push dev builds from non forks
- main # all main dev uploads will include a UI rebuild in build-distros
- dev-upload-docker:
<<: *dev-upload
context: consul-ci
- nomad-integration-main:
2019-01-28 16:18:30 +00:00
requires:
2019-03-08 20:33:03 +00:00
- dev-build
2019-01-28 16:18:30 +00:00
- nomad-integration-0_8:
requires:
- dev-build
- envoy-integration-test-1_17_4:
requires:
- dev-build
- envoy-integration-test-1_18_4:
requires:
- dev-build
2021-09-21 22:39:36 +00:00
- envoy-integration-test-1_19_1:
requires:
- dev-build
- envoy-integration-test-1_20_0:
requires:
- dev-build
website:
2021-01-14 04:30:00 +00:00
unless: << pipeline.parameters.trigger-load-test >>
jobs:
2020-04-13 23:41:41 +00:00
- build-website-docker-image:
context: website-docker-image
2019-04-05 14:38:27 +00:00
filters:
branches:
2020-04-13 23:41:41 +00:00
only:
- main
2020-06-29 16:14:43 +00:00
- algolia-index:
context: consul-docs
filters:
branches:
only:
- stable-website
2018-12-20 18:51:48 +00:00
frontend:
2021-01-14 04:30:00 +00:00
unless: << pipeline.parameters.trigger-load-test >>
2018-12-20 18:51:48 +00:00
jobs:
- frontend-cache:
filters:
branches:
only:
- main
- /^ui\/.*/
- workspace-tests:
requires:
- frontend-cache
2020-10-06 19:58:12 +00:00
- node-tests:
requires:
- frontend-cache
- ember-build-oss:
requires:
- frontend-cache
- ember-build-ent:
2018-12-20 18:51:48 +00:00
requires:
- frontend-cache
- ember-test-oss:
requires:
- ember-build-oss
- ember-test-ent:
2018-12-20 18:51:48 +00:00
requires:
- ember-build-ent
# ember-coverage in CI uses the dist/ folder to run tests so it requires
# either/or ent/oss to be built first
- ember-coverage:
requires:
- ember-build-ent
workflow-automation:
2021-01-14 04:30:00 +00:00
unless: << pipeline.parameters.trigger-load-test >>
jobs:
- trigger-oss-merge:
context: team-consul
filters:
branches:
only:
- main
- /release\/\d+\.\d+\.x$/
- cherry-picker:
context: team-consul
filters:
branches:
only:
- main
- /release\/\d+\.\d+\.x$/
2021-01-14 04:30:00 +00:00
load-test:
when: << pipeline.parameters.trigger-load-test >>
jobs:
- load-test
nightly-jobs:
triggers:
- schedule:
cron: "0 4 * * *" # 4AM UTC <> 12AM EST <> 9PM PST should have no impact
filters:
branches:
only:
- main
jobs:
- load-test