luxolus
/
open-consul
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity
open-consul/GNUmakefile

408 lines
14 KiB
Makefile
Raw Normal View History

build: Use bash for building
2017-04-27 18:49:43 +00:00
SHELL = bash
Generate JSON and Binary Marshalers for Protobuf Types (#6564) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 19:39:20 +00:00
GOGOVERSION?=$(shell grep github.com/gogo/protobuf go.mod | awk '{print $$2}')
Add `tools/cmd/cover` to GOTOOLS
2016-05-07 20:02:12 +00:00
GOTOOLS = \
ui: feature support templating for index.html (#6921)
2019-12-13 19:50:07 +00:00
github.com/elazarl/go-bindata-assetfs/go-bindata-assetfs@master \
github.com/hashicorp/go-bindata/go-bindata@master \
Add `tools/cmd/cover` to GOTOOLS
2016-05-07 20:02:12 +00:00
golang.org/x/tools/cmd/cover \
Missing tools in makefile (#3205)
2017-06-29 10:55:01 +00:00
golang.org/x/tools/cmd/stringer \
Generate JSON and Binary Marshalers for Protobuf Types (#6564) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 19:39:20 +00:00
github.com/gogo/protobuf/protoc-gen-gofast@$(GOGOVERSION) \
github.com/hashicorp/protoc-gen-go-binary \
CA Provider Plugins (#4751) This adds the `agent/connect/ca/plugin` library for consuming/serving Connect CA providers as [go-plugin](https://github.com/hashicorp/go-plugin) plugins. This **does not** wire this up in any way to Consul itself, so this will not enable using these plugins yet. ## Why? We want to enable CA providers to be pluggable without modifying Consul so that any CA or PKI system can potentially back the Connect certificates. This CA system may also be used in the future for easier bootstrapping and internal cluster security. ### go-plugin The benefit of `go-plugin` is that for the plugin consumer, the fact that the interface implementation is communicating over multi-process RPC is invisible. Internals of Consul will continue to just use `ca.Provider` interface implementations as if they're local. For plugin _authors_, they simply have to implement the interface. The network/transport/process management issues are handled by go-plugin itself. The CA provider plugins support both `net/rpc` and gRPC transports. This enables easy authoring in any language. go-plugin handles the actual protocol handshake and connection. This is just a feature of go-plugin. `go-plugin` is already in production use for years by Packer, Terraform, Nomad, Vault, and Sentinel. We've shown stability for both desktop and server-side software. It is very mature. ## Implementation Details ### `map[string]interface{}` The `Configure` method passes a `map[string]interface{}`. This map contains only Go primitives and containers of primitives (no funcs, chans, etc.). For `net/rpc` we encode as-is using Gob. For gRPC we marshal to JSON and transmit as a `bytes` type. This is the same approach we take with Vault and other software. Note that this is just the transport protocol, the end software views it fully decoded. ### `x509.Certificate` and `CertificateRequest` We transmit the raw ASN.1 bytes and decode on the other side. Unit tests are verifying we get the same cert/csrs across the wire. ### Testing `go-plugin` exposes test helpers that enable testing the full plugin RPC over real loopback network connections. We test all endpoints for success and error for both `net/rpc` and gRPC. ### Vendoring This PR doesn't introduce vendoring for two reasons: 1. @banks's `f-envoy` branch introduces a lot of these and I didn't want conflict. 2. The library isn't actually used yet so it doesn't introduce compile-time errors (it does introduce test errors). ## Next Steps With this in place, we need to figure out the proper way to actually hook these up to Consul, load them, etc. This discussion can happen elsewhere, since regardless of approach this plugin library implementation is the exact same.
2019-01-07 17:48:44 +00:00
github.com/vektra/mockery/cmd/mockery
build: build all packages together Build all packages together with a global timeout. Locally, the tests pass within 40 sec. On travis, we complete within 3-4 min. travis truncates the logs after 4MB which we are hitting so show the relevant information first and then whatever is left.
2017-06-29 12:50:47 +00:00
build: drop 'consul' build tag for OSS build
2017-08-30 11:25:14 +00:00
GOTAGS ?=
Move the watch package into the api module (#5664) * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)
2019-04-26 16:33:01 +00:00
GOMODULES ?= ./... ./api/... ./sdk/...
GOFILES ?= $(shell go list $(GOMODULES) | grep -v /vendor/)
Travis evaluates ENV before cloning git repo and cding so we need to delay gathering packages until the makefile
2018-02-21 12:53:35 +00:00
ifeq ($(origin GOTEST_PKGS_EXCLUDE), undefined)
Move the watch package into the api module (#5664) * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)
2019-04-26 16:33:01 +00:00
GOTEST_PKGS ?= $(GOMODULES)
Travis evaluates ENV before cloning git repo and cding so we need to delay gathering packages until the makefile
2018-02-21 12:53:35 +00:00
else
Move the watch package into the api module (#5664) * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)
2019-04-26 16:33:01 +00:00
GOTEST_PKGS=$(shell go list $(GOMODULES) | sed 's/github.com\/hashicorp\/consul/./' | egrep -v "^($(GOTEST_PKGS_EXCLUDE))$$")
Travis evaluates ENV before cloning git repo and cding so we need to delay gathering packages until the makefile
2018-02-21 12:53:35 +00:00
endif
Fix a couple find warnings on linux Additionally add the ability to use go install for dev builds rather than gox (travis doesn’t have gox)
2018-06-19 14:49:07 +00:00
GOOS?=$(shell go env GOOS)
GOARCH?=$(shell go env GOARCH)
Allow for default GOPATH
2018-01-04 19:38:20 +00:00
GOPATH=$(shell go env GOPATH)
`make test-docker` (#6059) * Implement the test-docker make target Running tests within docker allows us to resource constrain them better to not take over our systems. Additionally it allows us to run the tests on linux instead of the host OS which often times is macOS. * Use GOMAXPROCS instead of -p * Add a comment about docker cpus
2019-07-04 14:22:59 +00:00
MAIN_GOPATH=$(shell go env GOPATH | cut -d: -f1)
Faster dev builds (#2924) This patch runs 'go install' instead of gox which runs 'go build' for 'make dev' and copies the binary into ./bin and ./pkg/${GOOS}_${GOARCH} to mimick the previous behavior. This reduces the roundtrip times for a dev build from 11 sec to 500ms if there weren't any changes.
2017-04-18 22:03:51 +00:00
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
ASSETFS_PATH?=agent/bindata_assetfs.go
Faster dev builds (#2924) This patch runs 'go install' instead of gox which runs 'go build' for 'make dev' and copies the binary into ./bin and ./pkg/${GOOS}_${GOARCH} to mimick the previous behavior. This reduces the roundtrip times for a dev build from 11 sec to 500ms if there weren't any changes.
2017-04-18 22:03:51 +00:00
# Get the git commit
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
GIT_COMMIT?=$(shell git rev-parse --short HEAD)
Pass the git commits year through as an env var to the ui build (#7126)
2020-01-24 15:40:14 +00:00
GIT_COMMIT_YEAR?=$(shell git show -s --format=%cd --date=format:%Y HEAD)
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
GIT_DIRTY?=$(shell test -n "`git status --porcelain`" && echo "+CHANGES" || true)
build: use only version tags in version output now api is tagged too (#5622) * build: use only version tags in version output now api is tagged too Fixes #5621 Since we now have api package tags, our build tooling was picking up api tag when working out version to bake into builds. This fixes it by restricting to only tags that start with `v`. Before: ``` $ make version Version: 1.4.4 Version + release: 1.4.4-dev Version + git: api/v1.0.1-90-g3ce60db0c Version + release + git: api/v1.0.1-90-g3ce60db0c-dev (3ce60db0c) ``` After: ``` $ make version Version: 1.4.4 Version + release: 1.4.4-dev Version + git: v1.4.4-126-g3ce60db0c Version + release + git: v1.4.4-126-g3ce60db0c-dev (3ce60db0c) ``` * Update GNUmakefile
2019-04-10 11:54:03 +00:00
GIT_DESCRIBE?=$(shell git describe --tags --always --match "v*")
Faster dev builds (#2924) This patch runs 'go install' instead of gox which runs 'go build' for 'make dev' and copies the binary into ./bin and ./pkg/${GOOS}_${GOARCH} to mimick the previous behavior. This reduces the roundtrip times for a dev build from 11 sec to 500ms if there weren't any changes.
2017-04-18 22:03:51 +00:00
GIT_IMPORT=github.com/hashicorp/consul/version
GOLDFLAGS=-X $(GIT_IMPORT).GitCommit=$(GIT_COMMIT)$(GIT_DIRTY) -X $(GIT_IMPORT).GitDescribe=$(GIT_DESCRIBE)
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
PROTOFILES?=$(shell find . -name '*.proto' | grep -v 'vendor/')
PROTOGOFILES=$(PROTOFILES:.proto=.pb.go)
Generate JSON and Binary Marshalers for Protobuf Types (#6564) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 19:39:20 +00:00
PROTOGOBINFILES=$(PROTOFILES:.proto=.pb.binary.go)
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
ifeq ($(FORCE_REBUILD),1)
NOCACHE=--no-cache
else
NOCACHE=
endif
DOCKER_BUILD_QUIET?=1
ifeq (${DOCKER_BUILD_QUIET},1)
QUIET=-q
else
QUIET=
endif
CONSUL_DEV_IMAGE?=consul-dev
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
GO_BUILD_TAG?=consul-build-go
UI_BUILD_TAG?=consul-build-ui
BUILD_CONTAINER_NAME?=consul-builder
Build System Fixes for Go Modules (#5655) * Docker based builds can now use the module cache * Simplify building the consul-dev docker image. * Make sure to pull the latest consul image. * Allow selecting base image version for the dev image
2019-04-12 19:17:13 +00:00
CONSUL_IMAGE_VERSION?=latest
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
add dev docker image upload (#5879) * add dev docker image upload * remove the go cache since that isn't needed * add comment and image labels * get password from stdin
2019-07-25 13:19:09 +00:00
################
# CI Variables #
################
CI_DEV_DOCKER_NAMESPACE?=hashicorpdev
CI_DEV_DOCKER_IMAGE_NAME?=consul
CI_DEV_DOCKER_WORKDIR?=bin/
################
`make test-docker` (#6059) * Implement the test-docker make target Running tests within docker allows us to resource constrain them better to not take over our systems. Additionally it allows us to run the tests on linux instead of the host OS which often times is macOS. * Use GOMAXPROCS instead of -p * Add a comment about docker cpus
2019-07-04 14:22:59 +00:00
TEST_MODCACHE?=1
TEST_BUILDCACHE?=1
# You can only use as many CPUs as you have allocated to docker
ifdef TEST_DOCKER_CPUS
TEST_DOCKER_RESOURCE_CONSTRAINTS=--cpus $(TEST_DOCKER_CPUS)
TEST_PARALLELIZATION=-e GOMAXPROCS=$(TEST_DOCKER_CPUS)
else
TEST_DOCKER_RESOURCE_CONSTRAINTS=
TEST_PARALLELIZATION=
endif
ifeq ($(TEST_MODCACHE), 1)
TEST_MODCACHE_VOL=-v $(MAIN_GOPATH)/pkg/mod:/go/pkg/mod
else
TEST_MODCACHE_VOL=
endif
ifeq ($(TEST_BUILDCACHE), 1)
TEST_BUILDCACHE_VOL=-v $(shell go env GOCACHE):/root/.caches/go-build
else
TEST_BUILDCACHE_VOL=
endif
Redo the build system Improvements: - More modular - Building within docker doesn’t use volumes so can be run on a remote docker host - Build containers include only minimal context so they only rarely need to be rebuilt and most of the time can be used from the cache. - 3 build containers instead of 1. One based off of the upstream golang containers for building go stuff with all our required GOTOOLS installed. One like the old container based off ubuntu bionic for building the old UI (didn’t bother creating a much better container as this shouldn’t be needed once we completely remove the legacy UI). One for building the new UI. Its alpine based with all the node, ember, yarn stuff installed. - Top level makefile has the ability to do a container based build without running make dist - Can build for arbitrary platforms at the top level using: make consul-docker XC_OS=… XC_ARCH=… - overridable functionality to allow for customizations to the enterprise build (like to generate multiple binaries) - unified how we compile our go. always use gox even for dev-builds or rather always use the tooling around our scripts which will make sure things get copied to the correct places throughout the filesystem.
2018-06-12 20:55:52 +00:00
DIST_TAG?=1
DIST_BUILD?=1
DIST_SIGN?=1
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
ifdef DIST_VERSION
Quote some make vars
2018-06-18 18:36:24 +00:00
DIST_VERSION_ARG=-v "$(DIST_VERSION)"
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
else
DIST_VERSION_ARG=
endif
ifdef DIST_RELEASE_DATE
Quote some make vars
2018-06-18 18:36:24 +00:00
DIST_DATE_ARG=-d "$(DIST_RELEASE_DATE)"
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
else
DIST_DATE_ARG=
endif
Allow for building pre-releases/rcs/betas
2018-06-18 21:01:20 +00:00
ifdef DIST_PRERELEASE
DIST_REL_ARG=-r "$(DIST_PRERELEASE)"
else
DIST_REL_ARG=
endif
Generalize git pushing in a bash function
2018-06-15 19:23:26 +00:00
PUB_GIT?=1
PUB_WEBSITE?=1
ifeq ($(PUB_GIT),1)
PUB_GIT_ARG=-g
else
PUB_GIT_ARG=
endif
ifeq ($(PUB_WEBSITE),1)
Fix bug in makefile that prevented running the website publish.
2018-06-26 18:08:10 +00:00
PUB_WEBSITE_ARG=-w
Generalize git pushing in a bash function
2018-06-15 19:23:26 +00:00
else
PUB_WEBSITE_ARG=
endif
Default dev-build to not using GOX
2018-06-26 18:26:23 +00:00
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
export GO_BUILD_TAG
export UI_BUILD_TAG
export BUILD_CONTAINER_NAME
export GIT_COMMIT
Pass the git commits year through as an env var to the ui build (#7126)
2020-01-24 15:40:14 +00:00
export GIT_COMMIT_YEAR
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
export GIT_DIRTY
export GIT_DESCRIBE
export GOTAGS
Faster dev builds (#2924) This patch runs 'go install' instead of gox which runs 'go build' for 'make dev' and copies the binary into ./bin and ./pkg/${GOOS}_${GOARCH} to mimick the previous behavior. This reduces the roundtrip times for a dev build from 11 sec to 500ms if there weren't any changes.
2017-04-18 22:03:51 +00:00
export GOLDFLAGS
consul: adding basic skeleton
2013-12-06 23:43:07 +00:00
Connect: allow configuring Envoy for L7 Observability (#5558) * Add support for HTTP proxy listeners * Add customizable bootstrap configuration options * Debug logging for xDS AuthZ * Add Envoy Integration test suite with basic test coverage * Add envoy command tests to cover new cases * Add tracing integration test * Add gRPC support WIP * Merged changes from master Docker. get CI integration to work with same Dockerfile now * Make docker build optional for integration * Enable integration tests again! * http2 and grpc integration tests and fixes * Fix up command config tests * Store all container logs as artifacts in circle on fail * Add retries to outer part of stats measurements as we keep missing them in CI * Only dump logs on failing cases * Fix typos from code review * Review tidying and make tests pass again * Add debug logs to exec test. * Fix legit test failure caused by upstream rename in envoy config * Attempt to reduce cases of bad TLS handshake in CI integration tests * bring up the right service * Add prometheus integration test * Add test for denied AuthZ both HTTP and TCP * Try ANSI term for Circle
2019-04-29 16:27:57 +00:00
# Allow skipping docker build during integration tests in CI since we already
# have a built binary
ENVOY_INTEG_DEPS?=dev-docker
ifdef SKIP_DOCKER_BUILD
ENVOY_INTEG_DEPS=noop
endif
Default dev-build to not using GOX
2018-06-26 18:26:23 +00:00
Added capability to make dev-tree without pushing No push is the default
2018-06-26 15:46:37 +00:00
DEV_PUSH?=0
ifeq ($(DEV_PUSH),1)
DEV_PUSH_ARG=
else
DEV_PUSH_ARG=--no-push
endif
Tweaks some of the default makefile targets.
2016-02-18 04:36:48 +00:00
# all builds binaries for all targets
Only builds the binary stuff under master branch CI.
2016-10-25 20:49:57 +00:00
all: bin
Connect: allow configuring Envoy for L7 Observability (#5558) * Add support for HTTP proxy listeners * Add customizable bootstrap configuration options * Debug logging for xDS AuthZ * Add Envoy Integration test suite with basic test coverage * Add envoy command tests to cover new cases * Add tracing integration test * Add gRPC support WIP * Merged changes from master Docker. get CI integration to work with same Dockerfile now * Make docker build optional for integration * Enable integration tests again! * http2 and grpc integration tests and fixes * Fix up command config tests * Store all container logs as artifacts in circle on fail * Add retries to outer part of stats measurements as we keep missing them in CI * Only dump logs on failing cases * Fix typos from code review * Review tidying and make tests pass again * Add debug logs to exec test. * Fix legit test failure caused by upstream rename in envoy config * Attempt to reduce cases of bad TLS handshake in CI integration tests * bring up the right service * Add prometheus integration test * Add test for denied AuthZ both HTTP and TCP * Try ANSI term for Circle
2019-04-29 16:27:57 +00:00
# used to make integration dependencies conditional
noop: ;
Fix default make target to build everything
2018-06-27 18:25:49 +00:00
bin: tools
@$(SHELL) $(CURDIR)/build-support/scripts/build-local.sh
Use gox for building
2015-10-22 18:16:01 +00:00
Typos
2015-10-22 19:00:35 +00:00
# dev creates binaries for testing locally - these are put into ./bin and $GOPATH
remove remaining references to govendor and vendorfmt (#5587)
2019-04-01 14:55:48 +00:00
dev: changelogfmt dev-build
build: fix travis build
2017-07-18 07:22:49 +00:00
dev-build:
Fix a couple find warnings on linux Additionally add the ability to use go install for dev builds rather than gox (travis doesn’t have gox)
2018-06-19 14:49:07 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/build-local.sh -o $(GOOS) -a $(GOARCH)
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
Build System Fixes for Go Modules (#5655) * Docker based builds can now use the module cache * Simplify building the consul-dev docker image. * Make sure to pull the latest consul image. * Allow selecting base image version for the dev image
2019-04-12 19:17:13 +00:00
dev-docker: linux
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull consul:$(CONSUL_IMAGE_VERSION) >/dev/null
@echo "Building Consul Development container - $(CONSUL_DEV_IMAGE)"
@docker build $(NOCACHE) $(QUIET) -t '$(CONSUL_DEV_IMAGE)' --build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) $(CURDIR)/pkg/bin/linux_amd64 -f $(CURDIR)/build-support/docker/Consul-Dev.dockerfile
Use gox for building
2015-10-22 18:16:01 +00:00
add dev docker image upload (#5879) * add dev docker image upload * remove the go cache since that isn't needed * add comment and image labels * get password from stdin
2019-07-25 13:19:09 +00:00
# In CircleCI, the linux binary will be attached from a previous step at bin/. This make target
# should only run in CI and not locally.
ci.dev-docker:
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull consul:$(CONSUL_IMAGE_VERSION) >/dev/null
@echo "Building Consul Development container - $(CI_DEV_DOCKER_IMAGE_NAME)"
@docker build $(NOCACHE) $(QUIET) -t '$(CI_DEV_DOCKER_NAMESPACE)/$(CI_DEV_DOCKER_IMAGE_NAME):$(GIT_COMMIT)' \
--build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) \
--label COMMIT_SHA=$(CIRCLE_SHA1) \
--label PULL_REQUEST=$(CIRCLE_PULL_REQUEST) \
--label CIRCLE_BUILD_URL=$(CIRCLE_BUILD_URL) \
$(CI_DEV_DOCKER_WORKDIR) -f $(CURDIR)/build-support/docker/Consul-Dev.dockerfile
change DOCKER_LOGIN env name to DOCKER_USER (#6451)
2019-09-05 18:29:39 +00:00
@echo $(DOCKER_PASS) | docker login -u="$(DOCKER_USER)" --password-stdin
add dev docker image upload (#5879) * add dev docker image upload * remove the go cache since that isn't needed * add comment and image labels * get password from stdin
2019-07-25 13:19:09 +00:00
@echo "Pushing dev image to: https://cloud.docker.com/u/hashicorpdev/repository/docker/hashicorpdev/consul"
@docker push $(CI_DEV_DOCKER_NAMESPACE)/$(CI_DEV_DOCKER_IMAGE_NAME):$(GIT_COMMIT)
ifeq ($(CIRCLE_BRANCH), master)
@docker tag $(CI_DEV_DOCKER_NAMESPACE)/$(CI_DEV_DOCKER_IMAGE_NAME):$(GIT_COMMIT) $(CI_DEV_DOCKER_NAMESPACE)/$(CI_DEV_DOCKER_IMAGE_NAME):latest
@docker push $(CI_DEV_DOCKER_NAMESPACE)/$(CI_DEV_DOCKER_IMAGE_NAME):latest
endif
build: add changelogfmt target to format [GH-xxxx] references to links
2017-10-04 18:56:28 +00:00
changelogfmt:
@echo "--> Making [GH-xxxx] references clickable..."
@sed -E 's|([^\[])\[GH-([0-9]+)\]|\1[[GH-\2](https://github.com/hashicorp/consul/issues/\2)]|g' CHANGELOG.md > changelog.tmp && mv changelog.tmp CHANGELOG.md
fix type in make file
2017-06-28 14:48:00 +00:00
# linux builds a linux package independent of the source platform
Add make target for linux binary
2017-05-04 11:31:56 +00:00
linux:
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/build-local.sh -o linux -a amd64
Add make target for linux binary
2017-05-04 11:31:56 +00:00
Tweaks some of the default makefile targets.
2016-02-18 04:36:48 +00:00
# dist builds binaries for all platforms and packages them for distribution
dist:
Allow for building pre-releases/rcs/betas
2018-06-18 21:01:20 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/release.sh -t '$(DIST_TAG)' -b '$(DIST_BUILD)' -S '$(DIST_SIGN)' $(DIST_VERSION_ARG) $(DIST_DATE_ARG) $(DIST_REL_ARG)
Update verify.sh script
2018-06-26 16:08:33 +00:00
verify:
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/verify.sh
Update verify.sh script
2018-06-26 16:08:33 +00:00
Add more functionality related to verifying a build and publishing
2018-06-13 19:10:02 +00:00
publish:
Quote some make vars
2018-06-18 18:36:24 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/publish.sh $(PUB_GIT_ARG) $(PUB_WEBSITE_ARG)
Use gox for building
2015-10-22 18:16:01 +00:00
Add capability to put tree back into dev mode via make dev-tree
2018-06-15 12:00:12 +00:00
dev-tree:
Added capability to make dev-tree without pushing No push is the default
2018-06-26 15:46:37 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/dev.sh $(DEV_PUSH_ARG)
Add capability to put tree back into dev mode via make dev-tree
2018-06-15 12:00:12 +00:00
consul: adding basic skeleton
2013-12-06 23:43:07 +00:00
cov:
Move the watch package into the api module (#5664) * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)
2019-04-26 16:33:01 +00:00
go test $(GOMODULES) -coverprofile=coverage.out
removing gocov packages (#5534) * removing gocov packages * revise cov make target
2019-04-09 20:05:52 +00:00
go tool cover -html=coverage.out
consul: adding basic skeleton
2013-12-06 23:43:07 +00:00
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
test: other-consul dev-build vet test-install-deps test-internal
test-install-deps:
go test -tags '$(GOTAGS)' -i $(GOTEST_PKGS)
Move the watch package into the api module (#5664) * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)
2019-04-26 16:33:01 +00:00
update-vendor:
@echo "--> Running go mod vendor"
@go mod vendor
@echo "--> Removing vendoring of our own nested modules"
@rm -rf vendor/github.com/hashicorp/consul
@grep -v "hashicorp/consul/" < vendor/modules.txt > vendor/modules.txt.new
@mv vendor/modules.txt.new vendor/modules.txt
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
test-internal:
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
@echo "--> Running go test"
@rm -f test.log exit-code
Only output sparse lines to keep Travis happy while logging verbosely to disk
2018-02-20 15:32:44 +00:00
@# Dump verbose output to test.log so we can surface test names on failure but
@# hide it from travis as it exceeds their log limits and causes job to be
@# terminated (over 4MB and over 10k lines in the UI). We need to output
@# _something_ to stop them terminating us due to inactivity...
Improve resilience of api pkg tests (#4676) * Add function to wait for serfHealth in api tests * Disable connect when creating semaphore test clients * Wait for serfHealth when creating sessions in their tests * Add helper functions to create lock/semaphore sessions without checks * Log passing tests to prevent timeout in Travis due to lack of output
2018-09-18 16:47:01 +00:00
{ go test -v $(GOTEST_FLAGS) -tags '$(GOTAGS)' $(GOTEST_PKGS) 2>&1 ; echo $$? > exit-code ; } | tee test.log | egrep '^(ok|FAIL|panic:|--- FAIL|--- PASS)'
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
@echo "Exit code: $$(cat exit-code)"
Make test output more useful now we uses testify with multi-line error messages
2018-04-19 12:01:20 +00:00
@# This prints all the race report between ====== lines
@awk '/^WARNING: DATA RACE/ {do_print=1; print "=================="} do_print==1 {print} /^={10,}/ {do_print=0}' test.log || true
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list.
2018-04-20 13:24:24 +00:00
@grep -A10 'panic: ' test.log || true
Make test output more useful now we uses testify with multi-line error messages
2018-04-19 12:01:20 +00:00
@# Prints all the failure output until the next non-indented line - testify
Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list.
2018-04-20 13:24:24 +00:00
@# helpers often output multiple lines for readability but useless if we can't
@# see them. Un-intuitive order of matches is necessary. No || true because
@# awk always returns true even if there is no match and it breaks non-bash
@# shells locally.
@awk '/^[^[:space:]]/ {do_print=0} /--- SKIP/ {do_print=1} do_print==1 {print}' test.log
@awk '/^[^[:space:]]/ {do_print=0} /--- FAIL/ {do_print=1} do_print==1 {print}' test.log
build: build all packages together Build all packages together with a global timeout. Locally, the tests pass within 40 sec. On travis, we complete within 3-4 min. travis truncates the logs after 4MB which we are hitting so show the relevant information first and then whatever is left.
2017-06-29 12:50:47 +00:00
@grep '^FAIL' test.log || true
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
@if [ "$$(cat exit-code)" == "0" ] ; then echo "PASS" ; exit 0 ; else exit 1 ; fi
build: build all packages together Build all packages together with a global timeout. Locally, the tests pass within 40 sec. On travis, we complete within 3-4 min. travis truncates the logs after 4MB which we are hitting so show the relevant information first and then whatever is left.
2017-06-29 12:50:47 +00:00
test-race:
$(MAKE) GOTEST_FLAGS=-race
build: add target for running tests with race detector
2017-05-22 19:24:38 +00:00
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
# Run tests with config for CI so `make test` can still be local-dev friendly.
test-ci: other-consul dev-build vet test-install-deps
BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472) - Improve resilience of testrpc.WaitForLeader() - Add additionall retry to CI - Increase "go test" timeout to 8m - Add wait for cluster leader to several tests in the agent package - Add retry to some tests in the api and command packages
2018-08-06 23:46:09 +00:00
@ if ! GOTEST_FLAGS="-short -timeout 8m -p 3 -parallel 4" make test-internal; then \
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
echo " ============"; \
BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472) - Improve resilience of testrpc.WaitForLeader() - Add additionall retry to CI - Increase "go test" timeout to 8m - Add wait for cluster leader to several tests in the agent package - Add retry to some tests in the api and command packages
2018-08-06 23:46:09 +00:00
echo " Retrying 1/2"; \
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
echo " ============"; \
Keep same parameters on retry so results can be cached by go test (#4627)
2018-09-04 11:27:39 +00:00
if ! GOTEST_FLAGS="-timeout 9m -p 1 -parallel 1" make test-internal; then \
BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472) - Improve resilience of testrpc.WaitForLeader() - Add additionall retry to CI - Increase "go test" timeout to 8m - Add wait for cluster leader to several tests in the agent package - Add retry to some tests in the api and command packages
2018-08-06 23:46:09 +00:00
echo " ============"; \
echo " Retrying 2/2"; \
echo " ============"; \
GOTEST_FLAGS="-timeout 9m -p 1 -parallel 1" make test-internal; \
fi \
fi
Refactor test retry to only affect CI (#4436) * Refactor test retry to only affect CI * Move test install deps out of the retry loop * Add internal targets to PHONY too
2018-07-24 14:12:48 +00:00
Add script and makefile goal to help debug flaky tests
2018-09-10 15:44:07 +00:00
test-flake: other-consul vet test-install-deps
@$(SHELL) $(CURDIR)/build-support/scripts/test-flake.sh --pkg "$(FLAKE_PKG)" --test "$(FLAKE_TEST)" --cpus "$(FLAKE_CPUS)" --n "$(FLAKE_N)"
`make test-docker` (#6059) * Implement the test-docker make target Running tests within docker allows us to resource constrain them better to not take over our systems. Additionally it allows us to run the tests on linux instead of the host OS which often times is macOS. * Use GOMAXPROCS instead of -p * Add a comment about docker cpus
2019-07-04 14:22:59 +00:00
test-docker: linux go-build-image
@# -ti run in the foreground showing stdout
@# --rm removes the container once its finished running
@# GO_MODCACHE_VOL - args for mapping in the go module cache
@# GO_BUILD_CACHE_VOL - args for mapping in the go build cache
@# All the env vars are so we pass through all the relevant bits of information
@# Needed for running the tests
@# We map in our local linux_amd64 bin directory as thats where the linux dep
@# target dropped the binary. We could build the binary in the container too
@# but that might take longer as caching gets weird
@# Lastly we map the source dir here to the /consul workdir
@echo "Running tests within a docker container"
@docker run -ti --rm \
-e 'GOTEST_FLAGS=$(GOTEST_FLAGS)' \
-e 'GOTEST_PKGS=$(GOTEST_PKGS)' \
-e 'GOTAGS=$(GOTAGS)' \
-e 'GIT_COMMIT=$(GIT_COMMIT)' \
Pass the git commits year through as an env var to the ui build (#7126)
2020-01-24 15:40:14 +00:00
-e 'GIT_COMMIT_YEAR=$(GIT_COMMIT_YEAR)' \
`make test-docker` (#6059) * Implement the test-docker make target Running tests within docker allows us to resource constrain them better to not take over our systems. Additionally it allows us to run the tests on linux instead of the host OS which often times is macOS. * Use GOMAXPROCS instead of -p * Add a comment about docker cpus
2019-07-04 14:22:59 +00:00
-e 'GIT_DIRTY=$(GIT_DIRTY)' \
-e 'GIT_DESCRIBE=$(GIT_DESCRIBE)' \
$(TEST_PARALLELIZATION) \
$(TEST_DOCKER_RESOURCE_CONSTRAINTS) \
$(TEST_MODCACHE_VOL) \
$(TEST_BUILDCACHE_VOL) \
-v $(MAIN_GOPATH)/bin/linux_amd64/:/go/bin \
-v $(shell pwd):/consul \
$(GO_BUILD_TAG) \
make test-internal
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
2017-09-25 18:40:42 +00:00
other-consul:
@echo "--> Checking for other consul instances"
@if ps -ef | grep 'consul agent' | grep -v grep ; then \
echo "Found other running consul agents. This may affect your tests." ; \
exit 1 ; \
fi
Manage dependencies via Godep Embrace the future and use Go 1.6's vendor support via Godep. Go 1.5 users should `export GO15VENDOREXPERIMENT=1`
2016-02-13 00:50:37 +00:00
cover:
build: Simplify make test and log output
2017-04-26 22:47:57 +00:00
go test $(GOFILES) --cover
consul: adding basic skeleton
2013-12-06 23:43:07 +00:00
Manage dependencies via Godep Embrace the future and use Go 1.6's vendor support via Godep. Go 1.5 users should `export GO15VENDOREXPERIMENT=1`
2016-02-13 00:50:37 +00:00
format:
Add format task to all and silence large output
2014-05-06 05:45:54 +00:00
@echo "--> Running go fmt"
build: fix makefile and CI 'fmt' and 'vet' tasks (#6937)
2019-12-11 18:32:11 +00:00
@go fmt ./...
@cd api && go fmt ./... | sed 's@^@api/@'
@cd sdk && go fmt ./... | sed 's@^@sdk/@'
Add make format task
2014-05-06 05:38:21 +00:00
Makefile: add vet target Add a vet target in order to catch suspicious constructs reported by go vet. Vet has successfully detected problems in the past, for example, see c9333b1b9b472feb5cad80e2c8276d41b64bde88 Some vet flags are noisy. In particular, the following flags reports a large amount of generally unharmful constructs: ``` -assign: check for useless assignments -composites: check that composite literals used field-keyed elements -shadow: check for shadowed variables -shadowstrict: whether to be strict about shadowing -unreachable: check for unreachable code ``` In order to skip running the flags mentioned above, vet is invoked on a directory basis with `go tool vet .` since package- level type-checking with `go vet` doesn't accept flags. Hence, each file is vetted in isolation, which is weaker than package-level type-checking. But nevertheless, it might catch suspicious constructs that pose a real issue. The vet target runs the following flags on the entire repo: ``` -asmdecl: check assembly against Go declarations -atomic: check for common mistaken usages of the sync/atomic package -bool: check for mistakes involving boolean operators -buildtags: check that +build tags are valid -copylocks: check that locks are not passed by value -methods: check that canonically named methods are canonically defined -nilfunc: check for comparisons between functions and nil -printf: check printf-like invocations -rangeloops: check that range loop variables are used correctly -shift: check for useless shifts -structtags: check that struct field tags have canonical format and apply to exported fields as needed -unsafeptr: check for misuse of unsafe.Pointer ``` Now and then, it might make sense to check the output of the disabled flags manually. For example, `VETARGS=-unreachable make vet` can detect several lines of dead code that can be deleted, etc.
2015-01-17 06:44:10 +00:00
vet:
Modernize makefile a bit
2017-03-23 23:06:25 +00:00
@echo "--> Running go vet"
build: fix makefile and CI 'fmt' and 'vet' tasks (#6937)
2019-12-11 18:32:11 +00:00
@go vet -tags '$(GOTAGS)' ./... && \
(cd api && go vet -tags '$(GOTAGS)' ./...) && \
(cd sdk && go vet -tags '$(GOTAGS)' ./...); if [ $$? -ne 0 ]; then \
Makefile: add vet target Add a vet target in order to catch suspicious constructs reported by go vet. Vet has successfully detected problems in the past, for example, see c9333b1b9b472feb5cad80e2c8276d41b64bde88 Some vet flags are noisy. In particular, the following flags reports a large amount of generally unharmful constructs: ``` -assign: check for useless assignments -composites: check that composite literals used field-keyed elements -shadow: check for shadowed variables -shadowstrict: whether to be strict about shadowing -unreachable: check for unreachable code ``` In order to skip running the flags mentioned above, vet is invoked on a directory basis with `go tool vet .` since package- level type-checking with `go vet` doesn't accept flags. Hence, each file is vetted in isolation, which is weaker than package-level type-checking. But nevertheless, it might catch suspicious constructs that pose a real issue. The vet target runs the following flags on the entire repo: ``` -asmdecl: check assembly against Go declarations -atomic: check for common mistaken usages of the sync/atomic package -bool: check for mistakes involving boolean operators -buildtags: check that +build tags are valid -copylocks: check that locks are not passed by value -methods: check that canonically named methods are canonically defined -nilfunc: check for comparisons between functions and nil -printf: check printf-like invocations -rangeloops: check that range loop variables are used correctly -shift: check for useless shifts -structtags: check that struct field tags have canonical format and apply to exported fields as needed -unsafeptr: check for misuse of unsafe.Pointer ``` Now and then, it might make sense to check the output of the disabled flags manually. For example, `VETARGS=-unreachable make vet` can detect several lines of dead code that can be deleted, etc.
2015-01-17 06:44:10 +00:00
echo ""; \
echo "Vet found suspicious constructs. Please check the reported constructs"; \
Modernize makefile a bit
2017-03-23 23:06:25 +00:00
echo "and fix them if necessary before submitting the code for review."; \
exit 1; \
Makefile: add vet target Add a vet target in order to catch suspicious constructs reported by go vet. Vet has successfully detected problems in the past, for example, see c9333b1b9b472feb5cad80e2c8276d41b64bde88 Some vet flags are noisy. In particular, the following flags reports a large amount of generally unharmful constructs: ``` -assign: check for useless assignments -composites: check that composite literals used field-keyed elements -shadow: check for shadowed variables -shadowstrict: whether to be strict about shadowing -unreachable: check for unreachable code ``` In order to skip running the flags mentioned above, vet is invoked on a directory basis with `go tool vet .` since package- level type-checking with `go vet` doesn't accept flags. Hence, each file is vetted in isolation, which is weaker than package-level type-checking. But nevertheless, it might catch suspicious constructs that pose a real issue. The vet target runs the following flags on the entire repo: ``` -asmdecl: check assembly against Go declarations -atomic: check for common mistaken usages of the sync/atomic package -bool: check for mistakes involving boolean operators -buildtags: check that +build tags are valid -copylocks: check that locks are not passed by value -methods: check that canonically named methods are canonically defined -nilfunc: check for comparisons between functions and nil -printf: check printf-like invocations -rangeloops: check that range loop variables are used correctly -shift: check for useless shifts -structtags: check that struct field tags have canonical format and apply to exported fields as needed -unsafeptr: check for misuse of unsafe.Pointer ``` Now and then, it might make sense to check the output of the disabled flags manually. For example, `VETARGS=-unreachable make vet` can detect several lines of dead code that can be deleted, etc.
2015-01-17 06:44:10 +00:00
fi
Fixes checked in web assets and associated build scripts. (#3173)
2017-06-21 21:43:07 +00:00
# If you've run "make ui" manually then this will get called for you. This is
# also run as part of the release build script when it verifies that there are no
# changes to the UI assets that aren't checked in.
Manage dependencies via Godep Embrace the future and use Go 1.6's vendor support via Godep. Go 1.5 users should `export GO15VENDOREXPERIMENT=1`
2016-02-13 00:50:37 +00:00
static-assets:
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
@go-bindata-assetfs -pkg agent -prefix pkg -o $(ASSETFS_PATH) ./pkg/web_ui/...
fix remaining CI failures after Go 1.12.1 Upgrade (#5576)
2019-03-29 15:29:27 +00:00
@go fmt $(ASSETFS_PATH)
agent: compile web assets into consul binary
2015-11-30 19:24:08 +00:00
Add capability to put tree back into dev mode via make dev-tree
2018-06-15 12:00:12 +00:00
# Build the static web ui and build static assets inside a Docker container
Remove old UI, option to use it, and its build processes
2019-04-12 15:02:27 +00:00
ui: ui-docker static-assets-docker
Add capability to put tree back into dev mode via make dev-tree
2018-06-15 12:00:12 +00:00
Add a tools target that fetches various build-time tools
2016-02-13 01:09:18 +00:00
tools:
build: prevent 'make tools' from editing go.mod and go.sum (#6738)
2019-11-05 15:25:46 +00:00
@mkdir -p .gotools
@cd .gotools && if [[ ! -f go.mod ]]; then \
go mod init consul-tools ; \
fi
cd .gotools && go get -v $(GOTOOLS)
Add a tools target that fetches various build-time tools
2016-02-13 01:09:18 +00:00
Redo the build system Improvements: - More modular - Building within docker doesn’t use volumes so can be run on a remote docker host - Build containers include only minimal context so they only rarely need to be rebuilt and most of the time can be used from the cache. - 3 build containers instead of 1. One based off of the upstream golang containers for building go stuff with all our required GOTOOLS installed. One like the old container based off ubuntu bionic for building the old UI (didn’t bother creating a much better container as this shouldn’t be needed once we completely remove the legacy UI). One for building the new UI. Its alpine based with all the node, ember, yarn stuff installed. - Top level makefile has the ability to do a container based build without running make dist - Can build for arbitrary platforms at the top level using: make consul-docker XC_OS=… XC_ARCH=… - overridable functionality to allow for customizations to the enterprise build (like to generate multiple binaries) - unified how we compile our go. always use gox even for dev-builds or rather always use the tooling around our scripts which will make sure things get copied to the correct places throughout the filesystem.
2018-06-12 20:55:52 +00:00
version:
Add more functionality related to verifying a build and publishing
2018-06-13 19:10:02 +00:00
@echo -n "Version: "
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh
Add more functionality related to verifying a build and publishing
2018-06-13 19:10:02 +00:00
@echo -n "Version + release: "
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh -r
Add more functionality related to verifying a build and publishing
2018-06-13 19:10:02 +00:00
@echo -n "Version + git: "
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh -g
Add more functionality related to verifying a build and publishing
2018-06-13 19:10:02 +00:00
@echo -n "Version + release + git: "
Add capability to put tree back into dev mode via make dev-tree
2018-06-15 12:00:12 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh -r -g
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
Remove old UI, option to use it, and its build processes
2019-04-12 15:02:27 +00:00
docker-images: go-build-image ui-build-image
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
go-build-image:
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@echo "Building Golang build container"
@docker build $(NOCACHE) $(QUIET) --build-arg 'GOTOOLS=$(GOTOOLS)' -t $(GO_BUILD_TAG) - < build-support/docker/Build-Go.dockerfile
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
ui-build-image:
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@echo "Building UI build container"
@docker build $(NOCACHE) $(QUIET) -t $(UI_BUILD_TAG) - < build-support/docker/Build-UI.dockerfile
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
static-assets-docker: go-build-image
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/build-docker.sh static-assets
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
Redo the build system Improvements: - More modular - Building within docker doesn’t use volumes so can be run on a remote docker host - Build containers include only minimal context so they only rarely need to be rebuilt and most of the time can be used from the cache. - 3 build containers instead of 1. One based off of the upstream golang containers for building go stuff with all our required GOTOOLS installed. One like the old container based off ubuntu bionic for building the old UI (didn’t bother creating a much better container as this shouldn’t be needed once we completely remove the legacy UI). One for building the new UI. Its alpine based with all the node, ember, yarn stuff installed. - Top level makefile has the ability to do a container based build without running make dist - Can build for arbitrary platforms at the top level using: make consul-docker XC_OS=… XC_ARCH=… - overridable functionality to allow for customizations to the enterprise build (like to generate multiple binaries) - unified how we compile our go. always use gox even for dev-builds or rather always use the tooling around our scripts which will make sure things get copied to the correct places throughout the filesystem.
2018-06-12 20:55:52 +00:00
consul-docker: go-build-image
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/build-docker.sh consul
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
Initial progress on build system updates
2018-06-08 14:20:54 +00:00
ui-docker: ui-build-image
Update the scripting Automated putting the source tree into release mode.
2018-06-15 01:25:59 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/build-docker.sh ui
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
Connect: allow configuring Envoy for L7 Observability (#5558) * Add support for HTTP proxy listeners * Add customizable bootstrap configuration options * Debug logging for xDS AuthZ * Add Envoy Integration test suite with basic test coverage * Add envoy command tests to cover new cases * Add tracing integration test * Add gRPC support WIP * Merged changes from master Docker. get CI integration to work with same Dockerfile now * Make docker build optional for integration * Enable integration tests again! * http2 and grpc integration tests and fixes * Fix up command config tests * Store all container logs as artifacts in circle on fail * Add retries to outer part of stats measurements as we keep missing them in CI * Only dump logs on failing cases * Fix typos from code review * Review tidying and make tests pass again * Add debug logs to exec test. * Fix legit test failure caused by upstream rename in envoy config * Attempt to reduce cases of bad TLS handshake in CI integration tests * bring up the right service * Add prometheus integration test * Add test for denied AuthZ both HTTP and TCP * Try ANSI term for Circle
2019-04-29 16:27:57 +00:00
test-envoy-integ: $(ENVOY_INTEG_DEPS)
@$(SHELL) $(CURDIR)/test/integration/connect/envoy/run-tests.sh
Add CI test-integrations job for connect Vault CA provider (#6949) * vault-ca-provider: add make target and CI test-integration job for /agent/connect/ca/ Vault Provider tests * load env vars in the vault-ca-provider ci job * add checkout task, see if we need to attach workspace or not * ci: remove vault dependency from go-test job to ensure we only run the vault-provider tests in their job * ci: fix from RB and Alvin code review, add mod cache after checkout * ci: add CI context to make target and store test results * ci: fix whitespace * ci: create test results directory before we try to write to it
2019-12-17 21:22:32 +00:00
test-vault-ca-provider:
ifeq ("$(CIRCLECI)","true")
# Run in CI
test: run both the TestVaultProvider and TestVaultCAProvider tests in CI (#7097)
2020-01-21 18:01:35 +00:00
gotestsum --format=short-verbose --junitfile "$(TEST_RESULTS_DIR)/gotestsum-report.xml" -- $(CURDIR)/agent/connect/ca/* -run 'TestVault(CA)?Provider'
Add CI test-integrations job for connect Vault CA provider (#6949) * vault-ca-provider: add make target and CI test-integration job for /agent/connect/ca/ Vault Provider tests * load env vars in the vault-ca-provider ci job * add checkout task, see if we need to attach workspace or not * ci: remove vault dependency from go-test job to ensure we only run the vault-provider tests in their job * ci: fix from RB and Alvin code review, add mod cache after checkout * ci: add CI context to make target and store test results * ci: fix whitespace * ci: create test results directory before we try to write to it
2019-12-17 21:22:32 +00:00
else
# Run locally
test: run both the TestVaultProvider and TestVaultCAProvider tests in CI (#7097)
2020-01-21 18:01:35 +00:00
@echo "Running /agent/connect/ca TestVault(CA)?Provider tests in verbose mode"
@go test $(CURDIR)/agent/connect/ca/* -run 'TestVault(CA)?Provider' -v
Add CI test-integrations job for connect Vault CA provider (#6949) * vault-ca-provider: add make target and CI test-integration job for /agent/connect/ca/ Vault Provider tests * load env vars in the vault-ca-provider ci job * add checkout task, see if we need to attach workspace or not * ci: remove vault dependency from go-test job to ensure we only run the vault-provider tests in their job * ci: fix from RB and Alvin code review, add mod cache after checkout * ci: add CI context to make target and store test results * ci: fix whitespace * ci: create test results directory before we try to write to it
2019-12-17 21:22:32 +00:00
endif
Use encoding/json instead of jsonpb even for protobuf types (#6572) This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 19:32:15 +00:00
proto-delete:
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
@echo "Removing $(PROTOGOFILES)"
-@rm $(PROTOGOFILES)
Generate JSON and Binary Marshalers for Protobuf Types (#6564) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 19:39:20 +00:00
@echo "Removing $(PROTOGOBINFILES)"
-@rm $(PROTOGOBINFILES)
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
Use encoding/json instead of jsonpb even for protobuf types (#6572) This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 19:32:15 +00:00
proto-rebuild: proto-delete proto
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
Use encoding/json instead of jsonpb even for protobuf types (#6572) This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 19:32:15 +00:00
proto: $(PROTOGOFILES) $(PROTOGOBINFILES)
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
@echo "Generated all protobuf Go files"
Generate JSON and Binary Marshalers for Protobuf Types (#6564) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 19:39:20 +00:00
Use encoding/json instead of jsonpb even for protobuf types (#6572) This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 19:32:15 +00:00
%.pb.go %.pb.binary.go: %.proto
Generate JSON and Binary Marshalers for Protobuf Types (#6564) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions
2019-09-30 19:39:20 +00:00
@$(SHELL) $(CURDIR)/build-support/scripts/proto-gen.sh --grpc --import-replace "$<"
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
2019-09-20 18:37:22 +00:00
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 16:04:07 +00:00
remove remaining references to govendor and vendorfmt (#5587)
2019-04-01 14:55:48 +00:00
.PHONY: all ci bin dev dist cov test test-ci test-internal test-install-deps cover format vet ui static-assets tools
Remove old UI, option to use it, and its build processes
2019-04-12 15:02:27 +00:00
.PHONY: docker-images go-build-image ui-build-image static-assets-docker consul-docker ui-docker
Use encoding/json instead of jsonpb even for protobuf types (#6572) This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.
2019-10-02 19:32:15 +00:00
.PHONY: version proto proto-rebuild proto-delete test-envoy-integ