Commit Graph

178 Commits

Author SHA1 Message Date
Mahmood Ali 94913d2ad6
Adopt go-changelog in Nomad (#10825)
Adopts [`go-changelog`](https://github.com/hashicorp/go-changelog) for managing Nomad's changelog. `go-changelog` is becoming the HashiCorp defacto standard tool for managing changelog, e.g. [Consul](https://github.com/hashicorp/consul/pull/8387), [Vault](https://github.com/hashicorp/vault/pull/10363), [Waypoint](https://github.com/hashicorp/waypoint/pull/1179). [Consul](https://github.com/hashicorp/consul/pull/8387) seems to be the first product to adopt it, and its PR has the most context - though I've updated `.changelog/README.md` with the relevant info here.

## Changes to developers workflow

When opening PRs, developers should add a changelog entry in `.changelog/<PR#>.txt`. Check [`.changelog/README.md`](https://github.com/hashicorp/nomad/blob/docs-adopt-gochangelog/.changelog/README.md#developer-guide). 

For the WIP release, entries can be amended even after the PR merged, and new files may be added post-hoc (e.g. during transition period, missed accidentally, community PRs, etc).

### Transitioning

Pending PRs can start including the changelog entry files immediately.

For 1.1.3/1.0.9 cycle, the release coordinator should create the entries for any PR that gets merged without a changelog entry file. They should also move any 1.1.3 entry in CHANGELOG.md to a changelog entry file, as this PR done for GH-10818.

## Changes to release process

Before cutting a release, release coordinator should update the changelog by inserting the output of `make changelog` to CHANGELOG.md with appropriate headers. See [`.changelog/README.md`](https://github.com/hashicorp/nomad/blob/docs-adopt-gochangelog/.changelog/README.md#how-to-generate-changelog-entries-for-release) for more details.


## Details

go-changelog is a basic templating engine for maintaining changelog in HashiCorp environment.

It expects the changelog entries as files indexed by their PR number. The CLI generates the changelog section for a release by comparing two git references (e.g. `HEAD` and the latest release, e.g. `v1.1.2`), and still requires manual process for updating CHANGELOG.md and final formatting.

The approach has many nice advantages:
* Avoids changelog related merge conflicts: Each PR touches different file!
* Copes with amendments and post-PR updates: Just add or update a changelog entry file using the original PR numbers.
* Addresses the release backporting scenario: Cherry-picking PRs will cherry-pick the relevant changelog entry automatically!
* Only relies on data available through `git` - no reliance on GitHub metadata or require GitHub credentials

The approach has few downsides though:
* CHANGELOG.md going stale during development and must be updated manually before cutting the release
  * Repository watchers can no longer glance at the CHANGELOG.md to see upcoming changes
  * We can periodically update the file, but `go-changelog` tool does not aid with that
* `go-changelog` tool does not offer good error reporting. If an entry is has an invalid tag (e.g. uses `release-note:bugfix` instead of `release-note:bug`), the entry will be dropped silently
  * We should update go-changelog to warn against unexpected entry tags
  * TODO: Meanwhile, PR reviewers and release coordinators should watch out

## Potential follow ups

We should follow up with CI checks to ensure PR changes include a warning. I've opted not to include that now. We still make many non-changelog-worth PRs for website/docs, for large features that get merged in multiple small PRs. I did not want to include a check that fails often.

Also, we should follow up to have `go-changelog` emit better warnings on unexpected tag.
2021-07-06 10:46:53 -04:00
Mahmood Ali b2f9234b30
Merge pull request #10296 from hashicorp/c-script-tweaks-20210402
makefile cleanup
2021-04-07 16:54:43 -04:00
Seth Hoenig 7a6a7e4da1 build: install buf during bootstrap
Previously installing buf was left out of `make bootstrap` because it
had conflicts with the `tools/go.mod` file and dependencies used by
other tools. With Go 1.16 we eliminated that `go.mod` file, and can
now just install `buf` with `go install` like everything else.
2021-04-06 09:42:44 -06:00
Yoan Blanc ac0d5d8bd3
chore: bump golangci-lint from v1.24 to v1.39
Signed-off-by: Yoan Blanc <yoan@dosimple.ch>
2021-04-03 09:50:23 +02:00
Mahmood Ali 7237785972 Tweak cross compilation Make
Only use cross compile if not compiling in target architecture.

Also, use the latest gcc for cross compilation.
2021-04-02 16:29:57 -04:00
Mahmood Ali 24fe595de4 dev: ignore msgpack codegen code in dev by default
This change disables using msgpack generated serializers in dev by
default.

In released binaries, we use code-generated msgpack serializers to
improve performance.  However, in development, code generated
serializers are a pain. If a developer forgets to re-generate code, the
code generated gets out of sync with the go structs, and result into
subtle bugs where some values appear not to persist as expected.

The CI and release scripts will continue to use the msgpack
code-generation. Devs who want to test locally can set
`GO_TAGS=codegen_generated` as well.
2021-04-01 16:27:18 -04:00
Mahmood Ali fb6d49cc88 use `go install` for tools
Golang 1.16 added better support for installing tools, without updating
go.mod files: https://blog.golang.org/go116-module-changes#TOC_4. .
2021-03-09 14:26:44 -05:00
Mahmood Ali 5a1df47e38 update buf compare tag to latest release 2021-01-31 20:23:22 -05:00
Mahmood Ali ab92d495fc lint: Upgrade buf to 0.36.0, latest so far 2021-01-31 20:09:15 -05:00
Drew Bailey 0fd70d2cb1
tmp remove darwin arm build (#9786) 2021-01-12 15:52:30 -05:00
Kris Hicks 325eeae542
makefile: Set CC explicitly in go build (#9784)
This is required because Go does not pull CC from the make variable. This uses
whatever Go's default CC unless CC is overridden, as it is for the ARM targets.

This also makes it easier to build Nomad on a native ARM device, via:

```
make CC= pkg/linux_arm/nomad
```
2021-01-12 12:09:40 -08:00
Kris Hicks f1b50bcbe7
makefile: Re-scope VERBOSE=true, update build warnings (#9675)
* Set 'only' ALL_TARGETS rather than append

This is functionally no different than before, but it's more correct.

* Re-scope VERBOSE=true

Previously this was only set when the OS was Linux; this was added in
805ade7d3.

* Warn about unsupported OS rather than error

Also:

* Only print the warning when trying to build Nomad
* Print correct list of supported OSes
2020-12-18 09:54:14 -08:00
Kris Hicks 0bdea4435f
makefile: DRY out pkg targets (#9643)
This removes small differences between the targets, like the statement
about what's being built.

The CGO/Windows related comments were deleted as being not relevant. 
See https://github.com/hashicorp/nomad/pull/9643 for context.
2020-12-16 08:01:50 -08:00
Seth Hoenig 83aa566a0c build: add build target for darwin/arm64
Add a build target for Apple Silicon (m1) macs.

Note that Go must have been built with c4f497da6f for
Nomad to work on darwin/arm64 (i.e. wait for go1.16).

Closes #9408
2020-12-15 10:57:50 -06:00
Kris Hicks ad97e1f243
Bump buf check breaking tag to 1.0.0 (#9629) 2020-12-14 09:07:51 -08:00
Steffen c93e6a00d2
on s390x, only compile compile release for s390x (#9601) 2020-12-14 08:27:19 -05:00
Mahmood Ali 8ca33b24f0
Merge pull request #9414 from hashicorp/b-tweak-buf-linter
Parameterize buf compatibility check
2020-11-25 12:19:10 -05:00
Nick Ethier 3483f540b6 api: don't break public API 2020-11-23 10:36:10 -06:00
Mahmood Ali 14fe89003b lint protobuf files 2020-11-20 11:00:24 -05:00
Mahmood Ali ecbae03cf3 Parameterize buf compatibility target
Parameterize it so we can arbitrary target other versions, if we
are doing some manual checking, specially in the beginning when we may
want to validate compatibilities for skip release upgrades.

Also, introduce `checkbuf` target so we can run buf linter without the
rest.

use beta
2020-11-20 11:00:11 -05:00
Kris Hicks 511c2e9db2
proto: Switch to using buf (#9308)
This replaces all usage of `protoc` with `buf`. See `tools/buf/README.md` for more.
2020-11-17 07:01:48 -08:00
Kris Hicks 9d03cf4c5f
protos: Update .proto files not to use Go package name (#9301)
Previously, it was required that you `go get github.com/hashicorp/nomad` to be
able to build protos, as the protoc invocation added an include directive that
pointed to `$GOPATH/src`, which is how dependent protos were discovered. As
Nomad now uses Go modules, it won't necessarily be cloned to `$GOPATH`.
(Additionally, if you _had_ go-gotten Nomad at some point, protoc compilation
would have possibly used the _wrong_ protos, as those wouldn't necessarily be
the most up-to-date ones.)

This change modifies the proto files and the `protoc` invocation to handle
discovering dependent protos via protoc plugin modifier statements that are
specific to the protoc plugin being used.

In this change, `make proto` was run to recompile the protos, which results in
changes only to the gzipped `FileDescriptorProto`.
2020-11-10 08:42:35 -08:00
Seth Hoenig 7a3d8b31da deps: pin a8m/tree to specific hash 2020-10-26 13:44:35 -05:00
Ben Buzbee a5b5681248 Use GOPATH/src explicitly in proto build target
-I ../../.. is meant to navigate from `GOPATH/src/github.com/hashicorp/nomad` to `GOPATH/src`

This is fine but it assumes a few things about how the dev has setup nomad, which is also fine if that is the expected dev environment, however the `../../..` is not as explicit as "GOPATH/src" and it would also enable a few more scenarios so it seems strictly better to me.

Random example: nomad is a subrepo of ours, but with this change we can symlink from GOPATH/src/github.com/hashicorp/nomad and `make proto` will work.
2020-10-15 21:05:27 +00:00
Tim Gross 5dd77d1dc8
ci: make split between e2e and integration tests explicit (#9091)
Currently we compile (but don't run) the e2e tests as part of `test-other`,
which is skipped for branches named `e2e-*`. Move this check into the
`test-e2e` job. Split out the vault compatibility integration check as its own
makefile target for clarity.
2020-10-14 14:53:21 -04:00
Tim Gross 65282a7cf1
E2E: vault secrets (#9081)
* rename vault API compatibility test for clarity
* exercise vault secrets lease renewal
2020-10-14 08:43:28 -04:00
Mahmood Ali fdeafaa9c2 Add a linter for raftutil message type mapping 2020-08-31 08:46:28 -04:00
Tim Gross dde66bd8f8
lint: check for go.mod or go.sum drift (#8765) 2020-08-27 16:21:07 -04:00
Drew Bailey d87c7d3917
use specific sha (#8329)
* use specific sha

* use submodule version
2020-07-01 10:32:45 -04:00
Drew Bailey 463f1986b1
add hclog vet to check step (#8328)
* add hclog vet to check step

* ...
2020-07-01 10:08:37 -04:00
Seth Hoenig 2803ddadab deps: remove tidy from dev makefile target
With Go modules, `go mod tidy` supplants `vendorfmt`. Unfortunately,
`tidy` will try to reach out to the network and download modules, and
there is no way to disable that behavior (e.g. the -mod=vendor) option
does not apply. This means we cannot use the `tidy` target in nomad
enterprise, which will be unable to reach private repositories like
consul-enterprise.

This isn't a big deal, since `vendorfmt` served the purpose of rewriting
the output of `govendor`, wheras `tidy` is a part of the `sync` target
that is required to be run when modifying dependencies anyway.
2020-06-18 08:49:12 -05:00
Mahmood Ali 6ce56a9561 Ensure we are using the latest go-bindata tools
go-bindata released tags are very old, few years old, so ensure we use
the latest master as of now.
2020-06-04 18:49:12 -04:00
Mahmood Ali 82c87cf97b
Vendor gopsutil with go mod fix (#8108)
Point to gopsutil fork with shirou/gopsutil#889 changes. I used hashicorp/gopsutil rather than pointing to https://github.com/Lomanic/gopsutil/tree/issue832 directly just in case..
2020-06-03 16:09:27 -04:00
Seth Hoenig 435c0d9fc8 deps: Switch to Go modules for dependency management
This PR switches the Nomad repository from using govendor to Go modules
for managing dependencies. Aspects of the Nomad workflow remain pretty
much the same. The usual Makefile targets should continue to work as
they always did. The API submodule simply defers to the parent Nomad
version on the repository, keeping the semantics of API versioning that
currently exists.
2020-06-02 14:30:36 -05:00
Seth Hoenig 889e7ddd0c build: use hashicorp hclfmt
We have been using fatih/hclfmt which is long abandoned. Instead, switch
to HashiCorp's own hclfmt implementation. There are some trivial changes in
behavior around whitespace.
2020-05-24 18:31:57 -05:00
Mahmood Ali 4dc72b6a20 makefile to support windows 2020-05-15 12:19:59 -04:00
Drew Bailey 6f6513ffd3
pin golangci-lint dep to 1.24.0 2020-05-13 10:43:39 -04:00
Mahmood Ali 18dba6fdad Harmonize go-msgpack/codec/codecgen
Use v1.1.5 of go-msgpack/codec/codecgen, so go-msgpack codecgen matches
the library version.

We branched off earlier to pick up
f51b518921
, but apparently that's not needed as we could customize the package via
`-c` argument.
2020-04-28 17:12:31 -04:00
Seth Hoenig f2ef576510 demo: create a demo service for grpc healthchecks
Examples for HTTP based task-group service healthchecks are
covered by the `countdash` demo, but gRPC checks currently
have no runnable examples.

This PR adds a trivial gRPC enabled application that provides
a Service implementing the standard gRPC healthcheck interface.
2020-04-24 10:59:50 -06:00
Tim Gross bad9a82df8
ci: add a linting check for HCL files (#7791)
Running `make dev` runs `hclfmt`, but this isn't checked as part of
CI. That makes it possible to merge un-formatted HCL and Nomad
jobspecs that later will make for dirty git staging areas when
developers pull master.

This changeset adds HCL linting to the `make check` target.
2020-04-23 14:32:44 -04:00
Tim Gross e7e9c83aa3
website: fix path for spellchecking and correct errors (#7790) 2020-04-23 10:38:08 -04:00
Mahmood Ali 030e40ac5c
Merge pull request #7652 from hashicorp/v-gomod-msgpaack
dev: Use go mod for managing hashicorp/go-msgpack
2020-04-08 14:42:39 -04:00
Mahmood Ali 00ea28b0cd go mod for go-bindata
Use go mod for github.com/hashicorp/go-bindata/go-bindata and
github.com/elazarl/go-bindata-assetfs/go-bindata-assetfs but use
`@master` to pull the latest master.  These packages don't have release
tags so `@master` worksaround it.
2020-04-08 14:30:37 -04:00
Tim Gross 457b83a967
build: remove working dir from stack traces (#7653)
Adding `-trimpath` to builds removes the local working directory from
the goroutine stack traces, which makes our builds more reproducible
and doesn't leak information about our local development workstations
or CI environment.
2020-04-07 13:32:07 -04:00
Mahmood Ali 89be4e23c3 dev: Use go mod for managing hashicorp/go-msgpack 2020-04-07 10:52:04 -04:00
Mahmood Ali 5378b557e9 Use latest go-bindata and go-bindata-assetfs
These dep tools aren't tagged properly so shouldn't be installed with go
mod
2020-03-31 20:28:25 -04:00
Mahmood Ali 68462d3553 dev: use go mod to install dev dependencies
This allows using https download and go mod cache proxies, over using
git and downloading entire dependencies git history, hopefully,
resulting into a faster installation process.
2020-03-30 21:32:45 -04:00
Michael Schurter 83890f019d
Merge pull request #7254 from jboero/patch-1
Completed a ppc64le build for testing.
2020-03-19 11:07:20 -07:00
JohnnyB 5e58b434ec
Added s390x and ppc64le
IBM testing.
2020-03-12 13:48:29 +00:00
Mahmood Ali b6f3effc5b ci: add a check for non-vendored packages
This check fails the lint-go CI job if nomad references a package that
is not vendored.
2020-03-09 15:25:00 -04:00