Our testing so far indicates that ugorji/go/codec maintains backward
compatiblity with the version we are using now, for purposes of Nomad
serialization.
Using latest ugorji/go allows us to get back to using upstream library,
get get the optimizations benefits in RPC paths (including code
generation optimizations).
ugorji/go introduced two significant changes:
* time binary format in debb8e2d2e. Setting `h.BasicHandle.TimeNotBuiltin = true` restores old behavior
* ugorji/go started honoring `json` tag as well:
v1.1.4 is the latest but has a bug in handling RawString that's fixed in
d09a80c1e0
.
We run with ~120 apt sources, and apt-get update fails if any of them is
down.
True errors would be raised again at install phase as true dependencies
fetch would fail.
;
The current Travis setup scripts copy in rkt, but do not set up a
default container network.
Here we copy the container network setup over from the vagrant setup
scripts.
The changes introduces in #3256 require at least rkt 1.27.0 because of
a bug in the JSON output of `rkt status` in previous versions.
Here we upgrade all references to rkt's minimum version, and also make
travis and vagrant use this version when running tests.
Finally we add a CHANGELOG notice.
This commit reworks the Vagrantfile for Nomad in order to support
straightforward testing on more than one operating system, whilst
retaining the ability to stand up a test cluster running Ubuntu.
The following changes are made:
- Scripts have been extracted from the Vagrantfile into their own shell
script files, in order that editors lint them.
- All scripts have been edited to lint with no warnings or errors for
their respective shells.
- Scripts are named according to the operating system and privilege
level which they run. We prefer to run a whole shell script as root
versus prefixing (essentially) every command with `sudo` or an
equivalent.
- The Linux development box has been separated from the test cluster,
removing some of the more gnarly (and less portable) logic. The Linux
development box is still primary and autostarts.
- A FreeBSD target has been added. The base box works for both
Virtualbox and VMWare Fusion.
- A target is added to the GNUmakefile to stand up a test cluster, using
the default provider, or overriding the provider by setting the PROVIDER
variable in make:
- `make testcluster`
- `make testcluster PROVIDER=vmware_fusion`
- Machines in the test cluster have Avahi configured for zeroconf
discovery. Each machine can ping each other machine at `hostname.local`
- for example `nomad-server02.local`, `nomad-client03.local`.
This commit replaces the shell script-driven build process for Nomad
with one based around GNU Make (note we _do_ use GNU-specific
constructs), requiring no additional scripts for common cases of
development. The following targets are implemented:
Per-OS/arch combinations:
Binaries (Host - Mac OS X):
pkg/darwin_amd64/nomad
Binaries (Host - Linux):
pkg/linux_386/nomad
pkg/linux_amd64/nomad
pkg/linux_amd64-lxc/nomad
pkg/linux_arm/nomad
pkg/linux_arm64/nomad
pkg/windows_386/nomad
pkg/windows_amd64/nomad
Packages (Host - Mac OS X):
pkg/darwin_amd64.zip
Packages (Host - Linux):
pkg/linux_386.zip
pkg/linux_amd64.zip
pkg/linux_amd64-lxc.zip
pkg/linux_arm.zip
pkg/linux_arm64.zip
pkg/windows_386.zip
pkg/windows_amd64.zip
Phony targets:
dev - Builds for the current host GOOS/GOARCH (unless overriden
in the environment)
release - Builds all appropriate release packages for the
current host GOOS/GOARCH (i.e. Windows and Linux
packages on a Linux host, Darwin packages on an OSX
host)
generate - Generate code for the current host architecture using
`go generate`.
test - Runs the Nomad test suite
clean - Removes build artifacts
travis - Runs `make test` with the wrapper script to prevent
Travis CI from timing out.
help - Displays usage information about commonly used targets.
Note that there are some semantic differences from the previous version.
1. `generate` is no longer a dependency of `dev` builds. This is because
it causes a rebuild every time, even when no code has changed, since
`go generate` does not appear to leave file timestamps alone.
Regardless, it is insufficient to generate on one host OS - it needs
to be run on each target to ensure everything is generated correctly.
2. `gofmt` is no longer checked. This should be enabled as a linter once
the `gofmt -s` refactoring will pass on the whole code base, in order
to avoid special cased checks versus using go-metalinter.
Example Usages:
Make a development build for the current GOOS/GOARCH:
make dev
Make release build packages appropriate for the host OS:
make release
Update generated code for the host OS:
make generate
Run linting checks:
make check
Build a specific alternative GOOS/GOARCH/tags combination:
make pkg/linux_amd64-pkg/nomad
make pkg/linux_amd64-pkg.zip
* alloc_runner
* Random tests
* parallel task_runner and no exec compatible check
* Parallel client
* Fail fast and use random ports
* Fix docker port mapping
* Make concurrent pull less timing dependant
* up parallel
* Fixes
* don't build chroots in parallel on travis
* Reduce parallelism on travis with lxc/rkt
* make java test app not run forever
* drop parallelism a little
* use docker ports that are out of the os's ephemeral port range
* Limit even more on travis
* rkt deadline
The dev build is far simpler than the release build, so move it to its
own shell script. This simplifies the release build script slightly as
well at the cost of duplicating the version/tag logic.
Also don't even try to check for LXC if not running on Linux. I don't
think we want to try to support cross-compiling LXC from non-Linux
hosts.
Since lxc support requires linking to a C lib at compile and runtime
I'm putting it behind a build flag to avoid forcing all nomad users to
install liblxc (lxc-dev for development).
Tests are approaching 30min largely due to recompiling every package.
Out of a ~30min build, only ~5min are actually spent running the tests,
the remaining time is spent recompiling.
Mahmood Ali