Commit Graph

42 Commits

Author SHA1 Message Date
Michael Schurter 4ae115dc59 Allow custom ports for services and checks
Fixes #3380

Adds address_mode to checks (but no auto) and allows services and checks
to set literal port numbers when using address_mode=driver.

This allows SDNs, overlays, etc to advertise internal and host addresses
as well as do checks against either.
2017-12-08 12:03:00 -08:00
Jens Herrmann 5680fcccc2 Fix typos in metric names. #3610 2017-12-01 15:24:14 +01:00
Michael Schurter 0aace3d749 Don't set Interval on TTL health checks 2017-10-16 17:35:47 -07:00
Alex Dadgar 4173834231 Enable more linters 2017-09-26 15:26:33 -07:00
Michael Schurter 573a0df03d Watched -> TriggersRestart
Watched was a silly name
2017-09-14 16:48:39 -07:00
Michael Schurter 22690c5f4c Add check watcher for restarting unhealthy tasks 2017-09-14 16:46:54 -07:00
Michael Schurter bb8d5689d8 Add Header and Method support for HTTP checks 2017-08-17 16:44:21 -07:00
Alex Dadgar 6e20acb503 Merge pull request #2984 from hashicorp/b-tags
Fix alloc health with checks using interpolation
2017-08-10 13:07:25 -07:00
Alex Dadgar c8f74ac43b Address comments 2017-08-10 13:07:08 -07:00
Alex Dadgar d86b3977b9 Fix alloc health with checks using interpolation
Fixes an issue in which the allocation health watcher was checking for
allocations health based on un-interpolated services and checks. Change
the interface for retrieving check information from Consul to retrieving
all registered services and checks by allocation. In the future this
will allow us to output nicer messages.

Fixes https://github.com/hashicorp/nomad/issues/2969
2017-08-07 16:27:08 -07:00
Luke Farnell f0ced87b95 fixed all spelling mistakes for goreport 2017-08-07 17:13:05 -04:00
Michael Schurter 5794e5ece7 Use int32 for atomic ops to avoid alignment issues
From https://golang.org/pkg/sync/atomic/#pkg-note-BUG :

On both ARM and x86-32, it is the caller's responsibility to arrange for
64-bit alignment of 64-bit words accessed atomically. The first word in
a global variable or in an allocated struct or slice can be relied upon
to be 64-bit aligned.
2017-08-04 10:14:16 -07:00
Michael Schurter d2f8fdcad5 Fix comment 2017-07-25 12:13:05 -07:00
Michael Schurter 4b83eba599 Use seen more conservatively 2017-07-24 16:48:40 -07:00
Michael Schurter cdf138eb27 Always increment failures...
...as it's used in calculating the backoff
2017-07-24 15:37:53 -07:00
Michael Schurter 809724ad8d Track whether Consul has ever been seen
Need a way to squelch Consul operation errors on shutdown. If it's never
been seen don't log errors about deregs failing.
2017-07-24 12:12:02 -07:00
Michael Schurter edbe62a879 Synchronously deregister agent on shutdown
Fixes #2891

Previously the agent services and checks were being asynchrously
deregistered on shutdown, so it was a race between the sync goroutine
deregistering them and Nomad shutting down.

This switches to synchronously deregister agent serivces and checks
which doesn't really have a downside since the sync goroutines retry
behavior doesn't help on shutdown anyway.
2017-07-24 11:40:37 -07:00
Michael Schurter 99d1486f32 Never remove unknown agent services
Fixes #2827

This is a tradeoff. The pro is that you can run separate client and
server agents on the same node and advertise both. The con is that if a
Nomad agent crashes and isn't restarted on that node in the same mode
its entry will not be cleaned up.

That con scenario seems far less likely to occur than the scenario on
the pro side, and even if we do leak an agent entry the checks will be
failing so nothing should attempt to use it.
2017-07-18 13:23:01 -07:00
Alex Dadgar bf2dafb8e9 check id method name changed 2017-07-07 12:15:09 -07:00
Alex Dadgar 067ed86a47 Client watches for allocation health using task state and Consul checks
This PR adds watching of allocation health at the client. The client can
watch for health based on the tasks running on time and also based on
the consul checks passing.
2017-07-07 12:10:04 -07:00
Michael Schurter 9da78ae25f Remove debug logging 2017-06-21 17:19:08 -07:00
Michael Schurter c0eff81383 Fix Service.AddressMode changes during task updates 2017-06-21 17:19:08 -07:00
Michael Schurter 67d154a274 Test driver network advertisement and checks 2017-06-21 17:19:08 -07:00
Michael Schurter b9bfb84b53 Implement DriverNetwork and Service.AddressMode
Ideally DriverNetwork would be fully populated in Driver.Prestart, but
Docker doesn't assign the container's IP until you start the container.

However, it's important to setup the port env vars before calling
Driver.Start, so Prestart should populate that.
2017-06-21 17:19:08 -07:00
Michael Schurter cafefa049b Properly interpolate services on updated tasks
Previously was interpolating the original task's services again.

Fixes #2180

Also fixes a slight memory leak in the new consul agent. Script check
handles weren't being deleted after cancellation.
2017-04-26 11:22:01 -07:00
Michael Schurter e997ae44a5 Skip checks with TLSSkipVerify if it's unsupported
Fixes #2218
2017-04-19 12:45:34 -07:00
Michael Schurter 947e31e9c2 Only register HTTPS agent check when Consul>=0.7.2
Support for TLSSkipVerify in other checks coming soon!
2017-04-19 12:42:48 -07:00
Michael Schurter 383f21559e Explain weird timer logic 2017-04-19 12:42:48 -07:00
Michael Schurter 4db840b54e Metricsify new Consul client 2017-04-19 12:42:48 -07:00
Michael Schurter 2a11508cc8 Rework to account for ports not being in IDs
Previous implementation assumed all struct fields were included in
service and check IDs. Service IDs never include port labels and check
IDs *optionally* include port labels, so lots of things had to change.

Added a really big test to exercise this.
2017-04-19 12:42:47 -07:00
Michael Schurter 6e0fd86361 Remove commits return value
...and still protect against leaking agent entries in Consul on
shutdown.
2017-04-19 12:42:47 -07:00
Michael Schurter fb52bbfa45 Explain PortLabel handling in RegisterAgent 2017-04-19 12:42:47 -07:00
Michael Schurter 2728216f6f Plumb alloc id + task name into script check logs 2017-04-19 12:42:47 -07:00
Michael Schurter 01b20cbb59 Stop being lazy and just type out struct{}{} 2017-04-19 12:42:47 -07:00
Michael Schurter 7c8ad71da5 Fix comment to reflect reality 2017-04-19 12:42:47 -07:00
Michael Schurter 745ad9521f Move ScriptExecutor to driver 2017-04-19 12:42:47 -07:00
Michael Schurter 926d141532 Fix shutdown when consul is down 2017-04-19 12:42:47 -07:00
Michael Schurter 87b28cfb75 Switch ServiceClient to synchronizing state
Previously it applied a stream of operations. Reconciling state is less
complex and error prone at the cost of slightly higher CPU/memory usage.
2017-04-19 12:42:47 -07:00
Michael Schurter 244251490a Add UpdateTask method instead of Remove/Add 2017-04-19 12:42:47 -07:00
Michael Schurter 8118c41931 Remove some lies 2017-04-19 12:42:47 -07:00
Michael Schurter 0143bef8c7 Remove unused syncInterval 2017-04-19 12:42:47 -07:00
Michael Schurter e204a287ed Refactor Consul Syncer into new ServiceClient
Fixes #2478 #2474 #1995 #2294

The new client only handles agent and task service advertisement. Server
discovery is mostly unchanged.

The Nomad client agent now handles all Consul operations instead of the
executor handling task related operations. When upgrading from an
earlier version of Nomad existing executors will be told to deregister
from Consul so that the Nomad agent can re-register the task's services
and checks.

Drivers - other than qemu - now support an Exec method for executing
abritrary commands in a task's environment. This is used to implement
script checks.

Interfaces are used extensively to avoid interacting with Consul in
tests that don't assert any Consul related behavior.
2017-04-19 12:42:47 -07:00