41b540fbc8
Allow operators to opt into publishing node and alloc metrics
2016-08-01 19:52:20 -07:00
777bdf4a1e
fix setup consul syncer error message
2016-07-28 22:14:52 -05:00
ebac5cb283
Node.Register handles the case of transistioning to ready and creating evals
2016-07-21 15:22:02 -07:00
5b39a5db40
Fixed a debug message
2016-07-09 00:12:53 -07:00
03c571c61b
Consolidate fingerprinters into a single `map`.
2016-07-08 23:37:14 -07:00
8bdb38d016
Code golf
...
Pointed out by: @dadgar
2016-06-21 14:26:01 -07:00
df4fe2e502
Fix the shuffling of remote datacenters.
...
Pointed out by: @ryanuber
2016-06-21 13:37:22 -07:00
9a60999100
Pass a logger arg to `NewClient` and `NewServer`
2016-06-16 23:29:23 -07:00
fd18eb7fdb
Only register the Client services reaper when `consul.auto_advertise` is enabled
2016-06-16 18:24:58 -07:00
952b6ce7b5
Only auto-join clients if `client_auto_join` is true
2016-06-16 14:47:21 -07:00
af55b74114
Merge pull request #1276 from hashicorp/f-consul-server-autojoin
...
Teach Nomad servers how to fall back to Consul.
2016-06-16 14:40:45 -07:00
008d75184b
Use the `%+q` verb in log messages (vs `%q`).
2016-06-16 11:03:51 -07:00
7375d828e1
remove trace
2016-06-15 15:47:59 -07:00
5e0ced2ae7
Shuffle all datacenters vs only the nearest N datacenters.
...
Per discussion, we want to be aggressive about fanning out vs possibly
fixating on only local DCs. With RPC forwarding in place, a random walk
may be less optimal from a network latency perspective, but it is guaranteed
to eventually result in a converged state because all DCs are candidates
during the bootstrapping process.
2016-06-15 12:40:51 -07:00
2123460cf0
Bump various Consul search limits
...
Client: Search limit increased from 4 random DCs to 8 random DCs, plus nearest.
Server: Search factor increased from 3 to 5 times the bootstrap_expect.
This should allow for faster convergence in large environments (e.g.
sub-5min for 10K Consul DCs).
2016-06-15 12:40:51 -07:00
cf99fc3173
Use Status.Peers instead of Status.Ping
2016-06-15 12:00:20 -07:00
4b04e503f3
address comments
2016-06-13 17:32:18 -07:00
8bbf4a55e5
Fix IDs and domain scoping
2016-06-13 16:30:58 -07:00
d019d8ef8e
implemented reconciliation of unwanted services
2016-06-13 14:52:26 +02:00
a82c2bb058
Do not reconcile in client and cleanup executor a bit
2016-06-12 18:22:07 -07:00
8e231fa382
Rename ConsulService back to Service
2016-06-12 16:36:49 -07:00
fdda90229f
only support latest and remove ring buffer
2016-06-12 09:32:38 -07:00
e952540f6f
Allocation resources returned in a struct
2016-06-11 21:04:10 -07:00
2f036231e5
Merge pull request #1201 from hashicorp/f-dyn-server-list
...
Dynamic Server Lists/Client Bootstrapping via consul.
2016-06-11 18:58:25 -04:00
92e2cfb0ad
Walk the DCs from nearest to most remote.
2016-06-11 18:52:21 -04:00
2968545201
Walk the DCs from nearest to most remote, no limit on the search.
2016-06-11 18:23:06 -04:00
917766a3df
Prefer `%+q` over `%q` in log messages.
2016-06-11 18:17:20 -04:00
fd60cfd585
Emitting client resource usage metrics as guages instead of k/v pairs
2016-06-11 22:17:32 +02:00
bbd8dfa798
goling(1) compliance pass (e.g. Rpc* -> RPC)
2016-06-10 23:38:28 -04:00
bc771d35df
Query for the Nomad service across multiple Consul datacenters.
2016-06-10 23:05:14 -04:00
26b1e826d7
golint(1) police
2016-06-10 15:54:39 -04:00
f139d0c68b
Properly guard consulPullHeartbeatDeadline behind heartbeatLock
2016-06-10 15:54:39 -04:00
ed29946f5e
Populate the RPC Proxy's server list if heartbeat did not include a leader.
...
It's possible that a Nomad Client is heartbeating with a Nomad server that
has become issolated from the quorum of Nomad Servers. When 3x the
heartbeatTTL has been exceeded, append the Consul server list to the primary
primary server list. When the next RPCProxy rebalance occurs, there is a
chance one of the servers discovered from Consul will be in the majority.
When client reattaches to a Nomad Server in the majority, it will include
a heartbeat and will reset the TTLs *AND* will clear the primary server list
to include only values from the heartbeat.
2016-06-10 15:54:39 -04:00
9a223936bb
Generate and sync Consul ServiceIDs consistently
2016-06-10 15:54:39 -04:00
7956eb0c80
Rename structs.Task's `Service` attribute to `ConsulService`
2016-06-10 15:54:39 -04:00
8c813630e6
Move package client/consul/sync to command/agent/consul.
...
This has been done to allow the Server and Client to reuse the same
Syncer because the Agent may be running Client, Server, or both
simultaneously and we only want one Syncer object alive in the agent.
2016-06-10 15:54:39 -04:00
fda03c5c9e
Change the signature of the PeriodicCallback to return an error
...
I *KNEW* I should have done this when I wrote it, but didn't want to
go back and audit the handlers to include the appropriate return
handling, but now that the code is taking shape, make this change.
2016-06-10 15:54:39 -04:00
555f4fe135
Change client/consul.NewSyncer() to accept a shutdown channel
...
In addition to the API changing, consul.Syncer can now be signaled
to shutdown via the Shutdown() method, which will call the Run()'ing
sync task to exit gracefully.
2016-06-10 15:54:39 -04:00
484816f5e0
Ensure that all accesses to Client.alloc are wrapped by allocLock.
2016-06-10 15:50:11 -04:00
08cab4fdfa
Use client.getAllocRunners() where appropriate.
2016-06-10 15:50:11 -04:00
f9d0b9da32
Line wrap long line.
2016-06-10 15:50:11 -04:00
0d201631a3
Rename rpcproxy.UpdateFromNodeUpdateResponse to RefreshServerLists
...
While breaking the API within this PR, break out the individual
arguments to RefreshServerLists. The servers parameter is reusing
`structs.NodeServerInfo` for the time being, but this can be revisited
if the needs of the strucutre diverge in the future.
2016-06-10 15:50:11 -04:00
0997fb1669
Fix up the comments
...
Pointed out by: @dadgar
2016-06-10 15:50:11 -04:00
aaa7d6bf40
Make the locking protocol more explicit in client.NewClient
...
With an over abundance of caution, preevnt future copy/pasta by
using the right locks when bootstrapping a Client. Strictly speaking
this is not necessary, but it makes explicit the locking semantics
and guards against future concurrent or parallel initialization.
2016-06-10 15:50:11 -04:00
525554c008
Use the client configCopy and lock appropriately.
2016-06-10 15:50:11 -04:00
3060d6b33c
Flesh out the comment re: the client.rpcproxy.Run() task.
...
Requested by: Alex
2016-06-10 15:50:11 -04:00
b1ee131db8
Rename `backupServerDeadline` to `consulPullHeartbeatDeadline`
...
Suggested by: @alex
2016-06-10 15:50:11 -04:00
b9adfcecf5
Remove unused variable
2016-06-10 15:50:11 -04:00
f15eeb8f27
Clean up some docs and comments to be more accurate
2016-06-10 15:50:11 -04:00
c78b0a6567
Remove unused constants
2016-06-10 15:50:11 -04:00
39fb0f2469
Change the endpoint for `/v1/agent/servers` and fix tests.
...
When an agent is running a server, the list of servers includes the
Raft peers. When the agent is running a client (which is always the
case?), include a list of the servers found in the Client's RpcProxy.
Dedupe and provide a unique list back to the caller.
2016-06-10 15:50:11 -04:00
cb80e93a6b
Move client.DefaultConfig() to client/config.DefaultConfig()
...
Resolves an import cycle in testing and is more appropriate because
the default should reside next to its struct definition.
2016-06-10 15:50:11 -04:00
bff57a0dce
Reconcile, clean up, and centralize API version numbers (major and minor).
...
Reduce future confusion by introducing a minor version that is gossiped out
via the `mvn` Serf tag (Minor Version Number, `vsn` is already being used for
to communicate `Major Version Number`).
Background: hashicorp/consul/issues/1346#issuecomment-151663152
2016-06-10 15:50:11 -04:00
36340c654d
Improve language re: fingerprinting
2016-06-10 15:50:11 -04:00
438becb28b
Pass the datacenter name in the heartbeat
...
Servers that are part of a different datacenter are added as backup
servers instead of primary servers.
2016-06-10 15:50:11 -04:00
d914f262f9
Consolidate all consul sync periodic go routines to handlers.
...
Only one pump and periodic loop now.
2016-06-10 15:50:11 -04:00
9fb0104def
Teach Client to reuse an Agent's consulSyncer.
...
"There can be only one."
2016-06-10 15:50:11 -04:00
47891fb559
Register two services each for clients and servers, http and rpc.
...
In order to give clients a fighting chance to talk to the right port,
differentiate RPC services from HTTP services by registering two
services with different tags. This yields
`rpc.nomad-server.service.consul` and
`http.nomad-server.service.consul` which is immensely more useful to
clients attempting to bootstrap their world.
2016-06-10 15:50:11 -04:00
861f900225
s/RpcVersion/RPCVersion/g
2016-06-10 15:50:11 -04:00
9f8d6c9f67
Move struct member to reduce diff context
2016-06-10 15:50:11 -04:00
410d85cc78
Rename the package from `client/rpc_proxy` to `client/rpcproxy`
...
Also rename `NewRpcProxy()` to just `New()` to avoid package stutter.
2016-06-10 15:50:11 -04:00
ec7c78f533
Only poll Consul for servers when Nomad heartbeats begin to fail
...
When a deadline timer of 2x Server's last requested TTL expires,
begin polling Consul for Nomad Servers.
2016-06-10 15:50:11 -04:00
17116fc5a7
Rebalance Nomad client RPCs among different Nomad servers.
...
Implement client/rpc_proxy.RpcProxy.
2016-06-10 15:50:11 -04:00
88f3422d7c
Rename NewConsulService to NewSyncer
2016-06-10 15:49:37 -04:00
768aab015d
Rename client/consul/sync.ConsulService to client/consul/sync.Syncer
...
Syncer describes the responsibility and actions of the type.
2016-06-10 15:49:37 -04:00
73e173673e
Rename client/config/config's ConsulConfig to ConsulAgentConfig
...
A follow up commit to the previous rename. More to come.
2016-06-10 15:48:36 -04:00
e36686a17d
Use consul/lib's RandomStagger
...
Removes four redundant copies of the method in the process.
2016-06-10 15:48:36 -04:00
6bf38bb213
Fixed the svc id generation scheme while pruning services
2016-06-07 08:14:11 -07:00
ba1a92eb8c
Handle errors during stats collection
2016-06-03 14:23:18 -07:00
35e31c1b81
Enqueing metrics only if they are not nil
2016-06-02 17:14:15 -04:00
7efde782fa
Sending metrics for tasks as well
2016-06-01 16:42:16 +02:00
7f172f314f
Pushing host stats
2016-05-31 04:05:47 +02:00
c0dc6cfbf2
Changing the api of the stats endpoints
2016-05-28 19:59:20 -07:00
a0c279f3b2
comments
2016-05-28 19:59:20 -07:00
4c85f02072
Stopping the metrics collector timers using defer and starting to collect host stats right away
2016-05-28 19:59:20 -07:00
b2e8495971
Fixing the alloc runner tests
2016-05-28 19:59:20 -07:00
f93820f40c
Added comments
2016-05-28 19:59:20 -07:00
c2da19bf11
Refactored the api for NewHostStatsCollector
2016-05-28 19:59:20 -07:00
c46400597e
Making the stats collection interval and number of data points to keep in memory configurable
2016-05-28 19:59:20 -07:00
b6e5227e7a
Renamed monitorUsage method
2016-05-28 19:59:20 -07:00
666b419dba
Acquiring locks before iterating allocations and tasks
2016-05-28 19:59:20 -07:00
73fa700a88
Showing host resource usage stats
2016-05-28 19:59:20 -07:00
9a96dddc07
Added some docs to resource stats endpoint
2016-05-28 19:42:34 -07:00
b9feae89ce
Making the conversion to Stats simpler
2016-05-28 19:42:34 -07:00
7569b1af2e
Collecting host stats
2016-05-28 19:42:34 -07:00
f3d0aecafe
Reporting time series of stats
2016-05-28 19:42:34 -07:00
0fb0e0237f
Added a client API to display resource usage of an allocation
2016-05-28 19:42:34 -07:00
2a99a2cfb6
Removing addition of the client service while reconciling services
2016-05-13 10:34:21 -07:00
439d7bf326
Fixed an agent test
2016-05-11 17:26:53 -07:00
347cb890d2
Removed allocID and task name from consul service
2016-05-11 16:26:41 -07:00
7287376eca
Using consul config from client config instead of reading from client options
2016-05-11 16:10:57 -07:00
83fed62a0a
Implemented registering client and server services
2016-05-11 16:07:02 -07:00
2f8a3532ad
Refactored the signature of NewConsulService
2016-05-11 15:22:58 -07:00
53d4681b61
Emit various debugging information with the results of the fingerprinter
2016-05-09 12:21:51 -07:00
2941b26244
Reading consul attr from copy of node attributes
2016-04-11 20:13:28 -04:00
f3d9ecf354
When reserving ports don't reserve network interface speed
2016-04-07 15:47:02 -07:00
57b0bbcb8b
Watching for node updates after registration completes
2016-04-01 13:41:52 -07:00
c6e80582a6
Making the drivers fingerprint periodically if they are configured to do so
2016-03-31 15:15:00 -07:00
e677c43667
Client not syncing services with consul until fingerprinting succeeds
2016-03-30 21:51:50 -07:00
b886636f6f
Using tickers instead of creating new timers
2016-03-25 14:18:04 -07:00
91db8f44f1
Changing the logic of keep services
2016-03-24 19:19:13 -07:00
2a9e522ed4
Added an impl for Nomad Checks
2016-03-24 19:00:24 -07:00
6a62d4f452
Fixing check registration in perform sync
2016-03-24 14:12:09 -07:00
bf554992a4
Using a helper method to copy taskStates
2016-03-23 19:11:54 -07:00
12ac0b4a33
Reworded the log line
2016-03-23 19:04:59 -07:00
e98f5e4ee3
Locking the task states
2016-03-23 19:02:29 -07:00
31baa6ce4b
Renamed vars and methods
2016-03-23 18:21:27 -07:00
092f23a646
Locking on alloc runners before syncing with consul
2016-03-23 17:54:32 -07:00
62242595fc
Using the name of the task and the alloc id in the service name
2016-03-23 17:35:29 -07:00
ab35c187b3
Added comments
2016-03-23 15:39:25 -07:00
7dab719a66
Client sync with consul and removed unwanted services
2016-03-23 15:28:55 -07:00
f6a932194f
Removing references to old consul services and adding consul config to executor context
2016-03-23 12:19:19 -07:00
719f5d34ed
Merge pull request #910 from hashicorp/f-reserved-resources
...
Reserve Client Resources + Config Validation
2016-03-15 21:09:13 -07:00
7c3bb51cfa
codespell: minor spelling corrections
...
Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
2016-03-16 05:28:31 +11:00
7d4c19ed99
reserve resources on the node
2016-03-13 19:05:41 -07:00
75d5aad888
client: fix bug where pushing allocs is skipped
2016-03-10 16:18:20 -08:00
22f4fbd652
up cached connection time
2016-03-09 10:37:56 -08:00
cd889df20a
client: send correct node id
2016-02-22 22:43:55 -08:00
a40f734b77
Send NodeID when updating client allocation
2016-02-22 17:25:11 -08:00
51bacf674e
address feedback
2016-02-21 21:32:32 -08:00
281e2ca198
Batch client allocation updates to the server
2016-02-21 21:15:02 -08:00
2ec5d7de76
undo async update
2016-02-19 22:34:52 -08:00
c08e3dbee8
Make updating alloc status async
2016-02-19 21:44:23 -08:00
13e5597ca2
Reduce alloc lock contention in client
2016-02-19 19:51:55 -08:00
d47935b455
Don't re-register as initializing
2016-02-18 23:02:28 -08:00
96fd272422
Increase Alloc channel buffers
2016-02-18 20:43:48 -08:00
5473b6ae63
Extract the heartbeat and saveState into their own go routines
2016-02-17 11:32:17 -08:00
0e68c7c949
Initialize the config copy after client init
2016-02-10 19:01:57 -08:00
e6c2b6ae9d
Slightly less node copying
2016-02-10 14:09:23 -08:00
0c4c3fc4ee
safe but slow
2016-02-10 13:44:53 -08:00
071216a730
Fix concurrent r/w to heartbeat time
2016-02-09 22:43:16 -08:00
913f98f738
Make fingerprinting thread safe
2016-02-09 22:14:24 -08:00
73ab620f61
Add comments to hasNodeChanged and remove superfluous else block
2016-02-04 08:19:34 +01:00
d5a67aba86
Reregister node when periodic fingerprint changes node properties
2016-02-03 21:10:58 +01:00
b5260fc14e
Fix locks and use task runners state not alloc state
2016-02-01 15:43:59 -08:00
2d98c0eadd
Fix double pull with introduction of AllocModifyIndex
2016-02-01 15:43:59 -08:00
4527257647
allow group and others to have executable permissions
2016-01-31 10:54:32 -08:00
e053511232
Use net.JoinHostPort
2016-01-29 05:39:28 -05:00
78814cba28
Spelling
2016-01-29 05:11:50 -05:00
9a6bd0d7fe
Updates from comments, fix tests
2016-01-28 23:11:13 -05:00
ee1be79093
Allow ports to be optional when adding servers
...
When updating a clients servers, as nomad does not use the gossip
protocol over a specified port for clients, it was required to specify
ports along with server addresses.
Now specifying ports are optional, and if unspecified the default `4647`
port is used, reflecting a notice back to the user.
2016-01-28 22:08:28 -05:00
9c46eceeac
Cleanup code and add comments
2016-01-20 00:02:17 +01:00
149c55252d
Merge branch 'master' into f-cli-short-ids
2016-01-15 09:19:53 +01:00
e18f9d787e
Added the node id to agent info
2016-01-14 15:42:30 -08:00
39b263ed7f
Refactoring some comments and test names
2016-01-14 15:07:24 -08:00
4d94af74f8
Added some docs and removed a redundant method
2016-01-14 13:45:48 -08:00
08d8a7d527
changed the API of the client
2016-01-14 13:35:42 -08:00
91a9f2c4ce
Shorten CLI identifiers
...
* Truncate all UUID identifiers to eight characters by default
* Refactor the node identifier to an auto-generated UUID
* Created and updated tests and mocks
2016-01-14 21:57:43 +01:00
29d264ff7c
Renamed AllocFile to AllocFileInfo
2016-01-13 17:18:10 -08:00
9e5d6d7fe8
Implemeted readAt
2016-01-12 22:06:42 -08:00
74af0da4cd
Implemented the Stat API
2016-01-12 21:28:07 -08:00
a1453e6180
Added the Stat API
2016-01-12 15:25:51 -08:00
e3d7e693dc
Added methods for listing directories inside an alloc
2016-01-12 15:03:53 -08:00
3ba1c9b76b
merge
2016-01-11 09:58:26 -08:00
c7c0748cea
Making a struct to hold consul service config
2015-12-11 14:06:51 -08:00
d54ccfb0ec
Muted the consul debug messages
2015-12-11 09:06:03 -08:00
22e9711852
Verify Consul SSL by default
...
- Matches documentation default value and clarifies config
2015-11-28 19:58:09 -06:00
3f82deff60
Added options for adding more consul configuration
2015-11-25 13:39:16 -08:00
b35819f10a
Merge pull request #498 from hashicorp/b-consul-check
...
Handle updates of Service and Check definitions
2015-11-25 13:17:26 -08:00
135006699b
Renamed consul client to service
2015-11-24 12:34:26 -08:00
953ed6ac76
Client config option "fingerprint.whitelist"
2015-11-24 16:18:49 +01:00
835343f47d
Renamed whitelisted to skipped and update debug line
2015-11-20 14:07:35 -08:00
c78f69bdee
Implement the driver whitelist
2015-11-19 16:39:21 -08:00
2ee71ffb59
Added the option to configure consul address
2015-11-18 05:15:52 -08:00
e83387191e
Shutting down consul an not trying to de-register the consul service
2015-11-18 04:59:57 -08:00
cb34f34c12
Added a logger to consul client
2015-11-18 02:14:07 -08:00
404810043a
Added the implementation of consul client
2015-11-18 00:50:45 -08:00
933498e388
Client Restore test
2015-11-10 16:03:18 -08:00
19d0c97da7
Client restores state properly
2015-11-09 15:55:31 -08:00
285f2cd0a9
Make periodic fingerprinting log the error
2015-11-05 18:47:16 -08:00
5f1b6e70b0
Client handles periodic fingerprinters
2015-11-05 13:46:17 -08:00
f441c4c65d
Cleanup allocations when in dev mode
2015-10-04 13:36:03 -07:00
6ca7c6680d
Fix client init logging and test
2015-09-25 18:12:11 -07:00
5a76693bc6
Fix comments
2015-09-25 16:49:19 -07:00
3cf6d014ef
Log the alloc and state directory on Client Init()
2015-09-25 16:49:19 -07:00
72d892a518
Client init rebase
2015-09-25 16:46:41 -07:00
9d3e3c0704
AllocDirBuilder that creates the alloc directory structure
2015-09-25 16:46:41 -07:00
6f15160f5c
client: support modifying the servers list
2015-09-24 20:16:47 -07:00
8aa2c292a0
Merge pull request #77 from hashicorp/f-network-fingerprint
...
Network Fingerprinting
2015-09-23 14:36:29 -07:00
fe72c8ea59
client: reset response struct to avoid decode errors
2015-09-22 22:10:28 -07:00
b7f370ba07
client: create dir, handle not exist more gracefully
2015-09-22 22:00:24 -07:00
ef81b29d7a
client: fixing stats formating
2015-09-22 21:48:42 -07:00
088bc9e482
fix range error
2015-09-22 23:25:12 -05:00
ed2fc3a69f
client: persist the node ID
2015-09-22 10:31:47 -07:00
67fc3c9b87
client: optimize initial heartbeat time
2015-09-20 17:02:12 -07:00
2ff133c0e6
nomad: rename region1 to global. Fixes #41
2015-09-13 18:18:40 -07:00
3d31230ac1
client: always create alloc dir if it is non-empty
2015-09-13 12:14:12 -07:00
58216245ac
client: skip init in dev mode
2015-09-12 13:16:27 -07:00
823276993c
client: init the alloc dir
2015-09-12 11:47:44 -07:00
1ff724ab25
client: alloc dirs tolerate missing directories
2015-09-11 20:32:55 -07:00
a695e311dc
Replace logging and config with DriverContext, which allows us to expand the dependency injection without changing the interface
2015-09-09 18:06:23 -07:00
4eb8fc5188
Added config to drivers; needed for docker driver to get the socket endpoint
2015-09-08 12:43:02 -07:00
ea0795995d
Use a single implementation of GenerateUUID
2015-09-07 15:23:03 -07:00
0702f17989
Rename client endpoint to node endpoint
2015-09-06 20:31:32 -07:00
c7773feced
nomad: improve error messages at start for dev mode
2015-09-06 20:18:47 -07:00
ab069de9b9
client: more stats
2015-08-30 17:24:12 -07:00
5549a82f9f
client: testing state restore
2015-08-30 17:19:20 -07:00
7b3adc6cc4
client: refactor AllocRunner for testing
2015-08-30 16:35:04 -07:00
b6332b435d
client: first pass at save/restore of state
2015-08-29 18:16:49 -07:00
e57ee3fbfe
client: test watching allocations
2015-08-29 14:33:30 -07:00
Diptanu Choudhury