Commit graph

1590 commits

Author SHA1 Message Date
Diptanu Choudhury fa216199ce Added documentation 2016-06-15 02:42:15 +02:00
Diptanu Choudhury e08083acfe Added a client options for setting selinux options 2016-06-15 02:33:09 +02:00
Sean Chittenden 6e22b680ce
Disambiguate auto_join from auto_register, rename reg to auto_advertise.
Provide an option that describes the value to the user vs the
operation performed by the software.  Momentarily introducing
`auto_join`
2016-06-14 12:11:38 -07:00
Alex Dadgar 4b04e503f3 address comments 2016-06-13 17:32:18 -07:00
Alex Dadgar 8bbf4a55e5 Fix IDs and domain scoping 2016-06-13 16:30:58 -07:00
Diptanu Choudhury d019d8ef8e implemented reconciliation of unwanted services 2016-06-13 14:52:26 +02:00
Alex Dadgar 232654cdee register checks 2016-06-12 21:28:56 -07:00
Alex Dadgar a82c2bb058 Do not reconcile in client and cleanup executor a bit 2016-06-12 18:22:07 -07:00
Alex Dadgar 8e231fa382 Rename ConsulService back to Service 2016-06-12 16:36:49 -07:00
Alex Dadgar e931b42473 unify cli output 2016-06-12 13:16:07 -07:00
Alex Dadgar 6ef4bfd6bf skip docker test if no docker found 2016-06-12 11:28:43 -07:00
Alex Dadgar 2628827b7c Merge pull request #1262 from hashicorp/remove-artifact-check
Removing artifact check for java and qemu drivers
2016-06-12 11:21:18 -07:00
Alex Dadgar c4a819528a Merge pull request #1260 from hashicorp/f-alloc-stats-struct
Allocation resources returned in a struct
2016-06-12 11:18:57 -07:00
Alex Dadgar b671cd4134 Test fixes 2016-06-12 11:14:17 -07:00
Alex Dadgar fdda90229f only support latest and remove ring buffer 2016-06-12 09:32:38 -07:00
Diptanu Choudhury 34f85baab0 Fix the calculation of total ticks for docker and exec 2016-06-12 18:08:35 +02:00
Diptanu Choudhury beb362e202 Setting a flag to indicate whether fs isolation is indeed happening 2016-06-12 15:43:24 +02:00
Diptanu Choudhury 641cf50682 Not converting the abs path relative to task dir for drivers which enforce FS isolation only in linux 2016-06-12 13:54:30 +02:00
Alex Dadgar e952540f6f Allocation resources returned in a struct 2016-06-11 21:04:10 -07:00
Diptanu Choudhury a5e81ebc3a Removing un-used code 2016-06-12 01:23:49 +02:00
Diptanu Choudhury 86e4f295da Fixed the calculation of the host node ticks 2016-06-12 01:14:51 +02:00
Sean Chittenden 2f036231e5 Merge pull request #1201 from hashicorp/f-dyn-server-list
Dynamic Server Lists/Client Bootstrapping via consul.
2016-06-11 18:58:25 -04:00
Sean Chittenden 92e2cfb0ad
Walk the DCs from nearest to most remote. 2016-06-11 18:52:21 -04:00
Sean Chittenden 2968545201
Walk the DCs from nearest to most remote, no limit on the search. 2016-06-11 18:23:06 -04:00
Sean Chittenden 917766a3df
Prefer %+q over %q in log messages. 2016-06-11 18:17:20 -04:00
Sean Chittenden 445783889b
Remove default values and use nil for the executor. Much better. 2016-06-11 17:52:09 -04:00
Diptanu Choudhury 0fad17a1a9 Merge pull request #1258 from hashicorp/fix-statsd-metric-type
Emitting client resource usage metrics as guages instead of k/v pairs
2016-06-11 13:18:52 -07:00
Diptanu Choudhury fd60cfd585 Emitting client resource usage metrics as guages instead of k/v pairs 2016-06-11 22:17:32 +02:00
Diptanu Choudhury 19f4adbcf1 Using a different client for collecting stats and waiting on containers 2016-06-11 20:37:29 +02:00
Diptanu Choudhury 7fb507e810 Moving the clkspeed code to helper 2016-06-11 17:31:49 +02:00
Sean Chittenden f891fa0ec8
Perform a nil-check for Executor's consulServices.
Executors can `Shutdown()` before calling `SyncServices()`.
2016-06-10 23:43:54 -04:00
Sean Chittenden bbd8dfa798
goling(1) compliance pass (e.g. Rpc* -> RPC) 2016-06-10 23:38:28 -04:00
Sean Chittenden bc771d35df
Query for the Nomad service across multiple Consul datacenters. 2016-06-10 23:05:14 -04:00
Sean Chittenden d433001f04
Expose rpcproxy's ServerEndpoint() constructor, newServer() as NewServerEndpoint() 2016-06-10 22:14:03 -04:00
Diptanu Choudhury 59540c3e93 Extracted a method for getting clock speed 2016-06-11 02:07:28 +02:00
Diptanu Choudhury f94f89b6d7 Pruning out pids which are no longer present 2016-06-11 01:40:52 +02:00
Diptanu Choudhury 0a9a3918d6 Not reset-ing the list of pids if they don't change 2016-06-11 01:19:50 +02:00
Diptanu Choudhury c38a6fb3c5 Implementing the total ticks per task for the docker driver 2016-06-10 23:33:25 +02:00
Diptanu Choudhury 01054db4fa Calculating total ticks consumed in the nomad client 2016-06-10 23:14:33 +02:00
Sean Chittenden 5b0c2bf282
Restore old behavior and have AddPrimaryServer() return a pointer to the existing server (vs nil when the server already exists). 2016-06-10 16:46:49 -04:00
Diptanu Choudhury 2d3798b076 Calculating the cpu ticks in nomad client 2016-06-10 22:22:32 +02:00
Sean Chittenden df896c6ef2
Prevent duplicate servers being added in AddPrimaryServer.
This logic was already present elsewhere and was missed in this one
place.
2016-06-10 15:55:27 -04:00
Sean Chittenden d99467ef5e
Always create a consul.Syncer. Use a default Consul Config if necessary. 2016-06-10 15:55:27 -04:00
Sean Chittenden f6a0459ae5
Always create a consul.Syncer. Use a default Consul Config if necessary. 2016-06-10 15:55:27 -04:00
Sean Chittenden 03846fb754
Rename listLock to activatedListLock 2016-06-10 15:54:39 -04:00
Sean Chittenden 4c9067310b
Nomad does not use Serf at the client level. Use a hard lock. 2016-06-10 15:54:39 -04:00
Sean Chittenden 26b1e826d7
golint(1) police 2016-06-10 15:54:39 -04:00
Sean Chittenden 974c9927c7
Formatting nit: remove brackets 2016-06-10 15:54:39 -04:00
Sean Chittenden 1371ef85f5
Prefix all log entries in client/rpcproxy with client.rpcproxy 2016-06-10 15:54:39 -04:00
Sean Chittenden 009495c18a
Style nit: remove var block 2016-06-10 15:54:39 -04:00
Sean Chittenden f139d0c68b
Properly guard consulPullHeartbeatDeadline behind heartbeatLock 2016-06-10 15:54:39 -04:00
Sean Chittenden 8dc16ad5e3
Move RPCProxy.New() adjacent to its struct definition 2016-06-10 15:54:39 -04:00
Sean Chittenden cc6e8792e0
Create a consulContext using a client's consul config.
This is wrong and should be the Agent's Consul Config.  This is a
step in the right direction, so committing to mark the necessary
future change.
2016-06-10 15:54:39 -04:00
Sean Chittenden ed29946f5e
Populate the RPC Proxy's server list if heartbeat did not include a leader.
It's possible that a Nomad Client is heartbeating with a Nomad server that
has become issolated from the quorum of Nomad Servers.  When 3x the
heartbeatTTL has been exceeded, append the Consul server list to the primary
primary server list.  When the next RPCProxy rebalance occurs, there is a
chance one of the servers discovered from Consul will be in the majority.
When client reattaches to a Nomad Server in the majority, it will include
a heartbeat and will reset the TTLs *AND* will clear the primary server list
to include only values from the heartbeat.
2016-06-10 15:54:39 -04:00
Sean Chittenden 9a223936bb
Generate and sync Consul ServiceIDs consistently 2016-06-10 15:54:39 -04:00
Sean Chittenden 3892a0433e
Move the start of the UniversalExecutor's consulSyncer to initialize once
This should be handled via a sync.Once primative, but I don't want to
unpack that atm.
2016-06-10 15:54:39 -04:00
Sean Chittenden 7956eb0c80
Rename structs.Task's Service attribute to ConsulService 2016-06-10 15:54:39 -04:00
Sean Chittenden 8c813630e6
Move package client/consul/sync to command/agent/consul.
This has been done to allow the Server and Client to reuse the same
Syncer because the Agent may be running Client, Server, or both
simultaneously and we only want one Syncer object alive in the agent.
2016-06-10 15:54:39 -04:00
Sean Chittenden d6ef97911b
Rename Syncer.SetServiceIdentifier to SetServiceRegPrefix()
This attribute isn't actually an identifier because it can represent
a collection of services.  Rename `serviceIdentifier` to
`serviceRegPrefix which more accurately conveys the intention of this
Syncer attribute.

While here, also rename `SetServiceIdentifier()` to `SetServiceRegPrefix()`
and `GenerateServiceIdentifier()` to `GenerateServicePrefix()`.
2016-06-10 15:54:39 -04:00
Sean Chittenden 6b126ce488
Change the API signature of Syncer.SyncServices().
SyncServices() immediately attempts to sync whatever information
the process has with Consul.  Previously this method would take an
argument of the exclusive list of services that should exist,
however this is not condusive to having a Nomad Client and Nomad
Server share the same consul.Syncer.
2016-06-10 15:54:39 -04:00
Sean Chittenden fda03c5c9e
Change the signature of the PeriodicCallback to return an error
I *KNEW* I should have done this when I wrote it, but didn't want to
go back and audit the handlers to include the appropriate return
handling, but now that the code is taking shape, make this change.
2016-06-10 15:54:39 -04:00
Sean Chittenden 4973ec32bb
Rename structs.Services to structs.ConsulServices 2016-06-10 15:54:39 -04:00
Sean Chittenden 6430190d6a
Rename createCheck() to createDelegatedCheck() for clarity 2016-06-10 15:54:39 -04:00
Sean Chittenden 555f4fe135
Change client/consul.NewSyncer() to accept a shutdown channel
In addition to the API changing, consul.Syncer can now be signaled
to shutdown via the Shutdown() method, which will call the Run()'ing
sync task to exit gracefully.
2016-06-10 15:54:39 -04:00
Sean Chittenden 439baa0f8b
Remove named return parameters 2016-06-10 15:50:11 -04:00
Sean Chittenden ea0b35e303
Collapse server_endpoint_internal_test.go into server_endpoint_test.go
Requested by: @dadgar
2016-06-10 15:50:11 -04:00
Sean Chittenden 6e71bbc26b
Collapse rpcproxy_internal_test.go into rpcproxy_test.go
Requested by: @dadgar
2016-06-10 15:50:11 -04:00
Sean Chittenden 4b021b79b0
Bump the default Consul client timeout from 500ms to 5s.
Requsted by: @dadgar
2016-06-10 15:50:11 -04:00
Sean Chittenden 747a794c77
Move const block to the top of the file.
Requested by: @dadgar
2016-06-10 15:50:11 -04:00
Sean Chittenden 484816f5e0
Ensure that all accesses to Client.alloc are wrapped by allocLock. 2016-06-10 15:50:11 -04:00
Sean Chittenden 08cab4fdfa
Use client.getAllocRunners() where appropriate. 2016-06-10 15:50:11 -04:00
Sean Chittenden f9d0b9da32
Line wrap long line. 2016-06-10 15:50:11 -04:00
Sean Chittenden 0d201631a3
Rename rpcproxy.UpdateFromNodeUpdateResponse to RefreshServerLists
While breaking the API within this PR, break out the individual
arguments to RefreshServerLists.  The servers parameter is reusing
`structs.NodeServerInfo` for the time being, but this can be revisited
if the needs of the strucutre diverge in the future.
2016-06-10 15:50:11 -04:00
Sean Chittenden 0997fb1669
Fix up the comments
Pointed out by: @dadgar
2016-06-10 15:50:11 -04:00
Sean Chittenden aaa7d6bf40
Make the locking protocol more explicit in client.NewClient
With an over abundance of caution, preevnt future copy/pasta by
using the right locks when bootstrapping a Client.  Strictly speaking
this is not necessary, but it makes explicit the locking semantics
and guards against future concurrent or parallel initialization.
2016-06-10 15:50:11 -04:00
Sean Chittenden 525554c008
Use the client configCopy and lock appropriately. 2016-06-10 15:50:11 -04:00
Sean Chittenden 3060d6b33c
Flesh out the comment re: the client.rpcproxy.Run() task.
Requested by: Alex
2016-06-10 15:50:11 -04:00
Sean Chittenden a6a4345f27
Clean up various comments 2016-06-10 15:50:11 -04:00
Sean Chittenden 56a9981a13
Nuke the last of the explicit types in favor of using language idioms 2016-06-10 15:50:11 -04:00
Sean Chittenden b1ee131db8
Rename backupServerDeadline to consulPullHeartbeatDeadline
Suggested by: @alex
2016-06-10 15:50:11 -04:00
Sean Chittenden a8d2af692c
Don't clobber the default consul config in tests 2016-06-10 15:50:11 -04:00
Sean Chittenden b9adfcecf5
Remove unused variable 2016-06-10 15:50:11 -04:00
Sean Chittenden f15eeb8f27
Clean up some docs and comments to be more accurate 2016-06-10 15:50:11 -04:00
Sean Chittenden c78b0a6567
Remove unused constants 2016-06-10 15:50:11 -04:00
Sean Chittenden bcbec34937
Only actively test Consul when env CONSUL_HTTP_ADDR is set 2016-06-10 15:50:11 -04:00
Sean Chittenden 04c697c610
Update godoc for newServer to reflect DNS and IP-based inputs
Requested by: alex
2016-06-10 15:50:11 -04:00
Sean Chittenden 27413076fb
Pick the right DefaultConfig from the right package.
Overly zealous search && replace at work here.
2016-06-10 15:50:11 -04:00
Sean Chittenden 7167b7a357
Add a quick set of client/rpcproxy.ServerEndpoint equality tests 2016-06-10 15:50:11 -04:00
Sean Chittenden 837b387dcb
Fix building tests that used DefaultConfig() but didn't pickup the package move. 2016-06-10 15:50:11 -04:00
Sean Chittenden 3b5db4e390
Fix the client/rpcproxy unit tests. 2016-06-10 15:50:11 -04:00
Sean Chittenden 39fb0f2469
Change the endpoint for /v1/agent/servers and fix tests.
When an agent is running a server, the list of servers includes the
Raft peers.  When the agent is running a client (which is always the
case?), include a list of the servers found in the Client's RpcProxy.
Dedupe and provide a unique list back to the caller.
2016-06-10 15:50:11 -04:00
Sean Chittenden 6fdf9135cb
Provide a default ConsulConfig for client/config.DefaultConfig()
Change the unit test to only test if the consul link exists, not the
value of the link.  The old test was hostname specific and therefore
would always be different based on the environment running the tests.
2016-06-10 15:50:11 -04:00
Sean Chittenden cb80e93a6b
Move client.DefaultConfig() to client/config.DefaultConfig()
Resolves an import cycle in testing and is more appropriate because
the default should reside next to its struct definition.
2016-06-10 15:50:11 -04:00
Sean Chittenden d87085e040
Fix a comment to be more correct 2016-06-10 15:50:11 -04:00
Sean Chittenden c7e1879c4d
Unused code wasn't as unused as I thought. Restore. 2016-06-10 15:50:11 -04:00
Sean Chittenden bff57a0dce
Reconcile, clean up, and centralize API version numbers (major and minor).
Reduce future confusion by introducing a minor version that is gossiped out
via the `mvn` Serf tag (Minor Version Number, `vsn` is already being used for
to communicate `Major Version Number`).

Background: hashicorp/consul/issues/1346#issuecomment-151663152
2016-06-10 15:50:11 -04:00
Sean Chittenden b3fd455b1f
Register the serf service with the Nomad server service.
This will be unused in this PR.
2016-06-10 15:50:11 -04:00
Sean Chittenden 82d537fbd9
Update the nomad_server_service default from nomad-server to just nomad. 2016-06-10 15:50:11 -04:00
Sean Chittenden 6d9ec364e5
Change the constants used to match the struct definitions 2016-06-10 15:50:11 -04:00
Sean Chittenden c013957d49
When clearing the backup servers, set the length to zero. 2016-06-10 15:50:11 -04:00
Sean Chittenden 36340c654d
Improve language re: fingerprinting 2016-06-10 15:50:11 -04:00
Sean Chittenden bff82e4890
Remove unused function. 2016-06-10 15:50:11 -04:00
Sean Chittenden 498f21cdec
Clear the backup server list when a Nomad heartbeat arives with servers
If Nomad is heartbeating during a transition from using backup servers
to Nomad servers, make Nomad the canonical source of servers and flush
the list of servers populated from Consul.
2016-06-10 15:50:11 -04:00
Sean Chittenden 1fe979a5e4
Remove types.ShutdownChannel and replace with chan struct{} 2016-06-10 15:50:11 -04:00
Sean Chittenden 2395aa481c
Fix unit tests 2016-06-10 15:50:11 -04:00
Sean Chittenden 438becb28b
Pass the datacenter name in the heartbeat
Servers that are part of a different datacenter are added as backup
servers instead of primary servers.
2016-06-10 15:50:11 -04:00
Sean Chittenden d914f262f9
Consolidate all consul sync periodic go routines to handlers.
Only one pump and periodic loop now.
2016-06-10 15:50:11 -04:00
Sean Chittenden 9fb0104def
Teach Client to reuse an Agent's consulSyncer.
"There can be only one."
2016-06-10 15:50:11 -04:00
Sean Chittenden 47891fb559
Register two services each for clients and servers, http and rpc.
In order to give clients a fighting chance to talk to the right port,
differentiate RPC services from HTTP services by registering two
services with different tags.  This yields
`rpc.nomad-server.service.consul` and
`http.nomad-server.service.consul` which is immensely more useful to
clients attempting to bootstrap their world.
2016-06-10 15:50:11 -04:00
Sean Chittenden d2f9848348
Bump the cluster test minimums to 10min.
These ranges aren't too useful with the default 600s rebalance, but
will be useful if that default ever changes in the future.
2016-06-10 15:50:11 -04:00
Sean Chittenden 861f900225
s/RpcVersion/RPCVersion/g 2016-06-10 15:50:11 -04:00
Sean Chittenden bd0588ffa8
Bump shuffle interval per suggestion from Alex 2016-06-10 15:50:11 -04:00
Sean Chittenden 9f8d6c9f67
Move struct member to reduce diff context 2016-06-10 15:50:11 -04:00
Sean Chittenden 5570aa98e9
Fix package name in comments 2016-06-10 15:50:11 -04:00
Sean Chittenden 410d85cc78
Rename the package from client/rpc_proxy to client/rpcproxy
Also rename `NewRpcProxy()` to just `New()` to avoid package stutter.
2016-06-10 15:50:11 -04:00
Sean Chittenden be679c161c
Rename manager_test.go to rpc_proxy_test.go 2016-06-10 15:50:11 -04:00
Sean Chittenden 677f149b16
Rename RpcProxy's internal filename to match 2016-06-10 15:50:11 -04:00
Sean Chittenden 11d53fda00
Remove unused code 2016-06-10 15:50:11 -04:00
Sean Chittenden 10d1b6ba9b
Reduce all forms of ConsulConfig down to a single struct
nomad/structs/config/consul.go's ConsulConfig is the canonical definition
for all things Consul now.
2016-06-10 15:50:11 -04:00
Sean Chittenden 0dbd16aa5d
Rebalance Nomad client RPCs among different Nomad servers.
Implement client/rpc_proxy.RpcProxy.
2016-06-10 15:50:11 -04:00
Sean Chittenden a13cbe73c3
Rename ConsulConfig to ConsulAgentConfig 2016-06-10 15:50:11 -04:00
Sean Chittenden e727fd8c3c
Centralize the creation of a consul/api.Config struct.
While documented, the consul.timeout parameter wasn't ever set
except one-off in the Consul fingerprinter.
2016-06-10 15:50:11 -04:00
Sean Chittenden af96cb23a5
Rename mergedNomadMap to mergedPrimaryMap. Gratuitous, but more correct. 2016-06-10 15:50:11 -04:00
Sean Chittenden 9d5cdf4e68
Correct a mismerge
Somewhere along the lines when rebasing I mis-merged a patch.
2016-06-10 15:50:11 -04:00
Sean Chittenden d58632e93f
Rename RpcProxy's internal filename to match 2016-06-10 15:50:11 -04:00
Sean Chittenden 108d2ca981
index on f-dyn-server-list: bd38cb4 Remove unused code 2016-06-10 15:50:11 -04:00
Sean Chittenden cbe77754df
Remove unused code 2016-06-10 15:50:11 -04:00
Sean Chittenden ec7c78f533
Only poll Consul for servers when Nomad heartbeats begin to fail
When a deadline timer of 2x Server's last requested TTL expires,
begin polling Consul for Nomad Servers.
2016-06-10 15:50:11 -04:00
Sean Chittenden a4b32c62e2
s/availble/runChecks/g 2016-06-10 15:50:11 -04:00
Sean Chittenden d1442dc317
Reduce all forms of ConsulConfig down to a single struct
nomad/structs/config/consul.go's ConsulConfig is the canonical definition
for all things Consul now.
2016-06-10 15:50:11 -04:00
Sean Chittenden e60580b279
Define a type for the PeriodicCallback handlers and ShutdownChannel 2016-06-10 15:50:11 -04:00
Sean Chittenden 17116fc5a7
Rebalance Nomad client RPCs among different Nomad servers.
Implement client/rpc_proxy.RpcProxy.
2016-06-10 15:50:11 -04:00
Sean Chittenden 88f3422d7c
Rename NewConsulService to NewSyncer 2016-06-10 15:49:37 -04:00
Sean Chittenden b9789e0cf7
Rename client/consul/sync.PeriodicSync to Run 2016-06-10 15:49:37 -04:00
Sean Chittenden 768aab015d
Rename client/consul/sync.ConsulService to client/consul/sync.Syncer
Syncer describes the responsibility and actions of the type.
2016-06-10 15:49:37 -04:00
Sean Chittenden bfa4711df5
Rename ConsulConfig to ConsulAgentConfig 2016-06-10 15:48:36 -04:00
Sean Chittenden 73e173673e
Rename client/config/config's ConsulConfig to ConsulAgentConfig
A follow up commit to the previous rename.  More to come.
2016-06-10 15:48:36 -04:00
Sean Chittenden e97652bbfb
Rename consul.ConsulConfig to consul.AgentConfig
There were two `ConsulConfig` structs running around, one of them
needed to go away.  Rely on the package's path to provide context
for the type of AgentConfig.
2016-06-10 15:48:36 -04:00
Sean Chittenden e36686a17d
Use consul/lib's RandomStagger
Removes four redundant copies of the method in the process.
2016-06-10 15:48:36 -04:00
Alex Dadgar 26c9110e82 fix types 2016-06-10 11:52:15 -07:00
Alex Dadgar a3d41f2312 use time.Second instead of our own const 2016-06-10 11:38:59 -07:00
Alex Dadgar 3cf74e7fd8 Alloc-status only shows measured statistics and fixes to CPU calculations 2016-06-10 10:38:29 -07:00
Diptanu Choudhury 287e0eff22 Fixed merge conflicts 2016-06-10 04:47:34 +01:00
Diptanu Choudhury a69307cff2 Using named variables 2016-06-10 04:45:16 +01:00
Diptanu Choudhury 9056be6d86 Fixed calculation of cpu usage percent in user and sys mode for pids 2016-06-10 04:06:32 +01:00
Alex Dadgar 98d7547014 Annotate what is measured 2016-06-09 19:45:41 -07:00
Alex Dadgar b7e3a45fef fix channel being nil on restore 2016-06-07 15:03:08 -07:00
Diptanu Choudhury 6bf38bb213 Fixed the svc id generation scheme while pruning services 2016-06-07 08:14:11 -07:00
Diptanu Choudhury 4027ff56f0 Returning an error if stats collection hasn't started yet for a new docker container 2016-06-06 18:52:54 -07:00
Diptanu Choudhury c21d606ebb Getting inodes used percent back 2016-06-06 16:10:34 -07:00