353e7fc7f1
Moving the certs into tlsutil package
2016-10-25 16:01:53 -07:00
cf35aeac84
Moving the TLSConfig to structs
2016-10-25 15:57:38 -07:00
03eba049ed
Merge pull request #1848 from hashicorp/f-vault-error
...
Thread through whether DeriveToken error is recoverable or not
2016-10-24 15:01:18 -07:00
692a809919
Merge pull request #1842 from hashicorp/f-version-and-id
...
Print the version and client node ID
2016-10-24 10:13:33 -07:00
2e3118e69c
Implemented TLS support for http and rpc
2016-10-23 22:22:00 -07:00
ede3a814ba
Small fixes
2016-10-22 18:20:50 -07:00
0070178741
Thread through whether DeriveToken error is recoverable or not
2016-10-22 18:08:30 -07:00
285e80ac0f
Remove disk usage enforcement
...
Many thanks to @iverberk for the original PR (#1609 ), but we ended up
not wanting to ship this implementation with 0.5.
We'll come back to it after 0.5 and hopefully find a way to leverage
filesystem accounting and quotas, so we can skip the expensive polling.
2016-10-21 13:55:51 -07:00
aa0d8d0d8d
Print the version and client node ID
2016-10-20 17:46:04 -07:00
e7a98d5500
Make EvalSymlink errors more verbose
2016-10-12 17:07:21 -07:00
f8a65a3b9d
Resolve alloc/state directories to make Docker For Mac happy
...
* In -dev mode, `ioutil.TempDir` is used for the alloc and state
directories.
* `TempDir` uses `$TMPDIR`, which os OS X contains a per user
directory which is under `/var/folder`.
* `/var` is actually a symlink to `/private/var`
* Docker For Mac validates the directories that are passed to bind and on
OS X. That whitelist contains `/private`, but not `/var`. It does not
expand the path, and so any paths in `$TMPDIR` fail the whitelist check.
And thusly, by expanding the alloc/state directories the value passed
for binding does contain `/private` and Docker For Mac is happy.
2016-10-12 17:06:25 -07:00
6dea6df919
Restore lost chan inits
2016-10-03 14:56:50 -07:00
d50c395421
Getting snapshot of allocation from remote node ( #1741 )
...
* Added the alloc dir move
* Moving allocdirs when starting allocations
* Added the migrate flag to ephemeral disk
* Stopping migration if the allocation doesn't need migration any more
* Added the GetAllocDir method
* refactored code
* Added a test for alloc runner
* Incorporated review comments
2016-10-03 09:59:57 -07:00
b117725dc9
Only log consul errors once since last succesful run
2016-09-28 17:18:45 -07:00
d486de3804
Remove unused const
2016-09-27 16:04:01 -07:00
2e696c5e61
Fix lies found in comments by fact checkers
2016-09-26 16:51:53 -07:00
11cf9686a6
No need to put reaper ticker on the struct
2016-09-26 16:15:19 -07:00
2eb0062959
Drop clumsy timeout on discovery notifications
...
It's better to just let goroutines fallback to their longer retry
intervals then try to be clever here.
2016-09-26 16:05:21 -07:00
307e674eca
Flip disco chan; clarify method names/comments
2016-09-26 15:52:40 -07:00
888ee21270
Return csv of servers from Stats, not just count
2016-09-26 15:40:26 -07:00
7dc0079dd2
doDisco -> triggerDiscoveryCh; discovered -> serversDiscoveredCh
...
Also fix log line formatting
2016-09-26 15:21:28 -07:00
434e4be97c
noServers -> noServersErr
2016-09-26 15:12:35 -07:00
b2ddb85a78
consul -> Consul
2016-09-26 15:06:57 -07:00
37cfb2769c
Replace periodic handlers with event driven disco
...
Remove use of periodic consul handlers in the client and just use
goroutines. Consul Discovery is now triggered with a chan instead of
using a timer and deadline to trigger.
Once discovery is complete a chan is ticked so all goroutines waiting
for servers will run.
Should speed up bootstraping and recovery while decreasing spinning on
timers.
2016-09-23 17:02:48 -07:00
2ab5264595
Retry all servers on RPC call failure
...
rpcproxy is refactored into serverlist which prioritizes good servers
over servers in a remote DC or who have had a failure.
Registration, heartbeating, and alloc status updating will retry faster
when new servers are discovered.
Consul discovery will be retried more quickly when no servers are
available (eg on startup or an outage).
2016-09-23 11:44:48 -07:00
50efdb00e9
Merge pull request #1713 from hashicorp/f-alloc-runner-vault
...
Vault integration in client
2016-09-20 16:15:55 -07:00
64de46432a
Merge pull request #1677 from hashicorp/f-vault-implicit-constraint
...
Vault implicit Task Group constraint + allow root tokens
2016-09-20 16:15:32 -07:00
ec152a6d12
Clean up vault client
2016-09-14 18:10:56 -07:00
6702a29071
Vault token threaded
2016-09-14 13:30:01 -07:00
8dc19dbd10
Log adding of servers at INFO level
2016-09-14 22:24:17 +02:00
2c8dd8bbd3
Revert "Introduce a Secret/ directory"
2016-09-01 17:23:15 -07:00
b0adaa5301
Allow root token
2016-09-01 12:05:08 -07:00
1ed454dd60
Merge pull request #1671 from hashicorp/f-secret-dir2
...
Introduce a Secret/ directory
2016-09-01 09:56:17 -07:00
9fa23e3536
Symlink on windows
2016-08-31 21:41:44 -07:00
5d3b47e648
Address comments and reserve
2016-08-31 18:11:02 -07:00
55a6f06e15
Addressed review feedback
2016-08-30 13:08:13 -04:00
3808dd0ff8
Return only fatal error to renewal error channel
2016-08-30 12:46:59 -04:00
a0dbfe25b3
Fix tests
2016-08-29 21:30:06 -04:00
82f6209e97
tokenDeriver function pointer to derive tokens.
...
Remove rpc*, connPool, node and region from vaultclient.
2016-08-29 20:32:05 -04:00
14b7126511
Secret dir, hello world
2016-08-29 15:41:52 -07:00
56e42cf03d
Employ DeriveVaultToken API and flesh-up DeriveToken
2016-08-24 12:29:59 -04:00
6002e596c4
VaultClient for Nomad Client
2016-08-24 09:43:45 -04:00
1e1eef56a1
Putting the mock driver behind a build flag
2016-08-22 15:02:28 -05:00
4ca623bcfe
blocking chained allocations until previous allocation hasn't terminated
2016-08-22 11:34:24 -05:00
a90dafe9ab
handle the upgrade case
2016-08-18 19:01:24 -07:00
895c31f605
Nodes generate Secret ID and used for retrieving allocations and registering
2016-08-17 16:31:47 -07:00
84820db86f
If the client detects that a heartbeat has failed because it is not registered, reregister
2016-08-15 17:24:09 -07:00
28b3f511e0
Fixed some error messages
2016-08-10 15:17:32 -07:00
6a810e6f1e
Update after review
2016-08-09 08:57:26 +09:00
5c621b74e5
tiny: Return fmt.Errorf instead of duplicated error messages
2016-08-09 08:57:26 +09:00
41b540fbc8
Allow operators to opt into publishing node and alloc metrics
2016-08-01 19:52:20 -07:00
777bdf4a1e
fix setup consul syncer error message
2016-07-28 22:14:52 -05:00
ebac5cb283
Node.Register handles the case of transistioning to ready and creating evals
2016-07-21 15:22:02 -07:00
5b39a5db40
Fixed a debug message
2016-07-09 00:12:53 -07:00
03c571c61b
Consolidate fingerprinters into a single `map`.
2016-07-08 23:37:14 -07:00
8bdb38d016
Code golf
...
Pointed out by: @dadgar
2016-06-21 14:26:01 -07:00
df4fe2e502
Fix the shuffling of remote datacenters.
...
Pointed out by: @ryanuber
2016-06-21 13:37:22 -07:00
9a60999100
Pass a logger arg to `NewClient` and `NewServer`
2016-06-16 23:29:23 -07:00
fd18eb7fdb
Only register the Client services reaper when `consul.auto_advertise` is enabled
2016-06-16 18:24:58 -07:00
952b6ce7b5
Only auto-join clients if `client_auto_join` is true
2016-06-16 14:47:21 -07:00
af55b74114
Merge pull request #1276 from hashicorp/f-consul-server-autojoin
...
Teach Nomad servers how to fall back to Consul.
2016-06-16 14:40:45 -07:00
008d75184b
Use the `%+q` verb in log messages (vs `%q`).
2016-06-16 11:03:51 -07:00
7375d828e1
remove trace
2016-06-15 15:47:59 -07:00
5e0ced2ae7
Shuffle all datacenters vs only the nearest N datacenters.
...
Per discussion, we want to be aggressive about fanning out vs possibly
fixating on only local DCs. With RPC forwarding in place, a random walk
may be less optimal from a network latency perspective, but it is guaranteed
to eventually result in a converged state because all DCs are candidates
during the bootstrapping process.
2016-06-15 12:40:51 -07:00
2123460cf0
Bump various Consul search limits
...
Client: Search limit increased from 4 random DCs to 8 random DCs, plus nearest.
Server: Search factor increased from 3 to 5 times the bootstrap_expect.
This should allow for faster convergence in large environments (e.g.
sub-5min for 10K Consul DCs).
2016-06-15 12:40:51 -07:00
cf99fc3173
Use Status.Peers instead of Status.Ping
2016-06-15 12:00:20 -07:00
4b04e503f3
address comments
2016-06-13 17:32:18 -07:00
8bbf4a55e5
Fix IDs and domain scoping
2016-06-13 16:30:58 -07:00
d019d8ef8e
implemented reconciliation of unwanted services
2016-06-13 14:52:26 +02:00
a82c2bb058
Do not reconcile in client and cleanup executor a bit
2016-06-12 18:22:07 -07:00
8e231fa382
Rename ConsulService back to Service
2016-06-12 16:36:49 -07:00
fdda90229f
only support latest and remove ring buffer
2016-06-12 09:32:38 -07:00
e952540f6f
Allocation resources returned in a struct
2016-06-11 21:04:10 -07:00
2f036231e5
Merge pull request #1201 from hashicorp/f-dyn-server-list
...
Dynamic Server Lists/Client Bootstrapping via consul.
2016-06-11 18:58:25 -04:00
92e2cfb0ad
Walk the DCs from nearest to most remote.
2016-06-11 18:52:21 -04:00
2968545201
Walk the DCs from nearest to most remote, no limit on the search.
2016-06-11 18:23:06 -04:00
917766a3df
Prefer `%+q` over `%q` in log messages.
2016-06-11 18:17:20 -04:00
fd60cfd585
Emitting client resource usage metrics as guages instead of k/v pairs
2016-06-11 22:17:32 +02:00
bbd8dfa798
goling(1) compliance pass (e.g. Rpc* -> RPC)
2016-06-10 23:38:28 -04:00
bc771d35df
Query for the Nomad service across multiple Consul datacenters.
2016-06-10 23:05:14 -04:00
26b1e826d7
golint(1) police
2016-06-10 15:54:39 -04:00
f139d0c68b
Properly guard consulPullHeartbeatDeadline behind heartbeatLock
2016-06-10 15:54:39 -04:00
ed29946f5e
Populate the RPC Proxy's server list if heartbeat did not include a leader.
...
It's possible that a Nomad Client is heartbeating with a Nomad server that
has become issolated from the quorum of Nomad Servers. When 3x the
heartbeatTTL has been exceeded, append the Consul server list to the primary
primary server list. When the next RPCProxy rebalance occurs, there is a
chance one of the servers discovered from Consul will be in the majority.
When client reattaches to a Nomad Server in the majority, it will include
a heartbeat and will reset the TTLs *AND* will clear the primary server list
to include only values from the heartbeat.
2016-06-10 15:54:39 -04:00
9a223936bb
Generate and sync Consul ServiceIDs consistently
2016-06-10 15:54:39 -04:00
7956eb0c80
Rename structs.Task's `Service` attribute to `ConsulService`
2016-06-10 15:54:39 -04:00
8c813630e6
Move package client/consul/sync to command/agent/consul.
...
This has been done to allow the Server and Client to reuse the same
Syncer because the Agent may be running Client, Server, or both
simultaneously and we only want one Syncer object alive in the agent.
2016-06-10 15:54:39 -04:00
fda03c5c9e
Change the signature of the PeriodicCallback to return an error
...
I *KNEW* I should have done this when I wrote it, but didn't want to
go back and audit the handlers to include the appropriate return
handling, but now that the code is taking shape, make this change.
2016-06-10 15:54:39 -04:00
555f4fe135
Change client/consul.NewSyncer() to accept a shutdown channel
...
In addition to the API changing, consul.Syncer can now be signaled
to shutdown via the Shutdown() method, which will call the Run()'ing
sync task to exit gracefully.
2016-06-10 15:54:39 -04:00
484816f5e0
Ensure that all accesses to Client.alloc are wrapped by allocLock.
2016-06-10 15:50:11 -04:00
08cab4fdfa
Use client.getAllocRunners() where appropriate.
2016-06-10 15:50:11 -04:00
f9d0b9da32
Line wrap long line.
2016-06-10 15:50:11 -04:00
0d201631a3
Rename rpcproxy.UpdateFromNodeUpdateResponse to RefreshServerLists
...
While breaking the API within this PR, break out the individual
arguments to RefreshServerLists. The servers parameter is reusing
`structs.NodeServerInfo` for the time being, but this can be revisited
if the needs of the strucutre diverge in the future.
2016-06-10 15:50:11 -04:00
0997fb1669
Fix up the comments
...
Pointed out by: @dadgar
2016-06-10 15:50:11 -04:00
aaa7d6bf40
Make the locking protocol more explicit in client.NewClient
...
With an over abundance of caution, preevnt future copy/pasta by
using the right locks when bootstrapping a Client. Strictly speaking
this is not necessary, but it makes explicit the locking semantics
and guards against future concurrent or parallel initialization.
2016-06-10 15:50:11 -04:00
525554c008
Use the client configCopy and lock appropriately.
2016-06-10 15:50:11 -04:00
3060d6b33c
Flesh out the comment re: the client.rpcproxy.Run() task.
...
Requested by: Alex
2016-06-10 15:50:11 -04:00
b1ee131db8
Rename `backupServerDeadline` to `consulPullHeartbeatDeadline`
...
Suggested by: @alex
2016-06-10 15:50:11 -04:00
b9adfcecf5
Remove unused variable
2016-06-10 15:50:11 -04:00
f15eeb8f27
Clean up some docs and comments to be more accurate
2016-06-10 15:50:11 -04:00
c78b0a6567
Remove unused constants
2016-06-10 15:50:11 -04:00
39fb0f2469
Change the endpoint for `/v1/agent/servers` and fix tests.
...
When an agent is running a server, the list of servers includes the
Raft peers. When the agent is running a client (which is always the
case?), include a list of the servers found in the Client's RpcProxy.
Dedupe and provide a unique list back to the caller.
2016-06-10 15:50:11 -04:00
cb80e93a6b
Move client.DefaultConfig() to client/config.DefaultConfig()
...
Resolves an import cycle in testing and is more appropriate because
the default should reside next to its struct definition.
2016-06-10 15:50:11 -04:00
bff57a0dce
Reconcile, clean up, and centralize API version numbers (major and minor).
...
Reduce future confusion by introducing a minor version that is gossiped out
via the `mvn` Serf tag (Minor Version Number, `vsn` is already being used for
to communicate `Major Version Number`).
Background: hashicorp/consul/issues/1346#issuecomment-151663152
2016-06-10 15:50:11 -04:00
36340c654d
Improve language re: fingerprinting
2016-06-10 15:50:11 -04:00
438becb28b
Pass the datacenter name in the heartbeat
...
Servers that are part of a different datacenter are added as backup
servers instead of primary servers.
2016-06-10 15:50:11 -04:00
d914f262f9
Consolidate all consul sync periodic go routines to handlers.
...
Only one pump and periodic loop now.
2016-06-10 15:50:11 -04:00
9fb0104def
Teach Client to reuse an Agent's consulSyncer.
...
"There can be only one."
2016-06-10 15:50:11 -04:00
47891fb559
Register two services each for clients and servers, http and rpc.
...
In order to give clients a fighting chance to talk to the right port,
differentiate RPC services from HTTP services by registering two
services with different tags. This yields
`rpc.nomad-server.service.consul` and
`http.nomad-server.service.consul` which is immensely more useful to
clients attempting to bootstrap their world.
2016-06-10 15:50:11 -04:00
861f900225
s/RpcVersion/RPCVersion/g
2016-06-10 15:50:11 -04:00
9f8d6c9f67
Move struct member to reduce diff context
2016-06-10 15:50:11 -04:00
410d85cc78
Rename the package from `client/rpc_proxy` to `client/rpcproxy`
...
Also rename `NewRpcProxy()` to just `New()` to avoid package stutter.
2016-06-10 15:50:11 -04:00
ec7c78f533
Only poll Consul for servers when Nomad heartbeats begin to fail
...
When a deadline timer of 2x Server's last requested TTL expires,
begin polling Consul for Nomad Servers.
2016-06-10 15:50:11 -04:00
17116fc5a7
Rebalance Nomad client RPCs among different Nomad servers.
...
Implement client/rpc_proxy.RpcProxy.
2016-06-10 15:50:11 -04:00
88f3422d7c
Rename NewConsulService to NewSyncer
2016-06-10 15:49:37 -04:00
768aab015d
Rename client/consul/sync.ConsulService to client/consul/sync.Syncer
...
Syncer describes the responsibility and actions of the type.
2016-06-10 15:49:37 -04:00
73e173673e
Rename client/config/config's ConsulConfig to ConsulAgentConfig
...
A follow up commit to the previous rename. More to come.
2016-06-10 15:48:36 -04:00
e36686a17d
Use consul/lib's RandomStagger
...
Removes four redundant copies of the method in the process.
2016-06-10 15:48:36 -04:00
6bf38bb213
Fixed the svc id generation scheme while pruning services
2016-06-07 08:14:11 -07:00
ba1a92eb8c
Handle errors during stats collection
2016-06-03 14:23:18 -07:00
35e31c1b81
Enqueing metrics only if they are not nil
2016-06-02 17:14:15 -04:00
7efde782fa
Sending metrics for tasks as well
2016-06-01 16:42:16 +02:00
7f172f314f
Pushing host stats
2016-05-31 04:05:47 +02:00
c0dc6cfbf2
Changing the api of the stats endpoints
2016-05-28 19:59:20 -07:00
a0c279f3b2
comments
2016-05-28 19:59:20 -07:00
4c85f02072
Stopping the metrics collector timers using defer and starting to collect host stats right away
2016-05-28 19:59:20 -07:00
b2e8495971
Fixing the alloc runner tests
2016-05-28 19:59:20 -07:00
f93820f40c
Added comments
2016-05-28 19:59:20 -07:00
c2da19bf11
Refactored the api for NewHostStatsCollector
2016-05-28 19:59:20 -07:00
c46400597e
Making the stats collection interval and number of data points to keep in memory configurable
2016-05-28 19:59:20 -07:00
b6e5227e7a
Renamed monitorUsage method
2016-05-28 19:59:20 -07:00
666b419dba
Acquiring locks before iterating allocations and tasks
2016-05-28 19:59:20 -07:00
73fa700a88
Showing host resource usage stats
2016-05-28 19:59:20 -07:00
9a96dddc07
Added some docs to resource stats endpoint
2016-05-28 19:42:34 -07:00
b9feae89ce
Making the conversion to Stats simpler
2016-05-28 19:42:34 -07:00
7569b1af2e
Collecting host stats
2016-05-28 19:42:34 -07:00
f3d0aecafe
Reporting time series of stats
2016-05-28 19:42:34 -07:00
0fb0e0237f
Added a client API to display resource usage of an allocation
2016-05-28 19:42:34 -07:00
2a99a2cfb6
Removing addition of the client service while reconciling services
2016-05-13 10:34:21 -07:00
439d7bf326
Fixed an agent test
2016-05-11 17:26:53 -07:00
347cb890d2
Removed allocID and task name from consul service
2016-05-11 16:26:41 -07:00
7287376eca
Using consul config from client config instead of reading from client options
2016-05-11 16:10:57 -07:00
83fed62a0a
Implemented registering client and server services
2016-05-11 16:07:02 -07:00
2f8a3532ad
Refactored the signature of NewConsulService
2016-05-11 15:22:58 -07:00
53d4681b61
Emit various debugging information with the results of the fingerprinter
2016-05-09 12:21:51 -07:00
2941b26244
Reading consul attr from copy of node attributes
2016-04-11 20:13:28 -04:00
f3d9ecf354
When reserving ports don't reserve network interface speed
2016-04-07 15:47:02 -07:00
57b0bbcb8b
Watching for node updates after registration completes
2016-04-01 13:41:52 -07:00
c6e80582a6
Making the drivers fingerprint periodically if they are configured to do so
2016-03-31 15:15:00 -07:00
e677c43667
Client not syncing services with consul until fingerprinting succeeds
2016-03-30 21:51:50 -07:00
b886636f6f
Using tickers instead of creating new timers
2016-03-25 14:18:04 -07:00
Diptanu Choudhury