Commit Graph

297 Commits

Author SHA1 Message Date
Jasmine Dahilig e1c73cdab5
add validation for job_gc_interval (#6277) 2019-09-05 11:20:46 -07:00
Tim Gross 03433f35d4 client/template: configuration for function blacklist and sandboxing
When rendering a task template, the `plugin` function is no longer
permitted by default and will raise an error. An operator can opt-in
to permitting this function with the new `template.function_blacklist`
field in the client configuration.

When rendering a task template, path parameters for the `file`
function will be treated as relative to the task directory by
default. Relative paths or symlinks that point outside the task
directory will raise an error. An operator can opt-out of this
protection with the new `template.disable_file_sandbox` field in the
client configuration.
2019-08-12 16:34:48 -04:00
Danielle Lancashire 063e4240c1
client: Add parsing and registration of HostVolume configuration 2019-08-12 15:39:08 +02:00
Nick Ethier 7806f4c597
Revert "client: add autofetch for CNI plugins"
This reverts commit 0bd157cc3b04fb090dd0d54affcae71496102ce8.
2019-08-08 15:10:19 -04:00
Nick Ethier af6b191963
client: add autofetch for CNI plugins 2019-07-31 01:04:09 -04:00
Nick Ethier ef83f0831b
ar: plumb client config for networking into the network hook 2019-07-31 01:04:06 -04:00
Pete Woods b47c5ca467
Allow the job GC interval to be configured from default of 5 minutes 2019-07-26 10:11:25 +01:00
Danielle Lancashire 0422f1b0c2
Support for hot reloading log levels 2019-07-24 13:37:08 +02:00
Nick Ethier 1b7fa4fe29
Optional Consul service tags for nomad server and agent services (#5706)
Optional Consul service tags for nomad server and agent services
2019-06-13 09:00:35 -04:00
Mahmood Ali a9f81f2daa client config flag to disable remote exec
This exposes a client flag to disable nomad remote exec support in
environments where access to tasks ought to be restricted.

I used `disable_remote_exec` client flag that defaults to allowing
remote exec. Opted for a client config that can be used to disable
remote exec globally, or to a subset of the cluster if necessary.
2019-06-03 15:31:39 -04:00
Nick Ethier ade97bc91f
fixup #5172 and rebase against master 2019-05-14 14:37:34 -04:00
Nick Ethier cab6a95668
Merge branch 'master' into pr/5172
* master: (912 commits)
  Update redirects.txt
  Added redirect for Spark guide link
  client: log when server list changes
  docs: mention regression in task config validation
  fix update to changelog
  update CHANGELOG with datacenter config validation https://github.com/hashicorp/nomad/pull/5665
  typo: "atleast" -> "at least"
  implement nomad exec for rkt
  docs: fixed typo
  use pty/tty terminology similar to github.com/kr/pty
  vendor github.com/kr/pty
  drivers: implement streaming exec for executor based drivers
  executors: implement streaming exec
  executor: scaffolding for executor grpc handling
  client: expose allocated memory per task
  client improve a comment in updateNetworks
  stalebot: Add 'thinking' as an exempt label (#5684)
  Added Sparrow link
  update links to use new canonical location
  Add redirects for restructing done in GH-5667
  ...
2019-05-14 14:10:33 -04:00
Preetha Appan 8f9ec85fe6
fix import order 2019-02-22 13:40:13 -06:00
Preetha Appan 3ab2e431b6
Move logger initialization to earlier step 2019-02-21 12:41:54 -06:00
Michael Schurter 32daa7b47b goimports until make check is happy 2019-01-23 06:27:14 -08:00
Michael Schurter be0bab7c3f move pluginutils -> helper/pluginutils
I wanted a different color bikeshed, so I get to paint it
2019-01-22 15:50:08 -08:00
Alex Dadgar cdcd3c929c loader and singleton 2019-01-22 15:11:57 -08:00
Alex Dadgar 471fdb3ccf
Merge pull request #5173 from hashicorp/b-log-levels
Plugins use parent loggers
2019-01-14 16:14:30 -08:00
Preetha Appan 7bd1440710
REfactor statedb factory config to set it directly in client config 2019-01-12 10:38:20 -06:00
Preetha Appan f059ef8a47
Modified destroy failure handling to rely on allocrunner's destroy method
Added a unit test with custom statedb implementation that errors, to
use to verify destroy errors
2019-01-12 10:37:12 -06:00
Alex Dadgar 5621086f50 Enable json logs 2019-01-11 11:36:37 -08:00
Nick Wales 7a7b5da0df Adds optional Consul service tags to nomad server and agent services, gh#4297 2019-01-09 22:02:46 +00:00
Alex Dadgar 1e3c3cb287 Deprecate IOPS
IOPS have been modelled as a resource since Nomad 0.1 but has never
actually been detected and there is no plan in the short term to add
detection. This is because IOPS is a bit simplistic of a unit to define
the performance requirements from the underlying storage system. In its
current state it adds unnecessary confusion and can be removed without
impacting any users. This PR leaves IOPS defined at the jobspec parsing
level and in the api/ resources since these are the two public uses of
the field. These should be considered deprecated and only exist to allow
users to stop using them during the Nomad 0.9.x release. In the future,
there should be no expectation that the field will exist.
2018-12-06 15:09:26 -08:00
Nick Ethier 85b221a1d6
nomad: add flag to disable publishing of job_summary metrics for dispatched jobs 2018-11-19 23:21:19 -05:00
Alex Dadgar 6f0ed6184b Fix client reloading and pass the plugin loaders to server and client 2018-10-16 16:56:55 -07:00
Alex Dadgar 183561cf82 Plugin loader initialization 2018-10-16 16:54:12 -07:00
Alex Dadgar a78cefec18 use int64 2018-10-16 15:34:32 -07:00
Preetha Appan 7c0d8c646c
Change CPU/Disk/MemoryMB to int everywhere in new resource structs 2018-10-16 16:21:42 -05:00
Alex Dadgar 01f8e5b95f renames 2018-10-04 14:57:25 -07:00
Alex Dadgar 52f9cd7637 fixing tests 2018-10-04 14:26:19 -07:00
Alex Dadgar bac5cb1e8b Scheduler uses allocated resources 2018-10-02 17:08:25 -07:00
Alex Dadgar 5c8697667e Node reserved resources 2018-09-29 18:44:55 -07:00
Alex Dadgar 3c19d01d7a server 2018-09-15 16:23:13 -07:00
Alex Dadgar 7739ef51ce agent + consul 2018-09-13 10:43:40 -07:00
Alex Dadgar c0de218747 plugin dir parsing 2018-08-30 13:43:09 -07:00
Chelsea Holland Komlo 3b5d5c7be8 remove logic to reload RPC connections from agent 2018-06-08 13:14:40 -04:00
Michael Schurter f8e12e6ee7 agent: global logger should use the same flags
Prior to this change logs from the global logger only used seconds:

```
2018/06/06 18:25:58 http: TLS handshake error from ...
```

After this change they properly use the microseconds flag:

```
2018/06/06 18:39:50.702447 http: TLS handshake error ...
```

They still lack a log level unfortunately.
2018-06-06 11:40:08 -07:00
Preetha Appan 82837839eb
Fix bug with determining when agent is a client
This fixes a bug introduced in commit e27caadca6 that sets a boolean flag
when the agent is a client. It incorrectly checked state before initializing
the client. This leads to Nomad clients not deregistering any services registered
in Consul after allocs are destroyed
2018-06-05 19:19:52 -05:00
Preetha Appan ce6d4a8d7a
Fix tests and move isClient to constructor 2018-06-01 15:59:53 -05:00
Alex Dadgar aca8d5cece Actually disable the schedulers 2018-05-31 13:11:11 -07:00
Alex Dadgar 44697efd9a safety guard 2018-05-22 14:45:34 -07:00
Alex Dadgar 586895965c Unit test for dev agent 2018-05-22 14:45:34 -07:00
Alex Dadgar 58d2a4c7c2 Do not bypass normal RPC codepath when running both client and server at once 2018-05-22 14:45:34 -07:00
Preetha Appan f04c9d2b19
minor code review fix 2018-04-10 15:33:01 -05:00
Preetha Appan 6b6e97e50c
Dev mode should never persist nodeid 2018-04-10 12:34:14 -05:00
Preetha Appan e1c3c7ab6c
Lint fixes 2018-04-10 11:22:16 -05:00
Preetha Appan 6d0e1c9fea
Use preconfigured nodeID if there isn't a persisted node ID, and persist it if its not persisted. 2018-04-10 08:47:33 -05:00
Alex Dadgar d179a09b83 WIP: Not setting node id properlperly 2018-04-09 18:01:28 -07:00
Chelsea Komlo 607e631714
Merge pull request #4046 from hashicorp/tls-same-file-reload
Check file contents when determining if agent should reload TLS confi…
2018-03-29 10:51:32 -04:00
Chelsea Holland Komlo be15c07110 make check fix 2018-03-28 19:11:51 -04:00
Chelsea Holland Komlo 58ada9bc42 return error when setting checksum; don't reload 2018-03-28 18:15:50 -04:00
Chelsea Holland Komlo 770ac13bc4 check for nil, remove unnecessary set checksum call 2018-03-28 13:29:53 -04:00
Chelsea Holland Komlo 2d5af7ff4d set TLS checksum when parsing config
Refactor checksum comparison, always set checksum if it is empty
2018-03-28 09:56:11 -04:00
Mildred Ki'Lya 1017cbe8ab
Allow to specify total memory on agent configuration
Allow to set the total memory of an agent in its configuration file. This
can be used in case the automatic detection doesn't work or in specific
environments when memory overcommit (using swap for example) can be
desirable.
2018-03-27 15:46:18 -05:00
Chelsea Holland Komlo 6e6d6b7e33 check file contents when determining if agent should reload TLS configuration 2018-03-27 15:42:20 -04:00
Chelsea Holland Komlo 96df419fff code review feedback 2018-03-26 10:55:22 -04:00
Chelsea Holland Komlo 66e44cdb73 Allow TLS configurations for HTTP and RPC connections to be reloaded separately 2018-03-21 17:51:08 -04:00
James Rasell 121c3bc997 Update Consul check params from using health-check to check. 2018-03-20 16:03:58 +01:00
James Rasell 15afef9b77 Allow Nomads Consul health checks to be configurable.
This change allows the client HTTP and the server HTTP, Serf and
RPC health check names within Consul to be configurable with the
defaults as previous. The configuration can be done via either a
config file or using CLI flags.

Closes #3988
2018-03-19 19:37:56 +01:00
Alex Dadgar b8607ad6d6 Heartbeat uses client rpc advertise and server defaults server rpc advertise addr 2018-03-16 16:47:08 -07:00
Alex Dadgar 52b7fb5361 Separate client and server rpc advertise addresses 2018-03-16 16:47:08 -07:00
Michael Schurter 1044bc0feb
Merge pull request #3984 from hashicorp/f-loosen-consul-skipverify
Replace Consul TLSSkipVerify handling
2018-03-16 11:21:28 -07:00
Michael Schurter 0971114f0c Replace Consul TLSSkipVerify handling
Instead of checking Consul's version on startup to see if it supports
TLSSkipVerify, assume that it does and only log in the job service
handler if we discover Consul does not support TLSSkipVerify.

The old code would break TLSSkipVerify support if Nomad started before
Consul (such as on system boot) as TLSSkipVerify would default to false
if Consul wasn't running. Since TLSSkipVerify has been supported since
Consul 0.7.2, it's safe to relax our handling.
2018-03-14 17:43:06 -07:00
Alex Dadgar 92cb552ff6 Always add core scheduler and detect invalid schedulers 2018-03-14 10:53:27 -07:00
Kyle Havlovitz 2ccf565bf6 Refactor redundancy_zone/upgrade_version out of client meta 2018-01-29 20:03:38 -08:00
Chelsea Komlo d09cc2a69f
Merge pull request #3492 from hashicorp/f-client-tls-reload
Client/Server TLS dynamic reload
2018-01-23 05:51:32 -05:00
Chelsea Holland Komlo 6c9f9c8ac3 adding additional test assertions; differentiate reloading agent and http server 2018-01-16 07:34:39 -05:00
Chelsea Holland Komlo 0708d34135 call reload on agent, client, and server separately 2018-01-08 09:56:31 -05:00
Chelsea Holland Komlo 6a2432659a code review fixups 2018-01-08 09:21:06 -05:00
Chelsea Holland Komlo 4e0dbd23cf prevent races when reloading, fully shut down raft 2018-01-08 09:21:06 -05:00
Chelsea Holland Komlo c0ad9a4627 add ability to upgrade/downgrade nomad agents tls configurations via sighup 2018-01-08 09:21:06 -05:00
Kyle Havlovitz 1c07066064 Add autopilot functionality based on Consul's autopilot 2017-12-18 14:29:41 -08:00
Kyle Havlovitz b775fc7b33
Added support for v2 raft APIs and -raft-protocol option 2017-12-12 10:17:16 -06:00
Chelsea Komlo 2dfda33703 Nomad agent reload TLS configuration on SIGHUP (#3479)
* Allow server TLS configuration to be reloaded via SIGHUP

* dynamic tls reloading for nomad agents

* code cleanup and refactoring

* ensure keyloader is initialized, add comments

* allow downgrading from TLS

* initalize keyloader if necessary

* integration test for tls reload

* fix up test to assert success on reloaded TLS configuration

* failure in loading a new TLS config should remain at current

Reload only the config if agent is already using TLS

* reload agent configuration before specific server/client

lock keyloader before loading/caching a new certificate

* introduce a get-or-set method for keyloader

* fixups from code review

* fix up linting errors

* fixups from code review

* add lock for config updates; improve copy of tls config

* GetCertificate only reloads certificates dynamically for the server

* config updates/copies should be on agent

* improve http integration test

* simplify agent reloading storing a local copy of config

* reuse the same keyloader when reloading

* Test that server and client get reloaded but keep keyloader

* Keyloader exposes GetClientCertificate as well for outgoing connections

* Fix spelling

* correct changelog style
2017-11-14 17:53:23 -08:00
Diptanu Choudhury 46bc4280b2 Adding support for tagged metrics 2017-11-01 13:15:06 -07:00
Diptanu Choudhury 524a1f0712 Publishing metrics for job summary 2017-11-01 13:15:06 -07:00
Michael Schurter c53aac9eea Agent Health Endpoint 2017-10-13 15:37:44 -07:00
Alex Dadgar 4173834231 Enable more linters 2017-09-26 15:26:33 -07:00
Alex Dadgar e5ec915ac3 sync 2017-09-19 10:08:23 -05:00
Chelsea Holland Komlo 1238efc2a8 improve documentation
move metrics to telemetry; copy to client config
2017-09-06 21:38:06 +00:00
Chelsea Holland Komlo 66fa05405a tagged metrics config options should be on telemetry config
better api example, add telemetry documentation
2017-09-06 15:25:36 +00:00
Chelsea Holland Komlo 751fc5324e add http endpoint for in memory metrics
prevent against flaky test due to timing/initialization issues
2017-09-06 13:51:19 +00:00
Armon Dadgar 3e46094cee Passthrough replication token for token/policy replication 2017-09-04 13:05:53 -07:00
Armon Dadgar 792f176a44 agent: thread ACL config to client 2017-09-04 13:04:45 -07:00
Armon Dadgar 7d4aa1975f agent: thread through ACL config to Server 2017-09-04 13:04:45 -07:00
Alex Dadgar 1a86aecf55 Add version package
This PR adds a version package and consolidates version strings into a
Version struct.
2017-08-16 15:44:21 -07:00
Michael Schurter 4b83eba599 Use seen more conservatively 2017-07-24 16:48:40 -07:00
Michael Schurter 809724ad8d Track whether Consul has ever been seen
Need a way to squelch Consul operation errors on shutdown. If it's never
been seen don't log errors about deregs failing.
2017-07-24 12:12:02 -07:00
Alex Dadgar 747d67eb3f Allow tuning of heartbeat ttls
This PR allows tuning of heartbeat TTLs. An example of very aggressive
settings is as follows:

```
server {
  heartbeat_grace = "1s"
  min_heartbeat_ttl = "1s"
  max_heartbeats_per_second = 200.0
}
```
2017-07-19 09:38:35 -07:00
Alex Dadgar c643e6b0d1 Add config options 2017-07-07 12:05:56 -07:00
Michael Schurter b9c9e6e557 Fix no_host_uuid parsing
Need to pointerify it to default to true since we can't tell false from
unset if it's not a pointer.
2017-07-03 17:41:20 -07:00
Michael Schurter 95a00cbef1 Fix path used by Nomad Server HTTP Check
Fixes #2701
2017-06-21 10:41:28 -07:00
Michael Schurter 0453c2709c Add new gc_max_allocs tuneable
More than gc_max_allocs may be running on a node, but terminal allocs
will be garbage collected to try to keep the total number below the
limit.
2017-05-11 17:18:02 -07:00
Michael Schurter c4ab8211b6 Skip https health check if verify_https_client is true 2017-05-03 12:19:02 -07:00
Michael Schurter 66aea59650 Fix error check for consul skip tls verify support 2017-05-02 17:38:18 -07:00
Michael Schurter e997ae44a5 Skip checks with TLSSkipVerify if it's unsupported
Fixes #2218
2017-04-19 12:45:34 -07:00
Michael Schurter cdb7d2ebb6 Use go-version instead of manual version parsing 2017-04-19 12:42:48 -07:00
Michael Schurter 947e31e9c2 Only register HTTPS agent check when Consul>=0.7.2
Support for TLSSkipVerify in other checks coming soon!
2017-04-19 12:42:48 -07:00
Michael Schurter e204a287ed Refactor Consul Syncer into new ServiceClient
Fixes #2478 #2474 #1995 #2294

The new client only handles agent and task service advertisement. Server
discovery is mostly unchanged.

The Nomad client agent now handles all Consul operations instead of the
executor handling task related operations. When upgrading from an
earlier version of Nomad existing executors will be told to deregister
from Consul so that the Nomad agent can re-register the task's services
and checks.

Drivers - other than qemu - now support an Exec method for executing
abritrary commands in a task's environment. This is used to implement
script checks.

Interfaces are used extensively to avoid interacting with Consul in
tests that don't assert any Consul related behavior.
2017-04-19 12:42:47 -07:00
Alex Dadgar a1a7941dec Various fixes
This PR:
* Uses Go 1.8 executable lookup
* Stores any err message from stats init method
* Allows overriding of Cpu Compute for hosts where it can't be detected
2017-03-14 12:56:31 -07:00