Commit graph

14872 commits

Author SHA1 Message Date
Mahmood Ali a4640db7a6 drivers: implement streaming exec for executor based drivers
These simply delegate call to backend executor.
2019-05-10 19:17:14 -04:00
Mahmood Ali 3055fd53df executors: implement streaming exec
Implements streamign exec handling in both executors (i.e. universal and
libcontainer).

For creation of TTY, some incidental complexity leaked in.  The universal
executor uses github.com/kr/pty for creation of TTYs.

On the other hand, libcontainer expects a console socket and for libcontainer to
create the underlying console object on process start.  The caller can then use
`libcontainer.utils.RecvFd()` to get tty master end.

I chose github.com/kr/pty for managing TTYs here.  I tried
`github.com/containerd/console` package (which is already imported), but the
package did not work as expected on macOS.
2019-05-10 19:17:14 -04:00
Mahmood Ali 085d2ef759 executor: scaffolding for executor grpc handling
Prepare executor to handle streaming exec API calls that reuse drivers protobuf
structs.
2019-05-10 19:17:14 -04:00
Mahmood Ali 919827f2df
Merge pull request #5632 from hashicorp/f-nomad-exec-parts-01-base
nomad exec part 1: plumbing and docker driver
2019-05-09 18:09:27 -04:00
Mahmood Ali 2a555a7e74 add e2e tests for nomad exec 2019-05-09 16:49:08 -04:00
Mahmood Ali 09931bcdce add api support for nomad exec
Adds nomad exec support in our API, by hitting the websocket endpoint.

We introduce API structs that correspond to the drivers streaming exec structs.

For creating the websocket connection, we reuse the transport setting from api
http client.
2019-05-09 16:49:08 -04:00
Mahmood Ali 13c83ee38e drivers/docker: implement streaming exec 2019-05-09 16:49:08 -04:00
Mahmood Ali 57c18fec4e Add basic drivers conformance tests
Add consolidated testing package to serve as conformance tests for all drivers.
2019-05-09 16:49:08 -04:00
Mahmood Ali 050e88c668 vendor github.com/gorilla/websocket 2019-05-09 16:49:08 -04:00
Mahmood Ali 66982a1660 agent: add websocket handler for nomad exec
This adds a websocket endpoint for handling `nomad exec`.

The endpoint is a websocket interface, as we require a bi-directional
streaming (to handle both input and output), which is not very appropriate for
plain HTTP 1.0. Using websocket makes implementing the web ui a bit simpler. I
considered using golang http hijack capability to treat http request as a plain
connection, but the web interface would be too complicated potentially.

Furthermore, the API endpoint operates against the raw core nomad exec streaming
datastructures, defined in protobuf, with json serializer.  Our APIs use json
interfaces in general, and protobuf generates json friendly golang structs.
Reusing the structs here simplify interface and reduce conversion overhead.
2019-05-09 16:49:08 -04:00
Mahmood Ali 3c668732af server: server forwarding logic for nomad exec endpoint 2019-05-09 16:49:08 -04:00
Mahmood Ali ab2cae0625 implement client endpoint of nomad exec
Add a client streaming RPC endpoint for processing nomad exec tasks, by invoking
the relevant task handler for execution.
2019-05-09 16:49:08 -04:00
Mahmood Ali ce97689b79 drivers/mock: implement nomad exec interface 2019-05-09 16:49:08 -04:00
Mahmood Ali 13de875750 implemment streaming exec handling in driver grpc handlers
Also add a helper that converts the adapts the high level interface to the
low-level interface of nomad exec interfaces.
2019-05-09 16:49:08 -04:00
Yishan Lin 46bb7bbe08
Merge pull request #5667 from hashicorp/yishan/revised-nomadproject-structure
Revised NomadProject Structure
2019-05-09 13:03:50 -07:00
Yishan Lin 7df9d2570f Merge branch 'master' into yishan/revised-nomadproject-structure 2019-05-09 12:59:53 -07:00
Mahmood Ali 0d572b7cbe
Merge pull request #5660 from hashicorp/b-remove-EmitNodeEventsResponse-index
remove Index field from EmitNodeEventsResponse
2019-05-09 14:12:29 -04:00
Yishan Lin 4dbe029a99 Changed from "Container Management" to "Container Orchestration"
Removed Terraform mention in integrations without existing guide.
2019-05-09 10:14:01 -07:00
Danielle Lancashire 6f1bd7b52d changelog: Add missing entry for nomad alloc signal 2019-05-09 15:46:32 +02:00
Chris Baker 3f02119642
copied consts used by jobspec parsing from structs into api package, to avoid referencing structs package in jobspec (#5663) 2019-05-09 08:23:49 -04:00
Danielle Lancashire f56342121b changelog: Add changelog entries for consul improvements 2019-05-09 13:29:11 +02:00
Danielle 4a22fa0ee2
Merge pull request #5536 from hashicorp/dani/consul
Consul Catalog Integration Fixes
2019-05-09 13:22:54 +02:00
Danielle Lancashire 0da2924b2a consul: Document example check id 2019-05-09 13:22:22 +02:00
Mahmood Ali eac0e679aa
Merge pull request #5666 from hashicorp/docs-querystring
docs: Change querystring -> query string
2019-05-08 19:49:36 -04:00
Yishan Lin 968e67547a Revised NomadProject Structure
- Revised "What is Nomad" copy
- Added "Key Features" section with links to task drivers & device plugins with lift-and-shift from README
- Added "Who Uses Nomad" section with users, talks, blog posts
- Removed Hadoop YARN, Docker Swarm, HTCondor from comparisons
- Revamped Guides section
- Inserted "Installing Nomad", "Upgrading", "Integrations" as persistent in Guides navbar
- Split Installing Nomad into two paths for users (one for Sandbox with "Quickstart", one for Production)
- Surfaced "Upgrading" and "Integrations" section from documentation
- Changed "Job Lifecycle" section into "Deploying & Managing Applications"
- Reworked "Operations" into "Operating Nomad"
- Reworked "Security" into "Securing Nomad"
- Segmented Namespaces, Resource Quotas, Sentinel into "Governance & Policy" subsection
- Reworked "Spark integration" into its own "Analytical Workloads" section
2019-05-08 14:40:38 -07:00
Charlie Voiselle 45ff97cb7a querystring -> query string 2019-05-08 15:14:31 -04:00
Preetha Appan 4cceca18e9
update changelog for heartbeat fix 2019-05-08 13:38:29 -05:00
Preetha 1d02886bb6
Merge pull request #5654 from hashicorp/b-hearbeat-lockfix
Remove unnecessary locking and serverlist syncing in heartbeats
2019-05-08 13:36:39 -05:00
Charlie Voiselle ce5023ee95
Typo fixes (#5661)
* Typo fixes
* it's -> its
* plugable -> pluggable
2019-05-08 12:54:44 -04:00
Preetha Appan 3289e7f4a0
fix typo and add one more test scenario 2019-05-08 10:54:22 -05:00
Mahmood Ali 2e37cc4887 update changelog For GH-5587 and GH-5629 2019-05-08 09:49:04 -04:00
Mahmood Ali 6bf31702ca reorder CHANGELOG 2019-05-08 09:32:48 -04:00
Mahmood Ali 9d3f13e9b3 remove Index field from EmitNodeEventsResponse
`Index` is already included as part of `WriteMeta` embedding.

This is a backward compatible change: Clients never read the field; and
Server refernces to `EmitNodeEventsResponse.Index` would be using the
value in `WriteMeta`, which is consistent with other response structs.
2019-05-08 08:42:26 -04:00
Preetha Appan db6b291a5a
code review feedback 2019-05-07 16:23:32 -05:00
Chris Baker 7696c7336f update changelog to reflect metrics fix 2019-05-07 19:58:41 +00:00
Chris Baker 93ec1293be
stale allocation data leads to incorrect (and even negative) metrics (#5637)
* client: was not using up-to-date client state in determining which alloc count towards allocated resources

* Update client/client.go

Co-Authored-By: cgbaker <cgbaker@hashicorp.com>
2019-05-07 15:54:36 -04:00
Preetha 7d4fde7d3a
Fixed links in changelog 2019-05-07 14:24:18 -05:00
Michael Schurter d2742643fd
Merge pull request #5655 from hashicorp/docs-restarts
docs: fix default restart values
2019-05-07 10:43:53 -07:00
Lang Martin 014db9c023 CHANGELOG 2019-05-07 09:24:42 -04:00
Danielle 8bbfb609c6
Merge pull request #5656 from jweissig/patch-7
Fixed Typo
2019-05-07 14:12:44 +02:00
Justin Weissig 77e53de301
Fixed Typo
Fixed typo: schedular/scheduler.
2019-05-07 05:11:36 -07:00
Michael Schurter a1c3ce36bc
Merge pull request #5647 from hashicorp/e2e-tf
E2E Test Terraform/Packer Improvements
2019-05-06 15:42:52 -07:00
Michael Schurter e9526bec29 docs: fix default restart values 2019-05-06 14:52:44 -07:00
Preetha Appan 23b654a45a
Update changelog 2019-05-06 15:44:04 -05:00
Preetha 1538913a2a
Merge pull request #5628 from hashicorp/f-preemption-config
Add config to disable preemption for batch/service jobs
2019-05-06 15:40:35 -05:00
Michael Lange 3b1fc408e3
Merge pull request #5620 from Mongey/cm-color-ui
Converts ANSI terminal codes to HTML in logs
2019-05-06 13:01:44 -07:00
Preetha Appan b063fc81a4
Remove unnecessary locking and serverlist syncing in heartbeats
This removes an unnecessary shared lock between discovery and heartbeating
which was causing heartbeats to be missed upon retries when a single server
fails. Also made a drive by fix to call the periodic server shuffler goroutine.
2019-05-06 14:44:55 -05:00
Mahmood Ali f35ad92a8b
Merge pull request #5646 from hashicorp/some-ugorji-fixes
Codegen codec helpers for all nomad structs
2019-05-06 13:23:12 -04:00
Lang Martin 9f3f11df97
Merge pull request #5601 from hashicorp/b-config-parse-direct-hcl
config parse direct hcl
2019-05-06 12:05:19 -04:00
Lang Martin 2cae46bf31
Merge pull request #5631 from hashicorp/b-system-sched-constraint-errors
fix system sched constraint errors
2019-05-06 11:53:40 -04:00