Commit graph

14913 commits

Author SHA1 Message Date
Michael Schurter 7558747694 e2e: let the unindex clients do anything...
...and be debugable!
2019-05-03 07:54:55 -07:00
Michael Schurter f08dd66ffa e2e: ssh instructions + remove redundant naming 2019-05-03 07:54:34 -07:00
Michael Schurter 19889d6468 e2e: update deps and install nomad in packer
Nomad on the packer image will be overwritten by the sha specified in
the TF var, but including a base version on the packer image makes the
image valid for independent use.
2019-05-03 07:53:08 -07:00
Michael Schurter 13b62a68f7 e2e: enable systemd units so they start on boot 2019-05-03 07:52:03 -07:00
Jasmine Dahilig 016495c368 add non-empty string validation for datacenters 2019-05-03 06:48:02 -07:00
Preetha Appan b99a204582
Update deployment health on failed allocations only if health is unset
This fixes a confusing UX where a previously successful deployment's
healthy/unhealthy count would get updated if any allocations failed after
the deployment was already marked as successful.
2019-05-02 22:59:56 -05:00
Lang Martin c32cce51f0 client fingerprinting can keep multi ips on a device 2019-05-02 18:11:28 -04:00
Lang Martin 94f23016a2 client_test new test fingerprinting can keep multi ips on a device 2019-05-02 18:11:28 -04:00
Conor Mongey 49afa42af3
Return a htmlSafe string rather than use triple curlies 2019-05-02 21:47:24 +01:00
Conor Mongey 9ab4ef52c0
Converts ANSI terminal codes to HTML in logs 2019-05-02 21:47:23 +01:00
Mahmood Ali 85760a13ae
Merge pull request #5641 from hashicorp/b-task-networks-ports
client: handle 0.8 server network resources
2019-05-02 13:45:07 -04:00
Mahmood Ali 5c498112ff
Merge pull request #5640 from hashicorp/b-fix-omitempty
Update hashicorp/go-msgpack/codec
2019-05-02 12:12:56 -04:00
Hemanth Basappa 3fef02aa93 Add support in nomad for supporting raft 3 protocol peers.json 2019-05-02 09:11:23 -07:00
Mahmood Ali 7a32d3f3aa client: handle 0.8 server network resources
Fixes https://github.com/hashicorp/nomad/issues/5587

When a nomad 0.9 client is handling an alloc generated by a nomad 0.8
server, we should check the alloc.TaskResources for networking details
rather than task.Resources.

We check alloc.TaskResources for networking for other tasks in the task
group [1], so it's a bit odd that we used the task.Resources struct
here.  TaskRunner also uses `alloc.TaskResources`[2].

The task.Resources struct in 0.8 was sparsly populated, resulting to
storing of 0 in port mapping env vars:

```
vagrant@nomad-server-01:~$ nomad version
Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933+CHANGES)
vagrant@nomad-server-01:~$ nomad server members
Name                    Address      Port  Status  Leader  Protocol  Build  Datacenter  Region
nomad-server-01.global  10.199.0.11  4648  alive   true    2         0.8.7  dc1         global
vagrant@nomad-server-01:~$ nomad alloc status -json 5b34649b | jq '.Job.TaskGroups[0].Tasks[0].Resources.Networks'
[
  {
    "CIDR": "",
    "Device": "",
    "DynamicPorts": [
      {
        "Label": "db",
        "Value": 0
      }
    ],
    "IP": "",
    "MBits": 10,
    "ReservedPorts": null
  }
]
vagrant@nomad-server-01:~$ nomad alloc status -json 5b34649b | jq '.TaskResources'
{
  "redis": {
    "CPU": 500,
    "DiskMB": 0,
    "IOPS": 0,
    "MemoryMB": 256,
    "Networks": [
      {
        "CIDR": "",
        "Device": "eth1",
        "DynamicPorts": [
          {
            "Label": "db",
            "Value": 21722
          }
        ],
        "IP": "10.199.0.21",
        "MBits": 10,
        "ReservedPorts": null
      }
    ]
  }
}
```

Also, updated the test values to mimic how Nomad 0.8 structs are
represented, and made its result match the non compact values in
`TestEnvironment_AsList`.

[1] 24e9040b18/client/taskenv/env.go (L624-L639)
[2] https://github.com/hashicorp/nomad/blob/master/client/allocrunner/taskrunner/task_runner.go#L287-L303
2019-05-02 12:08:38 -04:00
Mahmood Ali a560f4aefc fix HashiStack eBay resource link 2019-05-02 11:19:44 -04:00
Mahmood Ali 7b72105802 Update hashicorp/go-msgpack/codec
To pick up the fix in https://github.com/hashicorp/go-msgpack/pull/8 .
2019-05-02 10:59:09 -04:00
Danielle Lancashire d824e00d1a consul: Do not deregister external checks
This commit causes sync to skip deregistering checks that are not
managed by nomad, such as service maintenance mode checks.  This is
handled in the same way as service registrations - by doing a Nomad
specific prefix match.
2019-05-02 16:54:18 +02:00
Danielle Lancashire 0b8e85118e consul: Use a stable identifier for services
The current implementation of Service Registration uses a hash of the
nomad-internal state of a service to register it with Consul, this means that
any update to the service invalidates this name and we then deregister, and
recreate the service in Consul.

While this behaviour slightly simplifies reasoning about service registration,
this becomes problematic when we add consul health checks to a service. When
the service is re-registered, so are the checks, which default to failing for
at least one check period.

This commit migrates us to using a stable identifier based on the
allocation, task, and service identifiers, and uses the difference
between the remote and local state to decide when to push updates.

It uses the existing hashing mechanic to decide when UpdateTask should
regenerate service registrations for providing to Sync, but this should
be removable as part of a future refactor.

It additionally introduces the _nomad-check- prefix for check
definitions, to allow for future allowing of consul features like
maintenance mode.
2019-05-02 16:54:18 +02:00
Yishan Lin 575814d4bb
Merge pull request #5636 from hashicorp/yishan/website-revised-resources-page
Yishan/website revised resources page
2019-05-01 18:45:47 -07:00
Yishan Lin 57302910be Cleaned spacing for syntax consistency. 2019-05-01 18:37:34 -07:00
Yishan Lin 53affa67b1 Fixed broken link to Nomad website in README 2019-05-01 11:46:03 -07:00
Yishan Lin 44f8e35586 Moved "Webinars" section below "Who Uses Nomad" 2019-05-01 11:29:22 -07:00
Mahmood Ali 914d219455 use ugorji/go/codec version that's in vendor.json 2019-05-01 12:47:14 -04:00
Mahmood Ali 21d21baf8b codegen codecs for nomad structs
`ls *[!_test].go` was ignoring any file that ends with `s.go` (or any of
the letter inside `[]`), including `structs.go`!
2019-05-01 12:42:55 -04:00
Lang Martin 29ea112586 system_sched & test cleanup comments 2019-05-01 12:25:26 -04:00
Lang Martin c490dacf76 system_sched_test extend the test to check ineligible nodes 2019-05-01 12:25:26 -04:00
Lang Martin c43bcbd35e system_sched when a node is filtered, don't mark failure 2019-05-01 12:25:26 -04:00
Lang Martin aecec5df1b system_sched_test create partially constrained job 2019-05-01 12:25:26 -04:00
Lang Martin 85dae46b51 dev/cluster meta tagged clients, raw_exec 2019-05-01 12:25:26 -04:00
Yishan Lin 0d326be223 Updated Resources Page
- Rewrote overview description
- Reformatted "Community" scetion for consistency with GitHub
- Removed IRC (redundant with Gitter)
- Added "Community Calls" section with links to prior sessions
- Added "Webinar" section with links to prior sessions
- Revised Bug Tracker copy
- Renamed "Tools for Provisioning & Experimentation" to "How to Install Nomad" section
- Bumped up the "How to Install Nomad" section right below to second on the list "Community"
- Removed "Videos" & "Blog Posts" section in favor of accurate segmentation between user talks, webinars, blog posts
- Added "Who Uses Nomad" section in consistency with new GitHub ReadME
- Removed the "Trusted By" section and disclaimer at bottom
2019-04-30 20:55:22 -07:00
Yishan Lin 15a175585e Update main tagline & boilerplate for Nomad 2019-04-30 20:51:18 -07:00
Yishan Lin 30bde7b25d
Merge pull request #5635 from hashicorp/yishan/revised-readme
Added Gitter to Resources + revised Overview in for ReadME.
2019-04-30 20:26:19 -07:00
Yishan Lin 76b4ca9c8c Added CircleCI post, Nic's two guides, and imgix Medium post into README 2019-04-30 18:42:51 -07:00
Yishan Lin 7c8b609a2b Corrected installation link via Omar's input + corrected typo 2019-04-30 13:56:13 -07:00
Yishan Lin 3e42c4067c Fixed link for Installing Nomad for Production 2019-04-30 13:39:32 -07:00
Lang Martin 9cf1c5a74d
Merge pull request #5581 from hashicorp/dev-make-boostrap-git-hooks
Dev make boostrap git hooks
2019-04-30 16:36:10 -04:00
Yishan Lin 9dcbeebbe1 Added Gitter to Resources + revised Overview for consistency. 2019-04-30 12:01:53 -07:00
Mahmood Ali c5150f81dd
Merge pull request #5621 from hashicorp/r-release-preflight
Update release/dist script with preflight
2019-04-30 14:52:03 -04:00
Mahmood Ali f47d3d5f8a add nomad streaming exec core data structures and interfaces
In this commit, we add two driver interfaces for supporting `nomad exec`
invocation:

* A high level `ExecTaskStreamingDriver`, that operates on io reader/writers.
  Drivers should prefer using this interface
* A low level `ExecTaskStreamingRawDriver` that operates on the raw stream of
  input structs; useful when a driver delegates handling to driver backend (e.g.
  across RPC/grpc).

The interfaces are optional for a driver, as `nomad exec` support is opt-in.
Existing drivers continue to compile without exec support, until their
maintainer add such support.

Furthermore, we create protobuf structures to represent exec stream entities:
`ExecTaskStreamingRequest` and `ExecTaskStreamingResponse`.  We aim to reuse the
protobuf generated code as much as possible, without translation to avoid
conversion overhead.

`ExecTaskStream` abstract fetching and sending stream entities.  It's influenced
by the grpc bi-directional stream interface, to avoid needing any adapter.  I
considered using channels, but the asynchronisity and concurrency makes buffer
reuse too complicated, which would put more pressure on GC and slows exec operation.
2019-04-30 14:02:29 -04:00
Mahmood Ali 40e62a6f17 Add ACL capabilities for nomad exec
This adds `alloc-exec` capability to allow operator to execute command into a
running task.  Furthermore, it adds `alloc-node-exec` capability, required when
the alloc task is raw_exec or a driver with no FSIsolation.
2019-04-30 14:02:16 -04:00
Mahmood Ali 7131ac3eef drivers/mock: extract command related operations
Extract command parsing and execution mocking into a separate struct.  Also,
allow mocking of different fs_isolation for testing.
2019-04-30 14:02:16 -04:00
Chris Baker a40477a7b8
test case for 5540 (#5590)
* client/metrics: modified metrics to use (updated) client copy of allocation instead of (unupdated) server copy

* updated armon/go-metrics to address race condition in DisplayMetrics
2019-04-30 10:31:35 -04:00
Lang Martin 2e643d26a2 config_parse leave the *HCL strings in place after converting times 2019-04-30 10:30:53 -04:00
Lang Martin 3ba6095fe3 config_parse_test additional config confirmation w/ sample json 2019-04-30 10:30:53 -04:00
Lang Martin fe9b31dcf9 config comment for future changes 2019-04-30 10:30:53 -04:00
Lang Martin 7e0e9b1ec1 govendor fetch github.com/hashicorp/hcl@99e2f22 2019-04-30 10:30:12 -04:00
Lang Martin 598112a1cc tag HCL bookkeeping keys with json:"-" to keep them out of the api 2019-04-30 10:29:14 -04:00
Lang Martin 43407cffe3 config_parse_test remove redundant parse direct test 2019-04-30 10:29:14 -04:00
Lang Martin b8e9c35cd0 config_parse remove unused multi-stage parsing via mapstructure 2019-04-30 10:29:14 -04:00
Lang Martin 1f86770456 config_parse_test test direct hcl parsing 2019-04-30 10:29:14 -04:00