open-nomad/website/source/docs/jobops/inspecting.html.md

---
layout: "docs"
page_title: "Operating a Job: Inspecting State"
sidebar_current: "docs-jobops-inspection"
description: |-
  Learn how to inspect a Nomad Job.
---

# Inspecting state

Once a job is submitted, the next step is to ensure it is running. This section
will assume we have submitted a job with the name _example_.

To get a high-level over view of our job we can use the [`nomad status`
command](/docs/commands/status.html). This command will display the list of
running allocations, as well as any recent placement failures. An example below
shows that the job has some allocations placed but did not have enough resources
to place all of the desired allocations. We run with `-evals` to see that there
is an outstanding evaluation for the job:

```
$ nomad status example
ID          = example
Name        = example
Type        = service
Priority    = 50
Datacenters = dc1
Status      = running
Periodic    = false

Evaluations
ID        Priority  Triggered By  Status    Placement Failures
5744eb15  50        job-register  blocked   N/A - In Progress
8e38e6cf  50        job-register  complete  true

Placement Failure
Task Group "cache":
  * Resources exhausted on 1 nodes
  * Dimension "cpu exhausted" exhausted on 1 nodes

Allocations
ID        Eval ID   Node ID   Task Group  Desired  Status
12681940  8e38e6cf  4beef22f  cache       run      running
395c5882  8e38e6cf  4beef22f  cache       run      running
4d7c6f84  8e38e6cf  4beef22f  cache       run      running
843b07b8  8e38e6cf  4beef22f  cache       run      running
a8bc6d3e  8e38e6cf  4beef22f  cache       run      running
b0beb907  8e38e6cf  4beef22f  cache       run      running
da21c1fd  8e38e6cf  4beef22f  cache       run      running
```

In the above example we see that the job has a "blocked" evaluation that is in
progress. When Nomad can not place all the desired allocations, it creates a
blocked evaluation that waits for more resources to become available. We can use
the [`eval-status` command](/docs/commands/eval-status.html) to examine any
evaluation in more detail. For the most part this should never be necessary but
can be useful to see why all of a job's allocations were not placed. For
example if we run it on the _example_ job, which had a placement failure
according to the above output, we see:

```
nomad eval-status 8e38e6cf
ID                 = 8e38e6cf
Status             = complete
Status Description = complete
Type               = service
TriggeredBy        = job-register
Job ID             = example
Priority           = 50
Placement Failures = true

Failed Placements
Task Group "cache" (failed to place 3 allocations):
  * Resources exhausted on 1 nodes
  * Dimension "cpu exhausted" exhausted on 1 nodes

Evaluation "5744eb15" waiting for additional capacity to place remainder
```

More interesting though is the [`alloc-status`
command](/docs/commands/alloc-status.html). This command gives us the most
recent events that occurred for a task, its resource usage, port allocations and
more:

```
nomad alloc-status 12
ID            = 12681940
Eval ID       = 8e38e6cf
Name          = example.cache[1]
Node ID       = 4beef22f
Job ID        = example
Client Status = running

Task "redis" is "running"
Task Resources
CPU    Memory           Disk     IOPS  Addresses
2/500  6.3 MiB/256 MiB  300 MiB  0     db: 127.0.0.1:57161

Recent Events:
Time                   Type        Description
06/28/16 15:46:42 UTC  Started     Task started by client
06/28/16 15:46:10 UTC  Restarting  Task restarting in 30.863215327s
06/28/16 15:46:10 UTC  Terminated  Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
06/28/16 15:37:46 UTC  Started     Task started by client
06/28/16 15:37:44 UTC  Received    Task received by client
```

In the above example we forced killed the Docker container so that we could see
in the event history that Nomad detected the failure and restarted the
allocation.

The `alloc-status` command is a good starting to point for debugging an
application that did not start. In this example task we are trying to start a
redis image using `redis:2.8` but the user has accidentally put a comma instead
of a period, typing `redis:2,8`.


When the job is run, it produces an allocation that fails. The `alloc-status`
command gives us the reason why:

```
nomad alloc-status c0f1
ID            = c0f1b34c
Eval ID       = 4df393cb
Name          = example.cache[0]
Node ID       = 13063955
Job ID        = example
Client Status = failed

Task "redis" is "dead"
Task Resources
CPU  Memory   Disk     IOPS  Addresses
500  256 MiB  300 MiB  0     db: 127.0.0.1:23285

Recent Events:
Time                   Type            Description
06/28/16 15:50:22 UTC  Not Restarting  Error was unrecoverable
06/28/16 15:50:22 UTC  Driver Failure  failed to create image: Failed to pull `redis:2,8`: API error (500): invalid tag format
06/28/16 15:50:22 UTC  Received        Task received by client
```

Not all failures are this easily debuggable. If the `alloc-status` command shows
many restarts occurring as in the example below, it is a good hint that the error
is occurring at the application level during start up. These failures can be
debugged by looking at logs which is covered in the [Nomad Job Logging
documentation](/docs/jobops/logs.html).

```
$ nomad alloc-status e6b6
ID            = e6b625a1
Eval ID       = 68b742e8
Name          = example.cache[0]
Node ID       = 83ef596c
Job ID        = example
Client Status = pending

Task "redis" is "pending"
Task Resources
CPU  Memory   Disk     IOPS  Addresses
500  256 MiB  300 MiB  0     db: 127.0.0.1:30153

Recent Events:
Time                   Type        Description
06/28/16 15:56:16 UTC  Restarting  Task restarting in 5.178426031s
06/28/16 15:56:16 UTC  Terminated  Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
06/28/16 15:56:16 UTC  Started     Task started by client
06/28/16 15:56:00 UTC  Restarting  Task restarting in 5.00123931s
06/28/16 15:56:00 UTC  Terminated  Exit Code: 1, Exit Message: "Docker container exited with non-zero exit code: 1"
06/28/16 15:55:59 UTC  Started     Task started by client
06/28/16 15:55:48 UTC  Received    Task received by client
```