Commit graph

1250 commits

Author SHA1 Message Date
James Phillips 96bff003b7
Adds basic support for node IDs. 2017-01-17 22:47:59 -08:00
Kyle Havlovitz b3fdfb09d8
Minor formatting tweaks as a follow-up to #2654 2017-01-17 19:20:29 -05:00
Kyle Havlovitz 9080006a37
Add support for multiple metadata filters to remaining endpoints
Enabled multiple meta filters for /v1/catalog/nodes and /v1/catalog/services
2017-01-13 20:49:13 -05:00
Kyle Havlovitz e37f25dc02
Add node metadata filtering to remaining health/catalog endpoints 2017-01-13 20:08:43 -05:00
James Phillips c837d055cd
Breaks up the state store into several files. 2017-01-13 11:47:16 -08:00
Kyle Havlovitz 3620e5e199
Fix inconsistency in TestStateStore_ServicesByNodeMeta 2017-01-12 19:46:58 -05:00
Kyle Havlovitz 51a1834667
Add -node-meta to agent command line options 2017-01-11 16:09:04 -05:00
Kyle Havlovitz d77890a011
Validate metadata config earlier and handle multiple filters 2017-01-11 15:12:03 -05:00
Kyle Havlovitz 6b5cf20b1c
Fix formatting 2017-01-09 13:49:33 -08:00
Kyle Havlovitz aee766baba
Add meta key validations and more tests 2017-01-09 11:21:49 -08:00
Kyle Havlovitz b25f4c7d35
Add tests for node metadata functionality 2017-01-05 17:21:56 -08:00
Kyle Havlovitz 2a423c6e2c
Add support for setting node metadata fields 2017-01-05 14:10:26 -08:00
James Phillips 927aa3789a
Adds ACL management support to the agent. 2016-12-14 07:07:41 -08:00
nobody d069c0805e fix fmt.Errorf error
missing argument for Errorf("%q"): format reads arg 1, have only 0 args
2016-12-14 18:57:12 +08:00
James Phillips 71a775d986 Merge pull request #2592 from hashicorp/acl-complete-node-session
Adds complete ACL coverage for nodes and sessions.
2016-12-13 13:55:07 -08:00
James Phillips 535d6b21b4
Adds complete ACL coverage for /v1/session endpoints. 2016-12-12 21:59:22 -08:00
James Phillips 4aa070834b
Adds complete ACL coverage for /v1/internal/ui/node endpoints. 2016-12-12 18:22:10 -08:00
James Phillips ac967176ee
Adds complete ACL support for /v1/query/<query id or name>/execute.
This was already supported by previous changes to the ACL filter, so
we just added a test to show it working.
2016-12-12 17:28:06 -08:00
James Phillips 7e7c1ebb13
Fixes implementation of node ACLs for /v1/catalog/node/<node>.
This would return a "permission denied" error, but this changes it to
return the same response as a node that doesn't exist (as was originally
intended and written in the code comments).
2016-12-12 16:53:31 -08:00
James Phillips 7a21a591b8
Adds full ACL coverage for /v1/health endpoints. 2016-12-12 16:28:52 -08:00
Kyle Havlovitz 402f30d52e Merge pull request #2591 from hashicorp/snapshot-interval
Change raft snapshot interval to 5 seconds
2016-12-12 19:15:10 -05:00
James Phillips 8db53b4ac2
Adds complete ACL coverage for /v1/coordinate/nodes and Coordinate.Update RPC. 2016-12-12 14:52:27 -08:00
James Phillips 67f1fe8d0b
Adds support for a new "acl_agent_token" which is used for internal
catalog operations.
2016-12-12 14:52:27 -08:00
James Phillips 99a5ae2737
Bans anonymous queries that aren't tied to a session.
This gets us coverage of PQ creation under the existing service
policy or the soon-to-be-added session policy.
2016-12-12 14:52:27 -08:00
Kyle Havlovitz 3ae8bedd6d
Change raft snapshot interval to 5 seconds 2016-12-12 13:31:42 -05:00
James Phillips 1ec02e5af4
Adds complete ACL coverage for /v1/catalog/service/<service>. 2016-12-12 08:34:15 -08:00
James Phillips 4dc3b17078
Adds complete ACL coverage for /v1/catalog/nodes. 2016-12-10 16:49:19 -08:00
James Phillips 63a6abe03c
Adds complete ACL coverage for /v1/catalog/node/<node>. 2016-12-10 16:49:19 -08:00
James Phillips 8ed0b81adb
Adds complete ACL coverage for /v1/catalog/deregister.
This included some state store helpers to make this more efficient.
2016-12-09 21:04:44 -08:00
James Phillips 7fa693a338
Adds complete ACL coverage for /v1/catalog/register. 2016-12-09 21:04:37 -08:00
James Phillips 2b19a5340c
Removes the exception for the "consul" service in the catalog. 2016-12-07 17:58:23 -08:00
Kyle Havlovitz 26a26a489d Add retry with backoff to initial bootstrap checks (#2561) 2016-12-01 17:05:02 -05:00
Seth Vargo 1c55429a38
Add an API method for determining the best status
Given a list of HealthChecks, this determines the "best" status for the
collective group. This is useful for nodes and services, which may have
multiple checks associated with them.
2016-11-29 18:41:46 -05:00
Kyle Havlovitz 2d37a07476 Add keyring http endpoints 2016-11-22 20:10:43 -05:00
Kyle Havlovitz 9adc3854d1 Retry with backoff on session invalidation failure (#2475) 2016-11-04 21:53:22 -07:00
James Phillips be4056789f Moves the snapshot package up one level. (#2472) 2016-11-03 21:36:25 -07:00
Kyle Havlovitz 169cae2203 Disallow -bootstrap-expect flag in dev mode (#2464) 2016-11-03 01:54:43 -04:00
Kyle Havlovitz 440611f9f7 Add snapshot inspect subcommand (#2451) 2016-10-31 19:37:27 -04:00
Kyle Havlovitz c6f461aa25 Enable snapshots in dev mode (#2453) 2016-10-31 14:39:47 -04:00
James Phillips bc29610124 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
Kyle Havlovitz f0aa65754b Wait for agent joins to finish in TestClient_RPC 2016-10-25 17:48:11 -07:00
Kyle Havlovitz 2379b822ad Add wait logic to TestClient_RPC_Pool 2016-10-25 17:48:11 -07:00
James Phillips b423dee303
Fixes port numbers in peers.info. 2016-10-05 18:09:15 -07:00
James Phillips 8e76af4311 Merge pull request #2319 from hashicorp/f-bootstrap-abort
Adds check that aborts bootstrap mode if there's an existing cluster.
2016-09-01 09:49:03 -07:00
James Phillips 94c0e961eb
Fixes error message in test. 2016-09-01 09:48:08 -07:00
James Phillips ce93b82e1e
Makes port selection atomic in unit tests. 2016-09-01 01:01:28 -07:00
James Phillips d04a706a7c
Tweaks comment to be more correct. 2016-08-31 23:54:53 -07:00
James Phillips 4dd9b4b08a
Adds check that aborts bootstrap mode if there's an existing cluster. 2016-08-31 21:25:56 -07:00
James Phillips 750e1751ac
Copies the member data instead of referencing by pointer. 2016-08-30 16:54:21 -07:00
James Phillips 6be1e07fec
Makes the Raft configuration API easier to consume. 2016-08-30 11:30:56 -07:00
James Phillips 5df4b6bef2
Adds a log warning when operator peer changes occur. 2016-08-30 10:23:32 -07:00
James Phillips 1b7a16b7d3
Adds new consul operator endpoint, CLI, and ACL and some basic Raft commands. 2016-08-30 00:02:50 -07:00
James Phillips 29e52307cb
Makes empty checkServiceNode return a nil.
The change in #2308 had an inadvertent interface change, so we fix that with
a special case in this fix.
2016-08-29 19:12:07 -07:00
James Phillips 327fe725d9
Preallocates result struct, which was a profiling hot spot. 2016-08-26 16:34:28 -07:00
James Phillips c5b6ac3655
Removes leader_lease_timeout from stats. 2016-08-25 15:39:19 -07:00
James Phillips 2f4c237cff
Adds a max raft multiplier and tweaks documentation. 2016-08-25 15:36:05 -07:00
James Phillips 5df36fbd82
Stops scaling the commit timeout. 2016-08-25 15:05:40 -07:00
James Phillips f65ef936cb
Increases RPC hold timeout for new default timing.
Rather than scale this we just bump it up a bit. It'll be on the edge in
the lower-performance default mode, and will have plenty of margin in the
high-performance mode. This seems like a reasonable compromise to keep the
logic here simple vs. scaling, and seems inline with the expectations of
the different modes of operation.
2016-08-24 23:35:28 -07:00
James Phillips b339b0d2fc
Adds performance tuning capability for Raft, detuned defaults, and supplemental docs. 2016-08-24 21:58:37 -07:00
James Phillips 0bdbdf1ba8 Merge pull request #2226 from abhinavdahiya/rm-health-unknown
Fixes #1775; Removes 'unknown' state
2016-08-17 17:51:04 -07:00
James Phillips 1f539d9914
Makes the filled-in parts of ServiceNode more explicit. 2016-08-12 18:25:36 -07:00
David van Geest 360e196c93
Translate Address to tagged WAN address in HTTP API when appropriate. 2016-08-12 18:25:36 -07:00
James Phillips d11a7a197c
Removes upper end of muxado handler. 2016-08-09 18:16:41 -07:00
James Phillips 97a25e8564
Closes the conn on bad protocol version. 2016-08-09 18:13:53 -07:00
James Phillips 359587f70e
Removes support for muxado and protocol version 1. 2016-08-09 18:10:04 -07:00
James Phillips 99ab3390c2
Updates hashicorp/hcl and hashicorp/hil.
This required a small mod to core Consul code to cope with an interface
change.
2016-08-09 17:24:13 -07:00
James Phillips ff6d42389c Merge pull request #2222 from hashicorp/f-raft-v2
Integrates Consul with "stage one" of HashiCorp Raft library v2.
2016-08-09 16:04:48 -07:00
James Phillips cce38f9a4b
Moves the peers.info content down into a constant. 2016-08-09 11:56:39 -07:00
James Phillips 7aaa4bc913
Adds peers back into bootstrap log, makes initial case consistent. 2016-08-09 11:52:41 -07:00
James Phillips 7f58b05dfe
Tweaks select style. 2016-08-09 11:33:42 -07:00
James Phillips 544169999c
Adds I/O-sensitive metrics to ACL replication operations. 2016-08-09 11:32:12 -07:00
James Phillips 820509760d
Switches to a smooth rate limit vs. a bursty one. 2016-08-09 11:29:12 -07:00
James Phillips 129e327bc9
Clarifies replication index shown in the log message. 2016-08-09 11:10:32 -07:00
James Phillips 4203612bd7
Returns from the shutdown wait right away. 2016-08-09 11:09:48 -07:00
James Phillips e03fbef6b3
Moves ACL ID sorting interface onto the iterator. 2016-08-09 11:08:26 -07:00
James Phillips 0fa059ec49
Switches all ACL caches to 2Q. 2016-08-09 11:00:22 -07:00
James Phillips 1e75fa0362
Moves ACL ID generation down into the endpoint.
We don't want ACL replication to have this behavior so it was a
little dangerous to have in the shared helper function.
2016-08-09 00:11:00 -07:00
James Phillips 06a510a808
Removes unsafe "recover to empty" code.
This isn't safe because it would implicitly commit all outstanding log
entries. The new Raft library already has logic to not start a vote if
the current node isn't in the configuration, so this shoudn't be needed.
2016-08-08 19:19:19 -07:00
James Phillips dd3169b395
Tweaks recovery based on interface changes. 2016-08-08 19:19:18 -07:00
James Phillips 19004e7095
Moves to a safer design where we don't ingest the initial peers.json file. 2016-08-08 19:19:18 -07:00
James Phillips 44c468995f
Touches up Raft integration after latest changes. 2016-08-08 19:19:18 -07:00
James Phillips fc25145e85
Formats log messages to be consistent. 2016-08-08 19:19:18 -07:00
James Phillips 6b157eada0
Adds more comments about the raftSafeFn. 2016-08-08 19:19:18 -07:00
James Phillips fcd8bb157a
Clarifies a comment about no-op peer operations. 2016-08-08 19:19:18 -07:00
James Phillips 2bf633f206
Adds back "safing" the configuration when a server leaves. 2016-08-08 19:19:18 -07:00
James Phillips 6c8e8271e2
Integrates Consul with new version of Raft library. 2016-08-08 19:19:17 -07:00
James Phillips 4a931ae12e
Adds an ACL replication status endpoint. 2016-08-04 23:30:16 -07:00
James Phillips c94f1e1b83
Increases the ACL cache size to 10k. 2016-08-04 18:03:07 -07:00
James Phillips 3906517f70
Adds a full integrated test for ACL replication. 2016-08-04 17:59:08 -07:00
James Phillips f639f49cc0
Adds remaining core replication tests. 2016-08-04 16:33:40 -07:00
James Phillips defb39f8d4
Removes a TODO comment.
Decided we don't need to log anything about the token here. If the
token is not valid then the client will get an error about that, so
anything that can happen here is related to talking to the server in
the ACL datacenter, so not specific to the token.
2016-08-04 07:46:59 -07:00
James Phillips 93a7fd0561
Adds tests for the ACL reconcile algorithm. 2016-08-03 21:24:09 -07:00
James Phillips 796933b45b
Activates fallback to replicated ACLs. 2016-08-03 21:24:09 -07:00
James Phillips 9cece515c0
Adds basic ACL replication plumbing. 2016-08-03 21:24:04 -07:00
Abhinav Dahiya 9dc52449e3 Fixes #1775; Removes 'unknown' state
Signed-off-by: Abhinav Dahiya <abhinavdtu2012@gmail.com>
2016-07-30 19:33:14 +05:30
James Phillips a1266e4164 Adds some supplemental tests for RPC "no leader" retries.
This adds some extra tests for #2175.
2016-07-11 17:32:26 -06:00
Armon Dadgar 2d8cf9ef4a consul: change tests to not expect ErrNoLeader 2016-07-10 13:24:18 -04:00
Armon Dadgar 5d0a977bdf consul: Refactor forward to hold RPC when no leader is known 2016-07-10 13:24:06 -04:00
Armon Dadgar 191876f87e consul: Add RPCHoldTimeout as tunable hold period 2016-07-10 13:23:43 -04:00
Ryan Uber d8fd470f4f Merge pull request #1837 from cleung2010/obfuscate-acl-token
Obfuscate token for lookupACL error
2016-07-05 13:56:49 -07:00
Calvin Leung Huang 38134f1b8c Fix substring length on obfuscated token 2016-07-05 15:53:30 -04:00
Ryan Uber 577523fc73 consul: sort source node first if at position <= 10 in PQ's 2016-07-01 14:28:58 -07:00
Ryan Uber e9960e6c85 Merge pull request #2137 from hashicorp/f-pq-near
Support "near" parameter in prepared query service block
2016-07-01 12:28:48 -07:00
Ryan Uber ccbe86d7a8 consul: mention magic _agent token in struct comments 2016-07-01 11:50:30 -07:00
Ryan Uber ebacaa2d67 consul: send agent source data as separate query source 2016-06-30 16:51:18 -07:00
Ryan Uber 782a081925 consul: use source parameter for near prepared queries 2016-06-30 12:11:20 -07:00
Ryan Uber 270270a33a consul: send origin node + dc when executing prepared queries 2016-06-21 15:34:26 -07:00
Ryan Uber 925915c6ac consul: test baked-in distance sort 2016-06-21 12:54:18 -07:00
Ryan Uber 114e57fff1 consul: use the Near field instead of PreferLocal 2016-06-21 12:39:40 -07:00
James Phillips 8358df599d Merge pull request #2127 from hashicorp/b-remote-consuls-locking
Ensure locking of `Server`'s `remoteConsuls`.
2016-06-21 10:00:04 -07:00
James Phillips f9e2900692 Merge pull request #2131 from hashicorp/b-misc-microoptimizations
Misc micro optimizations
2016-06-21 09:59:01 -07:00
Sean Chittenden ebdb72ce0a
Ensure locking of Server's remoteConsuls. 2016-06-20 22:59:49 -07:00
Sean Chittenden 72f7a4061c
Misc comment improvements 2016-06-20 15:29:38 -07:00
Sean Chittenden 9bf6e61655
Initialize a non-empty number of Consul Datacenters. No functional change. 2016-06-20 15:26:59 -07:00
Sean Chittenden b78c95d37e
Prefer rand.Int31n() over rand.Int31(). 2016-06-20 15:26:27 -07:00
Sean Chittenden e81bf2a505
Fix deadlock in Consul RTT.
- consul/rtt.go:388: s.getDatacentersByDistance().  Acquires RLock()
- consul/rtt.go:341: sortDatacentersByDistance() RLock still held.
- consul/rtt.go:282: getDatacenterDistance() RLock still held.
- consul/rtt.go:268: getNodesForDatacenter(). Attempts to reacquire RLock(), hangs indefinitely.
2016-06-20 14:59:54 -07:00
Ryan Uber 89fe991ab7 consul: test raw PreferLocal functionality 2016-06-20 14:53:13 -07:00
Ryan Uber 1fef85cd2e consul: support PreferLocal in PQ's 2016-06-20 14:24:40 -07:00
Sean Chittenden 7482a9207d
Chase casting types.CheckID to a string into the state_store.
It turns out the indexer can only use strings as arguments when
creating a query.  Cast `types.CheckID` to a `string` before calling
into `memdb`.

Ideally the indexer would be smart enough to do this at compile-time,
but I need to look into how to do this without reflection and the
runtime package.  For the time being statically cast `types.CheckID`
to a `string` at the call sites.
2016-06-07 16:59:02 -04:00
Sean Chittenden ff45f8c8ff
Revert "Move structs.CheckID to a new top-level package, types."
This reverts commit 2bbd52e3b44ff1b60939a8400264d534662d6d51.
2016-06-07 16:59:02 -04:00
Sean Chittenden a4554b945c
Move structs.CheckID to a new top-level package, types.
Per discussion w/ @slackpad, move this type to its own top-level package
2016-06-07 16:59:02 -04:00
Sean Chittenden cd68cd3868
Move structs.CheckID to a new top-level package, types.
Per discussion w/ @slackpad, move this type to its own top-level package
2016-06-07 16:59:02 -04:00
Sean Chittenden 0857e93d0b
Float a type balloon. Some strings are square pegs in round holes.
This experiment was brought about because of variable naming
confusion where name and checkIDs were interchanged.  Gave CheckID
an Qualified Type Name and chased downstream changes.
2016-06-07 16:59:02 -04:00
James Phillips ffcba3df58 Merge pull request #2028 from hashicorp/f-atomic-kv
Adds support for atomic transactions spanning multiple KV entries.
2016-05-15 13:46:05 -07:00
Sean Chittenden 3756fb23a6
Remove unused peers variable from setupRaft(). 2016-05-15 06:40:46 -07:00
James Phillips a11f32a1da Adds a get-tree verb to KV transaction operations. 2016-05-13 16:57:39 -07:00
James Phillips 0f94a7a326 Switches GETs to a filtering model for ACLs. 2016-05-13 15:58:55 -07:00
James Phillips 5fd99b13ef Removes null results for deletes, and preps for more than one result from an operation. 2016-05-13 01:47:55 -07:00
James Phillips 2649a6336e Adds a read-only optimized path for transactions. 2016-05-13 00:34:05 -07:00
James Phillips 0c34ed078c Adds a comment for the txnKVS() function. 2016-05-12 16:11:26 -07:00
James Phillips 88b1c7d054 Makes get fail a transaction if the key doesn't exist. 2016-05-11 14:18:31 -07:00
James Phillips 3d35acaa90 De-nests the KV output structure (removes DirEnt member). 2016-05-11 13:48:03 -07:00
James Phillips 04a13ec3d7 Switches to "KV" instead of "KV" for the KV operations. 2016-05-11 10:58:27 -07:00
James Phillips dc662f7e35 Refactors TxnRequest/TxnResponse into a form that will allow non-KV ops.
This isn't needed/used yet, but it's a good hook to get in there so we
can add more atomic operations in the future. The Go API hides this detail
so that feels like a KV-specific API. The implications on the REST API are
pretty minimal.
2016-05-11 01:39:10 -07:00
James Phillips d980cbcd9d Moves txn code into a new endpoint, not specific to KV. 2016-05-10 21:58:02 -07:00
James Phillips 907d8bab34 Fixes some go vet findings in a unit test. 2016-05-10 20:01:52 -07:00
Sean Chittenden 94e2766423
Remove stray type definition
Noticed while working on Nomad Client's server selection code.
2016-05-10 18:56:28 -07:00
James Phillips 4eb89481df Adds internal endpoint read ACL support and full unit tests. 2016-05-10 11:23:47 -07:00
James Phillips 6a96e052c4 Adds an empty get test case. 2016-05-09 22:18:26 -07:00
James Phillips 471160d8f0 Performs basic plumbing of KVS transactions through all the layers. 2016-05-09 22:15:49 -07:00
James Phillips dca00c96f7 Adds state store support for atomic KVS ops. 2016-05-05 15:46:59 -07:00
James Phillips a1a59bee73 Splits existing KVS operations into *Txn helpers for later reuse. 2016-05-04 14:20:11 -07:00
James Phillips 9185450fd5 Moves KVS-related state store code out into its own set of files. 2016-05-02 16:21:04 -07:00
Sean Chittenden c16b1ca178 Add the list of Raft peers to Consul's Stats
```
% consul info
[snip]
raft:
[snip]
	raft_peers = 127.0.0.1:8300
[snip]
```

Poached from: Nomad Project
2016-04-28 15:08:48 -07:00
James Phillips 79153c3014 Merge pull request #1884 from mtchavez/1541-data-dir-perms
command: Data directory permission error message
2016-04-12 22:06:49 -07:00
James Phillips 6e177a9b44 Merge pull request #1895 from shoenig/fixtypo
doc: fix trivial typo s/NewFSMPath/NewFSM/
2016-04-12 21:53:24 -07:00
James Phillips 3f340716fd Adds a clone method to HealthCheck and uses that in local.go. 2016-04-11 00:05:39 -07:00
Chavez c9602c561c Add description to rpc test client pool member failure message 2016-04-01 19:17:38 -07:00
Seth Hoenig 7f67c123b7 doc: fix trivial typo s/NewFSMPath/NewFSM/ 2016-03-29 20:52:17 -05:00
Sean Chittenden 5ae7835988 Rename server_details package to agent 2016-03-29 17:39:19 -07:00