luxolus
/
open-nomad
2
0
Fork 0
Code Issues 1 Pull Requests Packages Releases Wiki Activity

consul/connect: avoid warn messages on connect proxy errors 

When creating a TCP proxy bridge for Connect tasks, we are at the
mercy of either end for managing the connection state. For long
lived gRPC connections the proxy could reasonably expect to stay
open until the context was cancelled. For the HTTP connections used
by connect native tasks, we experience connection disconnects.
The proxy gets recreated as needed on follow up requests, however
we also emit a WARN log when the connection is broken. This PR
lowers the WARN to a TRACE, because these disconnects are to be
expected.

Ideally we would be able to proxy at the HTTP layer, however Consul
or the connect native task could be configured to expect mTLS, preventing
Nomad from MiTM the requests.

We also can't mange the proxy lifecycle more intelligently, because
we have no control over the HTTP client or server and how they wish
to manage connection state.

What we have now works, it's just noisy.

Fixes #10933
This commit is contained in:
James Rasell 2021-08-05 11:27:35 +02:00
parent a15d03556c
commit a9a04141a3
No known key found for this signature in database
GPG Key ID: AA7D460F5C8377AA
2 changed files with 10 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.changelog/10951.txt Normal file
View File

@ -0,0 +1,3 @@
```release-note:improvement
consul/connect: Reduced the noise of log messages emitted for connect native tasks
```

15
client/allocrunner/consul_grpc_sock_hook.go
View File

@ -313,10 +313,10 @@ func proxyConn(ctx context.Context, logger hclog.Logger, destAddr string, conn n
defer cancel() defer cancel()
n, err := io.Copy(dest, conn) n, err := io.Copy(dest, conn)
if ctx.Err() == nil && err != nil { if ctx.Err() == nil && err != nil {
logger.Warn("error proxying to Consul", "error", err, "dest", destAddr, // expect disconnects when proxying http
"src_local", conn.LocalAddr(), "src_remote", conn.RemoteAddr(), logger.Trace("error message received proxying to Consul",
"bytes", n, "msg", err, "dest", destAddr, "src_local", conn.LocalAddr(),
) "src_remote", conn.RemoteAddr(), "bytes", n)
return return
} }
logger.Trace("proxy to Consul complete", logger.Trace("proxy to Consul complete",
@ -332,10 +332,9 @@ func proxyConn(ctx context.Context, logger hclog.Logger, destAddr string, conn n
defer cancel() defer cancel()
n, err := io.Copy(conn, dest) n, err := io.Copy(conn, dest)
if ctx.Err() == nil && err != nil { if ctx.Err() == nil && err != nil {
logger.Warn("error proxying from Consul", "error", err, "dest", destAddr, logger.Trace("error message received proxying from Consul",
"src_local", conn.LocalAddr(), "src_remote", conn.RemoteAddr(), "msg", err, "dest", destAddr, "src_local", conn.LocalAddr(),
"bytes", n, "src_remote", conn.RemoteAddr(), "bytes", n)
)
return return
} }
logger.Trace("proxy from Consul complete", logger.Trace("proxy from Consul complete",