Introduce a low-level background connection expiration mechanism wherein connections will be recycled periodically based on the size and health of the cluster.
For the vast majority of consul users, this will mean an average connection age of 150s. For 10K node clusters it will take ~3min for clusters to rebalance their connections. In the pathological case for a 100K cluster where 99K clients are in the minority talking to 1x server it will take ~26min to rebalance all connections.
It's possibe for clients recovering from a parititon to become fixated on a single server until the server or agent is restarted. This is of particular interest to long-running environments with stable agents, where `allow_stale` is true, and partitions occur periodically.
Increase the max idle time for agents talking to servers from 30s to 127s in order to allow for the reuse of connections that are being initiated by cron.
127s was chosen as the first prime above 120s (arbitrarily chose to use a prime) with the intent of reusing connections who are used by once-a-minute cron(8) jobs *and* who use a 60s jitter window (e.g. in vixie cron job execution can drift by up to 59s per job, or 119s for a once-a-minute cron job).
Namely, don't check the DNS names in TLS certificates when connecting to
other servers.
As of golang 1.3, crypto/tls no longer natively supports doing partial
verification (verifying the cert issuer but not the hostname), so we
have to disable verification entirely and then do the issuer
verification ourselves. Fortunately, crypto/x509 makes this relatively
straightforward.
If the "server_name" configuration option is passed, we preserve the
existing behavior of checking that server name everywhere.
No option is provided to retain the current behavior of checking the
remote certificate against the local node name, since that behavior
seems clearly buggy and unintentional, and I have difficulty imagining
it is actually being used anywhere. It would be relatively
straightforward to restore if desired, however.
Sean Chittenden