Whoops, fix vendoring
This commit is contained in:
parent
a6682405a3
commit
04cfa4f88d
|
@ -1,27 +0,0 @@
|
|||
Copyright (c) 2015 The Go Authors. All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google Inc. nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@ -1,181 +0,0 @@
|
|||
# The Vendor Tool for Go
|
||||
`go get -u github.com/kardianos/govendor`
|
||||
|
||||
New users please read the [FAQ](doc/faq.md)
|
||||
|
||||
Package developers should read the [developer guide](doc/dev-guide.md).
|
||||
|
||||
For a high level overview read the [whitepaper](doc/whitepaper.md)
|
||||
|
||||
Uses the go1.5+ vendor folder. Multiple workflows supported, single tool.
|
||||
|
||||
[![Build Status](https://travis-ci.org/kardianos/govendor.svg?branch=master)](https://travis-ci.org/kardianos/govendor)
|
||||
[![GoDoc](https://godoc.org/github.com/kardianos/govendor?status.svg)](https://godoc.org/github.com/kardianos/govendor)
|
||||
|
||||
* Copy existing dependencies from $GOPATH with `govendor add/update`.
|
||||
* If you ignore `vendor/*/`, restore dependencies with `govendor sync`.
|
||||
* Pull in new dependencies or update existing dependencies directly from
|
||||
remotes with `govendor fetch`.
|
||||
* Migrate from legacy systems with `govendor migrate`.
|
||||
* Supports Linux, OS X, Windows, probably all others.
|
||||
* Supports git, hg, svn, bzr (must be installed an on the PATH).
|
||||
|
||||
## Notes
|
||||
|
||||
* The project must be within a $GOPATH.
|
||||
* If using go1.5, ensure you `set GO15VENDOREXPERIMENT=1`.
|
||||
|
||||
### Quick Start, also see the [FAQ](doc/faq.md)
|
||||
```
|
||||
# Setup your project.
|
||||
cd "my project in GOPATH"
|
||||
govendor init
|
||||
|
||||
# Add existing GOPATH files to vendor.
|
||||
govendor add +external
|
||||
|
||||
# View your work.
|
||||
govendor list
|
||||
|
||||
# Look at what is using a package
|
||||
govendor list -v fmt
|
||||
|
||||
# Specify a specific version or revision to fetch
|
||||
govendor fetch golang.org/x/net/context@a4bbce9fcae005b22ae5443f6af064d80a6f5a55
|
||||
govendor fetch golang.org/x/net/context@v1 # Get latest v1.*.* tag or branch.
|
||||
govendor fetch golang.org/x/net/context@=v1 # Get the tag or branch named "v1".
|
||||
|
||||
# Update a package to latest, given any prior version constraint
|
||||
govendor fetch golang.org/x/net/context
|
||||
|
||||
# Format your repository only
|
||||
govendor fmt +local
|
||||
|
||||
# Build everything in your repository only
|
||||
govendor install +local
|
||||
|
||||
# Test your repository only
|
||||
govendor test +local
|
||||
|
||||
```
|
||||
|
||||
## Sub-commands
|
||||
```
|
||||
init Create the "vendor" folder and the "vendor.json" file.
|
||||
list List and filter existing dependencies and packages.
|
||||
add Add packages from $GOPATH.
|
||||
update Update packages from $GOPATH.
|
||||
remove Remove packages from the vendor folder.
|
||||
status Lists any packages missing, out-of-date, or modified locally.
|
||||
fetch Add new or update vendor folder packages from remote repository.
|
||||
sync Pull packages into vendor folder from remote repository with revisions
|
||||
from vendor.json file.
|
||||
migrate Move packages from a legacy tool to the vendor folder with metadata.
|
||||
get Like "go get" but copies dependencies into a "vendor" folder.
|
||||
license List discovered licenses for the given status or import paths.
|
||||
shell Run a "shell" to make multiple sub-commands more efficient for large
|
||||
projects.
|
||||
|
||||
go tool commands that are wrapped:
|
||||
`+<status>` package selection may be used with them
|
||||
fmt, build, install, clean, test, vet, generate
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
Packages can be specified by their "status".
|
||||
```
|
||||
+local (l) packages in your project
|
||||
+external (e) referenced packages in GOPATH but not in current project
|
||||
+vendor (v) packages in the vendor folder
|
||||
+std (s) packages in the standard library
|
||||
|
||||
+excluded (x) external packages explicitely excluded from vendoring
|
||||
+unused (u) packages in the vendor folder, but unused
|
||||
+missing (m) referenced packages but not found
|
||||
|
||||
+program (p) package is a main package
|
||||
|
||||
+outside +external +missing
|
||||
+all +all packages
|
||||
```
|
||||
|
||||
Status can be referenced by their initial letters.
|
||||
|
||||
* `+std` same as `+s`
|
||||
* `+external` same as `+ext` same as `+e`
|
||||
* `+excluded` same as `+exc` same as `+x`
|
||||
|
||||
Status can be logically composed:
|
||||
|
||||
* `+local,program` (local AND program) local packages that are also programs
|
||||
* `+local +vendor` (local OR vendor) local packages or vendor packages
|
||||
* `+vendor,program +std` ((vendor AND program) OR std) vendor packages that are also programs
|
||||
or std library packages
|
||||
* `+vendor,^program` (vendor AND NOT program) vendor package that are not "main" packages.
|
||||
|
||||
## Package specifier
|
||||
|
||||
The full package-spec is:
|
||||
`<path>[::<origin>][{/...|/^}][@[<version-spec>]]`
|
||||
|
||||
Some examples:
|
||||
|
||||
* `github.com/kardianos/govendor` specifies a single package and single folder.
|
||||
* `github.com/kardianos/govendor/...` specifies `govendor` and all referenced
|
||||
packages under that path.
|
||||
* `github.com/kardianos/govendor/^` specifies the `govendor` folder and all
|
||||
sub-folders. Useful for resources or if you don't want a partial repository.
|
||||
* `github.com/kardianos/govendor/^::github.com/myself/govendor` same as above
|
||||
but fetch from user "myself".
|
||||
* `github.com/kardianos/govendor/...@abc12032` all referenced packages at
|
||||
revision `abc12032`.
|
||||
* `github.com/kardianos/govendor/...@v1` same as above, but get the most recent
|
||||
"v1" tag, such as "v1.4.3".
|
||||
* `github.com/kardianos/govendor/...@=v1` get the exact version "v1".
|
||||
|
||||
## Packages and Status
|
||||
|
||||
You may specify multiple package-specs and multiple status in a single command.
|
||||
Commands that accept status and package-spec:
|
||||
|
||||
* list
|
||||
* add
|
||||
* update
|
||||
* remove
|
||||
* fetch
|
||||
|
||||
You may pass arguments to govendor through stdin if the last argument is a "-".
|
||||
For example `echo +vendor | govendor list -` will list all vendor packages.
|
||||
|
||||
## Ignoring build tags and excluding packages
|
||||
Ignoring build tags is opt-out and is designed to be the opposite of the build
|
||||
file directives which are opt-in when specified. Typically a developer will
|
||||
want to support cross platform builds, but selectively opt out of tags, tests,
|
||||
and architectures as desired.
|
||||
|
||||
To ignore additional tags edit the "vendor.json" file and add tag to the vendor
|
||||
"ignore" file field. The field uses spaces to separate tags to ignore.
|
||||
For example the following will ignore both test and appengine files.
|
||||
```
|
||||
{
|
||||
"ignore": "test appengine",
|
||||
}
|
||||
```
|
||||
|
||||
Similarly, some specific packages can be excluded from the vendoring process.
|
||||
These packages will be listed as `excluded` (`x`), and will not be copied to the
|
||||
"vendor" folder when running `govendor add|fetch|update`.
|
||||
|
||||
Any sub-package `foo/bar` of an excluded package `foo` is also excluded (but
|
||||
package `bar/foo` is not). The import dependencies of excluded packages are not
|
||||
listed, and thus not vendored.
|
||||
|
||||
To exclude packages, also use the "ignore" field of the "vendor.json" file.
|
||||
Packages are identified by their name, they should contain a "/" character
|
||||
(possibly at the end):
|
||||
```
|
||||
{
|
||||
"ignore": "test appengine foo/",
|
||||
}
|
||||
```
|
|
@ -1,52 +0,0 @@
|
|||
// Copyright 2015 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// vendor tool to copy external source code from GOPATH or remote location to the
|
||||
// local vendor folder. See README.md for usage.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/kardianos/govendor/cliprompt"
|
||||
"github.com/kardianos/govendor/help"
|
||||
"github.com/kardianos/govendor/run"
|
||||
)
|
||||
|
||||
func main() {
|
||||
prompt := &cliprompt.Prompt{}
|
||||
|
||||
allArgs := os.Args
|
||||
|
||||
if allArgs[len(allArgs)-1] == "-" {
|
||||
stdin := &bytes.Buffer{}
|
||||
if _, err := io.Copy(stdin, os.Stdin); err == nil {
|
||||
stdinArgs := strings.Fields(stdin.String())
|
||||
allArgs = append(allArgs[:len(allArgs)-1], stdinArgs...)
|
||||
}
|
||||
}
|
||||
|
||||
msg, err := run.Run(os.Stdout, allArgs, prompt)
|
||||
if err == flag.ErrHelp {
|
||||
err = nil
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
|
||||
}
|
||||
msgText := msg.String()
|
||||
if len(msgText) > 0 {
|
||||
fmt.Fprint(os.Stderr, msgText)
|
||||
}
|
||||
if err != nil {
|
||||
os.Exit(2)
|
||||
}
|
||||
if msg != help.MsgNone {
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) [2016] [Seth Ammons]
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
|
@ -0,0 +1,126 @@
|
|||
# pester
|
||||
|
||||
`pester` wraps Go's standard lib http client to provide several options to increase resiliency in your request. If you experience poor network conditions or requests could experience varied delays, you can now pester the endpoint for data.
|
||||
- Send out multiple requests and get the first back (only used for GET calls)
|
||||
- Retry on errors
|
||||
- Backoff
|
||||
|
||||
### Simple Example
|
||||
Use `pester` where you would use the http client calls. By default, pester will use a concurrency of 1, and retry the endpoint 3 times with the `DefaultBackoff` strategy of waiting 1 second between retries.
|
||||
```go
|
||||
/* swap in replacement, just switch
|
||||
http.{Get|Post|PostForm|Head|Do} to
|
||||
pester.{Get|Post|PostForm|Head|Do}
|
||||
*/
|
||||
resp, err := pester.Get("http://sethammons.com")
|
||||
```
|
||||
|
||||
### Backoff Strategy
|
||||
Provide your own backoff strategy, or use one of the provided built in strategies:
|
||||
- `DefaultBackoff`: 1 second
|
||||
- `LinearBackoff`: n seconds where n is the retry number
|
||||
- `LinearJitterBackoff`: n seconds where n is the retry number, +/- 0-33%
|
||||
- `ExponentialBackoff`: n seconds where n is 2^(retry number)
|
||||
- `ExponentialJitterBackoff`: n seconds where n is 2^(retry number), +/- 0-33%
|
||||
|
||||
```go
|
||||
client := pester.New()
|
||||
client.Backoff = func(retry int) time.Duration {
|
||||
// set up something dynamic or use a look up table
|
||||
return time.Duration(retry) * time.Minute
|
||||
}
|
||||
```
|
||||
|
||||
### Complete example
|
||||
For a complete and working example, see the sample directory.
|
||||
`pester` allows you to use a constructor to control:
|
||||
- backoff strategy
|
||||
- reties
|
||||
- concurrency
|
||||
- keeping a log for debugging
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"log"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/sethgrid/pester"
|
||||
)
|
||||
|
||||
func main() {
|
||||
log.Println("Starting...")
|
||||
|
||||
{ // drop in replacement for http.Get and other client methods
|
||||
resp, err := pester.Get("http://example.com")
|
||||
if err != nil {
|
||||
log.Println("error GETing example.com", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
log.Printf("example.com %s", resp.Status)
|
||||
}
|
||||
|
||||
{ // control the resiliency
|
||||
client := pester.New()
|
||||
client.Concurrency = 3
|
||||
client.MaxRetries = 5
|
||||
client.Backoff = pester.ExponentialBackoff
|
||||
client.KeepLog = true
|
||||
|
||||
resp, err := client.Get("http://example.com")
|
||||
if err != nil {
|
||||
log.Println("error GETing example.com", client.LogString())
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
log.Printf("example.com %s", resp.Status)
|
||||
}
|
||||
|
||||
{ // use the pester version of http.Client.Do
|
||||
req, err := http.NewRequest("POST", "http://example.com", strings.NewReader("data"))
|
||||
if err != nil {
|
||||
log.Fatal("Unable to create a new http request", err)
|
||||
}
|
||||
resp, err := pester.Do(req)
|
||||
if err != nil {
|
||||
log.Println("error POSTing example.com", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
log.Printf("example.com %s", resp.Status)
|
||||
}
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
### Example Log
|
||||
`pester` also allows you to control the resiliency and can optionally log the errors.
|
||||
```go
|
||||
c := pester.New()
|
||||
c.KeepLog = true
|
||||
|
||||
nonExistantURL := "http://localhost:9000/foo"
|
||||
_, _ = c.Get(nonExistantURL)
|
||||
|
||||
fmt.Println(c.LogString())
|
||||
/*
|
||||
Output:
|
||||
|
||||
1432402837 Get [GET] http://localhost:9000/foo request-0 retry-0 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
|
||||
1432402838 Get [GET] http://localhost:9000/foo request-0 retry-1 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
|
||||
1432402839 Get [GET] http://localhost:9000/foo request-0 retry-2 error: Get http://localhost:9000/foo: dial tcp 127.0.0.1:9000: connection refused
|
||||
*/
|
||||
```
|
||||
|
||||
### Tests
|
||||
|
||||
You can run tests in the root directory with `$ go test`. There is a benchmark-like test available with `$ cd benchmarks; go test`.
|
||||
You can see `pester` in action with `$ cd sample; go run main.go`.
|
||||
|
||||
For watching open file descriptors, you can run `watch "lsof -i -P | grep main"` if you started the app with `go run main.go`.
|
||||
I did this for watching for FD leaks. My method was to alter `sample/main.go` to only run one case (`pester.Get with set backoff stategy, concurrency and retries increased`)
|
||||
and adding a sleep after the result came back. This let me verify if FDs were getting left open when they should have closed. If you know a better way, let me know!
|
||||
I was able to see that FDs are now closing when they should :)
|
||||
|
||||
![Are we there yet?](http://butchbellah.com/wp-content/uploads/2012/06/Are-We-There-Yet.jpg)
|
||||
|
||||
Are we there yet? Are we there yet? Are we there yet? Are we there yet? ...
|
|
@ -0,0 +1,423 @@
|
|||
package pester
|
||||
|
||||
// pester provides additional resiliency over the standard http client methods by
|
||||
// allowing you to control concurrency, retries, and a backoff strategy.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"math"
|
||||
"math/rand"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Client wraps the http client and exposes all the functionality of the http.Client.
|
||||
// Additionally, Client provides pester specific values for handling resiliency.
|
||||
type Client struct {
|
||||
// wrap it to provide access to http built ins
|
||||
hc *http.Client
|
||||
|
||||
Transport http.RoundTripper
|
||||
CheckRedirect func(req *http.Request, via []*http.Request) error
|
||||
Jar http.CookieJar
|
||||
Timeout time.Duration
|
||||
|
||||
// pester specific
|
||||
Concurrency int
|
||||
MaxRetries int
|
||||
Backoff BackoffStrategy
|
||||
KeepLog bool
|
||||
|
||||
SuccessReqNum int
|
||||
SuccessRetryNum int
|
||||
|
||||
wg *sync.WaitGroup
|
||||
|
||||
sync.Mutex
|
||||
ErrLog []ErrEntry
|
||||
}
|
||||
|
||||
// ErrEntry is used to provide the LogString() data and is populated
|
||||
// each time an error happens if KeepLog is set.
|
||||
// ErrEntry.Retry is deprecated in favor of ErrEntry.Attempt
|
||||
type ErrEntry struct {
|
||||
Time time.Time
|
||||
Method string
|
||||
URL string
|
||||
Verb string
|
||||
Request int
|
||||
Retry int
|
||||
Attempt int
|
||||
Err error
|
||||
}
|
||||
|
||||
// result simplifies the channel communication for concurrent request handling
|
||||
type result struct {
|
||||
resp *http.Response
|
||||
err error
|
||||
req int
|
||||
retry int
|
||||
}
|
||||
|
||||
// params represents all the params needed to run http client calls and pester errors
|
||||
type params struct {
|
||||
method string
|
||||
verb string
|
||||
req *http.Request
|
||||
url string
|
||||
bodyType string
|
||||
body io.Reader
|
||||
data url.Values
|
||||
}
|
||||
|
||||
// New constructs a new DefaultClient with sensible default values
|
||||
func New() *Client {
|
||||
return &Client{
|
||||
Concurrency: DefaultClient.Concurrency,
|
||||
MaxRetries: DefaultClient.MaxRetries,
|
||||
Backoff: DefaultClient.Backoff,
|
||||
ErrLog: DefaultClient.ErrLog,
|
||||
wg: &sync.WaitGroup{},
|
||||
}
|
||||
}
|
||||
|
||||
// NewExtendedClient allows you to pass in an http.Client that is previously set up
|
||||
// and extends it to have Pester's features of concurrency and retries.
|
||||
func NewExtendedClient(hc *http.Client) *Client {
|
||||
c := New()
|
||||
c.hc = hc
|
||||
return c
|
||||
}
|
||||
|
||||
// BackoffStrategy is used to determine how long a retry request should wait until attempted
|
||||
type BackoffStrategy func(retry int) time.Duration
|
||||
|
||||
// DefaultClient provides sensible defaults
|
||||
var DefaultClient = &Client{Concurrency: 1, MaxRetries: 3, Backoff: DefaultBackoff, ErrLog: []ErrEntry{}}
|
||||
|
||||
// DefaultBackoff always returns 1 second
|
||||
func DefaultBackoff(_ int) time.Duration {
|
||||
return 1 * time.Second
|
||||
}
|
||||
|
||||
// ExponentialBackoff returns ever increasing backoffs by a power of 2
|
||||
func ExponentialBackoff(i int) time.Duration {
|
||||
return time.Duration(math.Pow(2, float64(i))) * time.Second
|
||||
}
|
||||
|
||||
// ExponentialJitterBackoff returns ever increasing backoffs by a power of 2
|
||||
// with +/- 0-33% to prevent sychronized reuqests.
|
||||
func ExponentialJitterBackoff(i int) time.Duration {
|
||||
return jitter(int(math.Pow(2, float64(i))))
|
||||
}
|
||||
|
||||
// LinearBackoff returns increasing durations, each a second longer than the last
|
||||
func LinearBackoff(i int) time.Duration {
|
||||
return time.Duration(i) * time.Second
|
||||
}
|
||||
|
||||
// LinearJitterBackoff returns increasing durations, each a second longer than the last
|
||||
// with +/- 0-33% to prevent sychronized reuqests.
|
||||
func LinearJitterBackoff(i int) time.Duration {
|
||||
return jitter(i)
|
||||
}
|
||||
|
||||
// jitter keeps the +/- 0-33% logic in one place
|
||||
func jitter(i int) time.Duration {
|
||||
ms := i * 1000
|
||||
|
||||
maxJitter := ms / 3
|
||||
|
||||
rand.Seed(time.Now().Unix())
|
||||
jitter := rand.Intn(maxJitter + 1)
|
||||
|
||||
if rand.Intn(2) == 1 {
|
||||
ms = ms + jitter
|
||||
} else {
|
||||
ms = ms - jitter
|
||||
}
|
||||
|
||||
// a jitter of 0 messes up the time.Tick chan
|
||||
if ms <= 0 {
|
||||
ms = 1
|
||||
}
|
||||
|
||||
return time.Duration(ms) * time.Millisecond
|
||||
}
|
||||
|
||||
// Wait blocks until all pester requests have returned
|
||||
// Probably not that useful outside of testing.
|
||||
func (c *Client) Wait() {
|
||||
c.wg.Wait()
|
||||
}
|
||||
|
||||
// pester provides all the logic of retries, concurrency, backoff, and logging
|
||||
func (c *Client) pester(p params) (*http.Response, error) {
|
||||
resultCh := make(chan result)
|
||||
multiplexCh := make(chan result)
|
||||
finishCh := make(chan struct{})
|
||||
|
||||
// track all requests that go out so we can close the late listener routine that closes late incoming response bodies
|
||||
totalSentRequests := &sync.WaitGroup{}
|
||||
totalSentRequests.Add(1)
|
||||
defer totalSentRequests.Done()
|
||||
allRequestsBackCh := make(chan struct{})
|
||||
go func() {
|
||||
totalSentRequests.Wait()
|
||||
close(allRequestsBackCh)
|
||||
}()
|
||||
|
||||
// GET calls should be idempotent and can make use
|
||||
// of concurrency. Other verbs can mutate and should not
|
||||
// make use of the concurrency feature
|
||||
concurrency := c.Concurrency
|
||||
if p.verb != "GET" {
|
||||
concurrency = 1
|
||||
}
|
||||
|
||||
c.Lock()
|
||||
if c.hc == nil {
|
||||
c.hc = &http.Client{}
|
||||
c.hc.Transport = c.Transport
|
||||
c.hc.CheckRedirect = c.CheckRedirect
|
||||
c.hc.Jar = c.Jar
|
||||
c.hc.Timeout = c.Timeout
|
||||
}
|
||||
c.Unlock()
|
||||
|
||||
// re-create the http client so we can leverage the std lib
|
||||
httpClient := http.Client{
|
||||
Transport: c.hc.Transport,
|
||||
CheckRedirect: c.hc.CheckRedirect,
|
||||
Jar: c.hc.Jar,
|
||||
Timeout: c.hc.Timeout,
|
||||
}
|
||||
|
||||
// if we have a request body, we need to save it for later
|
||||
var originalRequestBody []byte
|
||||
var originalBody []byte
|
||||
var err error
|
||||
if p.req != nil && p.req.Body != nil {
|
||||
originalRequestBody, err = ioutil.ReadAll(p.req.Body)
|
||||
if err != nil {
|
||||
return &http.Response{}, errors.New("error reading request body")
|
||||
}
|
||||
p.req.Body.Close()
|
||||
}
|
||||
if p.body != nil {
|
||||
originalBody, err = ioutil.ReadAll(p.body)
|
||||
if err != nil {
|
||||
return &http.Response{}, errors.New("error reading body")
|
||||
}
|
||||
}
|
||||
|
||||
AttemptLimit := c.MaxRetries
|
||||
if AttemptLimit <= 0 {
|
||||
AttemptLimit = 1
|
||||
}
|
||||
|
||||
for req := 0; req < concurrency; req++ {
|
||||
c.wg.Add(1)
|
||||
totalSentRequests.Add(1)
|
||||
go func(n int, p params) {
|
||||
defer c.wg.Done()
|
||||
defer totalSentRequests.Done()
|
||||
|
||||
var err error
|
||||
for i := 1; i <= AttemptLimit; i++ {
|
||||
c.wg.Add(1)
|
||||
defer c.wg.Done()
|
||||
select {
|
||||
case <-finishCh:
|
||||
return
|
||||
default:
|
||||
}
|
||||
resp := &http.Response{}
|
||||
|
||||
// rehydrate the body (it is drained each read)
|
||||
if len(originalRequestBody) > 0 {
|
||||
p.req.Body = ioutil.NopCloser(bytes.NewBuffer(originalRequestBody))
|
||||
}
|
||||
if len(originalBody) > 0 {
|
||||
p.body = bytes.NewBuffer(originalBody)
|
||||
}
|
||||
|
||||
// route the calls
|
||||
switch p.method {
|
||||
case "Do":
|
||||
resp, err = httpClient.Do(p.req)
|
||||
case "Get":
|
||||
resp, err = httpClient.Get(p.url)
|
||||
case "Head":
|
||||
resp, err = httpClient.Head(p.url)
|
||||
case "Post":
|
||||
resp, err = httpClient.Post(p.url, p.bodyType, p.body)
|
||||
case "PostForm":
|
||||
resp, err = httpClient.PostForm(p.url, p.data)
|
||||
}
|
||||
|
||||
// Early return if we have a valid result
|
||||
// Only retry (ie, continue the loop) on 5xx status codes
|
||||
if err == nil && resp.StatusCode < 500 {
|
||||
multiplexCh <- result{resp: resp, err: err, req: n, retry: i}
|
||||
return
|
||||
}
|
||||
|
||||
c.log(ErrEntry{
|
||||
Time: time.Now(),
|
||||
Method: p.method,
|
||||
Verb: p.verb,
|
||||
URL: p.url,
|
||||
Request: n,
|
||||
Retry: i + 1, // would remove, but would break backward compatibility
|
||||
Attempt: i,
|
||||
Err: err,
|
||||
})
|
||||
|
||||
// if it is the last iteration, grab the result (which is an error at this point)
|
||||
if i == AttemptLimit {
|
||||
multiplexCh <- result{resp: resp, err: err}
|
||||
return
|
||||
}
|
||||
|
||||
// if we are retrying, we should close this response body to free the fd
|
||||
if resp != nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
|
||||
// prevent a 0 from causing the tick to block, pass additional microsecond
|
||||
<-time.Tick(c.Backoff(i) + 1*time.Microsecond)
|
||||
}
|
||||
}(req, p)
|
||||
}
|
||||
|
||||
// spin off the go routine so it can continually listen in on late results and close the response bodies
|
||||
go func() {
|
||||
gotFirstResult := false
|
||||
for {
|
||||
select {
|
||||
case res := <-multiplexCh:
|
||||
if !gotFirstResult {
|
||||
gotFirstResult = true
|
||||
close(finishCh)
|
||||
resultCh <- res
|
||||
} else if res.resp != nil {
|
||||
// we only return one result to the caller; close all other response bodies that come back
|
||||
// drain the body before close as to not prevent keepalive. see https://gist.github.com/mholt/eba0f2cc96658be0f717
|
||||
io.Copy(ioutil.Discard, res.resp.Body)
|
||||
res.resp.Body.Close()
|
||||
}
|
||||
case <-allRequestsBackCh:
|
||||
// don't leave this goroutine running
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
select {
|
||||
case res := <-resultCh:
|
||||
c.Lock()
|
||||
defer c.Unlock()
|
||||
c.SuccessReqNum = res.req
|
||||
c.SuccessRetryNum = res.retry
|
||||
return res.resp, res.err
|
||||
}
|
||||
}
|
||||
|
||||
// LogString provides a string representation of the errors the client has seen
|
||||
func (c *Client) LogString() string {
|
||||
c.Lock()
|
||||
defer c.Unlock()
|
||||
var res string
|
||||
for _, e := range c.ErrLog {
|
||||
res += fmt.Sprintf("%d %s [%s] %s request-%d retry-%d error: %s\n",
|
||||
e.Time.Unix(), e.Method, e.Verb, e.URL, e.Request, e.Retry, e.Err)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// LogErrCount is a helper method used primarily for test validation
|
||||
func (c *Client) LogErrCount() int {
|
||||
c.Lock()
|
||||
defer c.Unlock()
|
||||
return len(c.ErrLog)
|
||||
}
|
||||
|
||||
// EmbedHTTPClient allows you to extend an existing Pester client with an
|
||||
// underlying http.Client, such as https://godoc.org/golang.org/x/oauth2/google#DefaultClient
|
||||
func (c *Client) EmbedHTTPClient(hc *http.Client) {
|
||||
c.hc = hc
|
||||
}
|
||||
|
||||
func (c *Client) log(e ErrEntry) {
|
||||
if c.KeepLog {
|
||||
c.Lock()
|
||||
c.ErrLog = append(c.ErrLog, e)
|
||||
c.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// Do provides the same functionality as http.Client.Do
|
||||
func (c *Client) Do(req *http.Request) (resp *http.Response, err error) {
|
||||
return c.pester(params{method: "Do", req: req, verb: req.Method, url: req.URL.String()})
|
||||
}
|
||||
|
||||
// Get provides the same functionality as http.Client.Get
|
||||
func (c *Client) Get(url string) (resp *http.Response, err error) {
|
||||
return c.pester(params{method: "Get", url: url, verb: "GET"})
|
||||
}
|
||||
|
||||
// Head provides the same functionality as http.Client.Head
|
||||
func (c *Client) Head(url string) (resp *http.Response, err error) {
|
||||
return c.pester(params{method: "Head", url: url, verb: "HEAD"})
|
||||
}
|
||||
|
||||
// Post provides the same functionality as http.Client.Post
|
||||
func (c *Client) Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error) {
|
||||
return c.pester(params{method: "Post", url: url, bodyType: bodyType, body: body, verb: "POST"})
|
||||
}
|
||||
|
||||
// PostForm provides the same functionality as http.Client.PostForm
|
||||
func (c *Client) PostForm(url string, data url.Values) (resp *http.Response, err error) {
|
||||
return c.pester(params{method: "PostForm", url: url, data: data, verb: "POST"})
|
||||
}
|
||||
|
||||
////////////////////////////////////////
|
||||
// Provide self-constructing variants //
|
||||
////////////////////////////////////////
|
||||
|
||||
// Do provides the same functionality as http.Client.Do and creates its own constructor
|
||||
func Do(req *http.Request) (resp *http.Response, err error) {
|
||||
c := New()
|
||||
return c.Do(req)
|
||||
}
|
||||
|
||||
// Get provides the same functionality as http.Client.Get and creates its own constructor
|
||||
func Get(url string) (resp *http.Response, err error) {
|
||||
c := New()
|
||||
return c.Get(url)
|
||||
}
|
||||
|
||||
// Head provides the same functionality as http.Client.Head and creates its own constructor
|
||||
func Head(url string) (resp *http.Response, err error) {
|
||||
c := New()
|
||||
return c.Head(url)
|
||||
}
|
||||
|
||||
// Post provides the same functionality as http.Client.Post and creates its own constructor
|
||||
func Post(url string, bodyType string, body io.Reader) (resp *http.Response, err error) {
|
||||
c := New()
|
||||
return c.Post(url, bodyType, body)
|
||||
}
|
||||
|
||||
// PostForm provides the same functionality as http.Client.PostForm and creates its own constructor
|
||||
func PostForm(url string, data url.Values) (resp *http.Response, err error) {
|
||||
c := New()
|
||||
return c.PostForm(url, data)
|
||||
}
|
|
@ -574,12 +574,6 @@
|
|||
"revision": "0b12d6b521d83fc7f755e7cfc1b1fbdd35a01a74",
|
||||
"revisionTime": "2016-02-02T18:50:14Z"
|
||||
},
|
||||
{
|
||||
"checksumSHA1": "bQfc4zGh8WUri465COdQNhJJTw4=",
|
||||
"path": "github.com/kardianos/govendor",
|
||||
"revision": "7a0d30eab9e67b3ff60f13cdc483f003c01e04c1",
|
||||
"revisionTime": "2016-07-09T17:43:04Z"
|
||||
},
|
||||
{
|
||||
"checksumSHA1": "QK3MNUdQwUBuznCHZcPijU/3DyI=",
|
||||
"path": "github.com/lib/pq",
|
||||
|
@ -670,6 +664,12 @@
|
|||
"revision": "e64db453f3512cade908163702045e0f31137843",
|
||||
"revisionTime": "2016-06-16T02:49:54Z"
|
||||
},
|
||||
{
|
||||
"checksumSHA1": "8Lm8nsMCFz4+gr9EvQLqK8+w+Ks=",
|
||||
"path": "github.com/sethgrid/pester",
|
||||
"revision": "8053687f99650573b28fb75cddf3f295082704d7",
|
||||
"revisionTime": "2016-04-29T17:20:22Z"
|
||||
},
|
||||
{
|
||||
"checksumSHA1": "UADS3X1kxl+/qqeGcmTo0rBiXlQ=",
|
||||
"path": "github.com/ugorji/go/codec",
|
||||
|
|
Loading…
Reference in New Issue