Commit Graph

142 Commits

Author SHA1 Message Date
Snappy Team 1ce58af28e Fix the use of op + len when op is nullptr and len is non-zero.
See https://reviews.llvm.org/D67122 for some discussion of why this can matter.
I don't think this should have any noticeable effect on performance.

PiperOrigin-RevId: 340255083
2020-11-03 20:30:24 +00:00
Luca Versari 0b990db2b8 Run clang-format
PiperOrigin-RevId: 339897712
2020-11-03 20:30:11 +00:00
Snappy Team 4dd277fed4 Replace the division with a constant table in IncrementalCopy
PiperOrigin-RevId: 320686580
2020-07-11 01:54:52 +00:00
Snappy Team f16eda3466 Correct uninitialized variable.
PiperOrigin-RevId: 312741918
2020-06-01 23:46:44 +00:00
Victor Costan c98344f626 Fix Clang/GCC compilation warnings.
This makes it easier to adopt snappy in other projects.

PiperOrigin-RevId: 309958249
2020-05-05 16:15:02 +00:00
Victor Costan 113cd97ab3 Tighten types on a few for loops.
* Replace post-increment with pre-increment in for loops.
* Replace unsigned int counters with precise types, like uint8_t.
* Switch to C++11 iterating loops when possible.

PiperOrigin-RevId: 309724233
2020-05-04 12:32:00 +00:00
Victor Costan 5417da69b7 Switch from C headers to C++ headers.
This CL makes the following substitutions.

* assert.h -> cassert
* math.h -> cmath
* stdarg.h -> cstdarg
* stdio.h -> cstdio
* stdlib.h -> cstdlib
* string.h -> cstring

stddef.h and stdint.h are not migrated to C++ headers.

PiperOrigin-RevId: 309074805
2020-04-29 19:38:03 +00:00
Victor Costan a4cdb5d133 Introduce SNAPPY_ATTRIBUTE_ALWAYS_INLINE.
An internal CL started using ABSL_ATTRIBUTE_ALWAYS_INLINE
from Abseil. This CL introduces equivalent functionality as
SNAPPY_ALWAYS_INLINE.

PiperOrigin-RevId: 306289650
2020-04-13 19:51:05 +00:00
Victor Costan 231b8be076 Migrate to standard integral types.
The following changes are done via find/replace.
* int8 -> int8_t
* int16 -> int16_t
* int32 -> int32_t
* int64 -> int64_t

The aliases were removed from snappy-stubs-public.h.

PiperOrigin-RevId: 306141557
2020-04-12 20:10:03 +00:00
Victor Costan 14bef66290 Modernize memcpy() and memmove() usage.
This CL replaces memcpy() with std::memcpy()
and memmove() with std::memmove(), and #includes
<cstring> in files that use either function.

PiperOrigin-RevId: 306067788
2020-04-12 00:06:15 +00:00
Snappy Team d674348a0c Improve zippy with 5-10%.
BM_ZCord/0        [html   ]            1.26GB/s ± 0%           1.35GB/s ± 0%   +7.90%          (p=0.008 n=5+5)
BM_ZCord/1        [urls   ]             535MB/s ± 0%            562MB/s ± 0%   +5.05%          (p=0.008 n=5+5)
BM_ZCord/2        [jpg    ]            10.2GB/s ± 1%           10.2GB/s ± 0%     ~             (p=0.310 n=5+5)
BM_ZCord/3        [jpg_200]             841MB/s ± 1%            846MB/s ± 1%     ~             (p=0.421 n=5+5)
BM_ZCord/4        [pdf    ]            6.77GB/s ± 1%           7.06GB/s ± 1%   +4.28%          (p=0.008 n=5+5)
BM_ZCord/5        [html4  ]            1.00GB/s ± 0%           1.08GB/s ± 0%   +7.94%          (p=0.008 n=5+5)
BM_ZCord/6        [txt1   ]             391MB/s ± 0%            417MB/s ± 0%   +6.71%          (p=0.008 n=5+5)
BM_ZCord/7        [txt2   ]             363MB/s ± 0%            388MB/s ± 0%   +6.73%          (p=0.016 n=5+4)
BM_ZCord/8        [txt3   ]             400MB/s ± 0%            426MB/s ± 0%   +6.55%          (p=0.008 n=5+5)
BM_ZCord/9        [txt4   ]             328MB/s ± 0%            350MB/s ± 0%   +6.66%          (p=0.008 n=5+5)
BM_ZCord/10       [pb     ]            1.67GB/s ± 1%           1.80GB/s ± 0%   +7.52%          (p=0.008 n=5+5)

1) A key bottleneck in the data dependency chain is figuring out how many bytes are matched and loading the data for next hash value. The load-to-use latency is 5 cycles, in previous cl/303353110 we removed the load in lieu of "shrd" to align previous loads. Unfortunately "shrd" itself has a latency of 4 cycles, we'd prefer "shrx" which takes 1 cycle for variable shifts.
2)Maximally use data already computed. The above trick calculates 5 bytes of useful data. So in case we need to search for new match we can use this for the first search (which is one byte further).

PiperOrigin-RevId: 303875535
2020-04-11 04:41:15 +00:00
Snappy Team 4dfcad9f4e assertion failure on darwin_x86_64, have to investigage
PiperOrigin-RevId: 303428229
2020-04-11 04:41:07 +00:00
Snappy Team e19178748f assertion failure on darwin_x86_64, have to investigage
PiperOrigin-RevId: 303346402
2020-04-11 04:40:57 +00:00
Snappy Team 0faf56378e This cl does two things
1) It shaves of a few cycles from the data dependency chain. By using "shrd" instead of a load.
2) The important loop is finding small copies (4-12) which are either "copy 1", or "copy 2" depending if the offset fits <2048. It turns out that this is a branch that is mispredicted often. Due to the long dependency chain the CPU is running with IPC~1 anyway so we can freely add instructions to instead emit copies branchfree. This reduces the branch misspredicts from 15% to 11% (for BM_ZFlat/6 txt1) and from 5.6% to 4% (for BM_ZFlat/10 or pb).

PiperOrigin-RevId: 303328967
2020-04-11 04:40:48 +00:00
Snappy Team 0c7ed08a25 The result on protobuf benchmark is around 19%. Results vary by their propensity for compression. As the frequency of finding matches influences the amount of branch misspredicts and the amount of hashing.
Two ideas
1) The code uses "heuristic match skipping" has a quadratic interpolation. However for the first 32 bytes it's just every byte. Special case 16 bytes. This removes a lot of code.
2) Load 64 bit integers and shift instead of reload. The hashing loop has a very long chain data = Load32(ip) -> hash = Hash(data) -> offset = table[hash] -> copy_data = Load32(base_ip + offset) followed by a compare between data and copy_data. This chain is around 20 cycles. It's unreasonable for the branch predictor to be able to predict when it's a match (that is completely driven by the content of the data). So when it's a miss this chain is on the critical path. By loading 64 bits and shifting we can effectively remove the first load.

PiperOrigin-RevId: 302893821
2020-04-11 04:40:39 +00:00
Snappy Team 3c77e01459 1) Make the output pointer a local variable such it doesn't need a load add store on it's loop carried dependency chain.
2) Reduce the input pointer loop carried dependency chain from 7 cycles to 4 cycles by using pre-loading. This is a very subtle point.
3) Just brutally copy 64 bytes which removes a difficult to predict branch from the inner most loop. There is enough bandwidth to do so in the intrinsic cycles of the loop.
4) Implement limit pointers that include the slop region. This removes unnecessary instructions from the hot path.
5) It seems the removal of the difficult to predict branch has removed the code sensitivity to alignment, so remove the asm nop's.

PiperOrigin-RevId: 294692928
2020-04-11 04:40:29 +00:00
Snappy Team 9eabb7baba Cut a load from the critical dependency chain of the input pointer by speculating the uncommon case of COPY_4 is not happening.
PiperOrigin-RevId: 293803653
2020-04-11 04:40:20 +00:00
Snappy Team cddd9c0875 Improve comments in IncrementalCopy, add an assert.
PiperOrigin-RevId: 292506754
2020-04-11 04:40:09 +00:00
Snappy Team b5477a8457 Optimize IncrementalCopy: There are between 1 and 4 copy iterations. Allow FDO to work with full knowledge of the probabilities for each branch.
On skylake, this improves protobuf and html decompression speed by 15% and 9% respectively, and the rest by ~2%.
On haswell, this improves protobuf and html decompression speed by 23% and 16% respectively, and the rest by ~3%.

PiperOrigin-RevId: 289090401
2020-01-14 10:58:42 -08:00
Victor Costan 62363d9a79 Fully qualify std::string.
This is in preparation for removing the snappy::string alias
of std::string.

PiperOrigin-RevId: 271383199
2019-09-26 10:57:29 -07:00
Shahriar Rouf 4c7f2d5dfb Add BM_ZFlatAll, BM_ZFlatIncreasingTableSize benchmarks to see how good zippy performs when it is processing different data one after the other.
PiperOrigin-RevId: 257518137
2019-08-19 14:30:00 -07:00
Chris Mumford c76b053449 Sync TODO and comment processing with external repo.
Copybara transforms code slightly different than MOE. One
example is the TODO username stripping where Copybara
produces different results than MOE did. This change
moves the Copybara versions of comments to the public
repository.

Note: These changes didn't originate in cl/247950252.

PiperOrigin-RevId: 247950252
2019-05-14 11:02:57 -07:00
atdt 02cf187555 Remove MSan exemption for _bzhi_u32, since LLVM now handles it correctly.
This cleans up a TODO from cl/225463783 and cl/225655713.

PiperOrigin-RevId: 241933185
2019-05-13 10:11:12 -07:00
nafi c197d686a9 Optimize snappy compression by about 2.2%.
'jpg_200' is notably optimized by ~8%.

name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            41.8µs ± 0%             41.9µs ± 0%  +0.33%          (p=0.016 n=5+5)
BM_UFlat/1      [urls             ]             590µs ± 0%              590µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/2      [jpg              ]            7.14µs ± 1%             7.12µs ± 1%    ~             (p=0.310 n=5+5)
BM_UFlat/3      [jpg_200          ]              129ns ± 0%              129ns ± 0%    ~             (p=0.167 n=5+5)
BM_UFlat/4      [pdf              ]            8.21µs ± 0%             8.20µs ± 0%    ~             (p=0.310 n=5+5)
BM_UFlat/5      [html4            ]             220µs ± 1%              220µs ± 0%    ~             (p=0.421 n=5+5)
BM_UFlat/6      [txt1             ]             193µs ± 0%              193µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/7      [txt2             ]             171µs ± 0%              171µs ± 0%    ~             (p=0.056 n=5+5)
BM_UFlat/8      [txt3             ]             512µs ± 0%              511µs ± 0%    ~             (p=0.310 n=5+5)
BM_UFlat/9      [txt4             ]             716µs ± 0%              716µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/10     [pb               ]            38.8µs ± 1%             38.8µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/11     [gaviota          ]             190µs ± 0%              190µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/12     [cp               ]            14.4µs ± 1%             14.4µs ± 1%    ~             (p=0.151 n=5+5)
BM_UFlat/13     [c                ]            7.33µs ± 0%             7.32µs ± 0%    ~             (p=0.690 n=5+5)
BM_UFlat/14     [lsp              ]            2.30µs ± 0%             2.31µs ± 1%    ~             (p=0.548 n=5+5)
BM_UFlat/15     [xls              ]             984µs ± 0%              984µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/16     [xls_200          ]              213ns ± 0%              213ns ± 0%    ~             (p=0.310 n=5+5)
BM_UFlat/17     [bin              ]             277µs ± 0%              278µs ± 0%    ~             (p=0.690 n=5+5)
BM_UFlat/18     [bin_200          ]              101ns ± 0%              102ns ± 0%    ~             (p=0.190 n=5+4)
BM_UFlat/19     [sum              ]            29.6µs ± 0%             29.6µs ± 0%    ~             (p=0.310 n=5+5)
BM_UFlat/20     [man              ]            2.98µs ± 1%             2.98µs ± 0%    ~             (p=1.000 n=5+5)
BM_UValidate/0  [html             ]            33.5µs ± 0%             33.6µs ± 0%    ~             (p=0.310 n=5+5)
BM_UValidate/1  [urls             ]             443µs ± 0%              443µs ± 0%    ~             (p=0.841 n=5+5)
BM_UValidate/2  [jpg              ]              146ns ± 0%              146ns ± 0%    ~             (p=0.222 n=5+5)
BM_UValidate/3  [jpg_200          ]             95.6ns ± 0%             95.5ns ± 0%    ~             (p=0.421 n=5+5)
BM_UValidate/4  [pdf              ]            2.92µs ± 0%             2.92µs ± 0%    ~             (p=0.841 n=5+5)
BM_UIOVec/0     [html             ]             122µs ± 0%              122µs ± 0%    ~             (p=0.548 n=5+5)
BM_UIOVec/1     [urls             ]             1.08ms ± 0%             1.08ms ± 0%    ~             (p=0.151 n=5+5)
BM_UIOVec/2     [jpg              ]            7.48µs ± 5%             7.75µs ±12%    ~             (p=0.690 n=5+5)
BM_UIOVec/3     [jpg_200          ]              331ns ± 1%              327ns ± 1%    ~             (p=0.056 n=5+5)
BM_UIOVec/4     [pdf              ]            12.0µs ± 0%             12.0µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/0  [html             ]            41.7µs ± 0%             41.8µs ± 0%    ~             (p=0.421 n=5+5)
BM_UFlatSink/1  [urls             ]             591µs ± 0%              590µs ± 0%    ~             (p=0.151 n=5+5)
BM_UFlatSink/2  [jpg              ]            7.18µs ± 2%             7.31µs ± 3%    ~             (p=0.190 n=4+5)
BM_UFlatSink/3  [jpg_200          ]              134ns ± 2%              134ns ± 2%    ~             (p=1.000 n=5+5)
BM_UFlatSink/4  [pdf              ]            8.22µs ± 0%             8.23µs ± 0%    ~             (p=0.730 n=4+5)
BM_UFlatSink/5  [html4            ]             219µs ± 0%              219µs ± 0%    ~             (p=0.548 n=5+5)
BM_UFlatSink/6  [txt1             ]             193µs ± 0%              193µs ± 0%    ~             (p=0.095 n=5+5)
BM_UFlatSink/7  [txt2             ]             171µs ± 0%              171µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlatSink/8  [txt3             ]             512µs ± 0%              512µs ± 0%    ~             (p=0.548 n=5+5)
BM_UFlatSink/9  [txt4             ]             718µs ± 0%              718µs ± 0%    ~             (p=0.548 n=5+5)
BM_UFlatSink/10 [pb               ]            38.7µs ± 0%             38.6µs ± 0%    ~             (p=0.222 n=5+5)
BM_UFlatSink/11 [gaviota          ]             191µs ± 0%              190µs ± 0%    ~             (p=0.690 n=5+5)
BM_UFlatSink/12 [cp               ]            14.3µs ± 0%             14.4µs ± 0%    ~             (p=0.222 n=5+5)
BM_UFlatSink/13 [c                ]            7.33µs ± 0%             7.34µs ± 1%    ~             (p=0.690 n=5+5)
BM_UFlatSink/14 [lsp              ]            2.29µs ± 1%             2.30µs ± 1%    ~             (p=0.095 n=5+5)
BM_UFlatSink/15 [xls              ]             981µs ± 0%              980µs ± 0%    ~             (p=0.310 n=5+5)
BM_UFlatSink/16 [xls_200          ]              216ns ± 1%              216ns ± 1%    ~             (p=1.000 n=5+5)
BM_UFlatSink/17 [bin              ]             277µs ± 0%              277µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/18 [bin_200          ]              104ns ± 0%              104ns ± 1%    ~             (p=0.905 n=5+4)
BM_UFlatSink/19 [sum              ]            29.5µs ± 0%             29.5µs ± 0%    ~             (p=0.222 n=5+5)
BM_UFlatSink/20 [man              ]            3.01µs ± 1%             3.01µs ± 0%    ~             (p=0.730 n=5+4)
BM_ZFlat/0      [html (22.31 %)   ]             126µs ± 0%              124µs ± 0%  -1.66%          (p=0.008 n=5+5)
BM_ZFlat/1      [urls (47.78 %)   ]             1.68ms ± 0%             1.63ms ± 0%  -2.73%          (p=0.008 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]            11.6µs ± 8%             11.4µs ± 6%    ~             (p=0.310 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]              369ns ± 1%              340ns ± 1%  -7.93%          (p=0.008 n=5+5)
BM_ZFlat/4      [pdf (83.30 %)    ]            14.9µs ± 4%             14.4µs ± 1%  -3.56%          (p=0.008 n=5+5)
BM_ZFlat/5      [html4 (22.52 %)  ]             551µs ± 0%              545µs ± 0%  -1.21%          (p=0.008 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]             540µs ± 0%              534µs ± 0%  -1.15%          (p=0.008 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]             480µs ± 0%              475µs ± 0%  -1.13%          (p=0.008 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]             1.44ms ± 0%             1.43ms ± 0%  -1.14%          (p=0.008 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]             1.97ms ± 0%             1.95ms ± 0%  -1.00%          (p=0.008 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]             110µs ± 0%              107µs ± 0%  -2.77%          (p=0.008 n=5+5)
BM_ZFlat/11     [gaviota (37.72 %)]             413µs ± 0%              411µs ± 0%  -0.50%          (p=0.008 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            46.6µs ± 1%             44.8µs ± 1%  -3.89%          (p=0.008 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            17.8µs ± 0%             17.5µs ± 0%  -1.87%          (p=0.008 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            5.62µs ± 1%             5.35µs ± 1%  -4.81%          (p=0.008 n=5+5)
BM_ZFlat/15     [xls (41.23 %)    ]             1.63ms ± 0%             1.63ms ± 0%    ~             (p=0.310 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]              393ns ± 1%              384ns ± 2%  -2.45%          (p=0.008 n=5+5)
BM_ZFlat/17     [bin (18.11 %)    ]             510µs ± 0%              503µs ± 0%  -1.50%          (p=0.016 n=4+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]             83.2ns ± 3%             84.5ns ± 4%    ~             (p=0.206 n=5+5)
BM_ZFlat/19     [sum (48.96 %)    ]            80.0µs ± 0%             78.3µs ± 0%  -2.20%          (p=0.008 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            7.79µs ± 1%             7.45µs ± 1%  -4.38%          (p=0.008 n=5+5)

name                                          old allocs/op           new allocs/op           delta
BM_UFlat/0      [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/1      [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/2      [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/3      [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/4      [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/5      [html4            ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/6      [txt1             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/7      [txt2             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/8      [txt3             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/9      [txt4             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/10     [pb               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/11     [gaviota          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/12     [cp               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/13     [c                ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/14     [lsp              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/15     [xls              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/16     [xls_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/17     [bin              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/18     [bin_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/19     [sum              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/20     [man              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/0  [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/1  [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/2  [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/4  [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/0     [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/1     [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/2     [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/3     [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/0  [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/1  [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/3  [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/5  [html4            ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/6  [txt1             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/7  [txt2             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/8  [txt3             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/9  [txt4             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/10 [pb               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/11 [gaviota          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/12 [cp               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/13 [c                ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/14 [lsp              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/15 [xls              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/16 [xls_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/17 [bin              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/18 [bin_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/19 [sum              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/20 [man              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_ZFlat/0      [html (22.31 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/1      [urls (47.78 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/2      [jpg (99.95 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/3      [jpg_200 (73.00 %)]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/4      [pdf (83.30 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/5      [html4 (22.52 %)  ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/6      [txt1 (57.88 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/7      [txt2 (61.91 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/8      [txt3 (54.99 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/9      [txt4 (66.26 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/10     [pb (19.68 %)     ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/11     [gaviota (37.72 %)]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/12     [cp (48.12 %)     ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/13     [c (42.47 %)      ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/14     [lsp (48.37 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/15     [xls (41.23 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/16     [xls_200 (78.00 %)]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/17     [bin (18.11 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/18     [bin_200 (7.50 %) ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/19     [sum (48.96 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/20     [man (59.21 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)

name                                          old peak-mem(Bytes)/op  new peak-mem(Bytes)/op  delta
BM_UFlat/0      [html             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/1      [urls             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/2      [jpg              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/3      [jpg_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/4      [pdf              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/5      [html4            ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/6      [txt1             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/7      [txt2             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/8      [txt3             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/9      [txt4             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/10     [pb               ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/11     [gaviota          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/12     [cp               ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/13     [c                ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/14     [lsp              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/15     [xls              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/16     [xls_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/17     [bin              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/18     [bin_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/19     [sum              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/20     [man              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/0  [html             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/1  [urls             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/2  [jpg              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/4  [pdf              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/0     [html             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/1     [urls             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/2     [jpg              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/3     [jpg_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlatSink/0  [html             ]               102k ± 0%               102k ± 0%    ~     (all samples are equal)
BM_UFlatSink/1  [urls             ]               702k ± 0%               702k ± 0%    ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]               123k ± 0%               123k ± 0%    ~     (all samples are equal)
BM_UFlatSink/3  [jpg_200          ]                201 ± 0%                201 ± 0%    ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]               102k ± 0%               102k ± 0%    ~     (all samples are equal)
BM_UFlatSink/5  [html4            ]               410k ± 0%               410k ± 0%    ~     (all samples are equal)
BM_UFlatSink/6  [txt1             ]               152k ± 0%               152k ± 0%    ~     (all samples are equal)
BM_UFlatSink/7  [txt2             ]               125k ± 0%               125k ± 0%    ~     (all samples are equal)
BM_UFlatSink/8  [txt3             ]               427k ± 0%               427k ± 0%    ~     (all samples are equal)
BM_UFlatSink/9  [txt4             ]               482k ± 0%               482k ± 0%    ~     (all samples are equal)
BM_UFlatSink/10 [pb               ]               119k ± 0%               119k ± 0%    ~     (all samples are equal)
BM_UFlatSink/11 [gaviota          ]               184k ± 0%               184k ± 0%    ~     (all samples are equal)
BM_UFlatSink/12 [cp               ]              24.6k ± 0%              24.6k ± 0%    ~     (all samples are equal)
BM_UFlatSink/13 [c                ]              11.2k ± 0%              11.2k ± 0%    ~     (all samples are equal)
BM_UFlatSink/14 [lsp              ]              3.72k ± 0%              3.72k ± 0%    ~     (all samples are equal)
BM_UFlatSink/15 [xls              ]              1.03M ± 0%              1.03M ± 0%    ~     (all samples are equal)
BM_UFlatSink/16 [xls_200          ]                201 ± 0%                201 ± 0%    ~     (all samples are equal)
BM_UFlatSink/17 [bin              ]               513k ± 0%               513k ± 0%    ~     (all samples are equal)
BM_UFlatSink/18 [bin_200          ]                201 ± 0%                201 ± 0%    ~     (all samples are equal)
BM_UFlatSink/19 [sum              ]              38.2k ± 0%              38.2k ± 0%    ~     (all samples are equal)
BM_UFlatSink/20 [man              ]              4.23k ± 0%              4.23k ± 0%    ~     (all samples are equal)
BM_ZFlat/0      [html (22.31 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/1      [urls (47.78 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/2      [jpg (99.95 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/3      [jpg_200 (73.00 %)]              30.7k ± 0%              30.7k ± 0%    ~     (all samples are equal)
BM_ZFlat/4      [pdf (83.30 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/5      [html4 (22.52 %)  ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/6      [txt1 (57.88 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/7      [txt2 (61.91 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/8      [txt3 (54.99 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/9      [txt4 (66.26 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/10     [pb (19.68 %)     ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/11     [gaviota (37.72 %)]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/12     [cp (48.12 %)     ]              86.1k ± 0%              86.1k ± 0%    ~     (all samples are equal)
BM_ZFlat/13     [c (42.47 %)      ]              57.0k ± 0%              57.0k ± 0%    ~     (all samples are equal)
BM_ZFlat/14     [lsp (48.37 %)    ]              30.6k ± 0%              30.6k ± 0%    ~     (all samples are equal)
BM_ZFlat/15     [xls (41.23 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/16     [xls_200 (78.00 %)]              30.7k ± 0%              30.7k ± 0%    ~     (all samples are equal)
BM_ZFlat/17     [bin (18.11 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/18     [bin_200 (7.50 %) ]              30.7k ± 0%              30.7k ± 0%    ~     (all samples are equal)
BM_ZFlat/19     [sum (48.96 %)    ]               116k ± 0%               116k ± 0%    ~     (all samples are equal)
BM_ZFlat/20     [man (59.21 %)    ]              30.6k ± 0%              30.6k ± 0%    ~     (all samples are equal)

name                                          old speed               new speed               delta
BM_UFlat/0      [html             ]           2.46GB/s ± 0%           2.45GB/s ± 1%    ~             (p=0.841 n=5+5)
BM_UFlat/1      [urls             ]           1.19GB/s ± 1%           1.20GB/s ± 1%    ~             (p=0.310 n=5+5)
BM_UFlat/2      [jpg              ]           17.3GB/s ± 1%           17.4GB/s ± 1%    ~             (p=0.310 n=5+5)
BM_UFlat/3      [jpg_200          ]           1.56GB/s ± 0%           1.56GB/s ± 0%    ~             (p=0.190 n=4+5)
BM_UFlat/4      [pdf              ]           12.5GB/s ± 1%           12.5GB/s ± 0%    ~             (p=0.548 n=5+5)
BM_UFlat/5      [html4            ]           1.87GB/s ± 0%           1.87GB/s ± 1%    ~             (p=1.000 n=5+5)
BM_UFlat/6      [txt1             ]            791MB/s ± 1%            791MB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/7      [txt2             ]            737MB/s ± 0%            738MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/8      [txt3             ]            839MB/s ± 0%            839MB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/9      [txt4             ]            675MB/s ± 1%            674MB/s ± 0%    ~             (p=0.730 n=5+4)
BM_UFlat/10     [pb               ]           3.08GB/s ± 1%           3.06GB/s ± 0%    ~             (p=0.095 n=5+5)
BM_UFlat/11     [gaviota          ]            974MB/s ± 0%            976MB/s ± 0%    ~             (p=0.238 n=5+5)
BM_UFlat/12     [cp               ]           1.70GB/s ± 0%           1.72GB/s ± 0%  +1.07%          (p=0.016 n=4+5)
BM_UFlat/13     [c                ]           1.53GB/s ± 0%           1.53GB/s ± 1%    ~             (p=1.000 n=5+5)
BM_UFlat/14     [lsp              ]           1.62GB/s ± 1%           1.62GB/s ± 1%    ~             (p=1.000 n=5+5)
BM_UFlat/15     [xls              ]           1.05GB/s ± 1%           1.05GB/s ± 0%    ~             (p=0.556 n=5+4)
BM_UFlat/16     [xls_200          ]            943MB/s ± 0%            940MB/s ± 0%    ~             (p=0.151 n=5+5)
BM_UFlat/17     [bin              ]           1.86GB/s ± 1%           1.86GB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/18     [bin_200          ]           1.99GB/s ± 0%           1.97GB/s ± 1%    ~             (p=0.190 n=5+4)
BM_UFlat/19     [sum              ]           1.30GB/s ± 0%           1.30GB/s ± 1%    ~             (p=0.151 n=5+5)
BM_UFlat/20     [man              ]           1.42GB/s ± 1%           1.42GB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UValidate/0  [html             ]           3.06GB/s ± 0%           3.06GB/s ± 1%    ~             (p=1.000 n=5+5)
BM_UValidate/1  [urls             ]           1.59GB/s ± 0%           1.59GB/s ± 0%    ~             (p=0.095 n=5+5)
BM_UValidate/2  [jpg              ]            845GB/s ± 0%            845GB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UValidate/3  [jpg_200          ]           2.10GB/s ± 0%           2.10GB/s ± 0%    ~             (p=0.310 n=5+5)
BM_UValidate/4  [pdf              ]           35.1GB/s ± 0%           35.1GB/s ± 1%    ~             (p=0.690 n=5+5)
BM_UIOVec/0     [html             ]            843MB/s ± 0%            847MB/s ± 1%    ~             (p=0.222 n=5+5)
BM_UIOVec/1     [urls             ]            652MB/s ± 1%            652MB/s ± 1%    ~             (p=0.310 n=5+5)
BM_UIOVec/2     [jpg              ]           16.5GB/s ± 5%           16.0GB/s ±10%    ~             (p=0.841 n=5+5)
BM_UIOVec/3     [jpg_200          ]            606MB/s ± 1%            614MB/s ± 1%    ~             (p=0.056 n=5+5)
BM_UIOVec/4     [pdf              ]           8.57GB/s ± 0%           8.57GB/s ± 0%    ~             (p=0.343 n=4+4)
BM_UFlatSink/0  [html             ]           2.47GB/s ± 0%           2.45GB/s ± 0%  -0.58%          (p=0.016 n=5+5)
BM_UFlatSink/1  [urls             ]           1.19GB/s ± 0%           1.20GB/s ± 0%    ~             (p=0.548 n=5+5)
BM_UFlatSink/2  [jpg              ]           16.4GB/s ±19%           16.9GB/s ± 4%    ~             (p=0.690 n=5+5)
BM_UFlatSink/3  [jpg_200          ]           1.50GB/s ± 2%           1.50GB/s ± 2%    ~             (p=1.000 n=5+5)
BM_UFlatSink/4  [pdf              ]           12.5GB/s ± 0%           12.5GB/s ± 0%    ~             (p=0.730 n=4+5)
BM_UFlatSink/5  [html4            ]           1.87GB/s ± 1%           1.88GB/s ± 0%    ~             (p=0.421 n=5+5)
BM_UFlatSink/6  [txt1             ]            793MB/s ± 0%            792MB/s ± 1%    ~             (p=0.690 n=5+5)
BM_UFlatSink/7  [txt2             ]            736MB/s ± 0%            736MB/s ± 1%    ~             (p=0.841 n=5+5)
BM_UFlatSink/8  [txt3             ]            839MB/s ± 0%            839MB/s ± 0%    ~             (p=0.548 n=5+5)
BM_UFlatSink/9  [txt4             ]            675MB/s ± 0%            675MB/s ± 0%    ~             (p=0.222 n=5+5)
BM_UFlatSink/10 [pb               ]           3.07GB/s ± 0%           3.09GB/s ± 0%  +0.54%          (p=0.016 n=5+5)
BM_UFlatSink/11 [gaviota          ]            973MB/s ± 0%            971MB/s ± 0%    ~             (p=0.151 n=5+5)
BM_UFlatSink/12 [cp               ]           1.72GB/s ± 1%           1.71GB/s ± 1%    ~             (p=0.421 n=5+5)
BM_UFlatSink/13 [c                ]           1.53GB/s ± 1%           1.52GB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlatSink/14 [lsp              ]           1.63GB/s ± 0%           1.62GB/s ± 1%    ~             (p=0.222 n=5+5)
BM_UFlatSink/15 [xls              ]           1.06GB/s ± 0%           1.05GB/s ± 0%    ~             (p=0.111 n=4+5)
BM_UFlatSink/16 [xls_200          ]            932MB/s ± 1%            928MB/s ± 1%    ~             (p=0.548 n=5+5)
BM_UFlatSink/17 [bin              ]           1.86GB/s ± 0%           1.86GB/s ± 1%    ~             (p=1.000 n=5+5)
BM_UFlatSink/18 [bin_200          ]           1.93GB/s ± 1%           1.94GB/s ± 1%    ~             (p=0.730 n=5+4)
BM_UFlatSink/19 [sum              ]           1.30GB/s ± 0%           1.30GB/s ± 1%    ~             (p=0.690 n=5+5)
BM_UFlatSink/20 [man              ]           1.41GB/s ± 1%           1.41GB/s ± 2%    ~             (p=0.690 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]            815MB/s ± 1%            829MB/s ± 0%  +1.78%          (p=0.008 n=5+5)
BM_ZFlat/1      [urls (47.78 %)   ]            420MB/s ± 1%            432MB/s ± 1%  +2.87%          (p=0.008 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]           10.7GB/s ± 8%           10.9GB/s ± 6%    ~             (p=0.421 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]            544MB/s ± 2%            590MB/s ± 1%  +8.41%          (p=0.008 n=5+5)
BM_ZFlat/4      [pdf (83.30 %)    ]           6.92GB/s ± 3%           7.16GB/s ± 1%  +3.51%          (p=0.008 n=5+5)
BM_ZFlat/5      [html4 (22.52 %)  ]            745MB/s ± 0%            755MB/s ± 0%  +1.34%          (p=0.008 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]            282MB/s ± 0%            285MB/s ± 1%  +1.04%          (p=0.008 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]            262MB/s ± 0%            265MB/s ± 0%  +1.22%          (p=0.008 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]            297MB/s ± 0%            300MB/s ± 0%  +1.09%          (p=0.008 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]            246MB/s ± 1%            248MB/s ± 0%  +0.95%          (p=0.008 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]           1.08GB/s ± 1%           1.11GB/s ± 1%  +2.57%          (p=0.008 n=5+5)
BM_ZFlat/11     [gaviota (37.72 %)]            449MB/s ± 1%            451MB/s ± 0%    ~             (p=0.056 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            530MB/s ± 1%            552MB/s ± 0%  +4.17%          (p=0.008 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            628MB/s ± 1%            640MB/s ± 0%  +1.85%          (p=0.008 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            665MB/s ± 0%            697MB/s ± 1%  +4.71%          (p=0.008 n=5+5)
BM_ZFlat/15     [xls (41.23 %)    ]            635MB/s ± 0%            634MB/s ± 0%    ~             (p=0.310 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]            511MB/s ± 1%            522MB/s ± 2%  +2.23%          (p=0.008 n=5+5)
BM_ZFlat/17     [bin (18.11 %)    ]           1.01GB/s ± 1%           1.02GB/s ± 0%  +1.67%          (p=0.008 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]           2.41GB/s ± 3%           2.37GB/s ± 4%    ~             (p=0.222 n=5+5)
BM_ZFlat/19     [sum (48.96 %)    ]            480MB/s ± 0%            490MB/s ± 1%  +2.24%          (p=0.008 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            545MB/s ± 0%            569MB/s ± 1%  +4.38%          (p=0.008 n=5+5)
2019-02-26 18:27:31 -08:00
costan 3f194acb57 Convert DCHECK to assert.
A previous CL introduced a use of DCHECK. The open source build does not
support DCHECK, and this project uses assert() instead of DCHECK.
2019-01-08 13:49:15 -08:00
costan 97a20b480f Reduce the LeftShiftOverflows() table size.
A previous CL introduced LeftShiftOverflows(), which takes a uint32
input. However, the value it operates on is guaranteed to only have 8
bits set. This CL takes advantage of this restriction to reduce the size
of the static table used to compute LeftShiftOverflows().

The same methodology as the previous CL suggests a 0.6% improvement. The
improvement is likely bigger on mobile CPUs that have much smaller
caches.

Benchmark results:

name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            42.5µs ± 1%             42.1µs ± 0%  -0.87%        (p=0.000 n=20+20)
BM_UFlat/1      [urls             ]             575µs ± 0%              574µs ± 0%  -0.16%        (p=0.000 n=20+19)
BM_UFlat/2      [jpg              ]            7.13µs ± 1%             7.20µs ± 5%    ~           (p=0.422 n=16+19)
BM_UFlat/3      [jpg_200          ]              129ns ± 0%              130ns ± 0%  +0.82%        (p=0.000 n=20+17)
BM_UFlat/4      [pdf              ]            8.22µs ± 1%             8.21µs ± 0%    ~           (p=0.586 n=17+17)
BM_UFlat/5      [html4            ]             222µs ± 0%              222µs ± 0%  -0.11%        (p=0.047 n=19+20)
BM_UFlat/6      [txt1             ]             192µs ± 0%              191µs ± 0%  -0.69%        (p=0.000 n=20+20)
BM_UFlat/7      [txt2             ]             169µs ± 0%              169µs ± 0%  -0.28%        (p=0.000 n=20+20)
BM_UFlat/8      [txt3             ]             510µs ± 0%              507µs ± 0%  -0.50%        (p=0.000 n=20+20)
BM_UFlat/9      [txt4             ]             707µs ± 0%              703µs ± 0%  -0.53%        (p=0.000 n=20+20)
BM_UFlat/10     [pb               ]            39.1µs ± 0%             38.5µs ± 0%  -1.56%        (p=0.000 n=20+20)
BM_UFlat/11     [gaviota          ]             189µs ± 0%              189µs ± 0%  -0.42%        (p=0.000 n=20+20)
BM_UFlat/12     [cp               ]            14.2µs ± 0%             14.2µs ± 1%  -0.30%        (p=0.001 n=18+19)
BM_UFlat/13     [c                ]            7.29µs ± 0%             7.34µs ± 1%  +0.59%        (p=0.000 n=19+20)
BM_UFlat/14     [lsp              ]            2.28µs ± 0%             2.29µs ± 1%  +0.39%        (p=0.000 n=19+18)
BM_UFlat/15     [xls              ]             905µs ± 0%              904µs ± 0%  -0.12%        (p=0.030 n=20+20)
BM_UFlat/16     [xls_200          ]              213ns ± 2%              215ns ± 4%  +0.92%        (p=0.011 n=20+20)
BM_UFlat/17     [bin              ]             274µs ± 0%              275µs ± 0%  +0.55%        (p=0.000 n=20+20)
BM_UFlat/18     [bin_200          ]              101ns ± 1%              101ns ± 1%    ~           (p=0.913 n=18+18)
BM_UFlat/19     [sum              ]            27.9µs ± 1%             27.5µs ± 1%  -1.38%        (p=0.000 n=20+20)
BM_UFlat/20     [man              ]            2.97µs ± 1%             2.97µs ± 1%    ~           (p=0.835 n=20+19)
BM_UValidate/0  [html             ]            33.5µs ± 0%             34.2µs ± 0%  +2.32%        (p=0.000 n=20+20)
BM_UValidate/1  [urls             ]             441µs ± 0%              442µs ± 0%  +0.15%        (p=0.010 n=20+20)
BM_UValidate/2  [jpg              ]              144ns ± 0%              146ns ± 0%  +1.32%        (p=0.000 n=20+20)
BM_UValidate/3  [jpg_200          ]             95.3ns ± 0%             96.0ns ± 0%  +0.68%        (p=0.000 n=20+20)
BM_UValidate/4  [pdf              ]            2.86µs ± 0%             2.88µs ± 1%  +0.67%        (p=0.000 n=19+19)
BM_UIOVec/0     [html             ]             122µs ± 0%              122µs ± 0%  -0.25%        (p=0.000 n=20+20)
BM_UIOVec/1     [urls             ]             1.08ms ± 0%             1.08ms ± 0%    ~           (p=0.068 n=20+20)
BM_UIOVec/2     [jpg              ]            7.63µs ± 7%             7.76µs ±11%    ~           (p=0.396 n=19+20)
BM_UIOVec/3     [jpg_200          ]              325ns ± 0%              326ns ± 0%  +0.27%        (p=0.000 n=20+18)
BM_UIOVec/4     [pdf              ]            12.1µs ± 2%             12.1µs ± 3%    ~           (p=0.967 n=19+20)
BM_UFlatSink/0  [html             ]            42.4µs ± 0%             42.1µs ± 0%  -0.89%        (p=0.000 n=20+20)
BM_UFlatSink/1  [urls             ]             575µs ± 0%              575µs ± 0%    ~           (p=0.883 n=20+20)
BM_UFlatSink/2  [jpg              ]            7.58µs ±16%             7.52µs ±15%    ~           (p=0.945 n=19+20)
BM_UFlatSink/3  [jpg_200          ]              133ns ± 4%              133ns ± 4%    ~           (p=0.627 n=19+20)
BM_UFlatSink/4  [pdf              ]            8.29µs ± 4%             8.39µs ± 4%  +1.14%        (p=0.013 n=19+18)
BM_UFlatSink/5  [html4            ]             223µs ± 0%              222µs ± 0%  -0.18%        (p=0.001 n=20+20)
BM_UFlatSink/6  [txt1             ]             192µs ± 0%              191µs ± 0%  -0.71%        (p=0.000 n=20+20)
BM_UFlatSink/7  [txt2             ]             169µs ± 0%              169µs ± 0%  -0.26%        (p=0.000 n=20+20)
BM_UFlatSink/8  [txt3             ]             510µs ± 0%              508µs ± 0%  -0.50%        (p=0.000 n=20+20)
BM_UFlatSink/9  [txt4             ]             707µs ± 0%              704µs ± 0%  -0.44%        (p=0.000 n=20+20)
BM_UFlatSink/10 [pb               ]            39.1µs ± 0%             38.5µs ± 1%  -1.62%        (p=0.000 n=19+20)
BM_UFlatSink/11 [gaviota          ]             189µs ± 0%              189µs ± 0%  -0.39%        (p=0.000 n=20+20)
BM_UFlatSink/12 [cp               ]            14.2µs ± 0%             14.2µs ± 1%    ~           (p=0.435 n=19+19)
BM_UFlatSink/13 [c                ]            7.29µs ± 0%             7.33µs ± 1%  +0.57%        (p=0.000 n=19+20)
BM_UFlatSink/14 [lsp              ]            2.29µs ± 0%             2.29µs ± 1%    ~           (p=0.791 n=18+18)
BM_UFlatSink/15 [xls              ]             903µs ± 0%              902µs ± 0%  -0.11%        (p=0.044 n=20+19)
BM_UFlatSink/16 [xls_200          ]              215ns ± 1%              215ns ± 1%    ~           (p=0.885 n=19+19)
BM_UFlatSink/17 [bin              ]             274µs ± 0%              275µs ± 0%  +0.51%        (p=0.000 n=20+20)
BM_UFlatSink/18 [bin_200          ]              103ns ± 2%              103ns ± 0%  -0.41%        (p=0.016 n=20+15)
BM_UFlatSink/19 [sum              ]            27.9µs ± 1%             27.5µs ± 1%  -1.34%        (p=0.000 n=20+19)
BM_UFlatSink/20 [man              ]            2.98µs ± 1%             2.97µs ± 1%    ~           (p=0.358 n=18+19)
BM_ZFlat/0      [html (22.31 %)   ]             126µs ± 0%              126µs ± 0%  +0.14%        (p=0.011 n=20+20)
BM_ZFlat/1      [urls (47.78 %)   ]             1.67ms ± 0%             1.67ms ± 0%  +0.11%        (p=0.043 n=20+20)
BM_ZFlat/2      [jpg (99.95 %)    ]            11.5µs ± 6%             11.7µs ± 7%    ~           (p=0.142 n=20+20)
BM_ZFlat/3      [jpg_200 (73.00 %)]              349ns ± 3%              351ns ± 3%    ~           (p=0.573 n=18+20)
BM_ZFlat/4      [pdf (83.30 %)    ]            14.6µs ± 2%             14.7µs ± 4%    ~           (p=0.879 n=19+20)
BM_ZFlat/5      [html4 (22.52 %)  ]             553µs ± 0%              552µs ± 0%  -0.23%        (p=0.000 n=20+20)
BM_ZFlat/6      [txt1 (57.88 %)   ]             540µs ± 0%              540µs ± 0%    ~           (p=0.221 n=20+20)
BM_ZFlat/7      [txt2 (61.91 %)   ]             479µs ± 0%              481µs ± 1%  +0.47%        (p=0.000 n=20+20)
BM_ZFlat/8      [txt3 (54.99 %)   ]             1.44ms ± 0%             1.44ms ± 0%  +0.13%        (p=0.040 n=20+20)
BM_ZFlat/9      [txt4 (66.26 %)   ]             1.97ms ± 0%             1.97ms ± 0%  +0.16%        (p=0.009 n=20+20)
BM_ZFlat/10     [pb (19.68 %)     ]             110µs ± 1%              109µs ± 1%  -0.79%        (p=0.000 n=20+20)
BM_ZFlat/11     [gaviota (37.72 %)]             410µs ± 0%              410µs ± 0%    ~           (p=0.149 n=20+19)
BM_ZFlat/12     [cp (48.12 %)     ]            45.4µs ± 1%             44.9µs ± 1%  -1.23%        (p=0.000 n=20+20)
BM_ZFlat/13     [c (42.47 %)      ]            17.5µs ± 0%             17.5µs ± 1%    ~           (p=0.883 n=20+20)
BM_ZFlat/14     [lsp (48.37 %)    ]            5.51µs ± 1%             5.46µs ± 1%  -0.95%        (p=0.000 n=20+18)
BM_ZFlat/15     [xls (41.23 %)    ]             1.61ms ± 0%             1.62ms ± 0%    ~           (p=0.183 n=20+20)
BM_ZFlat/16     [xls_200 (78.00 %)]              389ns ± 2%              391ns ± 3%    ~           (p=0.740 n=18+20)
BM_ZFlat/17     [bin (18.11 %)    ]             508µs ± 0%              508µs ± 0%    ~           (p=0.779 n=20+20)
BM_ZFlat/18     [bin_200 (7.50 %) ]             87.4ns ± 5%             88.1ns ± 8%    ~           (p=0.367 n=16+19)
BM_ZFlat/19     [sum (48.96 %)    ]            79.1µs ± 0%             80.2µs ± 0%  +1.39%        (p=0.000 n=20+20)
BM_ZFlat/20     [man (59.21 %)    ]            7.55µs ± 1%             7.57µs ± 1%  +0.31%        (p=0.025 n=19+19)

name                                          old speed               new speed               delta
BM_UFlat/0      [html             ]           2.42GB/s ± 0%           2.44GB/s ± 0%  +0.77%        (p=0.000 n=19+19)
BM_UFlat/1      [urls             ]           1.22GB/s ± 0%           1.23GB/s ± 0%  +0.06%        (p=0.000 n=20+19)
BM_UFlat/2      [jpg              ]           17.3GB/s ± 2%           17.2GB/s ± 4%    ~           (p=0.433 n=17+19)
BM_UFlat/3      [jpg_200          ]           1.56GB/s ± 0%           1.54GB/s ± 0%  -0.82%        (p=0.000 n=20+20)
BM_UFlat/4      [pdf              ]           12.5GB/s ± 1%           12.5GB/s ± 1%    ~           (p=0.322 n=17+17)
BM_UFlat/5      [html4            ]           1.85GB/s ± 0%           1.85GB/s ± 0%  +0.16%        (p=0.000 n=20+20)
BM_UFlat/6      [txt1             ]            794MB/s ± 0%            800MB/s ± 0%  +0.68%        (p=0.000 n=18+20)
BM_UFlat/7      [txt2             ]            741MB/s ± 0%            743MB/s ± 0%  +0.30%        (p=0.000 n=19+19)
BM_UFlat/8      [txt3             ]            840MB/s ± 0%            844MB/s ± 0%  +0.53%        (p=0.000 n=18+20)
BM_UFlat/9      [txt4             ]            684MB/s ± 0%            688MB/s ± 0%  +0.57%        (p=0.000 n=20+17)
BM_UFlat/10     [pb               ]           3.04GB/s ± 0%           3.09GB/s ± 0%  +1.60%        (p=0.000 n=19+20)
BM_UFlat/11     [gaviota          ]            977MB/s ± 0%            981MB/s ± 0%  +0.45%        (p=0.000 n=19+19)
BM_UFlat/12     [cp               ]           1.74GB/s ± 0%           1.74GB/s ± 0%  +0.29%        (p=0.000 n=20+19)
BM_UFlat/13     [c                ]           1.53GB/s ± 0%           1.52GB/s ± 1%  -0.56%        (p=0.000 n=19+20)
BM_UFlat/14     [lsp              ]           1.64GB/s ± 0%           1.63GB/s ± 1%  -0.38%        (p=0.000 n=19+20)
BM_UFlat/15     [xls              ]           1.14GB/s ± 0%           1.14GB/s ± 0%  +0.11%        (p=0.000 n=19+20)
BM_UFlat/16     [xls_200          ]            941MB/s ± 1%            931MB/s ± 4%  -1.02%        (p=0.001 n=19+20)
BM_UFlat/17     [bin              ]           1.88GB/s ± 0%           1.87GB/s ± 0%  -0.51%        (p=0.000 n=20+20)
BM_UFlat/18     [bin_200          ]           1.98GB/s ± 0%           1.98GB/s ± 1%    ~           (p=0.767 n=18+18)
BM_UFlat/19     [sum              ]           1.37GB/s ± 0%           1.39GB/s ± 0%  +1.46%        (p=0.000 n=20+20)
BM_UFlat/20     [man              ]           1.43GB/s ± 0%           1.43GB/s ± 0%    ~           (p=0.501 n=18+18)
BM_UValidate/0  [html             ]           3.07GB/s ± 0%           3.00GB/s ± 0%  -2.25%        (p=0.000 n=20+20)
BM_UValidate/1  [urls             ]           1.60GB/s ± 0%           1.59GB/s ± 0%  -0.11%        (p=0.000 n=18+19)
BM_UValidate/2  [jpg              ]            859GB/s ± 0%            848GB/s ± 0%  -1.29%        (p=0.000 n=20+19)
BM_UValidate/3  [jpg_200          ]           2.10GB/s ± 0%           2.09GB/s ± 0%  -0.68%        (p=0.000 n=19+20)
BM_UValidate/4  [pdf              ]           35.9GB/s ± 0%           35.6GB/s ± 1%  -0.71%        (p=0.000 n=20+20)
BM_UIOVec/0     [html             ]            843MB/s ± 0%            844MB/s ± 0%  +0.21%        (p=0.000 n=20+20)
BM_UIOVec/1     [urls             ]            651MB/s ± 0%            650MB/s ± 0%  -0.10%        (p=0.000 n=20+20)
BM_UIOVec/2     [jpg              ]           16.2GB/s ± 6%           16.0GB/s ±10%    ~           (p=0.380 n=19+20)
BM_UIOVec/3     [jpg_200          ]            617MB/s ± 0%            615MB/s ± 0%  -0.24%        (p=0.000 n=20+17)
BM_UIOVec/4     [pdf              ]           8.52GB/s ± 3%           8.50GB/s ± 3%    ~           (p=0.771 n=19+20)
BM_UFlatSink/0  [html             ]           2.42GB/s ± 0%           2.44GB/s ± 0%  +0.93%        (p=0.000 n=20+20)
BM_UFlatSink/1  [urls             ]           1.23GB/s ± 0%           1.23GB/s ± 0%  +0.04%        (p=0.006 n=20+20)
BM_UFlatSink/2  [jpg              ]           16.4GB/s ±14%           16.5GB/s ±13%    ~           (p=0.879 n=19+20)
BM_UFlatSink/3  [jpg_200          ]           1.51GB/s ± 4%           1.51GB/s ± 4%    ~           (p=0.874 n=18+20)
BM_UFlatSink/4  [pdf              ]           12.4GB/s ± 4%           12.3GB/s ± 4%  -1.11%        (p=0.016 n=19+18)
BM_UFlatSink/5  [html4            ]           1.85GB/s ± 0%           1.85GB/s ± 0%  +0.20%        (p=0.000 n=20+20)
BM_UFlatSink/6  [txt1             ]            794MB/s ± 0%            799MB/s ± 0%  +0.72%        (p=0.000 n=19+20)
BM_UFlatSink/7  [txt2             ]            741MB/s ± 0%            743MB/s ± 0%  +0.30%        (p=0.000 n=18+20)
BM_UFlatSink/8  [txt3             ]            839MB/s ± 0%            843MB/s ± 0%  +0.52%        (p=0.000 n=20+18)
BM_UFlatSink/9  [txt4             ]            684MB/s ± 0%            687MB/s ± 0%  +0.46%        (p=0.000 n=20+20)
BM_UFlatSink/10 [pb               ]           3.04GB/s ± 0%           3.09GB/s ± 0%  +1.71%        (p=0.000 n=20+19)
BM_UFlatSink/11 [gaviota          ]            976MB/s ± 0%            980MB/s ± 0%  +0.45%        (p=0.000 n=20+20)
BM_UFlatSink/12 [cp               ]           1.74GB/s ± 1%           1.74GB/s ± 1%    ~           (p=0.904 n=20+20)
BM_UFlatSink/13 [c                ]           1.53GB/s ± 0%           1.53GB/s ± 1%  -0.50%        (p=0.000 n=19+20)
BM_UFlatSink/14 [lsp              ]           1.63GB/s ± 1%           1.63GB/s ± 1%    ~           (p=0.358 n=19+18)
BM_UFlatSink/15 [xls              ]           1.14GB/s ± 0%           1.15GB/s ± 0%  +0.12%        (p=0.000 n=20+20)
BM_UFlatSink/16 [xls_200          ]            931MB/s ± 1%            931MB/s ± 1%    ~           (p=0.686 n=19+19)
BM_UFlatSink/17 [bin              ]           1.88GB/s ± 0%           1.87GB/s ± 0%  -0.53%        (p=0.000 n=20+20)
BM_UFlatSink/18 [bin_200          ]           1.94GB/s ± 2%           1.95GB/s ± 1%  +0.42%        (p=0.014 n=20+15)
BM_UFlatSink/19 [sum              ]           1.37GB/s ± 0%           1.39GB/s ± 0%  +1.38%        (p=0.000 n=19+18)
BM_UFlatSink/20 [man              ]           1.42GB/s ± 1%           1.43GB/s ± 0%    ~           (p=0.284 n=18+19)
BM_ZFlat/0      [html (22.31 %)   ]            815MB/s ± 0%            814MB/s ± 0%  -0.15%        (p=0.000 n=20+20)
BM_ZFlat/1      [urls (47.78 %)   ]            423MB/s ± 0%            422MB/s ± 0%  -0.14%        (p=0.000 n=20+20)
BM_ZFlat/2      [jpg (99.95 %)    ]           10.8GB/s ± 5%           10.6GB/s ± 7%    ~           (p=0.142 n=20+20)
BM_ZFlat/3      [jpg_200 (73.00 %)]            574MB/s ± 2%            572MB/s ± 2%    ~           (p=0.613 n=18+20)
BM_ZFlat/4      [pdf (83.30 %)    ]           7.01GB/s ± 2%           7.01GB/s ± 4%    ~           (p=0.593 n=18+20)
BM_ZFlat/5      [html4 (22.52 %)  ]            743MB/s ± 0%            745MB/s ± 0%  +0.25%        (p=0.000 n=20+19)
BM_ZFlat/6      [txt1 (57.88 %)   ]            283MB/s ± 0%            282MB/s ± 0%    ~           (p=0.261 n=18+19)
BM_ZFlat/7      [txt2 (61.91 %)   ]            262MB/s ± 0%            261MB/s ± 0%  -0.35%        (p=0.000 n=20+19)
BM_ZFlat/8      [txt3 (54.99 %)   ]            298MB/s ± 0%            297MB/s ± 0%  -0.11%        (p=0.000 n=20+19)
BM_ZFlat/9      [txt4 (66.26 %)   ]            245MB/s ± 0%            245MB/s ± 0%  -0.13%        (p=0.000 n=19+20)
BM_ZFlat/10     [pb (19.68 %)     ]           1.08GB/s ± 0%           1.09GB/s ± 0%  +0.82%        (p=0.000 n=18+19)
BM_ZFlat/11     [gaviota (37.72 %)]            451MB/s ± 0%            451MB/s ± 0%  -0.05%        (p=0.004 n=19+20)
BM_ZFlat/12     [cp (48.12 %)     ]            543MB/s ± 1%            550MB/s ± 1%  +1.24%        (p=0.000 n=20+20)
BM_ZFlat/13     [c (42.47 %)      ]            638MB/s ± 0%            637MB/s ± 0%    ~           (p=0.708 n=19+19)
BM_ZFlat/14     [lsp (48.37 %)    ]            678MB/s ± 2%            684MB/s ± 1%  +0.89%        (p=0.000 n=20+19)
BM_ZFlat/15     [xls (41.23 %)    ]            640MB/s ± 0%            640MB/s ± 0%  -0.10%        (p=0.000 n=19+19)
BM_ZFlat/16     [xls_200 (78.00 %)]            515MB/s ± 2%            514MB/s ± 3%    ~           (p=0.916 n=18+19)
BM_ZFlat/17     [bin (18.11 %)    ]           1.01GB/s ± 0%           1.01GB/s ± 0%  +0.03%        (p=0.033 n=20+20)
BM_ZFlat/18     [bin_200 (7.50 %) ]           2.30GB/s ± 6%           2.28GB/s ± 9%    ~           (p=0.502 n=16+19)
BM_ZFlat/19     [sum (48.96 %)    ]            485MB/s ± 0%            478MB/s ± 0%  -1.39%        (p=0.000 n=19+20)
BM_ZFlat/20     [man (59.21 %)    ]            562MB/s ± 1%            560MB/s ± 1%  -0.37%        (p=0.016 n=18+19)
2019-01-08 13:48:30 -08:00
costan 4f0adca400 Wrap BMI2 instruction usage in support checks.
A previous version of this was submitted and rolled back due to breakage
-- an attempt to accommodate Visual Studio resulted in compiler errors
on GCC/Clang with -mavx2 but without -mbmi2. This version makes the BMI2
support check more strict, to avoid the errors.

A previous CL introduced _bzhi_u32 (part of Intel's BMI2 instruction
set, released in Haswell) gated by a check for the __BMI2__ preprocessor
macro. This works for Clang and GCC, but does not work on Visual Studio,
and may not work on other compilers.

This CL plumbs the BMI2 support checks through the CMake configuration
used by the open source build. It also replaces the <x86intrin.h>
header, which does not exist on Visual Studio, with the more scoped
headers <tmmintrin.h> (for SSSE3) and <immintrin.h> (for BMI2/AVX2).
Asides from fixing the open source build, the more scoped headers make
it slightly less likely that newer intrinsics will creep in without
proper gating.
2019-01-08 06:44:11 -08:00
nafi 46768e335d Optimize decompression by about 0.82%.
Assembly difference: https://godbolt.org/z/cvlH9b

name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            42.3µs ± 0%             42.5µs ± 0%   +0.57%          (p=0.008 n=5+5)
BM_UFlat/1      [urls             ]             590µs ± 0%              575µs ± 0%   -2.60%          (p=0.008 n=5+5)
BM_UFlat/2      [jpg              ]            7.16µs ± 1%             7.15µs ± 1%     ~             (p=0.841 n=5+5)
BM_UFlat/3      [jpg_200          ]              131ns ± 0%              129ns ± 0%   -1.41%          (p=0.008 n=5+5)
BM_UFlat/4      [pdf              ]            8.21µs ± 0%             8.22µs ± 1%     ~             (p=0.690 n=5+5)
BM_UFlat/5      [html4            ]             222µs ± 0%              223µs ± 0%     ~             (p=0.841 n=5+5)
BM_UFlat/6      [txt1             ]             193µs ± 0%              192µs ± 0%     ~             (p=0.095 n=5+5)
BM_UFlat/7      [txt2             ]             171µs ± 0%              169µs ± 0%   -0.83%          (p=0.008 n=5+5)
BM_UFlat/8      [txt3             ]             511µs ± 0%              510µs ± 0%     ~             (p=0.841 n=5+5)
BM_UFlat/9      [txt4             ]             717µs ± 0%              707µs ± 0%   -1.42%          (p=0.008 n=5+5)
BM_UFlat/10     [pb               ]            38.8µs ± 0%             39.3µs ± 0%   +1.26%          (p=0.008 n=5+5)
BM_UFlat/11     [gaviota          ]             190µs ± 0%              189µs ± 0%   -0.43%          (p=0.032 n=5+5)
BM_UFlat/12     [cp               ]            14.3µs ± 0%             14.2µs ± 0%   -0.92%          (p=0.008 n=5+5)
BM_UFlat/13     [c                ]            7.35µs ± 1%             7.30µs ± 0%   -0.66%          (p=0.032 n=5+5)
BM_UFlat/14     [lsp              ]            2.30µs ± 1%             2.28µs ± 0%     ~             (p=0.056 n=5+5)
BM_UFlat/15     [xls              ]             983µs ± 0%              904µs ± 0%   -7.99%          (p=0.008 n=5+5)
BM_UFlat/16     [xls_200          ]              213ns ± 0%              213ns ± 1%     ~             (p=0.690 n=5+5)
BM_UFlat/17     [bin              ]             278µs ± 0%              274µs ± 0%   -1.56%          (p=0.008 n=5+5)
BM_UFlat/18     [bin_200          ]              101ns ± 0%              101ns ± 1%     ~             (p=1.000 n=5+5)
BM_UFlat/19     [sum              ]            29.4µs ± 1%             28.0µs ± 1%   -4.98%          (p=0.008 n=5+5)
BM_UFlat/20     [man              ]            2.97µs ± 0%             2.97µs ± 0%     ~             (p=0.421 n=5+5)
BM_UValidate/0  [html             ]            33.6µs ± 0%             33.6µs ± 0%     ~             (p=0.548 n=5+5)
BM_UValidate/1  [urls             ]             443µs ± 0%              441µs ± 0%   -0.43%          (p=0.016 n=4+5)
BM_UValidate/2  [jpg              ]              146ns ± 0%              144ns ± 0%   -1.63%          (p=0.008 n=5+5)
BM_UValidate/3  [jpg_200          ]             98.6ns ± 0%             95.3ns ± 0%   -3.32%          (p=0.008 n=5+5)
BM_UValidate/4  [pdf              ]            2.89µs ± 1%             2.85µs ± 0%   -1.22%          (p=0.008 n=5+5)
BM_UIOVec/0     [html             ]             122µs ± 0%              122µs ± 0%     ~             (p=1.000 n=5+5)
BM_UIOVec/1     [urls             ]             1.08ms ± 0%             1.08ms ± 0%     ~             (p=0.095 n=5+5)
BM_UIOVec/2     [jpg              ]            7.51µs ± 4%             7.69µs ± 6%     ~             (p=0.421 n=5+5)
BM_UIOVec/3     [jpg_200          ]              327ns ± 0%              327ns ± 1%     ~             (p=0.730 n=4+5)
BM_UIOVec/4     [pdf              ]            12.0µs ± 1%             12.0µs ± 0%     ~             (p=0.286 n=5+4)
BM_UFlatSink/0  [html             ]            42.3µs ± 0%             42.5µs ± 0%   +0.46%          (p=0.008 n=5+5)
BM_UFlatSink/1  [urls             ]             589µs ± 0%              575µs ± 0%   -2.36%          (p=0.008 n=5+5)
BM_UFlatSink/2  [jpg              ]            7.40µs ± 8%             7.74µs ± 9%     ~             (p=0.310 n=5+5)
BM_UFlatSink/3  [jpg_200          ]              134ns ± 0%              131ns ± 0%   -1.78%          (p=0.008 n=5+5)
BM_UFlatSink/4  [pdf              ]            8.28µs ± 3%             8.35µs ± 6%     ~             (p=0.548 n=5+5)
BM_UFlatSink/5  [html4            ]             222µs ± 0%              222µs ± 0%     ~             (p=0.690 n=5+5)
BM_UFlatSink/6  [txt1             ]             193µs ± 0%              192µs ± 0%     ~             (p=0.222 n=5+5)
BM_UFlatSink/7  [txt2             ]             171µs ± 0%              169µs ± 0%   -0.91%          (p=0.008 n=5+5)
BM_UFlatSink/8  [txt3             ]             512µs ± 0%              510µs ± 0%   -0.28%          (p=0.032 n=5+5)
BM_UFlatSink/9  [txt4             ]             717µs ± 0%              707µs ± 0%   -1.32%          (p=0.008 n=5+5)
BM_UFlatSink/10 [pb               ]            38.7µs ± 0%             39.2µs ± 0%   +1.29%          (p=0.008 n=5+5)
BM_UFlatSink/11 [gaviota          ]             190µs ± 0%              189µs ± 0%   -0.47%          (p=0.008 n=5+5)
BM_UFlatSink/12 [cp               ]            14.3µs ± 0%             14.2µs ± 0%   -0.65%          (p=0.008 n=5+5)
BM_UFlatSink/13 [c                ]            7.36µs ± 1%             7.29µs ± 0%   -0.92%          (p=0.008 n=5+5)
BM_UFlatSink/14 [lsp              ]            2.30µs ± 1%             2.29µs ± 0%     ~             (p=0.841 n=5+5)
BM_UFlatSink/15 [xls              ]             980µs ± 0%              903µs ± 0%   -7.92%          (p=0.008 n=5+5)
BM_UFlatSink/16 [xls_200          ]              217ns ± 0%              215ns ± 0%   -0.94%          (p=0.008 n=5+5)
BM_UFlatSink/17 [bin              ]             278µs ± 0%              273µs ± 0%   -1.56%          (p=0.008 n=5+5)
BM_UFlatSink/18 [bin_200          ]              107ns ± 5%              104ns ± 0%     ~             (p=0.056 n=5+5)
BM_UFlatSink/19 [sum              ]            29.5µs ± 0%             27.9µs ± 0%   -5.32%          (p=0.008 n=5+5)
BM_UFlatSink/20 [man              ]            3.01µs ± 0%             3.00µs ± 1%     ~             (p=0.310 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]             127µs ± 0%              126µs ± 0%   -0.46%          (p=0.008 n=5+5)
BM_ZFlat/1      [urls (47.78 %)   ]             1.67ms ± 0%             1.67ms ± 0%     ~             (p=0.548 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]            11.5µs ± 3%             11.6µs ± 6%     ~             (p=0.841 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]              350ns ± 2%              347ns ± 0%     ~             (p=0.905 n=5+4)
BM_ZFlat/4      [pdf (83.30 %)    ]            14.6µs ± 4%             14.6µs ± 1%     ~             (p=0.421 n=5+5)
BM_ZFlat/5      [html4 (22.52 %)  ]             553µs ± 0%              553µs ± 0%     ~             (p=0.690 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]             540µs ± 0%              540µs ± 0%     ~             (p=1.000 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]             481µs ± 0%              479µs ± 0%   -0.54%          (p=0.008 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]             1.44ms ± 0%             1.44ms ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]             1.97ms ± 0%             1.97ms ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]             110µs ± 0%              110µs ± 0%     ~             (p=0.841 n=5+5)
BM_ZFlat/11     [gaviota (37.72 %)]             411µs ± 0%              410µs ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            46.1µs ± 1%             45.8µs ± 0%     ~             (p=0.056 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            17.6µs ± 0%             17.6µs ± 1%     ~             (p=0.310 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            5.46µs ± 1%             5.49µs ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/15     [xls (41.23 %)    ]             1.62ms ± 0%             1.61ms ± 0%     ~             (p=0.190 n=4+5)
BM_ZFlat/16     [xls_200 (78.00 %)]              392ns ± 2%              385ns ± 1%     ~             (p=0.200 n=4+4)
BM_ZFlat/17     [bin (18.11 %)    ]             509µs ± 0%              508µs ± 0%   -0.26%          (p=0.008 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]             90.2ns ±15%             80.8ns ± 0%  -10.39%          (p=0.016 n=5+4)
BM_ZFlat/19     [sum (48.96 %)    ]            81.1µs ± 0%             79.1µs ± 1%   -2.37%          (p=0.008 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            7.61µs ± 1%             7.57µs ± 1%     ~             (p=0.421 n=5+5)

name                                          old allocs/op           new allocs/op           delta
BM_UFlat/0      [html             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/1      [urls             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/2      [jpg              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/3      [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/4      [pdf              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/5      [html4            ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/6      [txt1             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/7      [txt2             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/8      [txt3             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/9      [txt4             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/10     [pb               ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/11     [gaviota          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/12     [cp               ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/13     [c                ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/14     [lsp              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/15     [xls              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/16     [xls_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/17     [bin              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/18     [bin_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/19     [sum              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/20     [man              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UValidate/0  [html             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UValidate/1  [urls             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UValidate/2  [jpg              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UValidate/4  [pdf              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UIOVec/0     [html             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UIOVec/1     [urls             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UIOVec/2     [jpg              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UIOVec/3     [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/0  [html             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/1  [urls             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/3  [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/5  [html4            ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/6  [txt1             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/7  [txt2             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/8  [txt3             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/9  [txt4             ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/10 [pb               ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/11 [gaviota          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/12 [cp               ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/13 [c                ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/14 [lsp              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/15 [xls              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/16 [xls_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/17 [bin              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/18 [bin_200          ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/19 [sum              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_UFlatSink/20 [man              ]               0.00 ±NaN%              0.00 ±NaN%     ~     (all samples are equal)
BM_ZFlat/0      [html (22.31 %)   ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/1      [urls (47.78 %)   ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/2      [jpg (99.95 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/3      [jpg_200 (73.00 %)]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/4      [pdf (83.30 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/5      [html4 (22.52 %)  ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/6      [txt1 (57.88 %)   ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/7      [txt2 (61.91 %)   ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/8      [txt3 (54.99 %)   ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/9      [txt4 (66.26 %)   ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/10     [pb (19.68 %)     ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/11     [gaviota (37.72 %)]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/12     [cp (48.12 %)     ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/13     [c (42.47 %)      ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/14     [lsp (48.37 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/15     [xls (41.23 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/16     [xls_200 (78.00 %)]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/17     [bin (18.11 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/18     [bin_200 (7.50 %) ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/19     [sum (48.96 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)
BM_ZFlat/20     [man (59.21 %)    ]               1.00 ± 0%               1.00 ± 0%     ~     (all samples are equal)

name                                          old peak-mem(Bytes)/op  new peak-mem(Bytes)/op  delta
BM_UFlat/0      [html             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/1      [urls             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/2      [jpg              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/3      [jpg_200          ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/4      [pdf              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/5      [html4            ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/6      [txt1             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/7      [txt2             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/8      [txt3             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/9      [txt4             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/10     [pb               ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/11     [gaviota          ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/12     [cp               ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/13     [c                ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/14     [lsp              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/15     [xls              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/16     [xls_200          ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/17     [bin              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/18     [bin_200          ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/19     [sum              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlat/20     [man              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UValidate/0  [html             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UValidate/1  [urls             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UValidate/2  [jpg              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UValidate/4  [pdf              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UIOVec/0     [html             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UIOVec/1     [urls             ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UIOVec/2     [jpg              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UIOVec/3     [jpg_200          ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]               4.00 ± 0%               4.00 ± 0%     ~     (all samples are equal)
BM_UFlatSink/0  [html             ]               102k ± 0%               102k ± 0%     ~     (all samples are equal)
BM_UFlatSink/1  [urls             ]               702k ± 0%               702k ± 0%     ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]               123k ± 0%               123k ± 0%     ~     (all samples are equal)
BM_UFlatSink/3  [jpg_200          ]                201 ± 0%                201 ± 0%     ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]               102k ± 0%               102k ± 0%     ~     (all samples are equal)
BM_UFlatSink/5  [html4            ]               410k ± 0%               410k ± 0%     ~     (all samples are equal)
BM_UFlatSink/6  [txt1             ]               152k ± 0%               152k ± 0%     ~     (all samples are equal)
BM_UFlatSink/7  [txt2             ]               125k ± 0%               125k ± 0%     ~     (all samples are equal)
BM_UFlatSink/8  [txt3             ]               427k ± 0%               427k ± 0%     ~     (all samples are equal)
BM_UFlatSink/9  [txt4             ]               482k ± 0%               482k ± 0%     ~     (all samples are equal)
BM_UFlatSink/10 [pb               ]               119k ± 0%               119k ± 0%     ~     (all samples are equal)
BM_UFlatSink/11 [gaviota          ]               184k ± 0%               184k ± 0%     ~     (all samples are equal)
BM_UFlatSink/12 [cp               ]              24.6k ± 0%              24.6k ± 0%     ~     (all samples are equal)
BM_UFlatSink/13 [c                ]              11.2k ± 0%              11.2k ± 0%     ~     (all samples are equal)
BM_UFlatSink/14 [lsp              ]              3.72k ± 0%              3.72k ± 0%     ~     (all samples are equal)
BM_UFlatSink/15 [xls              ]              1.03M ± 0%              1.03M ± 0%     ~     (all samples are equal)
BM_UFlatSink/16 [xls_200          ]                201 ± 0%                201 ± 0%     ~     (all samples are equal)
BM_UFlatSink/17 [bin              ]               513k ± 0%               513k ± 0%     ~     (all samples are equal)
BM_UFlatSink/18 [bin_200          ]                201 ± 0%                201 ± 0%     ~     (all samples are equal)
BM_UFlatSink/19 [sum              ]              38.2k ± 0%              38.2k ± 0%     ~     (all samples are equal)
BM_UFlatSink/20 [man              ]              4.23k ± 0%              4.23k ± 0%     ~     (all samples are equal)
BM_ZFlat/0      [html (22.31 %)   ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/1      [urls (47.78 %)   ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/2      [jpg (99.95 %)    ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/3      [jpg_200 (73.00 %)]              30.7k ± 0%              30.7k ± 0%     ~     (all samples are equal)
BM_ZFlat/4      [pdf (83.30 %)    ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/5      [html4 (22.52 %)  ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/6      [txt1 (57.88 %)   ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/7      [txt2 (61.91 %)   ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/8      [txt3 (54.99 %)   ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/9      [txt4 (66.26 %)   ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/10     [pb (19.68 %)     ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/11     [gaviota (37.72 %)]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/12     [cp (48.12 %)     ]              86.1k ± 0%              86.1k ± 0%     ~     (all samples are equal)
BM_ZFlat/13     [c (42.47 %)      ]              57.0k ± 0%              57.0k ± 0%     ~     (all samples are equal)
BM_ZFlat/14     [lsp (48.37 %)    ]              30.6k ± 0%              30.6k ± 0%     ~     (all samples are equal)
BM_ZFlat/15     [xls (41.23 %)    ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/16     [xls_200 (78.00 %)]              30.7k ± 0%              30.7k ± 0%     ~     (all samples are equal)
BM_ZFlat/17     [bin (18.11 %)    ]               175k ± 0%               175k ± 0%     ~     (all samples are equal)
BM_ZFlat/18     [bin_200 (7.50 %) ]              30.7k ± 0%              30.7k ± 0%     ~     (all samples are equal)
BM_ZFlat/19     [sum (48.96 %)    ]               116k ± 0%               116k ± 0%     ~     (all samples are equal)
BM_ZFlat/20     [man (59.21 %)    ]              30.6k ± 0%              30.6k ± 0%     ~     (all samples are equal)

name                                          old speed               new speed               delta
BM_UFlat/0      [html             ]           2.43GB/s ± 0%           2.41GB/s ± 0%   -0.59%          (p=0.032 n=5+5)
BM_UFlat/1      [urls             ]           1.19GB/s ± 1%           1.22GB/s ± 0%   +2.58%          (p=0.008 n=5+5)
BM_UFlat/2      [jpg              ]           17.2GB/s ± 1%           17.3GB/s ± 1%     ~             (p=0.421 n=5+5)
BM_UFlat/3      [jpg_200          ]           1.54GB/s ± 1%           1.56GB/s ± 1%   +1.23%          (p=0.008 n=5+5)
BM_UFlat/4      [pdf              ]           12.5GB/s ± 1%           12.5GB/s ± 0%     ~             (p=0.413 n=5+4)
BM_UFlat/5      [html4            ]           1.85GB/s ± 1%           1.85GB/s ± 0%     ~             (p=0.690 n=5+5)
BM_UFlat/6      [txt1             ]            793MB/s ± 0%            794MB/s ± 0%     ~             (p=0.690 n=5+5)
BM_UFlat/7      [txt2             ]            738MB/s ± 0%            742MB/s ± 1%     ~             (p=0.151 n=5+5)
BM_UFlat/8      [txt3             ]            839MB/s ± 0%            838MB/s ± 0%     ~             (p=0.310 n=5+5)
BM_UFlat/9      [txt4             ]            674MB/s ± 0%            684MB/s ± 0%   +1.55%          (p=0.008 n=5+5)
BM_UFlat/10     [pb               ]           3.07GB/s ± 1%           3.03GB/s ± 1%   -1.27%          (p=0.008 n=5+5)
BM_UFlat/11     [gaviota          ]            974MB/s ± 0%            978MB/s ± 0%   +0.50%          (p=0.032 n=5+5)
BM_UFlat/12     [cp               ]           1.72GB/s ± 0%           1.74GB/s ± 1%   +0.79%          (p=0.008 n=5+5)
BM_UFlat/13     [c                ]           1.52GB/s ± 1%           1.53GB/s ± 1%     ~             (p=0.421 n=5+5)
BM_UFlat/14     [lsp              ]           1.62GB/s ± 1%           1.64GB/s ± 0%     ~             (p=0.151 n=5+5)
BM_UFlat/15     [xls              ]           1.05GB/s ± 0%           1.14GB/s ± 1%   +8.60%          (p=0.008 n=5+5)
BM_UFlat/16     [xls_200          ]            942MB/s ± 0%            941MB/s ± 1%     ~             (p=0.690 n=5+5)
BM_UFlat/17     [bin              ]           1.85GB/s ± 0%           1.88GB/s ± 0%   +1.60%          (p=0.008 n=5+5)
BM_UFlat/18     [bin_200          ]           1.99GB/s ± 0%           1.99GB/s ± 0%     ~             (p=0.421 n=5+5)
BM_UFlat/19     [sum              ]           1.30GB/s ± 1%           1.37GB/s ± 1%   +5.28%          (p=0.008 n=5+5)
BM_UFlat/20     [man              ]           1.43GB/s ± 1%           1.42GB/s ± 0%     ~             (p=0.421 n=5+5)
BM_UValidate/0  [html             ]           3.07GB/s ± 0%           3.05GB/s ± 1%     ~             (p=0.222 n=5+5)
BM_UValidate/1  [urls             ]           1.59GB/s ± 0%           1.60GB/s ± 0%     ~             (p=0.310 n=5+5)
BM_UValidate/2  [jpg              ]            845GB/s ± 0%            860GB/s ± 0%   +1.75%          (p=0.008 n=5+5)
BM_UValidate/3  [jpg_200          ]           2.04GB/s ± 1%           2.11GB/s ± 1%   +3.61%          (p=0.008 n=5+5)
BM_UValidate/4  [pdf              ]           35.6GB/s ± 1%           36.1GB/s ± 1%   +1.40%          (p=0.016 n=5+5)
BM_UIOVec/0     [html             ]            845MB/s ± 1%            843MB/s ± 1%     ~             (p=0.310 n=5+5)
BM_UIOVec/1     [urls             ]            653MB/s ± 0%            651MB/s ± 1%     ~             (p=0.190 n=4+5)
BM_UIOVec/2     [jpg              ]           16.4GB/s ± 4%           16.1GB/s ± 5%     ~             (p=0.548 n=5+5)
BM_UIOVec/3     [jpg_200          ]            611MB/s ± 2%            614MB/s ± 0%     ~             (p=0.548 n=5+5)
BM_UIOVec/4     [pdf              ]           8.53GB/s ± 1%           8.52GB/s ± 3%     ~             (p=0.841 n=5+5)
BM_UFlatSink/0  [html             ]           2.43GB/s ± 1%           2.42GB/s ± 0%     ~             (p=0.222 n=5+5)
BM_UFlatSink/1  [urls             ]           1.20GB/s ± 0%           1.23GB/s ± 1%   +2.38%          (p=0.008 n=5+5)
BM_UFlatSink/2  [jpg              ]           16.7GB/s ± 8%           16.0GB/s ± 8%     ~             (p=0.151 n=5+5)
BM_UFlatSink/3  [jpg_200          ]           1.50GB/s ± 0%           1.53GB/s ± 0%   +2.13%          (p=0.008 n=5+5)
BM_UFlatSink/4  [pdf              ]           12.5GB/s ± 0%           12.3GB/s ± 5%     ~             (p=0.730 n=4+5)
BM_UFlatSink/5  [html4            ]           1.85GB/s ± 0%           1.84GB/s ± 0%     ~             (p=0.151 n=5+5)
BM_UFlatSink/6  [txt1             ]            791MB/s ± 0%            791MB/s ± 0%     ~             (p=1.000 n=5+5)
BM_UFlatSink/7  [txt2             ]            735MB/s ± 0%            739MB/s ± 0%   +0.51%          (p=0.016 n=5+4)
BM_UFlatSink/8  [txt3             ]            838MB/s ± 0%            840MB/s ± 0%     ~             (p=0.151 n=5+5)
BM_UFlatSink/9  [txt4             ]            674MB/s ± 0%            683MB/s ± 0%   +1.37%          (p=0.008 n=5+5)
BM_UFlatSink/10 [pb               ]           3.07GB/s ± 0%           3.03GB/s ± 1%   -1.34%          (p=0.008 n=5+5)
BM_UFlatSink/11 [gaviota          ]            973MB/s ± 0%            975MB/s ± 0%     ~             (p=0.310 n=5+5)
BM_UFlatSink/12 [cp               ]           1.73GB/s ± 1%           1.74GB/s ± 1%     ~             (p=0.056 n=5+5)
BM_UFlatSink/13 [c                ]           1.52GB/s ± 1%           1.53GB/s ± 1%   +0.76%          (p=0.032 n=5+5)
BM_UFlatSink/14 [lsp              ]           1.62GB/s ± 0%           1.63GB/s ± 0%     ~             (p=0.548 n=5+5)
BM_UFlatSink/15 [xls              ]           1.05GB/s ± 0%           1.14GB/s ± 0%   +8.57%          (p=0.008 n=5+5)
BM_UFlatSink/16 [xls_200          ]            925MB/s ± 0%            933MB/s ± 0%   +0.85%          (p=0.008 n=5+5)
BM_UFlatSink/17 [bin              ]           1.85GB/s ± 1%           1.88GB/s ± 0%   +1.47%          (p=0.008 n=5+5)
BM_UFlatSink/18 [bin_200          ]           1.88GB/s ± 5%           1.93GB/s ± 0%     ~             (p=0.421 n=5+5)
BM_UFlatSink/19 [sum              ]           1.30GB/s ± 1%           1.37GB/s ± 1%   +5.18%          (p=0.008 n=5+5)
BM_UFlatSink/20 [man              ]           1.41GB/s ± 0%           1.41GB/s ± 1%     ~             (p=0.222 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]            809MB/s ± 0%            814MB/s ± 1%   +0.61%          (p=0.016 n=5+5)
BM_ZFlat/1      [urls (47.78 %)   ]            423MB/s ± 0%            422MB/s ± 0%     ~             (p=0.548 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]           10.8GB/s ± 3%           10.6GB/s ± 5%     ~             (p=0.690 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]            575MB/s ± 2%            579MB/s ± 0%     ~             (p=1.000 n=5+4)
BM_ZFlat/4      [pdf (83.30 %)    ]           7.06GB/s ± 4%           7.05GB/s ± 2%     ~             (p=0.421 n=5+5)
BM_ZFlat/5      [html4 (22.52 %)  ]            745MB/s ± 0%            744MB/s ± 0%     ~             (p=0.421 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]            282MB/s ± 0%            282MB/s ± 1%     ~             (p=1.000 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]            261MB/s ± 0%            263MB/s ± 0%   +0.55%          (p=0.032 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]            297MB/s ± 1%            297MB/s ± 0%     ~             (p=1.000 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]            245MB/s ± 0%            246MB/s ± 0%     ~             (p=0.286 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]           1.08GB/s ± 1%           1.08GB/s ± 0%     ~             (p=0.056 n=5+5)
BM_ZFlat/11     [gaviota (37.72 %)]            450MB/s ± 0%            452MB/s ± 0%   +0.55%          (p=0.016 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            537MB/s ± 1%            538MB/s ± 0%     ~             (p=0.421 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            637MB/s ± 1%            634MB/s ± 1%     ~             (p=0.222 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            684MB/s ± 1%            680MB/s ± 0%     ~             (p=0.310 n=5+5)
BM_ZFlat/15     [xls (41.23 %)    ]            641MB/s ± 0%            640MB/s ± 1%     ~             (p=0.310 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]            501MB/s ± 9%            521MB/s ± 1%     ~             (p=0.111 n=5+4)
BM_ZFlat/17     [bin (18.11 %)    ]           1.01GB/s ± 0%           1.02GB/s ± 1%     ~             (p=0.151 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]           2.24GB/s ±14%           2.48GB/s ± 0%     ~             (p=0.063 n=5+4)
BM_ZFlat/19     [sum (48.96 %)    ]            473MB/s ± 1%            485MB/s ± 1%   +2.47%          (p=0.008 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            558MB/s ± 1%            558MB/s ± 1%     ~             (p=1.000 n=5+5)
2019-01-08 06:35:12 -08:00
costan 81d444e4e4 Remove direct use of _builtin_clz.
A previous CL introduced _builtin_clz in zippy.cc. This is a GCC / Clang
intrinsic, and is not supported in Visual Studio. The rest of the
project uses bit manipulation intrinsics via the functions in Bits::,
which are stubbed out for the open source build in
zippy-stubs-internal.h.

This CL extracts Bits::Log2FloorNonZero() out of Bits::Log2Floor() in
the stubbed version of Bits, adds assertions to the Bits::*NonZero()
functions in the stubs, and converts _builtin_clz to a
Bits::Log2FloorNonZero() call.

The latter part is not obvious. A mathematical proof of correctness is
outlined in added comments. An empirical proof is available at
https://godbolt.org/z/mPKWmh -- CalculateTableSizeOld(), which is the
current code, compiles to the same assembly on Clang as
CalculateTableSizeNew1(), which is the bigger jump in the proof.
CalculateTableSizeNew2() is a fairly obvious transformation from
CalculateTableSizeNew1(), and results in slightly better assembly on all
supported compilers.

Two benchmark runs with the same arguments as the original CL only
showed differences in completely disjoint tests, suggesting that the
differences are pure noise.
2019-01-06 12:49:08 -08:00
costan 925c3094c4 Convert DCHECK to assert.
The open source build does not support DCHECK, and this project uses
assert() instead of DCHECK.
2019-01-04 19:09:15 -08:00
atdt f7aece15e2 Add comment explaining MSan false-positive workaround 2019-01-04 19:09:01 -08:00
atdt 5913c5f8e4 Don't use _bzhi_u32 under MSan
MSan knows that x & 0xFF only uses the lower byte from x but it isn't as
smart about _bzhi_u32(val, 8). (I'll file an upstream bug.)
2019-01-04 19:08:53 -08:00
atdt 136b3ebc31 If BMI instructions are available, use BZHI to extract low bytes.
With --cpu=haswell, this results in some significant speed improvement
(notably 12-14% for html and pb). On k8, performance is not affected (as
expected). Full benchmark results for --cpu={k8,haswell} below.

Haswell
-------

name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            55.2µs ± 0%             49.0µs ± 0%  -11.34%          (p=0.008 n=5+5)
BM_UFlat/1      [urls             ]             612µs ± 0%              604µs ± 0%   -1.21%          (p=0.008 n=5+5)
BM_UFlat/2      [jpg              ]            6.11µs ± 2%             6.07µs ± 1%     ~             (p=0.421 n=5+5)
BM_UFlat/3      [jpg_200          ]              134ns ± 0%              132ns ± 5%   -1.49%          (p=0.048 n=5+5)
BM_UFlat/4      [pdf              ]            8.41µs ± 2%             8.34µs ± 1%     ~             (p=0.222 n=5+5)
BM_UFlat/5      [html4            ]             239µs ± 0%              234µs ± 0%   -2.24%          (p=0.008 n=5+5)
BM_UFlat/6      [txt1             ]             211µs ± 0%              205µs ± 0%   -2.73%          (p=0.008 n=5+5)
BM_UFlat/7      [txt2             ]             185µs ± 0%              181µs ± 0%   -2.34%          (p=0.008 n=5+5)
BM_UFlat/8      [txt3             ]             560µs ± 0%              545µs ± 0%   -2.55%          (p=0.008 n=5+5)
BM_UFlat/9      [txt4             ]             773µs ± 0%              753µs ± 0%   -2.61%          (p=0.008 n=5+5)
BM_UFlat/10     [pb               ]            51.6µs ± 0%             45.3µs ± 0%  -12.28%          (p=0.008 n=5+5)
BM_UFlat/11     [gaviota          ]             209µs ± 0%              204µs ± 0%   -2.28%          (p=0.008 n=5+5)
BM_UFlat/12     [cp               ]            17.3µs ± 0%             15.7µs ± 1%   -9.57%          (p=0.008 n=5+5)
BM_UFlat/13     [c                ]            8.08µs ± 0%             8.00µs ± 0%   -0.99%          (p=0.008 n=5+5)
BM_UFlat/14     [lsp              ]            2.48µs ± 0%             2.45µs ± 0%   -1.11%          (p=0.008 n=5+5)
BM_UFlat/15     [xls              ]             967µs ± 0%              954µs ± 0%   -1.36%          (p=0.008 n=5+5)
BM_UFlat/16     [xls_200          ]              219ns ± 1%              218ns ± 1%     ~             (p=0.444 n=5+5)
BM_UFlat/17     [bin              ]             278µs ± 0%              275µs ± 0%   -0.92%          (p=0.008 n=5+5)
BM_UFlat/18     [bin_200          ]              100ns ± 0%               99ns ± 1%   -1.04%          (p=0.008 n=5+5)
BM_UFlat/19     [sum              ]            34.0µs ± 0%             30.9µs ± 0%   -9.10%          (p=0.008 n=5+5)
BM_UFlat/20     [man              ]            3.21µs ± 0%             3.20µs ± 0%     ~             (p=0.063 n=5+5)
BM_UValidate/0  [html             ]            33.1µs ± 0%             33.6µs ± 0%   +1.69%          (p=0.008 n=5+5)
BM_UValidate/1  [urls             ]             436µs ± 0%              441µs ± 0%   +1.06%          (p=0.008 n=5+5)
BM_UValidate/2  [jpg              ]              141ns ± 0%              142ns ± 0%   +0.71%          (p=0.008 n=5+5)
BM_UValidate/3  [jpg_200          ]             94.3ns ± 0%             95.3ns ± 0%   +1.06%          (p=0.008 n=5+5)
BM_UValidate/4  [pdf              ]            2.87µs ± 0%             2.95µs ± 0%   +2.74%          (p=0.008 n=5+5)
BM_UIOVec/0     [html             ]             126µs ± 0%              124µs ± 0%   -1.50%          (p=0.008 n=5+5)
BM_UIOVec/1     [urls             ]             1.13ms ± 0%             1.11ms ± 0%   -1.95%          (p=0.008 n=5+5)
BM_UIOVec/2     [jpg              ]            6.31µs ± 3%             7.44µs ± 3%  +17.75%          (p=0.008 n=5+5)
BM_UIOVec/3     [jpg_200          ]              332ns ± 1%              318ns ± 1%   -4.22%          (p=0.008 n=5+5)
BM_UIOVec/4     [pdf              ]            12.7µs ± 3%             12.6µs ± 9%     ~             (p=0.222 n=5+5)
BM_UFlatSink/0  [html             ]            55.2µs ± 0%             49.0µs ± 0%  -11.31%          (p=0.008 n=5+5)
BM_UFlatSink/1  [urls             ]             612µs ± 0%              605µs ± 0%   -1.17%          (p=0.008 n=5+5)
BM_UFlatSink/2  [jpg              ]            6.29µs ±12%             6.57µs ± 9%     ~             (p=0.548 n=5+5)
BM_UFlatSink/3  [jpg_200          ]              138ns ± 2%              134ns ± 0%   -2.76%          (p=0.000 n=5+4)
BM_UFlatSink/4  [pdf              ]            8.35µs ± 0%             8.34µs ± 1%     ~             (p=0.905 n=4+5)
BM_UFlatSink/5  [html4            ]             239µs ± 0%              234µs ± 0%   -2.33%          (p=0.008 n=5+5)
BM_UFlatSink/6  [txt1             ]             211µs ± 0%              205µs ± 0%   -2.82%          (p=0.008 n=5+5)
BM_UFlatSink/7  [txt2             ]             185µs ± 0%              181µs ± 0%   -2.18%          (p=0.008 n=5+5)
BM_UFlatSink/8  [txt3             ]             560µs ± 0%              545µs ± 0%   -2.57%          (p=0.008 n=5+5)
BM_UFlatSink/9  [txt4             ]             773µs ± 0%              754µs ± 0%   -2.54%          (p=0.008 n=5+5)
BM_UFlatSink/10 [pb               ]            51.6µs ± 0%             45.3µs ± 0%  -12.19%          (p=0.008 n=5+5)
BM_UFlatSink/11 [gaviota          ]             209µs ± 0%              204µs ± 0%   -2.39%          (p=0.008 n=5+5)
BM_UFlatSink/12 [cp               ]            17.3µs ± 0%             15.6µs ± 0%   -9.98%          (p=0.008 n=5+5)
BM_UFlatSink/13 [c                ]            8.10µs ± 1%             7.98µs ± 0%   -1.53%          (p=0.008 n=5+5)
BM_UFlatSink/14 [lsp              ]            2.49µs ± 1%             2.47µs ± 0%   -0.84%          (p=0.008 n=5+5)
BM_UFlatSink/15 [xls              ]             968µs ± 0%              953µs ± 0%   -1.48%          (p=0.008 n=5+5)
BM_UFlatSink/16 [xls_200          ]              220ns ± 1%              220ns ± 0%     ~             (p=1.000 n=5+4)
BM_UFlatSink/17 [bin              ]             278µs ± 0%              275µs ± 0%   -0.99%          (p=0.008 n=5+5)
BM_UFlatSink/18 [bin_200          ]              102ns ± 1%              103ns ± 0%   +1.18%          (p=0.048 n=5+5)
BM_UFlatSink/19 [sum              ]            34.0µs ± 0%             30.9µs ± 0%   -9.21%          (p=0.008 n=5+5)
BM_UFlatSink/20 [man              ]            3.22µs ± 1%             3.20µs ± 0%   -0.76%          (p=0.032 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]             122µs ± 0%              122µs ± 0%     ~             (p=0.413 n=4+5)
BM_ZFlat/1      [urls (47.78 %)   ]             1.60ms ± 0%             1.60ms ± 0%   -0.06%          (p=0.032 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]            10.5µs ± 2%             10.7µs ± 9%     ~             (p=0.841 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]              310ns ± 1%              309ns ± 3%     ~             (p=0.349 n=4+5)
BM_ZFlat/4      [pdf (83.30 %)    ]            13.5µs ± 1%             13.6µs ± 2%     ~             (p=0.595 n=5+5)
BM_ZFlat/5      [html4 (22.52 %)  ]             533µs ± 0%              532µs ± 0%   -0.08%          (p=0.032 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]             529µs ± 0%              528µs ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]             469µs ± 0%              469µs ± 0%     ~             (p=0.690 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]             1.40ms ± 0%             1.40ms ± 0%     ~             (p=0.548 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]             1.93ms ± 0%             1.92ms ± 0%     ~             (p=0.421 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]             106µs ± 0%              106µs ± 0%     ~             (p=0.548 n=5+5)
BM_ZFlat/11     [gaviota (37.72 %)]             404µs ± 0%              404µs ± 0%     ~             (p=0.841 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            43.2µs ± 0%             43.3µs ± 1%     ~             (p=0.151 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            16.4µs ± 1%             16.4µs ± 0%     ~             (p=0.310 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            4.96µs ± 0%             4.96µs ± 1%     ~             (p=0.651 n=5+5)
BM_ZFlat/15     [xls (41.23 %)    ]             1.54ms ± 0%             1.54ms ± 0%     ~             (p=0.841 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]              352ns ± 2%              351ns ± 1%     ~             (p=0.762 n=5+5)
BM_ZFlat/17     [bin (18.11 %)    ]             491µs ± 0%              491µs ± 0%     ~             (p=0.310 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]             75.6ns ± 1%             77.2ns ± 0%   +2.06%          (p=0.016 n=5+4)
BM_ZFlat/19     [sum (48.96 %)    ]            76.9µs ± 0%             76.7µs ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            6.87µs ± 1%             6.81µs ± 0%   -0.87%          (p=0.008 n=5+5)

name                                          old speed               new speed               delta
BM_UFlat/0      [html             ]           1.85GB/s ± 0%           2.09GB/s ± 0%  +12.83%          (p=0.016 n=4+5)
BM_UFlat/1      [urls             ]           1.15GB/s ± 0%           1.16GB/s ± 0%   +1.25%          (p=0.008 n=5+5)
BM_UFlat/2      [jpg              ]           20.1GB/s ± 2%           20.3GB/s ± 1%     ~             (p=0.421 n=5+5)
BM_UFlat/3      [jpg_200          ]           1.49GB/s ± 0%           1.53GB/s ± 0%   +2.83%          (p=0.016 n=5+4)
BM_UFlat/4      [pdf              ]           12.2GB/s ± 2%           12.3GB/s ± 1%     ~             (p=0.222 n=5+5)
BM_UFlat/5      [html4            ]           1.71GB/s ± 0%           1.75GB/s ± 0%   +2.29%          (p=0.008 n=5+5)
BM_UFlat/6      [txt1             ]            722MB/s ± 0%            742MB/s ± 0%   +2.81%          (p=0.008 n=5+5)
BM_UFlat/7      [txt2             ]            676MB/s ± 0%            692MB/s ± 0%   +2.40%          (p=0.008 n=5+5)
BM_UFlat/8      [txt3             ]            762MB/s ± 0%            782MB/s ± 0%   +2.62%          (p=0.008 n=5+5)
BM_UFlat/9      [txt4             ]            623MB/s ± 0%            640MB/s ± 0%   +2.68%          (p=0.008 n=5+5)
BM_UFlat/10     [pb               ]           2.30GB/s ± 0%           2.62GB/s ± 0%  +13.99%          (p=0.008 n=5+5)
BM_UFlat/11     [gaviota          ]            883MB/s ± 0%            903MB/s ± 0%   +2.33%          (p=0.008 n=5+5)
BM_UFlat/12     [cp               ]           1.42GB/s ± 0%           1.57GB/s ± 1%  +10.57%          (p=0.008 n=5+5)
BM_UFlat/13     [c                ]           1.38GB/s ± 0%           1.39GB/s ± 0%   +1.00%          (p=0.008 n=5+5)
BM_UFlat/14     [lsp              ]           1.50GB/s ± 0%           1.52GB/s ± 0%   +1.12%          (p=0.008 n=5+5)
BM_UFlat/15     [xls              ]           1.06GB/s ± 0%           1.08GB/s ± 0%   +1.34%          (p=0.016 n=5+4)
BM_UFlat/16     [xls_200          ]            913MB/s ± 1%            918MB/s ± 1%     ~             (p=0.421 n=5+5)
BM_UFlat/17     [bin              ]           1.85GB/s ± 0%           1.86GB/s ± 0%   +0.92%          (p=0.008 n=5+5)
BM_UFlat/18     [bin_200          ]           2.01GB/s ± 0%           2.03GB/s ± 1%   +1.10%          (p=0.008 n=5+5)
BM_UFlat/19     [sum              ]           1.13GB/s ± 0%           1.24GB/s ± 0%   +9.99%          (p=0.008 n=5+5)
BM_UFlat/20     [man              ]           1.32GB/s ± 0%           1.32GB/s ± 1%     ~             (p=0.063 n=5+5)
BM_UValidate/0  [html             ]           3.10GB/s ± 0%           3.04GB/s ± 0%   -1.66%          (p=0.008 n=5+5)
BM_UValidate/1  [urls             ]           1.61GB/s ± 0%           1.59GB/s ± 0%   -1.04%          (p=0.008 n=5+5)
BM_UValidate/2  [jpg              ]            875GB/s ± 0%            866GB/s ± 0%   -1.11%          (p=0.008 n=5+5)
BM_UValidate/3  [jpg_200          ]           2.12GB/s ± 0%           2.10GB/s ± 0%   -1.01%          (p=0.016 n=5+4)
BM_UValidate/4  [pdf              ]           35.7GB/s ± 0%           34.7GB/s ± 0%   -2.66%          (p=0.008 n=5+5)
BM_UIOVec/0     [html             ]            813MB/s ± 0%            825MB/s ± 0%   +1.52%          (p=0.008 n=5+5)
BM_UIOVec/1     [urls             ]            622MB/s ± 0%            634MB/s ± 0%   +1.99%          (p=0.008 n=5+5)
BM_UIOVec/2     [jpg              ]           19.5GB/s ± 3%           16.6GB/s ± 3%  -15.08%          (p=0.008 n=5+5)
BM_UIOVec/3     [jpg_200          ]            603MB/s ± 1%            630MB/s ± 1%   +4.42%          (p=0.008 n=5+5)
BM_UIOVec/4     [pdf              ]           8.05GB/s ± 3%           8.12GB/s ± 8%     ~             (p=0.222 n=5+5)
BM_UFlatSink/0  [html             ]           1.85GB/s ± 0%           2.09GB/s ± 0%  +12.76%          (p=0.008 n=5+5)
BM_UFlatSink/1  [urls             ]           1.15GB/s ± 0%           1.16GB/s ± 0%   +1.18%          (p=0.008 n=5+5)
BM_UFlatSink/2  [jpg              ]           19.6GB/s ±11%           18.8GB/s ± 9%     ~             (p=0.548 n=5+5)
BM_UFlatSink/3  [jpg_200          ]           1.45GB/s ± 1%           1.49GB/s ± 0%   +2.82%          (p=0.016 n=5+4)
BM_UFlatSink/4  [pdf              ]           12.3GB/s ± 0%           12.3GB/s ± 1%     ~             (p=0.905 n=4+5)
BM_UFlatSink/5  [html4            ]           1.71GB/s ± 0%           1.75GB/s ± 0%   +2.41%          (p=0.008 n=5+5)
BM_UFlatSink/6  [txt1             ]            722MB/s ± 0%            743MB/s ± 0%   +2.90%          (p=0.008 n=5+5)
BM_UFlatSink/7  [txt2             ]            676MB/s ± 0%            691MB/s ± 0%   +2.23%          (p=0.008 n=5+5)
BM_UFlatSink/8  [txt3             ]            763MB/s ± 0%            783MB/s ± 0%   +2.64%          (p=0.008 n=5+5)
BM_UFlatSink/9  [txt4             ]            623MB/s ± 0%            639MB/s ± 0%   +2.61%          (p=0.008 n=5+5)
BM_UFlatSink/10 [pb               ]           2.30GB/s ± 0%           2.62GB/s ± 0%  +13.86%          (p=0.008 n=5+5)
BM_UFlatSink/11 [gaviota          ]            882MB/s ± 0%            904MB/s ± 0%   +2.45%          (p=0.008 n=5+5)
BM_UFlatSink/12 [cp               ]           1.42GB/s ± 0%           1.58GB/s ± 0%  +11.09%          (p=0.008 n=5+5)
BM_UFlatSink/13 [c                ]           1.38GB/s ± 1%           1.40GB/s ± 0%   +1.56%          (p=0.008 n=5+5)
BM_UFlatSink/14 [lsp              ]           1.50GB/s ± 1%           1.51GB/s ± 1%   +0.85%          (p=0.008 n=5+5)
BM_UFlatSink/15 [xls              ]           1.06GB/s ± 0%           1.08GB/s ± 0%   +1.51%          (p=0.016 n=5+4)
BM_UFlatSink/16 [xls_200          ]            908MB/s ± 1%            911MB/s ± 0%     ~             (p=0.730 n=5+4)
BM_UFlatSink/17 [bin              ]           1.85GB/s ± 0%           1.86GB/s ± 0%   +1.01%          (p=0.008 n=5+5)
BM_UFlatSink/18 [bin_200          ]           1.96GB/s ± 1%           1.94GB/s ± 1%   -1.18%          (p=0.016 n=5+5)
BM_UFlatSink/19 [sum              ]           1.12GB/s ± 0%           1.24GB/s ± 0%  +10.16%          (p=0.008 n=5+5)
BM_UFlatSink/20 [man              ]           1.31GB/s ± 1%           1.32GB/s ± 0%   +0.77%          (p=0.048 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]            839MB/s ± 0%            839MB/s ± 0%     ~             (p=0.413 n=4+5)
BM_ZFlat/1      [urls (47.78 %)   ]            439MB/s ± 0%            439MB/s ± 0%   +0.06%          (p=0.032 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]           11.7GB/s ± 2%           11.5GB/s ± 9%     ~             (p=0.841 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]            645MB/s ± 1%            647MB/s ± 3%     ~             (p=0.413 n=4+5)
BM_ZFlat/4      [pdf (83.30 %)    ]           7.57GB/s ± 1%           7.54GB/s ± 2%     ~             (p=0.595 n=5+5)
BM_ZFlat/5      [html4 (22.52 %)  ]            769MB/s ± 0%            770MB/s ± 0%   +0.08%          (p=0.032 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]            288MB/s ± 0%            288MB/s ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]            267MB/s ± 0%            267MB/s ± 0%     ~             (p=0.690 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]            305MB/s ± 0%            305MB/s ± 0%     ~             (p=0.548 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]            250MB/s ± 0%            251MB/s ± 0%     ~             (p=0.421 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]           1.12GB/s ± 0%           1.12GB/s ± 0%     ~             (p=0.635 n=5+5)
BM_ZFlat/11     [gaviota (37.72 %)]            457MB/s ± 0%            457MB/s ± 0%     ~             (p=0.841 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            570MB/s ± 0%            568MB/s ± 1%     ~             (p=0.151 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            682MB/s ± 1%            681MB/s ± 0%     ~             (p=0.310 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            750MB/s ± 0%            751MB/s ± 1%     ~             (p=0.690 n=5+5)
BM_ZFlat/15     [xls (41.23 %)    ]            668MB/s ± 0%            668MB/s ± 0%     ~             (p=0.841 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]            569MB/s ± 2%            570MB/s ± 1%     ~             (p=0.841 n=5+5)
BM_ZFlat/17     [bin (18.11 %)    ]           1.04GB/s ± 0%           1.04GB/s ± 0%     ~             (p=0.310 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]           2.64GB/s ± 1%           2.59GB/s ± 0%   -1.99%          (p=0.016 n=5+4)
BM_ZFlat/19     [sum (48.96 %)    ]            497MB/s ± 0%            498MB/s ± 0%     ~             (p=0.222 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            615MB/s ± 1%            621MB/s ± 0%   +0.87%          (p=0.008 n=5+5)

K8
--

name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            41.7µs ± 0%             41.7µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/1      [urls             ]             588µs ± 0%              588µs ± 0%    ~             (p=0.310 n=5+5)
BM_UFlat/2      [jpg              ]            7.11µs ± 1%             7.10µs ± 1%    ~             (p=0.556 n=5+4)
BM_UFlat/3      [jpg_200          ]              130ns ± 0%              130ns ± 0%    ~     (all samples are equal)
BM_UFlat/4      [pdf              ]            8.19µs ± 0%             8.26µs ± 2%    ~             (p=0.460 n=5+5)
BM_UFlat/5      [html4            ]             219µs ± 0%              219µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/6      [txt1             ]             192µs ± 0%              191µs ± 0%    ~             (p=0.341 n=5+5)
BM_UFlat/7      [txt2             ]             170µs ± 0%              170µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/8      [txt3             ]             509µs ± 0%              509µs ± 0%    ~             (p=0.151 n=5+5)
BM_UFlat/9      [txt4             ]             712µs ± 0%              712µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/10     [pb               ]            38.5µs ± 0%             38.5µs ± 0%    ~             (p=0.452 n=5+5)
BM_UFlat/11     [gaviota          ]             189µs ± 0%              189µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/12     [cp               ]            14.2µs ± 1%             14.2µs ± 0%    ~             (p=0.889 n=5+5)
BM_UFlat/13     [c                ]            7.32µs ± 0%             7.33µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlat/14     [lsp              ]            2.26µs ± 0%             2.27µs ± 0%    ~             (p=0.222 n=4+5)
BM_UFlat/15     [xls              ]             954µs ± 0%              955µs ± 0%    ~             (p=0.222 n=5+5)
BM_UFlat/16     [xls_200          ]              215ns ± 4%              212ns ± 0%    ~             (p=0.095 n=5+4)
BM_UFlat/17     [bin              ]             276µs ± 0%              276µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/18     [bin_200          ]              104ns ±10%              103ns ± 3%    ~             (p=0.825 n=5+5)
BM_UFlat/19     [sum              ]            29.2µs ± 0%             29.2µs ± 0%    ~             (p=0.690 n=5+5)
BM_UFlat/20     [man              ]            2.96µs ± 0%             2.97µs ± 0%  +0.43%          (p=0.032 n=5+5)
BM_UValidate/0  [html             ]            33.4µs ± 0%             33.4µs ± 0%    ~             (p=0.151 n=5+5)
BM_UValidate/1  [urls             ]             441µs ± 0%              441µs ± 0%    ~             (p=0.548 n=5+5)
BM_UValidate/2  [jpg              ]              146ns ± 0%              146ns ± 0%    ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]             98.0ns ± 0%             98.0ns ± 0%    ~             (p=1.000 n=5+5)
BM_UValidate/4  [pdf              ]            2.89µs ± 0%             2.89µs ± 0%    ~             (p=0.794 n=5+5)
BM_UIOVec/0     [html             ]             121µs ± 0%              121µs ± 0%    ~             (p=0.151 n=5+5)
BM_UIOVec/1     [urls             ]             1.08ms ± 0%             1.08ms ± 0%    ~             (p=0.095 n=5+5)
BM_UIOVec/2     [jpg              ]            7.47µs ± 5%             7.31µs ± 2%    ~             (p=0.222 n=5+5)
BM_UIOVec/3     [jpg_200          ]              330ns ± 0%              330ns ± 0%    ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]            12.3µs ± 2%             12.0µs ± 0%    ~             (p=0.063 n=5+5)
BM_UFlatSink/0  [html             ]            41.6µs ± 0%             41.6µs ± 0%    ~             (p=0.095 n=5+5)
BM_UFlatSink/1  [urls             ]             589µs ± 0%              589µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/2  [jpg              ]            7.84µs ±26%             7.23µs ± 5%    ~             (p=0.690 n=5+5)
BM_UFlatSink/3  [jpg_200          ]              132ns ± 0%              132ns ± 0%    ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]            8.43µs ± 3%             8.27µs ± 2%    ~             (p=0.254 n=5+5)
BM_UFlatSink/5  [html4            ]             219µs ± 0%              219µs ± 0%    ~             (p=0.524 n=5+5)
BM_UFlatSink/6  [txt1             ]             192µs ± 0%              192µs ± 0%    ~             (p=0.690 n=5+5)
BM_UFlatSink/7  [txt2             ]             170µs ± 0%              170µs ± 0%    ~             (p=0.421 n=5+5)
BM_UFlatSink/8  [txt3             ]             509µs ± 0%              509µs ± 0%    ~             (p=0.310 n=5+5)
BM_UFlatSink/9  [txt4             ]             712µs ± 0%              712µs ± 0%    ~             (p=0.841 n=5+5)
BM_UFlatSink/10 [pb               ]            38.5µs ± 0%             38.5µs ± 0%    ~             (p=0.421 n=5+5)
BM_UFlatSink/11 [gaviota          ]             189µs ± 0%              189µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/12 [cp               ]            14.2µs ± 0%             14.2µs ± 0%    ~             (p=0.421 n=5+5)
BM_UFlatSink/13 [c                ]            7.37µs ± 1%             7.36µs ± 1%    ~             (p=0.746 n=5+5)
BM_UFlatSink/14 [lsp              ]            2.27µs ± 0%             2.27µs ± 1%    ~             (p=0.714 n=5+5)
BM_UFlatSink/15 [xls              ]             954µs ± 0%              954µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/16 [xls_200          ]              215ns ± 1%              215ns ± 1%    ~             (p=0.921 n=5+5)
BM_UFlatSink/17 [bin              ]             276µs ± 0%              276µs ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/18 [bin_200          ]              103ns ± 2%              104ns ± 1%    ~             (p=0.429 n=5+5)
BM_UFlatSink/19 [sum              ]            29.2µs ± 0%             29.2µs ± 0%    ~             (p=0.452 n=5+5)
BM_UFlatSink/20 [man              ]            2.96µs ± 0%             2.97µs ± 1%    ~             (p=0.484 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]             126µs ± 0%              126µs ± 0%    ~             (p=1.000 n=5+5)
BM_ZFlat/1      [urls (47.78 %)   ]             1.67ms ± 0%             1.67ms ± 0%    ~             (p=0.841 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]            11.6µs ± 4%             11.6µs ± 3%    ~             (p=1.000 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]              368ns ± 1%              367ns ± 0%    ~             (p=0.159 n=5+5)
BM_ZFlat/4      [pdf (83.30 %)    ]            14.7µs ± 1%             14.6µs ± 0%    ~             (p=0.190 n=5+4)
BM_ZFlat/5      [html4 (22.52 %)  ]             550µs ± 0%              550µs ± 0%    ~             (p=0.841 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]             540µs ± 0%              540µs ± 0%    ~             (p=0.310 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]             479µs ± 0%              480µs ± 0%    ~             (p=1.000 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]             1.44ms ± 0%             1.44ms ± 0%    ~             (p=0.421 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]             1.97ms ± 0%             1.97ms ± 0%    ~             (p=0.421 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]             110µs ± 0%              109µs ± 0%    ~             (p=0.730 n=5+4)
BM_ZFlat/11     [gaviota (37.72 %)]             412µs ± 0%              412µs ± 0%    ~             (p=1.000 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            46.3µs ± 0%             46.3µs ± 1%    ~             (p=0.841 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            17.7µs ± 0%             17.7µs ± 1%    ~             (p=0.841 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            5.54µs ± 1%             5.55µs ± 0%    ~             (p=0.254 n=5+4)
BM_ZFlat/15     [xls (41.23 %)    ]             1.62ms ± 0%             1.63ms ± 0%    ~             (p=0.151 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]              395ns ± 2%              394ns ± 1%    ~             (p=1.000 n=5+5)
BM_ZFlat/17     [bin (18.11 %)    ]             507µs ± 0%              507µs ± 0%    ~             (p=0.056 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]             89.6ns ± 5%             89.8ns ± 5%    ~             (p=1.000 n=5+5)
BM_ZFlat/19     [sum (48.96 %)    ]            79.9µs ± 0%             79.9µs ± 0%    ~             (p=0.690 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            7.67µs ± 0%             7.67µs ± 1%    ~             (p=0.548 n=5+5)

name                                          old speed               new speed               delta
BM_UFlat/0      [html             ]           2.45GB/s ± 0%           2.45GB/s ± 0%    ~             (p=0.889 n=5+5)
BM_UFlat/1      [urls             ]           1.19GB/s ± 0%           1.19GB/s ± 0%    ~     (all samples are equal)
BM_UFlat/2      [jpg              ]           17.3GB/s ± 1%           17.3GB/s ± 1%    ~             (p=0.556 n=5+4)
BM_UFlat/3      [jpg_200          ]           1.54GB/s ± 0%           1.54GB/s ± 0%    ~             (p=0.833 n=5+5)
BM_UFlat/4      [pdf              ]           12.5GB/s ± 0%           12.4GB/s ± 2%    ~             (p=0.421 n=5+5)
BM_UFlat/5      [html4            ]           1.87GB/s ± 0%           1.87GB/s ± 0%    ~             (p=1.000 n=4+5)
BM_UFlat/6      [txt1             ]            794MB/s ± 0%            794MB/s ± 0%    ~             (p=0.310 n=5+5)
BM_UFlat/7      [txt2             ]            738MB/s ± 0%            738MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/8      [txt3             ]            839MB/s ± 0%            838MB/s ± 0%    ~             (p=0.151 n=5+5)
BM_UFlat/9      [txt4             ]            677MB/s ± 0%            677MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/10     [pb               ]           3.08GB/s ± 0%           3.08GB/s ± 0%    ~             (p=0.452 n=5+5)
BM_UFlat/11     [gaviota          ]            975MB/s ± 0%            975MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/12     [cp               ]           1.73GB/s ± 1%           1.73GB/s ± 0%    ~             (p=0.984 n=5+5)
BM_UFlat/13     [c                ]           1.52GB/s ± 0%           1.52GB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlat/14     [lsp              ]           1.64GB/s ± 0%           1.64GB/s ± 0%    ~             (p=0.254 n=4+5)
BM_UFlat/15     [xls              ]           1.08GB/s ± 0%           1.08GB/s ± 0%    ~             (p=0.095 n=5+4)
BM_UFlat/16     [xls_200          ]            931MB/s ± 4%            941MB/s ± 0%    ~             (p=0.151 n=5+5)
BM_UFlat/17     [bin              ]           1.86GB/s ± 0%           1.86GB/s ± 0%    ~             (p=0.762 n=5+5)
BM_UFlat/18     [bin_200          ]           1.92GB/s ± 9%           1.95GB/s ± 3%    ~             (p=1.000 n=5+5)
BM_UFlat/19     [sum              ]           1.31GB/s ± 1%           1.31GB/s ± 0%    ~             (p=0.548 n=5+5)
BM_UFlat/20     [man              ]           1.43GB/s ± 0%           1.42GB/s ± 1%  -0.42%          (p=0.040 n=5+5)
BM_UValidate/0  [html             ]           3.06GB/s ± 0%           3.06GB/s ± 0%    ~             (p=0.151 n=5+5)
BM_UValidate/1  [urls             ]           1.59GB/s ± 0%           1.59GB/s ± 0%    ~             (p=0.357 n=5+5)
BM_UValidate/2  [jpg              ]            845GB/s ± 0%            845GB/s ± 0%    ~             (p=0.548 n=5+5)
BM_UValidate/3  [jpg_200          ]           2.04GB/s ± 0%           2.04GB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UValidate/4  [pdf              ]           35.4GB/s ± 0%           35.4GB/s ± 0%    ~             (p=0.421 n=5+5)
BM_UIOVec/0     [html             ]            845MB/s ± 0%            845MB/s ± 0%    ~             (p=0.151 n=5+5)
BM_UIOVec/1     [urls             ]            650MB/s ± 0%            650MB/s ± 0%    ~             (p=0.087 n=5+5)
BM_UIOVec/2     [jpg              ]           16.5GB/s ± 5%           16.8GB/s ± 2%    ~             (p=0.222 n=5+5)
BM_UIOVec/3     [jpg_200          ]            605MB/s ± 0%            605MB/s ± 0%    ~             (p=0.690 n=5+5)
BM_UIOVec/4     [pdf              ]           8.36GB/s ± 2%           8.54GB/s ± 0%    ~             (p=0.063 n=5+5)
BM_UFlatSink/0  [html             ]           2.46GB/s ± 0%           2.46GB/s ± 0%    ~             (p=0.063 n=5+5)
BM_UFlatSink/1  [urls             ]           1.19GB/s ± 0%           1.19GB/s ± 0%    ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]           16.0GB/s ±22%           17.0GB/s ± 5%    ~             (p=0.690 n=5+5)
BM_UFlatSink/3  [jpg_200          ]           1.51GB/s ± 0%           1.51GB/s ± 2%    ~             (p=1.000 n=5+5)
BM_UFlatSink/4  [pdf              ]           12.2GB/s ± 3%           12.4GB/s ± 2%    ~             (p=0.254 n=5+5)
BM_UFlatSink/5  [html4            ]           1.87GB/s ± 0%           1.87GB/s ± 0%    ~             (p=0.532 n=5+5)
BM_UFlatSink/6  [txt1             ]            794MB/s ± 0%            794MB/s ± 0%    ~             (p=0.690 n=5+5)
BM_UFlatSink/7  [txt2             ]            738MB/s ± 0%            738MB/s ± 0%    ~             (p=0.421 n=5+5)
BM_UFlatSink/8  [txt3             ]            838MB/s ± 0%            838MB/s ± 0%    ~             (p=0.310 n=5+5)
BM_UFlatSink/9  [txt4             ]            676MB/s ± 0%            676MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_UFlatSink/10 [pb               ]           3.08GB/s ± 0%           3.08GB/s ± 0%    ~             (p=0.365 n=5+5)
BM_UFlatSink/11 [gaviota          ]            975MB/s ± 0%            975MB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/12 [cp               ]           1.73GB/s ± 0%           1.74GB/s ± 0%    ~             (p=0.286 n=5+5)
BM_UFlatSink/13 [c                ]           1.51GB/s ± 1%           1.52GB/s ± 1%    ~             (p=0.683 n=5+5)
BM_UFlatSink/14 [lsp              ]           1.64GB/s ± 0%           1.64GB/s ± 0%    ~             (p=0.444 n=5+5)
BM_UFlatSink/15 [xls              ]           1.08GB/s ± 0%           1.08GB/s ± 0%    ~             (p=0.333 n=4+5)
BM_UFlatSink/16 [xls_200          ]            930MB/s ± 1%            930MB/s ± 1%    ~             (p=0.841 n=5+5)
BM_UFlatSink/17 [bin              ]           1.86GB/s ± 0%           1.86GB/s ± 0%    ~             (p=1.000 n=5+5)
BM_UFlatSink/18 [bin_200          ]           1.93GB/s ± 2%           1.93GB/s ± 1%    ~             (p=0.651 n=5+5)
BM_UFlatSink/19 [sum              ]           1.31GB/s ± 0%           1.31GB/s ± 0%    ~             (p=0.508 n=5+5)
BM_UFlatSink/20 [man              ]           1.43GB/s ± 0%           1.42GB/s ± 1%    ~             (p=0.524 n=5+5)
BM_ZFlat/0      [html (22.31 %)   ]            815MB/s ± 0%            815MB/s ± 0%    ~             (p=1.000 n=5+5)
BM_ZFlat/1      [urls (47.78 %)   ]            420MB/s ± 0%            420MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_ZFlat/2      [jpg (99.95 %)    ]           10.6GB/s ± 4%           10.6GB/s ± 3%    ~             (p=1.000 n=5+5)
BM_ZFlat/3      [jpg_200 (73.00 %)]            543MB/s ± 1%            546MB/s ± 0%    ~             (p=0.095 n=5+5)
BM_ZFlat/4      [pdf (83.30 %)    ]           6.96GB/s ± 1%           7.01GB/s ± 0%    ~             (p=0.190 n=5+4)
BM_ZFlat/5      [html4 (22.52 %)  ]            745MB/s ± 0%            745MB/s ± 0%    ~             (p=0.841 n=5+5)
BM_ZFlat/6      [txt1 (57.88 %)   ]            282MB/s ± 0%            282MB/s ± 0%    ~             (p=0.310 n=5+5)
BM_ZFlat/7      [txt2 (61.91 %)   ]            261MB/s ± 0%            261MB/s ± 0%    ~             (p=1.000 n=5+5)
BM_ZFlat/8      [txt3 (54.99 %)   ]            297MB/s ± 0%            297MB/s ± 0%    ~             (p=0.421 n=5+5)
BM_ZFlat/9      [txt4 (66.26 %)   ]            244MB/s ± 0%            244MB/s ± 0%    ~             (p=0.389 n=5+5)
BM_ZFlat/10     [pb (19.68 %)     ]           1.08GB/s ± 0%           1.08GB/s ± 0%    ~             (p=0.238 n=5+4)
BM_ZFlat/11     [gaviota (37.72 %)]            448MB/s ± 0%            447MB/s ± 0%    ~             (p=1.000 n=5+5)
BM_ZFlat/12     [cp (48.12 %)     ]            532MB/s ± 0%            531MB/s ± 1%    ~             (p=0.841 n=5+5)
BM_ZFlat/13     [c (42.47 %)      ]            632MB/s ± 0%            631MB/s ± 1%    ~             (p=0.841 n=5+5)
BM_ZFlat/14     [lsp (48.37 %)    ]            672MB/s ± 1%            671MB/s ± 0%    ~             (p=0.286 n=5+4)
BM_ZFlat/15     [xls (41.23 %)    ]            634MB/s ± 0%            633MB/s ± 0%    ~             (p=0.151 n=5+5)
BM_ZFlat/16     [xls_200 (78.00 %)]            507MB/s ± 2%            508MB/s ± 1%    ~             (p=1.000 n=5+5)
BM_ZFlat/17     [bin (18.11 %)    ]           1.01GB/s ± 0%           1.01GB/s ± 0%    ~             (p=0.056 n=5+5)
BM_ZFlat/18     [bin_200 (7.50 %) ]           2.24GB/s ± 5%           2.23GB/s ± 5%    ~             (p=0.889 n=5+5)
BM_ZFlat/19     [sum (48.96 %)    ]            479MB/s ± 0%            479MB/s ± 0%    ~             (p=0.690 n=5+5)
BM_ZFlat/20     [man (59.21 %)    ]            551MB/s ± 0%            551MB/s ± 1%    ~             (p=0.548 n=5+5)
2019-01-04 19:08:39 -08:00
nafi eb47f79631 Optimize by about 0.5%.
How? Move boolean args of EmitLiteral, EmitCopyAtMost64 and EmitCopy to
template args so that compiler generates two separate pruned versions of
the functions for arg=true and arg=false. FWIW, CompressFragment
function calls 1) EmitLiteral inside from a 1-level loop and 2) EmitCopy
from a 2-level nested loop. CompressFragment is called from inside
another while-loop from the public 'Compress' function.

name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            41.9µs ± 0%             41.1µs ± 0%  -1.92%        (p=0.000 n=10+10)
BM_UFlat/1      [urls             ]             576µs ± 0%              572µs ± 0%  -0.68%        (p=0.000 n=10+10)
BM_UFlat/2      [jpg              ]            7.25µs ± 6%             7.13µs ± 1%    ~             (p=0.074 n=9+8)
BM_UFlat/3      [jpg_200          ]              132ns ± 1%              130ns ± 0%  -1.45%         (p=0.000 n=10+8)
BM_UFlat/4      [pdf              ]            8.27µs ± 3%             8.22µs ± 0%    ~             (p=0.277 n=9+8)
BM_UFlat/5      [html4            ]             220µs ± 0%              219µs ± 0%  -0.75%        (p=0.000 n=10+10)
BM_UFlat/6      [txt1             ]             192µs ± 0%              190µs ± 0%  -0.80%        (p=0.000 n=10+10)
BM_UFlat/7      [txt2             ]             169µs ± 0%              168µs ± 0%  -0.69%        (p=0.000 n=10+10)
BM_UFlat/8      [txt3             ]             510µs ± 0%              508µs ± 0%  -0.42%        (p=0.000 n=10+10)
BM_UFlat/9      [txt4             ]             707µs ± 0%              702µs ± 0%  -0.67%        (p=0.000 n=10+10)
BM_UFlat/10     [pb               ]            38.5µs ± 0%             37.4µs ± 1%  -2.84%        (p=0.000 n=10+10)
BM_UFlat/11     [gaviota          ]             189µs ± 0%              190µs ± 0%  +0.55%        (p=0.000 n=10+10)
BM_UFlat/12     [cp               ]            14.2µs ± 0%             14.1µs ± 0%  -0.44%        (p=0.000 n=10+10)
BM_UFlat/13     [c                ]            7.31µs ± 1%             7.35µs ± 0%  +0.54%        (p=0.002 n=10+10)
BM_UFlat/14     [lsp              ]            2.27µs ± 0%             2.27µs ± 1%    ~             (p=0.161 n=9+9)
BM_UFlat/15     [xls              ]             905µs ± 0%              903µs ± 0%  -0.25%        (p=0.000 n=10+10)
BM_UFlat/16     [xls_200          ]              214ns ± 1%              213ns ± 1%  -0.57%        (p=0.043 n=10+10)
BM_UFlat/17     [bin              ]             275µs ± 0%              274µs ± 0%  -0.31%        (p=0.000 n=10+10)
BM_UFlat/18     [bin_200          ]              102ns ± 5%              101ns ± 3%    ~             (p=0.161 n=9+9)
BM_UFlat/19     [sum              ]            27.9µs ± 0%             27.2µs ± 0%  -2.68%        (p=0.000 n=10+10)
BM_UFlat/20     [man              ]            2.97µs ± 1%             2.97µs ± 0%    ~            (p=0.400 n=9+10)
BM_UValidate/0  [html             ]            33.3µs ± 0%             33.7µs ± 0%  +1.18%        (p=0.000 n=10+10)
BM_UValidate/1  [urls             ]             442µs ± 0%              442µs ± 0%    ~           (p=0.353 n=10+10)
BM_UValidate/2  [jpg              ]              146ns ± 0%              146ns ± 0%    ~           (p=0.063 n=10+10)
BM_UValidate/3  [jpg_200          ]             98.4ns ± 0%             98.5ns ± 0%    ~           (p=0.184 n=10+10)
BM_UValidate/4  [pdf              ]            2.88µs ± 0%             2.90µs ± 1%  +0.68%        (p=0.000 n=10+10)
BM_UIOVec/0     [html             ]             122µs ± 0%              122µs ± 0%  -0.39%        (p=0.000 n=10+10)
BM_UIOVec/1     [urls             ]             1.08ms ± 0%             1.08ms ± 0%    ~           (p=0.529 n=10+10)
BM_UIOVec/2     [jpg              ]            7.71µs ±11%             7.76µs ± 9%    ~           (p=0.853 n=10+10)
BM_UIOVec/3     [jpg_200          ]              327ns ± 0%              328ns ± 0%    ~            (p=0.146 n=8+10)
BM_UIOVec/4     [pdf              ]            12.1µs ± 1%             12.1µs ± 3%    ~           (p=0.315 n=10+10)
BM_UFlatSink/0  [html             ]            41.8µs ± 0%             41.0µs ± 0%  -1.87%         (p=0.000 n=10+9)
BM_UFlatSink/1  [urls             ]             576µs ± 0%              572µs ± 0%  -0.74%         (p=0.000 n=9+10)
BM_UFlatSink/2  [jpg              ]            7.58µs ± 8%             7.56µs ± 9%    ~           (p=0.739 n=10+10)
BM_UFlatSink/3  [jpg_200          ]              133ns ± 0%              134ns ± 0%  +0.60%         (p=0.000 n=10+9)
BM_UFlatSink/4  [pdf              ]            8.44µs ± 3%             8.30µs ± 1%  -1.65%        (p=0.029 n=10+10)
BM_UFlatSink/5  [html4            ]             220µs ± 0%              218µs ± 0%  -0.81%        (p=0.000 n=10+10)
BM_UFlatSink/6  [txt1             ]             192µs ± 0%              190µs ± 0%  -0.78%        (p=0.000 n=10+10)
BM_UFlatSink/7  [txt2             ]             169µs ± 0%              168µs ± 0%  -0.59%        (p=0.000 n=10+10)
BM_UFlatSink/8  [txt3             ]             510µs ± 0%              508µs ± 0%  -0.39%        (p=0.000 n=10+10)
BM_UFlatSink/9  [txt4             ]             707µs ± 0%              703µs ± 0%  -0.62%        (p=0.000 n=10+10)
BM_UFlatSink/10 [pb               ]            38.4µs ± 0%             37.4µs ± 0%  -2.62%          (p=0.000 n=9+9)
BM_UFlatSink/11 [gaviota          ]             189µs ± 0%              190µs ± 0%  +0.63%        (p=0.000 n=10+10)
BM_UFlatSink/12 [cp               ]            14.2µs ± 0%             14.1µs ± 0%  -0.27%        (p=0.011 n=10+10)
BM_UFlatSink/13 [c                ]            7.33µs ± 1%             7.35µs ± 1%    ~            (p=0.243 n=10+9)
BM_UFlatSink/14 [lsp              ]            2.27µs ± 0%             2.26µs ± 0%  -0.39%          (p=0.000 n=9+9)
BM_UFlatSink/15 [xls              ]             904µs ± 0%              902µs ± 0%  -0.28%        (p=0.000 n=10+10)
BM_UFlatSink/16 [xls_200          ]              216ns ± 1%              217ns ± 1%    ~            (p=0.661 n=10+9)
BM_UFlatSink/17 [bin              ]             275µs ± 0%              274µs ± 0%  -0.24%          (p=0.000 n=8+9)
BM_UFlatSink/18 [bin_200          ]              104ns ± 2%              104ns ± 1%  -0.70%         (p=0.043 n=9+10)
BM_UFlatSink/19 [sum              ]            27.8µs ± 0%             27.1µs ± 0%  -2.51%         (p=0.000 n=9+10)
BM_UFlatSink/20 [man              ]            3.02µs ± 1%             3.00µs ± 1%    ~            (p=0.079 n=10+9)
BM_ZFlat/0      [html (22.31 %)   ]             126µs ± 0%              126µs ± 0%  -0.24%        (p=0.000 n=10+10)
BM_ZFlat/1      [urls (47.78 %)   ]             1.68ms ± 0%             1.67ms ± 0%  -1.06%        (p=0.000 n=10+10)
BM_ZFlat/2      [jpg (99.95 %)    ]            11.8µs ± 5%             11.6µs ± 5%    ~           (p=0.165 n=10+10)
BM_ZFlat/3      [jpg_200 (73.00 %)]              360ns ± 3%              358ns ± 1%    ~            (p=0.762 n=10+8)
BM_ZFlat/4      [pdf (83.30 %)    ]            14.8µs ± 2%             14.6µs ± 1%  -1.57%         (p=0.022 n=10+9)
BM_ZFlat/5      [html4 (22.52 %)  ]             556µs ± 0%              552µs ± 0%  -0.87%        (p=0.000 n=10+10)
BM_ZFlat/6      [txt1 (57.88 %)   ]             542µs ± 0%              540µs ± 0%  -0.47%        (p=0.000 n=10+10)
BM_ZFlat/7      [txt2 (61.91 %)   ]             483µs ± 0%              480µs ± 0%  -0.62%        (p=0.000 n=10+10)
BM_ZFlat/8      [txt3 (54.99 %)   ]             1.45ms ± 0%             1.44ms ± 0%  -0.47%        (p=0.000 n=10+10)
BM_ZFlat/9      [txt4 (66.26 %)   ]             1.98ms ± 0%             1.97ms ± 0%  -0.19%        (p=0.007 n=10+10)
BM_ZFlat/10     [pb (19.68 %)     ]             111µs ± 0%              109µs ± 0%  -1.75%        (p=0.000 n=10+10)
BM_ZFlat/11     [gaviota (37.72 %)]             411µs ± 0%              410µs ± 0%  -0.21%        (p=0.004 n=10+10)
BM_ZFlat/12     [cp (48.12 %)     ]            45.9µs ± 0%             45.5µs ± 0%  -0.76%        (p=0.000 n=10+10)
BM_ZFlat/13     [c (42.47 %)      ]            17.6µs ± 0%             17.5µs ± 0%  -0.80%        (p=0.000 n=10+10)
BM_ZFlat/14     [lsp (48.37 %)    ]            5.50µs ± 0%             5.44µs ± 0%  -1.19%         (p=0.000 n=9+10)
BM_ZFlat/15     [xls (41.23 %)    ]             1.63ms ± 0%             1.61ms ± 0%  -1.21%        (p=0.000 n=10+10)
BM_ZFlat/16     [xls_200 (78.00 %)]              389ns ± 2%              391ns ± 1%    ~            (p=0.182 n=10+9)
BM_ZFlat/17     [bin (18.11 %)    ]             509µs ± 0%              506µs ± 0%  -0.51%        (p=0.000 n=10+10)
BM_ZFlat/18     [bin_200 (7.50 %) ]             92.7ns ± 0%             89.4ns ± 1%  -3.55%          (p=0.000 n=8+8)
BM_ZFlat/19     [sum (48.96 %)    ]            80.2µs ± 0%             78.9µs ± 0%  -1.65%        (p=0.000 n=10+10)
BM_ZFlat/20     [man (59.21 %)    ]            7.59µs ± 1%             7.59µs ± 1%    ~           (p=0.912 n=10+10)

name                                          old allocs/op           new allocs/op           delta
BM_UFlat/0      [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/1      [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/2      [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/3      [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/4      [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/5      [html4            ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/6      [txt1             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/7      [txt2             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/8      [txt3             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/9      [txt4             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/10     [pb               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/11     [gaviota          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/12     [cp               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/13     [c                ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/14     [lsp              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/15     [xls              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/16     [xls_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/17     [bin              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/18     [bin_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/19     [sum              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlat/20     [man              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/0  [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/1  [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/2  [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UValidate/4  [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/0     [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/1     [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/2     [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/3     [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/0  [html             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/1  [urls             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/3  [jpg_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/5  [html4            ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/6  [txt1             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/7  [txt2             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/8  [txt3             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/9  [txt4             ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/10 [pb               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/11 [gaviota          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/12 [cp               ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/13 [c                ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/14 [lsp              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/15 [xls              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/16 [xls_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/17 [bin              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/18 [bin_200          ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/19 [sum              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_UFlatSink/20 [man              ]               0.00 ±NaN%              0.00 ±NaN%    ~     (all samples are equal)
BM_ZFlat/0      [html (22.31 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/1      [urls (47.78 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/2      [jpg (99.95 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/3      [jpg_200 (73.00 %)]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/4      [pdf (83.30 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/5      [html4 (22.52 %)  ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/6      [txt1 (57.88 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/7      [txt2 (61.91 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/8      [txt3 (54.99 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/9      [txt4 (66.26 %)   ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/10     [pb (19.68 %)     ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/11     [gaviota (37.72 %)]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/12     [cp (48.12 %)     ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/13     [c (42.47 %)      ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/14     [lsp (48.37 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/15     [xls (41.23 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/16     [xls_200 (78.00 %)]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/17     [bin (18.11 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/18     [bin_200 (7.50 %) ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/19     [sum (48.96 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)
BM_ZFlat/20     [man (59.21 %)    ]               1.00 ± 0%               1.00 ± 0%    ~     (all samples are equal)

name                                          old peak-mem(Bytes)/op  new peak-mem(Bytes)/op  delta
BM_UFlat/0      [html             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/1      [urls             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/2      [jpg              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/3      [jpg_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/4      [pdf              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/5      [html4            ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/6      [txt1             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/7      [txt2             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/8      [txt3             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/9      [txt4             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/10     [pb               ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/11     [gaviota          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/12     [cp               ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/13     [c                ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/14     [lsp              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/15     [xls              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/16     [xls_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/17     [bin              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/18     [bin_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/19     [sum              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlat/20     [man              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/0  [html             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/1  [urls             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/2  [jpg              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/3  [jpg_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UValidate/4  [pdf              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/0     [html             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/1     [urls             ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/2     [jpg              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/3     [jpg_200          ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UIOVec/4     [pdf              ]               4.00 ± 0%               4.00 ± 0%    ~     (all samples are equal)
BM_UFlatSink/0  [html             ]               102k ± 0%               102k ± 0%    ~     (all samples are equal)
BM_UFlatSink/1  [urls             ]               702k ± 0%               702k ± 0%    ~     (all samples are equal)
BM_UFlatSink/2  [jpg              ]               123k ± 0%               123k ± 0%    ~     (all samples are equal)
BM_UFlatSink/3  [jpg_200          ]                201 ± 0%                201 ± 0%    ~     (all samples are equal)
BM_UFlatSink/4  [pdf              ]               102k ± 0%               102k ± 0%    ~     (all samples are equal)
BM_UFlatSink/5  [html4            ]               410k ± 0%               410k ± 0%    ~     (all samples are equal)
BM_UFlatSink/6  [txt1             ]               152k ± 0%               152k ± 0%    ~     (all samples are equal)
BM_UFlatSink/7  [txt2             ]               125k ± 0%               125k ± 0%    ~     (all samples are equal)
BM_UFlatSink/8  [txt3             ]               427k ± 0%               427k ± 0%    ~     (all samples are equal)
BM_UFlatSink/9  [txt4             ]               482k ± 0%               482k ± 0%    ~     (all samples are equal)
BM_UFlatSink/10 [pb               ]               119k ± 0%               119k ± 0%    ~     (all samples are equal)
BM_UFlatSink/11 [gaviota          ]               184k ± 0%               184k ± 0%    ~     (all samples are equal)
BM_UFlatSink/12 [cp               ]              24.6k ± 0%              24.6k ± 0%    ~     (all samples are equal)
BM_UFlatSink/13 [c                ]              11.2k ± 0%              11.2k ± 0%    ~     (all samples are equal)
BM_UFlatSink/14 [lsp              ]              3.72k ± 0%              3.72k ± 0%    ~     (all samples are equal)
BM_UFlatSink/15 [xls              ]              1.03M ± 0%              1.03M ± 0%    ~     (all samples are equal)
BM_UFlatSink/16 [xls_200          ]                201 ± 0%                201 ± 0%    ~     (all samples are equal)
BM_UFlatSink/17 [bin              ]               513k ± 0%               513k ± 0%    ~     (all samples are equal)
BM_UFlatSink/18 [bin_200          ]                201 ± 0%                201 ± 0%    ~     (all samples are equal)
BM_UFlatSink/19 [sum              ]              38.2k ± 0%              38.2k ± 0%    ~     (all samples are equal)
BM_UFlatSink/20 [man              ]              4.23k ± 0%              4.23k ± 0%    ~     (all samples are equal)
BM_ZFlat/0      [html (22.31 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/1      [urls (47.78 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/2      [jpg (99.95 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/3      [jpg_200 (73.00 %)]              63.3k ± 0%              63.3k ± 0%    ~     (all samples are equal)
BM_ZFlat/4      [pdf (83.30 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/5      [html4 (22.52 %)  ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/6      [txt1 (57.88 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/7      [txt2 (61.91 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/8      [txt3 (54.99 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/9      [txt4 (66.26 %)   ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/10     [pb (19.68 %)     ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/11     [gaviota (37.72 %)]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/12     [cp (48.12 %)     ]              86.1k ± 0%              86.1k ± 0%    ~     (all samples are equal)
BM_ZFlat/13     [c (42.47 %)      ]              63.3k ± 0%              63.3k ± 0%    ~     (all samples are equal)
BM_ZFlat/14     [lsp (48.37 %)    ]              63.3k ± 0%              63.3k ± 0%    ~     (all samples are equal)
BM_ZFlat/15     [xls (41.23 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/16     [xls_200 (78.00 %)]              63.3k ± 0%              63.3k ± 0%    ~     (all samples are equal)
BM_ZFlat/17     [bin (18.11 %)    ]               175k ± 0%               175k ± 0%    ~     (all samples are equal)
BM_ZFlat/18     [bin_200 (7.50 %) ]              63.3k ± 0%              63.3k ± 0%    ~     (all samples are equal)
BM_ZFlat/19     [sum (48.96 %)    ]               116k ± 0%               116k ± 0%    ~     (all samples are equal)
BM_ZFlat/20     [man (59.21 %)    ]              63.3k ± 0%              63.3k ± 0%    ~     (all samples are equal)

name                                          old speed               new speed               delta
BM_UFlat/0      [html             ]           2.45GB/s ± 0%           2.50GB/s ± 0%  +1.96%        (p=0.000 n=10+10)
BM_UFlat/1      [urls             ]           1.22GB/s ± 0%           1.23GB/s ± 0%  +0.69%        (p=0.000 n=10+10)
BM_UFlat/2      [jpg              ]           17.0GB/s ± 5%           17.3GB/s ± 1%    ~             (p=0.074 n=9+8)
BM_UFlat/3      [jpg_200          ]           1.52GB/s ± 1%           1.54GB/s ± 0%  +1.44%         (p=0.000 n=10+8)
BM_UFlat/4      [pdf              ]           12.5GB/s ± 1%           12.5GB/s ± 0%    ~             (p=0.721 n=8+8)
BM_UFlat/5      [html4            ]           1.87GB/s ± 0%           1.88GB/s ± 0%  +0.76%        (p=0.000 n=10+10)
BM_UFlat/6      [txt1             ]            795MB/s ± 0%            801MB/s ± 0%  +0.79%        (p=0.000 n=10+10)
BM_UFlat/7      [txt2             ]            741MB/s ± 0%            746MB/s ± 0%  +0.68%        (p=0.000 n=10+10)
BM_UFlat/8      [txt3             ]            840MB/s ± 0%            844MB/s ± 0%  +0.44%        (p=0.000 n=10+10)
BM_UFlat/9      [txt4             ]            684MB/s ± 0%            688MB/s ± 0%  +0.65%         (p=0.000 n=9+10)
BM_UFlat/10     [pb               ]           3.09GB/s ± 0%           3.18GB/s ± 0%  +2.88%         (p=0.000 n=10+9)
BM_UFlat/11     [gaviota          ]            980MB/s ± 0%            975MB/s ± 0%  -0.57%        (p=0.000 n=10+10)
BM_UFlat/12     [cp               ]           1.74GB/s ± 0%           1.75GB/s ± 0%  +0.38%         (p=0.001 n=10+9)
BM_UFlat/13     [c                ]           1.53GB/s ± 1%           1.52GB/s ± 0%  -0.55%        (p=0.003 n=10+10)
BM_UFlat/14     [lsp              ]           1.64GB/s ± 0%           1.64GB/s ± 1%    ~            (p=0.400 n=9+10)
BM_UFlat/15     [xls              ]           1.14GB/s ± 0%           1.14GB/s ± 0%  +0.23%        (p=0.000 n=10+10)
BM_UFlat/16     [xls_200          ]            936MB/s ± 1%            941MB/s ± 1%    ~           (p=0.052 n=10+10)
BM_UFlat/17     [bin              ]           1.87GB/s ± 0%           1.88GB/s ± 0%  +0.28%        (p=0.000 n=10+10)
BM_UFlat/18     [bin_200          ]           1.97GB/s ± 5%           1.99GB/s ± 3%    ~             (p=0.136 n=9+9)
BM_UFlat/19     [sum              ]           1.37GB/s ± 0%           1.41GB/s ± 0%  +2.82%         (p=0.000 n=10+9)
BM_UFlat/20     [man              ]           1.42GB/s ± 1%           1.42GB/s ± 0%    ~           (p=0.579 n=10+10)
BM_UValidate/0  [html             ]           3.08GB/s ± 0%           3.05GB/s ± 0%  -1.18%        (p=0.000 n=10+10)
BM_UValidate/1  [urls             ]           1.59GB/s ± 0%           1.59GB/s ± 0%    ~           (p=0.247 n=10+10)
BM_UValidate/2  [jpg              ]            845GB/s ± 0%            846GB/s ± 0%  +0.09%        (p=0.000 n=10+10)
BM_UValidate/3  [jpg_200          ]           2.04GB/s ± 0%           2.04GB/s ± 0%  -0.09%        (p=0.019 n=10+10)
BM_UValidate/4  [pdf              ]           35.7GB/s ± 0%           35.4GB/s ± 1%  -0.70%        (p=0.000 n=10+10)
BM_UIOVec/0     [html             ]            841MB/s ± 0%            844MB/s ± 0%  +0.36%        (p=0.000 n=10+10)
BM_UIOVec/1     [urls             ]            650MB/s ± 0%            650MB/s ± 0%    ~           (p=0.105 n=10+10)
BM_UIOVec/2     [jpg              ]           16.1GB/s ±10%           15.9GB/s ± 8%    ~           (p=0.853 n=10+10)
BM_UIOVec/3     [jpg_200          ]            612MB/s ± 1%            612MB/s ± 0%    ~            (p=0.243 n=9+10)
BM_UIOVec/4     [pdf              ]           8.52GB/s ± 2%           8.46GB/s ± 3%    ~           (p=0.436 n=10+10)
BM_UFlatSink/0  [html             ]           2.46GB/s ± 0%           2.50GB/s ± 0%  +1.83%         (p=0.000 n=9+10)
BM_UFlatSink/1  [urls             ]           1.22GB/s ± 0%           1.23GB/s ± 0%  +0.73%        (p=0.000 n=10+10)
BM_UFlatSink/2  [jpg              ]           16.3GB/s ± 8%           16.4GB/s ± 9%    ~           (p=0.739 n=10+10)
BM_UFlatSink/3  [jpg_200          ]           1.51GB/s ± 0%           1.50GB/s ± 0%  -0.62%         (p=0.000 n=10+9)
BM_UFlatSink/4  [pdf              ]           12.2GB/s ± 3%           12.4GB/s ± 1%  +1.62%        (p=0.029 n=10+10)
BM_UFlatSink/5  [html4            ]           1.87GB/s ± 0%           1.88GB/s ± 0%  +0.79%        (p=0.000 n=10+10)
BM_UFlatSink/6  [txt1             ]            795MB/s ± 0%            801MB/s ± 0%  +0.74%         (p=0.000 n=10+9)
BM_UFlatSink/7  [txt2             ]            741MB/s ± 0%            745MB/s ± 0%  +0.59%         (p=0.000 n=10+9)
BM_UFlatSink/8  [txt3             ]            840MB/s ± 0%            843MB/s ± 0%  +0.37%         (p=0.000 n=9+10)
BM_UFlatSink/9  [txt4             ]            684MB/s ± 0%            688MB/s ± 0%  +0.57%         (p=0.000 n=9+10)
BM_UFlatSink/10 [pb               ]           3.10GB/s ± 0%           3.18GB/s ± 0%  +2.64%         (p=0.000 n=9+10)
BM_UFlatSink/11 [gaviota          ]            980MB/s ± 0%            974MB/s ± 0%  -0.64%        (p=0.000 n=10+10)
BM_UFlatSink/12 [cp               ]           1.74GB/s ± 0%           1.75GB/s ± 0%  +0.26%        (p=0.005 n=10+10)
BM_UFlatSink/13 [c                ]           1.52GB/s ± 1%           1.52GB/s ± 1%    ~           (p=0.123 n=10+10)
BM_UFlatSink/14 [lsp              ]           1.64GB/s ± 0%           1.65GB/s ± 0%  +0.46%         (p=0.000 n=10+8)
BM_UFlatSink/15 [xls              ]           1.14GB/s ± 0%           1.15GB/s ± 0%  +0.27%        (p=0.000 n=10+10)
BM_UFlatSink/16 [xls_200          ]            927MB/s ± 1%            926MB/s ± 1%    ~            (p=0.497 n=10+9)
BM_UFlatSink/17 [bin              ]           1.87GB/s ± 0%           1.88GB/s ± 0%  +0.27%        (p=0.000 n=10+10)
BM_UFlatSink/18 [bin_200          ]           1.92GB/s ± 2%           1.93GB/s ± 1%  +0.70%         (p=0.035 n=9+10)
BM_UFlatSink/19 [sum              ]           1.38GB/s ± 0%           1.41GB/s ± 0%  +2.59%         (p=0.000 n=9+10)
BM_UFlatSink/20 [man              ]           1.40GB/s ± 1%           1.41GB/s ± 1%    ~            (p=0.079 n=10+9)
BM_ZFlat/0      [html (22.31 %)   ]            814MB/s ± 0%            816MB/s ± 0%  +0.23%        (p=0.000 n=10+10)
BM_ZFlat/1      [urls (47.78 %)   ]            418MB/s ± 0%            423MB/s ± 0%  +1.06%        (p=0.000 n=10+10)
BM_ZFlat/2      [jpg (99.95 %)    ]           10.5GB/s ± 5%           10.7GB/s ± 5%    ~           (p=0.165 n=10+10)
BM_ZFlat/3      [jpg_200 (73.00 %)]            558MB/s ± 3%            560MB/s ± 1%    ~            (p=0.696 n=10+8)
BM_ZFlat/4      [pdf (83.30 %)    ]           6.94GB/s ± 2%           7.05GB/s ± 1%  +1.59%         (p=0.028 n=10+9)
BM_ZFlat/5      [html4 (22.52 %)  ]            739MB/s ± 0%            745MB/s ± 0%  +0.86%        (p=0.000 n=10+10)
BM_ZFlat/6      [txt1 (57.88 %)   ]            281MB/s ± 0%            283MB/s ± 0%  +0.46%        (p=0.000 n=10+10)
BM_ZFlat/7      [txt2 (61.91 %)   ]            260MB/s ± 0%            261MB/s ± 0%  +0.59%        (p=0.000 n=10+10)
BM_ZFlat/8      [txt3 (54.99 %)   ]            296MB/s ± 0%            297MB/s ± 0%  +0.45%        (p=0.000 n=10+10)
BM_ZFlat/9      [txt4 (66.26 %)   ]            244MB/s ± 0%            245MB/s ± 0%  +0.16%        (p=0.000 n=10+10)
BM_ZFlat/10     [pb (19.68 %)     ]           1.07GB/s ± 0%           1.09GB/s ± 0%  +1.75%        (p=0.000 n=10+10)
BM_ZFlat/11     [gaviota (37.72 %)]            450MB/s ± 0%            451MB/s ± 0%  +0.17%         (p=0.000 n=9+10)
BM_ZFlat/12     [cp (48.12 %)     ]            538MB/s ± 0%            542MB/s ± 0%  +0.74%        (p=0.000 n=10+10)
BM_ZFlat/13     [c (42.47 %)      ]            635MB/s ± 0%            640MB/s ± 0%  +0.80%        (p=0.000 n=10+10)
BM_ZFlat/14     [lsp (48.37 %)    ]            678MB/s ± 0%            686MB/s ± 1%  +1.18%         (p=0.000 n=9+10)
BM_ZFlat/15     [xls (41.23 %)    ]            633MB/s ± 0%            641MB/s ± 0%  +1.23%         (p=0.000 n=10+7)
BM_ZFlat/16     [xls_200 (78.00 %)]            516MB/s ± 2%            513MB/s ± 1%    ~            (p=0.156 n=10+9)
BM_ZFlat/17     [bin (18.11 %)    ]           1.01GB/s ± 0%           1.02GB/s ± 0%  +0.49%        (p=0.000 n=10+10)
BM_ZFlat/18     [bin_200 (7.50 %) ]           2.16GB/s ± 0%           2.24GB/s ± 1%  +3.65%          (p=0.000 n=8+8)
BM_ZFlat/19     [sum (48.96 %)    ]            478MB/s ± 0%            486MB/s ± 0%  +1.66%        (p=0.000 n=10+10)
BM_ZFlat/20     [man (59.21 %)    ]            558MB/s ± 1%            558MB/s ± 1%    ~           (p=0.912 n=10+10)
2019-01-04 19:08:30 -08:00
alkis 53a38e5e33 Reduce number of allocations when compressing and simplify the code.
Before we were allocating at least once: twice with large table and
thrice when we used a scratch buffer. With this approach we always
allocate once.

  name                                          old speed               new speed               delta
  BM_UFlat/0      [html             ]           2.45GB/s ± 0%           2.45GB/s ± 0%   -0.13%        (p=0.000 n=11+11)
  BM_UFlat/1      [urls             ]           1.19GB/s ± 0%           1.22GB/s ± 0%   +2.48%        (p=0.000 n=11+11)
  BM_UFlat/2      [jpg              ]           17.2GB/s ± 2%           17.3GB/s ± 1%     ~           (p=0.193 n=11+11)
  BM_UFlat/3      [jpg_200          ]           1.52GB/s ± 0%           1.51GB/s ± 0%   -0.78%         (p=0.000 n=10+9)
  BM_UFlat/4      [pdf              ]           12.5GB/s ± 1%           12.5GB/s ± 1%     ~             (p=0.881 n=9+9)
  BM_UFlat/5      [html4            ]           1.86GB/s ± 0%           1.86GB/s ± 0%     ~           (p=0.123 n=11+11)
  BM_UFlat/6      [txt1             ]            793MB/s ± 0%            799MB/s ± 0%   +0.78%         (p=0.000 n=11+9)
  BM_UFlat/7      [txt2             ]            739MB/s ± 0%            744MB/s ± 0%   +0.77%        (p=0.000 n=11+11)
  BM_UFlat/8      [txt3             ]            839MB/s ± 0%            845MB/s ± 0%   +0.71%        (p=0.000 n=11+11)
  BM_UFlat/9      [txt4             ]            678MB/s ± 0%            685MB/s ± 0%   +1.01%        (p=0.000 n=11+11)
  BM_UFlat/10     [pb               ]           3.08GB/s ± 0%           3.12GB/s ± 0%   +1.21%        (p=0.000 n=11+11)
  BM_UFlat/11     [gaviota          ]            975MB/s ± 0%            976MB/s ± 0%   +0.11%        (p=0.000 n=11+11)
  BM_UFlat/12     [cp               ]           1.73GB/s ± 1%           1.74GB/s ± 1%   +0.46%        (p=0.010 n=11+11)
  BM_UFlat/13     [c                ]           1.53GB/s ± 0%           1.53GB/s ± 0%     ~           (p=0.987 n=11+10)
  BM_UFlat/14     [lsp              ]           1.65GB/s ± 0%           1.63GB/s ± 1%   -1.04%        (p=0.000 n=11+11)
  BM_UFlat/15     [xls              ]           1.08GB/s ± 0%           1.15GB/s ± 0%   +6.12%        (p=0.000 n=10+11)
  BM_UFlat/16     [xls_200          ]            944MB/s ± 0%            920MB/s ± 3%   -2.51%         (p=0.000 n=9+11)
  BM_UFlat/17     [bin              ]           1.86GB/s ± 0%           1.87GB/s ± 0%   +0.68%        (p=0.000 n=10+11)
  BM_UFlat/18     [bin_200          ]           1.91GB/s ± 3%           1.92GB/s ± 5%     ~           (p=0.356 n=11+11)
  BM_UFlat/19     [sum              ]           1.31GB/s ± 0%           1.40GB/s ± 0%   +6.53%        (p=0.000 n=11+11)
  BM_UFlat/20     [man              ]           1.42GB/s ± 0%           1.42GB/s ± 0%   +0.33%        (p=0.000 n=10+10)
2019-01-04 19:07:49 -08:00
ckennelly df5548c0b3 Use sized deallocation when releasing Zippy's scratch buffers.
name                                          old time/op             new time/op             delta
BM_UFlat/0      [html             ]            41.7µs ± 0%             41.7µs ± 0%    ~             (p=0.222 n=5+5)
BM_UFlat/1      [urls             ]             587µs ± 0%              574µs ± 0%  -2.31%          (p=0.008 n=5+5)
BM_UFlat/2      [jpg              ]            7.24µs ± 2%             7.25µs ± 2%    ~             (p=0.690 n=5+5)
BM_UFlat/3      [jpg_200          ]              130ns ± 0%              131ns ± 1%    ~             (p=0.556 n=4+5)
BM_UFlat/4      [pdf              ]            8.21µs ± 0%             8.24µs ± 1%    ~             (p=0.278 n=5+5)
BM_UFlat/5      [html4            ]             219µs ± 0%              220µs ± 0%  +0.45%          (p=0.008 n=5+5)
BM_UFlat/6      [txt1             ]             192µs ± 0%              190µs ± 0%  -0.86%          (p=0.008 n=5+5)
BM_UFlat/7      [txt2             ]             169µs ± 0%              168µs ± 0%  -0.54%          (p=0.008 n=5+5)
BM_UFlat/8      [txt3             ]             509µs ± 0%              505µs ± 0%  -0.66%          (p=0.008 n=5+5)
BM_UFlat/9      [txt4             ]             710µs ± 0%              702µs ± 0%  -1.14%          (p=0.008 n=5+5)
BM_UFlat/10     [pb               ]            38.2µs ± 0%             37.9µs ± 0%  -0.82%          (p=0.008 n=5+5)
BM_UFlat/11     [gaviota          ]             189µs ± 0%              189µs ± 0%    ~             (p=0.746 n=5+5)
BM_UFlat/12     [cp               ]            14.2µs ± 0%             14.2µs ± 1%    ~             (p=0.421 n=5+5)
BM_UFlat/13     [c                ]            7.29µs ± 0%             7.34µs ± 1%  +0.69%          (p=0.016 n=5+5)
BM_UFlat/14     [lsp              ]            2.27µs ± 0%             2.28µs ± 0%  +0.34%          (p=0.008 n=5+5)
BM_UFlat/15     [xls              ]             954µs ± 0%              900µs ± 0%  -5.67%          (p=0.008 n=5+5)
BM_UFlat/16     [xls_200          ]              213ns ± 1%              217ns ± 2%    ~             (p=0.056 n=5+5)
BM_UFlat/17     [bin              ]             276µs ± 0%              274µs ± 0%  -0.94%          (p=0.008 n=5+5)
BM_UFlat/18     [bin_200          ]              101ns ± 1%              101ns ± 1%    ~             (p=0.524 n=5+5)
BM_UFlat/19     [sum              ]            29.3µs ± 0%             27.3µs ± 0%  -6.98%          (p=0.008 n=5+5)
BM_UFlat/20     [man              ]            2.95µs ± 0%             2.95µs ± 0%    ~             (p=0.651 n=5+5)

For microbenchmarks, the overhead of allocating/deallocating should be
small (the relevant metadata for TCMalloc's PageMap will be in cache),
but this helps demonstrate that the refactoring does not adversely
impact performance.
2019-01-04 19:07:40 -08:00
alkis 1b7466e143 Compute the wordmask instead of looking it up in a table.
Tested:
  name                                old speed               new speed               delta
  BM_UFlat/0      [html   ]           2.13GB/s ± 0%           2.46GB/s ± 0%  +15.70%         (p=0.000 n=10+8)
  BM_UFlat/1      [urls   ]           1.21GB/s ± 0%           1.20GB/s ± 0%   -1.49%         (p=0.000 n=9+10)
  BM_UFlat/2      [jpg    ]           17.1GB/s ± 1%           17.2GB/s ± 1%     ~           (p=0.120 n=11+11)
  BM_UFlat/3      [jpg_200]           1.55GB/s ± 0%           1.54GB/s ± 0%   -0.96%         (p=0.000 n=10+7)
  BM_UFlat/4      [pdf    ]           12.9GB/s ± 0%           12.6GB/s ± 0%   -1.98%         (p=0.000 n=11+9)
  BM_UFlat/5      [html4  ]           1.87GB/s ± 0%           1.87GB/s ± 0%   -0.06%        (p=0.033 n=11+11)
  BM_UFlat/6      [txt1   ]            816MB/s ± 0%            793MB/s ± 0%   -2.84%        (p=0.000 n=11+11)
  BM_UFlat/7      [txt2   ]            758MB/s ± 0%            737MB/s ± 0%   -2.77%        (p=0.000 n=11+11)
  BM_UFlat/8      [txt3   ]            865MB/s ± 0%            839MB/s ± 0%   -2.94%         (p=0.000 n=11+8)
  BM_UFlat/9      [txt4   ]            701MB/s ± 0%            679MB/s ± 0%   -3.11%        (p=0.000 n=11+10)
  BM_UFlat/10     [pb     ]           2.60GB/s ± 2%           3.07GB/s ± 0%  +17.81%        (p=0.000 n=11+11)
  BM_UFlat/11     [gaviota]           1.01GB/s ± 0%           0.97GB/s ± 0%   -3.83%        (p=0.000 n=11+10)
  BM_UFlat/12     [cp     ]           1.66GB/s ± 1%           1.73GB/s ± 1%   +4.32%        (p=0.000 n=11+11)
  BM_UFlat/13     [c      ]           1.52GB/s ± 1%           1.53GB/s ± 0%   +0.49%        (p=0.002 n=11+11)
  BM_UFlat/14     [lsp    ]           1.61GB/s ± 0%           1.64GB/s ± 0%   +2.10%        (p=0.000 n=10+11)
  BM_UFlat/15     [xls    ]           1.12GB/s ± 0%           1.08GB/s ± 0%   -3.95%         (p=0.000 n=11+7)
  BM_UFlat/16     [xls_200]            926MB/s ± 1%            935MB/s ± 1%     ~            (p=0.056 n=9+11)
  BM_UFlat/17     [bin    ]           1.89GB/s ± 0%           1.86GB/s ± 0%   -1.32%        (p=0.000 n=11+11)
  BM_UFlat/18     [bin_200]           1.96GB/s ± 0%           1.99GB/s ± 1%   +1.78%        (p=0.000 n=11+11)
  BM_UFlat/19     [sum    ]           1.32GB/s ± 0%           1.31GB/s ± 0%   -0.79%        (p=0.000 n=11+10)
  BM_UFlat/20     [man    ]           1.40GB/s ± 0%           1.43GB/s ± 0%   +2.51%         (p=0.000 n=9+10)
  BM_UValidate/0  [html   ]           2.95GB/s ± 1%           3.07GB/s ± 0%   +4.11%        (p=0.000 n=10+11)
  BM_UValidate/1  [urls   ]           1.57GB/s ± 0%           1.60GB/s ± 0%   +2.24%        (p=0.000 n=10+11)
  BM_UValidate/2  [jpg    ]            822GB/s ± 0%            850GB/s ± 0%   +3.42%        (p=0.000 n=10+11)
  BM_UValidate/3  [jpg_200]           2.01GB/s ± 0%           2.04GB/s ± 0%   +1.24%        (p=0.000 n=11+11)
  BM_UValidate/4  [pdf    ]           33.7GB/s ± 0%           35.9GB/s ± 1%   +6.51%        (p=0.000 n=10+11)
  BM_UIOVec/0     [html   ]            852MB/s ± 0%            852MB/s ± 0%     ~           (p=0.898 n=11+11)
  BM_UIOVec/1     [urls   ]            663MB/s ± 0%            652MB/s ± 0%   -1.61%        (p=0.000 n=11+11)
  BM_UIOVec/2     [jpg    ]           15.3GB/s ± 1%           15.3GB/s ± 2%     ~            (p=0.459 n=9+10)
  BM_UIOVec/3     [jpg_200]            652MB/s ± 0%            627MB/s ± 1%   -3.80%        (p=0.000 n=10+11)
  BM_UIOVec/4     [pdf    ]           8.80GB/s ± 1%           8.57GB/s ± 1%   -2.62%        (p=0.000 n=10+11)
  BM_UFlatSink/0  [html   ]           2.13GB/s ± 0%           2.46GB/s ± 0%  +15.63%        (p=0.000 n=11+11)
  BM_UFlatSink/1  [urls   ]           1.21GB/s ± 0%           1.20GB/s ± 0%   -1.42%        (p=0.000 n=11+10)
  BM_UFlatSink/2  [jpg    ]           17.1GB/s ± 2%           17.2GB/s ± 1%     ~            (p=0.175 n=11+9)
  BM_UFlatSink/3  [jpg_200]           1.52GB/s ± 1%           1.47GB/s ± 3%   -3.15%        (p=0.000 n=11+11)
  BM_UFlatSink/4  [pdf    ]           12.8GB/s ± 1%           12.6GB/s ± 1%   -1.76%        (p=0.000 n=11+11)
  BM_UFlatSink/5  [html4  ]           1.87GB/s ± 0%           1.87GB/s ± 0%   -0.19%        (p=0.000 n=11+10)
  BM_UFlatSink/6  [txt1   ]            816MB/s ± 0%            792MB/s ± 0%   -2.94%        (p=0.000 n=11+11)
  BM_UFlatSink/7  [txt2   ]            758MB/s ± 0%            736MB/s ± 0%   -2.83%        (p=0.000 n=11+11)
  BM_UFlatSink/8  [txt3   ]            865MB/s ± 0%            838MB/s ± 0%   -3.13%        (p=0.000 n=11+11)
  BM_UFlatSink/9  [txt4   ]            701MB/s ± 0%            678MB/s ± 0%   -3.20%        (p=0.000 n=11+11)
  BM_UFlatSink/10 [pb     ]           2.60GB/s ± 2%           3.07GB/s ± 0%  +18.27%        (p=0.000 n=11+10)
  BM_UFlatSink/11 [gaviota]           1.01GB/s ± 0%           0.97GB/s ± 0%   -3.90%        (p=0.000 n=11+11)
  BM_UFlatSink/12 [cp     ]           1.66GB/s ± 1%           1.73GB/s ± 1%   +4.62%        (p=0.000 n=11+10)
  BM_UFlatSink/13 [c      ]           1.52GB/s ± 0%           1.53GB/s ± 1%     ~            (p=0.180 n=9+11)
  BM_UFlatSink/14 [lsp    ]           1.61GB/s ± 0%           1.64GB/s ± 1%   +1.98%         (p=0.000 n=9+11)
  BM_UFlatSink/15 [xls    ]           1.12GB/s ± 0%           1.08GB/s ± 0%   -3.76%        (p=0.000 n=11+11)
  BM_UFlatSink/16 [xls_200]            909MB/s ± 2%            924MB/s ± 1%   +1.62%        (p=0.000 n=11+11)
  BM_UFlatSink/17 [bin    ]           1.88GB/s ± 0%           1.86GB/s ± 0%   -1.18%         (p=0.000 n=9+11)
  BM_UFlatSink/18 [bin_200]           1.94GB/s ± 2%           1.94GB/s ± 1%     ~           (p=0.090 n=11+11)
  BM_UFlatSink/19 [sum    ]           1.32GB/s ± 0%           1.31GB/s ± 0%   -0.76%        (p=0.000 n=11+11)
  BM_UFlatSink/20 [man    ]           1.39GB/s ± 2%           1.43GB/s ± 0%   +2.75%        (p=0.000 n=11+10)

  Assembly before:

*	44 8b 5c 85 a0       	mov    -0x60(%rbp,%rax,4),%r11d
	45 23 5d 00          	and    0x0(%r13),%r11d
	89 d6                	mov    %edx,%esi
	81 e6 00 07 00 00    	and    $0x700,%esi

  Assembly after:

*	89 c1                	mov    %eax,%ecx
*	c0 e1 03             	shl    $0x3,%cl
*	bf ff ff ff ff       	mov    $0xffffffff,%edi
*	48 d3 e7             	shl    %cl,%rdi
*	f7 d7                	not    %edi
	41 23 7d 00          	and    0x0(%r13),%edi
	41 89 d3             	mov    %edx,%r11d
	41 81 e3 00 07 00 00 	and    $0x700,%r11d
2019-01-04 19:07:28 -08:00
costan ea660b57d6 Fix unused private field warning in NDEBUG builds. 2018-08-17 14:31:23 -07:00
costan 7fefd231a1 C++11 guarantees <cstddef> and <cstdint>.
The build configuration can be cleaned up a bit.
2018-08-16 11:36:45 -07:00
costan ad82620f6f Move pshufb_fill_patterns from snappy-internal.h to snappy.cc.
The array of constants is only used in the SSSE3 fast-path in IncrementalCopy.
2018-08-09 12:08:12 -07:00
costan 73c31e824c Fix Visual Studio build.
Commit 8f469d97e2 introduced SSSE3 fast
paths that are gated by __SSE3__ macro checks and the <x86intrin.h>
header, neither of which exists in Visual Studio. This commit adds logic
for detecting SSSE3 compiler support that works for all compilers
supported by the open source release.

The commit also replaces the header with <tmmintrin.h>, which only
defines intrinsics supported by SSSE3 and below. This should help flag
any use of SIMD instructions that require more advanced SSE support, so
the uses can be gated by checks that also work in the open source
release.

Last, this commit requires C++11 support for the open source build. This is
needed by the alignas specifier, which was also introduced in commit
8f469d97e2.
2018-08-08 22:25:14 -07:00
jefflim 27ff0af12a Improve performance of zippy decompression to IOVecs by up to almost 50%
1) Simplify loop condition for small pattern IncrementalCopy
2) Use pointers rather than indices to track current iovec.
3) Use fast IncrementalCopy
4) Bypass Append check from within AppendFromSelf

While this code greatly improves the performance of ZippyIOVecWriter, a
bigger question is whether IOVec writing should be improved, or removed.

Perf tests:

name                                 old speed      new speed      delta
BM_UFlat/0      [html             ]  2.13GB/s ± 0%  2.14GB/s ± 1%     ~
BM_UFlat/1      [urls             ]  1.22GB/s ± 0%  1.24GB/s ± 0%   +1.87%
BM_UFlat/2      [jpg              ]  17.2GB/s ± 1%  17.1GB/s ± 0%     ~
BM_UFlat/3      [jpg_200          ]  1.55GB/s ± 0%  1.53GB/s ± 2%     ~
BM_UFlat/4      [pdf              ]  12.8GB/s ± 1%  12.7GB/s ± 2%   -0.36%
BM_UFlat/5      [html4            ]  1.89GB/s ± 0%  1.90GB/s ± 1%     ~
BM_UFlat/6      [txt1             ]   811MB/s ± 0%   829MB/s ± 1%   +2.24%
BM_UFlat/7      [txt2             ]   756MB/s ± 0%   774MB/s ± 1%   +2.41%
BM_UFlat/8      [txt3             ]   860MB/s ± 0%   879MB/s ± 1%   +2.16%
BM_UFlat/9      [txt4             ]   699MB/s ± 0%   715MB/s ± 1%   +2.31%
BM_UFlat/10     [pb               ]  2.64GB/s ± 0%  2.65GB/s ± 1%     ~
BM_UFlat/11     [gaviota          ]  1.00GB/s ± 0%  0.99GB/s ± 2%     ~
BM_UFlat/12     [cp               ]  1.66GB/s ± 1%  1.66GB/s ± 2%     ~
BM_UFlat/13     [c                ]  1.53GB/s ± 0%  1.47GB/s ± 5%   -3.97%
BM_UFlat/14     [lsp              ]  1.60GB/s ± 1%  1.55GB/s ± 5%   -3.41%
BM_UFlat/15     [xls              ]  1.12GB/s ± 0%  1.15GB/s ± 0%   +1.93%
BM_UFlat/16     [xls_200          ]   918MB/s ± 2%   929MB/s ± 1%   +1.15%
BM_UFlat/17     [bin              ]  1.86GB/s ± 0%  1.89GB/s ± 1%   +1.61%
BM_UFlat/18     [bin_200          ]  1.90GB/s ± 1%  1.97GB/s ± 1%   +3.67%
BM_UFlat/19     [sum              ]  1.32GB/s ± 0%  1.33GB/s ± 1%     ~
BM_UFlat/20     [man              ]  1.39GB/s ± 0%  1.36GB/s ± 3%     ~
BM_UValidate/0  [html             ]  2.85GB/s ± 3%  2.90GB/s ± 0%     ~
BM_UValidate/1  [urls             ]  1.57GB/s ± 0%  1.56GB/s ± 0%   -0.20%
BM_UValidate/2  [jpg              ]   824GB/s ± 0%   825GB/s ± 0%   +0.11%
BM_UValidate/3  [jpg_200          ]  2.01GB/s ± 0%  2.02GB/s ± 0%   +0.10%
BM_UValidate/4  [pdf              ]  30.4GB/s ±11%  33.5GB/s ± 0%     ~
BM_UIOVec/0     [html             ]   604MB/s ± 0%   856MB/s ± 0%  +41.70%
BM_UIOVec/1     [urls             ]   440MB/s ± 0%   660MB/s ± 0%  +49.91%
BM_UIOVec/2     [jpg              ]  15.1GB/s ± 1%  15.3GB/s ± 1%   +1.22%
BM_UIOVec/3     [jpg_200          ]   567MB/s ± 1%   629MB/s ± 0%  +10.89%
BM_UIOVec/4     [pdf              ]  7.16GB/s ± 2%  8.56GB/s ± 1%  +19.64%
BM_UFlatSink/0  [html             ]  2.13GB/s ± 0%  2.16GB/s ± 0%   +1.47%
BM_UFlatSink/1  [urls             ]  1.22GB/s ± 0%  1.25GB/s ± 0%   +2.18%
BM_UFlatSink/2  [jpg              ]  17.1GB/s ± 2%  17.1GB/s ± 2%     ~
BM_UFlatSink/3  [jpg_200          ]  1.51GB/s ± 1%  1.53GB/s ± 2%   +1.11%
BM_UFlatSink/4  [pdf              ]  12.7GB/s ± 2%  12.8GB/s ± 1%   +0.67%
BM_UFlatSink/5  [html4            ]  1.90GB/s ± 0%  1.92GB/s ± 0%   +1.31%
BM_UFlatSink/6  [txt1             ]   810MB/s ± 0%   835MB/s ± 0%   +3.04%
BM_UFlatSink/7  [txt2             ]   755MB/s ± 0%   779MB/s ± 0%   +3.19%
BM_UFlatSink/8  [txt3             ]   859MB/s ± 0%   884MB/s ± 0%   +2.86%
BM_UFlatSink/9  [txt4             ]   698MB/s ± 0%   718MB/s ± 0%   +2.96%
BM_UFlatSink/10 [pb               ]  2.64GB/s ± 0%  2.67GB/s ± 0%   +1.16%
BM_UFlatSink/11 [gaviota          ]  1.00GB/s ± 0%  1.01GB/s ± 0%   +1.04%
BM_UFlatSink/12 [cp               ]  1.66GB/s ± 1%  1.68GB/s ± 1%   +0.83%
BM_UFlatSink/13 [c                ]  1.52GB/s ± 1%  1.53GB/s ± 0%   +0.38%
BM_UFlatSink/14 [lsp              ]  1.60GB/s ± 1%  1.61GB/s ± 0%   +0.91%
BM_UFlatSink/15 [xls              ]  1.12GB/s ± 0%  1.15GB/s ± 0%   +1.96%
BM_UFlatSink/16 [xls_200          ]   906MB/s ± 3%   920MB/s ± 1%   +1.55%
BM_UFlatSink/17 [bin              ]  1.86GB/s ± 0%  1.90GB/s ± 0%   +2.15%
BM_UFlatSink/18 [bin_200          ]  1.85GB/s ± 2%  1.92GB/s ± 2%   +4.01%
BM_UFlatSink/19 [sum              ]  1.32GB/s ± 1%  1.35GB/s ± 0%   +2.23%
BM_UFlatSink/20 [man              ]  1.39GB/s ± 1%  1.40GB/s ± 0%   +1.12%
BM_ZFlat/0      [html (22.31 %)   ]   800MB/s ± 0%   793MB/s ± 0%   -0.95%
BM_ZFlat/1      [urls (47.78 %)   ]   423MB/s ± 0%   424MB/s ± 0%   +0.11%
BM_ZFlat/2      [jpg (99.95 %)    ]  12.0GB/s ± 2%  12.0GB/s ± 4%     ~
BM_ZFlat/3      [jpg_200 (73.00 %)]   592MB/s ± 3%   594MB/s ± 2%     ~
BM_ZFlat/4      [pdf (83.30 %)    ]  7.26GB/s ± 1%  7.23GB/s ± 2%   -0.49%
BM_ZFlat/5      [html4 (22.52 %)  ]   738MB/s ± 0%   739MB/s ± 0%   +0.17%
BM_ZFlat/6      [txt1 (57.88 %)   ]   286MB/s ± 0%   285MB/s ± 0%   -0.09%
BM_ZFlat/7      [txt2 (61.91 %)   ]   264MB/s ± 0%   264MB/s ± 0%   +0.08%
BM_ZFlat/8      [txt3 (54.99 %)   ]   300MB/s ± 0%   300MB/s ± 0%     ~
BM_ZFlat/9      [txt4 (66.26 %)   ]   248MB/s ± 0%   247MB/s ± 0%   -0.20%
BM_ZFlat/10     [pb (19.68 %)     ]  1.04GB/s ± 0%  1.03GB/s ± 0%   -1.17%
BM_ZFlat/11     [gaviota (37.72 %)]   451MB/s ± 0%   450MB/s ± 0%   -0.35%
BM_ZFlat/12     [cp (48.12 %)     ]   543MB/s ± 0%   538MB/s ± 0%   -1.04%
BM_ZFlat/13     [c (42.47 %)      ]   638MB/s ± 1%   643MB/s ± 0%   +0.68%
BM_ZFlat/14     [lsp (48.37 %)    ]   686MB/s ± 0%   691MB/s ± 1%   +0.76%
BM_ZFlat/15     [xls (41.23 %)    ]   636MB/s ± 0%   633MB/s ± 0%   -0.52%
BM_ZFlat/16     [xls_200 (78.00 %)]   523MB/s ± 2%   520MB/s ± 2%   -0.56%
BM_ZFlat/17     [bin (18.11 %)    ]  1.01GB/s ± 0%  1.01GB/s ± 0%   +0.50%
BM_ZFlat/18     [bin_200 (7.50 %) ]  2.45GB/s ± 1%  2.44GB/s ± 1%   -0.54%
BM_ZFlat/19     [sum (48.96 %)    ]   487MB/s ± 0%   478MB/s ± 0%   -1.89%
BM_ZFlat/20     [man (59.21 %)    ]   567MB/s ± 1%   566MB/s ± 1%     ~

The BM_UFlat/13 and BM_UFlat/14 results showed high variance, so I reran them:

name               old speed      new speed      delta
BM_UFlat/13 [c  ]  1.53GB/s ± 0%  1.53GB/s ± 1%    ~
BM_UFlat/14 [lsp]  1.61GB/s ± 1%  1.61GB/s ± 1%  +0.25%
2018-08-07 23:41:17 -07:00
atdt be490ef9ec Test for SSE3 suppport before using pshufb. 2018-08-04 18:51:13 -07:00
atdt 8f469d97e2 Avoid store-forwarding stalls in Zippy's IncrementalCopy
NEW: Annotate `pattern` as initialized, for MSan.

Snappy's IncrementalCopy routine optimizes for speed by reading and writing
memory in blocks of eight or sixteen bytes. If the gap between the source
and destination pointers is smaller than eight bytes, snappy's strategy is
to expand the gap by issuing a series of partly-overlapping eight-byte
loads+stores. Because the range of each load partly overlaps that of the
store which preceded it, the store buffer cannot be forwarded to the load,
and the load stalls while it waits for the store to retire. This is called a
store-forwarding stall.

We can use fewer loads and avoid most of the stalls by loading the first
eight bytes into an 128-bit XMM register, then using PSHUFB to permute the
register's contents in-place into the desired repeating sequence of bytes.
When falling back to IncrementalCopySlow, use memset if the pattern size == 1.
This eliminates around 60% of the stalls.

name                       old time/op    new time/op    delta
BM_UFlat/0 [html]        48.6µs ± 0%    48.2µs ± 0%   -0.92%        (p=0.000 n=19+18)
BM_UFlat/1 [urls]         589µs ± 0%     576µs ± 0%   -2.17%        (p=0.000 n=19+18)
BM_UFlat/2 [jpg]         7.12µs ± 0%    7.10µs ± 0%     ~           (p=0.071 n=19+18)
BM_UFlat/3 [jpg_200]      162ns ± 0%     151ns ± 0%   -7.06%        (p=0.000 n=19+18)
BM_UFlat/4 [pdf]         8.25µs ± 0%    8.19µs ± 0%   -0.74%        (p=0.000 n=19+18)
BM_UFlat/5 [html4]        218µs ± 0%     218µs ± 0%   +0.09%        (p=0.000 n=17+18)
BM_UFlat/6 [txt1]         191µs ± 0%     189µs ± 0%   -1.12%        (p=0.000 n=19+18)
BM_UFlat/7 [txt2]         168µs ± 0%     167µs ± 0%   -1.01%        (p=0.000 n=19+18)
BM_UFlat/8 [txt3]         502µs ± 0%     499µs ± 0%   -0.52%        (p=0.000 n=19+18)
BM_UFlat/9 [txt4]         704µs ± 0%     695µs ± 0%   -1.26%        (p=0.000 n=19+18)
BM_UFlat/10 [pb]         45.6µs ± 0%    44.2µs ± 0%   -3.13%        (p=0.000 n=19+15)
BM_UFlat/11 [gaviota]     188µs ± 0%     194µs ± 0%   +3.06%        (p=0.000 n=15+18)
BM_UFlat/12 [cp]         15.1µs ± 2%    14.7µs ± 1%   -2.09%        (p=0.000 n=18+18)
BM_UFlat/13 [c]          7.38µs ± 0%    7.36µs ± 0%   -0.28%        (p=0.000 n=16+18)
BM_UFlat/14 [lsp]        2.31µs ± 0%    2.37µs ± 0%   +2.64%        (p=0.000 n=19+18)
BM_UFlat/15 [xls]         984µs ± 0%     909µs ± 0%   -7.59%        (p=0.000 n=19+18)
BM_UFlat/16 [xls_200]     215ns ± 0%     217ns ± 0%   +0.71%        (p=0.000 n=19+15)
BM_UFlat/17 [bin]         289µs ± 0%     287µs ± 0%   -0.71%        (p=0.000 n=19+18)
BM_UFlat/18 [bin_200]     161ns ± 0%     116ns ± 0%  -28.09%        (p=0.000 n=19+16)
BM_UFlat/19 [sum]        31.9µs ± 0%    29.2µs ± 0%   -8.37%        (p=0.000 n=19+18)
BM_UFlat/20 [man]        3.13µs ± 1%    3.07µs ± 0%   -1.79%        (p=0.000 n=19+18)

name                       old allocs/op  new allocs/op  delta
BM_UFlat/0 [html]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/1 [urls]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/2 [jpg]          0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/3 [jpg_200]      0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/4 [pdf]          0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/5 [html4]        0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/6 [txt1]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/7 [txt2]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/8 [txt3]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/9 [txt4]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/10 [pb]          0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/11 [gaviota]     0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/12 [cp]          0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/13 [c]           0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/14 [lsp]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/15 [xls]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/16 [xls_200]     0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/17 [bin]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/18 [bin_200]     0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/19 [sum]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)
BM_UFlat/20 [man]         0.00 ±NaN%     0.00 ±NaN%     ~     (all samples are equal)

name                       old speed      new speed      delta
BM_UFlat/0 [html]      2.11GB/s ± 0%  2.13GB/s ± 0%   +0.92%        (p=0.000 n=19+18)
BM_UFlat/1 [urls]      1.19GB/s ± 0%  1.22GB/s ± 0%   +2.22%        (p=0.000 n=16+17)
BM_UFlat/2 [jpg]       17.3GB/s ± 0%  17.3GB/s ± 0%     ~           (p=0.074 n=19+18)
BM_UFlat/3 [jpg_200]   1.23GB/s ± 0%  1.33GB/s ± 0%   +7.58%        (p=0.000 n=19+18)
BM_UFlat/4 [pdf]       12.4GB/s ± 0%  12.5GB/s ± 0%   +0.74%        (p=0.000 n=19+18)
BM_UFlat/5 [html4]     1.88GB/s ± 0%  1.88GB/s ± 0%   -0.09%        (p=0.000 n=18+18)
BM_UFlat/6 [txt1]       798MB/s ± 0%   807MB/s ± 0%   +1.13%        (p=0.000 n=19+18)
BM_UFlat/7 [txt2]       743MB/s ± 0%   751MB/s ± 0%   +1.02%        (p=0.000 n=19+18)
BM_UFlat/8 [txt3]       850MB/s ± 0%   855MB/s ± 0%   +0.52%        (p=0.000 n=19+18)
BM_UFlat/9 [txt4]       684MB/s ± 0%   693MB/s ± 0%   +1.28%        (p=0.000 n=19+18)
BM_UFlat/10 [pb]       2.60GB/s ± 0%  2.69GB/s ± 0%   +3.25%        (p=0.000 n=19+16)
BM_UFlat/11 [gaviota]   979MB/s ± 0%   950MB/s ± 0%   -2.97%        (p=0.000 n=15+18)
BM_UFlat/12 [cp]       1.63GB/s ± 2%  1.67GB/s ± 1%   +2.13%        (p=0.000 n=18+18)
BM_UFlat/13 [c]        1.51GB/s ± 0%  1.52GB/s ± 0%   +0.29%        (p=0.000 n=16+18)
BM_UFlat/14 [lsp]      1.61GB/s ± 1%  1.57GB/s ± 0%   -2.57%        (p=0.000 n=19+18)
BM_UFlat/15 [xls]      1.05GB/s ± 0%  1.13GB/s ± 0%   +8.22%        (p=0.000 n=19+18)
BM_UFlat/16 [xls_200]   928MB/s ± 0%   921MB/s ± 0%   -0.81%        (p=0.000 n=19+17)
BM_UFlat/17 [bin]      1.78GB/s ± 0%  1.79GB/s ± 0%   +0.71%        (p=0.000 n=19+18)
BM_UFlat/18 [bin_200]  1.24GB/s ± 0%  1.72GB/s ± 0%  +38.92%        (p=0.000 n=19+18)
BM_UFlat/19 [sum]      1.20GB/s ± 0%  1.31GB/s ± 0%   +9.15%        (p=0.000 n=19+18)
BM_UFlat/20 [man]      1.35GB/s ± 1%  1.38GB/s ± 0%   +1.84%        (p=0.000 n=19+18)
2018-08-04 18:51:07 -07:00
jgorbe ca37ab7fb9 Ensure DecompressAllTags starts on a 32-byte boundary + 16 bytes.
First of all, I'm sorry about this ugly hack. I hope the following long
explanation is enough to justify it.

We have observed that, in some conditions, the results for dataset number 10
(pb) in the zippy benchmark can show a >20% regression on Skylake CPUs.

In order to diagnose this, we profiled the benchmark looking at hot functions
(99% of the time is spent on DecompressAllTags), then looked at the generated
code to see if there was any difference. In order to discard a minor difference
we observed in register allocation we replaced zippy.cc with a pre-built assembly
file so it was the same in both variants, and we still were able to reproduce the
regression.

After discarding a regression caused by the compiler, we digged a bit further
and noticed that the alignment of the function in the final binary was
different. Both were aligned to a 16-byte boundary, but the slower one was also
(by chance) aligned to a 32-byte boundary. A regression caused by alignment
differences would explain why I could reproduce it consistently on the same CitC
client, but not others: slight differences in the sources can cause the resulting
binary to have different layout.

Here are some detailed benchmark results before/after the fix. Note how fixing
the alignment makes the difference between baseline and experiment go away, but
regular 32-byte alignment puts both variants in the same ballpark as the
original regression:

Original (note BM_UCord_10 and BM_UDataBuffer_10 around the -24% line):

  BASELINE
  BM_UCord/10                    2938           2932          24194 3.767GB/s  pb
  BM_UDataBuffer/10              3008           3004          23316 3.677GB/s  pb

  EXPERIMENT
  BM_UCord/10                    3797           3789          18512 2.915GB/s  pb
  BM_UDataBuffer/10              4024           4016          17543 2.750GB/s  pb

Aligning DecompressAllTags to a 32-byte boundary:

  BASELINE
  BM_UCord/10                    3872           3862          18035 2.860GB/s  pb
  BM_UDataBuffer/10              4010           3998          17591 2.763GB/s  pb

  EXPERIMENT
  BM_UCord/10                    3884           3876          18126 2.850GB/s  pb
  BM_UDataBuffer/10              4037           4027          17199 2.743GB/s  pb

Aligning DecompressAllTags to a 32-byte boundary + 16 bytes (this patch):

  BASELINE
  BM_UCord/10                    3103           3095          22642 3.569GB/s  pb
  BM_UDataBuffer/10              3186           3177          21947 3.476GB/s  pb

  EXPERIMENT
  BM_UCord/10                    3104           3095          22632 3.569GB/s  pb
  BM_UDataBuffer/10              3167           3159          22076 3.496GB/s  pb

This change forces the "good" alignment for DecompressAllTags which, if
anything, should make benchmark results more stable (and maybe we'll improve
some unlucky application!).
2018-02-17 00:47:18 -08:00
scrubbed 15a2804cd2 Fix an incorrect analysis / comment in the "pattern doubling" code.
This should have a miniscule positive effect on performance; the
main idea of the CL is just to fix the incorrect comment.
2018-02-17 00:46:31 -08:00
chandlerc 4aba5426d4 Rework a very hot, very sensitive part of snappy to reduce the number of
instructions, the number of dynamic branches, and avoid a particular
loop structure than LLVM has a very hard time optimizing for this
particular case.

The code being changed is part of the hottest path for snappy
decompression. In the benchmarks for decompressing protocol buffers,
this has proven to be amazingly sensitive to the slightest changes in
code layout. For example, previously we added '.p2align 5' assembly
directive to the code. This essentially padded the loop out from the
function. Merely by doing this we saw significant performance
improvements.

As a consequence, several of the compiler's typically reasonable
optimizations can have surprising bad impacts. Loop unrolling is a
primary culprit, but in the next LLVM release we are seeing an issue due
to loop rotation. While some of the problems caused by the newly
triggered loop rotation in LLVM can be mitigated with ongoing work on
LLVM's code layout optimizations (specifically, loop header cloning),
that is a fairly long term project. And even minor fluctuations in how
that subsequent optimization is performed may prevent gaining the
performance back.

For now, we need some way to unblock the next LLVM release which
contains a generic improvement to the LLVM loop optimizer that enables
loop rotation in more places, but uncovers this sensitivity and weakness
in a particular case.

This CL restructures the loop to have a simpler structure. Specifically,
we eagerly test what the terminal condition will be and provide two
versions of the copy loop that use a single loop predicate.

The comments in the source code and benchmarks indicate that only one of
these two cases is actually hot: we expect to generally have enough slop
in the buffer. That in turn allows us to generate a much simpler branch
and loop structure for the hot path (especially for the protocol buffer
decompression benchmark).

However, structuring even this simple loop in a way that doesn't trigger
some other performance bubble (often a more severe one) is quite
challenging. We have to carefully manage the variables used in the loop
and the addressing pattern. We should teach LLVM how to do this
reliably, but that too is a *much* more significant undertaking and is
extremely rare to have this degree of importance. The desired structure
of the loop, as shown with IACA's analysis for the broadwell
micro-architecture (HSW and SKX are similar):

| Num Of |                    Ports pressure in cycles                     |    |
|  Uops  |  0  - DV  |  1  |  2  -  D  |  3  -  D  |  4  |  5  |  6  |  7  |    |
---------------------------------------------------------------------------------
|   1    |           |     | 1.0   1.0 |           |     |     |     |     |    | mov rcx, qword ptr [rdi+rdx*1-0x8]
|   2^   |           |     |           | 0.4       | 1.0 |     |     | 0.6 |    | mov qword ptr [rdi], rcx
|   1    |           |     |           | 1.0   1.0 |     |     |     |     |    | mov rcx, qword ptr [rdi+rdx*1]
|   2^   |           |     | 0.3       |           | 1.0 |     |     | 0.7 |    | mov qword ptr [rdi+0x8], rcx
|   1    | 0.5       |     |           |           |     | 0.5 |     |     |    | add rdi, 0x10
|   1    | 0.2       |     |           |           |     |     | 0.8 |     |    | cmp rdi, rax
|   0F   |           |     |           |           |     |     |     |     |    | jb 0xffffffffffffffe9

Specifically, the arrangement of addressing modes for the stores such
that micro-op fusion (indicated by the `^` on the `2` micro-op count) is
important to achieve good throughput for this loop.

The other thing necessary to make this change effective is to remove our
previous hack using `.p2align 5` to pad out the main decompression loop,
and to forcibly disable loop unrolling for critical loops. Because this
change simplifies the loop structure, more unrolling opportunities show
up. Also, the next LLVM release's generic loop optimization improvements
allow unrolling in more places, requiring still more disabling of
unrolling in this change.  Perhaps most surprising of these is that we
must disable loop unrolling in the *slow* path. While unrolling there
seems pointless, it should also be harmless.  This cold code is laid out
very far away from all of the hot code. All the samples shown in a
profile of the benchmark occur before this loop in the function. And
yet, if the loop gets unrolled (which seems to only happen reliably with
the next LLVM release) we see a nearly 20% regression in decompressing
protocol buffers!

With the current release of LLVM, we still observe some regression from
this source change, but it is fairly small (5% on decompressing protocol
buffers, less elsewhere). And with the next LLVM release it drops to
under 1% even in that case. Meanwhile, without this change, the next
release of LLVM will regress decompressing protocol buffers by more than
10%.
2018-01-04 15:27:15 -08:00
wmi 824e6718b5 Add a loop alignment directive to work around a performance regression.
We found LLVM upstream change at rL310792 degraded zippy benchmark by
~3%. Performance analysis showed the regression was caused by some
side-effect. The incidental loop alignment change (from 32 bytes to 16
bytes) led to increase of branch miss prediction and caused the
regression. The regression was reproducible on several intel
micro-architectures, like sandybridge, haswell and skylake. Sadly we
still don't have good understanding about the internal of intel branch
predictor and cannot explain how the branch miss prediction increases
when the loop alignment changes, so we cannot make a real fix here. The
workaround solution in the patch is to add a directive, align the hot
loop to 32 bytes, which can restore the performance. This is in order to
unblock the flip of default compiler to LLVM.
2017-08-24 16:54:12 -07:00
jueminyang 71b8f86887 Add SNAPPY_ prefix to PREDICT_{TRUE,FALSE} macros. 2017-08-01 14:36:26 -07:00
tmsriram f24f9d2d97 Explicitly copy internal::wordmask to the stack array to work around a compiler
optimization with LLVM that converts const stack arrays to global arrays.  This
is a temporary change and should be reverted when https://reviews.llvm.org/D30759
is fixed.

With PIE, accessing stack arrays is more efficient than global arrays and
wordmask was moved to the stack due to that.  However, the LLVM compiler
automatically converts stack arrays, detected as constant, to global arrays
and this transformation hurts PIE performance with LLVM.

We are working to fix this in the LLVM compiler, via
https://reviews.llvm.org/D30759, to not do this conversion in PIE mode.  Until
this patch is finished, please consider this source change as a temporary
work around to keep this array on the stack.  This source change is important
to allow some projects to flip the default compiler from GCC to LLVM for
optimized builds.

This change works for the following reason.  The LLVM compiler does not convert
non-const stack arrays to global arrays and explicitly copying the elements is
enough to make the compiler assume that this is a non-const array.

With GCC, this change does not affect code-gen in any significant way.  The
array initialization code is slightly different as it copies the constants
directly to the stack.

With LLVM, this keeps the array on the stack.

No change in performance with GCC (within noise range). With LLVM, ~0.7%
improvement in optimized mode (no FDO) and ~1.75% improvement in FDO
mode.
2017-06-28 18:34:54 -07:00
alkis 7b9532b878 Improve the SSE2 macro check on Windows.
This lands https://github.com/google/snappy/pull/37
2017-06-05 13:54:17 -07:00
costan ed3b7b242b Clean up unused function warnings in snappy. 2017-03-17 13:59:03 -07:00
costan 8b60aac4fd Remove "using namespace std;" from zippy-stubs-internal.h.
This makes it easier to build zippy, as some compiles require a warning
suppression to accept "using namespace std".
2017-03-13 13:03:01 -07:00
alkis 3c706d2230 Make UnalignedCopy64 not exhibit undefined behavior when src and dst overlap.
name         old speed      new speed      delta
BM_UFlat/0   3.09GB/s ± 3%  3.07GB/s ± 2%  -0.78%  (p=0.009 n=19+19)
BM_UFlat/1   1.63GB/s ± 2%  1.62GB/s ± 2%    ~     (p=0.099 n=19+20)
BM_UFlat/2   19.7GB/s ±19%  20.7GB/s ±11%    ~     (p=0.054 n=20+19)
BM_UFlat/3   1.61GB/s ± 2%  1.60GB/s ± 1%  -0.48%  (p=0.049 n=20+17)
BM_UFlat/4   15.8GB/s ± 7%  15.6GB/s ±10%    ~     (p=0.234 n=20+20)
BM_UFlat/5   2.47GB/s ± 1%  2.46GB/s ± 2%    ~     (p=0.608 n=19+19)
BM_UFlat/6   1.07GB/s ± 2%  1.07GB/s ± 1%    ~     (p=0.128 n=20+19)
BM_UFlat/7   1.01GB/s ± 1%  1.00GB/s ± 2%    ~     (p=0.656 n=15+19)
BM_UFlat/8   1.13GB/s ± 1%  1.13GB/s ± 1%    ~     (p=0.532 n=18+19)
BM_UFlat/9    918MB/s ± 1%   916MB/s ± 1%    ~     (p=0.443 n=19+18)
BM_UFlat/10  3.90GB/s ± 1%  3.90GB/s ± 1%    ~     (p=0.895 n=20+19)
BM_UFlat/11  1.30GB/s ± 1%  1.29GB/s ± 2%    ~     (p=0.156 n=19+19)
BM_UFlat/12  2.35GB/s ± 2%  2.34GB/s ± 1%    ~     (p=0.349 n=19+17)
BM_UFlat/13  2.07GB/s ± 1%  2.06GB/s ± 2%    ~     (p=0.475 n=18+19)
BM_UFlat/14  2.23GB/s ± 1%  2.23GB/s ± 1%    ~     (p=0.983 n=19+19)
BM_UFlat/15  1.55GB/s ± 1%  1.55GB/s ± 1%    ~     (p=0.314 n=19+19)
BM_UFlat/16  1.26GB/s ± 1%  1.26GB/s ± 1%    ~     (p=0.907 n=15+18)
BM_UFlat/17  2.32GB/s ± 1%  2.32GB/s ± 1%    ~     (p=0.604 n=18+19)
BM_UFlat/18  1.61GB/s ± 1%  1.61GB/s ± 1%    ~     (p=0.212 n=18+19)
BM_UFlat/19  1.78GB/s ± 1%  1.78GB/s ± 2%    ~     (p=0.350 n=19+19)
BM_UFlat/20  1.89GB/s ± 1%  1.90GB/s ± 2%    ~     (p=0.092 n=19+19)

Also tested the current version against UNALIGNED_STORE64(dst, UNALIGNED_LOAD64(src)), there is no difference (old is memcpy, new is UNALIGNED*):

name         old speed      new speed      delta
BM_UFlat/0   3.14GB/s ± 1%  3.16GB/s ± 2%    ~     (p=0.156 n=19+19)
BM_UFlat/1   1.62GB/s ± 1%  1.61GB/s ± 2%    ~     (p=0.102 n=19+20)
BM_UFlat/2   18.8GB/s ±17%  19.1GB/s ±11%    ~     (p=0.390 n=20+16)
BM_UFlat/3   1.59GB/s ± 1%  1.58GB/s ± 1%  -1.06%  (p=0.000 n=18+18)
BM_UFlat/4   15.8GB/s ± 6%  15.6GB/s ± 7%    ~     (p=0.184 n=19+20)
BM_UFlat/5   2.46GB/s ± 1%  2.44GB/s ± 1%  -0.95%  (p=0.000 n=19+18)
BM_UFlat/6   1.08GB/s ± 1%  1.06GB/s ± 1%  -1.17%  (p=0.000 n=19+18)
BM_UFlat/7   1.00GB/s ± 1%  0.99GB/s ± 1%  -1.16%  (p=0.000 n=19+18)
BM_UFlat/8   1.14GB/s ± 2%  1.12GB/s ± 1%  -1.12%  (p=0.000 n=19+18)
BM_UFlat/9    921MB/s ± 1%   914MB/s ± 1%  -0.84%  (p=0.000 n=20+17)
BM_UFlat/10  3.94GB/s ± 2%  3.92GB/s ± 1%    ~     (p=0.058 n=19+17)
BM_UFlat/11  1.29GB/s ± 1%  1.28GB/s ± 1%  -0.77%  (p=0.001 n=19+17)
BM_UFlat/12  2.34GB/s ± 1%  2.31GB/s ± 1%  -1.10%  (p=0.000 n=18+18)
BM_UFlat/13  2.06GB/s ± 1%  2.05GB/s ± 1%  -0.73%  (p=0.001 n=19+18)
BM_UFlat/14  2.22GB/s ± 1%  2.20GB/s ± 1%  -0.73%  (p=0.000 n=18+18)
BM_UFlat/15  1.55GB/s ± 1%  1.53GB/s ± 1%  -1.07%  (p=0.000 n=19+18)
BM_UFlat/16  1.26GB/s ± 1%  1.25GB/s ± 1%  -0.79%  (p=0.000 n=18+18)
BM_UFlat/17  2.31GB/s ± 1%  2.29GB/s ± 1%  -0.98%  (p=0.000 n=20+18)
BM_UFlat/18  1.61GB/s ± 1%  1.60GB/s ± 2%  -0.71%  (p=0.001 n=20+19)
BM_UFlat/19  1.77GB/s ± 1%  1.76GB/s ± 1%  -0.61%  (p=0.007 n=19+18)
BM_UFlat/20  1.89GB/s ± 1%  1.88GB/s ± 1%  -0.75%  (p=0.000 n=20+18)
2017-03-08 11:42:30 -08:00
skanev d3c6d20d0a Add compression size reporting hooks.
Also, force inlining util::compression::Sample().

The inlining change is necessary. Without it even with FDO+LIPO the call
doesn't get inlined and uses 4 registers to construct parameters (which
won't be used in the common case). In some of the more compute-bound
tests that causes extra spills and significant overhead (even if
call is sufficiently long).
For example, with inlining:
BM_UFlat/0         32.7µs ± 1%    33.1µs ± 1%  +1.41%
without:
BM_UFlat/0         32.7µs ± 1%    37.7µs ± 1%  +15.29%
2017-03-08 11:42:21 -08:00
alkis 626e1b9faa Use #ifdef __SSE2__ for the emmintrin.h include, otherwise snappy.cc does not compile with -march=prescott. 2017-03-07 18:09:49 -08:00
Alkis Evlogimenos 8bfb028b61 Improve zippy decompression speed.
The CL contains the following optimizations:

1) rewrite IncrementalCopy routine: single routine that splits the code into sections based on typical probabilities observed across a variety of inputs and helps reduce branch mispredictions both for FDO and non-FDO builds. IncrementalCopy is an adaptive routine that selects the best strategy based on input.
2) introduce UnalignedCopy128 that copies 128 bits per cycle using SSE2.
3) add branch hint for the main decoding loop. The non-literal case is taken more often in benchmarks. I expect this to be a noop in production with FDO. Note that this became apparent after step 1 above.
4) use the new IncrementalCopy in ZippyScatteredWriter.

I test two archs: x86_haswell and ppc_power8.

For x86_haswell I use FDO. For ppc_power8 I do not use FDO.

x86_haswell + FDO

name                   old speed      new speed      delta
BM_UCord/0             1.97GB/s ± 1%  3.19GB/s ± 1%  +62.08%  (p=0.000 n=19+18)
BM_UCord/1             1.28GB/s ± 1%  1.51GB/s ± 1%  +18.14%  (p=0.000 n=19+18)
BM_UCord/2             15.6GB/s ± 9%  15.5GB/s ± 7%     ~     (p=0.620 n=20+20)
BM_UCord/3              811MB/s ± 1%   808MB/s ± 1%   -0.38%  (p=0.009 n=17+18)
BM_UCord/4             12.4GB/s ± 4%  12.7GB/s ± 8%   +2.70%  (p=0.002 n=17+20)
BM_UCord/5             1.77GB/s ± 0%  2.33GB/s ± 1%  +31.37%  (p=0.000 n=18+18)
BM_UCord/6              900MB/s ± 1%  1006MB/s ± 1%  +11.71%  (p=0.000 n=18+17)
BM_UCord/7              858MB/s ± 1%   938MB/s ± 2%   +9.36%  (p=0.000 n=19+16)
BM_UCord/8              921MB/s ± 1%   985MB/s ±21%   +6.94%  (p=0.028 n=19+20)
BM_UCord/9              824MB/s ± 1%   800MB/s ±20%     ~     (p=0.113 n=19+20)
BM_UCord/10            2.60GB/s ± 1%  3.67GB/s ±21%  +41.31%  (p=0.000 n=19+20)
BM_UCord/11            1.07GB/s ± 1%  1.21GB/s ± 1%  +13.17%  (p=0.000 n=16+16)
BM_UCord/12            1.84GB/s ± 8%  2.18GB/s ± 1%  +18.44%  (p=0.000 n=16+19)
BM_UCord/13            1.83GB/s ±18%  1.89GB/s ± 1%   +3.14%  (p=0.000 n=17+19)
BM_UCord/14            1.96GB/s ± 2%  1.97GB/s ± 1%   +0.55%  (p=0.000 n=16+17)
BM_UCord/15            1.30GB/s ±20%  1.43GB/s ± 1%   +9.85%  (p=0.000 n=20+20)
BM_UCord/16             658MB/s ±20%   705MB/s ± 1%   +7.22%  (p=0.000 n=20+19)
BM_UCord/17            1.96GB/s ± 2%  2.15GB/s ± 1%   +9.73%  (p=0.000 n=16+19)
BM_UCord/18             555MB/s ± 1%   833MB/s ± 1%  +50.11%  (p=0.000 n=18+19)
BM_UCord/19            1.57GB/s ± 1%  1.75GB/s ± 1%  +11.34%  (p=0.000 n=20+20)
BM_UCord/20            1.72GB/s ± 2%  1.70GB/s ± 2%   -1.01%  (p=0.001 n=20+20)
BM_UCordStringSink/0   2.88GB/s ± 1%  3.15GB/s ± 1%   +9.56%  (p=0.000 n=17+20)
BM_UCordStringSink/1   1.50GB/s ± 1%  1.52GB/s ± 1%   +1.96%  (p=0.000 n=19+20)
BM_UCordStringSink/2   14.5GB/s ±10%  14.6GB/s ±10%     ~     (p=0.542 n=20+20)
BM_UCordStringSink/3   1.06GB/s ± 1%  1.08GB/s ± 1%   +1.77%  (p=0.000 n=18+20)
BM_UCordStringSink/4   12.6GB/s ± 7%  13.2GB/s ± 4%   +4.63%  (p=0.000 n=20+20)
BM_UCordStringSink/5   2.29GB/s ± 1%  2.36GB/s ± 1%   +3.05%  (p=0.000 n=19+20)
BM_UCordStringSink/6   1.01GB/s ± 2%  1.01GB/s ± 0%     ~     (p=0.055 n=20+18)
BM_UCordStringSink/7    945MB/s ± 1%   939MB/s ± 1%   -0.60%  (p=0.000 n=19+20)
BM_UCordStringSink/8   1.06GB/s ± 1%  1.07GB/s ± 1%   +0.62%  (p=0.000 n=18+20)
BM_UCordStringSink/9    866MB/s ± 1%   864MB/s ± 1%     ~     (p=0.107 n=19+20)
BM_UCordStringSink/10  3.64GB/s ± 2%  3.98GB/s ± 1%   +9.32%  (p=0.000 n=19+20)
BM_UCordStringSink/11  1.22GB/s ± 1%  1.22GB/s ± 1%   +0.61%  (p=0.001 n=19+20)
BM_UCordStringSink/12  2.23GB/s ± 1%  2.23GB/s ± 1%     ~     (p=0.692 n=19+20)
BM_UCordStringSink/13  1.96GB/s ± 1%  1.94GB/s ± 1%   -0.82%  (p=0.000 n=17+18)
BM_UCordStringSink/14  2.09GB/s ± 2%  2.08GB/s ± 1%     ~     (p=0.147 n=20+18)
BM_UCordStringSink/15  1.47GB/s ± 1%  1.45GB/s ± 1%   -0.88%  (p=0.000 n=20+19)
BM_UCordStringSink/16   908MB/s ± 1%   917MB/s ± 1%   +0.97%  (p=0.000 n=19+19)
BM_UCordStringSink/17  2.11GB/s ± 1%  2.20GB/s ± 1%   +4.35%  (p=0.000 n=18+20)
BM_UCordStringSink/18   804MB/s ± 2%  1106MB/s ± 1%  +37.52%  (p=0.000 n=20+20)
BM_UCordStringSink/19  1.67GB/s ± 1%  1.72GB/s ± 0%   +2.81%  (p=0.000 n=18+20)
BM_UCordStringSink/20  1.77GB/s ± 3%  1.77GB/s ± 3%     ~     (p=0.815 n=20+20)

ppc_power8

name                   old speed      new speed      delta
BM_UCord/0              918MB/s ± 6%  1262MB/s ± 0%   +37.56%  (p=0.000 n=17+16)
BM_UCord/1              671MB/s ±13%   879MB/s ± 2%   +30.99%  (p=0.000 n=18+16)
BM_UCord/2             12.6GB/s ± 8%  12.6GB/s ± 5%      ~     (p=0.452 n=17+19)
BM_UCord/3              285MB/s ±10%   284MB/s ± 4%    -0.50%  (p=0.021 n=19+17)
BM_UCord/4             5.21GB/s ±12%  6.59GB/s ± 1%   +26.37%  (p=0.000 n=17+16)
BM_UCord/5              913MB/s ± 4%  1253MB/s ± 1%   +37.27%  (p=0.000 n=16+17)
BM_UCord/6              461MB/s ±13%   547MB/s ± 1%   +18.67%  (p=0.000 n=18+16)
BM_UCord/7              455MB/s ± 2%   524MB/s ± 3%   +15.28%  (p=0.000 n=16+18)
BM_UCord/8              489MB/s ± 2%   584MB/s ± 2%   +19.47%  (p=0.000 n=17+17)
BM_UCord/9              410MB/s ±33%   490MB/s ± 1%   +19.64%  (p=0.000 n=17+18)
BM_UCord/10            1.10GB/s ± 3%  1.55GB/s ± 2%   +41.21%  (p=0.000 n=16+16)
BM_UCord/11             494MB/s ± 1%   558MB/s ± 1%   +12.92%  (p=0.000 n=17+18)
BM_UCord/12             608MB/s ± 3%   793MB/s ± 1%   +30.45%  (p=0.000 n=17+16)
BM_UCord/13             545MB/s ±18%   721MB/s ± 2%   +32.22%  (p=0.000 n=19+17)
BM_UCord/14             594MB/s ± 4%   748MB/s ± 3%   +25.99%  (p=0.000 n=17+17)
BM_UCord/15             628MB/s ± 1%   822MB/s ± 3%   +30.94%  (p=0.000 n=18+16)
BM_UCord/16             277MB/s ± 2%   280MB/s ±15%    +0.86%  (p=0.001 n=17+17)
BM_UCord/17             864MB/s ± 1%  1001MB/s ± 3%   +15.96%  (p=0.000 n=17+17)
BM_UCord/18             121MB/s ± 2%   284MB/s ± 4%  +134.08%  (p=0.000 n=17+18)
BM_UCord/19             594MB/s ± 0%   713MB/s ± 2%   +19.93%  (p=0.000 n=16+17)
BM_UCord/20             553MB/s ±10%   662MB/s ± 5%   +19.74%  (p=0.000 n=16+18)
BM_UCordStringSink/0   1.37GB/s ± 4%  1.48GB/s ± 2%    +8.51%  (p=0.000 n=16+16)
BM_UCordStringSink/1    969MB/s ± 1%   990MB/s ± 1%    +2.16%  (p=0.000 n=16+18)
BM_UCordStringSink/2   13.1GB/s ±11%  13.0GB/s ±14%      ~     (p=0.858 n=17+18)
BM_UCordStringSink/3    411MB/s ± 1%   415MB/s ± 1%    +0.93%  (p=0.000 n=16+17)
BM_UCordStringSink/4   6.81GB/s ± 8%  7.29GB/s ± 5%    +7.12%  (p=0.000 n=16+19)
BM_UCordStringSink/5   1.35GB/s ± 5%  1.45GB/s ±13%    +8.00%  (p=0.000 n=16+17)
BM_UCordStringSink/6    653MB/s ± 8%   653MB/s ± 3%    -0.12%  (p=0.007 n=17+19)
BM_UCordStringSink/7    618MB/s ±13%   597MB/s ±18%    -3.45%  (p=0.001 n=18+18)
BM_UCordStringSink/8    702MB/s ± 5%   702MB/s ± 1%    -0.10%  (p=0.012 n=17+16)
BM_UCordStringSink/9    590MB/s ± 2%   564MB/s ±13%    -4.46%  (p=0.000 n=16+17)
BM_UCordStringSink/10  1.63GB/s ± 2%  1.76GB/s ± 4%    +8.28%  (p=0.000 n=17+16)
BM_UCordStringSink/11   630MB/s ±14%   684MB/s ±15%    +8.51%  (p=0.000 n=19+17)
BM_UCordStringSink/12   858MB/s ±12%   903MB/s ± 9%    +5.17%  (p=0.000 n=19+17)
BM_UCordStringSink/13   806MB/s ±22%   879MB/s ± 1%    +8.98%  (p=0.000 n=19+19)
BM_UCordStringSink/14   854MB/s ±13%   901MB/s ± 5%    +5.60%  (p=0.000 n=19+17)
BM_UCordStringSink/15   930MB/s ± 2%   964MB/s ± 3%    +3.59%  (p=0.000 n=16+16)
BM_UCordStringSink/16   363MB/s ±10%   356MB/s ± 6%      ~     (p=0.050 n=20+19)
BM_UCordStringSink/17   976MB/s ±12%  1078MB/s ± 1%   +10.52%  (p=0.000 n=20+17)
BM_UCordStringSink/18   227MB/s ± 1%   355MB/s ± 3%   +56.45%  (p=0.000 n=16+17)
BM_UCordStringSink/19   751MB/s ± 4%   808MB/s ± 4%    +7.70%  (p=0.000 n=18+17)
BM_UCordStringSink/20   761MB/s ± 8%   786MB/s ± 4%    +3.23%  (p=0.000 n=18+17)
2017-01-27 09:10:36 +01:00
Behzad Nouri 818b583387 adds std:: to stl types (#061) 2017-01-26 21:43:13 +01:00
Geoff Pike 27c5d86527 Re-work fast path for handling copies in zippy decompression.
This is a performance-tuning change that shouldn't change the behavior
of the library.

This adds some complexity but the performance gain might make that
worthwhile: With FDO on perflab/haswell, a 4.0% gain (geometric mean).

SAMPLE (before)

Benchmark         Time(ns)    CPU(ns) Iterations
------------------------------------------------
BM_UFlat/0           36638      36552     100000 2.6GB/s  html
BM_UFlat/1          457153     455895       9173 1.4GB/s  urls
BM_UFlat/2            5850       5837     685481 19.6GB/s  jpg
BM_UFlat/3             122        122   34551988 1.5GB/s  jpg_200
BM_UFlat/4            6797       6781     620811 14.1GB/s  pdf
BM_UFlat/5          179485     179037      23471 2.1GB/s  html4
BM_UFlat/6          142734     142384      29525 1018.7MB/s  txt1
BM_UFlat/7          125233     124924      33709 955.6MB/s  txt2
BM_UFlat/8          382548     381533      10000 1066.7MB/s  txt3
BM_UFlat/9          525614     524297       8018 876.5MB/s  txt4
BM_UFlat/10          34946      34868     100000 3.2GB/s  pb
BM_UFlat/11         149548     149208      28063 1.2GB/s  gaviota
BM_UFlat/12          10684      10663     392580 2.1GB/s  cp
BM_UFlat/13           5494       5484     766584 1.9GB/s  c
BM_UFlat/14           1691       1688    2488784 2.1GB/s  lsp
BM_UFlat/15         676443     674726       6129 1.4GB/s  xls
BM_UFlat/16            156        156   26656909 1.2GB/s  xls_200
BM_UFlat/17         239911     239297      17558 2.0GB/s  bin
BM_UFlat/18            182        182   23072932 1047.9MB/s  bin_200
BM_UFlat/19          21544      21499     194484 1.7GB/s  sum
BM_UFlat/20           2236       2232    1877810 1.8GB/s  man
BM_UFlatSink/0       42266      42179      99732 2.3GB/s  html
BM_UFlatSink/1      461810     460633       9055 1.4GB/s  urls
BM_UFlatSink/2        5816       5804     632829 19.8GB/s  jpg
BM_UFlatSink/3         124        123   34351698 1.5GB/s  jpg_200
BM_UFlatSink/4        7173       7157     609929 13.3GB/s  pdf
BM_UFlatSink/5      184795     184302      22660 2.1GB/s  html4
BM_UFlatSink/6      143552     143223      29272 1012.7MB/s  txt1
BM_UFlatSink/7      127160     126890      33178 940.8MB/s  txt2
BM_UFlatSink/8      382219     381313      10000 1067.3MB/s  txt3
BM_UFlatSink/9      528042     526713       7988 872.5MB/s  txt4
BM_UFlatSink/10      41389      41305     100000 2.7GB/s  pb
BM_UFlatSink/11     147215     146877      28854 1.2GB/s  gaviota
BM_UFlatSink/12      12008      11984     348139 1.9GB/s  cp
BM_UFlatSink/13       5444       5433     775084 1.9GB/s  c
BM_UFlatSink/14       1647       1644    2552119 2.1GB/s  lsp
BM_UFlatSink/15     665011     663424       6320 1.4GB/s  xls
BM_UFlatSink/16        153        153   27571837 1.2GB/s  xls_200
BM_UFlatSink/17     239735     239169      17411 2.0GB/s  bin
BM_UFlatSink/18        183        182   23005573 1046.8MB/s  bin_200
BM_UFlatSink/19      22544      22498     187705 1.6GB/s  sum
BM_UFlatSink/20       2190       2186    1917894 1.8GB/s  man

SAMPLE (after)

Benchmark         Time(ns)    CPU(ns) Iterations
------------------------------------------------
BM_UFlat/0           33940      33889     100000 2.8GB/s  html
BM_UFlat/1          440728     439944       9586 1.5GB/s  urls
BM_UFlat/2            5652       5641     744776 20.3GB/s  jpg
BM_UFlat/3             123        123   34647884 1.5GB/s  jpg_200
BM_UFlat/4            6628       6615     631892 14.4GB/s  pdf
BM_UFlat/5          169523     169227      24197 2.3GB/s  html4
BM_UFlat/6          144139     143892      29232 1008.0MB/s  txt1
BM_UFlat/7          127148     126915      33144 940.6MB/s  txt2
BM_UFlat/8          380267     379233      10000 1073.2MB/s  txt3
BM_UFlat/9          529495     528194       7957 870.0MB/s  txt4
BM_UFlat/10          31844      31784     100000 3.5GB/s  pb
BM_UFlat/11         146822     146476      28737 1.2GB/s  gaviota
BM_UFlat/12          10784      10762     392176 2.1GB/s  cp
BM_UFlat/13           5528       5518     760934 1.9GB/s  c
BM_UFlat/14           1721       1719    2449291 2.0GB/s  lsp
BM_UFlat/15         673304     671774       6255 1.4GB/s  xls
BM_UFlat/16            155        155   27092003 1.2GB/s  xls_200
BM_UFlat/17         230424     229902      18285 2.1GB/s  bin
BM_UFlat/18            185        184   22818199 1033.9MB/s  bin_200
BM_UFlat/19          21035      20996     200765 1.7GB/s  sum
BM_UFlat/20           2242       2238    1864380 1.8GB/s  man
BM_UFlatSink/0       33487      33405     100000 2.9GB/s  html
BM_UFlatSink/1      431108     430226       9764 1.5GB/s  urls
BM_UFlatSink/2        5927       5916     648112 19.4GB/s  jpg
BM_UFlatSink/3         123        122   34704423 1.5GB/s  jpg_200
BM_UFlatSink/4        6472       6461     653462 14.8GB/s  pdf
BM_UFlatSink/5      164309     163988      25567 2.3GB/s  html4
BM_UFlatSink/6      138274     138020      30311 1050.9MB/s  txt1
BM_UFlatSink/7      120844     120637      34708 989.6MB/s  txt2
BM_UFlatSink/8      371046     370366      10000 1098.9MB/s  txt3
BM_UFlatSink/9      510021     508982       8269 902.9MB/s  txt4
BM_UFlatSink/10      30889      30844     100000 3.6GB/s  pb
BM_UFlatSink/11     140752     140521      29903 1.2GB/s  gaviota
BM_UFlatSink/12      10162      10146     413600 2.3GB/s  cp
BM_UFlatSink/13       5264       5256     762398 2.0GB/s  c
BM_UFlatSink/14       1622       1619    2606069 2.1GB/s  lsp
BM_UFlatSink/15     646897     645756       6512 1.5GB/s  xls
BM_UFlatSink/16        150        150   28223595 1.2GB/s  xls_200
BM_UFlatSink/17     226096     225650      18629 2.1GB/s  bin
BM_UFlatSink/18        185        184   22907935 1035.3MB/s  bin_200
BM_UFlatSink/19      21369      21335     198881 1.7GB/s  sum
BM_UFlatSink/20       2139       2136    1953637 1.8GB/s  man
2017-01-26 21:42:26 +01:00
Sriraman Tallam 4a74094080 Speed up Zippy decompression in PIE mode by removing the penalty for
global array access.

With PIE, accessing global arrays needs two instructions whereas it can be
done with a single instruction without PIE.  See []
For example, without PIE the access looks like:
mov    0x400780(,%rdi,4),%eax  // One instruction to access arr[i]

and with PIE the access looks like:
lea    0x149(%rip),%rax        # 400780 <_ZL3arr>
mov    (%rax,%rdi,4),%eax

This causes a slow down in zippy as it has two global arrays, wordmask and
char_table.  There is no equivalent PC-relative insn. with PIE to do this in
one instruction.

The slow down can be seen as an increase in dynamic instruction count and
cycles with a similar IPC.  We have seen this affect REDACTED recently and this
is causing a ~1% perf. slow down.

One of the mitigation techniques for small arrays is to move it onto the stack,
use the stack pointer to make the access a single instruction.  The downside to
this is the extra instructions at function call to mov the array onto the stack
which is why we want to do this only for small arrays.  I tried moving
wordmask onto the stack since it is a small array. The performance numbers look
good overall. There is an improvement in the dynamic instruction count for
almost all BM_UFlat benchmarks.  BM_UFlat/2 and BM_UFlat/3 are pretty noisy.
The only case where there is a regression is BM_UFlat/10.  Here, the instruction
count does go down but the IPC also goes down affecting performance. This also
looks noisy but I do see a small IPC drop with this change.  Otherwise, the
numbers look good and consistent.  I measured this on a perflab ivybridge
machine multiple times.  Numbers are given below.  For Improv. (improvements),
positive is good.

Binaries built as: blaze build -c opt --dynamic_mode=off

Benchmark	Base CPU(ns)	Opt CPU(ns)	Improv.	Base Cycles	Opt Cycles	Improv.	Base Insns	Opt Insns	Improv.

BM_UFlat/1	541711		537052		0.86%	46068129918	45442732684	1.36%	85113352848	83917656016	1.40%
BM_UFlat/2	6228		6388		-2.57%	582789808	583267855	-0.08%	1261517746	1261116553	0.03%
BM_UFlat/3	159		120		24.53%	61538641	58783800	4.48%	90008672	90980060	-1.08%
BM_UFlat/4	7878		7787		1.16%	710491888	703718556	0.95%	1914898283	1525060250	20.36%
BM_UFlat/5	208854		207673		0.57%	17640846255	17609530720	0.18%	36546983483	36008920788	1.47%
BM_UFlat/6	172595		167225		3.11%	14642082831	14232371166	2.80%	33647820489	33056659600	1.76%
BM_UFlat/7	152364		147901		2.93%	12904338645	12635220582	2.09%	28958390984	28457982504	1.73%
BM_UFlat/8	463764		448244		3.35%	39423576973	37917435891	3.82%	88350964483	86800265943	1.76%
BM_UFlat/9	639517		621811		2.77%	54275945823	52555988926	3.17%	119503172410	117432599704	1.73%
BM_UFlat/10	41929		42358		-1.02%	3593125535	3647231492	-1.51%	8559206066	8446526639	1.32%
BM_UFlat/11	174754		173936		0.47%	14885371426	14749410955	0.91%	36693421142	35987215897	1.92%
BM_UFlat/12	13388		13257		0.98%	1192648670	1179645044	1.09%	3506482177	3454962579	1.47%
BM_UFlat/13	6801		6588		3.13%	627960003	608367286	3.12%	1847877894	1818368400	1.60%
BM_UFlat/14	2057		1989		3.31%	229005588	217393157	5.07%	609686274	599419511	1.68%
BM_UFlat/15	831618		799881		3.82%	70440388955	67911853013	3.59%	167178603105	164653652416	1.51%
BM_UFlat/16	199		199		0.00%	70109081	68747579	1.94%	106263639	105569531	0.65%
BM_UFlat/17	279031		273890		1.84%	23361373312	23294246637	0.29%	40474834585	39981682217	1.22%
BM_UFlat/18	233		199		14.59%	74530664	67841101	8.98%	94305848	92271053	2.16%
BM_UFlat/19	26743		25309		5.36%	2327215133	2206712016	5.18%	6024314357	5935228694	1.48%
BM_UFlat/20	2731		2625		3.88%	282018757	276772813	1.86%	768382519	758277029	1.32%

Is this a reasonable work-around for the problem?  Do you need more performance
measurements?  haih@ is evaluating this change for [] and I will update those
numbers once we have it.

Tested:
   Performance with zippy_unittest.
2017-01-26 21:42:11 +01:00
Geoff Pike 38a5ec5fca Re-work fast path that emits copies in zippy compression.
The primary motivation for the change is that FindMatchLength is
likely to discover a difference in the first 8 bytes it compares.
If that occurs then we know the length of the match is less than 12,
because FindMatchLength is invoked after a 4-byte match is found.
When emitting a copy, it is useful to know that the length is less
than 12 because the two-byte variant of an emitted copy requires that.

This is a performance-tuning change that should not affect the
library's behavior.

With FDO on perflab/Haswell the geometric mean for ZFlat/* went from
47,290ns to 45,741ns, an improvement of 3.4%.

SAMPLE (before)

BM_ZFlat/0      102824     102650      40691 951.4MB/s  html (22.31 %)
BM_ZFlat/1     1293512    1290442       3225 518.9MB/s  urls (47.78 %)
BM_ZFlat/2       10373      10353     417959 11.1GB/s  jpg (99.95 %)
BM_ZFlat/3         268        268   15745324 712.4MB/s  jpg_200 (73.00 %)
BM_ZFlat/4       12137      12113     342462 7.9GB/s  pdf (83.30 %)
BM_ZFlat/5      430672     429720       9724 909.0MB/s  html4 (22.52 %)
BM_ZFlat/6      420541     419636       9833 345.6MB/s  txt1 (57.88 %)
BM_ZFlat/7      373829     373158      10000 319.9MB/s  txt2 (61.91 %)
BM_ZFlat/8     1119014    1116604       3755 364.5MB/s  txt3 (54.99 %)
BM_ZFlat/9     1544203    1540657       2748 298.3MB/s  txt4 (66.26 %)
BM_ZFlat/10      91041      90866      46002 1.2GB/s  pb (19.68 %)
BM_ZFlat/11     332766     331990      10000 529.5MB/s  gaviota (37.72 %)
BM_ZFlat/12      39960      39886     100000 588.3MB/s  cp (48.12 %)
BM_ZFlat/13      14493      14465     287181 735.1MB/s  c (42.47 %)
BM_ZFlat/14       4447       4440     947927 799.3MB/s  lsp (48.37 %)
BM_ZFlat/15    1316362    1313350       3196 747.7MB/s  xls (41.23 %)
BM_ZFlat/16        312        311   10000000 613.0MB/s  xls_200 (78.00 %)
BM_ZFlat/17     388471     387502      10000 1.2GB/s  bin (18.11 %)
BM_ZFlat/18         65         64   64838208 2.9GB/s  bin_200 (7.50 %)
BM_ZFlat/19      65900      65787      63099 554.3MB/s  sum (48.96 %)
BM_ZFlat/20       6188       6177     681951 652.6MB/s  man (59.21 %)

SAMPLE (after)

Benchmark     Time(ns)    CPU(ns) Iterations
--------------------------------------------
BM_ZFlat/0       99259      99044      42428 986.0MB/s  html (22.31 %)
BM_ZFlat/1     1257039    1255276       3341 533.4MB/s  urls (47.78 %)
BM_ZFlat/2       10044      10030     405781 11.4GB/s  jpg (99.95 %)
BM_ZFlat/3         268        267   15732282 713.3MB/s  jpg_200 (73.00 %)
BM_ZFlat/4       11675      11657     358629 8.2GB/s  pdf (83.30 %)
BM_ZFlat/5      420951     419818       9739 930.5MB/s  html4 (22.52 %)
BM_ZFlat/6      415460     414632      10000 349.8MB/s  txt1 (57.88 %)
BM_ZFlat/7      367191     366436      10000 325.8MB/s  txt2 (61.91 %)
BM_ZFlat/8     1098345    1096036       3819 371.3MB/s  txt3 (54.99 %)
BM_ZFlat/9     1508701    1505306       2758 305.3MB/s  txt4 (66.26 %)
BM_ZFlat/10      87195      87031      47289 1.3GB/s  pb (19.68 %)
BM_ZFlat/11     322338     321637      10000 546.5MB/s  gaviota (37.72 %)
BM_ZFlat/12      36739      36668     100000 639.9MB/s  cp (48.12 %)
BM_ZFlat/13      13646      13618     304009 780.9MB/s  c (42.47 %)
BM_ZFlat/14       4249       4240     992456 837.0MB/s  lsp (48.37 %)
BM_ZFlat/15    1262925    1260012       3314 779.4MB/s  xls (41.23 %)
BM_ZFlat/16        308        308   10000000 619.8MB/s  xls_200 (78.00 %)
BM_ZFlat/17     379750     378944      10000 1.3GB/s  bin (18.11 %)
BM_ZFlat/18         62         62   67443280 3.0GB/s  bin_200 (7.50 %)
BM_ZFlat/19      61706      61587      67645 592.1MB/s  sum (48.96 %)
BM_ZFlat/20       5968       5958     698974 676.6MB/s  man (59.21 %)
2017-01-26 21:39:39 +01:00
ckennelly 094c67de88 Speed up the EmitLiteral fast path, +1.62% for ZFlat benchmarks.
This is inspired by the Go version in
//third_party/golang/snappy/encode_amd64.s (emitLiteralFastPath)

        Benchmark         Base:Reference   (1)
--------------------------------------------------
(BM_ZFlat_0 1/cputime_ns)        9.669e-06  +1.65%
(BM_ZFlat_1 1/cputime_ns)        7.643e-07  +2.53%
(BM_ZFlat_10 1/cputime_ns)       1.107e-05  -0.97%
(BM_ZFlat_11 1/cputime_ns)       3.002e-06  +0.71%
(BM_ZFlat_12 1/cputime_ns)       2.338e-05  +7.22%
(BM_ZFlat_13 1/cputime_ns)       6.386e-05  +9.18%
(BM_ZFlat_14 1/cputime_ns)       0.0002256  -0.05%
(BM_ZFlat_15 1/cputime_ns)       7.608e-07  -1.29%
(BM_ZFlat_16 1/cputime_ns)        0.003236  -1.28%
(BM_ZFlat_17 1/cputime_ns)        2.58e-06  +0.52%
(BM_ZFlat_18 1/cputime_ns)         0.01538  +0.00%
(BM_ZFlat_19 1/cputime_ns)       1.436e-05  +6.21%
(BM_ZFlat_2 1/cputime_ns)        0.0001044  +4.99%
(BM_ZFlat_20 1/cputime_ns)       0.0001608  -0.18%
(BM_ZFlat_3 1/cputime_ns)         0.003745  +0.38%
(BM_ZFlat_4 1/cputime_ns)        8.144e-05  +6.21%
(BM_ZFlat_5 1/cputime_ns)        2.328e-06  -1.60%
(BM_ZFlat_6 1/cputime_ns)        2.391e-06  +0.06%
(BM_ZFlat_7 1/cputime_ns)         2.68e-06  -0.61%
(BM_ZFlat_8 1/cputime_ns)        8.852e-07  +0.19%
(BM_ZFlat_9 1/cputime_ns)        6.441e-07  +1.06%

geometric mean                              +1.62%
2017-01-26 21:38:49 +01:00
Geoff Pike fce661fa8c Speed up zippy decompression by removing some zero-extensions.
This is a performance tuning change that should not affect
correctness.  On perflab with FDO on Haswell the performance gain is
21,776ns before vs 21,255ns after, about 2.4%.  (Using geometric means.)

SAMPLE PERFORMANCE with FDO on HASWELL (NEW)

Benchmark         Time(ns)    CPU(ns) Iterations
------------------------------------------------
BM_UFlat/0           37366      37279     100000 2.6GB/s  html
BM_UFlat/1          471153     470204       8975 1.4GB/s  urls
BM_UFlat/2            6116       6105     639496 18.8GB/s  jpg
BM_UFlat/3             123        123   34709908 1.5GB/s  jpg_200
BM_UFlat/4            6724       6714     623318 14.2GB/s  pdf
BM_UFlat/5          183122     182722      23138 2.1GB/s  html4
BM_UFlat/6          144981     144689      29384 1002.5MB/s  txt1
BM_UFlat/7          125939     125691      33423 949.8MB/s  txt2
BM_UFlat/8          383101     382241      10000 1064.7MB/s  txt3
BM_UFlat/9          527824     526606       7958 872.6MB/s  txt4
BM_UFlat/10          34849      34790     100000 3.2GB/s  pb
BM_UFlat/11         150213     149937      28131 1.1GB/s  gaviota
BM_UFlat/12          10850      10830     393231 2.1GB/s  cp
BM_UFlat/13           5532       5523     735739 1.9GB/s  c
BM_UFlat/14           1698       1695    2478035 2.0GB/s  lsp
BM_UFlat/15         678396     676917       6200 1.4GB/s  xls
BM_UFlat/16            155        155   26909789 1.2GB/s  xls_200
BM_UFlat/17         241235     240698      17416 2.0GB/s  bin
BM_UFlat/18            183        183   23000841 1043.5MB/s  bin_200
BM_UFlat/19          21461      21424     193275 1.7GB/s  sum
BM_UFlat/20           2232       2228    1887191 1.8GB/s  man
BM_UFlatSink/0       42272      42199      98528 2.3GB/s  html
BM_UFlatSink/1      460814     459898       9092 1.4GB/s  urls
BM_UFlatSink/2        5558       5547     768629 20.7GB/s  jpg
BM_UFlatSink/3         124        123   33629141 1.5GB/s  jpg_200
BM_UFlatSink/4        6634       6621     629989 14.4GB/s  pdf
BM_UFlatSink/5      182883     182491      23030 2.1GB/s  html4
BM_UFlatSink/6      143269     142964      29410 1014.5MB/s  txt1
BM_UFlatSink/7      127041     126809      33136 941.4MB/s  txt2
BM_UFlatSink/8      384367     383577      10000 1061.0MB/s  txt3
BM_UFlatSink/9      529979     528890       7898 868.9MB/s  txt4
BM_UFlatSink/10      41154      41075     100000 2.7GB/s  pb
BM_UFlatSink/11     146446     146155      28742 1.2GB/s  gaviota
BM_UFlatSink/12      11939      11918     352663 1.9GB/s  cp
BM_UFlatSink/13       5430       5421     770451 1.9GB/s  c
BM_UFlatSink/14       1665       1662    2538921 2.1GB/s  lsp
BM_UFlatSink/15     666840     665617       6309 1.4GB/s  xls
BM_UFlatSink/16        152        152   27639460 1.2GB/s  xls_200
BM_UFlatSink/17     240076     239573      17643 2.0GB/s  bin
BM_UFlatSink/18        183        182   23128210 1046.0MB/s  bin_200
BM_UFlatSink/19      22570      22528     185839 1.6GB/s  sum
BM_UFlatSink/20       2183       2180    1899526 1.8GB/s  man

SAMPLE PERFORMANCE with FDO on HASWELL (OLD)

Benchmark         Time(ns)    CPU(ns) Iterations
------------------------------------------------
BM_UFlat/0           37041      36990     100000 2.6GB/s  html
BM_UFlat/1          471384     470574       8930 1.4GB/s  urls
BM_UFlat/2            5997       5986     722354 19.2GB/s  jpg
BM_UFlat/3             124        123   34964717 1.5GB/s  jpg_200
BM_UFlat/4            6850       6838     621414 13.9GB/s  pdf
BM_UFlat/5          182578     182271      23001 2.1GB/s  html4
BM_UFlat/6          148338     147989      28132 980.1MB/s  txt1
BM_UFlat/7          130682     130471      32347 915.0MB/s  txt2
BM_UFlat/8          397420     396553      10000 1026.3MB/s  txt3
BM_UFlat/9          550126     548872       7736 837.2MB/s  txt4
BM_UFlat/10          35013      34958     100000 3.2GB/s  pb
BM_UFlat/11         152270     151889      27508 1.1GB/s  gaviota
BM_UFlat/12          11117      11096     379059 2.1GB/s  cp
BM_UFlat/13           5812       5801     725240 1.8GB/s  c
BM_UFlat/14           1780       1777    2383982 2.0GB/s  lsp
BM_UFlat/15         707871     706139       5946 1.4GB/s  xls
BM_UFlat/16            157        157   26889747 1.2GB/s  xls_200
BM_UFlat/17         239160     238556      17512 2.0GB/s  bin
BM_UFlat/18            181        180   23326040 1057.5MB/s  bin_200
BM_UFlat/19          22706      22656     186285 1.6GB/s  sum
BM_UFlat/20           2319       2315    1813186 1.7GB/s  man
BM_UFlatSink/0       42657      42574      99000 2.2GB/s  html
BM_UFlatSink/1      466316     465262       9036 1.4GB/s  urls
BM_UFlatSink/2        6873       6859     648525 16.7GB/s  jpg
BM_UFlatSink/3         124        124   34434643 1.5GB/s  jpg_200
BM_UFlatSink/4        6804       6790     624282 14.0GB/s  pdf
BM_UFlatSink/5      185468     185062      22746 2.1GB/s  html4
BM_UFlatSink/6      148511     148209      28284 978.6MB/s  txt1
BM_UFlatSink/7      130865     130607      32144 914.0MB/s  txt2
BM_UFlatSink/8      393931     392983      10000 1035.6MB/s  txt3
BM_UFlatSink/9      545548     544275       7740 844.3MB/s  txt4
BM_UFlatSink/10      41659      41584     100000 2.7GB/s  pb
BM_UFlatSink/11     152062     151721      27854 1.1GB/s  gaviota
BM_UFlatSink/12      11987      11968     350909 1.9GB/s  cp
BM_UFlatSink/13       5652       5641     743280 1.8GB/s  c
BM_UFlatSink/14       1728       1725    2446140 2.0GB/s  lsp
BM_UFlatSink/15     687879     686231       6138 1.4GB/s  xls
BM_UFlatSink/16        155        155   27254484 1.2GB/s  xls_200
BM_UFlatSink/17     240689     240083      17450 2.0GB/s  bin
BM_UFlatSink/18        183        182   22932858 1046.8MB/s  bin_200
BM_UFlatSink/19      22718      22674     185207 1.6GB/s  sum
BM_UFlatSink/20       2272       2268    1851664 1.7GB/s  man
2017-01-26 21:38:36 +01:00
ckennelly e788e527d3 Avoid calling memset when resizing the buffer.
This buffer will be initialized and then trimmed down to size during the
compression phase.
2017-01-26 21:35:55 +01:00
Steinar H. Gunderson d53de18799 Make heuristic match skipping more aggressive.
This causes compression to be much faster on incompressible inputs
(such as the jpeg and pdf tests), and is neutral or even positive on the other
tests. The test set shows only microscopic density regressions; I attempted to
construct a worst-case test set containing ~1500 different cases of mixed
plaintext + /dev/urandom, and even those seemed to be only 0.38 percentage
points less dense on average (the single worst case was 87.8% -> 89.0%), which
we can live with given that this is already an edge case.

The original idea is by Klaus Post; I only tweaked the implementation.
Ironically, the new implementation is almost more in line with the
comment that was there, so I've left that largely alone, albeit
with a small modification.

Microbenchmark results (opt mode, 64-bit, static linking):

Ivy Bridge:

Benchmark                 Base (ns)  New (ns)                                Improvement
----------------------------------------------------------------------------------------
BM_ZFlat/0                   120284    115480  847.0MB/s  html (22.31 %)        +4.2%
BM_ZFlat/1                  1527911   1522242  440.7MB/s  urls (47.78 %)        +0.4%
BM_ZFlat/2                    17591     10582  10.9GB/s  jpg (99.95 %)         +66.2%
BM_ZFlat/3                      323       322  593.3MB/s  jpg_200 (73.00 %)     +0.3%
BM_ZFlat/4                    53691     14063  6.8GB/s  pdf (83.30 %)         +281.8%
BM_ZFlat/5                   495442    492347  794.8MB/s  html4 (22.52 %)       +0.6%
BM_ZFlat/6                   473523    473622  306.7MB/s  txt1 (57.88 %)        -0.0%
BM_ZFlat/7                   421406    420120  284.5MB/s  txt2 (61.91 %)        +0.3%
BM_ZFlat/8                  1265632   1270538  320.8MB/s  txt3 (54.99 %)        -0.4%
BM_ZFlat/9                  1742688   1737894  264.8MB/s  txt4 (66.26 %)        +0.3%
BM_ZFlat/10                  107950    103404  1095.1MB/s  pb (19.68 %)         +4.4%
BM_ZFlat/11                  372660    371818  473.5MB/s  gaviota (37.72 %)     +0.2%
BM_ZFlat/12                   53239     49528  474.4MB/s  cp (48.12 %)          +7.5%
BM_ZFlat/13                   18940     17349  613.9MB/s  c (42.47 %)           +9.2%
BM_ZFlat/14                    5155      5075  700.3MB/s  lsp (48.37 %)         +1.6%
BM_ZFlat/15                 1474757   1474471  667.2MB/s  xls (41.23 %)         +0.0%
BM_ZFlat/16                     363       362  528.0MB/s  xls_200 (78.00 %)     +0.3%
BM_ZFlat/17                  453849    456931  1073.2MB/s  bin (18.11 %)        -0.7%
BM_ZFlat/18                      90        87  2.1GB/s  bin_200 (7.50 %)        +3.4%
BM_ZFlat/19                   82163     80498  453.7MB/s  sum (48.96 %)         +2.1%
BM_ZFlat/20                    7174      7124  566.7MB/s  man (59.21 %)         +0.7%
Sum of all benchmarks       8694831   8623857                                   +0.8%

Sandy Bridge:

Benchmark                 Base (ns)  New (ns)                                Improvement
----------------------------------------------------------------------------------------
BM_ZFlat/0                   117426    112649  868.2MB/s  html (22.31 %)        +4.2%
BM_ZFlat/1                  1517095   1498522  447.5MB/s  urls (47.78 %)        +1.2%
BM_ZFlat/2                    18601     10649  10.8GB/s  jpg (99.95 %)         +74.7%
BM_ZFlat/3                      359       356  536.0MB/s  jpg_200 (73.00 %)     +0.8%
BM_ZFlat/4                    60249     13832  6.9GB/s  pdf (83.30 %)         +335.6%
BM_ZFlat/5                   481246    475571  822.7MB/s  html4 (22.52 %)       +1.2%
BM_ZFlat/6                   460541    455693  318.8MB/s  txt1 (57.88 %)        +1.1%
BM_ZFlat/7                   407751    404147  295.8MB/s  txt2 (61.91 %)        +0.9%
BM_ZFlat/8                  1228255   1222519  333.4MB/s  txt3 (54.99 %)        +0.5%
BM_ZFlat/9                  1678299   1666379  276.2MB/s  txt4 (66.26 %)        +0.7%
BM_ZFlat/10                  106499    101715  1113.4MB/s  pb (19.68 %)         +4.7%
BM_ZFlat/11                  361913    360222  488.7MB/s  gaviota (37.72 %)     +0.5%
BM_ZFlat/12                   53137     49618  473.6MB/s  cp (48.12 %)          +7.1%
BM_ZFlat/13                   18801     17812  597.8MB/s  c (42.47 %)           +5.6%
BM_ZFlat/14                    5394      5383  660.2MB/s  lsp (48.37 %)         +0.2%
BM_ZFlat/15                 1435411   1432870  686.4MB/s  xls (41.23 %)         +0.2%
BM_ZFlat/16                     389       395  483.3MB/s  xls_200 (78.00 %)     -1.5%
BM_ZFlat/17                  447255    445510  1100.4MB/s  bin (18.11 %)        +0.4%
BM_ZFlat/18                      86        86  2.2GB/s  bin_200 (7.50 %)        +0.0%
BM_ZFlat/19                   82555     79512  459.3MB/s  sum (48.96 %)         +3.8%
BM_ZFlat/20                    7527      7553  534.5MB/s  man (59.21 %)         -0.3%
Sum of all benchmarks       8488789   8360993                                   +1.5%

Haswell:

Benchmark                 Base (ns)  New (ns)                                Improvement
----------------------------------------------------------------------------------------
BM_ZFlat/0                   107512    105621  925.6MB/s  html (22.31 %)        +1.8%
BM_ZFlat/1                  1344306   1332479  503.1MB/s  urls (47.78 %)        +0.9%
BM_ZFlat/2                    14752      9471  12.1GB/s  jpg (99.95 %)         +55.8%
BM_ZFlat/3                      287       275  694.0MB/s  jpg_200 (73.00 %)     +4.4%
BM_ZFlat/4                    48810     12263  7.8GB/s  pdf (83.30 %)         +298.0%
BM_ZFlat/5                   443013    442064  884.6MB/s  html4 (22.52 %)       +0.2%
BM_ZFlat/6                   429239    432124  336.0MB/s  txt1 (57.88 %)        -0.7%
BM_ZFlat/7                   381765    383681  311.5MB/s  txt2 (61.91 %)        -0.5%
BM_ZFlat/8                  1136667   1154304  353.0MB/s  txt3 (54.99 %)        -1.5%
BM_ZFlat/9                  1579925   1592431  288.9MB/s  txt4 (66.26 %)        -0.8%
BM_ZFlat/10                   98345     92411  1.2GB/s  pb (19.68 %)            +6.4%
BM_ZFlat/11                  340397    340466  516.8MB/s  gaviota (37.72 %)     -0.0%
BM_ZFlat/12                   47076     43536  539.5MB/s  cp (48.12 %)          +8.1%
BM_ZFlat/13                   16680     15637  680.8MB/s  c (42.47 %)           +6.7%
BM_ZFlat/14                    4616      4539  782.6MB/s  lsp (48.37 %)         +1.7%
BM_ZFlat/15                 1331231   1334094  736.9MB/s  xls (41.23 %)         -0.2%
BM_ZFlat/16                     326       322  593.5MB/s  xls_200 (78.00 %)     +1.2%
BM_ZFlat/17                  404383    400326  1.2GB/s  bin (18.11 %)           +1.0%
BM_ZFlat/18                      69        69  2.7GB/s  bin_200 (7.50 %)        +0.0%
BM_ZFlat/19                   74771     71348  511.7MB/s  sum (48.96 %)         +4.8%
BM_ZFlat/20                    6461      6383  632.2MB/s  man (59.21 %)         +1.2%
Sum of all benchmarks       7810631   7773844                                   +0.5%

I've done a quick test that there are no performance regressions on external
GCC (4.9.2, Debian, Haswell, 64-bit), too.
2016-04-05 11:50:26 +02:00
Steinar H. Gunderson 7525a1600d Fix an issue where the ByteSource path (used for parsing std::string)
would incorrectly accept some invalid varints that the other path would not,
causing potential CHECK-failures if the unit test were run with
--write_uncompressed and a corrupted input file.

Found by the afl fuzzer.
2016-01-04 12:52:15 +01:00
Steinar H. Gunderson 0852af7606 Move the logic from ComputeTable into the unit test, which means it's run
automatically together with the other tests, and also removes the stray
function ComputeTable() (which was never referenced by anything else
in the open-source version, causing compiler warnings for some)
out of the core library.

Fixes public issue 96.

A=sesse
R=sanjay
2015-08-19 11:37:51 +02:00
Steinar H. Gunderson d80342922c Fix signed-vs.-unsigned comparison warnings.
These were found by compiling Chromium's external copy of this code with MSVC
with warning C4018 enabled.

A=pkasting
R=sanjay
2015-08-03 13:17:04 +02:00
Steinar H. Gunderson eb66d8176b Initialized members of SnappyArrayWriter and SnappyDecompressionValidator.
These members were almost surely initialized before use by other member
functions, but Coverity was warning about this. Eliminating these warnings
minimizes clutter in that report and the likelihood of overlooking a real bug.

A=cmumford
R=jeff
2015-07-06 14:21:16 +02:00
Steinar H. Gunderson b2312c4c25 Add support for Uncompress(source, sink). Various changes to allow
Uncompress(source, sink) to get the same performance as the different
variants of Uncompress to Cord/DataBuffer/String/FlatBuffer.

Changes to efficiently support Uncompress(source, sink)
--------

a) For strings - we add support to StringByteSink to do GetAppendBuffer so we
   can write to it without copying.
b) For flat array buffers, we do GetAppendBuffer and see if we can get a full buffer.

With the above changes we get performance with ByteSource/ByteSink
that is	very close to directly using flat arrays and strings.

We add various benchmark cases to demonstrate that.

Orthogonal change
------------------

Add support for TryFastAppend() for SnappyScatteredWriter.

Benchmark results are below

CPU: Intel Core2 dL1:32KB dL2:4096KB
Benchmark              Time(ns)    CPU(ns) Iterations
-----------------------------------------------------
BM_UFlat/0               109065     108996       6410 896.0MB/s  html
BM_UFlat/1              1012175    1012343        691 661.4MB/s  urls
BM_UFlat/2                26775      26771      26149 4.4GB/s  jpg
BM_UFlat/3                48947      48940      14363 1.8GB/s  pdf
BM_UFlat/4               441029     440835       1589 886.1MB/s  html4
BM_UFlat/5                39861      39880      17823 588.3MB/s  cp
BM_UFlat/6                18315      18300      38126 581.1MB/s  c
BM_UFlat/7                 5254       5254     100000 675.4MB/s  lsp
BM_UFlat/8              1568060    1567376        447 626.6MB/s  xls
BM_UFlat/9               337512     337734       2073 429.5MB/s  txt1
BM_UFlat/10              287269     287054       2434 415.9MB/s  txt2
BM_UFlat/11              890098     890219        787 457.2MB/s  txt3
BM_UFlat/12             1186593    1186863        590 387.2MB/s  txt4
BM_UFlat/13              573927     573318       1000 853.7MB/s  bin
BM_UFlat/14               64250      64294      10000 567.2MB/s  sum
BM_UFlat/15                7301       7300      96153 552.2MB/s  man
BM_UFlat/16              109617     109636       6375 1031.5MB/s  pb
BM_UFlat/17              364438     364497       1921 482.3MB/s  gaviota
BM_UFlatSink/0           108518     108465       6450 900.4MB/s  html
BM_UFlatSink/1           991952     991997        705 675.0MB/s  urls
BM_UFlatSink/2            26815      26798      26065 4.4GB/s  jpg
BM_UFlatSink/3            49127      49122      14255 1.8GB/s  pdf
BM_UFlatSink/4           436674     436731       1604 894.4MB/s  html4
BM_UFlatSink/5            39738      39733      17345 590.5MB/s  cp
BM_UFlatSink/6            18413      18416      37962 577.4MB/s  c
BM_UFlatSink/7             5677       5676     100000 625.2MB/s  lsp
BM_UFlatSink/8          1552175    1551026        451 633.2MB/s  xls
BM_UFlatSink/9           338526     338489       2065 428.5MB/s  txt1
BM_UFlatSink/10          289387     289307       2420 412.6MB/s  txt2
BM_UFlatSink/11          893803     893706        783 455.4MB/s  txt3
BM_UFlatSink/12         1195919    1195459        586 384.4MB/s  txt4
BM_UFlatSink/13          559637     559779       1000 874.3MB/s  bin
BM_UFlatSink/14           65073      65094      10000 560.2MB/s  sum
BM_UFlatSink/15            7618       7614      92823 529.5MB/s  man
BM_UFlatSink/16          110085     110121       6352 1027.0MB/s  pb
BM_UFlatSink/17          369196     368915       1896 476.5MB/s  gaviota
BM_UValidate/0            46954      46957      14899 2.0GB/s  html
BM_UValidate/1           500621     500868       1000 1.3GB/s  urls
BM_UValidate/2              283        283    2481447 417.2GB/s  jpg
BM_UValidate/3            16230      16228      43137 5.4GB/s  pdf
BM_UValidate/4           189129     189193       3701 2.0GB/s  html4

A=uday
R=sanjay
2015-07-06 14:21:00 +02:00
Steinar H. Gunderson 86eb8b152b Change a few branch annotations that profiling found to be wrong.
Overall performance is neutral or slightly positive.

Westmere (64-bit, opt):

Benchmark               Base (ns)  New (ns)                                Improvement
--------------------------------------------------------------------------------------
BM_UFlat/0                  73798     71464  1.3GB/s  html                    +3.3%
BM_UFlat/1                 715223    704318  953.5MB/s  urls                  +1.5%
BM_UFlat/2                   8137      8871  13.0GB/s  jpg                    -8.3%
BM_UFlat/3                    200       204  935.5MB/s  jpg_200               -2.0%
BM_UFlat/4                  21627     21281  4.5GB/s  pdf                     +1.6%
BM_UFlat/5                 302806    290350  1.3GB/s  html4                   +4.3%
BM_UFlat/6                 218920    219017  664.1MB/s  txt1                  -0.0%
BM_UFlat/7                 190437    191212  626.1MB/s  txt2                  -0.4%
BM_UFlat/8                 584192    580484  703.4MB/s  txt3                  +0.6%
BM_UFlat/9                 776537    779055  591.6MB/s  txt4                  -0.3%
BM_UFlat/10                 76056     72606  1.5GB/s  pb                      +4.8%
BM_UFlat/11                235962    239043  737.4MB/s  gaviota               -1.3%
BM_UFlat/12                 28049     28000  840.1MB/s  cp                    +0.2%
BM_UFlat/13                 12225     12021  886.9MB/s  c                     +1.7%
BM_UFlat/14                  3362      3544  1004.0MB/s  lsp                  -5.1%
BM_UFlat/15                937015    939206  1048.9MB/s  xls                  -0.2%
BM_UFlat/16                   236       233  823.1MB/s  xls_200               +1.3%
BM_UFlat/17                373170    361947  1.3GB/s  bin                     +3.1%
BM_UFlat/18                   264       264  725.5MB/s  bin_200               +0.0%
BM_UFlat/19                 42834     43577  839.2MB/s  sum                   -1.7%
BM_UFlat/20                  4770      4736  853.6MB/s  man                   +0.7%
BM_UValidate/0              39671     39944  2.4GB/s  html                    -0.7%
BM_UValidate/1             443391    443391  1.5GB/s  urls                    +0.0%
BM_UValidate/2                163       163  703.3GB/s  jpg                   +0.0%
BM_UValidate/3                113       112  1.7GB/s  jpg_200                 +0.9%
BM_UValidate/4               7555      7608  12.6GB/s  pdf                    -0.7%
BM_ZFlat/0                 157616    157568  621.5MB/s  html (22.31 %)        +0.0%
BM_ZFlat/1                1997290   2014486  333.4MB/s  urls (47.77 %)        -0.9%
BM_ZFlat/2                  23035     22237  5.2GB/s  jpg (99.95 %)           +3.6%
BM_ZFlat/3                    539       540  354.5MB/s  jpg_200 (73.00 %)     -0.2%
BM_ZFlat/4                  80709     81369  1.2GB/s  pdf (81.85 %)           -0.8%
BM_ZFlat/5                 639059    639220  613.0MB/s  html4 (22.51 %)       -0.0%
BM_ZFlat/6                 577203    583370  249.3MB/s  txt1 (57.87 %)        -1.1%
BM_ZFlat/7                 510887    516094  232.0MB/s  txt2 (61.93 %)        -1.0%
BM_ZFlat/8                1535843   1556973  262.2MB/s  txt3 (54.92 %)        -1.4%
BM_ZFlat/9                2070068   2102380  219.3MB/s  txt4 (66.22 %)        -1.5%
BM_ZFlat/10                152396    152148  745.5MB/s  pb (19.64 %)          +0.2%
BM_ZFlat/11                447367    445859  395.4MB/s  gaviota (37.72 %)     +0.3%
BM_ZFlat/12                 76375     76797  306.3MB/s  cp (48.12 %)          -0.5%
BM_ZFlat/13                 31518     31987  333.3MB/s  c (42.40 %)           -1.5%
BM_ZFlat/14                 10598     10827  328.6MB/s  lsp (48.37 %)         -2.1%
BM_ZFlat/15               1782243   1802728  546.5MB/s  xls (41.23 %)         -1.1%
BM_ZFlat/16                   526       539  355.0MB/s  xls_200 (78.00 %)     -2.4%
BM_ZFlat/17                598141    597311  822.1MB/s  bin (18.11 %)         +0.1%
BM_ZFlat/18                   121       120  1.6GB/s  bin_200 (7.50 %)        +0.8%
BM_ZFlat/19                109981    112173  326.0MB/s  sum (48.96 %)         -2.0%
BM_ZFlat/20                 14355     14575  277.4MB/s  man (59.36 %)         -1.5%
Sum of all benchmarks    33882722  33879325                                   +0.0%

Sandy Bridge (64-bit, opt):

Benchmark               Base (ns)  New (ns)                                Improvement
--------------------------------------------------------------------------------------
BM_UFlat/0                  43764     41600  2.3GB/s  html                    +5.2%
BM_UFlat/1                 517990    507058  1.3GB/s  urls                    +2.2%
BM_UFlat/2                   6625      5529  20.8GB/s  jpg                   +19.8%
BM_UFlat/3                    154       155  1.2GB/s  jpg_200                 -0.6%
BM_UFlat/4                  12795     11747  8.1GB/s  pdf                     +8.9%
BM_UFlat/5                 200335    193413  2.0GB/s  html4                   +3.6%
BM_UFlat/6                 156574    156426  929.2MB/s  txt1                  +0.1%
BM_UFlat/7                 137574    137464  870.4MB/s  txt2                  +0.1%
BM_UFlat/8                 422551    421603  967.4MB/s  txt3                  +0.2%
BM_UFlat/9                 577749    578985  795.6MB/s  txt4                  -0.2%
BM_UFlat/10                 42329     39362  2.8GB/s  pb                      +7.5%
BM_UFlat/11                170615    169751  1037.9MB/s  gaviota              +0.5%
BM_UFlat/12                 12800     12719  1.8GB/s  cp                      +0.6%
BM_UFlat/13                  6585      6579  1.6GB/s  c                       +0.1%
BM_UFlat/14                  2066      2044  1.7GB/s  lsp                     +1.1%
BM_UFlat/15                750861    746911  1.3GB/s  xls                     +0.5%
BM_UFlat/16                   188       192  996.0MB/s  xls_200               -2.1%
BM_UFlat/17                271622    264333  1.8GB/s  bin                     +2.8%
BM_UFlat/18                   208       207  923.6MB/s  bin_200               +0.5%
BM_UFlat/19                 24667     24845  1.4GB/s  sum                     -0.7%
BM_UFlat/20                  2663      2662  1.5GB/s  man                     +0.0%
BM_ZFlat/0                 115173    115624  846.5MB/s  html (22.31 %)        -0.4%
BM_ZFlat/1                1530331   1537769  436.5MB/s  urls (47.77 %)        -0.5%
BM_ZFlat/2                  17503     17013  6.8GB/s  jpg (99.95 %)           +2.9%
BM_ZFlat/3                    385       385  496.3MB/s  jpg_200 (73.00 %)     +0.0%
BM_ZFlat/4                  61753     61540  1.6GB/s  pdf (81.85 %)           +0.3%
BM_ZFlat/5                 484806    483356  810.1MB/s  html4 (22.51 %)       +0.3%
BM_ZFlat/6                 464143    467609  310.9MB/s  txt1 (57.87 %)        -0.7%
BM_ZFlat/7                 410315    413319  289.5MB/s  txt2 (61.93 %)        -0.7%
BM_ZFlat/8                1244082   1249381  326.5MB/s  txt3 (54.92 %)        -0.4%
BM_ZFlat/9                1696914   1709685  269.4MB/s  txt4 (66.22 %)        -0.7%
BM_ZFlat/10                104148    103372  1096.7MB/s  pb (19.64 %)         +0.8%
BM_ZFlat/11                363522    359722  489.8MB/s  gaviota (37.72 %)     +1.1%
BM_ZFlat/12                 47021     50095  469.3MB/s  cp (48.12 %)          -6.1%
BM_ZFlat/13                 16888     16985  627.4MB/s  c (42.40 %)           -0.6%
BM_ZFlat/14                  5496      5469  650.3MB/s  lsp (48.37 %)         +0.5%
BM_ZFlat/15               1460713   1448760  679.5MB/s  xls (41.23 %)         +0.8%
BM_ZFlat/16                   387       393  486.8MB/s  xls_200 (78.00 %)     -1.5%
BM_ZFlat/17                457654    451462  1086.6MB/s  bin (18.11 %)        +1.4%
BM_ZFlat/18                    97        87  2.1GB/s  bin_200 (7.50 %)       +11.5%
BM_ZFlat/19                 77904     80924  451.7MB/s  sum (48.96 %)         -3.7%
BM_ZFlat/20                  7648      7663  527.1MB/s  man (59.36 %)         -0.2%
Sum of all benchmarks    25493635  25482069                                   +0.0%

A=dehao
R=sesse
2015-06-22 16:09:56 +02:00
Steinar H. Gunderson 11ccdfb868 Sync with various Google-internal changes.
Should not mean much for the open-source version.
2015-06-22 16:08:38 +02:00
snappy.mirrorbot@gmail.com 7c3c01df77 When we compare the number of bytes produced with the offset for a
backreference, make the signedness of the bytes produced clear,
by sticking it into a size_t. This avoids a signed/unsigned compare
warning from MSVC (public issue 71), and also is slightly clearer.

Since the line is now so long the explanatory comment about the -1u
trick has to go somewhere else anyway, I used the opportunity to
explain it in slightly more detail.

This is a purely stylistic change; the emitted assembler from GCC
is identical.

R=jeff


git-svn-id: https://snappy.googlecode.com/svn/trunk@79 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2013-07-29 11:06:44 +00:00
snappy.mirrorbot@gmail.com 2f0aaf8631 In the fast path for decompressing literals, instead of checking
whether there's 16 bytes free and then checking right afterwards
(when having subtracted the literal size) that there are now 
5 bytes free, just check once for 21 bytes. This skips a compare
and a branch; although it is easily predictable, it is still
a few cycles on a fast path that we would like to get rid of.

Benchmarking this yields very confusing results. On open-source
GCC 4.8.1 on Haswell, we get exactly the expected results; the
benchmarks where we hit the fast path for literals (in particular
the two HTML benchmarks and the protobuf benchmark) give very nice
speedups, and the others are not really affected.

However, benchmarks with Google's GCC branch on other hardware
is much less clear. It seems that we have a weak loss in some cases
(and the win for the “typical” win cases are not nearly as clear),
but that it depends on microarchitecture and plain luck in how we run
the benchmark. Looking at the generated assembler, it seems that
the removal of the if causes other large-scale changes in how the
function is laid out, which makes it likely that this is just bad luck.

Thus, we should keep this change, even though its exact current impact is
unclear; it's a sensible change per se, and dropping it on the basis of
microoptimization for a given compiler (or even branch of a compiler)
would seem like a bad strategy in the long run.

Microbenchmark results (all in 64-bit, opt mode):

  Nehalem, Google GCC:

  Benchmark                Base (ns)  New (ns)                       Improvement
  ------------------------------------------------------------------------------
  BM_UFlat/0                   76747     75591  1.3GB/s  html           +1.5%
  BM_UFlat/1                  765756    757040  886.3MB/s  urls         +1.2%
  BM_UFlat/2                   10867     10893  10.9GB/s  jpg           -0.2%
  BM_UFlat/3                     124       131  1.4GB/s  jpg_200        -5.3%
  BM_UFlat/4                   31663     31596  2.8GB/s  pdf            +0.2%
  BM_UFlat/5                  314162    308176  1.2GB/s  html4          +1.9%
  BM_UFlat/6                   29668     29746  790.6MB/s  cp           -0.3%
  BM_UFlat/7                   12958     13386  796.4MB/s  c            -3.2%
  BM_UFlat/8                    3596      3682  966.0MB/s  lsp          -2.3%
  BM_UFlat/9                 1019193   1033493  953.3MB/s  xls          -1.4%
  BM_UFlat/10                    239       247  775.3MB/s  xls_200      -3.2%
  BM_UFlat/11                 236411    240271  606.9MB/s  txt1         -1.6%
  BM_UFlat/12                 206639    209768  571.2MB/s  txt2         -1.5%
  BM_UFlat/13                 627803    635722  641.4MB/s  txt3         -1.2%
  BM_UFlat/14                 845932    857816  538.2MB/s  txt4         -1.4%
  BM_UFlat/15                 402107    391670  1.2GB/s  bin            +2.7%
  BM_UFlat/16                    283       279  683.6MB/s  bin_200      +1.4%
  BM_UFlat/17                  46070     46815  781.5MB/s  sum          -1.6%
  BM_UFlat/18                   5053      5163  782.0MB/s  man          -2.1%
  BM_UFlat/19                  79721     76581  1.4GB/s  pb             +4.1%
  BM_UFlat/20                 251158    252330  697.5MB/s  gaviota      -0.5%
  Sum of all benchmarks      4966150   4980396                          -0.3%


  Sandy Bridge, Google GCC:
  
  Benchmark                Base (ns)  New (ns)                       Improvement
  ------------------------------------------------------------------------------
  BM_UFlat/0                   42850     42182  2.3GB/s  html           +1.6%
  BM_UFlat/1                  525660    515816  1.3GB/s  urls           +1.9%
  BM_UFlat/2                    7173      7283  16.3GB/s  jpg           -1.5%
  BM_UFlat/3                      92        91  2.1GB/s  jpg_200        +1.1%
  BM_UFlat/4                   15147     14872  5.9GB/s  pdf            +1.8%
  BM_UFlat/5                  199936    192116  2.0GB/s  html4          +4.1%
  BM_UFlat/6                   12796     12443  1.8GB/s  cp             +2.8%
  BM_UFlat/7                    6588      6400  1.6GB/s  c              +2.9%
  BM_UFlat/8                    2010      1951  1.8GB/s  lsp            +3.0%
  BM_UFlat/9                  761124    763049  1.3GB/s  xls            -0.3%
  BM_UFlat/10                    186       189  1016.1MB/s  xls_200     -1.6%
  BM_UFlat/11                 159354    158460  918.6MB/s  txt1         +0.6%
  BM_UFlat/12                 139732    139950  856.1MB/s  txt2         -0.2%
  BM_UFlat/13                 429917    425027  961.7MB/s  txt3         +1.2%
  BM_UFlat/14                 585255    587324  785.8MB/s  txt4         -0.4%
  BM_UFlat/15                 276186    266173  1.8GB/s  bin            +3.8%
  BM_UFlat/16                    205       207  925.5MB/s  bin_200      -1.0%
  BM_UFlat/17                  24925     24935  1.4GB/s  sum            -0.0%
  BM_UFlat/18                   2632      2576  1.5GB/s  man            +2.2%
  BM_UFlat/19                  40546     39108  2.8GB/s  pb             +3.7%
  BM_UFlat/20                 175803    168209  1048.9MB/s  gaviota     +4.5%
  Sum of all benchmarks      3408117   3368361                          +1.2%


  Haswell, upstream GCC 4.8.1:

  Benchmark                Base (ns)  New (ns)                       Improvement
  ------------------------------------------------------------------------------
  BM_UFlat/0                   46308     40641  2.3GB/s  html          +13.9%
  BM_UFlat/1                  513385    514706  1.3GB/s  urls           -0.3%
  BM_UFlat/2                    6197      6151  19.2GB/s  jpg           +0.7%
  BM_UFlat/3                      61        61  3.0GB/s  jpg_200        +0.0%
  BM_UFlat/4                   13551     13429  6.5GB/s  pdf            +0.9%
  BM_UFlat/5                  198317    190243  2.0GB/s  html4          +4.2%
  BM_UFlat/6                   14768     12560  1.8GB/s  cp            +17.6%
  BM_UFlat/7                    6453      6447  1.6GB/s  c              +0.1%
  BM_UFlat/8                    1991      1980  1.8GB/s  lsp            +0.6%
  BM_UFlat/9                  766947    770424  1.2GB/s  xls            -0.5%
  BM_UFlat/10                    170       169  1.1GB/s  xls_200        +0.6%
  BM_UFlat/11                 164350    163554  888.7MB/s  txt1         +0.5%
  BM_UFlat/12                 145444    143830  832.1MB/s  txt2         +1.1%
  BM_UFlat/13                 437849    438413  929.2MB/s  txt3         -0.1%
  BM_UFlat/14                 603587    605309  759.8MB/s  txt4         -0.3%
  BM_UFlat/15                 249799    248067  1.9GB/s  bin            +0.7%
  BM_UFlat/16                    191       188  1011.4MB/s  bin_200     +1.6%
  BM_UFlat/17                  26064     24778  1.4GB/s  sum            +5.2%
  BM_UFlat/18                   2620      2601  1.5GB/s  man            +0.7%
  BM_UFlat/19                  44551     37373  3.0GB/s  pb            +19.2%
  BM_UFlat/20                 165408    164584  1.0GB/s  gaviota        +0.5%
  Sum of all benchmarks      3408011   3385508                          +0.7%


git-svn-id: https://snappy.googlecode.com/svn/trunk@78 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2013-06-30 19:24:03 +00:00
snappy.mirrorbot@gmail.com 062bf544a6 Make the two IncrementalCopy* functions take in an ssize_t instead of a len,
in order to avoid having to do 32-to-64-bit signed conversions on a hot path
during decompression. (Also fixes some MSVC warnings, mentioned in public
issue 75, but more of those remain.) They cannot be size_t because we expect
them to go negative and test for that.

This saves a few movzwl instructions, yielding ~2% speedup in decompression.


Sandy Bridge:

Benchmark                          Base (ns)  New (ns)                                Improvement
-------------------------------------------------------------------------------------------------
BM_UFlat/0                             48009     41283  2.3GB/s  html                   +16.3%
BM_UFlat/1                            531274    513419  1.3GB/s  urls                    +3.5%
BM_UFlat/2                              7378      7062  16.8GB/s  jpg                    +4.5%
BM_UFlat/3                                92        92  2.0GB/s  jpg_200                 +0.0%
BM_UFlat/4                             15057     14974  5.9GB/s  pdf                     +0.6%
BM_UFlat/5                            204323    193140  2.0GB/s  html4                   +5.8%
BM_UFlat/6                             13282     12611  1.8GB/s  cp                      +5.3%
BM_UFlat/7                              6511      6504  1.6GB/s  c                       +0.1%
BM_UFlat/8                              2014      2030  1.7GB/s  lsp                     -0.8%
BM_UFlat/9                            775909    768336  1.3GB/s  xls                     +1.0%
BM_UFlat/10                              182       184  1043.2MB/s  xls_200              -1.1%
BM_UFlat/11                           167352    161630  901.2MB/s  txt1                  +3.5%
BM_UFlat/12                           147393    142246  842.8MB/s  txt2                  +3.6%
BM_UFlat/13                           449960    432853  944.4MB/s  txt3                  +4.0%
BM_UFlat/14                           620497    594845  775.9MB/s  txt4                  +4.3%
BM_UFlat/15                           265610    267356  1.8GB/s  bin                     -0.7%
BM_UFlat/16                              206       205  932.7MB/s  bin_200               +0.5%
BM_UFlat/17                            25561     24730  1.4GB/s  sum                     +3.4%
BM_UFlat/18                             2620      2644  1.5GB/s  man                     -0.9%
BM_UFlat/19                            45766     38589  2.9GB/s  pb                     +18.6%
BM_UFlat/20                           171107    169832  1039.5MB/s  gaviota              +0.8%
Sum of all benchmarks                3500103   3394565                                   +3.1%


Westmere:

Benchmark                          Base (ns)  New (ns)                                Improvement
-------------------------------------------------------------------------------------------------
BM_UFlat/0                             72624     71526  1.3GB/s  html                    +1.5%
BM_UFlat/1                            735821    722917  930.8MB/s  urls                  +1.8%
BM_UFlat/2                             10450     10172  11.7GB/s  jpg                    +2.7%
BM_UFlat/3                               117       117  1.6GB/s  jpg_200                 +0.0%
BM_UFlat/4                             29817     29648  3.0GB/s  pdf                     +0.6%
BM_UFlat/5                            297126    293073  1.3GB/s  html4                   +1.4%
BM_UFlat/6                             28252     27994  842.0MB/s  cp                    +0.9%
BM_UFlat/7                             12672     12391  862.1MB/s  c                     +2.3%
BM_UFlat/8                              3507      3425  1040.9MB/s  lsp                  +2.4%
BM_UFlat/9                           1004268    969395  1018.0MB/s  xls                  +3.6%
BM_UFlat/10                              233       227  844.8MB/s  xls_200               +2.6%
BM_UFlat/11                           230054    224981  647.8MB/s  txt1                  +2.3%
BM_UFlat/12                           201229    196447  610.5MB/s  txt2                  +2.4%
BM_UFlat/13                           609547    596761  685.3MB/s  txt3                  +2.1%
BM_UFlat/14                           824362    804821  573.8MB/s  txt4                  +2.4%
BM_UFlat/15                           371095    374899  1.3GB/s  bin                     -1.0%
BM_UFlat/16                              267       267  717.8MB/s  bin_200               +0.0%
BM_UFlat/17                            44623     43828  835.9MB/s  sum                   +1.8%
BM_UFlat/18                             5077      4815  841.0MB/s  man                   +5.4%
BM_UFlat/19                            74964     73210  1.5GB/s  pb                      +2.4%
BM_UFlat/20                           237987    236745  746.0MB/s  gaviota               +0.5%
Sum of all benchmarks                4794092   4697659                                   +2.1%


Istanbul:

Benchmark                          Base (ns)  New (ns)                                Improvement
-------------------------------------------------------------------------------------------------
BM_UFlat/0                             98614     96376  1020.4MB/s  html                 +2.3%
BM_UFlat/1                            963740    953241  707.2MB/s  urls                  +1.1%
BM_UFlat/2                             25042     24769  4.8GB/s  jpg                     +1.1%
BM_UFlat/3                               180       180  1065.6MB/s  jpg_200              +0.0%
BM_UFlat/4                             45942     45403  1.9GB/s  pdf                     +1.2%
BM_UFlat/5                            400135    390226  1008.2MB/s  html4                +2.5%
BM_UFlat/6                             37768     37392  631.9MB/s  cp                    +1.0%
BM_UFlat/7                             18585     18200  588.2MB/s  c                     +2.1%
BM_UFlat/8                              5751      5690  627.7MB/s  lsp                   +1.1%
BM_UFlat/9                           1543154   1542209  641.4MB/s  xls                   +0.1%
BM_UFlat/10                              381       388  494.6MB/s  xls_200               -1.8%
BM_UFlat/11                           339715    331973  440.1MB/s  txt1                  +2.3%
BM_UFlat/12                           294807    289418  415.4MB/s  txt2                  +1.9%
BM_UFlat/13                           906160    884094  463.3MB/s  txt3                  +2.5%
BM_UFlat/14                          1224221   1198435  386.1MB/s  txt4                  +2.2%
BM_UFlat/15                           516277    502923  979.5MB/s  bin                   +2.7%
BM_UFlat/16                              405       402  477.2MB/s  bin_200               +0.7%
BM_UFlat/17                            61640     60621  605.6MB/s  sum                   +1.7%
BM_UFlat/18                             7326      7383  549.5MB/s  man                   -0.8%
BM_UFlat/19                            94720     92653  1.2GB/s  pb                      +2.2%
BM_UFlat/20                           360435    346687  510.6MB/s  gaviota               +4.0%
Sum of all benchmarks                6944998   6828663                                   +1.7%


git-svn-id: https://snappy.googlecode.com/svn/trunk@77 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2013-06-14 21:42:26 +00:00
snappy.mirrorbot@gmail.com 328aafa198 Add support for uncompressing to iovecs (scatter I/O).
Windows does not have struct iovec defined anywhere,
so we define our own version that's equal to what UNIX
typically has.

The bulk of this patch was contributed by Mohit Aron.

R=jeff


git-svn-id: https://snappy.googlecode.com/svn/trunk@76 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2013-06-13 16:19:52 +00:00
snappy.mirrorbot@gmail.com cd92eb0852 Some code reorganization needed for an internal change.
R=fikes


git-svn-id: https://snappy.googlecode.com/svn/trunk@75 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2013-06-12 19:51:15 +00:00
snappy.mirrorbot@gmail.com 698af469b4 Change a few ORs to additions where they don't matter. This helps the compiler
use the LEA instruction more efficiently, since e.g. a + (b << 2) can be encoded
as one instruction. Even more importantly, it can constant-fold the
COPY_* enums together with the shifted negative constants, which also saves
some instructions. (We don't need it for LITERAL, since it happens to be 0.)

I am unsure why the compiler couldn't do this itself, but the theory is that
it cannot prove that len-1 and len-4 cannot underflow/wrap, and thus can't
do the optimization safely.

The gains are small but measurable; 0.5-1.0% over the BM_Z* benchmarks
(measured on Westmere, Sandy Bridge and Istanbul).

R=sanjay


git-svn-id: https://snappy.googlecode.com/svn/trunk@69 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2013-01-04 11:54:20 +00:00
snappy.mirrorbot@gmail.com 8b95464146 Snappy library no longer depends on iostream.
Achieved by moving logging macro definitions to a test-only
header file, and by changing non-test code to use assert,
fprintf, and abort instead of LOG/CHECK macros.

R=sesse


git-svn-id: https://snappy.googlecode.com/svn/trunk@62 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2012-05-22 09:32:50 +00:00
snappy.mirrorbot@gmail.com dc63e0ad96 For 32-bit platforms, do not try to accelerate multiple neighboring
32-bit loads with a 64-bit load during compression (it's not a win).

The main target for this optimization is ARM, but 32-bit x86 gets
a small gain, too, although there is noise in the microbenchmarks.
It's a no-op for 64-bit x86. It does not affect decompression.

Microbenchmark results on a Cortex-A9 1GHz, using g++ 4.6.2 (from
Ubuntu/Linaro), -O2 -DNDEBUG -Wa,-march=armv7a -mtune=cortex-a9
-mthumb-interwork, minimum 1000 iterations:

  Benchmark            Time(ns)    CPU(ns) Iterations
  ---------------------------------------------------
  BM_ZFlat/0            1158277    1160000       1000 84.2MB/s  html (23.57 %)    [ +4.3%]
  BM_ZFlat/1           14861782   14860000       1000 45.1MB/s  urls (50.89 %)    [ +1.1%]
  BM_ZFlat/2             393595     390000       1000 310.5MB/s  jpg (99.88 %)    [ +0.0%]
  BM_ZFlat/3             650583     650000       1000 138.4MB/s  pdf (82.13 %)    [ +3.1%]
  BM_ZFlat/4            4661480    4660000       1000 83.8MB/s  html4 (23.55 %)   [ +4.3%]
  BM_ZFlat/5             491973     490000       1000 47.9MB/s  cp (48.12 %)      [ +2.0%]
  BM_ZFlat/6             193575     192678       1038 55.2MB/s  c (42.40 %)       [ +9.0%]
  BM_ZFlat/7              62343      62754       3187 56.5MB/s  lsp (48.37 %)     [ +2.6%]
  BM_ZFlat/8           17708468   17710000       1000 55.5MB/s  xls (41.34 %)     [ -0.3%]
  BM_ZFlat/9            3755345    3760000       1000 38.6MB/s  txt1 (59.81 %)    [ +8.2%]
  BM_ZFlat/10           3324217    3320000       1000 36.0MB/s  txt2 (64.07 %)    [ +4.2%]
  BM_ZFlat/11          10139932   10140000       1000 40.1MB/s  txt3 (57.11 %)    [ +6.4%]
  BM_ZFlat/12          13532109   13530000       1000 34.0MB/s  txt4 (68.35 %)    [ +5.0%]
  BM_ZFlat/13           4690847    4690000       1000 104.4MB/s  bin (18.21 %)    [ +4.1%]
  BM_ZFlat/14            830682     830000       1000 43.9MB/s  sum (51.88 %)     [ +1.2%]
  BM_ZFlat/15             84784      85011       2235 47.4MB/s  man (59.36 %)     [ +1.1%]
  BM_ZFlat/16           1293254    1290000       1000 87.7MB/s  pb (23.15 %)      [ +2.3%]
  BM_ZFlat/17           2775155    2780000       1000 63.2MB/s  gaviota (38.27 %) [+12.2%]

Core i7 in 32-bit mode (only one run and 100 iterations, though, so noisy):

  Benchmark            Time(ns)    CPU(ns) Iterations
  ---------------------------------------------------
  BM_ZFlat/0             227582     223464       3043 437.0MB/s  html (23.57 %)    [ +7.4%]
  BM_ZFlat/1            2982430    2918455        233 229.4MB/s  urls (50.89 %)    [ +2.9%]
  BM_ZFlat/2              46967      46658      15217 2.5GB/s  jpg (99.88 %)       [ +0.0%]
  BM_ZFlat/3             115298     114864       5833 783.2MB/s  pdf (82.13 %)     [ +1.5%]
  BM_ZFlat/4             913440     899743        778 434.2MB/s  html4 (23.55 %)   [ +0.3%]
  BM_ZFlat/5             110302     108571       7000 216.1MB/s  cp (48.12 %)      [ +0.0%]
  BM_ZFlat/6              44409      43372      15909 245.2MB/s  c (42.40 %)       [ +0.8%]
  BM_ZFlat/7              15713      15643      46667 226.9MB/s  lsp (48.37 %)     [ +2.7%]
  BM_ZFlat/8            2625539    2602230        269 377.4MB/s  xls (41.34 %)     [ +1.4%]
  BM_ZFlat/9             808884     811429        875 178.8MB/s  txt1 (59.81 %)    [ -3.9%]
  BM_ZFlat/10            709532     700000       1000 170.5MB/s  txt2 (64.07 %)    [ +0.0%]
  BM_ZFlat/11           2177682    2162162        333 188.2MB/s  txt3 (57.11 %)    [ -1.4%]
  BM_ZFlat/12           2849640    2840000        250 161.8MB/s  txt4 (68.35 %)    [ -1.4%]
  BM_ZFlat/13            849760     835476        778 585.8MB/s  bin (18.21 %)     [ +1.2%]
  BM_ZFlat/14            165940     164571       4375 221.6MB/s  sum (51.88 %)     [ +1.4%]
  BM_ZFlat/15             20939      20571      35000 196.0MB/s  man (59.36 %)     [ +2.1%]
  BM_ZFlat/16            239209     236544       2917 478.1MB/s  pb (23.15 %)      [ +4.2%]
  BM_ZFlat/17            616206     610000       1000 288.2MB/s  gaviota (38.27 %) [ -1.6%]

R=sanjay


git-svn-id: https://snappy.googlecode.com/svn/trunk@60 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2012-02-23 17:00:36 +00:00
snappy.mirrorbot@gmail.com f8829ea39d Enable the use of unaligned loads and stores for ARM-based architectures
where they are available (ARMv7 and higher). This gives a significant 
speed boost on ARM, both for compression and decompression. 
It should not affect x86 at all. 
 
There are more changes possible to speed up ARM, but it might not be 
that easy to do without hurting x86 or making the code uglier. 
Also, we de not try to use NEON yet. 
 
Microbenchmark results on a Cortex-A9 1GHz, using g++ 4.6.2 (from Ubuntu/Linaro), 
-O2 -DNDEBUG -Wa,-march=armv7a -mtune=cortex-a9 -mthumb-interwork: 
 
Benchmark            Time(ns)    CPU(ns) Iterations
---------------------------------------------------
BM_UFlat/0             524806     529100        378 184.6MB/s  html            [+33.6%]
BM_UFlat/1            5139790    5200000        100 128.8MB/s  urls            [+28.8%]
BM_UFlat/2              86540      84166       1901 1.4GB/s  jpg               [ +0.6%]
BM_UFlat/3             215351     210176        904 428.0MB/s  pdf             [+29.8%]
BM_UFlat/4            2144490    2100000        100 186.0MB/s  html4           [+33.3%]
BM_UFlat/5             194482     190000       1000 123.5MB/s  cp              [+36.2%]
BM_UFlat/6              91843      90175       2107 117.9MB/s  c               [+38.6%]
BM_UFlat/7              28535      28426       6684 124.8MB/s  lsp             [+34.7%]
BM_UFlat/8            9206600    9200000        100 106.7MB/s  xls             [+42.4%]
BM_UFlat/9            1865273    1886792        106 76.9MB/s  txt1             [+32.5%]
BM_UFlat/10           1576809    1587301        126 75.2MB/s  txt2             [+32.3%]
BM_UFlat/11           4968450    4900000        100 83.1MB/s  txt3             [+32.7%]
BM_UFlat/12           6673970    6700000        100 68.6MB/s  txt4             [+32.8%]
BM_UFlat/13           2391470    2400000        100 203.9MB/s  bin             [+29.2%]
BM_UFlat/14            334601     344827        522 105.8MB/s  sum             [+30.6%]
BM_UFlat/15             37404      38080       5252 105.9MB/s  man             [+33.8%]
BM_UFlat/16            535470     540540        370 209.2MB/s  pb              [+31.2%]
BM_UFlat/17           1875245    1886792        106 93.2MB/s  gaviota          [+37.8%]
BM_UValidate/0         178425     179533       1114 543.9MB/s  html            [ +2.7%]
BM_UValidate/1        2100450    2000000        100 334.8MB/s  urls            [ +5.0%]
BM_UValidate/2           1039       1044     172413 113.3GB/s  jpg             [ +3.4%]
BM_UValidate/3          59423      59470       3363 1.5GB/s  pdf               [ +7.8%]
BM_UValidate/4         760716     766283        261 509.8MB/s  html4           [ +6.5%]
BM_ZFlat/0            1204632    1204819        166 81.1MB/s  html (23.57 %)   [+32.8%]
BM_ZFlat/1           15656190   15600000        100 42.9MB/s  urls (50.89 %)   [+27.6%]
BM_ZFlat/2             403336     410677        487 294.8MB/s  jpg (99.88 %)   [+16.5%]
BM_ZFlat/3             664073     671140        298 134.0MB/s  pdf (82.13 %)   [+28.4%]
BM_ZFlat/4            4961940    4900000        100 79.7MB/s  html4 (23.55 %)  [+30.6%]
BM_ZFlat/5             500664     501253        399 46.8MB/s  cp (48.12 %)     [+33.4%]
BM_ZFlat/6             217276     215982        926 49.2MB/s  c (42.40 %)      [+25.0%]
BM_ZFlat/7              64122      65487       3054 54.2MB/s  lsp (48.37 %)    [+36.1%]
BM_ZFlat/8           18045730   18000000        100 54.6MB/s  xls (41.34 %)    [+34.4%]
BM_ZFlat/9            4051530    4000000        100 36.3MB/s  txt1 (59.81 %)   [+25.0%]
BM_ZFlat/10           3451800    3500000        100 34.1MB/s  txt2 (64.07 %)   [+25.7%]
BM_ZFlat/11          11052340   11100000        100 36.7MB/s  txt3 (57.11 %)   [+24.3%]
BM_ZFlat/12          14538690   14600000        100 31.5MB/s  txt4 (68.35 %)   [+24.7%]
BM_ZFlat/13           5041850    5000000        100 97.9MB/s  bin (18.21 %)    [+32.0%]
BM_ZFlat/14            908840     909090        220 40.1MB/s  sum (51.88 %)    [+22.2%]
BM_ZFlat/15             86921      86206       1972 46.8MB/s  man (59.36 %)    [+42.2%]
BM_ZFlat/16           1312315    1315789        152 86.0MB/s  pb (23.15 %)     [+34.5%]
BM_ZFlat/17           3173120    3200000        100 54.9MB/s  gaviota (38.27%) [+28.1%]


The move from 64-bit to 32-bit operations for the copies also affected 32-bit x86;
positive on the decompression side, and slightly negative on the compression side
(unless that is noise; I only ran once):

Benchmark              Time(ns)    CPU(ns) Iterations
-----------------------------------------------------
BM_UFlat/0                86279      86140       7778 1.1GB/s  html             [ +7.5%]
BM_UFlat/1               839265     822622        778 813.9MB/s  urls           [ +9.4%]
BM_UFlat/2                 9180       9143      87500 12.9GB/s  jpg             [ +1.2%]
BM_UFlat/3                35080      35000      20000 2.5GB/s  pdf              [+10.1%]
BM_UFlat/4               350318     345000       2000 1.1GB/s  html4            [ +7.0%]
BM_UFlat/5                33808      33472      21212 701.0MB/s  cp             [ +9.0%]
BM_UFlat/6                15201      15214      46667 698.9MB/s  c              [+14.9%]
BM_UFlat/7                 4652       4651     159091 762.9MB/s  lsp            [ +7.5%]
BM_UFlat/8              1285551    1282528        538 765.7MB/s  xls            [+10.7%]
BM_UFlat/9               282510     281690       2414 514.9MB/s  txt1           [+13.6%]
BM_UFlat/10              243494     239286       2800 498.9MB/s  txt2           [+14.4%]
BM_UFlat/11              743625     740000       1000 550.0MB/s  txt3           [+14.3%]
BM_UFlat/12              999441     989717        778 464.3MB/s  txt4           [+16.1%]
BM_UFlat/13              412402     410076       1707 1.2GB/s  bin              [ +7.3%]
BM_UFlat/14               54876      54000      10000 675.3MB/s  sum            [+13.0%]
BM_UFlat/15                6146       6100     100000 660.8MB/s  man            [+14.8%]
BM_UFlat/16               90496      90286       8750 1.2GB/s  pb               [ +4.0%]
BM_UFlat/17              292650     292000       2500 602.0MB/s  gaviota        [+18.1%]
BM_UValidate/0            49620      49699      14286 1.9GB/s  html             [ +0.0%]
BM_UValidate/1           501371     500000       1000 1.3GB/s  urls             [ +0.0%]
BM_UValidate/2              232        227    3043478 521.5GB/s  jpg            [ +1.3%]
BM_UValidate/3            17250      17143      43750 5.1GB/s  pdf              [ -1.3%]
BM_UValidate/4           198643     200000       3500 1.9GB/s  html4            [ -0.9%]
BM_ZFlat/0               227128     229415       3182 425.7MB/s  html (23.57 %) [ -1.4%]
BM_ZFlat/1              2970089    2960000        250 226.2MB/s  urls (50.89 %) [ -1.9%]
BM_ZFlat/2                45683      44999      15556 2.6GB/s  jpg (99.88 %)    [ +2.2%]
BM_ZFlat/3               114661     113136       6364 795.1MB/s  pdf (82.13 %)  [ -1.5%]
BM_ZFlat/4               919702     914286        875 427.2MB/s  html4 (23.55%) [ -1.3%]
BM_ZFlat/5               108189     108422       6364 216.4MB/s  cp (48.12 %)   [ -1.2%]
BM_ZFlat/6                44525      44000      15909 241.7MB/s  c (42.40 %)    [ -2.9%]
BM_ZFlat/7                15973      15857      46667 223.8MB/s  lsp (48.37 %)  [ +0.0%]
BM_ZFlat/8              2677888    2639405        269 372.1MB/s  xls (41.34 %)  [ -1.4%]
BM_ZFlat/9               800715     780000       1000 186.0MB/s  txt1 (59.81 %) [ -0.4%]
BM_ZFlat/10              700089     700000       1000 170.5MB/s  txt2 (64.07 %) [ -2.9%]
BM_ZFlat/11             2159356    2138365        318 190.3MB/s  txt3 (57.11 %) [ -0.3%]
BM_ZFlat/12             2796143    2779923        259 165.3MB/s  txt4 (68.35 %) [ -1.4%]
BM_ZFlat/13              856458     835476        778 585.8MB/s  bin (18.21 %)  [ -0.1%]
BM_ZFlat/14              166908     166857       4375 218.6MB/s  sum (51.88 %)  [ -1.4%]
BM_ZFlat/15               21181      20857      35000 193.3MB/s  man (59.36 %)  [ -0.8%]
BM_ZFlat/16              244009     239973       2917 471.3MB/s  pb (23.15 %)   [ -1.4%]
BM_ZFlat/17              596362     590000       1000 297.9MB/s  gaviota (38.27%) [ +0.0%]

R=sanjay


git-svn-id: https://snappy.googlecode.com/svn/trunk@59 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2012-02-21 17:02:17 +00:00
snappy.mirrorbot@gmail.com e750dc0f05 Minor refactoring to accomodate changes in Google's internal code tree.
git-svn-id: https://snappy.googlecode.com/svn/trunk@57 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2012-01-08 17:55:48 +00:00
snappy.mirrorbot@gmail.com d9068ee301 Fix public issue r57: Fix most warnings with -Wall, mostly signed/unsigned
warnings. There are still some in the unit test, but the main .cc file should
be clean. We haven't enabled -Wall for the default build, since the unit test
is still not clean.

This also fixes a real bug in the open-source implementation of
ReadFileToStringOrDie(); it would not detect errors correctly.

I had to go through some pains to avoid performance loss as the types
were changed; I think there might still be some with 32-bit if and only if LFS
is enabled (ie., size_t is 64-bit), but for regular 32-bit and 64-bit I can't
see any losses, and I've diffed the generated GCC assembler between the old and
new code without seeing any significant choices. If anything, it's ever so
slightly faster.

This may or may not enable compression of very large blocks (>2^32 bytes)
when size_t is 64-bit, but I haven't checked, and it is still not a supported
case.


git-svn-id: https://snappy.googlecode.com/svn/trunk@56 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2012-01-04 13:10:46 +00:00
snappy.mirrorbot@gmail.com d7eb2dc413 Speed up decompression by moving the refill check to the end of the loop.
This seems to work because in most of the branches, the compiler can evaluate
“ip_limit_ - ip” in a more efficient way than reloading ip_limit_ from memory
(either by already having the entire expression in a register, or reconstructing
it from “avail”, or something else). Memory loads, even from L1, are seemingly
costly in the big picture at the current decompression speeds.

Microbenchmarks (64-bit, opt mode):

Westmere (Intel Core i7):

  Benchmark     Time(ns)    CPU(ns) Iterations
  --------------------------------------------
  BM_UFlat/0       74492      74491     187894 1.3GB/s  html      [ +5.9%]
  BM_UFlat/1      712268     712263      19644 940.0MB/s  urls    [ +3.8%]
  BM_UFlat/2       10591      10590    1000000 11.2GB/s  jpg      [ -6.8%]
  BM_UFlat/3       29643      29643     469915 3.0GB/s  pdf       [ +7.9%]
  BM_UFlat/4      304669     304667      45930 1.3GB/s  html4     [ +4.8%]
  BM_UFlat/5       28508      28507     490077 823.1MB/s  cp      [ +4.0%]
  BM_UFlat/6       12415      12415    1000000 856.5MB/s  c       [ +8.6%]
  BM_UFlat/7        3415       3415    4084723 1039.0MB/s  lsp    [+18.0%]
  BM_UFlat/8      979569     979563      14261 1002.5MB/s  xls    [ +5.8%]
  BM_UFlat/9      230150     230148      60934 630.2MB/s  txt1    [ +5.2%]
  BM_UFlat/10     197167     197166      71135 605.5MB/s  txt2    [ +4.7%]
  BM_UFlat/11     607394     607390      23041 670.1MB/s  txt3    [ +5.6%]
  BM_UFlat/12     808502     808496      17316 568.4MB/s  txt4    [ +5.0%]
  BM_UFlat/13     372791     372788      37564 1.3GB/s  bin       [ +3.3%]
  BM_UFlat/14      44541      44541     313969 818.8MB/s  sum     [ +5.7%]
  BM_UFlat/15       4833       4833    2898697 834.1MB/s  man     [ +4.8%]
  BM_UFlat/16      79855      79855     175356 1.4GB/s  pb        [ +4.8%]
  BM_UFlat/17     245845     245843      56838 715.0MB/s  gaviota [ +5.8%]

Clovertown (Intel Core 2):

  Benchmark     Time(ns)    CPU(ns) Iterations
  --------------------------------------------
  BM_UFlat/0      107911     107890     100000 905.1MB/s  html    [ +2.2%]
  BM_UFlat/1     1011237    1011041      10000 662.3MB/s  urls    [ +2.5%]
  BM_UFlat/2       26775      26770     523089 4.4GB/s  jpg       [ +0.0%]
  BM_UFlat/3       48103      48095     290618 1.8GB/s  pdf       [ +3.4%]
  BM_UFlat/4      437724     437644      31937 892.6MB/s  html4   [ +2.1%]
  BM_UFlat/5       39607      39600     358284 592.5MB/s  cp      [ +2.4%]
  BM_UFlat/6       18227      18224     768191 583.5MB/s  c       [ +2.7%]
  BM_UFlat/7        5171       5170    2709437 686.4MB/s  lsp     [ +3.9%]
  BM_UFlat/8     1560291    1559989       8970 629.5MB/s  xls     [ +3.6%]
  BM_UFlat/9      335401     335343      41731 432.5MB/s  txt1    [ +3.0%]
  BM_UFlat/10     287014     286963      48758 416.0MB/s  txt2    [ +2.8%]
  BM_UFlat/11     888522     888356      15752 458.1MB/s  txt3    [ +2.9%]
  BM_UFlat/12    1186600    1186378      10000 387.3MB/s  txt4    [ +3.1%]
  BM_UFlat/13     572295     572188      24468 855.4MB/s  bin     [ +2.1%]
  BM_UFlat/14      64060      64049     218401 569.4MB/s  sum     [ +4.1%]
  BM_UFlat/15       7264       7263    1916168 555.0MB/s  man     [ +1.4%]
  BM_UFlat/16     108853     108836     100000 1039.1MB/s  pb     [ +1.7%]
  BM_UFlat/17     364289     364223      38419 482.6MB/s  gaviota [ +4.9%]

Barcelona (AMD Opteron):

  Benchmark     Time(ns)    CPU(ns) Iterations
  --------------------------------------------
  BM_UFlat/0      103900     103871     100000 940.2MB/s  html    [ +8.3%]
  BM_UFlat/1     1000435    1000107      10000 669.5MB/s  urls    [ +6.6%]
  BM_UFlat/2       24659      24652     567362 4.8GB/s  jpg       [ +0.1%]
  BM_UFlat/3       48206      48193     291121 1.8GB/s  pdf       [ +5.0%]
  BM_UFlat/4      421980     421850      33174 926.0MB/s  html4   [ +7.3%]
  BM_UFlat/5       40368      40357     346994 581.4MB/s  cp      [ +8.7%]
  BM_UFlat/6       19836      19830     708695 536.2MB/s  c       [ +8.0%]
  BM_UFlat/7        6100       6098    2292774 581.9MB/s  lsp     [ +9.0%]
  BM_UFlat/8     1693093    1692514       8261 580.2MB/s  xls     [ +8.0%]
  BM_UFlat/9      365991     365886      38225 396.4MB/s  txt1    [ +7.1%]
  BM_UFlat/10     311330     311238      44950 383.6MB/s  txt2    [ +7.6%]
  BM_UFlat/11     975037     974737      14376 417.5MB/s  txt3    [ +6.9%]
  BM_UFlat/12    1303558    1303175      10000 352.6MB/s  txt4    [ +7.3%]
  BM_UFlat/13     517448     517290      27144 946.2MB/s  bin     [ +5.5%]
  BM_UFlat/14      66537      66518     210352 548.3MB/s  sum     [ +7.5%]
  BM_UFlat/15       7976       7974    1760383 505.6MB/s  man     [ +5.6%]
  BM_UFlat/16     103121     103092     100000 1097.0MB/s  pb     [ +8.7%]
  BM_UFlat/17     391431     391314      35733 449.2MB/s  gaviota [ +6.5%]

R=sanjay


git-svn-id: https://snappy.googlecode.com/svn/trunk@54 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-12-05 21:27:26 +00:00
snappy.mirrorbot@gmail.com 5ed51ce15f Speed up decompression by making the fast path for literals faster.
We do the fast-path step as soon as possible; in fact, as soon as we know the
literal length. Since we usually hit the fast path, we can then skip the checks
for long literals and available input space (beyond what the fast path check
already does).

Note that this changes the decompression Writer API; however, it does not
change the ABI, since writers are always templatized and as such never
cross compilation units. The new API is slightly more general, in that it
doesn't hard-code the value 16. Note that we also take care to check
for len <= 16 first, since the other two checks almost always succeed
(so we don't want to waste time checking for them until we have to).

The improvements are most marked on Nehalem, but are generally positive
on other platforms as well. All microbenchmarks are 64-bit, opt.

Clovertown (Core 2):

  Benchmark     Time(ns)    CPU(ns) Iterations
  --------------------------------------------
  BM_UFlat/0      110226     110224     100000 886.0MB/s  html    [ +1.5%]
  BM_UFlat/1     1036523    1036508      10000 646.0MB/s  urls    [ -0.8%]
  BM_UFlat/2       26775      26775     522570 4.4GB/s  jpg       [ +0.0%]
  BM_UFlat/3       49738      49737     280974 1.8GB/s  pdf       [ +0.3%]
  BM_UFlat/4      446790     446792      31334 874.3MB/s  html4   [ +0.8%]
  BM_UFlat/5       40561      40562     350424 578.5MB/s  cp      [ +1.3%]
  BM_UFlat/6       18722      18722     746903 568.0MB/s  c       [ +1.4%]
  BM_UFlat/7        5373       5373    2608632 660.5MB/s  lsp     [ +8.3%]
  BM_UFlat/8     1615716    1615718       8670 607.8MB/s  xls     [ +2.0%]
  BM_UFlat/9      345278     345281      40481 420.1MB/s  txt1    [ +1.4%]
  BM_UFlat/10     294855     294855      47452 404.9MB/s  txt2    [ +1.6%]
  BM_UFlat/11     914263     914263      15316 445.2MB/s  txt3    [ +1.1%]
  BM_UFlat/12    1222694    1222691      10000 375.8MB/s  txt4    [ +1.4%]
  BM_UFlat/13     584495     584489      23954 837.4MB/s  bin     [ -0.6%]
  BM_UFlat/14      66662      66662     210123 547.1MB/s  sum     [ +1.2%]
  BM_UFlat/15       7368       7368    1881856 547.1MB/s  man     [ +4.0%]
  BM_UFlat/16     110727     110726     100000 1021.4MB/s  pb     [ +2.3%]
  BM_UFlat/17     382138     382141      36616 460.0MB/s  gaviota [ -0.7%]

Westmere (Core i7):

  Benchmark     Time(ns)    CPU(ns) Iterations
  --------------------------------------------
  BM_UFlat/0       78861      78853     177703 1.2GB/s  html      [ +2.1%]
  BM_UFlat/1      739560     739491      18912 905.4MB/s  urls    [ +3.4%]
  BM_UFlat/2        9867       9866    1419014 12.0GB/s  jpg      [ +3.4%]
  BM_UFlat/3       31989      31986     438385 2.7GB/s  pdf       [ +0.2%]
  BM_UFlat/4      319406     319380      43771 1.2GB/s  html4     [ +1.9%]
  BM_UFlat/5       29639      29636     472862 791.7MB/s  cp      [ +5.2%]
  BM_UFlat/6       13478      13477    1000000 789.0MB/s  c       [ +2.3%]
  BM_UFlat/7        4030       4029    3475364 880.7MB/s  lsp     [ +8.7%]
  BM_UFlat/8     1036585    1036492      10000 947.5MB/s  xls     [ +6.9%]
  BM_UFlat/9      242127     242105      57838 599.1MB/s  txt1    [ +3.0%]
  BM_UFlat/10     206499     206480      67595 578.2MB/s  txt2    [ +3.4%]
  BM_UFlat/11     641635     641570      21811 634.4MB/s  txt3    [ +2.4%]
  BM_UFlat/12     848847     848769      16443 541.4MB/s  txt4    [ +3.1%]
  BM_UFlat/13     384968     384938      36366 1.2GB/s  bin       [ +0.3%]
  BM_UFlat/14      47106      47101     297770 774.3MB/s  sum     [ +4.4%]
  BM_UFlat/15       5063       5063    2772202 796.2MB/s  man     [ +7.7%]
  BM_UFlat/16      83663      83656     167697 1.3GB/s  pb        [ +1.8%]
  BM_UFlat/17     260224     260198      53823 675.6MB/s  gaviota [ -0.5%]

Barcelona (Opteron):

  Benchmark     Time(ns)    CPU(ns) Iterations
  --------------------------------------------
  BM_UFlat/0      112490     112457     100000 868.4MB/s  html    [ -0.4%]
  BM_UFlat/1     1066719    1066339      10000 627.9MB/s  urls    [ +1.0%]
  BM_UFlat/2       24679      24672     563802 4.8GB/s  jpg       [ +0.7%]
  BM_UFlat/3       50603      50589     277285 1.7GB/s  pdf       [ +2.6%]
  BM_UFlat/4      452982     452849      30900 862.6MB/s  html4   [ -0.2%]
  BM_UFlat/5       43860      43848     319554 535.1MB/s  cp      [ +1.2%]
  BM_UFlat/6       21419      21413     653573 496.6MB/s  c       [ +1.0%]
  BM_UFlat/7        6646       6645    2105405 534.1MB/s  lsp     [ +0.3%]
  BM_UFlat/8     1828487    1827886       7658 537.3MB/s  xls     [ +2.6%]
  BM_UFlat/9      391824     391714      35708 370.3MB/s  txt1    [ +2.2%]
  BM_UFlat/10     334913     334816      41885 356.6MB/s  txt2    [ +1.7%]
  BM_UFlat/11    1042062    1041674      10000 390.7MB/s  txt3    [ +1.1%]
  BM_UFlat/12    1398902    1398456      10000 328.6MB/s  txt4    [ +1.7%]
  BM_UFlat/13     545706     545530      25669 897.2MB/s  bin     [ -0.4%]
  BM_UFlat/14      71512      71505     196035 510.0MB/s  sum     [ +1.4%]
  BM_UFlat/15       8422       8421    1665036 478.7MB/s  man     [ +2.6%]
  BM_UFlat/16     112053     112048     100000 1009.3MB/s  pb     [ -0.4%]
  BM_UFlat/17     416723     416713      33612 421.8MB/s  gaviota [ -2.0%]

R=sanjay


git-svn-id: https://snappy.googlecode.com/svn/trunk@53 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-11-23 11:14:17 +00:00
snappy.mirrorbot@gmail.com 57e7cd7255 Fix public issue #44: Make the definition and declaration of CompressFragment
identical, even regarding cv-qualifiers.

This is required to work around a bug in the Solaris Studio C++ compiler
(it does not properly disregard cv-qualifiers when doing name mangling).

R=sanjay


git-svn-id: https://snappy.googlecode.com/svn/trunk@44 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-06-28 11:40:25 +00:00
snappy.mirrorbot@gmail.com f540673740 Speed up decompression by removing a fast-path attempt.
Whenever we try to enter a copy fast-path, there is a certain cost in checking
that all the preconditions are in place, but it's normally offset by the fact
that we can usually take the cheaper path. However, in a certain path we've
already established that "avail < literal_length", which usually means that
either the available space is small, or the literal is big. Both will disqualify
us from taking the fast path, and thus we take the hit from the precondition
checking without gaining much from having a fast path. Thus, simply don't try
the fast path in this situation -- we're already on a slow path anyway
(one where we need to refill more data from the reader).

I'm a bit surprised at how much this gained; it could be that this path is
more common than I thought, or that the simpler structure somehow makes the
compiler happier. I haven't looked at the assembler, but it's a win across
the board on both Core 2, Core i7 and Opteron, at least for the cases we
typically care about. The gains seem to be the largest on Core i7, though.
Results from my Core i7 workstation:


  Benchmark            Time(ns)    CPU(ns) Iterations
  ---------------------------------------------------
  BM_UFlat/0              73337      73091     190996 1.3GB/s  html      [ +1.7%]
  BM_UFlat/1             696379     693501      20173 965.5MB/s  urls    [ +2.7%]
  BM_UFlat/2               9765       9734    1472135 12.1GB/s  jpg      [ +0.7%]
  BM_UFlat/3              29720      29621     472973 3.0GB/s  pdf       [ +1.8%]
  BM_UFlat/4             294636     293834      47782 1.3GB/s  html4     [ +2.3%]
  BM_UFlat/5              28399      28320     494700 828.5MB/s  cp      [ +3.5%]
  BM_UFlat/6              12795      12760    1000000 833.3MB/s  c       [ +1.2%]
  BM_UFlat/7               3984       3973    3526448 893.2MB/s  lsp     [ +5.7%]
  BM_UFlat/8             991996     989322      14141 992.6MB/s  xls     [ +3.3%]
  BM_UFlat/9             228620     227835      61404 636.6MB/s  txt1    [ +4.0%]
  BM_UFlat/10            197114     196494      72165 607.5MB/s  txt2    [ +3.5%]
  BM_UFlat/11            605240     603437      23217 674.4MB/s  txt3    [ +3.7%]
  BM_UFlat/12            804157     802016      17456 573.0MB/s  txt4    [ +3.9%]
  BM_UFlat/13            347860     346998      40346 1.4GB/s  bin       [ +1.2%]
  BM_UFlat/14             44684      44559     315315 818.4MB/s  sum     [ +2.3%]
  BM_UFlat/15              5120       5106    2739726 789.4MB/s  man     [ +3.3%]
  BM_UFlat/16             76591      76355     183486 1.4GB/s  pb        [ +2.8%]
  BM_UFlat/17            238564     237828      58824 739.1MB/s  gaviota [ +1.6%]
  BM_UValidate/0          42194      42060     333333 2.3GB/s  html      [ -0.1%]
  BM_UValidate/1         433182     432005      32407 1.5GB/s  urls      [ -0.1%]
  BM_UValidate/2            197        196   71428571 603.3GB/s  jpg     [ +0.5%]
  BM_UValidate/3          14494      14462     972222 6.1GB/s  pdf       [ +0.5%]
  BM_UValidate/4         168444     167836      83832 2.3GB/s  html4     [ +0.1%]
	
R=jeff

Revision created by MOE tool push_codebase.


git-svn-id: https://snappy.googlecode.com/svn/trunk@42 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-06-03 20:53:06 +00:00
snappy.mirrorbot@gmail.com 197f3ee9f9 Speed up decompression by not needing a lookup table for literal items.
Looking up into and decoding the values from char_table has long shown up as a
hotspot in the decompressor. While it turns out that it's hard to make a more
efficient decoder for the copy ops, the literals are simple enough that we can
decode them without needing a table lookup. (This means that 1/4 of the table
is now unused, although that in itself doesn't buy us anything.)

The gains are small, but definitely present; some tests win as much as 10%,
but 1-4% is more typical. These results are from Core i7, in 64-bit mode;
Core 2 and Opteron show similar results. (I've run with more iterations
than unusual to make sure the smaller gains don't drown entirely in noise.)

  Benchmark            Time(ns)    CPU(ns) Iterations
  ---------------------------------------------------
  BM_UFlat/0              74665      74428     182055 1.3GB/s  html      [ +3.1%]
  BM_UFlat/1             714106     711997      19663 940.4MB/s  urls    [ +4.4%]
  BM_UFlat/2               9820       9789    1427115 12.1GB/s  jpg      [ -1.2%]
  BM_UFlat/3              30461      30380     465116 2.9GB/s  pdf       [ +0.8%]
  BM_UFlat/4             301445     300568      46512 1.3GB/s  html4     [ +2.2%]
  BM_UFlat/5              29338      29263     479452 801.8MB/s  cp      [ +1.6%]
  BM_UFlat/6              13004      12970    1000000 819.9MB/s  c       [ +2.1%]
  BM_UFlat/7               4180       4168    3349282 851.4MB/s  lsp     [ +1.3%]
  BM_UFlat/8            1026149    1024000      10000 959.0MB/s  xls     [+10.7%]
  BM_UFlat/9             237441     236830      59072 612.4MB/s  txt1    [ +0.3%]
  BM_UFlat/10            203966     203298      69307 587.2MB/s  txt2    [ +0.8%]
  BM_UFlat/11            627230     625000      22400 651.2MB/s  txt3    [ +0.7%]
  BM_UFlat/12            836188     833979      16787 551.0MB/s  txt4    [ +1.3%]
  BM_UFlat/13            351904     350750      39886 1.4GB/s  bin       [ +3.8%]
  BM_UFlat/14             45685      45562     308370 800.4MB/s  sum     [ +5.9%]
  BM_UFlat/15              5286       5270    2656546 764.9MB/s  man     [ +1.5%]
  BM_UFlat/16             78774      78544     178117 1.4GB/s  pb        [ +4.3%]
  BM_UFlat/17            242270     241345      58091 728.3MB/s  gaviota [ +1.2%]
  BM_UValidate/0          42149      42000     333333 2.3GB/s  html      [ -3.0%]
  BM_UValidate/1         432741     431303      32483 1.5GB/s  urls      [ +7.8%]
  BM_UValidate/2            198        197   71428571 600.7GB/s  jpg     [+16.8%]
  BM_UValidate/3          14560      14521     965517 6.1GB/s  pdf       [ -4.1%]
  BM_UValidate/4         169065     168671      83832 2.3GB/s  html4     [ -2.9%]

R=jeff

Revision created by MOE tool push_codebase.


git-svn-id: https://snappy.googlecode.com/svn/trunk@41 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-06-03 20:47:14 +00:00
snappy.mirrorbot@gmail.com 2e12124bd8 Remove an unneeded goto in the decompressor; it turns out that the
state of ip_ after decompression (or attempted decompresion) is
completely irrelevant, so we don't need the trailer.

Performance is, as expected, mostly flat -- there's a curious ~3-5%
loss in the "lsp" test, but that test case is so short it is hard to say
anything definitive about why (most likely, it's some sort of
unrelated effect).

R=jeff


git-svn-id: https://snappy.googlecode.com/svn/trunk@39 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-06-02 18:06:54 +00:00
snappy.mirrorbot@gmail.com c266bbf321 Speed up decompression by caching ip_.
It is seemingly hard for the compiler to understand that ip_, the current input
pointer into the compressed data stream, can not alias on anything else, and
thus using it directly will incur memory traffic as it cannot be kept in a
register. The code already knew about this and cached it into a local
variable, but since Step() only decoded one tag, it had to move ip_ back into
place between every tag. This seems to have cost us a significant amount of
performance, so changing Step() into a function that decodes as much as it can
before it saves ip_ back and returns. (Note that Step() was already inlined,
so it is not the manual inlining that buys the performance here.)

The wins are about 3-6% for Core 2, 6-13% on Core i7 and 5-12% on Opteron
(for plain array-to-array decompression, in 64-bit opt mode).

There is a tiny difference in the behavior here; if an invalid literal is
encountered (ie., the writer refuses the Append() operation), ip_ will now
point to the byte past the tag byte, instead of where the literal was
originally thought to end. However, we don't use ip_ for anything after
DecompressAllTags() has returned, so this should not change external behavior
in any way.

Microbenchmark results for Core i7, 64-bit (Opteron results are similar):

Benchmark            Time(ns)    CPU(ns) Iterations
---------------------------------------------------
BM_UFlat/0              79134      79110       8835 1.2GB/s  html      [ +6.2%]
BM_UFlat/1             786126     786096        891 851.8MB/s  urls    [+10.0%]
BM_UFlat/2               9948       9948      69125 11.9GB/s  jpg      [ -1.3%]
BM_UFlat/3              31999      31998      21898 2.7GB/s  pdf       [ +6.5%]
BM_UFlat/4             318909     318829       2204 1.2GB/s  html4     [ +6.5%]
BM_UFlat/5              31384      31390      22363 747.5MB/s  cp      [ +9.2%]
BM_UFlat/6              14037      14034      49858 757.7MB/s  c       [+10.6%]
BM_UFlat/7               4612       4612     151395 769.5MB/s  lsp     [ +9.5%]
BM_UFlat/8            1203174    1203007        582 816.3MB/s  xls     [+19.3%]
BM_UFlat/9             253869     253955       2757 571.1MB/s  txt1    [+11.4%]
BM_UFlat/10            219292     219290       3194 544.4MB/s  txt2    [+12.1%]
BM_UFlat/11            672135     672131       1000 605.5MB/s  txt3    [+11.2%]
BM_UFlat/12            902512     902492        776 509.2MB/s  txt4    [+12.5%]
BM_UFlat/13            372110     371998       1881 1.3GB/s  bin       [ +5.8%]
BM_UFlat/14             50407      50407      10000 723.5MB/s  sum     [+13.5%]
BM_UFlat/15              5699       5701     100000 707.2MB/s  man     [+12.4%]
BM_UFlat/16             83448      83424       8383 1.3GB/s  pb        [ +5.7%]
BM_UFlat/17            256958     256963       2723 684.1MB/s  gaviota [ +7.9%]
BM_UValidate/0          42795      42796      16351 2.2GB/s  html      [+25.8%]
BM_UValidate/1         490672     490622       1427 1.3GB/s  urls      [+22.7%]
BM_UValidate/2            237        237    2950297 499.0GB/s  jpg     [+24.9%]
BM_UValidate/3          14610      14611      47901 6.0GB/s  pdf       [+26.8%]
BM_UValidate/4         171973     171990       4071 2.2GB/s  html4     [+25.7%]




git-svn-id: https://snappy.googlecode.com/svn/trunk@38 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-06-02 17:59:40 +00:00
snappy.mirrorbot@gmail.com 7e8ca8f831 Change on 2011-03-25 19:18:00-07:00 by sesse
Replace the Apache 2.0 license header by the BSD-type license header;
	somehow a lot of the files were missed in the last round.

	R=dannyb,csilvers
	DELTA=147  (74 added, 2 deleted, 71 changed)

Change on 2011-03-25 19:25:07-07:00 by sesse

	Unbreak the build; the relicensing removed a bit too much (only comments
	were intended, but I also accidentially removed some of the top lines of
	the actual source).



Revision created by MOE tool push_codebase.
MOE_MIGRATION=1072


git-svn-id: https://snappy.googlecode.com/svn/trunk@21 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-03-26 02:34:34 +00:00
snappy.mirrorbot@gmail.com 28a6440239 Revision created by MOE tool push_codebase.
MOE_MIGRATION=


git-svn-id: https://snappy.googlecode.com/svn/trunk@2 03e5f5b5-db94-4691-08a0-1a8bf15f6143
2011-03-18 17:14:15 +00:00