mirror of https://github.com/facebook/rocksdb.git
14eca6bf04
Summary: The implementation of GetApproximateSizes was inconsistent in its treatment of the size of non-data blocks of SST files, sometimes including and sometimes now. This was at its worst with large portion of table file used by filters and querying a small range that crossed a table boundary: the size estimate would include large filter size. It's conceivable that someone might want only to know the size in terms of data blocks, but I believe that's unlikely enough to ignore for now. Similarly, there's no evidence the internal function AppoximateOffsetOf is used for anything other than a one-sided ApproximateSize, so I intend to refactor to remove redundancy in a follow-up commit. So to fix this, GetApproximateSizes (and implementation details ApproximateSize and ApproximateOffsetOf) now consistently include in their returned sizes a portion of table file metadata (incl filters and indexes) based on the size portion of the data blocks in range. In other words, if a key range covers data blocks that are X% by size of all the table's data blocks, returned approximate size is X% of the total file size. It would technically be more accurate to attribute metadata based on number of keys, but that's not computationally efficient with data available and rarely a meaningful difference. Also includes miscellaneous comment improvements / clarifications. Also included is a new approximatesizerandom benchmark for db_bench. No significant performance difference seen with this change, whether ~700 ops/sec with cache_index_and_filter_blocks and small cache or ~150k ops/sec without cache_index_and_filter_blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6784 Test Plan: Test added to DBTest.ApproximateSizesFilesWithErrorMargin. Old code running new test... [ RUN ] DBTest.ApproximateSizesFilesWithErrorMargin db/db_test.cc:1562: Failure Expected: (size) <= (11 * 100), actual: 9478 vs 1100 Other tests updated to reflect consistent accounting of metadata. Reviewed By: siying Differential Revision: D21334706 Pulled By: pdillinger fbshipit-source-id: 6f86870e45213334fedbe9c73b4ebb1d8d611185 |
||
|---|---|---|
| .. | ||
| adaptive | ||
| block_based | ||
| cuckoo | ||
| plain | ||
| block_fetcher.cc | ||
| block_fetcher.h | ||
| block_fetcher_test.cc | ||
| cleanable_test.cc | ||
| format.cc | ||
| format.h | ||
| get_context.cc | ||
| get_context.h | ||
| internal_iterator.h | ||
| iter_heap.h | ||
| iterator.cc | ||
| iterator_wrapper.h | ||
| merger_test.cc | ||
| merging_iterator.cc | ||
| merging_iterator.h | ||
| meta_blocks.cc | ||
| meta_blocks.h | ||
| mock_table.cc | ||
| mock_table.h | ||
| multiget_context.h | ||
| persistent_cache_helper.cc | ||
| persistent_cache_helper.h | ||
| persistent_cache_options.h | ||
| scoped_arena_iterator.h | ||
| sst_file_reader.cc | ||
| sst_file_reader_test.cc | ||
| sst_file_writer.cc | ||
| sst_file_writer_collectors.h | ||
| table_builder.h | ||
| table_properties.cc | ||
| table_properties_internal.h | ||
| table_reader.h | ||
| table_reader_bench.cc | ||
| table_reader_caller.h | ||
| table_test.cc | ||
| two_level_iterator.cc | ||
| two_level_iterator.h | ||