mirror of https://github.com/facebook/rocksdb.git
91687d70ea
Summary: Bloom filters generated by pre-7.0 releases are not read by 7.0.x releases (and vice-versa) due to changes to FilterPolicy::Name() in https://github.com/facebook/rocksdb/issues/9590. This can severely impact read performance and read I/O on upgrade or downgrade with existing DB, but not data correctness. To fix, we go back using the old, unified name in SST metadata but (for a while anyway) recognize the aliases that could be generated by early 7.0.x releases. This unfortunately requires a public API change to avoid interfering with all the good changes from https://github.com/facebook/rocksdb/issues/9590, but the API change only affects users with custom FilterPolicy, which should be very few. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9736 Test Plan: manual Generate DBs with ``` ./db_bench.7.0 -db=/dev/shm/rocksdb.7.0 -bloom_bits=10 -cache_index_and_filter_blocks=1 -benchmarks=fillrandom -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 ``` and similar. Compare with ``` for IMPL in 6.29 7.0 fixed; do for DB in 6.29 7.0 fixed; do echo "Testing $IMPL on $DB:"; ./db_bench.$IMPL -db=/dev/shm/rocksdb.$DB -use_existing_db -readonly -bloom_bits=10 -benchmarks=readrandom -num=10000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -duration=10 2>&1 | grep micros/op; done; done ``` Results: ``` Testing 6.29 on 6.29: readrandom : 34.381 micros/op 29085 ops/sec; 3.2 MB/s (291999 of 291999 found) Testing 6.29 on 7.0: readrandom : 190.443 micros/op 5249 ops/sec; 0.6 MB/s (52999 of 52999 found) Testing 6.29 on fixed: readrandom : 40.148 micros/op 24907 ops/sec; 2.8 MB/s (249999 of 249999 found) Testing 7.0 on 6.29: readrandom : 229.430 micros/op 4357 ops/sec; 0.5 MB/s (43999 of 43999 found) Testing 7.0 on 7.0: readrandom : 33.348 micros/op 29986 ops/sec; 3.3 MB/s (299999 of 299999 found) Testing 7.0 on fixed: readrandom : 152.734 micros/op 6546 ops/sec; 0.7 MB/s (65999 of 65999 found) Testing fixed on 6.29: readrandom : 32.024 micros/op 31224 ops/sec; 3.5 MB/s (312999 of 312999 found) Testing fixed on 7.0: readrandom : 33.990 micros/op 29390 ops/sec; 3.3 MB/s (294999 of 294999 found) Testing fixed on fixed: readrandom : 28.714 micros/op 34825 ops/sec; 3.9 MB/s (348999 of 348999 found) ``` Just paying attention to order of magnitude of ops/sec (short test durations, lots of noise), it's clear that with the fix we can read <= 6.29 & >= 7.0 at full speed, where neither 6.29 nor 7.0 can on both. And 6.29 release can properly read fixed DB at full speed. Reviewed By: siying, ajkr Differential Revision: D35057844 Pulled By: pdillinger fbshipit-source-id: a46893a6af4bf084375ebe4728066d00eb08f050 |
||
|---|---|---|
| .. | ||
| adaptive | ||
| block_based | ||
| cuckoo | ||
| plain | ||
| block_fetcher.cc | ||
| block_fetcher.h | ||
| block_fetcher_test.cc | ||
| cleanable_test.cc | ||
| format.cc | ||
| format.h | ||
| get_context.cc | ||
| get_context.h | ||
| internal_iterator.h | ||
| iter_heap.h | ||
| iterator.cc | ||
| iterator_wrapper.h | ||
| merger_test.cc | ||
| merging_iterator.cc | ||
| merging_iterator.h | ||
| meta_blocks.cc | ||
| meta_blocks.h | ||
| mock_table.cc | ||
| mock_table.h | ||
| multiget_context.h | ||
| persistent_cache_helper.cc | ||
| persistent_cache_helper.h | ||
| persistent_cache_options.h | ||
| scoped_arena_iterator.h | ||
| sst_file_dumper.cc | ||
| sst_file_dumper.h | ||
| sst_file_reader.cc | ||
| sst_file_reader_test.cc | ||
| sst_file_writer.cc | ||
| sst_file_writer_collectors.h | ||
| table_builder.h | ||
| table_factory.cc | ||
| table_properties.cc | ||
| table_properties_internal.h | ||
| table_reader.h | ||
| table_reader_bench.cc | ||
| table_reader_caller.h | ||
| table_test.cc | ||
| two_level_iterator.cc | ||
| two_level_iterator.h | ||
| unique_id.cc | ||
| unique_id_impl.h | ||