mirror of https://github.com/facebook/rocksdb.git
a92bd0a183
Summary: The filter bits builder collects all the hashes to add in memory before adding them (because the number of keys is not known until we've walked over all the keys). Existing code uses a std::vector for this, which can mean up to 2x than necessary space allocated (and not freed) and up to ~2x write amplification in memory. Using std::deque uses close to minimal space (for large filters, the only time it matters), no write amplification, frees memory while building, and no need for large contiguous memory area. The only cost is more calls to allocator, which does not appear to matter, at least in benchmark test. For now, this change only applies to the new (format_version=5) Bloom filter implementation, to ease before-and-after comparison downstream. Temporary memory use during build is about the only way the new Bloom filter could regress vs. the old (because of upgrade to 64-bit hash) and that should only matter for full filters. This change should largely mitigate that potential regression. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6175 Test Plan: Using filter_bench with -new_builder option and 6M keys per filter is like large full filter (improvement). 10k keys and no -new_builder is like partitioned filters (about the same). (Corresponding configurations run simultaneously on devserver.) std::vector impl (before) $ /usr/bin/time -v ./filter_bench -impl=2 -quick -new_builder -working_mem_size_mb=1000 - average_keys_per_filter=6000000 Build avg ns/key: 52.2027 Maximum resident set size (kbytes): 1105016 $ /usr/bin/time -v ./filter_bench -impl=2 -quick -working_mem_size_mb=1000 - average_keys_per_filter=10000 Build avg ns/key: 30.5694 Maximum resident set size (kbytes): 1208152 std::deque impl (after) $ /usr/bin/time -v ./filter_bench -impl=2 -quick -new_builder -working_mem_size_mb=1000 - average_keys_per_filter=6000000 Build avg ns/key: 39.0697 Maximum resident set size (kbytes): 1087196 $ /usr/bin/time -v ./filter_bench -impl=2 -quick -working_mem_size_mb=1000 - average_keys_per_filter=10000 Build avg ns/key: 30.9348 Maximum resident set size (kbytes): 1207980 Differential Revision: D19053431 Pulled By: pdillinger fbshipit-source-id: 2888e748723a19d9ea40403934f13cbb8483430c |
||
|---|---|---|
| .. | ||
| aligned_buffer.h | ||
| autovector.h | ||
| autovector_test.cc | ||
| bloom_impl.h | ||
| bloom_test.cc | ||
| build_version.cc.in | ||
| build_version.h | ||
| cast_util.h | ||
| channel.h | ||
| coding.cc | ||
| coding.h | ||
| coding_test.cc | ||
| compaction_job_stats_impl.cc | ||
| comparator.cc | ||
| compression.h | ||
| compression_context_cache.cc | ||
| compression_context_cache.h | ||
| concurrent_task_limiter_impl.cc | ||
| concurrent_task_limiter_impl.h | ||
| core_local.h | ||
| crc32c.cc | ||
| crc32c.h | ||
| crc32c_arm64.cc | ||
| crc32c_arm64.h | ||
| crc32c_ppc.c | ||
| crc32c_ppc.h | ||
| crc32c_ppc_asm.S | ||
| crc32c_ppc_constants.h | ||
| crc32c_test.cc | ||
| duplicate_detector.h | ||
| dynamic_bloom.cc | ||
| dynamic_bloom.h | ||
| dynamic_bloom_test.cc | ||
| file_reader_writer_test.cc | ||
| filelock_test.cc | ||
| filter_bench.cc | ||
| gflags_compat.h | ||
| hash.cc | ||
| hash.h | ||
| hash_map.h | ||
| hash_test.cc | ||
| heap.h | ||
| heap_test.cc | ||
| kv_map.h | ||
| log_write_bench.cc | ||
| murmurhash.cc | ||
| murmurhash.h | ||
| mutexlock.h | ||
| ppc-opcode.h | ||
| random.cc | ||
| random.h | ||
| random_test.cc | ||
| rate_limiter.cc | ||
| rate_limiter.h | ||
| rate_limiter_test.cc | ||
| repeatable_thread.h | ||
| repeatable_thread_test.cc | ||
| set_comparator.h | ||
| slice.cc | ||
| slice_transform_test.cc | ||
| status.cc | ||
| stderr_logger.h | ||
| stop_watch.h | ||
| string_util.cc | ||
| string_util.h | ||
| thread_list_test.cc | ||
| thread_local.cc | ||
| thread_local.h | ||
| thread_local_test.cc | ||
| thread_operation.h | ||
| threadpool_imp.cc | ||
| threadpool_imp.h | ||
| timer_queue.h | ||
| timer_queue_test.cc | ||
| user_comparator_wrapper.h | ||
| util.h | ||
| vector_iterator.h | ||
| xxh3p.h | ||
| xxhash.cc | ||
| xxhash.h | ||