dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
rocksdb/util
History
Vijay Nadimpalli 4c49e38f15 MultiGet batching in memtable (#5818) 
Summary:
RocksDB has a MultiGet() API that implements batched key lookup for higher performance (https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h#L468). Currently, batching is implemented in BlockBasedTableReader::MultiGet() for SST file lookups. One of the ways it improves performance is by pipelining bloom filter lookups (by prefetching required cachelines for all the keys in the batch, and then doing the probe) and thus hiding the cache miss latency. The same concept can be extended to the memtable as well. This PR involves implementing a pipelined bloom filter lookup in DynamicBloom, and implementing MemTable::MultiGet() that can leverage it.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5818

Test Plan:
Existing tests

Performance Test:
Ran the below command which fills up the memtable and makes sure there are no flushes and then call multiget. Ran it on master and on the new change and see atleast 1% performance improvement across all the test runs I did. Sometimes the improvement was upto 5%.

TEST_TMPDIR=/data/users/$USER/benchmarks/feature/ numactl -C 10 ./db_bench -benchmarks="fillseq,multireadrandom" -num=600000 -compression_type="none" -level_compaction_dynamic_level_bytes -write_buffer_size=200000000 -target_file_size_base=200000000 -max_bytes_for_level_base=16777216 -reads=90000 -threads=1 -compression_type=none -cache_size=4194304000 -batch_size=32 -disable_auto_compactions=true -bloom_bits=10 -cache_index_and_filter_blocks=true -pin_l0_filter_and_index_blocks_in_cache=true -multiread_batched=true -multiread_stride=4 -statistics -memtable_whole_key_filtering=true -memtable_bloom_size_ratio=10

Differential Revision: D17578869

Pulled By: vjnadimpalli

fbshipit-source-id: 23dc651d9bf49db11d22375bf435708875a1f192
 		2019-10-10 09:39:39 -07:00
..
aligned_buffer.h Document AlignedBuffer (#5345) 2019-05-24 10:05:40 -07:00
autovector.h Use placement new and delete in autovector (#5080) 2019-03-20 10:42:04 -07:00
autovector_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
bloom.cc Add FullBloomTest.CorruptFilters,RawSchema (#5834) 2019-10-02 15:33:48 -07:00
bloom_impl.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
bloom_test.cc Fix type in shift operation in bloom_test (#5882) 2019-10-03 13:19:20 -07:00
build_version.cc.in Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
build_version.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cast_util.h Add a missing "once" in .h 2017-07-31 12:12:03 -07:00
channel.h Support pragma once in all header files and cleanup some warnings (#4339) 2018-09-05 18:13:31 -07:00
coding.cc Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
coding.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
coding_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
compaction_job_stats_impl.cc Refresh snapshot list during long compactions (2nd attempt) (#5278) 2019-05-03 17:30:22 -07:00
comparator.cc Add support for timestamp in Get/Put (#5079) 2019-06-05 23:10:47 -07:00
compression.h Revert to storing UncompressionDicts in the cache (#5645) 2019-08-23 08:27:30 -07:00
compression_context_cache.cc run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
compression_context_cache.h run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
concurrent_task_limiter_impl.cc Compaction limiter miscs (#4795) 2018-12-26 13:59:35 -08:00
concurrent_task_limiter_impl.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
core_local.h Change RocksDB License 2017-07-15 16:11:23 -07:00
crc32c.cc Cleanup the Arm64 CRC32 unused warning (#5565) 2019-07-15 11:20:26 -07:00
crc32c.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_arm64.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
crc32c_arm64.h Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
crc32c_ppc.c C file should not include <cinttypes>, it is a C++ header. (#5499) 2019-06-24 16:12:39 -07:00
crc32c_ppc.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_ppc_asm.S Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_ppc_constants.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
duplicate_detector.h simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
dynamic_bloom.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dynamic_bloom.h MultiGet batching in memtable (#5818) 2019-10-10 09:39:39 -07:00
dynamic_bloom_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
file_reader_writer_test.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
filelock_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filter_bench.cc Fix some implicit conversions in filter_bench (#5894) 2019-10-08 19:22:07 -07:00
filter_policy.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
gflags_compat.h filter_bench - a prelim tool for SST filter benchmarking (#5825) 2019-10-07 20:10:53 -07:00
hash.cc Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
hash.h Faster new DynamicBloom implementation (for memtable) (#5762) 2019-09-05 14:59:25 -07:00
hash_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
heap.h Add compaction logic to RangeDelAggregatorV2 (#4758) 2018-12-17 13:20:51 -08:00
heap_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
kv_map.h Consolidate hash function used for non-persistent data in a new function (#5155) 2019-04-08 13:32:06 -07:00
log_write_bench.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
murmurhash.cc Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
murmurhash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
mutexlock.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
ppc-opcode.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
random.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
random.h Fix some implicit conversions in filter_bench (#5894) 2019-10-08 19:22:07 -07:00
rate_limiter.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
rate_limiter.h rate limit auto-tuning 2017-10-04 19:15:01 -07:00
rate_limiter_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
repeatable_thread.h Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
repeatable_thread_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
set_comparator.h WritePrepared Txn: Move DuplicateDetector to util 2018-03-05 23:57:12 -08:00
slice.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
slice_transform_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
status.cc Allow users to stop manual compactions (#3971) 2019-09-16 21:01:47 -07:00
stderr_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
stop_watch.h Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
string_util.cc Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
string_util.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
thread_list_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
thread_local.cc Enable building of ARM32 (#4349) 2018-10-09 16:58:25 -07:00
thread_local.h Provide a way to override windows memory allocator with jemalloc for ZSTD 2018-06-04 12:12:48 -07:00
thread_local_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
thread_operation.h Add inline comments to flush job (#4464) 2018-10-05 15:41:17 -07:00
threadpool_imp.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
threadpool_imp.h Support lowering CPU priority of background threads 2018-04-24 08:41:51 -07:00
timer_queue.h Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
timer_queue_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
user_comparator_wrapper.h Fix perf_context.user_key_comparison_count for range scan (#5098) 2019-03-27 10:34:27 -07:00
util.h Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
vector_iterator.h Make clang-analyzer happy (#5821) 2019-09-18 15:25:48 -07:00
xxhash.cc Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00
xxhash.h Add copyright headers per FB open-source checkup tool. (#5199) 2019-04-18 10:55:01 -07:00