dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-26 07:30:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/table
History
Andrew Kryczka fea2b1dfb2 Copy Get() result when file reads use mmap 
Summary:
For iterator reads, a `SuperVersion` is pinned to preserve a snapshot of SST files, and `Block`s are pinned to allow `key()` and `value()` to return pointers directly into a RocksDB memory region. This works for both non-mmap reads, where the block owns the memory region, and mmap reads, where the file owns the memory region.

For point reads with `PinnableSlice`, only the `Block` object is pinned. This works for non-mmap reads because the block owns the memory region, so even if the file is deleted after compaction, the memory region survives. However, for mmap reads, file deletion causes the memory region to which the `PinnableSlice` refers to be unmapped.   The result is usually a segfault upon accessing the `PinnableSlice`, although sometimes it returned wrong results (I repro'd this a bunch of times with `db_stress`).

This PR copies the value into the `PinnableSlice` when it comes from mmap'd memory. We can tell whether the `Block` owns its memory using `Block::cachable()`, which is unset when reads do not use the provided buffer as is the case with mmap file reads. When that is false we ensure the result of `Get()` is copied.

This feels like a short-term solution as ideally we'd have the `PinnableSlice` pin the mmap'd memory so we can do zero-copy reads. It seemed hard so I chose this approach to fix correctness in the meantime.
Closes https://github.com/facebook/rocksdb/pull/3881

Differential Revision: D8076288

Pulled By: ajkr

fbshipit-source-id: 31d78ec010198723522323dbc6ea325122a46b08
2018-06-01 16:57:58 -07:00
..
adaptive_table_factory.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
adaptive_table_factory.h Comment out unused variables 2018-03-05 13:13:41 -08:00
block.cc Copy Get() result when file reads use mmap 2018-06-01 16:57:58 -07:00
block.h Copy Get() result when file reads use mmap 2018-06-01 16:57:58 -07:00
block_based_filter_block.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_filter_block.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_filter_block_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_table_builder.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
block_based_table_builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_table_factory.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
block_based_table_factory.h Align SST file data blocks to avoid spanning multiple pages 2018-03-26 20:26:10 -07:00
block_based_table_reader.cc Copy Get() result when file reads use mmap 2018-06-01 16:57:58 -07:00
block_based_table_reader.h Exclude seq from index keys 2018-05-25 18:42:43 -07:00
block_builder.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_fetcher.cc Imporve perf of random read and insert compare by suggesting inlining to the compiler 2018-03-23 13:26:55 -07:00
block_fetcher.h Fix BlockFetcher ASAN error 2017-12-12 12:12:38 -08:00
block_prefix_index.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
block_prefix_index.h Change RocksDB License 2017-07-15 16:11:23 -07:00
block_test.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
bloom_block.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
bloom_block.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cleanable_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder.cc Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
cuckoo_table_builder.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cuckoo_table_builder_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
cuckoo_table_factory.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
cuckoo_table_factory.h comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
cuckoo_table_reader.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
cuckoo_table_reader.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
cuckoo_table_reader_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
filter_block.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
flush_block_policy.cc Align SST file data blocks to avoid spanning multiple pages 2018-03-26 20:26:10 -07:00
format.cc Several small "fixes" 2018-02-15 16:57:37 -08:00
format.h comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
full_filter_bits_builder.h Skip duplicate bloom keys when whole_key and prefix are mixed 2018-04-24 10:58:16 -07:00
full_filter_block.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
full_filter_block.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
full_filter_block_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
get_context.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
get_context.h Stats for false positive rate of full filtesr 2018-04-05 15:58:48 -07:00
index_builder.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
index_builder.h Exclude seq from index keys 2018-05-25 18:42:43 -07:00
internal_iterator.h Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
iter_heap.h Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
iterator.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
iterator_wrapper.h Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
merger_test.cc Make InternalKeyComparator final and directly use it in merging iterator 2017-09-11 12:04:21 -07:00
merging_iterator.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
merging_iterator.h fix DBImpl::NewInternalIterator super-version leak on failure 2017-10-11 14:57:43 -07:00
meta_blocks.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
meta_blocks.h Support prefetch last 512KB with direct I/O in block based file reader 2017-08-11 12:16:45 -07:00
mock_table.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
mock_table.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
partitioned_filter_block.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
partitioned_filter_block.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
partitioned_filter_block_test.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
persistent_cache_helper.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
persistent_cache_helper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
persistent_cache_options.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_builder.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_factory.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_factory.h Comment out unused variables 2018-03-05 13:13:41 -08:00
plain_table_index.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_index.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_key_coding.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
plain_table_key_coding.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_reader.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
plain_table_reader.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
scoped_arena_iterator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sst_file_writer.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
sst_file_writer_collectors.h Comment out unused variables 2018-03-05 13:13:41 -08:00
table_builder.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
table_properties.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
table_properties_internal.h Eliminate some redundant block reads. 2018-01-10 17:11:58 -08:00
table_reader.h Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
table_reader_bench.cc Move prefix_extractor to MutableCFOptions 2018-05-21 14:43:11 -07:00
table_test.cc Exclude seq from index keys 2018-05-25 18:42:43 -07:00
two_level_iterator.cc Change and clarify the relationship between Valid(), status() and Seek*() for all iterators. Also fix some bugs 2018-05-17 02:56:56 -07:00
two_level_iterator.h fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00