dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-29 00:34:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/util
History
Peter Dillinger ad135f3ffd Document design/specification bugs with auto_prefix_mode (#10144) 
Summary:
auto_prefix_mode is designed to use prefix filtering in a
particular "safe" set of cases where the upper bound and the seek key
have different prefixes: where the upper bound is the "same length
immediate successor". These conditions are not sufficient to guarantee
the same iteration results as total_order_seek if the DB contains
"short" keys, less than the "full" (maximum) prefix length.

We are not simply disabling the optimization in these successor cases
because it is likely that users are essentially getting what they want
out of existing usage. Especially if users are constructing successor
bounds with the intention of doing a prefix-bounded seek, the existing
behavior is more expected than the total_order_seek behavior.
Consequently, for now we reconcile the bad specification of behavior by
documenting the existing mismatch with total_order_seek.

A closely related issue affects hypothetical comparators like
ReverseBytewiseComparator: if they "correctly" implement
IsSameLengthImmediateSuccessor, auto_prefix_mode could omit more
entries (other than "short" keys noted above). Luckily, the built-in
ReverseBytewiseComparator has an "incorrect" implementation of
IsSameLengthImmediateSuccessor that effectively prevents prefix
optimization and, thus, the bug. This is now documented as a new
constraint on IsSameLengthImmediateSuccessor, and the implementation
tweaked to be simply "safe" rather than "incorrect".

This change also includes unit test updates to demonstrate the above
issues. (Test was cleaned up for readability and simplicity.)

Intended follow-up:
* Tweak documented axioms for prefix_extractor (more details then)
* Consider some sort of fix for this case. I don't know what that would
look like without breaking the performance of existing code. Perhaps
if all keys in an SST file have prefixes that are "full length," we can track
that fact and use it to allow optimization with the "same length
immediate successor", but that would only apply to new files.
* Consider a better system of specifying prefix bounds

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10144

Test Plan: test updates included

Reviewed By: siying

Differential Revision: D37052710

Pulled By: pdillinger

fbshipit-source-id: 5f63b7d65f3f214e4b143e0f9aa1749527c587db
2022-06-13 11:08:50 -07:00
..
aligned_buffer.h Remove code that only compiles for Visual Studio versions older than 2015 (#10065) 2022-05-26 16:55:08 -07:00
async_file_reader.cc Multi file concurrency in MultiGet using coroutines and async IO (#9968) 2022-05-19 15:36:27 -07:00
async_file_reader.h Fix fbcode internal build failure (#10041) 2022-05-23 20:06:14 -07:00
autovector.h Add wide column serialization primitives (#9915) 2022-06-03 20:54:48 -07:00
autovector_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
bloom_impl.h FilterPolicy API changes for 7.0 (#9501) 2022-02-08 13:56:46 -08:00
bloom_test.cc Rewrite memory-charging feature's option API (#9926) 2022-05-17 15:01:51 -07:00
build_version.cc.in Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
cast_util.h More refactoring ahead of footer & meta changes (#9240) 2021-12-10 08:13:26 -08:00
channel.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
cleanable.cc Fix compile error in Clang 13 (#10033) 2022-05-28 00:15:28 -07:00
coding.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
coding.h More refactoring ahead of footer & meta changes (#9240) 2021-12-10 08:13:26 -08:00
coding_lean.h New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
coding_test.cc Fix potential overflow of unsigned type in for loop (#6902) 2020-06-02 15:05:07 -07:00
compaction_job_stats_impl.cc Update compaction statistics to include the amount of data read from blob files (#8022) 2021-03-04 00:43:48 -08:00
comparator.cc Document design/specification bugs with auto_prefix_mode (#10144) 2022-06-13 11:08:50 -07:00
compression.cc Integrate WAL compression into log reader/writer. (#9642) 2022-03-09 15:49:53 -08:00
compression.h Remove code that only compiles for Visual Studio versions older than 2015 (#10065) 2022-05-26 16:55:08 -07:00
compression_context_cache.cc Remove using namespace (#9369) 2022-01-12 09:31:12 -08:00
compression_context_cache.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
concurrent_task_limiter_impl.cc Remove TaskLimiterToken::ReleaseOnce for fix (#8567) 2021-07-21 17:37:53 -07:00
concurrent_task_limiter_impl.h Remove TaskLimiterToken::ReleaseOnce for fix (#8567) 2021-07-21 17:37:53 -07:00
core_local.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
coro_utils.h Multi file concurrency in MultiGet using coroutines and async IO (#9968) 2022-05-19 15:36:27 -07:00
crc32c.cc Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
crc32c.h Implementation of Crc32c combine function (#8305) 2021-06-16 18:30:34 -07:00
crc32c_arm64.cc Mac M1 crc32 intrinsics ARM64 check support proposal (#7893) 2021-03-10 09:05:56 -08:00
crc32c_arm64.h Fix compilation on Apple Silicon (#7714) 2020-12-04 15:22:33 -08:00
crc32c_ppc.c Fix Compilation on ppc64le using Clang 11 (#7713) 2020-12-01 11:21:44 -08:00
crc32c_ppc.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
crc32c_ppc_asm.S Fix Compilation on ppc64le using Clang 11 (#7713) 2020-12-01 11:21:44 -08:00
crc32c_ppc_constants.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
crc32c_test.cc Implementation of Crc32c combine function (#8305) 2021-06-16 18:30:34 -07:00
defer.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
defer_test.cc Fix insecure internal API for GetImpl (#8590) 2021-07-29 17:23:01 -07:00
duplicate_detector.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
dynamic_bloom.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
dynamic_bloom.h Fix major bug with MultiGet, DeleteRange, and memtable Bloom (#9453) 2022-01-27 14:55:04 -08:00
dynamic_bloom_test.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
fastrange.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
file_checksum_helper.cc Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362) 2022-01-11 06:33:48 -08:00
file_checksum_helper.h New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
file_reader_writer_test.cc Support read rate-limiting in SequentialFileReader (#9973) 2022-05-24 10:28:57 -07:00
filelock_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
filter_bench.cc Rewrite memory-charging feature's option API (#9926) 2022-05-17 15:01:51 -07:00
gflags_compat.h Require C++17 (#9481) 2022-02-04 17:13:10 -08:00
hash.cc Experimental support for SST unique IDs (#8990) 2021-10-18 23:32:01 -07:00
hash.h Experimental support for SST unique IDs (#8990) 2021-10-18 23:32:01 -07:00
hash128.h Upgrade xxhash, add Hash128 (#8634) 2021-08-20 18:41:51 -07:00
hash_containers.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
hash_map.h Change HashMap::Insert()'s value to a const reference (#6567) 2020-03-20 14:59:54 -07:00
hash_test.cc Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
heap.h Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
heap_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
kv_map.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
log_write_bench.cc Add a SystemClock class to capture the time functions of an Env (#7858) 2021-01-25 22:09:11 -08:00
math.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
math128.h New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
murmurhash.cc C++20 compatibility (#6697) 2020-04-20 13:24:25 -07:00
murmurhash.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
mutexlock.h Prevents Table Cache to open same files more times (#6707) 2020-04-21 13:16:31 -07:00
ppc-opcode.h Remove PATENTS text from a few straggler files (#5326) 2019-05-21 16:22:35 -07:00
random.cc Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015) 2022-05-18 15:25:19 -07:00
random.h Experimental support for SST unique IDs (#8990) 2021-10-18 23:32:01 -07:00
random_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
rate_limiter.cc Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
rate_limiter.h remove unused instance variable in GenericRateLimiter (#9484) 2022-02-01 14:04:12 -08:00
rate_limiter_test.cc Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
repeatable_thread.h Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
repeatable_thread_test.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
ribbon_alg.h util/ribbon_alg.h: removed duplicate word "vector" (#9216) 2022-05-06 18:38:13 -07:00
ribbon_config.cc Refine Ribbon configuration, improve testing, add Homogeneous (#7879) 2021-02-26 08:50:42 -08:00
ribbon_config.h Refine Ribbon configuration, improve testing, add Homogeneous (#7879) 2021-02-26 08:50:42 -08:00
ribbon_impl.h Account Bloom/Ribbon filter construction memory in global memory limit (#9073) 2021-11-18 09:42:20 -08:00
ribbon_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
set_comparator.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
single_thread_executor.h Multi file concurrency in MultiGet using coroutines and async IO (#9968) 2022-05-19 15:36:27 -07:00
slice.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
slice_test.cc Require C++17 (#9481) 2022-02-04 17:13:10 -08:00
slice_transform_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
status.cc Combine data members of IOStatus with Status (#9549) 2022-02-22 11:23:01 -08:00
stderr_logger.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
stop_watch.h Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
string_util.cc Fix potential ambiguities in/around port/sys_time.h (#10045) 2022-05-24 18:20:17 -07:00
string_util.h Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
thread_guard.h Introduce a ThreadGuard class and use it in ExternalSSTFileTest.PickedLevelBug (#8112) 2021-03-25 22:08:58 -07:00
thread_list_test.cc fix thread status synchronization in thread_list_test (#7825) 2021-01-04 10:46:24 -08:00
thread_local.cc Remove ROCKSDB_SUPPORT_THREAD_LOCAL define because it's a part of C++11 (#10015) 2022-05-18 15:25:19 -07:00
thread_local.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
thread_local_test.cc Add StartThread type checking wrapper (#8303) 2021-05-19 16:51:13 -07:00
thread_operation.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
threadpool_imp.cc Remove incremental ID from background thread pool names (#9165) 2021-11-16 18:26:12 -08:00
threadpool_imp.h Make it able to lower cpu priority to specific level in threadpool (#6969) 2020-06-13 13:25:20 -07:00
timer.h Fix a timer crash caused by invalid memory management (#9656) 2022-03-12 11:45:56 -08:00
timer_queue.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
timer_queue_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
timer_test.cc Fix a timer crash caused by invalid memory management (#9656) 2022-03-12 11:45:56 -08:00
user_comparator_wrapper.h Enable backward iterator for keys with user-defined timestamp (#8035) 2021-03-10 11:15:46 -08:00
vector_iterator.h Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
work_queue.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
work_queue_test.cc Add pipelined & parallel compression optimization (#6262) 2020-04-01 16:40:18 -07:00
xxhash.cc Upgrade xxhash, add Hash128 (#8634) 2021-08-20 18:41:51 -07:00
xxhash.h Upgrade xxhash, add Hash128 (#8634) 2021-08-20 18:41:51 -07:00
xxph3.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00