rocksdb/db
Gang Liao 8d178090be Add a blob-specific cache priority (#10309)
Summary:
RocksDB's `Cache` abstraction currently supports two priority levels for items: high (used for frequently accessed/highly valuable SST metablocks like index/filter blocks) and low (used for SST data blocks). Blobs are typically lower-value targets for caching than data blocks, since 1) with BlobDB, data blocks containing blob references conceptually form an index structure which has to be consulted before we can read the blob value, and 2) cached blobs represent only a single key-value, while cached data blocks generally contain multiple KVs. Since we would like to make it possible to use the same backing cache for the block cache and the blob cache, it would make sense to add a new, lower-than-low cache priority level (bottom level) for blobs so data blocks are prioritized over them.

This task is a part of https://github.com/facebook/rocksdb/issues/10156

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10309

Reviewed By: ltamasi

Differential Revision: D38211655

Pulled By: gangliao

fbshipit-source-id: 65ef33337db4d85277cc6f9782d67c421ad71dd5
2022-07-27 19:09:24 -07:00
..
blob Add a blob-specific cache priority (#10309) 2022-07-27 19:09:24 -07:00
compaction Allow sufficient subcompactions under round-robin compaction priority (#10422) 2022-07-26 20:37:34 -07:00
db_impl Support subcmpct using reserved resources for round-robin priority (#10341) 2022-07-24 11:12:44 -07:00
wide Add API for writing wide-column entities (#10242) 2022-06-25 15:30:47 -07:00
arena_wrapped_db_iter.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
arena_wrapped_db_iter.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
builder.cc Support prepopulating/warming the blob cache (#10298) 2022-07-17 07:13:59 -07:00
builder.h Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
c.cc Support prepopulating/warming the blob cache (#10298) 2022-07-17 07:13:59 -07:00
c_test.c Support prepopulating/warming the blob cache (#10298) 2022-07-17 07:13:59 -07:00
column_family.cc fix typos in some code and comment (#10139) 2022-07-22 19:25:52 -07:00
column_family.h Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
column_family_test.cc Deflake FlushStaleColumnFamilies test (#10409) 2022-07-26 12:50:27 -07:00
compact_files_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
comparator_db_test.cc Document design/specification bugs with auto_prefix_mode (#10144) 2022-06-13 11:08:50 -07:00
convenience.cc Specify largest_seqno in VerifyChecksum (#9919) 2022-05-02 10:22:08 -07:00
corruption_test.cc Multi-File Trivial Move in L0->L1 (#10188) 2022-06-30 18:04:23 -07:00
cuckoo_table_db_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_basic_test.cc Add helper function to get debug type name (#10243) 2022-07-15 14:42:00 -07:00
db_block_cache_test.cc Add a blob-specific cache priority (#10309) 2022-07-27 19:09:24 -07:00
db_bloom_filter_test.cc Update/clarify required properties for prefix extractors (#10245) 2022-06-28 16:08:30 -07:00
db_compaction_filter_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_compaction_test.cc Allow sufficient subcompactions under round-robin compaction priority (#10422) 2022-07-26 20:37:34 -07:00
db_dynamic_level_test.cc Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit (#9452) 2022-01-27 13:01:09 -08:00
db_encryption_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_filesnapshot.cc Reduce risk of backup or checkpoint missing a WAL file (#10083) 2022-06-01 11:02:27 -07:00
db_flush_test.cc Dynamically changeable MemPurge option (#10011) 2022-06-23 09:42:18 -07:00
db_info_dumper.cc Printing IO Error in DumpDBFileSummary (#9940) 2022-05-04 10:19:53 -07:00
db_info_dumper.h Add a DB Session ID (#6959) 2020-06-15 10:47:02 -07:00
db_inplace_update_test.cc Fix in-place updates for value types other than kTypeValue (#10254) 2022-06-27 16:37:09 -07:00
db_io_failure_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_iter.cc Fix assertion error with read_opts.iter_start_ts (#10279) 2022-06-30 10:16:03 -07:00
db_iter.h Support iter_start_ts for backward iteration (#10200) 2022-06-28 19:51:05 -07:00
db_iter_stress_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_iter_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_iterator_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_kv_checksum_test.cc Handoff checksum during WAL replay (#10212) 2022-07-05 15:44:35 -07:00
db_log_iter_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_logical_block_size_cache_test.cc Attempt to deflake DBLogicalBlockSizeCacheTest.CreateColumnFamilies (#9516) 2022-03-04 11:35:28 -08:00
db_memtable_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_merge_operand_test.cc Fix GetMergeOperands() heap-use-after-free on flushed memtable (#9805) 2022-04-05 12:26:36 -07:00
db_merge_operator_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_options_test.cc Fix typo in comments and code (#10233) 2022-06-22 15:45:21 -07:00
db_properties_test.cc Add blob cache tickers, perf context statistics, and DB properties (#10203) 2022-06-28 13:52:35 -07:00
db_range_del_test.cc Try to trivial move more than one files (#10190) 2022-07-05 10:10:37 -07:00
db_rate_limiter_test.cc Add rate-limiting support to batched MultiGet() (#10159) 2022-06-17 16:40:47 -07:00
db_readonly_with_timestamp_test.cc Add timestamp support to CompactedDBImpl (#10030) 2022-05-24 12:14:10 -07:00
db_secondary_test.cc Fail DB::Open() if logger cannot be created (#9984) 2022-05-27 07:23:31 -07:00
db_sst_test.cc Pass the size of blob files to SstFileManager during DB open (#10062) 2022-05-27 05:58:43 -07:00
db_statistics_test.cc Bytes read stat for VerifyChecksum() and VerifyFileChecksums() APIs (#8741) 2021-09-07 13:28:29 -07:00
db_table_properties_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_tailing_iter_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_test.cc Remove Travis CI (#10407) 2022-07-22 20:16:45 -07:00
db_test2.cc Add a blob-specific cache priority (#10309) 2022-07-27 19:09:24 -07:00
db_test_util.cc Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
db_test_util.h Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
db_universal_compaction_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_wal_test.cc Added WAL compression checksum (#10319) 2022-07-13 15:29:20 -07:00
db_with_timestamp_basic_test.cc Fix assertion error with read_opts.iter_start_ts (#10279) 2022-06-30 10:16:03 -07:00
db_with_timestamp_compaction_test.cc Use the comparator from the sst file table properties in sst_dump_tool (#9491) 2022-02-08 12:15:35 -08:00
db_with_timestamp_test_util.cc Add timestamp support to DBImplReadOnly (#10004) 2022-05-19 18:39:41 -07:00
db_with_timestamp_test_util.h Add timestamp support to DBImplReadOnly (#10004) 2022-05-19 18:39:41 -07:00
db_write_buffer_manager_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_write_test.cc Fix race condition with WAL tracking and FlushWAL(true /* sync */) (#10185) 2022-06-17 16:45:28 -07:00
dbformat.cc Make InternalKeyComparator not configurable (#10342) 2022-07-14 10:09:31 -07:00
dbformat.h Add helper function to get debug type name (#10243) 2022-07-15 14:42:00 -07:00
dbformat_test.cc Make InternalKeyComparator not configurable (#10342) 2022-07-14 10:09:31 -07:00
deletefile_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
error_handler.cc Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
error_handler.h Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
error_handler_fs_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
event_helpers.cc Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
event_helpers.h Add a listener callback for end of auto error recovery (#9244) 2021-12-08 14:30:57 -08:00
experimental.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
external_sst_file_basic_test.cc Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
external_sst_file_ingestion_job.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
external_sst_file_ingestion_job.h Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
external_sst_file_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
fault_injection_test.cc Fix a bug causing duplicate trailing entries in WritableFile (buffered IO) (#9236) 2021-12-13 09:00:36 -08:00
file_indexer.cc
file_indexer.h Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
file_indexer_test.cc
filename_test.cc fixing issue #8345 RocksDB does not work when using UNC network paths (#9384) 2022-03-30 15:55:31 -07:00
flush_job.cc Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
flush_job.h Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
flush_job_test.cc Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
flush_scheduler.cc
flush_scheduler.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
forward_iterator.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
forward_iterator.h Make InternalKeyComparator not configurable (#10342) 2022-07-14 10:09:31 -07:00
forward_iterator_bench.cc Remove using namespace (#9369) 2022-01-12 09:31:12 -08:00
history_trimming_iterator.h Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410) 2022-03-11 16:13:23 -08:00
import_column_family_job.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
import_column_family_job.h New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
import_column_family_test.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
internal_stats.cc Add blob cache tickers, perf context statistics, and DB properties (#10203) 2022-06-28 13:52:35 -07:00
internal_stats.h Tiered Compaction: per key placement support (#9964) 2022-07-13 20:54:49 -07:00
job_context.h CompactionIterator sees consistent view of which keys are committed (#9830) 2022-04-14 11:11:04 -07:00
kv_checksum.h fix compile errors in db/kv_checksum.h (#9173) 2021-11-16 10:20:50 -08:00
listener_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
log_format.h Add record to set WAL compression type if enabled (#9556) 2022-02-17 16:19:31 -08:00
log_reader.cc Add checksum handshake for WAL fragment decompression (#10339) 2022-07-25 16:27:26 -07:00
log_reader.h Add checksum handshake for WAL fragment decompression (#10339) 2022-07-25 16:27:26 -07:00
log_test.cc Add checksum handshake for WAL fragment decompression (#10339) 2022-07-25 16:27:26 -07:00
log_writer.cc Integrate WAL compression into log reader/writer. (#9642) 2022-03-09 15:49:53 -08:00
log_writer.h Fix typo in comments and code (#10233) 2022-06-22 15:45:21 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
lookup_key.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
malloc_stats.cc Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
malloc_stats.h
manual_compaction_test.cc Remove using namespace (#9369) 2022-01-12 09:31:12 -08:00
memtable.cc Fix in-place updates for value types other than kTypeValue (#10254) 2022-06-27 16:37:09 -07:00
memtable.h Fix in-place updates for value types other than kTypeValue (#10254) 2022-06-27 16:37:09 -07:00
memtable_list.cc Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
memtable_list.h Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
memtable_list_test.cc Add blob source to retrieve blobs in RocksDB (#10198) 2022-06-20 20:58:11 -07:00
merge_context.h Add Merge Operator support to WriteBatchWithIndex (#8135) 2021-05-10 12:50:25 -07:00
merge_helper.cc Add API for writing wide-column entities (#10242) 2022-06-25 15:30:47 -07:00
merge_helper.h Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_helper_test.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_operator.cc
merge_test.cc Make the Env class Customizable (#9293) 2022-01-04 16:45:49 -08:00
obsolete_files_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
options_file_test.cc No elide constructors (#7798) 2020-12-23 16:55:53 -08:00
output_validator.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
output_validator.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
perf_context_test.cc Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
periodic_work_scheduler.cc Fix seqno->time worker not scheduled with multi DB instances (#10383) 2022-07-18 19:08:39 -07:00
periodic_work_scheduler.h Fix seqno->time worker not scheduled with multi DB instances (#10383) 2022-07-18 19:08:39 -07:00
periodic_work_scheduler_test.cc Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
pinned_iterators_manager.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
plain_table_db_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
post_memtable_callback.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00
pre_release_callback.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
prefix_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
range_del_aggregator.cc In ParseInternalKey(), include corrupt key info in Status (#7515) 2020-10-28 10:12:58 -07:00
range_del_aggregator.h Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
range_del_aggregator_bench.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
range_del_aggregator_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
range_tombstone_fragmenter.cc Added memtable garbage statistics (#8411) 2021-06-18 04:57:27 -07:00
range_tombstone_fragmenter.h Added memtable garbage statistics (#8411) 2021-06-18 04:57:27 -07:00
range_tombstone_fragmenter_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
read_callback.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
repair.cc Add seqno to time mapping (#10338) 2022-07-14 21:49:34 -07:00
repair_test.cc Remove customized naming from InternalKeyComparator (#10343) 2022-07-12 13:30:35 -07:00
seqno_time_test.cc Fix seqno->time worker not scheduled with multi DB instances (#10383) 2022-07-18 19:08:39 -07:00
seqno_to_time_mapping.cc Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
seqno_to_time_mapping.h Tiered compaction: integrate Seqno time mapping with per key placement (#10370) 2022-07-15 19:01:30 -07:00
snapshot_checker.h Use STATIC_AVOID_DESTRUCTION for static objects with non-trivial destructors (#9958) 2022-05-17 09:39:22 -07:00
snapshot_impl.cc
snapshot_impl.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00
table_cache.cc Improve SubCompaction Partitioning (#10393) 2022-07-23 17:38:49 -07:00
table_cache.h Improve SubCompaction Partitioning (#10393) 2022-07-23 17:38:49 -07:00
table_cache_sync_and_async.h Multi file concurrency in MultiGet using coroutines and async IO (#9968) 2022-05-19 15:36:27 -07:00
table_properties_collector.cc Apply sample_for_compression to all block-based tables (#8105) 2021-03-25 15:00:45 -07:00
table_properties_collector.h Track each SST's timestamp information as user properties (#9093) 2021-11-19 11:37:06 -08:00
table_properties_collector_test.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
transaction_log_impl.cc Support read rate-limiting in SequentialFileReader (#9973) 2022-05-24 10:28:57 -07:00
transaction_log_impl.h Add checks to GetUpdatesSince (#9459) 2022-04-14 17:12:16 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Add basic kRoundRobin compaction policy (#10107) 2022-06-21 11:56:53 -07:00
version_builder.h Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
version_builder_test.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
version_edit.cc ldb to display public unique id and dump work with key range (#10417) 2022-07-26 20:40:18 -07:00
version_edit.h Fix typo in comments and code (#10233) 2022-06-22 15:45:21 -07:00
version_edit_handler.cc Add blob source to retrieve blobs in RocksDB (#10198) 2022-06-20 20:58:11 -07:00
version_edit_handler.h Fixed manifest_dump issues when printing keys and values containing null characters (#8378) 2021-06-10 12:55:20 -07:00
version_edit_test.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
version_set.cc Support subcmpct using reserved resources for round-robin priority (#10341) 2022-07-24 11:12:44 -07:00
version_set.h Support subcmpct using reserved resources for round-robin priority (#10341) 2022-07-24 11:12:44 -07:00
version_set_sync_and_async.h Enable blob caching for MultiGetBlob in RocksDB (#10272) 2022-06-30 13:24:35 -07:00
version_set_test.cc Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
version_util.h Add blob source to retrieve blobs in RocksDB (#10198) 2022-06-20 20:58:11 -07:00
wal_edit.cc Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
wal_edit.h Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
wal_edit_test.cc Do not hold mutex when write keys if not necessary (#7516) 2022-07-21 13:35:36 -07:00
wal_manager.cc Fix bug for WalManager with compressed WAL (#10130) 2022-06-08 14:16:43 -07:00
wal_manager.h Fix bug for WalManager with compressed WAL (#10130) 2022-06-08 14:16:43 -07:00
wal_manager_test.cc Add blob source to retrieve blobs in RocksDB (#10198) 2022-06-20 20:58:11 -07:00
write_batch.cc Handoff checksum during WAL replay (#10212) 2022-07-05 15:44:35 -07:00
write_batch_base.cc
write_batch_internal.h Handoff checksum during WAL replay (#10212) 2022-07-05 15:44:35 -07:00
write_batch_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
write_callback.h
write_callback_test.cc Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
write_controller.cc Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_controller.h Set Write rate limiter priority dynamically and pass it to FS (#9988) 2022-05-18 00:41:41 -07:00
write_controller_test.cc Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_thread.cc Verify write batch checksum before WAL (#10114) 2022-06-15 13:43:58 -07:00
write_thread.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00