dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-26 16:30:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Andrew Kryczka 75d3b6fdf0 Redesign block cache pinning API (#7520) 
Summary:
The old flag-based APIs (`BlockBasedTableOptions::pin_l0_filter_and_index_blocks_in_cache` and `BlockBasedTableOptions::pin_top_level_index_and_filter`) were insufficient for our needs. For example, it was impossible to pin only unpartitioned meta-blocks, which could prevent block cache contention when turning on dictionary compression or during a migration to partitioned indexes/filters. It was also impossible to pin all meta-blocks in memory while having predictable memory usage via block cache. If we had continued adding flags to address these scenarios, they would have had significant overlap causing confusion. Instead, this PR deprecates the flags and starts a new API with non-overlapping options.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7520

Test Plan:
- new unit test
- added new options to stress/crash test and ran for a while: `$ python tools/db_crashtest.py blackbox --simple --max_key=1000000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 --interval=10 -value_size_mult=33 -column_families=1 -reopen=0`

Reviewed By: pdillinger

Differential Revision: D24200034

Pulled By: ajkr

fbshipit-source-id: 3fa7cfc71e7960f7a867511dd6ae5834dd73b13e
2020-10-11 14:58:24 -07:00
..
blob Do not rely on the two-argument std::pair constructor being constexpr (#7519) 2020-10-08 10:49:40 -07:00
compaction Make sure assert(false) handles failures too (#7483) 2020-10-06 13:52:42 -07:00
db_impl Add few unit test cases in ASSERT_STATUS_CHECKED (#7500) 2020-10-08 11:22:44 -07:00
arena_wrapped_db_iter.cc dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
arena_wrapped_db_iter.h dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
builder.cc Enforce status check for corruption_test (#7453) 2020-10-02 22:11:00 -07:00
builder.h Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
c.cc RocksJava - Add errorIfLogFileExists parameter to RocksDB.openReadOnly (#7046) 2020-09-17 15:41:25 -07:00
c_test.c Fix MSVC-related build issues (#7439) 2020-10-01 09:23:04 -07:00
column_family.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
column_family.h Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
column_family_test.cc Add status check enforcement for column_family_test.cc (#7484) 2020-10-02 13:35:15 -07:00
compact_files_test.cc Replace reinterpret_cast with static_cast_with_check (#7067) 2020-07-02 19:25:41 -07:00
compacted_db_impl.cc Periodically flush info log out of application buffer (#7488) 2020-10-01 19:14:14 -07:00
compacted_db_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
comparator_db_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
convenience.cc sst_dump to reduce number of file reads (#6836) 2020-05-12 18:23:33 -07:00
corruption_test.cc Enforce status check for corruption_test (#7453) 2020-10-02 22:11:00 -07:00
cuckoo_table_db_test.cc Replace reinterpret_cast with static_cast_with_check (#7067) 2020-07-02 19:25:41 -07:00
db_basic_test.cc Add Stats for MultiGet (#7366) 2020-10-07 13:28:48 -07:00
db_block_cache_test.cc Redesign block cache pinning API (#7520) 2020-10-11 14:58:24 -07:00
db_bloom_filter_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_compaction_filter_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_compaction_test.cc Disallow trivial move if BottommostLevelCompaction is kForce* (#7368) 2020-10-07 13:19:31 -07:00
db_dynamic_level_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_encryption_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_filesnapshot.cc Fix checkpoint file deletion race with avoid_unnecessary_blocking_io (#7369) 2020-09-10 22:35:25 -07:00
db_flush_test.cc Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
db_info_dumper.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
db_info_dumper.h Add a DB Session ID (#6959) 2020-06-15 10:47:02 -07:00
db_inplace_update_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_io_failure_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_iter.cc Make sure assert(false) handles failures too (#7483) 2020-10-06 13:52:42 -07:00
db_iter.h Make db_basic_test pass assert status checked (#7452) 2020-09-29 09:49:04 -07:00
db_iter_stress_test.cc Test CircleCI with CLANG-10 (#7025) 2020-06-24 16:22:49 -07:00
db_iter_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_iterator_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_log_iter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_logical_block_size_cache_test.cc Get block size only in direct IO mode (#6522) 2020-03-20 15:26:10 -07:00
db_memtable_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_merge_operand_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_merge_operator_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_options_test.cc Track WAL in MANIFEST: add option track_and_verify_wals_in_manifest (#7275) 2020-10-09 16:42:19 -07:00
db_properties_test.cc Fix db_properties_test for ASSERT_STATUS_CHECKED (#7490) 2020-10-01 17:47:09 -07:00
db_range_del_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_sst_test.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
db_statistics_test.cc Add a new stats level to exclude tickers (#7329) 2020-09-04 23:25:03 -07:00
db_table_properties_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_tailing_iter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_test.cc Reduce the number of iterations in DBTest.FileCreationRandomFailure (#7481) 2020-10-01 10:42:58 -07:00
db_test2.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_test_util.cc Enable ASSERT_STATUS_CHECKED for db_universal_compaction_test (#7460) 2020-10-06 14:42:12 -07:00
db_test_util.h add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
db_universal_compaction_test.cc Enable ASSERT_STATUS_CHECKED for db_universal_compaction_test (#7460) 2020-10-06 14:42:12 -07:00
db_wal_test.cc Integrate blob file writing with recovery (#7388) 2020-09-15 17:14:10 -07:00
db_with_timestamp_basic_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_with_timestamp_compaction_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_write_test.cc Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508) 2020-10-06 12:44:20 -07:00
dbformat.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
dbformat.h Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
dbformat_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
deletefile_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
error_handler.cc Status check enforcement for timestamp_basic_test (#7454) 2020-09-29 18:23:27 -07:00
error_handler.h Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-17 20:25:45 -07:00
error_handler_fs_test.cc Status check enforcement for error_handler_fs_test (#7342) 2020-10-02 16:41:13 -07:00
event_helpers.cc Status check enforcement for error_handler_fs_test (#7342) 2020-10-02 16:41:13 -07:00
event_helpers.h Pass SST file checksum information through OnTableFileCreated (#7108) 2020-08-25 10:46:11 -07:00
experimental.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
external_sst_file_basic_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
external_sst_file_ingestion_job.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
external_sst_file_ingestion_job.h Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
external_sst_file_test.cc Disable fsync in some ExternalSSTFileTest tests (#7303) 2020-08-24 11:26:09 -07:00
fault_injection_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
file_indexer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filename_test.cc Add more tests to ASSERT_STATUS_CHECKED (#7211) 2020-08-06 17:19:41 -07:00
flush_job.cc Add flush_job_test to the list of ASSERT_STATUS_CHECKED tests (#7445) 2020-09-28 14:59:02 -07:00
flush_job.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
flush_job_test.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
flush_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator.cc Add status check enforcement for column_family_test.cc (#7484) 2020-10-02 13:35:15 -07:00
forward_iterator.h Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
forward_iterator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
import_column_family_job.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
import_column_family_job.h Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
import_column_family_test.cc Add few unit test cases in ASSERT_STATUS_CHECKED (#7500) 2020-10-08 11:22:44 -07:00
internal_stats.cc Introduce a blob file reader class (#7461) 2020-10-07 15:44:53 -07:00
internal_stats.h Introduce a blob file reader class (#7461) 2020-10-07 15:44:53 -07:00
job_context.h Expose the set of live blob files from Version/VersionSet (#6785) 2020-05-04 15:08:13 -07:00
listener_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
log_format.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
log_reader.cc Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_reader.h Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
log_writer.cc Fail recovery when MANIFEST record checksum mismatch (#6996) 2020-06-18 10:09:12 -07:00
log_writer.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
logs_with_prep_tracker.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
logs_with_prep_tracker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
lookup_key.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
manual_compaction_test.cc Skip high levels with no key falling in the range in CompactRange (#6482) 2020-03-04 20:15:25 -08:00
memtable.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
memtable.h Provide users with option to opt-in to get corrupt data in logs/messages (#7420) 2020-09-29 23:17:45 -07:00
memtable_list.cc Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
memtable_list.h Bug Fix for memtables not trimmed down. (#7296) 2020-08-21 13:29:05 -07:00
memtable_list_test.cc Add few unit test cases in ASSERT_STATUS_CHECKED build (#7427) 2020-09-24 21:48:57 -07:00
merge_context.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
merge_helper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_operator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
obsolete_files_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
options_file_test.cc Add more tests to ASSERT_STATUS_CHECKED (#7367) 2020-09-16 15:48:07 -07:00
output_validator.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
output_validator.h Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
perf_context_test.cc Fix MSVC-related build issues (#7439) 2020-10-01 09:23:04 -07:00
periodic_work_scheduler.cc Periodically flush info log out of application buffer (#7488) 2020-10-01 19:14:14 -07:00
periodic_work_scheduler.h Periodically flush info log out of application buffer (#7488) 2020-10-01 19:14:14 -07:00
periodic_work_scheduler_test.cc Periodically flush info log out of application buffer (#7488) 2020-10-01 19:14:14 -07:00
pinned_iterators_manager.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
plain_table_db_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
pre_release_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
prefix_test.cc Fix prefix_test for status check (#7495) 2020-10-02 17:01:15 -07:00
range_del_aggregator.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
range_del_aggregator.h Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
range_del_aggregator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
read_callback.h Get() with timestamp should respect snapshot (#7227) 2020-08-14 19:20:58 -07:00
repair.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
repair_test.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
snapshot_checker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_cache.cc Fix db_properties_test for ASSERT_STATUS_CHECKED (#7490) 2020-10-01 17:47:09 -07:00
table_cache.h Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
table_properties_collector.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
table_properties_collector.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_properties_collector_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
transaction_log_impl.cc Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
transaction_log_impl.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
trim_history_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
trim_history_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
version_builder.cc Enforce status check for corruption_test (#7453) 2020-10-02 22:11:00 -07:00
version_builder.h make L0 index/filter pinned memory usage predictable (#6911) 2020-06-09 16:51:23 -07:00
version_builder_test.cc Add version_builder_test to the list of ASSERT_STATUS_CHECKED tests (#7444) 2020-09-28 12:12:40 -07:00
version_edit.cc Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
version_edit.h Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
version_edit_handler.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_edit_handler.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_edit_test.cc Track WAL in MANIFEST: update WalMetadata for WAL syncing (#7414) 2020-09-22 14:35:14 -07:00
version_set.cc Add ldb_cmd_test to ASSERT_STATUS_CHECKED list (#7499) 2020-10-08 00:00:48 -07:00
version_set.h Fix TSAN build and re-enable the tests (#7386) 2020-09-25 14:46:28 -07:00
version_set_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
wal_edit.cc Track WAL in MANIFEST: add method to check WAL consistency (#7236) 2020-09-25 13:25:54 -07:00
wal_edit.h Track WAL in MANIFEST: add method to check WAL consistency (#7236) 2020-09-25 13:25:54 -07:00
wal_edit_test.cc Track WAL in MANIFEST: add method to check WAL consistency (#7236) 2020-09-25 13:25:54 -07:00
wal_manager.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
wal_manager.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
wal_manager_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
write_batch.cc Add trace_analyzer_test to ASSERT_STATUS_CHECKED list (#7480) 2020-10-01 15:58:52 -07:00
write_batch_base.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_internal.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
write_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_callback_test.cc Divide WriteCallbackTest.WriteWithCallbackTest (#7037) 2020-06-30 12:31:30 -07:00
write_controller.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_thread.cc Fix StallWrite crash with mixed of slowdown/no_slowdown writes (#7508) 2020-10-06 12:44:20 -07:00
write_thread.h Enforce status check for corruption_test (#7453) 2020-10-02 22:11:00 -07:00