dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-28 15:33:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Peter Dillinger 8a36543326 Steps toward preserve/preclude options mutable (#13124) 
Summary:
Follow-up to https://github.com/facebook/rocksdb/issues/13114

This change makes the options mutable in testing only through some internal hooks, so that we can keep the easier mechanics and testing of making the options mutable separate from a more interesting and critical fix needed for the options to be *safely* mutable. See https://github.com/facebook/rocksdb/pull/9964/files#r1024449523 for some background on the interesting remaining problem, which we've added a test for here, with the failing piece commented out (because it puts the DB in a failure state): PrecludeLastLevelTest.RangeTombstoneSnapshotMigrateFromLast.

The mechanics of making the options mutable turned out to be smaller than expected because `RegisterRecordSeqnoTimeWorker()` and `RecordSeqnoToTimeMapping()` are already robust to things like frequently switching between preserve/preclude durations e.g. with new and dropped column families, based on work from
 https://github.com/facebook/rocksdb/issues/11920, https://github.com/facebook/rocksdb/issues/11929, and https://github.com/facebook/rocksdb/issues/12253. Mostly, `options_mutex_` prevents races
in applying the options changes, and smart capacity enforcement in `SeqnoToTimeMapping` means it doesn't really matter if the periodic task wakes up too often by being re-scheduled repeatedly.

Functional changes needed other than marking mutable:
* Update periodic task registration (as needed) from SetOptions, with a mapping recorded then also in case it's needed.
* Install SuperVersion(s) with updated mapping when the registration function itself updates the mapping.

Possible follow-up (aside from already mentioned):
* Some FIXME code in RangeTombstoneSnapshotMigrateFromLast is present because Flush does not automatically include a seqno to time mapping entry that puts an upper bound on how new the flushed data is. This has the potential to be a measurable CPU impact so needs to be done carefully.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13124

Test Plan:
updated/refactored tests in tiered_compaction_test to parametrically use dynamic configuration changes (or DB restarts) when changing operating parameters such as these.

CheckInternalKeyRange test got some heavier refactoring in preparation for follow-up, and manually verified that the test still fails when relevant `if (!safe_to_penultimate_level) ...` code is disabled.

Reviewed By: jowlyzhang

Differential Revision: D65634146

Pulled By: pdillinger

fbshipit-source-id: 25c9d00fd5b7fd1b408b5f36d58dc48647970528
2024-11-18 19:34:01 -08:00
..
blob Fix a leak of open Blob files (#13106) 2024-10-31 15:29:30 -07:00
compaction Steps toward preserve/preclude options mutable (#13124) 2024-11-18 19:34:01 -08:00
db_impl Steps toward preserve/preclude options mutable (#13124) 2024-11-18 19:34:01 -08:00
wide Fix the output of ldb dump_wal for PutEntity records (#12677) 2024-05-20 17:04:14 -07:00
arena_wrapped_db_iter.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
arena_wrapped_db_iter.h Support the on-demand loading of blobs during iteration (#13069) 2024-10-16 12:34:57 -07:00
attribute_group_iterator_impl.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
attribute_group_iterator_impl.h Some small improvements around allow_unprepared_value and multi-CF iterators (#13113) 2024-11-04 18:06:07 -08:00
builder.cc Fix and test for leaks of open SST files (#13117) 2024-11-08 10:54:43 -08:00
builder.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
c.cc Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
c_test.c Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
coalescing_iterator.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
coalescing_iterator.h Remove EXPERIMENTAL tag for MultiCfIterators (#13142) 2024-11-18 11:23:17 -08:00
column_family.cc Steps toward preserve/preclude options mutable (#13124) 2024-11-18 19:34:01 -08:00
column_family.h Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
column_family_test.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
compact_files_test.cc Prevent data block compression with BlockBasedTableOptions::block_align (#12592) 2024-04-26 20:05:30 -07:00
comparator_db_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
convenience.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
convenience_impl.h
corruption_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
cuckoo_table_db_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
db_basic_test.cc Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
db_block_cache_test.cc Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
db_bloom_filter_test.cc Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
db_clip_test.cc
db_compaction_filter_test.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
db_compaction_test.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
db_dynamic_level_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_encryption_test.cc
db_filesnapshot.cc Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
db_flush_test.cc Refactor SyncWAL and SyncClosedLogs for code sharing (#12707) 2024-05-30 14:53:13 -07:00
db_follower_test.cc Move file tracking in VersionEditHandlerPointInTime to VersionBuilder (#12928) 2024-08-12 21:09:37 -07:00
db_info_dumper.cc
db_info_dumper.h
db_inplace_update_test.cc
db_io_failure_test.cc Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
db_iter.cc Add a temporary hook for custom yielding in long-running op (#13103) 2024-10-30 20:37:28 -07:00
db_iter.h Support the on-demand loading of blobs during iteration (#13069) 2024-10-16 12:34:57 -07:00
db_iter_stress_test.cc
db_iter_test.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
db_iterator_test.cc Add Iterator property "rocksdb.iterator.is-value-pinned" (#12659) 2024-05-15 19:11:52 -07:00
db_kv_checksum_test.cc Fix MultiGet dropping memtable kv checksum corruption (#12842) 2024-08-08 13:34:11 -07:00
db_log_iter_test.cc Disable flaky part of TransactionLogIteratorCheckWhenArchive (#12423) 2024-03-12 12:54:53 -07:00
db_logical_block_size_cache_test.cc
db_memtable_test.cc Add an option to verify memtable key order during reads (#12889) 2024-08-19 13:53:25 -07:00
db_merge_operand_test.cc Add ContinueCallback to GetMergeOperands() (#12438) 2024-03-15 12:25:49 -07:00
db_merge_operator_test.cc
db_options_test.cc Refactoring toward making preserve/preclude options mutable (#13114) 2024-11-04 16:15:10 -08:00
db_properties_test.cc GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
db_range_del_test.cc Fail DeleteRange() early when row_cache is configured (#12710) 2024-05-29 15:03:15 -07:00
db_rate_limiter_test.cc Fix db_rate_limiter_test for win (#12816) 2024-07-01 16:14:19 -07:00
db_readonly_with_timestamp_test.cc
db_secondary_test.cc Fix Compaction Stats (#13071) 2024-10-16 19:20:37 -07:00
db_sst_test.cc Fix orphaned files in SstFileManager (#13015) 2024-09-18 13:27:44 -07:00
db_statistics_test.cc
db_table_properties_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
db_tailing_iter_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_test.cc Improve DBTest.DynamicLevelCompressionPerLevel (#13044) 2024-10-09 12:51:19 -07:00
db_test2.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
db_test_util.cc Steps toward preserve/preclude options mutable (#13124) 2024-11-18 19:34:01 -08:00
db_test_util.h Steps toward preserve/preclude options mutable (#13124) 2024-11-18 19:34:01 -08:00
db_universal_compaction_test.cc Option to decouple index and filter partitions (#12939) 2024-08-16 15:34:31 -07:00
db_wal_test.cc Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
db_with_timestamp_basic_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
db_with_timestamp_compaction_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_with_timestamp_test_util.cc
db_with_timestamp_test_util.h
db_write_buffer_manager_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_write_test.cc Disallow memtable flush and sst ingest while WAL is locked (#12652) 2024-05-21 10:17:34 -07:00
dbformat.cc Add some per key optimization for UDT in memtable only feature (#13031) 2024-10-03 17:57:50 -07:00
dbformat.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
dbformat_test.cc Refactor IndexBuilder::AddIndexEntry (#12867) 2024-07-22 14:27:31 -07:00
deletefile_test.cc Add an option to wait for purge in WaitForCompact (#12520) 2024-04-17 17:33:27 -07:00
error_handler.cc Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
error_handler.h Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
error_handler_fs_test.cc Remove the return value of SetBGError() (#12792) 2024-06-26 18:17:05 -07:00
event_helpers.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
event_helpers.h
experimental.cc Fix and generalize framework for filtering range queries, etc. (#13005) 2024-09-18 15:26:37 -07:00
external_sst_file_basic_test.cc Add compaction stats for filtered files (#13136) 2024-11-14 10:10:38 -08:00
external_sst_file_ingestion_job.cc Introduce a WriteBatchWithIndex-based implementation of ReadOnlyMemTable (#13123) 2024-11-12 09:27:11 -08:00
external_sst_file_ingestion_job.h Add some checks for the file ingestion flow (#13100) 2024-11-05 15:44:56 -08:00
external_sst_file_test.cc Add an ingestion option to not fill block cache (#13067) 2024-10-16 14:11:22 -07:00
fault_injection_test.cc FaultInjectionTestFS follow-up and clean-up (#12861) 2024-07-15 10:28:34 -07:00
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc Refactoring toward making preserve/preclude options mutable (#13114) 2024-11-04 16:15:10 -08:00
flush_job.h Fix a bug that can retain old WAL longer than needed (#13127) 2024-11-11 14:19:45 -08:00
flush_job_test.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
forward_iterator.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
forward_iterator_bench.cc
history_trimming_iterator.h
import_column_family_job.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
import_column_family_job.h
import_column_family_test.cc Introduce minimum compaction debt requirement for parallel compaction (#13054) 2024-10-04 15:01:54 -07:00
internal_stats.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
internal_stats.h Add compaction stats for filtered files (#13136) 2024-11-14 10:10:38 -08:00
job_context.h Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
kv_checksum.h
listener_test.cc Add some checks at property block creation side (#12898) 2024-07-31 13:28:17 -07:00
log_format.h
log_reader.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_reader.h Enable recycle_log_file_num option for point in time recovery (#12403) 2024-03-21 12:29:35 -07:00
log_test.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_writer.cc Handle injected write error after successful WAL write in crash test + misc (#12838) 2024-07-29 13:51:49 -07:00
log_writer.h Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
lookup_key.h
malloc_stats.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
malloc_stats.h
manual_compaction_test.cc
memtable.cc Introduce a WriteBatchWithIndex-based implementation of ReadOnlyMemTable (#13123) 2024-11-12 09:27:11 -08:00
memtable.h Introduce a WriteBatchWithIndex-based implementation of ReadOnlyMemTable (#13123) 2024-11-12 09:27:11 -08:00
memtable_list.cc Fix a bug that can retain old WAL longer than needed (#13127) 2024-11-11 14:19:45 -08:00
memtable_list.h Fix a bug that can retain old WAL longer than needed (#13127) 2024-11-11 14:19:45 -08:00
memtable_list_test.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
merge_context.h Add ContinueCallback to GetMergeOperands() (#12438) 2024-03-15 12:25:49 -07:00
merge_helper.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
merge_helper.h
merge_helper_test.cc
merge_operator.cc
merge_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
multi_cf_iterator_impl.h Some small improvements around allow_unprepared_value and multi-CF iterators (#13113) 2024-11-04 18:06:07 -08:00
multi_cf_iterator_test.cc Support allow_unprepared_value for multi-CF iterators (#13079) 2024-10-20 20:53:08 -07:00
obsolete_files_test.cc Remove the force mode for EnableFileDeletions API (#12337) 2024-02-13 18:36:25 -08:00
options_file_test.cc
output_validator.cc
output_validator.h
perf_context_test.cc Add write_memtable_time to perf level kEnableWait (#12394) 2024-02-29 15:08:26 -08:00
periodic_task_scheduler.cc Remove extra semi colon from instagram/ranking/mezql/shots/parser/fast/Token.cpp 2024-03-04 06:32:50 -08:00
periodic_task_scheduler.h
periodic_task_scheduler_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
pinned_iterators_manager.h Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
plain_table_db_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
post_memtable_callback.h
pre_release_callback.h
prefix_test.cc
range_del_aggregator.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator.h Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator_bench.cc
range_del_aggregator_test.cc
range_tombstone_fragmenter.cc
range_tombstone_fragmenter.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
range_tombstone_fragmenter_test.cc
read_callback.h
repair.cc Fix write committed transactions replay when UDT setting toggles (#13121) 2024-11-06 17:32:03 -08:00
repair_test.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
seqno_time_test.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
seqno_to_time_mapping.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
seqno_to_time_mapping.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
snapshot_checker.h Optimize compaction for standalone range deletion files (#13078) 2024-10-25 09:32:14 -07:00
snapshot_impl.cc
snapshot_impl.h
table_cache.cc Fix and test for leaks of open SST files (#13117) 2024-11-08 10:54:43 -08:00
table_cache.h Fix and test for leaks of open SST files (#13117) 2024-11-08 10:54:43 -08:00
table_cache_sync_and_async.h Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
table_properties_collector.cc
table_properties_collector.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
table_properties_collector_test.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
transaction_log_impl.cc Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
transaction_log_impl.h Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Fix a leak of open Blob files (#13106) 2024-10-31 15:29:30 -07:00
version_builder.h Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
version_builder_test.cc
version_edit.cc Support ingesting SST files generated by a live DB (#12750) 2024-07-19 16:14:54 -07:00
version_edit.h Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
version_edit_handler.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
version_edit_handler.h Add some documentation for version edit handlers (#12948) 2024-08-21 10:09:10 -07:00
version_edit_test.cc
version_set.cc Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
version_set.h Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
version_set_sync_and_async.h Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
version_set_test.cc Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
version_util.h
wal_edit.cc
wal_edit.h
wal_edit_test.cc
wal_manager.cc Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
wal_manager.h Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
wal_manager_test.cc
write_batch.cc Fix write committed transactions replay when UDT setting toggles (#13121) 2024-11-06 17:32:03 -08:00
write_batch_base.cc
write_batch_internal.h Fix write committed transactions replay when UDT setting toggles (#13121) 2024-11-06 17:32:03 -08:00
write_batch_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
write_callback.h
write_callback_test.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_controller.cc
write_controller.h
write_controller_test.cc
write_stall_stats.cc
write_stall_stats.h
write_thread.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_thread.h Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00