rocksdb/db
Hui Xiao f69a5fe8ee Cap compaction_readahead_size by max_sectors_kb (#12937)
Summary:
**Context/Summary:**
https://github.com/facebook/rocksdb/issues/12038 reported a regression where compaction read ahead does not work when `compaction_readahead_size ` is greater than `max_sectors_kb` defined in linux (i.e, largest I/O size that the OS issues to a block device, see https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt for more).

This PR fixes it by capping the `compaction_readahead_size` by `max_sectors_kb` if any. A refactoring of reading queue sys file is also included to reuse existing code.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/12937

Test Plan: https://github.com/facebook/rocksdb/issues/12038#issuecomment-2327618031 verified the regression was fixed

Reviewed By: anand1976

Differential Revision: D61350188

Pulled By: hx235

fbshipit-source-id: e10677f2f5854c22ebf6318b052557db94b98abe
2024-11-18 15:08:21 -08:00
..
blob Fix a leak of open Blob files (#13106) 2024-10-31 15:29:30 -07:00
compaction Rely on PurgeObsoleteFiles Only for Options file clean up when remote compaction is enabled (#13139) 2024-11-15 14:21:32 -08:00
db_impl Cap compaction_readahead_size by max_sectors_kb (#12937) 2024-11-18 15:08:21 -08:00
wide Fix the output of ldb dump_wal for PutEntity records (#12677) 2024-05-20 17:04:14 -07:00
arena_wrapped_db_iter.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
arena_wrapped_db_iter.h Support the on-demand loading of blobs during iteration (#13069) 2024-10-16 12:34:57 -07:00
attribute_group_iterator_impl.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
attribute_group_iterator_impl.h Some small improvements around allow_unprepared_value and multi-CF iterators (#13113) 2024-11-04 18:06:07 -08:00
builder.cc Fix and test for leaks of open SST files (#13117) 2024-11-08 10:54:43 -08:00
builder.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
c.cc Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
c_test.c Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
coalescing_iterator.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
coalescing_iterator.h Remove EXPERIMENTAL tag for MultiCfIterators (#13142) 2024-11-18 11:23:17 -08:00
column_family.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
column_family.h Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
column_family_test.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
compact_files_test.cc Prevent data block compression with BlockBasedTableOptions::block_align (#12592) 2024-04-26 20:05:30 -07:00
comparator_db_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
convenience.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
convenience_impl.h
corruption_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
cuckoo_table_db_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
db_basic_test.cc Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
db_block_cache_test.cc Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
db_bloom_filter_test.cc Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
db_clip_test.cc
db_compaction_filter_test.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
db_compaction_test.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
db_dynamic_level_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_encryption_test.cc
db_filesnapshot.cc Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
db_flush_test.cc Refactor SyncWAL and SyncClosedLogs for code sharing (#12707) 2024-05-30 14:53:13 -07:00
db_follower_test.cc Move file tracking in VersionEditHandlerPointInTime to VersionBuilder (#12928) 2024-08-12 21:09:37 -07:00
db_info_dumper.cc Don't log an error when an auxiliary dir is missing (#12326) 2024-02-05 10:26:41 -08:00
db_info_dumper.h
db_inplace_update_test.cc
db_io_failure_test.cc Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
db_iter.cc Add a temporary hook for custom yielding in long-running op (#13103) 2024-10-30 20:37:28 -07:00
db_iter.h Support the on-demand loading of blobs during iteration (#13069) 2024-10-16 12:34:57 -07:00
db_iter_stress_test.cc Automated modernization (#12210) 2024-01-05 11:53:57 -08:00
db_iter_test.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
db_iterator_test.cc Add Iterator property "rocksdb.iterator.is-value-pinned" (#12659) 2024-05-15 19:11:52 -07:00
db_kv_checksum_test.cc Fix MultiGet dropping memtable kv checksum corruption (#12842) 2024-08-08 13:34:11 -07:00
db_log_iter_test.cc Disable flaky part of TransactionLogIteratorCheckWhenArchive (#12423) 2024-03-12 12:54:53 -07:00
db_logical_block_size_cache_test.cc
db_memtable_test.cc Add an option to verify memtable key order during reads (#12889) 2024-08-19 13:53:25 -07:00
db_merge_operand_test.cc Add ContinueCallback to GetMergeOperands() (#12438) 2024-03-15 12:25:49 -07:00
db_merge_operator_test.cc
db_options_test.cc Refactoring toward making preserve/preclude options mutable (#13114) 2024-11-04 16:15:10 -08:00
db_properties_test.cc GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
db_range_del_test.cc Fail DeleteRange() early when row_cache is configured (#12710) 2024-05-29 15:03:15 -07:00
db_rate_limiter_test.cc Fix db_rate_limiter_test for win (#12816) 2024-07-01 16:14:19 -07:00
db_readonly_with_timestamp_test.cc Enforce status checking after Valid() returns false for IteratorWrapper (#11975) 2023-10-18 09:38:38 -07:00
db_secondary_test.cc Fix Compaction Stats (#13071) 2024-10-16 19:20:37 -07:00
db_sst_test.cc Fix orphaned files in SstFileManager (#13015) 2024-09-18 13:27:44 -07:00
db_statistics_test.cc Fix double counting of BYTES_WRITTEN ticker (#12111) 2023-12-08 17:12:11 -08:00
db_table_properties_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
db_tailing_iter_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_test.cc Improve DBTest.DynamicLevelCompressionPerLevel (#13044) 2024-10-09 12:51:19 -07:00
db_test2.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
db_test_util.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
db_test_util.h Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
db_universal_compaction_test.cc Option to decouple index and filter partitions (#12939) 2024-08-16 15:34:31 -07:00
db_wal_test.cc Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
db_with_timestamp_basic_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
db_with_timestamp_compaction_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_with_timestamp_test_util.cc
db_with_timestamp_test_util.h
db_write_buffer_manager_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_write_test.cc Disallow memtable flush and sst ingest while WAL is locked (#12652) 2024-05-21 10:17:34 -07:00
dbformat.cc Add some per key optimization for UDT in memtable only feature (#13031) 2024-10-03 17:57:50 -07:00
dbformat.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
dbformat_test.cc Refactor IndexBuilder::AddIndexEntry (#12867) 2024-07-22 14:27:31 -07:00
deletefile_test.cc Add an option to wait for purge in WaitForCompact (#12520) 2024-04-17 17:33:27 -07:00
error_handler.cc Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
error_handler.h Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
error_handler_fs_test.cc Remove the return value of SetBGError() (#12792) 2024-06-26 18:17:05 -07:00
event_helpers.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
event_helpers.h
experimental.cc Fix and generalize framework for filtering range queries, etc. (#13005) 2024-09-18 15:26:37 -07:00
external_sst_file_basic_test.cc Add compaction stats for filtered files (#13136) 2024-11-14 10:10:38 -08:00
external_sst_file_ingestion_job.cc Introduce a WriteBatchWithIndex-based implementation of ReadOnlyMemTable (#13123) 2024-11-12 09:27:11 -08:00
external_sst_file_ingestion_job.h Add some checks for the file ingestion flow (#13100) 2024-11-05 15:44:56 -08:00
external_sst_file_test.cc Add an ingestion option to not fill block cache (#13067) 2024-10-16 14:11:22 -07:00
fault_injection_test.cc FaultInjectionTestFS follow-up and clean-up (#12861) 2024-07-15 10:28:34 -07:00
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc Refactoring toward making preserve/preclude options mutable (#13114) 2024-11-04 16:15:10 -08:00
flush_job.h Fix a bug that can retain old WAL longer than needed (#13127) 2024-11-11 14:19:45 -08:00
flush_job_test.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
forward_iterator.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
forward_iterator_bench.cc
history_trimming_iterator.h
import_column_family_job.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
import_column_family_job.h
import_column_family_test.cc Introduce minimum compaction debt requirement for parallel compaction (#13054) 2024-10-04 15:01:54 -07:00
internal_stats.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
internal_stats.h Add compaction stats for filtered files (#13136) 2024-11-14 10:10:38 -08:00
job_context.h Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
kv_checksum.h
listener_test.cc Add some checks at property block creation side (#12898) 2024-07-31 13:28:17 -07:00
log_format.h
log_reader.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_reader.h Enable recycle_log_file_num option for point in time recovery (#12403) 2024-03-21 12:29:35 -07:00
log_test.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_writer.cc Handle injected write error after successful WAL write in crash test + misc (#12838) 2024-07-29 13:51:49 -07:00
log_writer.h Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
lookup_key.h
malloc_stats.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
malloc_stats.h
manual_compaction_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
memtable.cc Introduce a WriteBatchWithIndex-based implementation of ReadOnlyMemTable (#13123) 2024-11-12 09:27:11 -08:00
memtable.h Introduce a WriteBatchWithIndex-based implementation of ReadOnlyMemTable (#13123) 2024-11-12 09:27:11 -08:00
memtable_list.cc Fix a bug that can retain old WAL longer than needed (#13127) 2024-11-11 14:19:45 -08:00
memtable_list.h Fix a bug that can retain old WAL longer than needed (#13127) 2024-11-11 14:19:45 -08:00
memtable_list_test.cc Introduce an interface ReadOnlyMemTable for immutable memtables (#13107) 2024-11-04 16:09:34 -08:00
merge_context.h Add ContinueCallback to GetMergeOperands() (#12438) 2024-03-15 12:25:49 -07:00
merge_helper.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
merge_helper.h Eliminate some code duplication in MergeHelper (#12121) 2023-12-05 14:07:42 -08:00
merge_helper_test.cc
merge_operator.cc
merge_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
multi_cf_iterator_impl.h Some small improvements around allow_unprepared_value and multi-CF iterators (#13113) 2024-11-04 18:06:07 -08:00
multi_cf_iterator_test.cc Support allow_unprepared_value for multi-CF iterators (#13079) 2024-10-20 20:53:08 -07:00
obsolete_files_test.cc Remove the force mode for EnableFileDeletions API (#12337) 2024-02-13 18:36:25 -08:00
options_file_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
output_validator.cc Removed check_flush_compaction_key_order (#12311) 2024-01-31 16:30:26 -08:00
output_validator.h Removed check_flush_compaction_key_order (#12311) 2024-01-31 16:30:26 -08:00
perf_context_test.cc Add write_memtable_time to perf level kEnableWait (#12394) 2024-02-29 15:08:26 -08:00
periodic_task_scheduler.cc Remove extra semi colon from instagram/ranking/mezql/shots/parser/fast/Token.cpp 2024-03-04 06:32:50 -08:00
periodic_task_scheduler.h
periodic_task_scheduler_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
pinned_iterators_manager.h Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
plain_table_db_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
post_memtable_callback.h
pre_release_callback.h
prefix_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
range_del_aggregator.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator.h Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator_bench.cc
range_del_aggregator_test.cc
range_tombstone_fragmenter.cc Add support for range deletion when user timestamps are not persisted (#12254) 2024-01-29 11:37:34 -08:00
range_tombstone_fragmenter.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
range_tombstone_fragmenter_test.cc
read_callback.h
repair.cc Fix write committed transactions replay when UDT setting toggles (#13121) 2024-11-06 17:32:03 -08:00
repair_test.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
seqno_time_test.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
seqno_to_time_mapping.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
seqno_to_time_mapping.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
snapshot_checker.h Optimize compaction for standalone range deletion files (#13078) 2024-10-25 09:32:14 -07:00
snapshot_impl.cc
snapshot_impl.h
table_cache.cc Fix and test for leaks of open SST files (#13117) 2024-11-08 10:54:43 -08:00
table_cache.h Fix and test for leaks of open SST files (#13117) 2024-11-08 10:54:43 -08:00
table_cache_sync_and_async.h Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
table_properties_collector.cc
table_properties_collector.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
table_properties_collector_test.cc Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
transaction_log_impl.cc Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
transaction_log_impl.h Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Fix a leak of open Blob files (#13106) 2024-10-31 15:29:30 -07:00
version_builder.h Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
version_builder_test.cc Fix blob files not reclaimed after deleting all SSTs (#12235) 2024-01-16 11:15:23 -08:00
version_edit.cc Support ingesting SST files generated by a live DB (#12750) 2024-07-19 16:14:54 -07:00
version_edit.h Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
version_edit_handler.cc Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
version_edit_handler.h Add some documentation for version edit handlers (#12948) 2024-08-21 10:09:10 -07:00
version_edit_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
version_set.cc Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
version_set.h Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
version_set_sync_and_async.h Refactor table_factory into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
version_set_test.cc Fix missing cases of corruption retries (#13122) 2024-11-08 12:43:21 -08:00
version_util.h Group SST write in flush, compaction and db open with new stats (#11910) 2023-12-29 15:29:23 -08:00
wal_edit.cc
wal_edit.h
wal_edit_test.cc
wal_manager.cc Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
wal_manager.h Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
wal_manager_test.cc Group SST write in flush, compaction and db open with new stats (#11910) 2023-12-29 15:29:23 -08:00
write_batch.cc Fix write committed transactions replay when UDT setting toggles (#13121) 2024-11-06 17:32:03 -08:00
write_batch_base.cc
write_batch_internal.h Fix write committed transactions replay when UDT setting toggles (#13121) 2024-11-06 17:32:03 -08:00
write_batch_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
write_callback.h
write_callback_test.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_controller.cc
write_controller.h
write_controller_test.cc
write_stall_stats.cc
write_stall_stats.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
write_thread.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_thread.h Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00