mirror of https://github.com/facebook/rocksdb.git
cc01985db0
Summary:
When we had a single thread pool for compactions, a thread could be busy for a long time (minutes) executing a compaction involving the bottom level. In multi-instance setups, the entire thread pool could be consumed by such bottom-level compactions. Then, top-level compactions (e.g., a few L0 files) would be blocked for a long time ("head-of-line blocking"). Such top-level compactions are critical to prevent compaction stalls as they can quickly reduce number of L0 files / sorted runs.
This diff introduces a bottom-priority queue for universal compactions including the bottom level. This alleviates the head-of-line blocking situation for fast, top-level compactions.
- Added `Env::Priority::BOTTOM` thread pool. This feature is only enabled if user explicitly configures it to have a positive number of threads.
- Changed `ThreadPoolImpl`'s default thread limit from one to zero. This change is invisible to users as we call `IncBackgroundThreadsIfNeeded` on the low-pri/high-pri pools during `DB::Open` with values of at least one. It is necessary, though, for bottom-pri to start with zero threads so the feature is disabled by default.
- Separated `ManualCompaction` into two parts in `PrepickedCompaction`. `PrepickedCompaction` is used for any compaction that's picked outside of its execution thread, either manual or automatic.
- Forward universal compactions involving last level to the bottom pool (worker thread's entry point is `BGWorkBottomCompaction`).
- Track `bg_bottom_compaction_scheduled_` so we can wait for bottom-level compactions to finish. We don't count them against the background jobs limits. So users of this feature will get an extra compaction for free.
Closes https://github.com/facebook/rocksdb/pull/2580
Differential Revision: D5422916
Pulled By: ajkr
fbshipit-source-id: a74bd11f1ea4933df3739b16808bb21fcd512333
|
||
|---|---|---|
| .. | ||
| aligned_buffer.h | ||
| allocator.h | ||
| arena.cc | ||
| arena.h | ||
| arena_test.cc | ||
| auto_roll_logger.cc | ||
| auto_roll_logger.h | ||
| auto_roll_logger_test.cc | ||
| autovector.h | ||
| autovector_test.cc | ||
| bloom.cc | ||
| bloom_test.cc | ||
| build_version.cc.in | ||
| build_version.h | ||
| cast_util.h | ||
| channel.h | ||
| coding.cc | ||
| coding.h | ||
| coding_test.cc | ||
| compaction_job_stats_impl.cc | ||
| comparator.cc | ||
| compression.h | ||
| concurrent_arena.cc | ||
| concurrent_arena.h | ||
| core_local.h | ||
| crc32c.cc | ||
| crc32c.h | ||
| crc32c_test.cc | ||
| delete_scheduler.cc | ||
| delete_scheduler.h | ||
| delete_scheduler_test.cc | ||
| dynamic_bloom.cc | ||
| dynamic_bloom.h | ||
| dynamic_bloom_test.cc | ||
| event_logger.cc | ||
| event_logger.h | ||
| event_logger_test.cc | ||
| fault_injection_test_env.cc | ||
| fault_injection_test_env.h | ||
| file_reader_writer.cc | ||
| file_reader_writer.h | ||
| file_reader_writer_test.cc | ||
| file_util.cc | ||
| file_util.h | ||
| filelock_test.cc | ||
| filename.cc | ||
| filename.h | ||
| filter_policy.cc | ||
| hash.cc | ||
| hash.h | ||
| hash_map.h | ||
| hash_test.cc | ||
| heap.h | ||
| heap_test.cc | ||
| kv_map.h | ||
| log_buffer.cc | ||
| log_buffer.h | ||
| log_write_bench.cc | ||
| logging.h | ||
| memory_usage.h | ||
| mpsc.h | ||
| murmurhash.cc | ||
| murmurhash.h | ||
| mutexlock.h | ||
| random.cc | ||
| random.h | ||
| rate_limiter.cc | ||
| rate_limiter.h | ||
| rate_limiter_test.cc | ||
| slice.cc | ||
| slice_transform_test.cc | ||
| sst_file_manager_impl.cc | ||
| sst_file_manager_impl.h | ||
| status.cc | ||
| status_message.cc | ||
| stderr_logger.h | ||
| stop_watch.h | ||
| string_util.cc | ||
| string_util.h | ||
| sync_point.cc | ||
| sync_point.h | ||
| testharness.cc | ||
| testharness.h | ||
| testutil.cc | ||
| testutil.h | ||
| thread_list_test.cc | ||
| thread_local.cc | ||
| thread_local.h | ||
| thread_local_test.cc | ||
| thread_operation.h | ||
| threadpool_imp.cc | ||
| threadpool_imp.h | ||
| timer_queue.h | ||
| timer_queue_test.cc | ||
| transaction_test_util.cc | ||
| transaction_test_util.h | ||
| xxhash.cc | ||
| xxhash.h | ||