mirror of
https://github.com/facebook/rocksdb.git
synced 2024-11-27 02:44:18 +00:00
f02c708aa3
Summary: This PR is the first step for Issue https://github.com/facebook/rocksdb/issues/4811. Currently compaction output files are cut at point keys, and the decision is made mainly in `CompactionOutputs::ShouldStopBefore()`. This makes it possible for range tombstones to cause large compactions that does not respect `max_compaction_bytes`. For example, we can have a large range tombstone that overlaps with too many files from the next level. Another example is when there is a gap between a range tombstone and another key. The first issue may be more acceptable, as a lot of data is deleted. This PR address the second issue by calling `ShouldStopBefore()` for range tombstone start keys. The main change is for `CompactionIterator` to emit range tombstone start keys to be processed by `CompactionOutputs`. A new `CompactionMergingIterator` is introduced and only used under `CompactionIterator` for this purpose. Further improvement after this PR include 1) cut compaction output at some grandparent boundary key instead of at the next point key or range tombstone start key and 2) cut compaction output file within a large range tombstone (it may be easier and reasonable to only do it for range tombstones at the end of a compaction output). Pull Request resolved: https://github.com/facebook/rocksdb/pull/10802 Test Plan: - added unit tests in db_range_del_test. - stress test: `python3 tools/db_crashtest.py whitebox --[simple|enable_ts] --verify_iterator_with_expected_state_one_in=5 --delrangepercent=5 --prefixpercent=2 --writepercent=58 --readpercen=21 --duration=36000 --range_deletion_width=1000000` Reviewed By: ajkr, jay-zhuang Differential Revision: D40308827 Pulled By: cbi42 fbshipit-source-id: a8fd6f70a3f09d0ef7a40e006f6c964bba8c00df |
||
|---|---|---|
| .. | ||
| clipping_iterator.h | ||
| clipping_iterator_test.cc | ||
| compaction.cc | ||
| compaction.h | ||
| compaction_iteration_stats.h | ||
| compaction_iterator.cc | ||
| compaction_iterator.h | ||
| compaction_iterator_test.cc | ||
| compaction_job.cc | ||
| compaction_job.h | ||
| compaction_job_stats_test.cc | ||
| compaction_job_test.cc | ||
| compaction_outputs.cc | ||
| compaction_outputs.h | ||
| compaction_picker.cc | ||
| compaction_picker.h | ||
| compaction_picker_fifo.cc | ||
| compaction_picker_fifo.h | ||
| compaction_picker_level.cc | ||
| compaction_picker_level.h | ||
| compaction_picker_test.cc | ||
| compaction_picker_universal.cc | ||
| compaction_picker_universal.h | ||
| compaction_service_job.cc | ||
| compaction_service_test.cc | ||
| compaction_state.cc | ||
| compaction_state.h | ||
| file_pri.h | ||
| sst_partitioner.cc | ||
| subcompaction_state.cc | ||
| subcompaction_state.h | ||
| tiered_compaction_test.cc | ||