rocksdb/db/compaction
Hui Xiao 9502856edd Add missing range conflict check between file ingestion and RefitLevel() (#10988)
Summary:
**Context:**
File ingestion never checks whether the key range it acts on overlaps with an ongoing RefitLevel() (used in `CompactRange()` with `change_level=true`). That's because RefitLevel() doesn't register and make its key range known to file ingestion. Though it checks overlapping with other compactions by https://github.com/facebook/rocksdb/blob/7.8.fb/db/external_sst_file_ingestion_job.cc#L998.

RefitLevel() (used in `CompactRange()` with `change_level=true`) doesn't check whether the key range it acts on overlaps with an ongoing file ingestion. That's because file ingestion does not register and make its key range known to other compactions.
- Note that non-refitlevel-compaction (e.g, manual compaction w/o RefitLevel() or general compaction) also does not check key range overlap with ongoing file ingestion for the same reason.
- But it's fine. Credited to cbi42's discovery, `WaitForIngestFile` was called by background and foreground compactions. They were introduced in 0f88160f67, 5c64fb67d2 and 87dfc1d23e.
- Regardless, this PR registers file ingestion like a compaction is a general approach that will also add range conflict check between file ingestion and non-refitlevel-compaction, though it has not been the issue motivated this PR.

Above are bugs resulting in two bad consequences:
- If file ingestion and RefitLevel() creates files in the same level, then range-overlapped files will be created at that level and caught as corruption by `force_consistency_checks=true`
- If file ingestion and RefitLevel() creates file in different levels, then with one further compaction on the ingested file, it can result in two same keys both with seqno 0 in two different levels. Then with iterator's [optimization](c62f322169/db/db_iter.cc (L342-L343)) that assumes no two same keys both with seqno 0, it will either break this assertion in debug build or, even worst, return value of this same key for the key after it, which is the wrong value to return, in release build.

Therefore we decide to introduce range conflict check for file ingestion and RefitLevel() inspired from the existing range conflict check among compactions.

**Summary:**
- Treat file ingestion job and RefitLevel() as `Compaction` of new compaction reasons: `CompactionReason::kExternalSstIngestion` and `CompactionReason::kRefitLevel` and register/unregister them.  File ingestion is treated as compaction from L0 to different levels and RefitLevel() as compaction from source level to target level.
- Check for `RangeOverlapWithCompaction` with other ongoing compactions, `RegisterCompaction()` on this "compaction" before changing the LSM state in `VersionStorageInfo`, and `UnregisterCompaction()` after changing.
- Replace scattered fixes (0f88160f67, 5c64fb67d2 and 87dfc1d23e.) that prevents overlapping between file ingestion and non-refit-level compaction with this fix cuz those practices are easy to overlook.
- Misc: logic cleanup, see PR comments

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10988

Test Plan:
- New unit test `DBCompactionTestWithOngoingFileIngestionParam*` that failed pre-fix and passed afterwards.
- Made compatible with existing tests, see PR comments
- make check
- [Ongoing] Stress test rehearsal with normal value and aggressive CI value https://github.com/facebook/rocksdb/pull/10761

Reviewed By: cbi42

Differential Revision: D41535685

Pulled By: hx235

fbshipit-source-id: 549833a577ba1496d20a870583d4caa737da1258
2022-12-29 15:05:36 -08:00
..
clipping_iterator.h Consider range tombstone in compaction output file cutting (#10802) 2022-12-15 09:11:54 -08:00
clipping_iterator_test.cc Print stack traces on frozen tests in CI (#10828) 2022-10-18 00:35:35 -07:00
compaction.cc Add missing range conflict check between file ingestion and RefitLevel() (#10988) 2022-12-29 15:05:36 -08:00
compaction.h Consider range tombstone in compaction output file cutting (#10802) 2022-12-15 09:11:54 -08:00
compaction_iteration_stats.h Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
compaction_iterator.cc Consider range tombstone in compaction output file cutting (#10802) 2022-12-15 09:11:54 -08:00
compaction_iterator.h Consider range tombstone in compaction output file cutting (#10802) 2022-12-15 09:11:54 -08:00
compaction_iterator_test.cc Basic Support for Merge with user-defined timestamp (#10819) 2022-10-31 22:28:58 -07:00
compaction_job.cc Add missing range conflict check between file ingestion and RefitLevel() (#10988) 2022-12-29 15:05:36 -08:00
compaction_job.h Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_job_stats_test.cc clang-format for db/compaction (#10882) 2022-10-26 12:35:12 -07:00
compaction_job_test.cc Include estimated bytes deleted by range tombstones in compensated file size (#10734) 2022-12-29 13:28:24 -08:00
compaction_outputs.cc Include estimated bytes deleted by range tombstones in compensated file size (#10734) 2022-12-29 13:28:24 -08:00
compaction_outputs.h Consider range tombstone in compaction output file cutting (#10802) 2022-12-15 09:11:54 -08:00
compaction_picker.cc Add missing range conflict check between file ingestion and RefitLevel() (#10988) 2022-12-29 15:05:36 -08:00
compaction_picker.h Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_picker_fifo.cc Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_picker_fifo.h Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_picker_level.cc Add missing range conflict check between file ingestion and RefitLevel() (#10988) 2022-12-29 15:05:36 -08:00
compaction_picker_level.h Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_picker_test.cc Include estimated bytes deleted by range tombstones in compensated file size (#10734) 2022-12-29 13:28:24 -08:00
compaction_picker_universal.cc Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_picker_universal.h Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_service_job.cc Sort L0 files by newly introduced epoch_num (#10922) 2022-12-13 13:29:37 -08:00
compaction_service_test.cc Deflake CompactionServiceTest.BasicCompactions (#10697) 2022-09-19 14:10:05 -07:00
compaction_state.cc Tiered Compaction: per key placement support (#9964) 2022-07-13 20:54:49 -07:00
compaction_state.h Tiered Compaction: per key placement support (#9964) 2022-07-13 20:54:49 -07:00
file_pri.h Try to start TTL earlier with kMinOverlappingRatio is used (#8749) 2021-11-01 14:36:31 -07:00
sst_partitioner.cc Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362) 2022-01-11 06:33:48 -08:00
subcompaction_state.cc Refactor Compaction file cut ShouldStopBefore() (#10629) 2022-09-14 22:09:12 -07:00
subcompaction_state.h Consider range tombstone in compaction output file cutting (#10802) 2022-12-15 09:11:54 -08:00
tiered_compaction_test.cc Add an unittest for Periodic compaction conflict with ongoing compaction (#10908) 2022-12-12 10:37:55 -08:00