rocksdb/utilities
Yu Zhang c73cf7a878 Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760)
Summary:
This PR adds user property collector factory `CompactForTieringCollectorFactory` to support observe SST file and mark it as need compaction for fast tracking data to the proper tier.

A triggering ratio `compaction_trigger_ratio_` can be configured to achieve the following:
1) Setting the ratio to be equal to or smaller than 0 disables this collector
2) Setting the ratio to be within (0, 1] will write the number of observed eligible entries into a user property and marks a file as need-compaction when aforementioned condition is met.
3) Setting the ratio to be higher than 1 can be used to just writes the user table property, and not mark any file as need compaction.
 For a column family that does not enable tiering feature, even if an effective configuration is provided, this collector is still disabled. For a file that is already on the last level, this collector is also disabled.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/12760

Test Plan: Added unit tests

Reviewed By: pdillinger

Differential Revision: D58734976

Pulled By: jowlyzhang

fbshipit-source-id: 6daab2c4f62b5c6689c3c03e3b3907bbbe6b7a81
2024-06-18 10:51:29 -07:00
..
agg_merge Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
backup fix gcc warning about dangling-reference in backup_engine_test (#12637) 2024-05-18 18:01:19 -07:00
blob_db Add GetLiveFilesStorageInfo to legacy BlobDB (#12468) 2024-04-05 13:50:27 -07:00
cassandra Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
checkpoint Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
compaction_filters Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
convenience Run clang-format on utilities/ (except utilities/transactions/) (#10853) 2022-10-24 16:38:09 -07:00
leveldb_options Put Cache and CacheWrapper in new public header (#11192) 2023-02-09 12:12:02 -08:00
memory Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
merge_operators Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
option_change_migration Disallow refitting more than 1 file from non-L0 to L0 (#12481) 2024-03-29 10:52:36 -07:00
options Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
persistent_cache Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
simulator_cache Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
table_properties_collectors Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
trace Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
transactions Add public API `WriteWithCallback` to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
ttl Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
write_batch_with_index Fix compile errors in C++23 (#12106) 2024-05-28 15:33:57 -07:00
cache_dump_load.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
cache_dump_load_impl.cc Add dump all keys for cache dumper impl (#12500) 2024-04-12 10:47:13 -07:00
cache_dump_load_impl.h Add dump all keys for cache dumper impl (#12500) 2024-04-12 10:47:13 -07:00
compaction_filters.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
counted_fs.cc Fix serious FSDirectory use-after-Close bug (missing fsync) (#10460) 2022-08-02 10:54:32 -07:00
counted_fs.h Explicitly closing all directory file descriptors (#10049) 2022-06-01 18:03:34 -07:00
debug.cc Add timestamp support in dump_wal/dump/idump (#12690) 2024-05-23 20:26:57 -07:00
env_mirror.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
env_mirror_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_timed.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_timed.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_timed_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
fault_injection_env.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
fault_injection_env.h Remove extra semi colon from internal_repo_rocksdb/repo/util/filelock_test.cc 2024-03-19 16:17:57 -07:00
fault_injection_fs.cc Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
fault_injection_fs.h Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
fault_injection_secondary_cache.cc Add some compressed and tiered secondary cache stats (#12150) 2023-12-15 11:34:08 -08:00
fault_injection_secondary_cache.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
memory_allocators.h Major Cache refactoring, CPU efficiency improvement (#10975) 2023-01-11 14:20:40 -08:00
merge_operators.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
merge_operators.h Run clang-format on utilities/ (except utilities/transactions/) (#10853) 2022-10-24 16:38:09 -07:00
object_registry.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
object_registry_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
types_util.cc Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
types_util_test.cc Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
util_merge_operators_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
wal_filter.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00