dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
rocksdb/include/rocksdb
History
Andrew Ryan Chang af2a36d2c7 Record newest_key_time as a table property (#13083) 
Summary:
This PR does two things:
1. Adds a new table property `newest_key_time`
2. Uses this property to improve TTL and temperature change compaction.

### Context

The current `creation_time` table property should really be named `oldest_ancestor_time`. For flush output files, this is the oldest key time in the file. For compaction output files, this is the minimum among all oldest key times in the input files.

The problem with using the oldest ancestor time for TTL compaction is that we may end up dropping files earlier than we should. What we really want is the newest (i.e. "youngest") key time. Right now we take a roundabout way to estimate this value -- we take the value of the _oldest_ key time for the _next_ (newer) SST file. This is also why the current code has checks for `index >= 1`.

Our new property `newest_key_time` is set to the file creation time during flushes, and the max over all input files for compactions.

There were some additional smaller changes that I had to make for testing purposes:
- Refactoring the mock table reader to support specifying my own table properties
- Refactoring out a test utility method `GetLevelFileMetadatas`  that would otherwise be copy/pasted in 3 places

Credit to cbi42 for the problem explanation and proposed solution

### Testing

- Added a dedicated unit test to my `newest_key_time` logic in isolation (i.e. are we populating the property on flush and compaction)
- Updated the existing unit tests (for TTL/temperate change compaction), which were comprehensive enough to break when I first made my code changes. I removed the test setup code which set the file metadata `oldest_ancestor_time`, so we know we are actually only using the new table property instead.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13083

Reviewed By: cbi42

Differential Revision: D65298604

Pulled By: archang19

fbshipit-source-id: 898ef91b692ab33f5129a2a16b64ecadd4c32432
 		2024-11-01 10:08:35 -07:00
..
utilities Include TableProperties in the CompactionServiceResult (#13089) 2024-10-25 13:13:12 -07:00
advanced_cache.h Make Cache a customizable class (#13024) 2024-09-20 12:13:19 -07:00
advanced_options.h Compact one file at a time for FIFO temperature change compactions (#13018) 2024-09-19 15:50:41 -07:00
attribute_groups.h Couple of small improvements for (Iterator)AttributeGroup (#13076) 2024-10-16 20:58:26 -07:00
block_cache_trace_writer.h Update header file to include right copyright (#10854) 2022-10-24 16:13:16 -07:00
c.h Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
cache.h Implement secondary cache admission policy to allow all evicted blocks (#12599) 2024-05-02 11:23:35 -07:00
cache_bench_tool.h Allow cache_bench/db_bench to use a custom secondary cache (#8312) 2021-05-19 15:26:18 -07:00
cleanable.h Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899) 2022-04-26 21:59:24 -07:00
compaction_filter.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
compaction_job_stats.h Remove unnecessary semi-colon (#13034) 2024-09-25 15:32:22 -07:00
comparator.h Fix and generalize framework for filtering range queries, etc. (#13005) 2024-09-18 15:26:37 -07:00
compression_type.h Add CompressionOptions to the compressed secondary cache (#12234) 2024-01-16 12:21:27 -08:00
concurrent_task_limiter.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
configurable.h Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
convenience.h Fix and clarify ignore_unknown_options (#12989) 2024-09-04 11:42:04 -07:00
customizable.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
data_structure.h Improve SmallEnumSet (#11178) 2023-02-08 20:14:57 -08:00
db.h Clarify GetProperty API doc (#12829) 2024-07-02 13:15:00 -07:00
db_bench_tool.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
db_dump_tool.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_stress_tool.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
env.h Add Temperature to FileAttributes (#12965) 2024-08-23 20:06:41 -07:00
env_encryption.h Remove extra semi colon from internal_repo_rocksdb/repo/include/rocksdb/env_encryption.h 2024-01-23 08:35:47 -08:00
experimental.h Fix and generalize framework for filtering range queries, etc. (#13005) 2024-09-18 15:26:37 -07:00
file_checksum.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
file_system.h Options for file temperature for more files (#12957) 2024-08-23 19:49:25 -07:00
filter_policy.h Optimize, simplify filter block building (fix regression) (#12931) 2024-08-14 15:13:16 -07:00
flush_block_policy.h Some API clarifications (#9080) 2021-11-02 20:30:07 -07:00
functor_wrapper.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
io_status.h clang format files under include/ (#10850) 2022-10-24 14:09:32 -07:00
iostats_context.h clang format files under include/ (#10850) 2022-10-24 14:09:32 -07:00
iterator.h Add AttributeGroupIterator to Stress Test (#12776) 2024-06-17 11:25:30 -07:00
iterator_base.h Support allow_unprepared_value for multi-CF iterators (#13079) 2024-10-20 20:53:08 -07:00
ldb_tool.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
listener.h Add a OnManualFlushScheduled callback in event listener (#12631) 2024-06-06 17:29:01 -07:00
memory_allocator.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
memtablerep.h Add an option to verify memtable key order during reads (#12889) 2024-08-19 13:53:25 -07:00
merge_operator.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
metadata.h Support range deletion tombstones in `CreateColumnFamilyWithImport` (#11252) 2023-03-13 11:06:59 -07:00
options.h Support allow_unprepared_value for multi-CF iterators (#13079) 2024-10-20 20:53:08 -07:00
perf_context.h initialize member variables in `PerfContext`'s default constructor (#12581) 2024-04-25 10:24:34 -07:00
perf_level.h Add a perf level for measuring user thread block time (#12368) 2024-02-22 12:14:53 -08:00
persistent_cache.h clang format files under include/ (#10850) 2022-10-24 14:09:32 -07:00
port_defs.h Add SystemClock::TimedWait() function (#11753) 2023-08-29 18:39:10 -07:00
rate_limiter.h Decouple `RateLimiter` burst size and refill period (#12379) 2024-02-26 16:55:13 -08:00
rocksdb_namespace.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
secondary_cache.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
slice.h Avoid allocations/copies for large `GetMergeOperands()` results (#10458) 2022-08-04 00:42:13 -07:00
slice_transform.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
snapshot.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00
sst_dump_tool.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
sst_file_manager.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
sst_file_reader.h Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
sst_file_writer.h Add an API to check if an SST file is generated by SstFileWriter (#13072) 2024-10-16 16:57:05 -07:00
sst_partitioner.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
statistics.h Add ticker stats for read corruption retries (#12923) 2024-08-12 15:32:07 -07:00
stats_history.h More refactoring ahead of footer & meta changes (#9240) 2021-12-10 08:13:26 -08:00
status.h Improve efficiency of create_missing_column_families, light refactor (#11920) 2023-10-04 14:14:22 -07:00
system_clock.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
table.h Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
table_properties.h Record newest_key_time as a table property (#13083) 2024-11-01 10:08:35 -07:00
table_reader_caller.h Refactor block cache tracing APIs (#10811) 2022-10-21 12:15:35 -07:00
thread_status.h GetEntity Support for ReadOnlyDB and SecondaryDB (#11799) 2023-09-15 08:30:44 -07:00
threadpool.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
trace_reader_writer.h Update comments, fix typos. (#8721) 2021-08-27 13:16:32 -07:00
trace_record.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
trace_record_result.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
transaction_log.h Rename, deprecate `LogFile` and `VectorLogPtr` (#12695) 2024-05-28 09:24:49 -07:00
types.h Add Temperature to FileAttributes (#12965) 2024-08-23 20:06:41 -07:00
unique_id.h Adjust public APIs to prefer 128-bit SST unique ID (#10009) 2022-05-17 18:43:48 -07:00
universal_compaction.h Improve universal compaction sorted-run trigger (#12477) 2024-05-24 10:10:31 -07:00
user_write_callback.h Fix "no new line at end of file" (#12806) 2024-06-24 14:50:08 -07:00
version.h Start version 9.9.0 (#13093) 2024-10-25 13:47:29 -07:00
wal_filter.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
wide_columns.h Reset user-facing wide-column stuctures upon deserialization failures (#12562) 2024-04-18 13:08:34 -07:00
write_batch.h Add comment for memory usage in BeginTransaction() and WriteBatch::Clear() (#13042) 2024-09-30 10:27:45 -07:00
write_batch_base.h MultiCFIterator Refactor - CoalescingIterator & AttributeGroupIterator (#12480) 2024-04-11 11:34:04 -07:00
write_buffer_manager.h Add `SetAllowStall()` (#11335) 2023-03-30 09:43:33 -07:00