dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-28 05:43:50 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Yu Zhang 8592517c89 Remove stale entries from L0 files when UDT is not persisted (#13035) 
Summary:
When user-defined timestamps are not persisted, currently we replace the actual timestamp with min timestamp after an entry is output from compaction iterator. Compaction iterator won't be able to help with removing stale entries this way. This PR adds a wrapper iterator `TimestampStrippingIterator` for `MemTableIterator` that does the min timestamp replacement at the memtable iteration step. It is used by flush and can help remove stale entries from landing in L0 files.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13035

Test Plan: Added unit test

Reviewed By: pdillinger, cbi42

Differential Revision: D63423682

Pulled By: jowlyzhang

fbshipit-source-id: 087dcc9cee97b9ea51b8d2b88dc91c2984d54e55
2024-10-14 12:28:35 -07:00
..
blob Fixed MultiGet() error handling to not skip blob dereference (#12597) 2024-04-29 14:18:42 -07:00
compaction Fix a bug for surfacing write unix time (#13057) 2024-10-08 11:31:51 -07:00
db_impl Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
wide Fix the output of ldb dump_wal for PutEntity records (#12677) 2024-05-20 17:04:14 -07:00
arena_wrapped_db_iter.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
arena_wrapped_db_iter.h Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
attribute_group_iterator_impl.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
attribute_group_iterator_impl.h Templatize MultiCfIteratorImpl to avoid std::function's overhead (#13052) 2024-10-02 19:18:15 -07:00
builder.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
builder.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
c.cc Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
c_test.c Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
coalescing_iterator.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
coalescing_iterator.h Templatize MultiCfIteratorImpl to avoid std::function's overhead (#13052) 2024-10-02 19:18:15 -07:00
column_family.cc Introduce minimum compaction debt requirement for parallel compaction (#13054) 2024-10-04 15:01:54 -07:00
column_family.h Fix needs_flush assertion in file ingestion (#13045) 2024-10-02 17:19:18 -07:00
column_family_test.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
compact_files_test.cc Prevent data block compression with BlockBasedTableOptions::block_align (#12592) 2024-04-26 20:05:30 -07:00
comparator_db_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
convenience.cc Inject more errors to more files in stress test (#12713) 2024-06-19 08:42:00 -07:00
convenience_impl.h
corruption_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
cuckoo_table_db_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
db_basic_test.cc Steps toward making IDENTITY file obsolete (#13019) 2024-09-19 14:05:21 -07:00
db_block_cache_test.cc Fix rare failure in DBBlockCacheTypeTest.Uncache (#12775) 2024-06-14 20:50:36 -07:00
db_bloom_filter_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
db_clip_test.cc
db_compaction_filter_test.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
db_compaction_test.cc Compact one file at a time for FIFO temperature change compactions (#13018) 2024-09-19 15:50:41 -07:00
db_dynamic_level_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_encryption_test.cc
db_filesnapshot.cc Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
db_flush_test.cc Refactor SyncWAL and SyncClosedLogs for code sharing (#12707) 2024-05-30 14:53:13 -07:00
db_follower_test.cc Move file tracking in VersionEditHandlerPointInTime to VersionBuilder (#12928) 2024-08-12 21:09:37 -07:00
db_info_dumper.cc Don't log an error when an auxiliary dir is missing (#12326) 2024-02-05 10:26:41 -08:00
db_info_dumper.h
db_inplace_update_test.cc
db_io_failure_test.cc Fix a couple of missing cases of retry on corruption (#13007) 2024-09-13 13:56:49 -07:00
db_iter.cc Refactor the BlobDB-related parts of DBIter (#13061) 2024-10-10 15:38:59 -07:00
db_iter.h Refactor the BlobDB-related parts of DBIter (#13061) 2024-10-10 15:38:59 -07:00
db_iter_stress_test.cc Automated modernization (#12210) 2024-01-05 11:53:57 -08:00
db_iter_test.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
db_iterator_test.cc Add Iterator property "rocksdb.iterator.is-value-pinned" (#12659) 2024-05-15 19:11:52 -07:00
db_kv_checksum_test.cc Fix MultiGet dropping memtable kv checksum corruption (#12842) 2024-08-08 13:34:11 -07:00
db_log_iter_test.cc Disable flaky part of TransactionLogIteratorCheckWhenArchive (#12423) 2024-03-12 12:54:53 -07:00
db_logical_block_size_cache_test.cc
db_memtable_test.cc Add an option to verify memtable key order during reads (#12889) 2024-08-19 13:53:25 -07:00
db_merge_operand_test.cc Add ContinueCallback to GetMergeOperands() (#12438) 2024-03-15 12:25:49 -07:00
db_merge_operator_test.cc
db_options_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_properties_test.cc GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
db_range_del_test.cc Fail DeleteRange() early when row_cache is configured (#12710) 2024-05-29 15:03:15 -07:00
db_rate_limiter_test.cc Fix db_rate_limiter_test for win (#12816) 2024-07-01 16:14:19 -07:00
db_readonly_with_timestamp_test.cc Enforce status checking after Valid() returns false for IteratorWrapper (#11975) 2023-10-18 09:38:38 -07:00
db_secondary_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_sst_test.cc Fix orphaned files in SstFileManager (#13015) 2024-09-18 13:27:44 -07:00
db_statistics_test.cc Fix double counting of BYTES_WRITTEN ticker (#12111) 2023-12-08 17:12:11 -08:00
db_table_properties_test.cc Run internal cpp modernizer on RocksDB repo (#12398) 2024-03-04 10:08:32 -08:00
db_tailing_iter_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_test.cc Improve DBTest.DynamicLevelCompressionPerLevel (#13044) 2024-10-09 12:51:19 -07:00
db_test2.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
db_test_util.cc Option to decouple index and filter partitions (#12939) 2024-08-16 15:34:31 -07:00
db_test_util.h Options for file temperature for more files (#12957) 2024-08-23 19:49:25 -07:00
db_universal_compaction_test.cc Option to decouple index and filter partitions (#12939) 2024-08-16 15:34:31 -07:00
db_wal_test.cc Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
db_with_timestamp_basic_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
db_with_timestamp_compaction_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_with_timestamp_test_util.cc
db_with_timestamp_test_util.h
db_write_buffer_manager_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
db_write_test.cc Disallow memtable flush and sst ingest while WAL is locked (#12652) 2024-05-21 10:17:34 -07:00
dbformat.cc Add some per key optimization for UDT in memtable only feature (#13031) 2024-10-03 17:57:50 -07:00
dbformat.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
dbformat_test.cc Refactor IndexBuilder::AddIndexEntry (#12867) 2024-07-22 14:27:31 -07:00
deletefile_test.cc Add an option to wait for purge in WaitForCompact (#12520) 2024-04-17 17:33:27 -07:00
error_handler.cc Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
error_handler.h Fix a bug with auto recovery on WAL write error (#12995) 2024-09-17 14:10:33 -07:00
error_handler_fs_test.cc Remove the return value of SetBGError() (#12792) 2024-06-26 18:17:05 -07:00
event_helpers.cc Record largest seqno in table properties and verify in file ingestion (#12951) 2024-08-21 16:24:18 -07:00
event_helpers.h
experimental.cc Fix and generalize framework for filtering range queries, etc. (#13005) 2024-09-18 15:26:37 -07:00
external_sst_file_basic_test.cc Use same logic to assign level for non-overlapping files in universal compaction (#13059) 2024-10-11 11:18:45 -07:00
external_sst_file_ingestion_job.cc Use same logic to assign level for non-overlapping files in universal compaction (#13059) 2024-10-11 11:18:45 -07:00
external_sst_file_ingestion_job.h Use extended file boundary for key range overlap check during file ingestion (#12735) 2024-06-04 13:39:51 -07:00
external_sst_file_test.cc Use same logic to assign level for non-overlapping files in universal compaction (#13059) 2024-10-11 11:18:45 -07:00
fault_injection_test.cc FaultInjectionTestFS follow-up and clean-up (#12861) 2024-07-15 10:28:34 -07:00
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
flush_job.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
flush_job_test.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc Fix a bug for surfacing write unix time (#13057) 2024-10-08 11:31:51 -07:00
forward_iterator.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
forward_iterator_bench.cc
history_trimming_iterator.h
import_column_family_job.cc Fix a corruption bug in CreateColumnFamilyWithImport() (#12602) 2024-05-06 11:01:38 -07:00
import_column_family_job.h
import_column_family_test.cc Introduce minimum compaction debt requirement for parallel compaction (#13054) 2024-10-04 15:01:54 -07:00
internal_stats.cc GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
internal_stats.h GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
job_context.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
kv_checksum.h
listener_test.cc Add some checks at property block creation side (#12898) 2024-07-31 13:28:17 -07:00
log_format.h
log_reader.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_reader.h Enable recycle_log_file_num option for point in time recovery (#12403) 2024-03-21 12:29:35 -07:00
log_test.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_writer.cc Handle injected write error after successful WAL write in crash test + misc (#12838) 2024-07-29 13:51:49 -07:00
log_writer.h Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
lookup_key.h
malloc_stats.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
malloc_stats.h
manual_compaction_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
memtable.cc Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
memtable.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
memtable_list.cc Fix needs_flush assertion in file ingestion (#13045) 2024-10-02 17:19:18 -07:00
memtable_list.h Fix needs_flush assertion in file ingestion (#13045) 2024-10-02 17:19:18 -07:00
memtable_list_test.cc Fix MultiGet dropping memtable kv checksum corruption (#12842) 2024-08-08 13:34:11 -07:00
merge_context.h Add ContinueCallback to GetMergeOperands() (#12438) 2024-03-15 12:25:49 -07:00
merge_helper.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
merge_helper.h Eliminate some code duplication in MergeHelper (#12121) 2023-12-05 14:07:42 -08:00
merge_helper_test.cc
merge_operator.cc
merge_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
multi_cf_iterator_impl.h Templatize MultiCfIteratorImpl to avoid std::function's overhead (#13052) 2024-10-02 19:18:15 -07:00
multi_cf_iterator_test.cc Fix IteratorsConsistentView tests (#12582) 2024-04-25 14:06:46 -07:00
obsolete_files_test.cc Remove the force mode for EnableFileDeletions API (#12337) 2024-02-13 18:36:25 -08:00
options_file_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
output_validator.cc Removed check_flush_compaction_key_order (#12311) 2024-01-31 16:30:26 -08:00
output_validator.h Removed check_flush_compaction_key_order (#12311) 2024-01-31 16:30:26 -08:00
perf_context_test.cc Add write_memtable_time to perf level kEnableWait (#12394) 2024-02-29 15:08:26 -08:00
periodic_task_scheduler.cc Remove extra semi colon from instagram/ranking/mezql/shots/parser/fast/Token.cpp 2024-03-04 06:32:50 -08:00
periodic_task_scheduler.h
periodic_task_scheduler_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
pinned_iterators_manager.h Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
plain_table_db_test.cc Prefer static_cast in place of most reinterpret_cast (#12308) 2024-02-07 10:44:11 -08:00
post_memtable_callback.h
pre_release_callback.h
prefix_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
range_del_aggregator.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator.h Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator_bench.cc
range_del_aggregator_test.cc
range_tombstone_fragmenter.cc Add support for range deletion when user timestamps are not persisted (#12254) 2024-01-29 11:37:34 -08:00
range_tombstone_fragmenter.h Remove stale entries from L0 files when UDT is not persisted (#13035) 2024-10-14 12:28:35 -07:00
range_tombstone_fragmenter_test.cc
read_callback.h
repair.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
repair_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
seqno_time_test.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
seqno_to_time_mapping.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
seqno_to_time_mapping.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
snapshot_checker.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
snapshot_impl.cc
snapshot_impl.h
table_cache.cc Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
table_cache.h Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
table_cache_sync_and_async.h Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
table_properties_collector.cc
table_properties_collector.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
table_properties_collector_test.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
transaction_log_impl.cc Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
transaction_log_impl.h Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Add some documentation for version edit handlers (#12948) 2024-08-21 10:09:10 -07:00
version_builder.h Best efforts recovery recover seqno prefix (#12938) 2024-08-16 17:18:54 -07:00
version_builder_test.cc Fix blob files not reclaimed after deleting all SSTs (#12235) 2024-01-16 11:15:23 -08:00
version_edit.cc Support ingesting SST files generated by a live DB (#12750) 2024-07-19 16:14:54 -07:00
version_edit.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
version_edit_handler.cc Best efforts recovery recover seqno prefix (#12938) 2024-08-16 17:18:54 -07:00
version_edit_handler.h Add some documentation for version edit handlers (#12948) 2024-08-21 10:09:10 -07:00
version_edit_test.cc internal_repo_rocksdb (4372117296613874540) (#12117) 2023-12-04 11:17:32 -08:00
version_set.cc Fix a bug for surfacing write unix time (#13057) 2024-10-08 11:31:51 -07:00
version_set.h Deshim coro in fbcode/internal_repo_rocksdb 2024-09-14 09:48:21 -07:00
version_set_sync_and_async.h
version_set_test.cc Change the semantics of blob_garbage_collection_force_threshold to provide better control over space amp (#13022) 2024-09-19 15:47:13 -07:00
version_util.h Group SST write in flush, compaction and db open with new stats (#11910) 2023-12-29 15:29:23 -08:00
wal_edit.cc
wal_edit.h
wal_edit_test.cc
wal_manager.cc Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
wal_manager.h Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831) 2024-07-01 23:29:02 -07:00
wal_manager_test.cc Group SST write in flush, compaction and db open with new stats (#11910) 2023-12-29 15:29:23 -08:00
write_batch.cc Print unknown writebatch tag (#13062) 2024-10-10 15:34:35 -07:00
write_batch_base.cc
write_batch_internal.h Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
write_batch_test.cc Steps toward deprecating implicit prefix seek, related fixes (#13026) 2024-09-20 15:54:19 -07:00
write_callback.h
write_callback_test.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_controller.cc
write_controller.h
write_controller_test.cc
write_stall_stats.cc
write_stall_stats.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
write_thread.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_thread.h Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00