Yu Zhang dc34a0ff1e Add some checks for the file ingestion flow (#13100) ... Summary: This PR does a few misc things for file ingestion flow: - Add an invalid argument status return for the combination of `allow_global_seqno = false` and external files' key range overlap in `Prepare` stage. - Add a MemTables status check for when column family is flushed before `Run`. - Replace the column family dropped check with an assertion after thread enters the write queue and before it exits the write queue, since dropping column family can only happen in the single threaded write queue too and we already checked once after enter write queue. - Add an `ExternalSstFileIngestionJob::GetColumnFamilyData` API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13100 Test Plan: Added unit tests, and stress tested the ingestion path Reviewed By: hx235 Differential Revision: D65180472 Pulled By: jowlyzhang fbshipit-source-id: 180145dd248a7507a13a543481b135e5a31ebe2d		2024-11-05 15:44:56 -08:00
..
blob	Fix a leak of open Blob files (#13106)	2024-10-31 15:29:30 -07:00
compaction	Remove early return when scanning files for temperature change compaction (#13112)	2024-11-05 09:12:39 -08:00
db_impl	Add some checks for the file ingestion flow (#13100)	2024-11-05 15:44:56 -08:00
wide	Fix the output of `ldb dump_wal` for PutEntity records (#12677)	2024-05-20 17:04:14 -07:00
arena_wrapped_db_iter.cc	Steps toward deprecating implicit prefix seek, related fixes (#13026)	2024-09-20 15:54:19 -07:00
arena_wrapped_db_iter.h	Support the on-demand loading of blobs during iteration (#13069)	2024-10-16 12:34:57 -07:00
attribute_group_iterator_impl.cc	MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534)	2024-04-16 08:45:38 -07:00
attribute_group_iterator_impl.h	Some small improvements around allow_unprepared_value and multi-CF iterators (#13113)	2024-11-04 18:06:07 -08:00
builder.cc	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
builder.h	Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760)	2024-06-18 10:51:29 -07:00
c.cc	Steps toward making IDENTITY file obsolete (#13019)	2024-09-19 14:05:21 -07:00
c_test.c	Steps toward making IDENTITY file obsolete (#13019)	2024-09-19 14:05:21 -07:00
coalescing_iterator.cc	MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534)	2024-04-16 08:45:38 -07:00
coalescing_iterator.h	Some small improvements around allow_unprepared_value and multi-CF iterators (#13113)	2024-11-04 18:06:07 -08:00
column_family.cc	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
column_family.h	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
column_family_test.cc	Remove stale entries from L0 files when UDT is not persisted (#13035)	2024-10-14 12:28:35 -07:00
compact_files_test.cc	Prevent data block compression with `BlockBasedTableOptions::block_align` (#12592)	2024-04-26 20:05:30 -07:00
comparator_db_test.cc	Run internal cpp modernizer on RocksDB repo (#12398)	2024-03-04 10:08:32 -08:00
convenience.cc	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
convenience_impl.h	…
corruption_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
cuckoo_table_db_test.cc	Run internal cpp modernizer on RocksDB repo (#12398)	2024-03-04 10:08:32 -08:00
db_basic_test.cc	Steps toward making IDENTITY file obsolete (#13019)	2024-09-19 14:05:21 -07:00
db_block_cache_test.cc	Fix race to make BlockBasedTableOptions effectively mutable (#13082)	2024-10-25 10:24:54 -07:00
db_bloom_filter_test.cc	Fix race to make BlockBasedTableOptions effectively mutable (#13082)	2024-10-25 10:24:54 -07:00
db_clip_test.cc	…
db_compaction_filter_test.cc	Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470)	2024-03-22 13:40:42 -07:00
db_compaction_test.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
db_dynamic_level_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
db_encryption_test.cc	…
db_filesnapshot.cc	Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831)	2024-07-01 23:29:02 -07:00
db_flush_test.cc	Refactor SyncWAL and SyncClosedLogs for code sharing (#12707)	2024-05-30 14:53:13 -07:00
db_follower_test.cc	Move file tracking in VersionEditHandlerPointInTime to VersionBuilder (#12928)	2024-08-12 21:09:37 -07:00
db_info_dumper.cc	…
db_info_dumper.h	…
db_inplace_update_test.cc	…
db_io_failure_test.cc	Fix a couple of missing cases of retry on corruption (#13007)	2024-09-13 13:56:49 -07:00
db_iter.cc	Add a temporary hook for custom yielding in long-running op (#13103)	2024-10-30 20:37:28 -07:00
db_iter.h	Support the on-demand loading of blobs during iteration (#13069)	2024-10-16 12:34:57 -07:00
db_iter_stress_test.cc	…
db_iter_test.cc	Add initial support for TimedPut API (#12419)	2024-03-14 15:44:55 -07:00
db_iterator_test.cc	Add `Iterator` property "rocksdb.iterator.is-value-pinned" (#12659)	2024-05-15 19:11:52 -07:00
db_kv_checksum_test.cc	Fix MultiGet dropping memtable kv checksum corruption (#12842)	2024-08-08 13:34:11 -07:00
db_log_iter_test.cc	Disable flaky part of TransactionLogIteratorCheckWhenArchive (#12423)	2024-03-12 12:54:53 -07:00
db_logical_block_size_cache_test.cc	…
db_memtable_test.cc	Add an option to verify memtable key order during reads (#12889)	2024-08-19 13:53:25 -07:00
db_merge_operand_test.cc	Add `ContinueCallback` to `GetMergeOperands()` (#12438)	2024-03-15 12:25:49 -07:00
db_merge_operator_test.cc	…
db_options_test.cc	Refactoring toward making preserve/preclude options mutable (#13114)	2024-11-04 16:15:10 -08:00
db_properties_test.cc	`GetAggregatedIntProperty` accumulates property once per block cache (#12755)	2024-06-18 10:46:55 -07:00
db_range_del_test.cc	Fail DeleteRange() early when row_cache is configured (#12710)	2024-05-29 15:03:15 -07:00
db_rate_limiter_test.cc	Fix db_rate_limiter_test for win (#12816)	2024-07-01 16:14:19 -07:00
db_readonly_with_timestamp_test.cc	…
db_secondary_test.cc	Fix Compaction Stats (#13071)	2024-10-16 19:20:37 -07:00
db_sst_test.cc	Fix orphaned files in SstFileManager (#13015)	2024-09-18 13:27:44 -07:00
db_statistics_test.cc	…
db_table_properties_test.cc	Run internal cpp modernizer on RocksDB repo (#12398)	2024-03-04 10:08:32 -08:00
db_tailing_iter_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
db_test.cc	Improve DBTest.DynamicLevelCompressionPerLevel (#13044)	2024-10-09 12:51:19 -07:00
db_test2.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
db_test_util.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
db_test_util.h	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
db_universal_compaction_test.cc	Option to decouple index and filter partitions (#12939)	2024-08-16 15:34:31 -07:00
db_wal_test.cc	Fix a bug with auto recovery on WAL write error (#12995)	2024-09-17 14:10:33 -07:00
db_with_timestamp_basic_test.cc	Steps toward deprecating implicit prefix seek, related fixes (#13026)	2024-09-20 15:54:19 -07:00
db_with_timestamp_compaction_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
db_with_timestamp_test_util.cc	…
db_with_timestamp_test_util.h	…
db_write_buffer_manager_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
db_write_test.cc	Disallow memtable flush and sst ingest while WAL is locked (#12652)	2024-05-21 10:17:34 -07:00
dbformat.cc	Add some per key optimization for UDT in memtable only feature (#13031)	2024-10-03 17:57:50 -07:00
dbformat.h	Remove stale entries from L0 files when UDT is not persisted (#13035)	2024-10-14 12:28:35 -07:00
dbformat_test.cc	Refactor IndexBuilder::AddIndexEntry (#12867)	2024-07-22 14:27:31 -07:00
deletefile_test.cc	Add an option to wait for purge in WaitForCompact (#12520)	2024-04-17 17:33:27 -07:00
error_handler.cc	Fix a bug with auto recovery on WAL write error (#12995)	2024-09-17 14:10:33 -07:00
error_handler.h	Fix a bug with auto recovery on WAL write error (#12995)	2024-09-17 14:10:33 -07:00
error_handler_fs_test.cc	Remove the return value of `SetBGError()` (#12792)	2024-06-26 18:17:05 -07:00
event_helpers.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
event_helpers.h	…
experimental.cc	Fix and generalize framework for filtering range queries, etc. (#13005)	2024-09-18 15:26:37 -07:00
external_sst_file_basic_test.cc	Add some checks for the file ingestion flow (#13100)	2024-11-05 15:44:56 -08:00
external_sst_file_ingestion_job.cc	Add some checks for the file ingestion flow (#13100)	2024-11-05 15:44:56 -08:00
external_sst_file_ingestion_job.h	Add some checks for the file ingestion flow (#13100)	2024-11-05 15:44:56 -08:00
external_sst_file_test.cc	Add an ingestion option to not fill block cache (#13067)	2024-10-16 14:11:22 -07:00
fault_injection_test.cc	FaultInjectionTestFS follow-up and clean-up (#12861)	2024-07-15 10:28:34 -07:00
file_indexer.cc	…
file_indexer.h	…
file_indexer_test.cc	…
filename_test.cc	…
flush_job.cc	Refactoring toward making preserve/preclude options mutable (#13114)	2024-11-04 16:15:10 -08:00
flush_job.h	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
flush_job_test.cc	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
flush_scheduler.cc	…
flush_scheduler.h	…
forward_iterator.cc	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
forward_iterator.h	Support returning write unix time in iterator property (#12428)	2024-03-15 15:37:37 -07:00
forward_iterator_bench.cc	…
history_trimming_iterator.h	…
import_column_family_job.cc	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
import_column_family_job.h	…
import_column_family_test.cc	Introduce minimum compaction debt requirement for parallel compaction (#13054)	2024-10-04 15:01:54 -07:00
internal_stats.cc	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
internal_stats.h	`GetAggregatedIntProperty` accumulates property once per block cache (#12755)	2024-06-18 10:46:55 -07:00
job_context.h	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
kv_checksum.h	…
listener_test.cc	Add some checks at property block creation side (#12898)	2024-07-31 13:28:17 -07:00
log_format.h	…
log_reader.cc	Fix recycled WAL detection when wal_compression is enabled (#12643)	2024-05-22 15:34:37 -07:00
log_reader.h	Enable recycle_log_file_num option for point in time recovery (#12403)	2024-03-21 12:29:35 -07:00
log_test.cc	Fix recycled WAL detection when wal_compression is enabled (#12643)	2024-05-22 15:34:37 -07:00
log_writer.cc	Handle injected write error after successful WAL write in crash test + misc (#12838)	2024-07-29 13:51:49 -07:00
log_writer.h	Ensure Close() before LinkFile() for WALs in Checkpoint (#12734)	2024-06-12 11:48:45 -07:00
logs_with_prep_tracker.cc	…
logs_with_prep_tracker.h	…
lookup_key.h	…
malloc_stats.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
malloc_stats.h	…
manual_compaction_test.cc	…
memtable.cc	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
memtable.h	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
memtable_list.cc	Add some checks for the file ingestion flow (#13100)	2024-11-05 15:44:56 -08:00
memtable_list.h	Add some checks for the file ingestion flow (#13100)	2024-11-05 15:44:56 -08:00
memtable_list_test.cc	Introduce an interface `ReadOnlyMemTable` for immutable memtables (#13107)	2024-11-04 16:09:34 -08:00
merge_context.h	Add `ContinueCallback` to `GetMergeOperands()` (#12438)	2024-03-15 12:25:49 -07:00
merge_helper.cc	Add initial support for TimedPut API (#12419)	2024-03-14 15:44:55 -07:00
merge_helper.h	…
merge_helper_test.cc	…
merge_operator.cc	…
merge_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
multi_cf_iterator_impl.h	Some small improvements around allow_unprepared_value and multi-CF iterators (#13113)	2024-11-04 18:06:07 -08:00
multi_cf_iterator_test.cc	Support allow_unprepared_value for multi-CF iterators (#13079)	2024-10-20 20:53:08 -07:00
obsolete_files_test.cc	Remove the force mode for EnableFileDeletions API (#12337)	2024-02-13 18:36:25 -08:00
options_file_test.cc	…
output_validator.cc	…
output_validator.h	…
perf_context_test.cc	Add `write_memtable_time` to perf level `kEnableWait` (#12394)	2024-02-29 15:08:26 -08:00
periodic_task_scheduler.cc	Remove extra semi colon from instagram/ranking/mezql/shots/parser/fast/Token.cpp	2024-03-04 06:32:50 -08:00
periodic_task_scheduler.h	…
periodic_task_scheduler_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
pinned_iterators_manager.h	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
plain_table_db_test.cc	Prefer static_cast in place of most reinterpret_cast (#12308)	2024-02-07 10:44:11 -08:00
post_memtable_callback.h	…
pre_release_callback.h	…
prefix_test.cc	…
range_del_aggregator.cc	Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470)	2024-03-22 13:40:42 -07:00
range_del_aggregator.h	Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470)	2024-03-22 13:40:42 -07:00
range_del_aggregator_bench.cc	…
range_del_aggregator_test.cc	…
range_tombstone_fragmenter.cc	…
range_tombstone_fragmenter.h	Remove stale entries from L0 files when UDT is not persisted (#13035)	2024-10-14 12:28:35 -07:00
range_tombstone_fragmenter_test.cc	…
read_callback.h	…
repair.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
repair_test.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
seqno_time_test.cc	Add initial support for TimedPut API (#12419)	2024-03-14 15:44:55 -07:00
seqno_to_time_mapping.cc	Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760)	2024-06-18 10:51:29 -07:00
seqno_to_time_mapping.h	Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760)	2024-06-18 10:51:29 -07:00
snapshot_checker.h	Optimize compaction for standalone range deletion files (#13078)	2024-10-25 09:32:14 -07:00
snapshot_impl.cc	…
snapshot_impl.h	…
table_cache.cc	Fix a leak of open Blob files (#13106)	2024-10-31 15:29:30 -07:00
table_cache.h	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
table_cache_sync_and_async.h	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
table_properties_collector.cc	…
table_properties_collector.h	Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760)	2024-06-18 10:51:29 -07:00
table_properties_collector_test.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
transaction_log_impl.cc	Rename, deprecate `LogFile` and `VectorLogPtr` (#12695)	2024-05-28 09:24:49 -07:00
transaction_log_impl.h	Rename, deprecate `LogFile` and `VectorLogPtr` (#12695)	2024-05-28 09:24:49 -07:00
trim_history_scheduler.cc	…
trim_history_scheduler.h	…
version_builder.cc	Fix a leak of open Blob files (#13106)	2024-10-31 15:29:30 -07:00
version_builder.h	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
version_builder_test.cc	…
version_edit.cc	Support ingesting SST files generated by a live DB (#12750)	2024-07-19 16:14:54 -07:00
version_edit.h	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
version_edit_handler.cc	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
version_edit_handler.h	Add some documentation for version edit handlers (#12948)	2024-08-21 10:09:10 -07:00
version_edit_test.cc	…
version_set.cc	Remove early return when scanning files for temperature change compaction (#13112)	2024-11-05 09:12:39 -08:00
version_set.h	Fix a leak of open Blob files (#13106)	2024-10-31 15:29:30 -07:00
version_set_sync_and_async.h	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
version_set_test.cc	Record newest_key_time as a table property (#13083)	2024-11-01 10:08:35 -07:00
version_util.h	…
wal_edit.cc	…
wal_edit.h	…
wal_edit_test.cc	…
wal_manager.cc	Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831)	2024-07-01 23:29:02 -07:00
wal_manager.h	Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831)	2024-07-01 23:29:02 -07:00
wal_manager_test.cc	…
write_batch.cc	Refactor `table_factory` into MutableCFOptions (#13077)	2024-10-17 14:13:20 -07:00
write_batch_base.cc	…
write_batch_internal.h	Add initial support for TimedPut API (#12419)	2024-03-14 15:44:55 -07:00
write_batch_test.cc	Steps toward deprecating implicit prefix seek, related fixes (#13026)	2024-09-20 15:54:19 -07:00
write_callback.h	…
write_callback_test.cc	Add public API `WriteWithCallback` to support custom callbacks (#12603)	2024-05-31 19:30:19 -07:00
write_controller.cc	…
write_controller.h	…
write_controller_test.cc	…
write_stall_stats.cc	…
write_stall_stats.h	…
write_thread.cc	Add public API `WriteWithCallback` to support custom callbacks (#12603)	2024-05-31 19:30:19 -07:00
write_thread.h	Add public API `WriteWithCallback` to support custom callbacks (#12603)	2024-05-31 19:30:19 -07:00