rocksdb/db
Peter Dillinger 3ee4d5a11a Fix possible crash in failure to sync some WALs (#12789)
Summary:
I believe this was possible with recyclable logs before recent work like https://github.com/facebook/rocksdb/issues/12734, but this cleans up a couple of possible crashes revealed by the crash test.  A WAL with a nullptr file writer (already closed) can persist in `logs_` if a later WAL fails to sync. In case of any WAL sync failures, we don't record WAL syncs to the manifest. Thus, even if a WAL is fully synced and closed, we might need to keep it on the `logs_` list so that we know to record its sync to the manifest if there should be a successful sync next time. (However, I believe that's future-looking because currently any failure in WAL sync is considered non-recoverable.)

I don't believe this was likely enough before recent changes to warrant a release note (if it was possible).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/12789

Test Plan: A unit test that would reveal the crashes, now fixed

Reviewed By: cbi42

Differential Revision: D58874154

Pulled By: pdillinger

fbshipit-source-id: bc69407cd9cbcd080af9585d502d4e33dafc3d29
2024-06-21 12:56:21 -07:00
..
blob Fixed MultiGet() error handling to not skip blob dereference (#12597) 2024-04-29 14:18:42 -07:00
compaction Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
db_impl Fix possible crash in failure to sync some WALs (#12789) 2024-06-21 12:56:21 -07:00
wide Fix the output of ldb dump_wal for PutEntity records (#12677) 2024-05-20 17:04:14 -07:00
arena_wrapped_db_iter.cc
arena_wrapped_db_iter.h
attribute_group_iterator_impl.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
attribute_group_iterator_impl.h MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
builder.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
builder.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
c.cc c.h: Add set_track_and_verify_wals_in_manifest to C API (#12749) 2024-06-10 16:26:52 -07:00
c_test.c c.h: Add set_track_and_verify_wals_in_manifest to C API (#12749) 2024-06-10 16:26:52 -07:00
coalescing_iterator.cc MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
coalescing_iterator.h MultiCfIterator - AttributeGroupIter Impl & CoalescingIter Optimization (#12534) 2024-04-16 08:45:38 -07:00
column_family.cc Change the behavior of manual flush to not retain UDT (#12737) 2024-06-13 13:18:10 -07:00
column_family.h Fix manual flush hanging on waiting for no stall for UDT in memtable … (#12771) 2024-06-14 13:37:37 -07:00
column_family_test.cc Fix manual flush hanging on waiting for no stall for UDT in memtable … (#12771) 2024-06-14 13:37:37 -07:00
compact_files_test.cc Prevent data block compression with BlockBasedTableOptions::block_align (#12592) 2024-04-26 20:05:30 -07:00
comparator_db_test.cc
convenience.cc Inject more errors to more files in stress test (#12713) 2024-06-19 08:42:00 -07:00
convenience_impl.h
corruption_test.cc
cuckoo_table_db_test.cc
db_basic_test.cc DeleteRange() return NotSupported if row_cache is configured (#12512) 2024-04-29 16:33:13 -07:00
db_block_cache_test.cc Fix rare failure in DBBlockCacheTypeTest.Uncache (#12775) 2024-06-14 20:50:36 -07:00
db_bloom_filter_test.cc Set optimize_filters_for_memory by default (#12377) 2024-04-30 08:33:31 -07:00
db_clip_test.cc
db_compaction_filter_test.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
db_compaction_test.cc Fix crash in CompactFiles() of conflict range under preclude_last_level_data_seconds > 0 (#12628) 2024-05-13 13:12:06 -07:00
db_dynamic_level_test.cc
db_encryption_test.cc
db_filesnapshot.cc Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
db_flush_test.cc Refactor SyncWAL and SyncClosedLogs for code sharing (#12707) 2024-05-30 14:53:13 -07:00
db_follower_test.cc Implement obsolete file deletion (GC) in follower (#12657) 2024-05-17 19:13:33 -07:00
db_info_dumper.cc
db_info_dumper.h
db_inplace_update_test.cc
db_io_failure_test.cc Retry DB::Open upon a corruption detected while reading the MANIFEST (#12518) 2024-04-18 17:36:33 -07:00
db_iter.cc Add Iterator property "rocksdb.iterator.is-value-pinned" (#12659) 2024-05-15 19:11:52 -07:00
db_iter.h Follow ups for TimedPut and write time property (#12455) 2024-03-21 10:00:15 -07:00
db_iter_stress_test.cc
db_iter_test.cc
db_iterator_test.cc Add Iterator property "rocksdb.iterator.is-value-pinned" (#12659) 2024-05-15 19:11:52 -07:00
db_kv_checksum_test.cc Fix locking for ColumnFamilyOptions::inplace_update_support (#12624) 2024-05-08 08:30:12 -07:00
db_log_iter_test.cc
db_logical_block_size_cache_test.cc
db_memtable_test.cc fix DeleteRange+memtable_insert_with_hint_prefix_extractor interaction (#12558) 2024-04-22 20:13:58 -07:00
db_merge_operand_test.cc
db_merge_operator_test.cc
db_options_test.cc
db_properties_test.cc GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
db_range_del_test.cc Fail DeleteRange() early when row_cache is configured (#12710) 2024-05-29 15:03:15 -07:00
db_rate_limiter_test.cc
db_readonly_with_timestamp_test.cc
db_secondary_test.cc
db_sst_test.cc Fix delete obsolete files on recovery not rate limited (#12590) 2024-05-01 12:26:54 -07:00
db_statistics_test.cc
db_table_properties_test.cc
db_tailing_iter_test.cc
db_test.cc Disable "uncache" behavior in DB shutdown (#12751) 2024-06-11 15:57:40 -07:00
db_test2.cc Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
db_test_util.cc Add TimedPut to stress test (#12559) 2024-04-30 15:40:35 -07:00
db_test_util.h Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
db_universal_compaction_test.cc
db_wal_test.cc Fix possible crash in failure to sync some WALs (#12789) 2024-06-21 12:56:21 -07:00
db_with_timestamp_basic_test.cc
db_with_timestamp_compaction_test.cc
db_with_timestamp_test_util.cc
db_with_timestamp_test_util.h
db_write_buffer_manager_test.cc
db_write_test.cc Disallow memtable flush and sst ingest while WAL is locked (#12652) 2024-05-21 10:17:34 -07:00
dbformat.cc Add EntryType for TimedPut (#12669) 2024-05-16 15:18:12 -07:00
dbformat.h Support read timestamp in ldb (#12641) 2024-05-13 15:43:12 -07:00
dbformat_test.cc
deletefile_test.cc Add an option to wait for purge in WaitForCompact (#12520) 2024-04-17 17:33:27 -07:00
error_handler.cc
error_handler.h
error_handler_fs_test.cc Refactor SyncWAL and SyncClosedLogs for code sharing (#12707) 2024-05-30 14:53:13 -07:00
event_helpers.cc
event_helpers.h
experimental.cc
external_sst_file_basic_test.cc Disallow memtable flush and sst ingest while WAL is locked (#12652) 2024-05-21 10:17:34 -07:00
external_sst_file_ingestion_job.cc Use extended file boundary for key range overlap check during file ingestion (#12735) 2024-06-04 13:39:51 -07:00
external_sst_file_ingestion_job.h Use extended file boundary for key range overlap check during file ingestion (#12735) 2024-06-04 13:39:51 -07:00
external_sst_file_test.cc Use extended file boundary for key range overlap check during file ingestion (#12735) 2024-06-04 13:39:51 -07:00
fault_injection_test.cc Inject more errors to more files in stress test (#12713) 2024-06-19 08:42:00 -07:00
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
flush_job.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
flush_job_test.cc Follow ups for TimedPut and write time property (#12455) 2024-03-21 10:00:15 -07:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc
forward_iterator.h
forward_iterator_bench.cc
history_trimming_iterator.h
import_column_family_job.cc Fix a corruption bug in CreateColumnFamilyWithImport() (#12602) 2024-05-06 11:01:38 -07:00
import_column_family_job.h
import_column_family_test.cc Fix a corruption bug in CreateColumnFamilyWithImport() (#12602) 2024-05-06 11:01:38 -07:00
internal_stats.cc GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
internal_stats.h GetAggregatedIntProperty accumulates property once per block cache (#12755) 2024-06-18 10:46:55 -07:00
job_context.h
kv_checksum.h
listener_test.cc
log_format.h
log_reader.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_reader.h Enable recycle_log_file_num option for point in time recovery (#12403) 2024-03-21 12:29:35 -07:00
log_test.cc Fix recycled WAL detection when wal_compression is enabled (#12643) 2024-05-22 15:34:37 -07:00
log_writer.cc Inject more errors to more files in stress test (#12713) 2024-06-19 08:42:00 -07:00
log_writer.h Ensure Close() before LinkFile() for WALs in Checkpoint (#12734) 2024-06-12 11:48:45 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
lookup_key.h
malloc_stats.cc
malloc_stats.h
manual_compaction_test.cc
memtable.cc Use std::optional instead of std::unique_ptr to conditionally create a read lock. (#12704) 2024-05-28 15:31:45 -07:00
memtable.h Fix max_successive_merges counting CPU overhead regression (#12546) 2024-04-17 12:11:24 -07:00
memtable_list.cc
memtable_list.h
memtable_list_test.cc
merge_context.h
merge_helper.cc
merge_helper.h
merge_helper_test.cc
merge_operator.cc
merge_test.cc
multi_cf_iterator_impl.h Fix heap-use-after-free in MultiCfIteratorImpl (#12784) 2024-06-21 11:56:10 -07:00
multi_cf_iterator_test.cc Fix IteratorsConsistentView tests (#12582) 2024-04-25 14:06:46 -07:00
obsolete_files_test.cc
options_file_test.cc
output_validator.cc
output_validator.h
perf_context_test.cc
periodic_task_scheduler.cc
periodic_task_scheduler.h
periodic_task_scheduler_test.cc
pinned_iterators_manager.h
plain_table_db_test.cc
post_memtable_callback.h
pre_release_callback.h
prefix_test.cc
range_del_aggregator.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator.h Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
range_del_aggregator_bench.cc
range_del_aggregator_test.cc
range_tombstone_fragmenter.cc
range_tombstone_fragmenter.h Fix compile errors in C++23 (#12106) 2024-05-28 15:33:57 -07:00
range_tombstone_fragmenter_test.cc
read_callback.h
repair.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
repair_test.cc
seqno_time_test.cc
seqno_to_time_mapping.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
seqno_to_time_mapping.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
snapshot_checker.h
snapshot_impl.cc
snapshot_impl.h
table_cache.cc Support pro-actively erasing obsolete block cache entries (#12694) 2024-06-07 08:57:11 -07:00
table_cache.h Support pro-actively erasing obsolete block cache entries (#12694) 2024-06-07 08:57:11 -07:00
table_cache_sync_and_async.h Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
table_properties_collector.cc
table_properties_collector.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
table_properties_collector_test.cc Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
transaction_log_impl.cc Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
transaction_log_impl.h Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc
version_builder.h
version_builder_test.cc
version_edit.cc
version_edit.h
version_edit_handler.cc Implement obsolete file deletion (GC) in follower (#12657) 2024-05-17 19:13:33 -07:00
version_edit_handler.h Fix version edit dump in json (#12703) 2024-05-28 16:44:25 -07:00
version_edit_test.cc
version_set.cc Disable "uncache" behavior in DB shutdown (#12751) 2024-06-11 15:57:40 -07:00
version_set.h Support pro-actively erasing obsolete block cache entries (#12694) 2024-06-07 08:57:11 -07:00
version_set_sync_and_async.h
version_set_test.cc Implement obsolete file deletion (GC) in follower (#12657) 2024-05-17 19:13:33 -07:00
version_util.h
wal_edit.cc
wal_edit.h
wal_edit_test.cc
wal_manager.cc Inject more errors to more files in stress test (#12713) 2024-06-19 08:42:00 -07:00
wal_manager.h Rename, deprecate LogFile and VectorLogPtr (#12695) 2024-05-28 09:24:49 -07:00
wal_manager_test.cc
write_batch.cc Fail DeleteRange() early when row_cache is configured (#12710) 2024-05-29 15:03:15 -07:00
write_batch_base.cc
write_batch_internal.h
write_batch_test.cc Replace ScopedArenaIterator with ScopedArenaPtr<InternalIterator> (#12470) 2024-03-22 13:40:42 -07:00
write_callback.h
write_callback_test.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_controller.cc
write_controller.h
write_controller_test.cc
write_stall_stats.cc
write_stall_stats.h
write_thread.cc Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00
write_thread.h Add public API WriteWithCallback to support custom callbacks (#12603) 2024-05-31 19:30:19 -07:00