dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-30 13:41:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Peter Dillinger 2a67d475f1 Fix bug affecting GetSortedWalFiles, Backups, Checkpoint (#9208) 
Summary:
Saw error like this:
`Backup failed -- IO error: No such file or directory: While opening a
file for sequentially reading:
/dev/shm/rocksdb/rocksdb_crashtest_blackbox/004426.log: No such file or
directory`

Unfortunately, GetSortedWalFiles (used by Backups, Checkpoint, etc.)
relies on no file deletions happening while its operating, which
means not only disabling (more) deletions, but ensuring any pending
deletions are completed. Two fixes related to this:

* There was a gap in several places between decrementing
pending_purge_obsolete_files_ and incrementing bg_purge_scheduled_ where
the db mutex would be released and GetSortedWalFiles (and others) could
get false information that no deletions are pending.

* The fix to https://github.com/facebook/rocksdb/issues/8591 (disabling deletions in GetSortedWalFiles) seems
incomplete because it doesn't prevent pending deletions from occuring
during the operation (if deletions not already disabled, the case that
was to be fixed by the change).

Pull Request resolved: https://github.com/facebook/rocksdb/pull/9208

Test Plan:
existing tests (it's hard to write a test for interleavings
that are now excluded - this is what stress test is for)

Reviewed By: ajkr

Differential Revision: D32630675

Pulled By: pdillinger

fbshipit-source-id: a121e3da648de130cd24d44c524232f4eb22f178
2021-11-24 14:52:00 -08:00
..
blob Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
compaction Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
db_impl Fix bug affecting GetSortedWalFiles, Backups, Checkpoint (#9208) 2021-11-24 14:52:00 -08:00
arena_wrapped_db_iter.cc Rename ImmutableOptions variables (#8409) 2021-06-16 16:51:38 -07:00
arena_wrapped_db_iter.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
builder.cc Don't call OnTableFileCreated with OK for empty+deleted file (#9118) 2021-11-03 08:43:27 -07:00
builder.h Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
c.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
c_test.c Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
column_family.cc Fix bug affecting GetSortedWalFiles, Backups, Checkpoint (#9208) 2021-11-24 14:52:00 -08:00
column_family.h Add DB properties for BlobDB (#8734) 2021-09-08 12:22:04 -07:00
column_family_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
compact_files_test.cc Compaction should not move data to up level (#8116) 2021-03-29 17:10:42 -07:00
comparator_db_test.cc
convenience.cc Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
corruption_test.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
cuckoo_table_db_test.cc Experimental support for SST unique IDs (#8990) 2021-10-18 23:32:01 -07:00
db_basic_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_block_cache_test.cc Check for and disallow shared key space in block caches (#9172) 2021-11-16 11:16:05 -08:00
db_bloom_filter_test.cc Account Bloom/Ribbon filter construction memory in global memory limit (#9073) 2021-11-18 09:42:20 -08:00
db_compaction_filter_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_compaction_test.cc Adhere to per-DB concurrency limit when bottom-pri compactions exist (#9179) 2021-11-18 17:31:50 -08:00
db_dynamic_level_test.cc Make SystemClock into a Customizable Class (#8636) 2021-09-21 09:23:48 -07:00
db_encryption_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_filesnapshot.cc Fix bug affecting GetSortedWalFiles, Backups, Checkpoint (#9208) 2021-11-24 14:52:00 -08:00
db_flush_test.cc Fix a bug in FlushJob picking more memtables beyond synced WALs (#9142) 2021-11-19 09:56:00 -08:00
db_info_dumper.cc Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
db_info_dumper.h
db_inplace_update_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_io_failure_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_iter.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
db_iter.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
db_iter_stress_test.cc Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
db_iter_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
db_iterator_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_kv_checksum_test.cc Allow WriteBatch to have keys with different timestamp sizes (#8725) 2021-09-12 15:34:26 -07:00
db_log_iter_test.cc Attempt to deflake DBTestXactLogIterator.TransactionLogIteratorCorruptedLog (#8627) 2021-08-10 11:10:07 -07:00
db_logical_block_size_cache_test.cc Add further tests to ASSERT_STATUS_CHECKED (2) (#7698) 2020-12-09 21:21:16 -08:00
db_memtable_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_merge_operand_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_merge_operator_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_options_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_properties_test.cc Add support for building on s390x platform (#8962) 2021-10-22 10:13:15 -07:00
db_range_del_test.cc Incremental Space Amp Compactions in Universal Style (#8655) 2021-10-20 10:04:13 -07:00
db_secondary_test.cc Fix an assertion failure when ManifestTailer switches to new Manifest in multi-cf mode (#9143) 2021-11-19 19:53:40 -08:00
db_sst_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_statistics_test.cc Bytes read stat for VerifyChecksum() and VerifyFileChecksums() APIs (#8741) 2021-09-07 13:28:29 -07:00
db_table_properties_test.cc Fix backward compatibility with 2.5 through 2.7 (#9189) 2021-11-19 17:31:01 -08:00
db_tailing_iter_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_test2.cc Fix flaky DBTest2.RateLimitedCompactionReads (#9185) 2021-11-19 10:08:59 -08:00
db_test_util.cc Implement XXH3 block checksum type (#9069) 2021-10-28 22:15:17 -07:00
db_test_util.h Implement XXH3 block checksum type (#9069) 2021-10-28 22:15:17 -07:00
db_universal_compaction_test.cc Adhere to per-DB concurrency limit when bottom-pri compactions exist (#9179) 2021-11-18 17:31:50 -08:00
db_wal_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
db_with_timestamp_basic_test.cc Track each SST's timestamp information as user properties (#9093) 2021-11-19 11:37:06 -08:00
db_with_timestamp_compaction_test.cc Make MemTableRepFactory into a Customizable class (#8419) 2021-09-08 07:46:44 -07:00
db_write_buffer_manager_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_write_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
dbformat.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
dbformat.h Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
dbformat_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
deletefile_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
error_handler.cc cleanup error_handler related code (#9098) 2021-11-08 15:49:17 -08:00
error_handler.h cleanup error_handler related code (#9098) 2021-11-08 15:49:17 -08:00
error_handler_fs_test.cc cleanup error_handler related code (#9098) 2021-11-08 15:49:17 -08:00
event_helpers.cc Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
event_helpers.h Expose blob file information through the EventListener interface (#8675) 2021-09-16 17:23:36 -07:00
experimental.cc
external_sst_file_basic_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
external_sst_file_ingestion_job.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
external_sst_file_ingestion_job.h Fix sequence number bump logic in multi-CF SST ingestion (#9005) 2021-10-12 20:39:52 -07:00
external_sst_file_test.cc Implement XXH3 block checksum type (#9069) 2021-10-28 22:15:17 -07:00
fault_injection_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
flush_job.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
flush_job_test.cc Fix NotifyOnFlushCompleted() for atomic flush (#8585) 2021-08-03 13:31:10 -07:00
flush_scheduler.cc
flush_scheduler.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
forward_iterator.cc Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
forward_iterator.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
forward_iterator_bench.cc
import_column_family_job.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
import_column_family_job.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
import_column_family_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
internal_stats.cc InternalStats::DumpCFMapStat: fix sum.w_amp (#9065) 2021-10-31 23:11:43 -07:00
internal_stats.h Support GetMapProperty() with "rocksdb.dbstats" (#9057) 2021-10-20 13:17:00 -07:00
job_context.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
kv_checksum.h fix compile errors in db/kv_checksum.h (#9173) 2021-11-16 10:20:50 -08:00
listener_test.cc Don't call OnTableFileCreated with OK for empty+deleted file (#9118) 2021-11-03 08:43:27 -07:00
log_format.h
log_reader.cc Fix kPointInTimeRecovery handling of truncated WAL (#7701) 2020-11-30 18:11:38 -08:00
log_reader.h
log_test.cc Make StringEnv, StringSink, StringSource use FS classes (#7786) 2021-01-04 16:01:01 -08:00
log_writer.cc Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412) 2021-06-25 00:47:17 -07:00
log_writer.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h Include C++ standard library headers instead of C compatibility headers (#8068) 2021-03-19 12:09:47 -07:00
lookup_key.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
malloc_stats.cc Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
malloc_stats.h
manual_compaction_test.cc Add more tests for assert status checked (#7524) 2020-12-22 23:45:58 -08:00
memtable.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
memtable.h Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
memtable_list.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
memtable_list.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
memtable_list_test.cc Fix NotifyOnFlushCompleted() for atomic flush (#8585) 2021-08-03 13:31:10 -07:00
merge_context.h Add Merge Operator support to WriteBatchWithIndex (#8135) 2021-05-10 12:50:25 -07:00
merge_helper.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_helper.h Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_helper_test.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_operator.cc
merge_test.cc Built-in support for generating unique IDs, bug fix (#8708) 2021-08-30 15:20:41 -07:00
obsolete_files_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
options_file_test.cc No elide constructors (#7798) 2020-12-23 16:55:53 -08:00
output_validator.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
output_validator.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
perf_context_test.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
periodic_work_scheduler.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
periodic_work_scheduler.h Add a SystemClock class to capture the time functions of an Env (#7858) 2021-01-25 22:09:11 -08:00
periodic_work_scheduler_test.cc Fix a minor issue with initializing the test path (#8555) 2021-07-23 08:38:45 -07:00
pinned_iterators_manager.h Replace most typedef with using= (#8751) 2021-09-07 11:31:59 -07:00
plain_table_db_test.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
pre_release_callback.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
prefix_test.cc Use SystemClock* instead of std::shared_ptr<SystemClock> in lower level routines (#8033) 2021-03-15 04:34:11 -07:00
range_del_aggregator.cc
range_del_aggregator.h Fix some typos in comments (#8066) 2021-03-25 21:18:08 -07:00
range_del_aggregator_bench.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
range_del_aggregator_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
range_tombstone_fragmenter.cc Added memtable garbage statistics (#8411) 2021-06-18 04:57:27 -07:00
range_tombstone_fragmenter.h Added memtable garbage statistics (#8411) 2021-06-18 04:57:27 -07:00
range_tombstone_fragmenter_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
read_callback.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
repair.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
repair_test.cc Some fixes and enhancements to ldb repair (#8544) 2021-07-28 16:44:14 -07:00
snapshot_checker.h
snapshot_impl.cc
snapshot_impl.h Fix and detect headers with missing dependencies (#8893) 2021-09-10 10:00:26 -07:00
table_cache.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
table_cache.h Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
table_properties_collector.cc Apply sample_for_compression to all block-based tables (#8105) 2021-03-25 15:00:45 -07:00
table_properties_collector.h Track each SST's timestamp information as user properties (#9093) 2021-11-19 11:37:06 -08:00
table_properties_collector_test.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
transaction_log_impl.cc Add file operation callbacks to SequentialFileReader (#8982) 2021-10-05 10:51:59 -07:00
transaction_log_impl.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Refactor and unify blob file saving and the logic that finds the oldest live blob file (#9122) 2021-11-05 16:50:52 -07:00
version_builder.h Refactor the detailed consistency checks and the SST saving logic in VersionBuilder (#9099) 2021-11-03 11:52:47 -07:00
version_builder_test.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
version_edit.cc Print file checksum in hex (#9196) 2021-11-22 09:30:47 -08:00
version_edit.h Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
version_edit_handler.cc Fix an assertion failure when ManifestTailer switches to new Manifest in multi-cf mode (#9143) 2021-11-19 19:53:40 -08:00
version_edit_handler.h Fixed manifest_dump issues when printing keys and values containing null characters (#8378) 2021-06-10 12:55:20 -07:00
version_edit_test.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
version_set.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
version_set.h Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
version_set_test.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
wal_edit.cc Always track WAL obsoletion (#7759) 2020-12-09 16:02:12 -08:00
wal_edit.h Always track WAL obsoletion (#7759) 2020-12-09 16:02:12 -08:00
wal_edit_test.cc Always track WAL obsoletion (#7759) 2020-12-09 16:02:12 -08:00
wal_manager.cc Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
wal_manager.h Allow WAL dir to change with db dir (#8582) 2021-07-30 12:16:44 -07:00
wal_manager_test.cc Make SystemClock into a Customizable Class (#8636) 2021-09-21 09:23:48 -07:00
write_batch.cc Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
write_batch_base.cc
write_batch_internal.h Adapt key-value checksum for timestamp-suffixed keys (#8914) 2021-09-14 13:14:39 -07:00
write_batch_test.cc Remove some unneeded code (#8736) 2021-09-01 14:28:58 -07:00
write_callback.h
write_callback_test.cc Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475) 2021-07-07 11:14:05 -07:00
write_controller.cc Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_controller.h Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_controller_test.cc Revamp WriteController (#8064) 2021-03-18 09:47:31 -07:00
write_thread.cc Stall writes in WriteBufferManager when memory_usage exceeds buffer_size (#7898) 2021-04-21 13:54:02 -07:00
write_thread.h typo: fix typo in db/write_thread's state (#8423) 2021-06-18 17:14:51 -07:00