Mike Kolupaev eef63ef807 Fixed CompactFiles() spuriously failing or corrupting DB ... Summary: We started getting two kinds of crashes since we started using `DB::CompactFiles()`: (1) `CompactFiles()` fails saying something like "/data/logdevice/4440/shard12/012302.sst: No such file or directory", and presumably makes DB read-only, (2) DB fails to open saying "Corruption: Can't access /267000.sst: IO error: /data/logdevice/4440/shard1/267000.sst: No such file or directory". AFAICT, both can be explained by background thread deleting compaction output as "obsolete" while it's being written, before it's committed to manifest. If it ends up committed to the manifest, we get (2); if compaction notices the disappearance and fails, we get (1). The internal tasks t10068021 and t10134177 have some details about the investigation that led to this. Test Plan: `make -j check`; the new test fails to reopen the DB without the fix Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba, sdong Differential Revision: https://reviews.facebook.net/D54561		2016-02-22 13:54:58 -08:00
..
auto_roll_logger.cc	Fix race conditions in auto-rolling logger	2016-02-17 12:06:45 -08:00
auto_roll_logger.h	This addresses build issues on Windows	2016-02-19 12:29:54 -08:00
auto_roll_logger_test.cc	Use condition variable in log roller test	2016-02-18 18:03:53 -08:00
builder.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
builder.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
c.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
c_test.c	Deprecate CompactionFilterV2	2015-07-17 18:59:11 +02:00
column_family.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
column_family.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
column_family_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compact_files_test.cc	Fixed CompactFiles() spuriously failing or corrupting DB	2016-02-22 13:54:58 -08:00
compacted_db_impl.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compacted_db_impl.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_iterator.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_iterator.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_iterator_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_job.cc	Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage	2016-02-17 15:20:23 -08:00
compaction_job.h	Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage	2016-02-17 15:20:23 -08:00
compaction_job_stats_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_job_test.cc	Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage	2016-02-17 15:20:23 -08:00
compaction_picker.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_picker.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
compaction_picker_test.cc	Add a new compaction priority that picks file whose overlapping ratio is smallest	2016-02-11 15:59:19 -08:00
comparator_db_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
convenience.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
corruption_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
cuckoo_table_db_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_compaction_filter_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_compaction_test.cc	Add a new compaction priority that picks file whose overlapping ratio is smallest	2016-02-11 15:59:19 -08:00
db_dynamic_level_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_filesnapshot.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_impl.cc	Fixed CompactFiles() spuriously failing or corrupting DB	2016-02-22 13:54:58 -08:00
db_impl.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_impl_debug.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_impl_experimental.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_impl_readonly.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_impl_readonly.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_info_dumper.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_info_dumper.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_inplace_update_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_iter.cc	Use user_comparator when comparing against iterate_upper_bound.	2016-02-11 08:47:16 -05:00
db_iter.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_iter_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_log_iter_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_properties_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_table_properties_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_tailing_iter_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_test.cc	Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage	2016-02-17 15:20:23 -08:00
db_test_util.cc	Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage	2016-02-17 15:20:23 -08:00
db_test_util.h	Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage	2016-02-17 15:20:23 -08:00
db_universal_compaction_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
db_wal_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
dbformat.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
dbformat.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
dbformat_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
deletefile_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
event_helpers.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
event_helpers.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
experimental.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
fault_injection_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
file_indexer.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
file_indexer.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
file_indexer_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
filename.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
filename.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
filename_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
flush_job.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
flush_job.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
flush_job_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
flush_scheduler.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
flush_scheduler.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
forward_iterator.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
forward_iterator.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
forward_iterator_bench.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
inlineskiplist.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
inlineskiplist_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
internal_stats.cc	Eliminate duplicated property constants	2016-02-02 19:14:56 -08:00
internal_stats.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
job_context.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
listener_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
log_format.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
log_reader.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
log_reader.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
log_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
log_writer.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
log_writer.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
managed_iterator.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
managed_iterator.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
manual_compaction_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable_allocator.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable_allocator.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable_list.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable_list.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtable_list_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
memtablerep_bench.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
merge_context.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
merge_helper.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
merge_helper.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
merge_helper_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
merge_operator.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
merge_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
options_file_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
perf_context_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
plain_table_db_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
prefix_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
repair.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
skiplist.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
skiplist_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
slice.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
snapshot_impl.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
snapshot_impl.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
table_cache.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
table_cache.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
table_properties_collector.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
table_properties_collector.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
table_properties_collector_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
transaction_log_impl.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
transaction_log_impl.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_builder.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_builder.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_builder_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_edit.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_edit.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_edit_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_set.cc	Add a new compaction priority that picks file whose overlapping ratio is smallest	2016-02-11 15:59:19 -08:00
version_set.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
version_set_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
wal_manager.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
wal_manager.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
wal_manager_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_batch.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_batch_base.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_batch_internal.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_batch_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_callback.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_callback_test.cc	Improve write_with_callback_test to sync WAL	2016-02-16 14:04:14 -08:00
write_controller.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_controller.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_controller_test.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_thread.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
write_thread.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
writebuffer.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
xfunc_test_points.cc	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00
xfunc_test_points.h	Updated all copyright headers to the new format.	2016-02-09 15:12:00 -08:00