Yi Wu b1ad6ebba8 WritePrepared: fix two versions in compaction see different status for released snapshots (#4890) ... Summary: Fix how CompactionIterator::findEarliestVisibleSnapshots handles released snapshot. It fixing the two scenarios: Scenario 1: key1 has two values v1 and v2. There're two snapshots s1 and s2 taken after v1 and v2 are committed. Right after compaction output v2, s1 is released. Now findEarliestVisibleSnapshot may see s1 being released, and return the next snapshot, which is s2. That's larger than v2's earliest visible snapshot, which was s1. The fix: the only place we check against last snapshot and current key snapshot is when we decide whether to compact out a value if it is hidden by a later value. In the check if we see current snapshot is even larger than last snapshot, we know last snapshot is released, and we are safe to compact out current key. Scenario 2: key1 has two values v1 and v2. there are two snapshots s1 and s2 taken after v1 and v2 are committed. During compaction before we process the key, s1 is released. When compaction process v2, snapshot checker may return kSnapshotReleased, and the earliest visible snapshot for v2 become s2. When compaction process v1, snapshot checker may return kIsInSnapshot (for WritePrepared transaction, it could be because v1 is still in commit cache). The result will become inconsistent here. The fix: remember the set of released snapshots ever reported by snapshot checker, and ignore them when finding result for findEarliestVisibleSnapshot. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4890 Differential Revision: D13705538 Pulled By: maysamyabandeh fbshipit-source-id: e577f0d9ee1ff5a6035f26859e56902ecc85a5a4		2019-01-18 17:24:06 -08:00
..
builder.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
builder.h	Add compaction logic to RangeDelAggregatorV2 (#4758)	2018-12-17 13:20:51 -08:00
c.cc	Support setting options on column families via C bindings (#4785)	2018-12-17 13:52:12 -08:00
c_test.c	Get CompactionJobInfo from CompactFiles	2018-12-13 14:21:24 -08:00
column_family.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
column_family.h	Lock free MultiGet (#4754)	2019-01-02 11:42:54 -08:00
column_family_test.cc	Fix spelling errors (#4827)	2019-01-02 11:17:57 -08:00
compact_files_test.cc	Get CompactionJobInfo from CompactFiles	2018-12-13 14:21:24 -08:00
compacted_db_impl.cc
compacted_db_impl.h	Enable checkpoint of read-only db (#4681)	2018-12-07 17:06:02 -08:00
compaction.cc
compaction.h
compaction_iteration_stats.h
compaction_iterator.cc	WritePrepared: fix two versions in compaction see different status for released snapshots (#4890)	2019-01-18 17:24:06 -08:00
compaction_iterator.h	WritePrepared: fix two versions in compaction see different status for released snapshots (#4890)	2019-01-18 17:24:06 -08:00
compaction_iterator_test.cc	WritePrepared: fix issue with snapshot released during compaction (#4858)	2019-01-16 09:55:32 -08:00
compaction_job.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
compaction_job.h	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
compaction_job_stats_test.cc
compaction_job_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
compaction_picker.cc	Move FIFOCompactionPicker to a separate file (#4724)	2018-11-29 16:04:52 -08:00
compaction_picker.h	Move FIFOCompactionPicker to a separate file (#4724)	2018-11-29 16:04:52 -08:00
compaction_picker_fifo.cc	Move FIFOCompactionPicker to a separate file (#4724)	2018-11-29 16:04:52 -08:00
compaction_picker_fifo.h	Move FIFOCompactionPicker to a separate file (#4724)	2018-11-29 16:04:52 -08:00
compaction_picker_test.cc	Move FIFOCompactionPicker to a separate file (#4724)	2018-11-29 16:04:52 -08:00
compaction_picker_universal.cc
compaction_picker_universal.h
comparator_db_test.cc
convenience.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
corruption_test.cc	fix DeleteRange memory leak for mmap and block cache (#4810)	2018-12-20 21:59:49 -08:00
cuckoo_table_db_test.cc
db_basic_test.cc	Lock free MultiGet (#4754)	2019-01-02 11:42:54 -08:00
db_blob_index_test.cc
db_block_cache_test.cc	Fix spelling errors (#4827)	2019-01-02 11:17:57 -08:00
db_bloom_filter_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
db_compaction_filter_test.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
db_compaction_test.cc	Preload some files even if options.max_open_files (#3340)	2018-12-28 18:02:28 -08:00
db_dynamic_level_test.cc	Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668)	2018-11-12 16:42:16 -08:00
db_encryption_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
db_filesnapshot.cc	Remove redundant member var and set options (#4631)	2018-11-12 12:24:26 -08:00
db_flush_test.cc	Improve flushing multiple column families (#4708)	2018-12-13 15:12:40 -08:00
db_impl.cc	WritePrepared: Fix visible key compacted out by compaction (#4883)	2019-01-15 21:34:38 -08:00
db_impl.h	WritePrepared: Fix visible key compacted out by compaction (#4883)	2019-01-15 21:34:38 -08:00
db_impl_compaction_flush.cc	WritePrepared: Fix visible key compacted out by compaction (#4883)	2019-01-15 21:34:38 -08:00
db_impl_debug.cc	Enable atomic flush (#4023)	2018-10-26 15:08:43 -07:00
db_impl_experimental.cc
db_impl_files.cc	Fix #3840: only SyncClosedLogs for multiple CFs (#4460)	2018-11-13 11:32:16 -08:00
db_impl_open.cc	Improve Error Message When wal_dir doesn't exist (#4874)	2019-01-15 16:46:04 -08:00
db_impl_readonly.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
db_impl_readonly.h	Get CompactionJobInfo from CompactFiles	2018-12-13 14:21:24 -08:00
db_impl_write.cc	Fix skip WAL for whole write_group when leader's callback fail (#4838)	2019-01-03 12:40:42 -08:00
db_info_dumper.cc
db_info_dumper.h
db_inplace_update_test.cc
db_io_failure_test.cc	Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596)	2018-10-29 14:36:31 -07:00
db_iter.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
db_iter.h	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
db_iter_stress_test.cc
db_iter_test.cc
db_iterator_test.cc	Fix ticker stat for number files closed (#4703)	2018-11-21 18:31:34 -08:00
db_log_iter_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
db_memtable_test.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
db_merge_operator_test.cc	WritePrepared: fix issue with snapshot released during compaction (#4858)	2019-01-16 09:55:32 -08:00
db_options_test.cc
db_properties_test.cc	fix accounting for range tombstones in TableProperties (#4841)	2019-01-02 15:08:53 -08:00
db_range_del_test.cc	Fix point lookup on range tombstone sentinel endpoint (#4829)	2019-01-04 11:24:08 -08:00
db_sst_test.cc	Try to fix DBSSTTest.RateLimitedDelete flakiness (#4840)	2019-01-03 11:05:19 -08:00
db_statistics_test.cc
db_table_properties_test.cc
db_tailing_iter_test.cc
db_test.cc	Get CompactionJobInfo from CompactFiles	2018-12-13 14:21:24 -08:00
db_test2.cc	Remove duplicates from SnapshotList::GetAll (#4860)	2019-01-09 16:25:42 -08:00
db_test_util.cc	Lock free MultiGet (#4754)	2019-01-02 11:42:54 -08:00
db_test_util.h	Lock free MultiGet (#4754)	2019-01-02 11:42:54 -08:00
db_universal_compaction_test.cc
db_wal_test.cc	Preload some files even if options.max_open_files (#3340)	2018-12-28 18:02:28 -08:00
db_write_test.cc
dbformat.cc
dbformat.h
dbformat_test.cc
deletefile_test.cc
error_handler.cc
error_handler.h
error_handler_test.cc
event_helpers.cc
event_helpers.h
experimental.cc
external_sst_file_basic_test.cc	Change directory where ExternalSSTFileBasicTest runs (#4766)	2018-12-11 10:21:37 -08:00
external_sst_file_ingestion_job.cc
external_sst_file_ingestion_job.h
external_sst_file_test.cc	Try to fix ExternalSSTFileTest.IngestNonExistingFile flakines (#4625)	2018-11-02 17:26:35 -07:00
fault_injection_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc	Add compaction logic to RangeDelAggregatorV2 (#4758)	2018-12-17 13:20:51 -08:00
flush_job.h
flush_job_test.cc	Remove duplicates from SnapshotList::GetAll (#4860)	2019-01-09 16:25:42 -08:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
forward_iterator.h
forward_iterator_bench.cc
internal_stats.cc	Add DB property for SST files kept from deletion (#4618)	2018-11-05 20:24:40 -08:00
internal_stats.h	Add DB property for SST files kept from deletion (#4618)	2018-11-05 20:24:40 -08:00
job_context.h	WritePrepared: Fix visible key compacted out by compaction (#4883)	2019-01-15 21:34:38 -08:00
listener_test.cc	Use chrono::time_point instead of time_t (#4868)	2019-01-16 09:51:05 -08:00
log_format.h
log_reader.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
log_reader.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
log_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
log_writer.cc
log_writer.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
malloc_stats.cc	Detect if Jemalloc is linked with the binary (#4844)	2019-01-03 16:30:12 -08:00
malloc_stats.h
manual_compaction_test.cc
memtable.cc	Change is_range_del_table_empty_ flag to atomic (#4801)	2018-12-19 17:21:14 -08:00
memtable.h	Refactor atomic flush result installation to MANIFEST (#4791)	2019-01-03 20:56:24 -08:00
memtable_list.cc	Refactor atomic flush result installation to MANIFEST (#4791)	2019-01-03 20:56:24 -08:00
memtable_list.h	Refactor atomic flush result installation to MANIFEST (#4791)	2019-01-03 20:56:24 -08:00
memtable_list_test.cc	Refactor atomic flush result installation to MANIFEST (#4791)	2019-01-03 20:56:24 -08:00
merge_context.h	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
merge_helper.cc	WritePrepared: fix issue with snapshot released during compaction (#4858)	2019-01-16 09:55:32 -08:00
merge_helper.h	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
merge_helper_test.cc
merge_operator.cc
merge_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
obsolete_files_test.cc
options_file_test.cc
perf_context_test.cc	Add a new per level counter for block cache hit (#4796)	2018-12-21 13:20:05 -08:00
pinned_iterators_manager.h
plain_table_db_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
pre_release_callback.h
prefix_test.cc
range_del_aggregator.cc	Remove stale TODO (#4800)	2018-12-19 15:45:37 -08:00
range_del_aggregator.h	Fix unused member compile error	2018-12-18 14:28:42 -08:00
range_del_aggregator_bench.cc	Fix Windows broken build error due to non-const override (#4798)	2018-12-19 13:29:51 -08:00
range_del_aggregator_test.cc	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
range_tombstone_fragmenter.cc	Add compaction logic to RangeDelAggregatorV2 (#4758)	2018-12-17 13:20:51 -08:00
range_tombstone_fragmenter.h	Add compaction logic to RangeDelAggregatorV2 (#4758)	2018-12-17 13:20:51 -08:00
range_tombstone_fragmenter_test.cc	Prepare FragmentedRangeTombstoneIterator for use in compaction (#4740)	2018-12-11 12:10:48 -08:00
read_callback.h
repair.cc	Add compaction logic to RangeDelAggregatorV2 (#4758)	2018-12-17 13:20:51 -08:00
repair_test.cc
snapshot_checker.h	WritePrepared: fix issue with snapshot released during compaction (#4858)	2019-01-16 09:55:32 -08:00
snapshot_impl.cc
snapshot_impl.h	Remove duplicates from SnapshotList::GetAll (#4860)	2019-01-09 16:25:42 -08:00
table_cache.cc	Introduce a CPU time counter in perf_context (#4741)	2018-12-20 12:03:44 -08:00
table_cache.h	Preload some files even if options.max_open_files (#3340)	2018-12-28 18:02:28 -08:00
table_properties_collector.cc	Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594)	2018-10-30 15:34:27 -07:00
table_properties_collector.h	Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594)	2018-10-30 15:34:27 -07:00
table_properties_collector_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
transaction_log_impl.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
transaction_log_impl.h	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
version_builder.cc	Preload some files even if options.max_open_files (#3340)	2018-12-28 18:02:28 -08:00
version_builder.h	Preload some files even if options.max_open_files (#3340)	2018-12-28 18:02:28 -08:00
version_builder_test.cc
version_edit.cc	Fix spelling errors (#4827)	2019-01-02 11:17:57 -08:00
version_edit.h
version_edit_test.cc
version_set.cc	Non-initial file preloading should always prefetch index and filter (#4852)	2019-01-08 12:47:34 -08:00
version_set.h	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
version_set_test.cc	Refactor atomic flush result installation to MANIFEST (#4791)	2019-01-03 20:56:24 -08:00
wal_manager.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
wal_manager.h
wal_manager_test.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
write_batch.cc	Update all unique/shared_ptr instances to be qualified with namespace std (#4638)	2018-11-09 11:19:58 -08:00
write_batch_base.cc
write_batch_internal.h
write_batch_test.cc	Clean up FragmentedRangeTombstoneList (#4692)	2018-11-28 15:29:02 -08:00
write_callback.h
write_callback_test.cc
write_controller.cc
write_controller.h
write_controller_test.cc
write_thread.cc	Add more sync point to fix flaky test GroupCommitTest	2018-11-07 14:07:53 -08:00
write_thread.h	Fix skip WAL for whole write_group when leader's callback fail (#4838)	2019-01-03 12:40:42 -08:00