rocksdb/db
sdong 7508175558 Introduce options.check_flush_compaction_key_order (#7467)
Summary:
Introduce an new option options.check_flush_compaction_key_order, by default set to true, which checks key order of flush and compaction, and fail the operation if the order is violated.
Also did minor refactor hash checking code, which consolidates the hashing logic to a vlidation class, where the key ordering logic is added.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/7467

Test Plan: Add unit tests to validate the check can catch reordering in flush and compaction, and can be properly disabled.

Reviewed By: riversand963

Differential Revision: D24010683

fbshipit-source-id: 8dd6292d2cda8006054e9ded7cfa4bf405f0527c
2020-10-01 10:10:26 -07:00
..
blob Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
compaction Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
db_impl Status check enforcement for timestamp_basic_test (#7454) 2020-09-29 18:23:27 -07:00
arena_wrapped_db_iter.cc dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
arena_wrapped_db_iter.h dedup ReadOptions in iterator hierarchy (#7210) 2020-08-03 15:23:04 -07:00
builder.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
builder.h Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
c.cc RocksJava - Add errorIfLogFileExists parameter to RocksDB.openReadOnly (#7046) 2020-09-17 15:41:25 -07:00
c_test.c Fix MSVC-related build issues (#7439) 2020-10-01 09:23:04 -07:00
column_family.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
column_family.h Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
column_family_test.cc Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-17 20:25:45 -07:00
compact_files_test.cc Replace reinterpret_cast with static_cast_with_check (#7067) 2020-07-02 19:25:41 -07:00
compacted_db_impl.cc Make db_basic_test pass assert status checked (#7452) 2020-09-29 09:49:04 -07:00
compacted_db_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
comparator_db_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
convenience.cc sst_dump to reduce number of file reads (#6836) 2020-05-12 18:23:33 -07:00
corruption_test.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
cuckoo_table_db_test.cc Replace reinterpret_cast with static_cast_with_check (#7067) 2020-07-02 19:25:41 -07:00
db_basic_test.cc Make db_basic_test pass assert status checked (#7452) 2020-09-29 09:49:04 -07:00
db_block_cache_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_bloom_filter_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_compaction_filter_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_compaction_test.cc Fix flaky intra-L0 consistency failure regression tests (#7477) 2020-09-30 16:50:24 -07:00
db_dynamic_level_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_encryption_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_filesnapshot.cc Fix checkpoint file deletion race with avoid_unnecessary_blocking_io (#7369) 2020-09-10 22:35:25 -07:00
db_flush_test.cc Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
db_info_dumper.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
db_info_dumper.h Add a DB Session ID (#6959) 2020-06-15 10:47:02 -07:00
db_inplace_update_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_io_failure_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_iter.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_iter.h Make db_basic_test pass assert status checked (#7452) 2020-09-29 09:49:04 -07:00
db_iter_stress_test.cc Test CircleCI with CLANG-10 (#7025) 2020-06-24 16:22:49 -07:00
db_iter_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_iterator_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_log_iter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_logical_block_size_cache_test.cc Get block size only in direct IO mode (#6522) 2020-03-20 15:26:10 -07:00
db_memtable_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_merge_operand_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_merge_operator_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_options_test.cc Add few unit test cases in ASSERT_STATUS_CHECKED build (#7427) 2020-09-24 21:48:57 -07:00
db_properties_test.cc Disable fsync in DB tests with timeouts (#7380) 2020-09-15 18:55:08 -07:00
db_range_del_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_sst_test.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
db_statistics_test.cc Add a new stats level to exclude tickers (#7329) 2020-09-04 23:25:03 -07:00
db_table_properties_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_tailing_iter_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_test.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
db_test2.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
db_test_util.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_test_util.h add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
db_universal_compaction_test.cc Disable fsync in DB tests with timeouts (#7380) 2020-09-15 18:55:08 -07:00
db_wal_test.cc Integrate blob file writing with recovery (#7388) 2020-09-15 17:14:10 -07:00
db_with_timestamp_basic_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
db_with_timestamp_compaction_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
db_write_test.cc Possible fix to flaky db_write_test (#7418) 2020-09-22 09:57:05 -07:00
dbformat.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
dbformat.h Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
dbformat_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
deletefile_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
error_handler.cc Status check enforcement for timestamp_basic_test (#7454) 2020-09-29 18:23:27 -07:00
error_handler.h Map retryable IO error during Flush without WAL to soft error and no switch memtable during resume (#7310) 2020-09-17 20:25:45 -07:00
error_handler_fs_test.cc fix the flaky test failure (#7415) 2020-09-19 17:57:54 -07:00
event_helpers.cc Add more tests to ASSERT_STATUS_CHECKED (#7367) 2020-09-16 15:48:07 -07:00
event_helpers.h Pass SST file checksum information through OnTableFileCreated (#7108) 2020-08-25 10:46:11 -07:00
experimental.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
external_sst_file_basic_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
external_sst_file_ingestion_job.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
external_sst_file_ingestion_job.h Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
external_sst_file_test.cc Disable fsync in some ExternalSSTFileTest tests (#7303) 2020-08-24 11:26:09 -07:00
fault_injection_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
file_indexer.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
file_indexer_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
filename_test.cc Add more tests to ASSERT_STATUS_CHECKED (#7211) 2020-08-06 17:19:41 -07:00
flush_job.cc Add flush_job_test to the list of ASSERT_STATUS_CHECKED tests (#7445) 2020-09-28 14:59:02 -07:00
flush_job.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
flush_job_test.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
flush_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
flush_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
forward_iterator.cc make L0 index/filter pinned memory usage predictable (#6911) 2020-06-09 16:51:23 -07:00
forward_iterator.h Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) 2020-04-15 17:40:44 -07:00
forward_iterator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
import_column_family_job.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
import_column_family_job.h Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
import_column_family_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
internal_stats.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
internal_stats.h Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
job_context.h Expose the set of live blob files from Version/VersionSet (#6785) 2020-05-04 15:08:13 -07:00
listener_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
log_format.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
log_reader.cc Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_reader.h Real fix for race in backup custom checksum checking (#7309) 2020-08-26 10:39:20 -07:00
log_test.cc Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) 2020-06-03 15:55:03 -07:00
log_writer.cc Fail recovery when MANIFEST record checksum mismatch (#6996) 2020-06-18 10:09:12 -07:00
log_writer.h Pass IOStatus to write path and set retryable IO Error as hard error in BG jobs (#6487) 2020-03-27 16:04:43 -07:00
logs_with_prep_tracker.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
logs_with_prep_tracker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
lookup_key.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
malloc_stats.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
manual_compaction_test.cc Skip high levels with no key falling in the range in CompactRange (#6482) 2020-03-04 20:15:25 -08:00
memtable.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
memtable.h Provide users with option to opt-in to get corrupt data in logs/messages (#7420) 2020-09-29 23:17:45 -07:00
memtable_list.cc Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
memtable_list.h Bug Fix for memtables not trimmed down. (#7296) 2020-08-21 13:29:05 -07:00
memtable_list_test.cc Add few unit test cases in ASSERT_STATUS_CHECKED build (#7427) 2020-09-24 21:48:57 -07:00
merge_context.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
merge_helper.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_helper_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_operator.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
merge_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
obsolete_files_test.cc Whole DBTest to skip fsync (#7274) 2020-08-17 18:42:25 -07:00
options_file_test.cc Add more tests to ASSERT_STATUS_CHECKED (#7367) 2020-09-16 15:48:07 -07:00
output_validator.cc Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
output_validator.h Introduce options.check_flush_compaction_key_order (#7467) 2020-10-01 10:10:26 -07:00
perf_context_test.cc Fix MSVC-related build issues (#7439) 2020-10-01 09:23:04 -07:00
pinned_iterators_manager.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
plain_table_db_test.cc More Makefile Cleanup (#7097) 2020-07-09 14:35:17 -07:00
pre_release_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
prefix_test.cc Fix cmake build on MacOS (#7205) 2020-08-10 10:48:05 -07:00
range_del_aggregator.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
range_del_aggregator.h Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
range_del_aggregator_bench.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_del_aggregator_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
range_tombstone_fragmenter_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
read_callback.h Get() with timestamp should respect snapshot (#7227) 2020-08-14 19:20:58 -07:00
repair.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
repair_test.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
snapshot_checker.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
snapshot_impl.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_cache.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
table_cache.h Store FSRandomAccessPtr object in RandomAccessFileReader (#7192) 2020-08-27 11:21:52 -07:00
table_properties_collector.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
table_properties_collector.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
table_properties_collector_test.cc Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
transaction_log_impl.cc Store FSSequentialFilePtr object in SequenceFileReader (#7190) 2020-08-18 16:20:54 -07:00
transaction_log_impl.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
trim_history_scheduler.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
trim_history_scheduler.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
version_builder.cc Enable force_consistency_checks by default (#7446) 2020-09-30 11:57:32 -07:00
version_builder.h make L0 index/filter pinned memory usage predictable (#6911) 2020-06-09 16:51:23 -07:00
version_builder_test.cc Add version_builder_test to the list of ASSERT_STATUS_CHECKED tests (#7444) 2020-09-28 12:12:40 -07:00
version_edit.cc Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
version_edit.h Integrate blob file writing with the flush logic (#7345) 2020-09-14 21:11:43 -07:00
version_edit_handler.cc Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_edit_handler.h Store FSWritableFilePtr object in WritableFileWriter (#7193) 2020-09-08 10:56:08 -07:00
version_edit_test.cc Track WAL in MANIFEST: update WalMetadata for WAL syncing (#7414) 2020-09-22 14:35:14 -07:00
version_set.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
version_set.h Fix TSAN build and re-enable the tests (#7386) 2020-09-25 14:46:28 -07:00
version_set_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
wal_edit.cc Track WAL in MANIFEST: add method to check WAL consistency (#7236) 2020-09-25 13:25:54 -07:00
wal_edit.h Track WAL in MANIFEST: add method to check WAL consistency (#7236) 2020-09-25 13:25:54 -07:00
wal_edit_test.cc Track WAL in MANIFEST: add method to check WAL consistency (#7236) 2020-09-25 13:25:54 -07:00
wal_manager.cc add Status check assertions for repair_test (#7455) 2020-09-29 16:30:08 -07:00
wal_manager.h Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
wal_manager_test.cc Store FileSystemPtr object that contains FileSystem ptr (#7180) 2020-08-12 17:31:23 -07:00
write_batch.cc Add some simulator cache and block tracer tests to ASSERT_STATUS_CHECKED (#7305) 2020-08-24 16:43:31 -07:00
write_batch_base.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_internal.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_batch_test.cc Change ParseInternalKey() to return Status instead of bool (#7457) 2020-09-30 19:16:47 -07:00
write_callback.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_callback_test.cc Divide WriteCallbackTest.WriteWithCallbackTest (#7037) 2020-06-30 12:31:30 -07:00
write_controller.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller.h Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_controller_test.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
write_thread.cc Assert valid linked list for write group (#7375) 2020-09-11 07:58:31 -07:00
write_thread.h Add EnvTestWithParam::OptionsTest to the ASSERT_STATUS_CHECKED passes (#7283) 2020-08-20 19:18:35 -07:00