mirror of
https://github.com/facebook/rocksdb.git
synced 2024-12-01 16:15:59 +00:00
12925 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Peter Dillinger
|
0e3e43f4d1 |
FaultInjectionTestFS follow-up and clean-up (#12861)
Summary: In follow-up to https://github.com/facebook/rocksdb/issues/12852: * Use std::copy in place of copy_n for potentially overlapping buffer * Get rid of troublesome -1 idiom from `pos_at_last_append_` and `pos_at_last_sync_` * Small improvements to test FaultInjectionFSTest.ReadUnsyncedData Pull Request resolved: https://github.com/facebook/rocksdb/pull/12861 Test Plan: CI, crash test, etc. Reviewed By: cbi42 Differential Revision: D59757484 Pulled By: pdillinger fbshipit-source-id: c6fbdc2e97c959983184925a855cc8b0285fa23f |
||
|
Changyu Bi
|
b800b5eb6a |
Deflake ThreadStatus related unit tests (#12858)
Summary:
Unit tests `DBTest.ThreadStatusFlush` and `DBTestWithParam.ThreadStatusSingleCompaction` have been flaky and fail with error message
```
[ RUN ] DBTest.ThreadStatusFlush
op_count: 0, expected_count 1
thread id: 718113, thread status: , cf_name
thread id: 718114, thread status: , cf_name pikachu
/__w/rocksdb/rocksdb/db/db_test.cc:4817: Failure
Value of: VerifyOperationCount(env_, ThreadStatus::OP_FLUSH, 1)
Actual: false
Expected: true
[ FAILED ] DBTest.ThreadStatusFlush (106 ms)
[ RUN ] DBTestWithParam/DBTestWithParam.ThreadStatusSingleCompaction/0
db/db_test.cc:4673: Failure
Expected equality of these values:
op_count
Which is: 0
expected_count
Which is: 1
[ FAILED ] DBTestWithParam/DBTestWithParam.ThreadStatusSingleCompaction/0, where GetParam() = (1, false)
```
One cause for this is that before flush/compaction finishes, we will go through `~WritableFileWriter()`, either for WAL or SST file, and temporarily set thread_operation to UNKNOWN. This UNKNOWN thread operation seem to be there for some stress test verification. This PR fixes these tests by setting the IOActivity in ~WritableFileWriter() for debug build.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12858
Test Plan: monitor future test failure.
Reviewed By: hx235
Differential Revision: D59691564
Pulled By: cbi42
fbshipit-source-id: 3f96998bba9d42aba50d1830c2b51bef2dd6705f
|
||
|
Peter Dillinger
|
72438a6788 |
Support read & write with unsynced data in FaultInjectionTestFS (#12852)
Summary: Follow-up to https://github.com/facebook/rocksdb/issues/12729 and others to fix FaultInjectionTestFS handling the case where a live WAL is being appended to and synced while also being copied for checkpoint or backup, up to a known flushed (but not necessarily synced) prefix of the file. It was tricky to structure the code in a way that could handle a tricky race with Sync in another thread (see code comments, thanks Changyu) while maintaining good performance and test-ability. For more context, see the call to FlushWAL() in DBImpl::GetLiveFilesStorageInfo(). Also, the unit test for https://github.com/facebook/rocksdb/issues/12729 was neutered by https://github.com/facebook/rocksdb/issues/12797, and this re-enables the functionality it is testing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12852 Test Plan: unit test expanded/updated. Local runs of blackbox_crash_test. The implementation is structured so that a multi-threaded unit test is not needed to cover at least the code lines, as the race handling is folded into "catch up after returning unsynced and then a sync." Reviewed By: cbi42 Differential Revision: D59594045 Pulled By: pdillinger fbshipit-source-id: 94667bb72255e2952586c53bae2c2dd384e85a50 |
||
|
Yu Zhang
|
3db030d7ee |
Fix bug for recovering a prepared but not committed txn (#12856)
Summary:
This PR fix a bug for recovering a prepared Transaction that can contain user-defined timestamps.
The `Transaction::Put` type of APIs expect the key provided to be user key without timestamps. When the original transaction added a key for a column family that enables user-defined timestamps, say of size 8. Internally `WriteBatch::Put` will leave a placeholder 8 bytes for the final commit timestamp. For example:
|
||
|
Changyu Bi
|
cec28aa90f |
Fix SetOptions() failure in stress test (#12854)
Summary: fix SetOptions() so that max_read_amp is at least level0_file_num_compaction_trigger. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12854 Test Plan: monitor stress test new failure Reviewed By: hx235 Differential Revision: D59618547 Pulled By: cbi42 fbshipit-source-id: b83371f293b87097ee9cdd32d662e9965cde57e6 |
||
|
anand76
|
37b81bd28f |
Avoid SyncWAL if flushing during shutdown (#12853)
Summary: https://github.com/facebook/rocksdb/issues/12746 added calls to FlushWAL/SyncWAL in db_stress during reopen, in order to ensure persistence of unpersisted data and avoid false alarms due to lack of prefix recovery support in db_stress reopen. However, there's no need to flush/sync the WAL if avoid_flush_during_shutdown is false, as the WAL will not be needed during recovery. This allows file systems that don't support SyncWAL (not thread safe) to avoid the need by requesting flush during shutdown. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12853 Reviewed By: hx235 Differential Revision: D59604138 Pulled By: anand1976 fbshipit-source-id: 4c4470b3c956d6bf64f5b8a1a5727a8b888f1a5f |
||
|
Jay Huh
|
6997dd909c |
Disable attribute group txn tests (#12851)
Summary: Transactions are not yet supported in AttributeGroup APIs. Disabling `use_attribute_group` for txn tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/12851 Test Plan: Verified output that `--use_attribute_group=0` ``` python3 tools/db_crashtest.py whitebox --txn ``` ``` python3 tools/db_crashtest.py whitebox --optimistic_txn ``` Reviewed By: hx235 Differential Revision: D59565635 Pulled By: jaykorean fbshipit-source-id: 7d618f475b6d2e5a53c3c59cdf1e694f3893ae58 |
||
|
Changyu Bi
|
d6f265f9d6 |
Fix race in multiops txn stress test (#12847)
Summary: `MultiOpsTxnsStressListener::OnCompactionCompleted()` access `db_` and can be called while db_ is being destroyed in ~StressTest(). This causes TSAN to complain about data race. This PR fixes this issue by calling db_->Close() first to stop all background work. Also moved the cleanup out of StressTest destructor to avoid race between the listener and ~StressTest(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/12847 Test Plan: monitor crash test failure. Reviewed By: hx235 Differential Revision: D59492691 Pulled By: cbi42 fbshipit-source-id: afcbab084cc9ac0904d6b04809b0888498ca8e66 |
||
|
Hui Xiao
|
ebe2116240 |
Remove false-postive assertion in FaultInjectionTestFS::RenameFile (#12828)
Summary: **Context/Summary:** The assertion `tlist.find(tdn.second) == tlist.end()` |
||
|
Konstantin Ilin
|
5ecb92760a |
Create C API function to iterate over WriteBatch for custom Column Families (#12718)
Summary: Create C API function for iterating over WriteBatch for custom Column Families Adding function to C API that exposes column family specific methods to iterate over WriteBatch: put_cf, delete_cf and merge_cf. This is required when the one needs to read changes for any non-default column family. Without that functionality it is impossible to iterate over changes in WAL that are relevant to custom column families. Fixes https://github.com/facebook/rocksdb/issues/12790 Testing: Added WriteBatch iteration test to "columnfamilies" section of C API unit tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/12718 Reviewed By: cbi42 Differential Revision: D59483601 Pulled By: ajkr fbshipit-source-id: b68b900636304528a38620a8c3ad82fdce4b60cb |
||
|
w41ter
|
b837d41ab1 |
Expose SizeApproximationFlags to C API (#12836)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12836 Reviewed By: cbi42 Differential Revision: D59502673 Pulled By: ajkr fbshipit-source-id: fc9f77d6740d8efa45d9357662f0f827dbd0511f |
||
|
Yu Zhang
|
2e1b3f921f |
Remove unreachable code (#12846)
Summary: Removing some unreachable code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12846 Reviewed By: cbi42 Differential Revision: D59498423 Pulled By: jowlyzhang fbshipit-source-id: 6b2c51732d94b1f69a8ba7474b16a171d4e6d640 |
||
|
Jeffery
|
62b62cf135 |
Fix CondVar::TimedWait for Windows (#12815)
Summary: Based on https://github.com/microsoft/STL/issues/369 They fixed the issue in `std::condition_variable_any` but not in `std::condition_variable`, which is currently used in rocksdb repo. So we need to implement the work around regardless of `_MSVC_STL_UPDATE`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12815 Reviewed By: cbi42 Differential Revision: D59493690 Pulled By: ajkr fbshipit-source-id: ad0fc9ef9f2357347d21e271c2f1d0a3a97d89be |
||
|
Zixuan Tan
|
a97a1f3247 |
Fix incorrect refillPeriodMicros unit in the document (#12832)
Summary: The default value for `refillPeriodMicros` is `100 * 1000`, which means 100ms (or 100,000us). The document comments say 100,000ms (equivalent to 100 seconds), which is incorrect and misleading. This PR fixes this typo. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12832 Reviewed By: cbi42 Differential Revision: D59492336 Pulled By: ajkr fbshipit-source-id: c2f55a8b996fe078a1510fcbebaea92ec0075929 |
||
|
WangQian
|
f471e56190 |
fix the non initialized bug in StderrLogger. (#12839)
Summary: This PR is intended to fix a potential uninitialized variable bug. Fixes https://github.com/facebook/rocksdb/issues/12837 Pull Request resolved: https://github.com/facebook/rocksdb/pull/12839 Reviewed By: ajkr Differential Revision: D59398888 Pulled By: cbi42 fbshipit-source-id: 337391d7c1e73c0ff61797f88fbb4a8379500211 |
||
|
Chdy
|
110ce5f4a3 |
fix: Round-Robin pri under leveled compaction allows subcompactions b… (#12843)
Summary:
### Summary: Round-Robin pri under leveled compaction allows subcompactions by default is not compatible with PlainTable
```c++
bool Compaction::ShouldFormSubcompactions() const {
if (cfd_ == nullptr) {
return false;
}
// Round-Robin pri under leveled compaction allows subcompactions by default
// and the number of subcompactions can be larger than max_subcompactions_
if (cfd_->ioptions()->compaction_pri == kRoundRobin &&
cfd_->ioptions()->compaction_style == kCompactionStyleLevel) {
return output_level_ > 0;
}
if (max_subcompactions_ <= 1) {
return false;
}
```
PlainTable does not support Subcompaction, including when AdaptiveTable is applied to PlainTable. subcompaction by default will result in the following error in some scenarios.
```c++
void PlainTableIterator::Seek(const Slice& target) {
if (use_prefix_seek_ != !table_->IsTotalOrderMode()) {
// This check is done here instead of NewIterator() to permit creating an
// iterator with total_order_seek = true even if we won't be able to Seek()
// it. This is needed for compaction: it creates iterator with
// total_order_seek = true but usually never does Seek() on it,
// only SeekToFirst().
status_ = Status::InvalidArgument(
"total_order_seek not implemented for PlainTable.");
offset_ = next_offset_ = table_->file_info_.data_end_offset;
return;
}
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12843
Reviewed By: ajkr
Differential Revision: D59433477
Pulled By: cbi42
fbshipit-source-id: fb780ba7f7e8efdfedb7480abf14dd38e0b63677
|
||
|
Radek Hubner
|
b6c3495a71 |
Update snappy dependency for Java releases. (#12207)
Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12207 Reviewed By: hx235 Differential Revision: D59299915 Pulled By: cbi42 fbshipit-source-id: 3f5fa88b0c5e8366a08734f99db1d3de942cd60b |
||
|
Hui Xiao
|
1f589a3f73 |
Clarify GetProperty API doc (#12829)
Summary:
**Context/Summary:** as titled since
|
||
|
Changyu Bi
|
de16611a50 |
Fix WAL corruption in stress test (#12834)
Summary: We are seeing WAL corruption in crash tests where wal_compression and recycled_wal are enabled. With wal_compression, we write a SetCompression record when creating a WAL, which can happen during DB open time. Our current stress test set up may write directly to the underlying WAL file during DB open, while writing to a buffer under TestFSWritableFile later for sync fault injection. This causes the last synced position to be inaccurately tracked in TestFSWritableFile and causes reads to return incorrect data. This PR removes the line that causes this mixture of WAL writes. Also updated TestFSWritableFile to avoid such a mixture of buffered and direct writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12834 Test Plan: the following command repros WAL corruption before this PR ``` ./db_stress --WAL_size_limit_MB=0 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --adm_policy=1 --advise_random_on_open=1 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=0 --async_io=1 --auto_readahead_size=0 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=0 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=8 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=1 --block_size=16384 --bloom_before_level=0 --bloom_bits=8 --bottommost_compression_type=snappy --bottommost_file_compaction_delay=600 --bytes_per_sync=0 --cache_index_and_filter_blocks=0 --cache_index_and_filter_blocks_with_high_priority=1 --cache_size=33554432 --cache_type=auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=1 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=10000 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000 --compact_range_one_in=1000000 --compaction_pri=4 --compaction_readahead_size=1048576 --compaction_ttl=0 --compress_format_version=1 --compressed_secondary_cache_size=16777216 --compression_checksum=1 --compression_max_dict_buffer_bytes=1099511627775 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=zstd --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=1 --db_write_buffer_size=134217728 --default_temperature=kHot --default_write_temperature=kCold --delete_obsolete_files_period_micros=21600000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_file_deletions_one_in=1000000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=1 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=1 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=0 --exclude_wal_from_write_fault_injection=1 --fail_if_options_file_error=0 --fifo_allow_compaction=1 --file_checksum_impl=crc32c --fill_cache=0 --flush_one_in=1000000 --format_version=2 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=1000000 --get_properties_of_all_tables_one_in=1000000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0.5 --index_block_restart_interval=13 --index_shortening=0 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=0 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log_file_time_to_roll=0 --log_readahead_size=0 --long_running_snapshots=1 --low_pri_pool_ratio=0 --lowest_used_cache_tier=0 --manifest_preallocation_size=5120 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=8 --max_total_wal_size=0 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=10 --max_write_buffer_size_to_maintain=4194304 --memtable_insert_hint_per_batch=1 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.5 --memtable_protection_bytes_per_key=0 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=1 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=1 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=1 --num_file_reads_for_auto_readahead=1 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=16 --ops_per_thread=100000000 --optimize_filters_for_hits=1 --optimize_filters_for_memory=1 --optimize_multiget_for_io=1 --paranoid_file_checks=0 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=10000 --periodic_compaction_seconds=0 --prefix_size=8 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=1000 --readahead_size=16384 --readpercent=45 --recycle_log_file_num=1 --reopen=0 --report_bg_io_stats=1 --reset_stats_one_in=10000 --sample_for_compression=0 --secondary_cache_fault_one_in=0 --secondary_cache_uri= --set_options_one_in=0 --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=1048576 --sqfc_name=bar --sqfc_version=2 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=4 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=-1 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=3 --uncache_aggressiveness=113 --universal_max_read_amp=4 --unpartitioned_pinning=3 --use_adaptive_mutex=1 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=1 --use_direct_io_for_flush_and_compaction=1 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=1 --use_multiget=0 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=5 --use_write_buffer_manager=1 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=0 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=zstd --write_buffer_size=33554432 --write_dbid_to_manifest=0 --write_fault_one_in=128 --writepercent=35 --preserve_unverified_changes=1 --db=/dev/shm/rocksdb_test/blackbox --expected_values_dir=/dev/shm/rocksdb_test/expected Choosing random keys with no overwrite ... (Re-)verified 0 unique IDs 2024/07/01-16:42:46 Initializing worker threads Crash-recovery verification passed :) 2024/07/01-16:42:46 Starting database operations ^C ./db_stress --WAL_size_limit_MB=0 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=10000 --adaptive_readahead=0 --adm_policy=1 --advise_random_on_open=1 --allow_concurrent_memtable_write=0 --allow_data_in_errors=True --allow_fallocate=0 --async_io=1 --auto_readahead_size=0 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=0 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=1000 --batch_protection_bytes_per_key=8 --bgerror_resume_retry_interval=1000000 --block_align=0 --block_protection_bytes_per_key=1 --block_size=16384 --bloom_before_level=0 --bloom_bits=8 --bottommost_compression_type=snappy --bottommost_file_compaction_delay=600 --bytes_per_sync=0 --cache_index_and_filter_blocks=0 --cache_index_and_filter_blocks_with_high_priority=1 --cache_size=33554432 --cache_type=auto_hyper_clock_cache --charge_compression_dictionary_building_buffer=1 --charge_file_metadata=0 --charge_filter_construction=1 --charge_table_reader=1 --check_multiget_consistency=0 --check_multiget_entity_consistency=0 --checkpoint_one_in=10000 --checksum_type=kxxHash64 --clear_column_family_one_in=0 --column_families=1 --compact_files_one_in=1000 --compact_range_one_in=1000000 --compaction_pri=4 --compaction_readahead_size=1048576 --compaction_ttl=0 --compress_format_version=1 --compressed_secondary_cache_size=16777216 --compression_checksum=1 --compression_max_dict_buffer_bytes=1099511627775 --compression_max_dict_bytes=16384 --compression_parallel_threads=1 --compression_type=zstd --compression_use_zstd_dict_trainer=1 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=1 --db_write_buffer_size=134217728 --default_temperature=kHot --default_write_temperature=kCold --delete_obsolete_files_period_micros=21600000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=1 --disable_file_deletions_one_in=1000000 --disable_manual_compaction_one_in=10000 --disable_wal=0 --dump_malloc_stats=0 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=1 --enable_index_compression=1 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=1 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=1 --error_recovery_with_no_fault_injection=0 --exclude_wal_from_write_fault_injection=1 --fail_if_options_file_error=0 --fifo_allow_compaction=1 --file_checksum_impl=crc32c --fill_cache=0 --flush_one_in=1000000 --format_version=2 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=1000000 --get_properties_of_all_tables_one_in=1000000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0.5 --index_block_restart_interval=13 --index_shortening=0 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=0 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kUnknown --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log_file_time_to_roll=0 --log_readahead_size=0 --long_running_snapshots=1 --low_pri_pool_ratio=0 --lowest_used_cache_tier=0 --manifest_preallocation_size=5120 --manual_wal_flush_one_in=0 --mark_for_compaction_one_file_in=0 --max_auto_readahead_size=524288 --max_background_compactions=1 --max_bytes_for_level_base=67108864 --max_key=100000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=8 --max_total_wal_size=0 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=10 --max_write_buffer_size_to_maintain=4194304 --memtable_insert_hint_per_batch=1 --memtable_max_range_deletions=0 --memtable_prefix_bloom_size_ratio=0.5 --memtable_protection_bytes_per_key=0 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=1 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=1 --mmap_read=0 --mock_direct_io=True --nooverwritepercent=1 --num_file_reads_for_auto_readahead=1 --open_files=-1 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=16 --ops_per_thread=100000000 --optimize_filters_for_hits=1 --optimize_filters_for_memory=1 --optimize_multiget_for_io=1 --paranoid_file_checks=0 --partition_filters=0 --partition_pinning=3 --pause_background_one_in=10000 --periodic_compaction_seconds=0 --prefix_size=8 --prefixpercent=5 --prepopulate_block_cache=1 --preserve_internal_time_seconds=0 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=1000 --readahead_size=16384 --readpercent=45 --recycle_log_file_num=1 --reopen=0 --report_bg_io_stats=1 --reset_stats_one_in=10000 --sample_for_compression=0 --secondary_cache_fault_one_in=0 --secondary_cache_uri= --set_options_one_in=0 --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=1048576 --sqfc_name=bar --sqfc_version=2 --sst_file_manager_bytes_per_sec=0 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=10 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=4 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=-1 --target_file_size_base=16777216 --target_file_size_multiplier=1 --test_batches_snapshots=0 --top_level_index_pinning=3 --uncache_aggressiveness=113 --universal_max_read_amp=4 --unpartitioned_pinning=3 --use_adaptive_mutex=1 --use_adaptive_mutex_lru=1 --use_attribute_group=0 --use_delta_encoding=1 --use_direct_io_for_flush_and_compaction=1 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=1 --use_multiget=0 --use_put_entity_one_in=0 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=5 --use_write_buffer_manager=1 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=0 --verify_db_one_in=10000 --verify_file_checksums_one_in=1000 --verify_iterator_with_expected_state_one_in=5 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=zstd --write_buffer_size=33554432 --write_dbid_to_manifest=0 --write_fault_one_in=128 --writepercent=35 --preserve_unverified_changes=1 --db=/dev/shm/rocksdb_test/blackbox --expected_values_dir=/dev/shm/rocksdb_test/expected Choosing random keys with no overwrite ... Crash-recovery verification passed :) 2024/07/01-16:43:02 Starting database operations Failure in BackupEngine::CreateNewBackup with: Corruption: bad record length under specified BackupEngineOptions: share_table_files: 1, share_files_with_checksum: 1, share_files_with_checksum_naming: 2147483650, schema_version: 1, max_background_operations: 1, backup_rate_limiter: 0x7f2373676280, restore_rate_limiter: 0, current_temperatures_override_manifest: 1, CreateBackupOptions: flush_before_backup: 0, decrease_background_thread_cpu_priority: 0, background_thread_cpu_priority: 2, RestoreOptions: keep_log_files: 1 (Empty string or missing field indicates default option or value is used) Verification failed: Backup/restore failed: Corruption: bad record length db_stress: db_stress_tool/db_stress_test_base.cc:528: void rocksdb::StressTest::ProcessStatus(rocksdb::SharedState*, std::string, const rocksdb::Status&, bool) const: Assertion `false' failed. Received signal 6 (Aborted) Invoking GDB for stack trace... ^CCouldn't get CS register: No such process. Couldn't get registers: No such process. [Inferior 1 (process 2097222) detached] ``` Reviewed By: pdillinger Differential Revision: D59260401 Pulled By: cbi42 fbshipit-source-id: fdcdaaab2e14b527b26fbdfa819b4fe3f745a4de |
||
|
Peter Dillinger
|
0bb939611d |
Avoid unnecessary work in internal calls to GetSortedWalFiles (#12831)
Summary: We are seeing a number of crash test failures coming from checkpoint and backup code, likely from WalManager::GetSortedWalFiles -> ... -> WalManager::ReadFirstLine and this code path is not needed, because we don't need to know the sequence numbers of WAL files going into a checkpoint or backup. We can minimize the impact of whatever inconsistency is causing that problem by not relying on it where it's not needed. Similarly, when we only need a roughly accurate set of current WAL files, we don't need to query all the archived WAL files (and redundantly the live ones again). So this reduces filesystem queries and DB mutex acquires in creating backups and checkpoints. Needed follow-up: Figure out what is causing various failures with an apparent inconsistency where GetSortedWalFiles fails on reading a WAL file. If it's an injected failure, perhaps it's not propagating that injected failure appropriately. It might also be an inconsistency between what the DB knows is flushed and what WalManager reads from the filesystem (which we know is dubious and should be phased out, which this is arguably another step toward). Or completing that phase-out might solve the problem without a full diagnosis. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12831 Test Plan: existing tests (easily caught when I went too far in initally developing this change) Update to BackupUsingDirectIO test so that there's a WAL file in what is backed up. (Was relying on some oddity.) Reviewed By: cbi42 Differential Revision: D59252649 Pulled By: pdillinger fbshipit-source-id: 7ad4187a1c70caa59a6d6c1c643ef95232b929f5 |
||
|
Hui Xiao
|
84296bc248 |
Reset seen_injected_error_ with seen_error_ (#12830)
Summary: **Context/Summary** : as titled as seen_injected_error_ is a subcategory of seen_error_ Pull Request resolved: https://github.com/facebook/rocksdb/pull/12830 Test Plan: existing CI as it only affects crash test code Reviewed By: jaykorean Differential Revision: D59249018 Pulled By: hx235 fbshipit-source-id: 20e4c22cade57e12a104a03999e4c841a3648b11 |
||
|
Jeffery
|
093f4ef82c |
Fix db_rate_limiter_test for win (#12816)
Summary: We didn't implement file system prefetch for OS Win. During table open, it uses `FilePrefetchBuffer` instead and only do 1 read instead of 4 in BufferedIO. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12816 Reviewed By: jaykorean Differential Revision: D59181835 Pulled By: ajkr fbshipit-source-id: 18b8f0247408cd1a80f289357ede5232ae5a3c66 |
||
|
Changyu Bi
|
9eebaf11cb |
Fix stress test SetOptions() setting incompatible options (#12827)
Summary: To fix errors like "Verification failed: SetOptions failed: Invalid argument: max_successive_merges > 0 is incompatible with unordered_write". Pull Request resolved: https://github.com/facebook/rocksdb/pull/12827 Test Plan: no new crash test failure due to this option combination. Reviewed By: hx235 Differential Revision: D59233002 Pulled By: cbi42 fbshipit-source-id: 2a3e4d57a56f07bdda49ea36f0f9f6a30f17bbc3 |
||
|
Hui Xiao
|
69ad597b46 |
Disable fault injection for TestGetProperty (#12825)
Summary: **Context/Summary:** See titled; along with one more minor fix to other disabling Pull Request resolved: https://github.com/facebook/rocksdb/pull/12825 Test Plan: CI won't show `Failed to get DB property: rocksdb.aggregated-table-properties` Reviewed By: jaykorean Differential Revision: D59231819 Pulled By: hx235 fbshipit-source-id: a8e73c9e06eeceb4c6025a4885823a3eba25c359 |
||
|
Hui Xiao
|
c7e94bc878 |
Disable WAL write error injection when reopen with WAL (#12820)
Summary:
**Context/Summary:**
Right now we need to persist WAL data before closing for reopen when `reopen > 0` and `disable_wal = false`
|
||
|
Jay Huh
|
22fe23edc8 |
Fix unknown flag "manual_wal_flush" (#12823)
Summary: - Fix `manual_wal_flush` -> `manual_wal_flush_one_in` - auto-formatter fixed format in other settings Pull Request resolved: https://github.com/facebook/rocksdb/pull/12823 Test Plan: CI Reviewed By: hx235 Differential Revision: D59177107 Pulled By: jaykorean fbshipit-source-id: 2400b2822f42299d03e150e3a098c62e7fdaf1f8 |
||
|
Yu Zhang
|
8c1558a3e0 |
Add some documentation for Env related interfaces (#12813)
Summary: As titled. Added some documentations for some `Env` interfaces and removed some obsolete doc for `Options.env`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12813 Reviewed By: ajkr Differential Revision: D59119632 Pulled By: jowlyzhang fbshipit-source-id: 114b13f0f843cde6ebc0746156b80c94ea2ce143 |
||
|
Hui Xiao
|
0b10e7dbba |
Ignore more non-critical IO error in BlockCacheLookupForReadAheadSize() in crash test (#12822)
Summary: **Context/Summary:** Similar to https://github.com/facebook/rocksdb/pull/12814#issue-2376803461 Pull Request resolved: https://github.com/facebook/rocksdb/pull/12822 Test Plan: CI Reviewed By: jaykorean Differential Revision: D59166188 Pulled By: hx235 fbshipit-source-id: 68c2eb7b319103ede0ba34944a0737440aecb17f |
||
|
HypenZou
|
5bb7f95ed6 |
Don't take archived log size into account when calculating log size for flush (#12680)
Summary: **Context/Summary:** It seems unreasonable to take the archived log size into account when calculating log size **for flush** in method CreateCheckpoint. If the user sets WAL_ttl_seconds or WAL_size_limit_MB, the argument _log_size_for_flush_ can easily be reached due to the size of the archived dir. As a result, the flush may always be triggered. **Test** corverd by ./checkpoint_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/12680 Reviewed By: jaykorean Differential Revision: D59097904 Pulled By: ajkr fbshipit-source-id: 0ed29c1b078d8f40b85288541b008e00dbc517d3 |
||
|
Hui Xiao
|
aec15eebec |
Ignore non-critical IO error in BlockCacheLookupForReadAheadSize() in crash test (#12814)
Summary:
**Context/Summary:**
Error in `BlockCacheLookupForReadAheadSize()` is not critical enough to return such error in read path. That's because the worst case is to not have any read ahead. See below comment.
|
||
|
Changyu Bi
|
a31fe52173 |
Remove the return value of SetBGError() (#12792)
Summary: the return value for `ErrorHandler::SetBGError(error)` seems to be not well-defined, it can be `bg_error_` (no matter if the `bg_error_` is set to the input error), ok status or [`recovery_error_`]( |
||
|
Hui Xiao
|
0d93c8a6ca |
Decouple sync fault and write injection in FaultInjectionTestFS & fix tracing issue under WAL write error injection (#12797)
Summary:
**Context/Summary:**
After injecting write error to WAL, we started to see crash recovery verification failure in prefix recovery. That's because the current tracing implementation traces every write before it writes to WAL even when the WAL write can fail with write error injection. One consequence of that is the traced writes in trace files does not corresponding to write sequence sequence anymore e.g, it has more traced writes that the actual assigned sequence number to successful writes. Therefore
|
||
|
Hui Xiao
|
41c6b4549a |
Revert back to previous ReadAsync error injection (#12811)
Summary:
**Context/Summary:**
https://github.com/facebook/rocksdb/pull/12713 adjusted the error injection in ReadAsync. See original behavior here
|
||
|
Hui Xiao
|
58bc4db456 |
Print more debugging info & further disable backup/restore error inejction (#12812)
Summary: **Context/Summary:** Print more info for debugging a TestCheckpoint error; further disable backup/restore error injection as it has not been stabilized with our new thread-local error injection. Will need to enable it separately later. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12812 Test Plan: CI Reviewed By: jaykorean Differential Revision: D59072678 Pulled By: hx235 fbshipit-source-id: 9481ccf62db952288e7f47ee4b68a34ad0651d5c |
||
|
Jason Volk
|
8bf1f6f87f |
Add info logging via callback to C api. (#12537)
Summary: I'd like to get this in so the Rust folks can integrate with their splendid logging/tracing frameworks; will be hugely appreciated. 🙏🏻 The infolog capabilities for C embeddings are quite spartan. LOG files were generated involuntarily until redirection to stderr was added by https://github.com/facebook/rocksdb/issues/12262; still insufficient for apps which cannot tolerate pollution of their stdio and tend to have existing logging frameworks to tie into for that. Adds a very minimal derive of Logger around a C callback, written in the spirit, useful for FFI interfaces from other languages to integrate infolog. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12537 Reviewed By: ajkr Differential Revision: D57597766 Pulled By: cbi42 fbshipit-source-id: ec684ce4ddf77a0a6ebbf013a1bacb4ff2e49eb0 |
||
|
Richard Barnes
|
a06a7fdc88 |
Remove extra semi colon from internal_repo_rocksdb/repo/util/xxhash.h
Summary: `-Wextra-semi` or `-Wextra-semi-stmt` If the code compiles, this is safe to land. Reviewed By: dmm-fb Differential Revision: D59007259 fbshipit-source-id: ee0e01e1cc14ebe183d3b74153ef77f11625d983 |
||
|
Hui Xiao
|
6f79496475 |
Tag FaultInjectionIOType::kRead for FaultInjectionTestFS new read file & fix unrelease snapshot (#12810)
Summary:
**Context/Summary:** It makes more sense to mark error injection during creation as read file as "kread" so we don't get confusing msg like below
```
stderr:
error : Get() returns IO error: injected metadata write error for key: 000000000000004F000000000000012B00000000000000EF.
Verification failed :(
```
Also an early return here
|
||
|
Hui Xiao
|
e0ddbee76f |
Remove unnecessary injected error logging in crash test (#12807)
Summary: Context/Summary: as titled, since injected error log isn't that useful for debugging and takes up a lot of console printing space Pull Request resolved: https://github.com/facebook/rocksdb/pull/12807 Test Plan: CI Reviewed By: pdillinger, jowlyzhang Differential Revision: D58969796 Pulled By: hx235 fbshipit-source-id: 1663fb0779d7a049fc3b101ddefd263be7bdd4b5 |
||
|
Yu Zhang
|
ff204d8ecd |
Add entry for #12803: fix race between event listener and error handler (#12809)
Summary: As titled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12809 Reviewed By: hx235 Differential Revision: D58974154 Pulled By: jowlyzhang fbshipit-source-id: 7e44b54d9fa3bfbd58a4154a2c7e91aec905c34b |
||
|
Hui Xiao
|
56f7ef50d7 |
Fix nullptr access and race to fault_fs_guard (#12799)
Summary:
**Context/Summary:**
There are a couple places where we forgot to check fault_fs_guard before accessing it. So we can see something like this occasionally
```
=138831==Hint: address points to the zero page.
SCARINESS: 10 (null-deref)
AddressSanitizer:DEADLYSIGNAL
#0 0x18b9e0b in rocksdb::ThreadLocalPtr::Get() const fbcode/internal_repo_rocksdb/repo/util/thread_local.cc:503
https://github.com/facebook/rocksdb/issues/1 0x83d8b7 in rocksdb::StressTest::TestCompactRange(rocksdb::ThreadState*, long, rocksdb::Slice const&, rocksdb::ColumnFamilyHandle*) fbcode/internal_repo_rocksdb/repo/utilities/fault_injection_fs.h
```
Also accessing of `io_activties_exempted_from_fault_injection.find` not fully synced so we see the following
```
WARNING: ThreadSanitizer: data race (pid=90939)
Write of size 8 at 0x7b4c000004d0 by thread T762 (mutexes: write M0):
#0 std::_Rb_tree<rocksdb::Env::IOActivity, rocksdb::Env::IOActivity, std::_Identity<rocksdb::Env::IOActivity>, std::less<rocksdb::Env::IOActivity>, std::allocator<rocksdb::Env::IOActivity>>::operator=(std::_Rb_tree<rocksdb::Env::IOActivity, rocksdb::Env::IOActivity, std::_Identity<rocksdb::Env::IOActivity>, std::less<rocksdb::Env::IOActivity>, std::allocator<rocksdb::Env::IOActivity>> const&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_tree.h:208 (db_stress+0x411c32) (BuildId: b803e5aca22c6b080defed8e85b7bfec)
https://github.com/facebook/rocksdb/issues/1 rocksdb::DbStressListener::OnErrorRecoveryCompleted(rocksdb::Status) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_set.h:298 (db_stress+0x4112e5) (BuildId: b803e5aca22c6b080defed8e85b7bfec)
https://github.com/facebook/rocksdb/issues/2 rocksdb::EventHelpers::NotifyOnErrorRecoveryEnd(std::vector<std::shared_ptr<rocksdb::EventListener>, std::allocator<std::shared_ptr<rocksdb::EventListener>>> const&, rocksdb::Status const&, rocksdb::Status const&, rocksdb::InstrumentedMutex*) fbcode/internal_repo_rocksdb/repo/db/event_helpers.cc:239 (db_stress+0xa09d60) (BuildId: b803e5aca22c6b080defed8e85b7bfec)
Previous read of size 8 at 0x7b4c000004d0 by thread T131 (mutexes: write M1):
#0 rocksdb::FaultInjectionTestFS::MaybeInjectThreadLocalError(rocksdb::FaultInjectionIOType, rocksdb::IOOptions const&, rocksdb::FaultInjectionTestFS::ErrorOperation, rocksdb::Slice*, bool, char*, bool, bool*) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_tree.h:798 (db_stress+0xf7d0f3) (BuildId: b803e5aca22c6b080defed8e85b7bfec)
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12799
Test Plan: CI
Reviewed By: jowlyzhang
Differential Revision: D58917449
Pulled By: hx235
fbshipit-source-id: f24fc1acc2a7d91f9f285447a97ba41397f48dbd
|
||
|
Changyu Bi
|
e3f5125ff1 |
Fix "no new line at end of file" (#12806)
Summary: internal CI with -Wnewline-eof complains about this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12806 Reviewed By: jowlyzhang Differential Revision: D58969551 Pulled By: cbi42 fbshipit-source-id: f97bd23f82129bac8dfd97b0ff4dbf7d2ded95d4 |
||
|
Changyu Bi
|
748f74aca3 |
Update main branch for 9.4 release (#12802)
Summary:
Main branch cut at
|
||
|
Peter Dillinger
|
39455974cb |
Fix possible double-free on TruncatedRangeDelIterator (#12805)
Summary: Not sure where or how it happens, but using a recent CircleCI failure I got a reliable db_stress reproducer. Using std::unique_ptr appropriately for managing them has apparently (and unsurprisingly) fixed the problem without needing to know exactly where the problem was. Suggested follow-up: * Three or even four levels of pointers is very confusing to work with. Surely this part can be cleaned up to be simpler. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12805 Test Plan: Reproducer passes, plus ASAN test and crash test runs. I don't think it's worth the extra work to track down the details and create a careful unit test. ``` ./db_stress --WAL_size_limit_MB=1 --WAL_ttl_seconds=60 --acquire_snapshot_one_in=10000 --adaptive_readahead=1 --adm_policy=2 --advise_random_on_open=1 --allow_data_in_errors=True --allow_fallocate=1 --async_io=0 --auto_readahead_size=1 --avoid_flush_during_recovery=0 --avoid_flush_during_shutdown=1 --avoid_unnecessary_blocking_io=1 --backup_max_size=104857600 --backup_one_in=100000 --batch_protection_bytes_per_key=0 --bgerror_resume_retry_interval=1000000 --block_align=1 --block_protection_bytes_per_key=4 --block_size=16384 --bloom_before_level=2147483646 --bloom_bits=15 --bottommost_compression_type=none --bottommost_file_compaction_delay=3600 --bytes_per_sync=262144 --cache_index_and_filter_blocks=0 --cache_index_and_filter_blocks_with_high_priority=0 --cache_size=33554432 --cache_type=tiered_lru_cache --charge_compression_dictionary_building_buffer=0 --charge_file_metadata=1 --charge_filter_construction=0 --charge_table_reader=0 --check_multiget_consistency=1 --check_multiget_entity_consistency=1 --checkpoint_one_in=10000 --checksum_type=kxxHash --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000 --compaction_pri=0 --compaction_readahead_size=0 --compaction_ttl=0 --compress_format_version=2 --compressed_secondary_cache_ratio=0.2 --compressed_secondary_cache_size=0 --compression_checksum=0 --compression_max_dict_buffer_bytes=0 --compression_max_dict_bytes=0 --compression_parallel_threads=1 --compression_type=none --compression_use_zstd_dict_trainer=0 --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --daily_offpeak_time_utc= --data_block_index_type=0 --db=/dev/shm/rocksdb.gpxs/rocksdb_crashtest_blackbox --db_write_buffer_size=0 --default_temperature=kWarm --default_write_temperature=kCold --delete_obsolete_files_period_micros=21600000000 --delpercent=4 --delrangepercent=1 --destroy_db_initially=0 --detect_filter_construct_corruption=0 --disable_file_deletions_one_in=10000 --disable_manual_compaction_one_in=1000000 --disable_wal=0 --dump_malloc_stats=1 --enable_checksum_handoff=1 --enable_compaction_filter=0 --enable_custom_split_merge=0 --enable_do_not_compress_roles=0 --enable_index_compression=0 --enable_memtable_insert_with_hint_prefix_extractor=0 --enable_pipelined_write=1 --enable_sst_partitioner_factory=0 --enable_thread_tracking=1 --enable_write_thread_adaptive_yield=0 --error_recovery_with_no_fault_injection=0 --expected_values_dir=/dev/shm/rocksdb.gpxs/rocksdb_crashtest_expected --fail_if_options_file_error=0 --fifo_allow_compaction=0 --file_checksum_impl=none --fill_cache=1 --flush_one_in=1000000 --format_version=3 --get_all_column_family_metadata_one_in=1000000 --get_current_wal_file_one_in=0 --get_live_files_apis_one_in=10000 --get_properties_of_all_tables_one_in=100000 --get_property_one_in=100000 --get_sorted_wal_files_one_in=0 --hard_pending_compaction_bytes_limit=274877906944 --high_pri_pool_ratio=0 --index_block_restart_interval=4 --index_shortening=0 --index_type=0 --ingest_external_file_one_in=0 --initial_auto_readahead_size=16384 --inplace_update_support=0 --iterpercent=10 --key_len_percent_dist=1,30,69 --key_may_exist_one_in=100 --last_level_temperature=kHot --level_compaction_dynamic_level_bytes=0 --lock_wal_one_in=1000000 --log_file_time_to_roll=0 --log_readahead_size=0 --long_running_snapshots=1 --low_pri_pool_ratio=0 --lowest_used_cache_tier=2 --manifest_preallocation_size=5120 --manual_wal_flush_one_in=1000 --mark_for_compaction_one_file_in=10 --max_auto_readahead_size=16384 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=2500000 --max_key_len=3 --max_log_file_size=0 --max_manifest_file_size=1073741824 --max_sequential_skip_in_iterations=1 --max_total_wal_size=0 --max_write_batch_group_size_bytes=16 --max_write_buffer_number=3 --max_write_buffer_size_to_maintain=0 --memtable_insert_hint_per_batch=1 --memtable_max_range_deletions=100 --memtable_prefix_bloom_size_ratio=0 --memtable_protection_bytes_per_key=4 --memtable_whole_key_filtering=0 --memtablerep=skip_list --metadata_charge_policy=0 --metadata_read_fault_one_in=32 --metadata_write_fault_one_in=0 --min_write_buffer_number_to_merge=2 --mmap_read=1 --mock_direct_io=False --nooverwritepercent=1 --num_file_reads_for_auto_readahead=0 --open_files=100 --open_metadata_read_fault_one_in=0 --open_metadata_write_fault_one_in=8 --open_read_fault_one_in=0 --open_write_fault_one_in=16 --ops_per_thread=100000000 --optimize_filters_for_hits=1 --optimize_filters_for_memory=0 --optimize_multiget_for_io=1 --paranoid_file_checks=1 --partition_filters=0 --partition_pinning=1 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefix_size=-1 --prefixpercent=0 --prepopulate_block_cache=1 --preserve_internal_time_seconds=60 --progress_reports=0 --promote_l0_one_in=0 --read_amp_bytes_per_bit=0 --read_fault_one_in=32 --readahead_size=524288 --readpercent=50 --recycle_log_file_num=1 --reopen=0 --report_bg_io_stats=1 --reset_stats_one_in=10000 --sample_for_compression=5 --secondary_cache_fault_one_in=32 --secondary_cache_uri= --set_options_one_in=10000 --skip_stats_update_on_db_open=0 --snapshot_hold_ops=100000 --soft_pending_compaction_bytes_limit=68719476736 --sqfc_name=bar --sqfc_version=1 --sst_file_manager_bytes_per_sec=104857600 --sst_file_manager_bytes_per_truncate=0 --stats_dump_period_sec=0 --stats_history_buffer_size=1048576 --strict_bytes_per_sync=1 --subcompactions=3 --sync=0 --sync_fault_injection=1 --table_cache_numshardbits=0 --target_file_size_base=524288 --target_file_size_multiplier=2 --test_batches_snapshots=0 --test_cf_consistency=1 --top_level_index_pinning=1 --uncache_aggressiveness=5 --universal_max_read_amp=-1 --unpartitioned_pinning=2 --use_adaptive_mutex=0 --use_adaptive_mutex_lru=0 --use_attribute_group=1 --use_delta_encoding=1 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=0 --use_get_entity=0 --use_merge=0 --use_multi_cf_iterator=0 --use_multi_get_entity=0 --use_multiget=1 --use_put_entity_one_in=1 --use_sqfc_for_range_queries=1 --use_timed_put_one_in=0 --use_write_buffer_manager=0 --user_timestamp_size=0 --value_size_mult=32 --verification_only=0 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_compression=1 --verify_db_one_in=100000 --verify_file_checksums_one_in=0 --verify_iterator_with_expected_state_one_in=0 --verify_sst_unique_id_in_manifest=1 --wal_bytes_per_sync=0 --wal_compression=none --write_buffer_size=1048576 --write_dbid_to_manifest=1 --write_fault_one_in=0 --writepercent=35 ``` Reviewed By: cbi42 Differential Revision: D58958390 Pulled By: pdillinger fbshipit-source-id: 1271cfdcc3c574f78cd59f3c68148f7ed4a19c47 |
||
|
Yu Zhang
|
fa4ffc816c |
Fix race condition between event listener and error handler (#12803)
Summary: Fix a race for accessing `bg_error_` after mutex is released. We make some copies before releasing to avoid this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12803 Reviewed By: cbi42 Differential Revision: D58957557 Pulled By: jowlyzhang fbshipit-source-id: 3c7369a3b8c8707aebc0044ff98288c898c05cb8 |
||
|
Andrew Kryczka
|
13549817af |
Update pinned folly version (#12801)
Summary:
|
||
|
Richard Barnes
|
5c391c7713 |
Fix deprecated dynamic exception in infra_asic_fpga/validation/freya/bmtc/ssp/src/gcc-arm-none-eabi-9-2019-q4-major/arm-none-eabi/include/c++/9.2.1/tr1/special_function_util.h +5
Summary: LLVM has detected a violation of `-Wdeprecated-dynamic-exception-spec`. Dynamic exceptions were removed in C++17. This diff fixes the deprecated instance(s). See [Dynamic exception specification](https://en.cppreference.com/w/cpp/language/except_spec) and [noexcept specifier](https://en.cppreference.com/w/cpp/language/noexcept_spec). Reviewed By: dmm-fb Differential Revision: D58953050 fbshipit-source-id: 8559dc925ea5ed0a6dbf938aa02ee810f01047ae |
||
|
Hui Xiao
|
e90e9153d5 |
Calculate injected_error_count even when SharedState::ignore_read_error (#12800)
Summary:
**Context/Summary:**
`injected_error_count` is needed to verify read error injection. For example, when injected_error_count == 0, the read call should not return error.
|
||
|
Hui Xiao
|
9d64ca55b7 |
Proceed for new memtable on okay status (#12798)
Summary: **Context/Summary:** The relevant code logs info of newly created WAL and proceeds to "ConstructFragmentedRangeTombstones()" even when the previous step fails. This PR fixes it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12798 Test Plan: Existing tests Reviewed By: cbi42 Differential Revision: D58917246 Pulled By: hx235 fbshipit-source-id: f395210d91e50617195cb9a8047cf5d82db0c40e |
||
|
Andrew Kryczka
|
40944cbbdb |
Fix folly build (#12795)
Summary: - Updated pinned folly version to the latest - gcc/g++ 10 is required since https://github.com/facebook/folly/commit/2c1c617e9e so we had to modify the tests using gcc/g++ 7 - libsodium 1.0.17 is no longer downloadable from GitHub so I found it elsewhere. I will submit a PR for that upstream to folly - USE_FOLLY_LITE changes - added boost header dependency instead of commenting out the `#include`s since that approach stopped working - added "folly/lang/Exception.cpp" to the compilation Pull Request resolved: https://github.com/facebook/rocksdb/pull/12795 Reviewed By: hx235 Differential Revision: D58916693 Pulled By: ajkr fbshipit-source-id: b5f9bca2d929825846ac898b785972b071db62b1 |
||
|
Changyu Bi
|
b4a84efb4e |
Fix assertion failure in ConstructFragmentedRangeTombstones() (#12796)
Summary: the assertion `assert(!IsFragmentedRangeTombstonesConstructed(false));` assumes ConstructFragmentedRangeTombstones() is called only once for a memtable. This is not true since SwitchMemtable() can be called multiple times on the same live memtable, if a previous attempt fails. So remove the assertion in this PR and simplify relevant code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12796 Test Plan: the exact condition to trigger manifest write in SwitchMemtable() is complicated. Will monitor crash test to see if there's no more failure. Reviewed By: hx235 Differential Revision: D58913310 Pulled By: cbi42 fbshipit-source-id: 458bb9eebcf6743e9001186fcb757e4b50e8a5d2 |