mirror of https://github.com/facebook/rocksdb.git
d2daa10afc
Summary: Thanks ltamasi and ajkr for initial investigations on the test failure. Per the investigations, the following scenario is likely causing the test to fail. 1. Recovery is needed (could be any reason during crash test) 2. Trying to recover from the latest manifest fails (likely due to read error injection) 3. DB opens with recovery from the next manifest which is different from step 2. 4. Expected state is based on the manifest we tried and failed in step 2. 5. Two manifests used in step 2 and 3 are confirmed to have difference in LSM trees (Thanks ltamasi again for the finding). ``` 2023/10/05-11:24:18.942189 56341 [db/version_set.cc:6079] Trying to recover from manifest: /dev/shm/rocksdb_test/rocksdb_crashtest_blackbox/MANIFEST-007184 ... 2023/10/05-11:24:18.978007 56341 [db/version_set.cc:6079] Trying to recover from manifest: /dev/shm/rocksdb_test/rocksdb_crashtest_blackbox/MANIFEST-007180 ``` ``` [ltamasi@devbig1024.prn1 /tmp/x]$ ldb manifest_dump --hex --path=MANIFEST-007184_renamed_ > 2 [ltamasi@devbig1024.prn1 /tmp/x]$ ldb manifest_dump --hex --path=MANIFEST-007180_renamed_ > 1 [ltamasi@devbig1024.prn1 /tmp/x]$ diff 1 2 --- 1 2023-10-09 10:29:16.966215207 -0700 +++ 2 2023-10-09 10:29:11.984241645 -0700 @@ -13,7 +13,7 @@ 7174:3950254[1875617 .. 2203952]['000000000003415B000000000000012B000000000000007D' seq:1906214, type:1 .. '000000000003CA59000000000000012B000000000000005C' seq:2039838, type:1] 7175:88060[2074748 .. 2203892]['000000000003CA6300000000000000CF78787878787878' seq:2167539, type:2 .. '000000000003D08F000000000000012B0000000000000130' seq:2112478, type:0] --- level 6 --- version# 1 --- - 7057:3132633[0 .. 2046144]['0000000000000009000000000000000978' seq:0, type:1 .. '0000000000005F8B000000000000012B00000000000002AC' seq:0, type:1] + 7219:2135565[0 .. 2046144]['0000000000000009000000000000000978' seq:0, type:1 .. '0000000000005F8B000000000000012B00000000000002AC' seq:0, type:1] 7061:827724[0 .. 2046131]['0000000000005F95000000000000000778787878787878' seq:0, type:1 .. '000000000000784F000000000000012B0000000000000113' seq:0, type:1] 6763:1352[0 .. 0]['000000000000784F000000000000012B0000000000000129' seq:0, type:1 .. '000000000000784F000000000000012B0000000000000129' seq:0, type:1] 7173:4812291[0 .. 2203957]['000000000000784F000000000000012B0000000000000138' seq:0, type:1 .. '0000000000020FAE787878787878' seq:0, type:1] @@ -77,4 +77,4 @@ --- level 61 --- version# 1 --- --- level 62 --- version# 1 --- --- level 63 --- version# 1 --- -next_file_number 7182 last_sequence 2203963 prev_log_number 0 max_column_family 0 min_log_number_to_keep 7015 +next_file_number 7221 last_sequence 2203963 prev_log_number 0 max_column_family 0 min_log_number_to_keep 7015 ``` We have two options to fix this. Either skip verification against expected state or disable read injection when BE recovery is enabled. I chose to skip verification against expected state per discussion. (See comments in this PR) Please note that some linter changes were included in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/11938 Test Plan: ``` TEST_TMPDIR=/dev/shm/rocksdb make crash_test_with_best_efforts_recovery ``` Reviewed By: ltamasi Differential Revision: D50136341 Pulled By: jaykorean fbshipit-source-id: ac7434d592aebc148bfc3a4fcaa34936f136b95c |
||
|---|---|---|
| .. | ||
| advisor | ||
| block_cache_analyzer | ||
| dump | ||
| CMakeLists.txt | ||
| Dockerfile | ||
| analyze_txn_stress_test.sh | ||
| auto_sanity_test.sh | ||
| backup_db.sh | ||
| benchmark.sh | ||
| benchmark_ci.py | ||
| benchmark_compare.sh | ||
| benchmark_leveldb.sh | ||
| blob_dump.cc | ||
| check_all_python.py | ||
| check_format_compatible.sh | ||
| db_bench.cc | ||
| db_bench_tool.cc | ||
| db_bench_tool_test.cc | ||
| db_crashtest.py | ||
| db_repl_stress.cc | ||
| db_sanity_test.cc | ||
| dbench_monitor | ||
| generate_random_db.sh | ||
| ingest_external_sst.sh | ||
| io_tracer_parser.cc | ||
| io_tracer_parser_test.cc | ||
| io_tracer_parser_tool.cc | ||
| io_tracer_parser_tool.h | ||
| ldb.cc | ||
| ldb_cmd.cc | ||
| ldb_cmd_impl.h | ||
| ldb_cmd_test.cc | ||
| ldb_test.py | ||
| ldb_tool.cc | ||
| pflag | ||
| reduce_levels_test.cc | ||
| regression_test.sh | ||
| restore_db.sh | ||
| rocksdb_dump_test.sh | ||
| run_blob_bench.sh | ||
| run_flash_bench.sh | ||
| run_leveldb.sh | ||
| sample-dump.dmp | ||
| simulated_hybrid_file_system.cc | ||
| simulated_hybrid_file_system.h | ||
| sst_dump.cc | ||
| sst_dump_test.cc | ||
| sst_dump_tool.cc | ||
| trace_analyzer.cc | ||
| trace_analyzer_test.cc | ||
| trace_analyzer_tool.cc | ||
| trace_analyzer_tool.h | ||
| verify_random_db.sh | ||
| write_external_sst.sh | ||
| write_stress.cc | ||
| write_stress_runner.py | ||