dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-26 16:30:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db_stress_tool
History
Hui Xiao 9a034801ce Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 
Summary:
**Context/Summary:**
- Similar to https://github.com/facebook/rocksdb/pull/11288 but for user read such as `Get(), MultiGet(), DBIterator::XXX(), Verify(File)Checksum()`.
   - For this, I refactored some user-facing `MultiGet` calls in `TransactionBase` and various types of `DB` so that it does not call a user-facing `Get()` but `GetImpl()` for passing the `ReadOptions::io_activity` check (see PR conversation)
   - New user read stats breakdown are guarded by `kExceptDetailedTimers` since measurement shows they have 4-5% regression to the upstream/main.

- Misc
   - More refactoring: with https://github.com/facebook/rocksdb/pull/11288, we complete passing `ReadOptions/IOOptions` to FS level. So we can now replace the previously [added](https://github.com/facebook/rocksdb/pull/9424) `rate_limiter_priority` parameter in `RandomAccessFileReader`'s `Read/MultiRead/Prefetch()` with `IOOptions::rate_limiter_priority`
   - Also, `ReadAsync()` call time is measured in `SST_READ_MICRO` now

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11444

Test Plan:
- CI fake db crash/stress test
- Microbenchmarking

**Build** `make clean && ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make -jN db_basic_bench`
- google benchmark version: 604f6fd3f4
- db_basic_bench_base: upstream
- db_basic_bench_pr: db_basic_bench_base + this PR
- asyncread_db_basic_bench_base: upstream + [db basic bench patch for IteratorNext](https://github.com/facebook/rocksdb/compare/main...hx235:rocksdb:micro_bench_async_read)
- asyncread_db_basic_bench_pr: asyncread_db_basic_bench_base + this PR

**Test**

Get
```
TEST_TMPDIR=/dev/shm ./db_basic_bench_{null_stat|base|pr} --benchmark_filter=DBGet/comp_style:0/max_data:134217728/per_key_size:256/enable_statistics:1/negative_query:0/enable_filter:0/mmap:1/threads:1 --benchmark_repetitions=1000
```

Result
```
Coming soon
```

AsyncRead
```
TEST_TMPDIR=/dev/shm ./asyncread_db_basic_bench_{base|pr} --benchmark_filter=IteratorNext/comp_style:0/max_data:134217728/per_key_size:256/enable_statistics:1/async_io:1/include_detailed_timers:0 --benchmark_repetitions=1000 > syncread_db_basic_bench_{base|pr}.out
```

Result
```
Base:
1956,1956,1968,1977,1979,1986,1988,1988,1988,1990,1991,1991,1993,1993,1993,1993,1994,1996,1997,1997,1997,1998,1999,2001,2001,2002,2004,2007,2007,2008,

PR (2.3% regression, due to measuring `SST_READ_MICRO` that wasn't measured before):
1993,2014,2016,2022,2024,2027,2027,2028,2028,2030,2031,2031,2032,2032,2038,2039,2042,2044,2044,2047,2047,2047,2048,2049,2050,2052,2052,2052,2053,2053,
```

Reviewed By: ajkr

Differential Revision: D45918925

Pulled By: hx235

fbshipit-source-id: 58a54560d9ebeb3a59b6d807639692614dad058a
2023-08-08 17:26:50 -07:00
..
batched_ops_stress.cc Support parallel read and write/delete to same key in NonBatchedOpsStressTest (#11058) 2023-05-15 15:34:22 -07:00
cf_consistency_stress.cc Extend the stress test coverage of MultiGetEntity (#11336) 2023-03-29 20:35:15 -07:00
CMakeLists.txt Improve comment of ExpectedValue in db stress (#11456) 2023-05-18 09:44:15 -07:00
db_stress.cc Disable tiered storage + BlobDB stress test (#10699) 2022-09-19 15:39:31 -07:00
db_stress_common.cc Increase the stress test coverage of GetEntity (#11303) 2023-03-17 14:47:29 -07:00
db_stress_common.h Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
db_stress_compaction_filter.h Enable compaction filter for db_stress with user-defined timestamp (#10259) 2022-06-27 11:53:09 -07:00
db_stress_driver.cc Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
db_stress_driver.h fix shared state used after free (#11059) 2023-01-04 19:35:34 -08:00
db_stress_env_wrapper.h Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
db_stress_gflags.cc Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
db_stress_listener.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_stress_listener.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_stress_shared_state.cc
db_stress_shared_state.h Allow TryAgain in db_stress with optimistic txn, and refactoring (#11653) 2023-07-28 16:25:29 -07:00
db_stress_stat.cc
db_stress_stat.h
db_stress_table_properties_collector.h
db_stress_test_base.cc Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
db_stress_test_base.h Allow TryAgain in db_stress with optimistic txn, and refactoring (#11653) 2023-07-28 16:25:29 -07:00
db_stress_tool.cc Stress/Crash Test for OptimisticTransactionDB (#11513) 2023-06-17 16:27:37 -07:00
expected_state.cc Improve comment of ExpectedValue in db stress (#11456) 2023-05-18 09:44:15 -07:00
expected_state.h Improve comment of ExpectedValue in db stress (#11456) 2023-05-18 09:44:15 -07:00
expected_value.cc Improve comment of ExpectedValue in db stress (#11456) 2023-05-18 09:44:15 -07:00
expected_value.h Refactor WriteUnpreparedStressTest to be a unit test (#11424) 2023-05-22 12:31:52 -07:00
multi_ops_txns_stress.cc db_stress: Reinstate Transaction::Rollback() calls before destruction (#11656) 2023-07-30 17:30:01 -07:00
multi_ops_txns_stress.h Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
no_batched_ops_stress.cc Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00