rocksdb/db
Hui Xiao dcc6fc99f9 Fix StopWatch bug; Remove setting `record_read_stats` (#11474)
Summary:
**Context/Summary:**
- StopWatch enable stats even when `StatsLevel::kExceptTimers` is set. It's a harmless bug though since `reportTimeToHistogram()` will not report it anyway according to https://github.com/facebook/rocksdb/blob/main/include/rocksdb/statistics.h#L705
-  https://github.com/facebook/rocksdb/pull/11288 should have removed logics of setting `record_read_stats = !for_compaction` as we don't differentiate `RandomAccessFileReader`'s stats behavior based on compaction or not (instead we now report stats of different IO activities including compaction to different stats). Fixing this should report more compaction related file read micros that aren't reported previously due to `for_compaction==true`

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11474

Test Plan:
- DB bench pre vs post fix with small max_open_files

Setup command
`./db_ bench  -db=/dev/shm/testdb/ -statistics=true -benchmarks=fillseq -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=true -compression_type=none -bloom_bits=3`

Run command
`./db_bench --open_files=1 -use_existing_db=true -db=/dev/shm/testdb2/ -statistics=true -benchmarks=compactall -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=true -compression_type=none -bloom_bits=3`

Pre-fix
```
rocksdb.sst.read.micros P50 : 2.056175 P95 : 4.647739 P99 : 8.948475 P100 : 25.000000 COUNT : 4451 SUM : 12827
rocksdb.file.read.flush.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0
rocksdb.file.read.compaction.micros P50 : 2.057397 P95 : 4.625253 P99 : 8.749474 P100 : 25.000000 COUNT : 4382 SUM : 12608
rocksdb.file.read.db.open.micros P50 : 1.985294 P95 : 9.100000 P99 : 13.000000 P100 : 13.000000 COUNT : 69 SUM : 219
```

Post-fix (with a higher `rocksdb.file.read.compaction.micros` count)
```
rocksdb.sst.read.micros P50 : 1.858968 P95 : 3.653086 P99 : 5.968000 P100 : 21.000000 COUNT : 3548 SUM : 9119
rocksdb.file.read.flush.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0
rocksdb.file.read.compaction.micros P50 : 1.857027 P95 : 3.627614 P99 : 5.738621 P100 : 21.000000 COUNT : 3479 SUM : 8904
rocksdb.file.read.db.open.micros P50 : 2.000000 P95 : 6.733333 P99 : 11.000000 P100 : 11.000000 COUNT : 69 SUM : 215
```
- CI

Reviewed By: ajkr

Differential Revision: D46137221

Pulled By: hx235

fbshipit-source-id: e5b4ee7001af26f2ee0377bc6334f43b2a527388
2023-05-25 10:16:58 -07:00
..
blob Improve memory efficiency of many OptimisticTransactionDBs (#11439) 2023-05-24 11:57:15 -07:00
compaction Remove wait_unscheduled from waitForCompact internal API (#11443) 2023-05-17 18:13:50 -07:00
db_impl Add utils to use for handling user defined timestamp size record in WAL (#11451) 2023-05-22 14:28:58 -07:00
wide Add a new MultiGetEntity API (#11222) 2023-02-15 09:34:17 -08:00
arena_wrapped_db_iter.cc Ignore async_io ReadOption if FileSystem doesn't support it (#11296) 2023-03-17 14:57:09 -07:00
arena_wrapped_db_iter.h
builder.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
builder.h Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
c.cc Block fetch CPU time counters in perf context (#11342) 2023-04-15 11:09:44 -07:00
c_test.c Add in-transaction multi-get-for-update to the C interface (#11107) 2023-03-27 12:14:18 -07:00
column_family.cc remove unnecessary code in super version getter (#11452) 2023-05-23 12:16:24 -07:00
column_family.h New stat rocksdb.{cf|db}-write-stall-stats exposed in a structural way (#11300) 2023-03-18 09:51:58 -07:00
column_family_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
compact_files_test.cc Drain unnecessary levels when `level_compaction_dynamic_level_bytes=true` (#11340) 2023-04-06 11:20:43 -07:00
comparator_db_test.cc
convenience.cc Block per key-value checksum (#11287) 2023-04-25 12:08:23 -07:00
corruption_test.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
cuckoo_table_db_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_basic_test.cc Ensure VerifyFileChecksums reads don't exceed readahead_size (#11328) 2023-04-05 16:22:08 -07:00
db_block_cache_test.cc Add hash_seed to Caches (#11391) 2023-05-09 22:24:26 -07:00
db_bloom_filter_test.cc Much better stats for seeks and prefix filtering (#11460) 2023-05-19 15:25:49 -07:00
db_clip_test.cc Support Clip DB to KeyRange (#11379) 2023-05-18 13:25:01 -07:00
db_compaction_filter_test.cc Always allow L0->L1 trivial move during manual compaction (#11375) 2023-04-20 11:10:48 -07:00
db_compaction_test.cc Remove wait_unscheduled from waitForCompact internal API (#11443) 2023-05-17 18:13:50 -07:00
db_dynamic_level_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_encryption_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_filesnapshot.cc Fix bug of prematurely excluded CF in atomic flush contains unflushed data that should've been included in the atomic flush (#11148) 2023-03-14 16:53:20 -07:00
db_flush_test.cc Fix bug of prematurely excluded CF in atomic flush contains unflushed data that should've been included in the atomic flush (#11148) 2023-03-14 16:53:20 -07:00
db_info_dumper.cc
db_info_dumper.h
db_inplace_update_test.cc
db_io_failure_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_iter.cc DBIter::FindValueForCurrentKey: remove unused Status s (#11394) 2023-05-03 08:52:03 -07:00
db_iter.h Fix backward iteration issue when user defined timestamp is enabled in BlobDB (#11258) 2023-03-01 13:28:54 -08:00
db_iter_stress_test.cc
db_iter_test.cc
db_iterator_test.cc Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
db_kv_checksum_test.cc
db_log_iter_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_logical_block_size_cache_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_memtable_test.cc
db_merge_operand_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_merge_operator_test.cc Better support for merge operation with data block hash index (#11356) 2023-04-07 10:06:03 -07:00
db_options_test.cc Delete temp OPTIONS file on failure to write it (#11423) 2023-05-12 22:39:39 -07:00
db_properties_test.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
db_range_del_test.cc Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113) 2023-02-22 12:28:18 -08:00
db_rate_limiter_test.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
db_readonly_with_timestamp_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_secondary_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_sst_test.cc Remove wait_unscheduled from waitForCompact internal API (#11443) 2023-05-17 18:13:50 -07:00
db_statistics_test.cc Add block checksum mismatch ticker stat (#11438) 2023-05-12 18:16:11 -07:00
db_table_properties_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_tailing_iter_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_test.cc Compatibility step for separating BlockCache and GeneralCache APIs (#11450) 2023-05-18 20:40:19 -07:00
db_test2.cc Much better stats for seeks and prefix filtering (#11460) 2023-05-19 15:25:49 -07:00
db_test_util.cc Fix flaky test `DBTestUniversalManualCompactionOutputPathId.ManualCompactionOutputPathId` (#11412) 2023-05-03 11:12:20 -07:00
db_test_util.h Much better stats for seeks and prefix filtering (#11460) 2023-05-19 15:25:49 -07:00
db_universal_compaction_test.cc Fix flaky test `DBTestUniversalManualCompactionOutputPathId.ManualCompactionOutputPathId` (#11412) 2023-05-03 11:12:20 -07:00
db_wal_test.cc Delete empty WAL files on reopen (#11409) 2023-05-02 15:54:29 -07:00
db_with_timestamp_basic_test.cc Filter table files by timestamp: Get operator (#11332) 2023-04-06 15:39:38 -07:00
db_with_timestamp_compaction_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
db_with_timestamp_test_util.cc
db_with_timestamp_test_util.h
db_write_buffer_manager_test.cc Add `SetAllowStall()` (#11335) 2023-03-30 09:43:33 -07:00
db_write_test.cc Deflake DBWriteTest.LockWALInEffect (#11382) 2023-04-17 11:00:08 -07:00
dbformat.cc
dbformat.h Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113) 2023-02-22 12:28:18 -08:00
dbformat_test.cc
deletefile_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
error_handler.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
error_handler.h
error_handler_fs_test.cc Remove wait_unscheduled from waitForCompact internal API (#11443) 2023-05-17 18:13:50 -07:00
event_helpers.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
event_helpers.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
experimental.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
external_sst_file_basic_test.cc Clarify `SstFileWriter::DeleteRange()` ordering requirements (#11390) 2023-04-20 13:02:16 -07:00
external_sst_file_ingestion_job.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
external_sst_file_ingestion_job.h
external_sst_file_test.cc Remove wait_unscheduled from waitForCompact internal API (#11443) 2023-05-17 18:13:50 -07:00
fault_injection_test.cc
file_indexer.cc
file_indexer.h
file_indexer_test.cc
filename_test.cc
flush_job.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
flush_job.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
flush_job_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc Block per key-value checksum (#11287) 2023-04-25 12:08:23 -07:00
forward_iterator.h Ignore async_io ReadOption if FileSystem doesn't support it (#11296) 2023-03-17 14:57:09 -07:00
forward_iterator_bench.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
history_trimming_iterator.h Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113) 2023-02-22 12:28:18 -08:00
import_column_family_job.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
import_column_family_job.h Support range deletion tombstones in `CreateColumnFamilyWithImport` (#11252) 2023-03-13 11:06:59 -07:00
import_column_family_test.cc Fix the wrong calculation of largest_key in import_column_family_job (#11381) 2023-04-15 10:33:23 -07:00
internal_stats.cc Fix write stall stats dump format (#11445) 2023-05-15 11:47:17 -07:00
internal_stats.h Add hash_seed to Caches (#11391) 2023-05-09 22:24:26 -07:00
job_context.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
kv_checksum.h Block per key-value checksum (#11287) 2023-04-25 12:08:23 -07:00
listener_test.cc Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
log_format.h Add support in log writer and reader for a user-defined timestamp size record (#11433) 2023-05-11 17:26:19 -07:00
log_reader.cc Add support in log writer and reader for a user-defined timestamp size record (#11433) 2023-05-11 17:26:19 -07:00
log_reader.h Add support in log writer and reader for a user-defined timestamp size record (#11433) 2023-05-11 17:26:19 -07:00
log_test.cc Add support in log writer and reader for a user-defined timestamp size record (#11433) 2023-05-11 17:26:19 -07:00
log_writer.cc Add support in log writer and reader for a user-defined timestamp size record (#11433) 2023-05-11 17:26:19 -07:00
log_writer.h Add support in log writer and reader for a user-defined timestamp size record (#11433) 2023-05-11 17:26:19 -07:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
lookup_key.h
malloc_stats.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
malloc_stats.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
manual_compaction_test.cc
memtable.cc Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
memtable.h Improve memory efficiency of many OptimisticTransactionDBs (#11439) 2023-05-24 11:57:15 -07:00
memtable_list.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
memtable_list.h
memtable_list_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
merge_context.h
merge_helper.cc Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
merge_helper.h
merge_helper_test.cc
merge_operator.cc
merge_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
obsolete_files_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
options_file_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
output_validator.cc
output_validator.h
perf_context_test.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
periodic_task_scheduler.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
periodic_task_scheduler.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
periodic_task_scheduler_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
pinned_iterators_manager.h
plain_table_db_test.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
post_memtable_callback.h
pre_release_callback.h
prefix_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
range_del_aggregator.cc Improve documentation for MergingIterator (#11161) 2023-03-03 12:17:30 -08:00
range_del_aggregator.h Improve documentation for MergingIterator (#11161) 2023-03-03 12:17:30 -08:00
range_del_aggregator_bench.cc
range_del_aggregator_test.cc Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113) 2023-02-22 12:28:18 -08:00
range_tombstone_fragmenter.cc
range_tombstone_fragmenter.h Refactor AddRangeDels() + consider range tombstone during compaction file cutting (#11113) 2023-02-22 12:28:18 -08:00
range_tombstone_fragmenter_test.cc
read_callback.h
repair.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
repair_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
seqno_time_test.cc Remove wait_unscheduled from waitForCompact internal API (#11443) 2023-05-17 18:13:50 -07:00
seqno_to_time_mapping.cc
seqno_to_time_mapping.h
snapshot_checker.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
snapshot_impl.cc
snapshot_impl.h
table_cache.cc Fix StopWatch bug; Remove setting `record_read_stats` (#11474) 2023-05-25 10:16:58 -07:00
table_cache.h Fix StopWatch bug; Remove setting `record_read_stats` (#11474) 2023-05-25 10:16:58 -07:00
table_cache_sync_and_async.h Fix StopWatch bug; Remove setting `record_read_stats` (#11474) 2023-05-25 10:16:58 -07:00
table_properties_collector.cc
table_properties_collector.h
table_properties_collector_test.cc Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
transaction_log_impl.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
transaction_log_impl.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Fix StopWatch bug; Remove setting `record_read_stats` (#11474) 2023-05-25 10:16:58 -07:00
version_builder.h Block per key-value checksum (#11287) 2023-04-25 12:08:23 -07:00
version_builder_test.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
version_edit.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
version_edit.h Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
version_edit_handler.cc Block per key-value checksum (#11287) 2023-04-25 12:08:23 -07:00
version_edit_handler.h Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
version_edit_test.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
version_set.cc Fix StopWatch bug; Remove setting `record_read_stats` (#11474) 2023-05-25 10:16:58 -07:00
version_set.h Support Clip DB to KeyRange (#11379) 2023-05-18 13:25:01 -07:00
version_set_sync_and_async.h Block per key-value checksum (#11287) 2023-04-25 12:08:23 -07:00
version_set_test.cc Record and use the tail size to prefetch table tail (#11406) 2023-05-08 13:14:28 -07:00
version_util.h Group rocksdb.sst.read.micros stat by IOActivity flush and compaction (#11288) 2023-04-21 09:07:18 -07:00
wal_edit.cc
wal_edit.h
wal_edit_test.cc
wal_manager.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
wal_manager.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
wal_manager_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
write_batch.cc Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
write_batch_base.cc
write_batch_internal.h
write_batch_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
write_callback.h
write_callback_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
write_controller.cc
write_controller.h
write_controller_test.cc
write_stall_stats.cc Fix initialization-order-fiasco in write_stall_stats.cc (#11355) 2023-04-05 14:42:31 -07:00
write_stall_stats.h Fix initialization-order-fiasco in write_stall_stats.cc (#11355) 2023-04-05 14:42:31 -07:00
write_thread.cc Ensure LockWAL() stall cleared for UnlockWAL() return (#11172) 2023-02-03 12:08:37 -08:00
write_thread.h Ensure LockWAL() stall cleared for UnlockWAL() return (#11172) 2023-02-03 12:08:37 -08:00