dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-27 11:43:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Peter Dillinger 1aac814578 Use optimized folly DistributedMutex in LRUCache when available (#10179) 
Summary:
folly DistributedMutex is faster than standard mutexes though
imposes some static obligations on usage. See
https://github.com/facebook/folly/blob/main/folly/synchronization/DistributedMutex.h
for details. Here we use this alternative for our Cache implementations
(especially LRUCache) for better locking performance, when RocksDB is
compiled with folly.

Also added information about which distributed mutex implementation is
being used to cache_bench output and to DB LOG.

Intended follow-up:
* Use DMutex in more places, perhaps improving API to support non-scoped
locking
* Fix linking with fbcode compiler (needs ROCKSDB_NO_FBCODE=1 currently)

Credit: Thanks Siying for reminding me about this line of work that was previously
left unfinished.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10179

Test Plan:
for correctness, existing tests. CircleCI config updated.
Also Meta-internal buck build updated.

For performance, ran simultaneous before & after cache_bench. Out of three
comparison runs, the middle improvement to ops/sec was +21%:

Baseline: USE_CLANG=1 DEBUG_LEVEL=0 make -j24 cache_bench (fbcode
compiler)

```
Complete in 20.201 s; Rough parallel ops/sec = 1584062
Thread ops/sec = 107176

Operation latency (ns):
Count: 32000000 Average: 9257.9421  StdDev: 122412.04
Min: 134  Median: 3623.0493  Max: 56918500
Percentiles: P50: 3623.05 P75: 10288.02 P99: 30219.35 P99.9: 683522.04 P99.99: 7302791.63
```

New: (add USE_FOLLY=1)

```
Complete in 16.674 s; Rough parallel ops/sec = 1919135  (+21%)
Thread ops/sec = 135487

Operation latency (ns):
Count: 32000000 Average: 7304.9294  StdDev: 108530.28
Min: 132  Median: 3777.6012  Max: 91030902
Percentiles: P50: 3777.60 P75: 10169.89 P99: 24504.51 P99.9: 59721.59 P99.99: 1861151.83
```

Reviewed By: anand1976

Differential Revision: D37182983

Pulled By: pdillinger

fbshipit-source-id: a17eb05f25b832b6a2c1356f5c657e831a5af8d1
2022-06-17 13:08:45 -07:00
..
blob Fix LITE build (#10106) 2022-06-02 23:42:41 -07:00
compaction Abort in dbg mode after logging (#10183) 2022-06-15 22:00:24 -07:00
db_impl Use optimized folly DistributedMutex in LRUCache when available (#10179) 2022-06-17 13:08:45 -07:00
wide Add wide column serialization primitives (#9915) 2022-06-03 20:54:48 -07:00
arena_wrapped_db_iter.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
arena_wrapped_db_iter.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
builder.cc Consolidate manual_compaction_paused_ check (#10070) 2022-06-06 18:32:26 -07:00
builder.h CompactionIterator sees consistent view of which keys are committed (#9830) 2022-04-14 11:11:04 -07:00
c.cc Add blob cache option in the column family options (#10155) 2022-06-14 14:19:26 -07:00
c_test.c Make it possible to enable blob files starting from a certain LSM tree level (#10077) 2022-06-02 20:04:33 -07:00
column_family.cc Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
column_family.h Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
column_family_test.cc Deflake column_family_test to avoid hang (#10060) 2022-05-30 12:31:46 -07:00
compact_files_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
comparator_db_test.cc Document design/specification bugs with auto_prefix_mode (#10144) 2022-06-13 11:08:50 -07:00
convenience.cc Specify largest_seqno in VerifyChecksum (#9919) 2022-05-02 10:22:08 -07:00
corruption_test.cc Persist the new MANIFEST after successfully syncing the new WAL during recovery (#9922) 2022-06-01 10:52:26 -07:00
cuckoo_table_db_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_basic_test.cc Update stats to help users estimate MultiGet async IO impact (#10182) 2022-06-16 12:12:43 -07:00
db_block_cache_test.cc Remove deprecated block-based filter (#10184) 2022-06-16 15:51:33 -07:00
db_bloom_filter_test.cc Fix a false negative merge conflict (#10192) 2022-06-16 21:14:10 -07:00
db_compaction_filter_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_compaction_test.cc Support specifying blob garbage collection parameters when CompactRange() (#10073) 2022-06-01 19:40:26 -07:00
db_dynamic_level_test.cc Remove deprecated API AdvancedColumnFamilyOptions::soft_rate_limit/hard_rate_limit (#9452) 2022-01-27 13:01:09 -08:00
db_encryption_test.cc
db_filesnapshot.cc Reduce risk of backup or checkpoint missing a WAL file (#10083) 2022-06-01 11:02:27 -07:00
db_flush_test.cc Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
db_info_dumper.cc Printing IO Error in DumpDBFileSummary (#9940) 2022-05-04 10:19:53 -07:00
db_info_dumper.h
db_inplace_update_test.cc fix a bug, c api, if enable inplace_update_support, and use create sn… (#9471) 2022-03-21 12:04:33 -07:00
db_io_failure_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_iter.cc Remove dead code (#9825) 2022-04-11 10:26:55 -07:00
db_iter.h Remove dead code (#9825) 2022-04-11 10:26:55 -07:00
db_iter_stress_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_iter_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_iterator_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_kv_checksum_test.cc Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
db_log_iter_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_logical_block_size_cache_test.cc Attempt to deflake DBLogicalBlockSizeCacheTest.CreateColumnFamilies (#9516) 2022-03-04 11:35:28 -08:00
db_memtable_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_merge_operand_test.cc Fix GetMergeOperands() heap-use-after-free on flushed memtable (#9805) 2022-04-05 12:26:36 -07:00
db_merge_operator_test.cc
db_options_test.cc Add blob cache option in the column family options (#10155) 2022-06-14 14:19:26 -07:00
db_properties_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_range_del_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_rate_limiter_test.cc Address comments for PR #9988 and #9996 (#10020) 2022-05-19 15:23:53 -07:00
db_readonly_with_timestamp_test.cc Add timestamp support to CompactedDBImpl (#10030) 2022-05-24 12:14:10 -07:00
db_secondary_test.cc Fail DB::Open() if logger cannot be created (#9984) 2022-05-27 07:23:31 -07:00
db_sst_test.cc Pass the size of blob files to SstFileManager during DB open (#10062) 2022-05-27 05:58:43 -07:00
db_statistics_test.cc
db_table_properties_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_tailing_iter_test.cc
db_test.cc Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
db_test2.cc Document design/specification bugs with auto_prefix_mode (#10144) 2022-06-13 11:08:50 -07:00
db_test_util.cc Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
db_test_util.h Explicitly closing all directory file descriptors (#10049) 2022-06-01 18:03:34 -07:00
db_universal_compaction_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
db_wal_test.cc Bypass tests instead of skipping to resolve internal failure (#10148) 2022-06-12 12:05:11 -07:00
db_with_timestamp_basic_test.cc Return try again when full_history_ts_low is higher than requested ts (#10126) 2022-06-10 08:21:08 -07:00
db_with_timestamp_compaction_test.cc Use the comparator from the sst file table properties in sst_dump_tool (#9491) 2022-02-08 12:15:35 -08:00
db_with_timestamp_test_util.cc Add timestamp support to DBImplReadOnly (#10004) 2022-05-19 18:39:41 -07:00
db_with_timestamp_test_util.h Add timestamp support to DBImplReadOnly (#10004) 2022-05-19 18:39:41 -07:00
db_write_buffer_manager_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
db_write_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
dbformat.cc Track per-SST user-defined timestamp information in MANIFEST (#9092) 2021-11-10 10:49:04 -08:00
dbformat.h Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
dbformat_test.cc Enable a few unit tests to use custom Env objects (#9087) 2021-11-08 11:05:59 -08:00
deletefile_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
error_handler.cc Expand auto recovery to background read errors (#9679) 2022-03-15 14:45:34 -07:00
error_handler.h Expand auto recovery to background read errors (#9679) 2022-03-15 14:45:34 -07:00
error_handler_fs_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
event_helpers.cc Fix a race condition in WAL tracking causing DB open failure (#9715) 2022-03-23 19:41:31 -07:00
event_helpers.h Add a listener callback for end of auto error recovery (#9244) 2021-12-08 14:30:57 -08:00
experimental.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
external_sst_file_basic_test.cc Fix unittest ExternalSSTFileBasicTest.StableSnapshotWhileLoggingToManifest (#10066) 2022-05-31 08:48:57 -07:00
external_sst_file_ingestion_job.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
external_sst_file_ingestion_job.h Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
external_sst_file_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
fault_injection_test.cc Fix a bug causing duplicate trailing entries in WritableFile (buffered IO) (#9236) 2021-12-13 09:00:36 -08:00
file_indexer.cc
file_indexer.h Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
file_indexer_test.cc
filename_test.cc fixing issue #8345 RocksDB does not work when using UNC network paths (#9384) 2022-03-30 15:55:31 -07:00
flush_job.cc Consolidate manual_compaction_paused_ check (#10070) 2022-06-06 18:32:26 -07:00
flush_job.h Set Write rate limiter priority dynamically and pass it to FS (#9988) 2022-05-18 00:41:41 -07:00
flush_job_test.cc Set Write rate limiter priority dynamically and pass it to FS (#9988) 2022-05-18 00:41:41 -07:00
flush_scheduler.cc
flush_scheduler.h
forward_iterator.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
forward_iterator.h Fast path for detecting unchanged prefix_extractor (#9407) 2022-01-21 11:37:46 -08:00
forward_iterator_bench.cc Remove using namespace (#9369) 2022-01-12 09:31:12 -08:00
history_trimming_iterator.h Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410) 2022-03-11 16:13:23 -08:00
import_column_family_job.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
import_column_family_job.h New stable, fixed-length cache keys (#9126) 2021-12-16 17:15:13 -08:00
import_column_family_test.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
internal_stats.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
internal_stats.h Expose the amount of garbage in live blob files as a dedicated DB property (#9835) 2022-04-13 13:36:30 -07:00
job_context.h CompactionIterator sees consistent view of which keys are committed (#9830) 2022-04-14 11:11:04 -07:00
kv_checksum.h fix compile errors in db/kv_checksum.h (#9173) 2021-11-16 10:20:50 -08:00
listener_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
log_format.h Add record to set WAL compression type if enabled (#9556) 2022-02-17 16:19:31 -08:00
log_reader.cc Support read rate-limiting in SequentialFileReader (#9973) 2022-05-24 10:28:57 -07:00
log_reader.h Fix bug for WalManager with compressed WAL (#10130) 2022-06-08 14:16:43 -07:00
log_test.cc Integrate WAL compression into log reader/writer. (#9642) 2022-03-09 15:49:53 -08:00
log_writer.cc Integrate WAL compression into log reader/writer. (#9642) 2022-03-09 15:49:53 -08:00
log_writer.h Integrate WAL compression into log reader/writer. (#9642) 2022-03-09 15:49:53 -08:00
logs_with_prep_tracker.cc
logs_with_prep_tracker.h
lookup_key.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
malloc_stats.cc
malloc_stats.h
manual_compaction_test.cc Remove using namespace (#9369) 2022-01-12 09:31:12 -08:00
memtable.cc Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
memtable.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00
memtable_list.cc Encode min_log_number_to_keep and delete_wals_before in one version edit (#9766) 2022-03-31 20:00:52 -07:00
memtable_list.h Fix various spelling errors still found in code (#9653) 2022-05-05 19:45:32 -07:00
memtable_list_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
merge_context.h
merge_helper.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_helper.h Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_helper_test.cc Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
merge_operator.cc
merge_test.cc Make the Env class Customizable (#9293) 2022-01-04 16:45:49 -08:00
obsolete_files_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
options_file_test.cc
output_validator.cc Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
output_validator.h Cleanup includes in dbformat.h (#8930) 2021-09-29 04:04:40 -07:00
perf_context_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
periodic_work_scheduler.cc Fix a timer crash caused by invalid memory management (#9656) 2022-03-12 11:45:56 -08:00
periodic_work_scheduler.h Fix a timer crash caused by invalid memory management (#9656) 2022-03-12 11:45:56 -08:00
periodic_work_scheduler_test.cc
pinned_iterators_manager.h
plain_table_db_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
post_memtable_callback.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00
pre_release_callback.h
prefix_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
range_del_aggregator.cc
range_del_aggregator.h
range_del_aggregator_bench.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
range_del_aggregator_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
range_tombstone_fragmenter.cc
range_tombstone_fragmenter.h
range_tombstone_fragmenter_test.cc Cleanup multiple implementations of VectorIterator (#8901) 2021-10-06 07:48:31 -07:00
read_callback.h
repair.cc Support read rate-limiting in SequentialFileReader (#9973) 2022-05-24 10:28:57 -07:00
repair_test.cc Fix failed VerifySstUniqueIds unittests (#10043) 2022-05-24 09:00:06 -07:00
snapshot_checker.h Use STATIC_AVOID_DESTRUCTION for static objects with non-trivial destructors (#9958) 2022-05-17 09:39:22 -07:00
snapshot_impl.cc
snapshot_impl.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00
table_cache.cc Fix crash due to MultiGet async IO and direct IO (#10024) 2022-05-20 12:38:21 -07:00
table_cache.h Multi file concurrency in MultiGet using coroutines and async IO (#9968) 2022-05-19 15:36:27 -07:00
table_cache_sync_and_async.h Multi file concurrency in MultiGet using coroutines and async IO (#9968) 2022-05-19 15:36:27 -07:00
table_properties_collector.cc
table_properties_collector.h Track each SST's timestamp information as user properties (#9093) 2021-11-19 11:37:06 -08:00
table_properties_collector_test.cc Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
transaction_log_impl.cc Support read rate-limiting in SequentialFileReader (#9973) 2022-05-24 10:28:57 -07:00
transaction_log_impl.h Add checks to GetUpdatesSince (#9459) 2022-04-14 17:12:16 -07:00
trim_history_scheduler.cc
trim_history_scheduler.h
version_builder.cc Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
version_builder.h Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
version_builder_test.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
version_edit.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
version_edit.h Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
version_edit_handler.cc Fix a race condition in WAL tracking causing DB open failure (#9715) 2022-03-23 19:41:31 -07:00
version_edit_handler.h
version_edit_test.cc Track SST unique id in MANIFEST and verify (#9990) 2022-05-19 11:04:21 -07:00
version_set.cc Update stats to help users estimate MultiGet async IO impact (#10182) 2022-06-16 12:12:43 -07:00
version_set.h Update stats to help users estimate MultiGet async IO impact (#10182) 2022-06-16 12:12:43 -07:00
version_set_sync_and_async.h Update stats to help users estimate MultiGet async IO impact (#10182) 2022-06-16 12:12:43 -07:00
version_set_test.cc Account memory of FileMetaData in global memory limit (#9924) 2022-06-14 13:06:40 -07:00
version_util.h Add manifest fix-up utility for file temperatures (#9683) 2022-03-18 16:35:51 -07:00
wal_edit.cc
wal_edit.h Use std::numeric_limits<> (#9954) 2022-05-05 13:08:21 -07:00
wal_edit_test.cc
wal_manager.cc Fix bug for WalManager with compressed WAL (#10130) 2022-06-08 14:16:43 -07:00
wal_manager.h Fix bug for WalManager with compressed WAL (#10130) 2022-06-08 14:16:43 -07:00
wal_manager_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
write_batch.cc Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
write_batch_base.cc
write_batch_internal.h Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
write_batch_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
write_callback.h
write_callback_test.cc Add WriteOptions::protection_bytes_per_key (#10037) 2022-06-16 23:10:07 -07:00
write_controller.cc
write_controller.h Set Write rate limiter priority dynamically and pass it to FS (#9988) 2022-05-18 00:41:41 -07:00
write_controller_test.cc
write_thread.cc Verify write batch checksum before WAL (#10114) 2022-06-15 13:43:58 -07:00
write_thread.h Snapshots with user-specified timestamps (#9879) 2022-06-10 16:07:03 -07:00