Maysam Yabandeh 638d239507 Charge block cache for cache internal usage (#5797) ... Summary: For our default block cache, each additional entry has extra memory overhead. It include LRUHandle (72 bytes currently) and the cache key (two varint64, file id and offset). The usage is not negligible. For example for block_size=4k, the overhead accounts for an extra 2% memory usage for the cache. The patch charging the cache for the extra usage, reducing untracked memory usage outside block cache. The feature is enabled by default and can be disabled by passing kDontChargeCacheMetadata to the cache constructor. This PR builds up on https://github.com/facebook/rocksdb/issues/4258 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5797 Test Plan: - Existing tests are updated to either disable the feature when the test has too much dependency on the old way of accounting the usage or increasing the cache capacity to account for the additional charge of metadata. - The Usage tests in cache_test.cc are augmented to test the cache usage under kFullChargeCacheMetadata. Differential Revision: D17396833 Pulled By: maysamyabandeh fbshipit-source-id: 7684ccb9f8a40ca595e4f5efcdb03623afea0c6f		2019-09-16 15:26:21 -07:00
..
compaction	Refactor UniversalCompactionPicker code a little bit (#5639)	2019-09-16 10:51:11 -07:00
db_impl	Charge block cache for cache internal usage (#5797)	2019-09-16 15:26:21 -07:00
arena_wrapped_db_iter.cc	Refactor ArenaWrappedDBIter into separate files (#5801)	2019-09-13 13:50:43 -07:00
arena_wrapped_db_iter.h	Refactor ArenaWrappedDBIter into separate files (#5801)	2019-09-13 13:50:43 -07:00
builder.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
builder.h	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
c.cc	Add insert hints for each writebatch (#5728)	2019-09-12 17:15:18 -07:00
c_test.c	Add C binding for secondary instance (#5505)	2019-06-27 08:58:54 -07:00
column_family.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
column_family.h	Bump up memory order of ref counting of ColumnFamilyData (#5723)	2019-08-20 10:34:33 -07:00
column_family_test.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
compact_files_test.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
compacted_db_impl.cc	New API to get all merge operands for a Key (#5604)	2019-08-06 14:26:44 -07:00
compacted_db_impl.h	Use delete to disable automatic generated methods. (#5009)	2019-09-11 18:09:00 -07:00
comparator_db_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
convenience.cc	Do readahead in VerifyChecksum() (#5713)	2019-08-16 16:42:56 -07:00
corruption_test.cc	Do readahead in VerifyChecksum() (#5713)	2019-08-16 16:42:56 -07:00
cuckoo_table_db_test.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
db_basic_test.cc	Persistent globally unique DB ID in manifest (#5725)	2019-09-03 08:52:24 -07:00
db_blob_index_test.cc	Refactor ArenaWrappedDBIter into separate files (#5801)	2019-09-13 13:50:43 -07:00
db_block_cache_test.cc	Charge block cache for cache internal usage (#5797)	2019-09-16 15:26:21 -07:00
db_bloom_filter_test.cc	Unordered Writes (#5218)	2019-05-13 17:47:21 -07:00
db_compaction_filter_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
db_compaction_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
db_dynamic_level_test.cc	…
db_encryption_test.cc	Fix EncryptedEnv assert (#5735)	2019-09-05 17:21:42 -07:00
db_filesnapshot.cc	Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765)	2019-09-04 12:10:17 -07:00
db_flush_test.cc	Ref and unref cfd before and after calling WaitForFlushMemTables (#5513)	2019-07-01 14:12:02 -07:00
db_info_dumper.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
db_info_dumper.h	…
db_inplace_update_test.cc	…
db_io_failure_test.cc	…
db_iter.cc	Refactor ArenaWrappedDBIter into separate files (#5801)	2019-09-13 13:50:43 -07:00
db_iter.h	Refactor ArenaWrappedDBIter into separate files (#5801)	2019-09-13 13:50:43 -07:00
db_iter_stress_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
db_iter_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
db_iterator_test.cc	Charge block cache for cache internal usage (#5797)	2019-09-16 15:26:21 -07:00
db_log_iter_test.cc	Apply modernize-use-override (2nd iteration)	2019-02-14 14:41:36 -08:00
db_memtable_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc	New API to get all merge operands for a Key (#5604)	2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc	New API to get all merge operands for a Key (#5604)	2019-08-06 14:26:44 -07:00
db_options_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
db_properties_test.cc	Charge block cache for cache internal usage (#5797)	2019-09-16 15:26:21 -07:00
db_range_del_test.cc	Fix merging range tombstone covering put during flush/compaction (#5406)	2019-06-04 10:24:14 -07:00
db_sst_test.cc	Fix bugs in WAL trash file handling (#5520)	2019-07-06 21:07:32 -07:00
db_statistics_test.cc	Make statistics's stats_level change thread-safe (#5030)	2019-03-01 10:42:09 -08:00
db_table_properties_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc	…
db_test.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
db_test2.cc	Charge block cache for cache internal usage (#5797)	2019-09-16 15:26:21 -07:00
db_test_util.cc	exclude TEST_ENV_URI from rocksdb lite (#5686)	2019-08-10 19:15:05 -07:00
db_test_util.h	Support loading custom objects in unit tests (#5676)	2019-08-09 15:12:08 -07:00
db_universal_compaction_test.cc	Refactor UniversalCompactionPicker code a little bit (#5639)	2019-09-16 10:51:11 -07:00
db_wal_test.cc	Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765)	2019-09-04 12:10:17 -07:00
db_write_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
dbformat.cc	Avoid user key copying for Get/Put/Write with user-timestamp (#5502)	2019-07-25 15:27:39 -07:00
dbformat.h	Use delete to disable automatic generated methods. (#5009)	2019-09-11 18:09:00 -07:00
dbformat_test.cc	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
deletefile_test.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
error_handler.cc	Make format	2019-05-31 15:24:43 -07:00
error_handler.h	…
error_handler_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
event_helpers.cc	Fix wrong info log printing for num_range_deletions (#5617)	2019-07-23 19:38:16 -07:00
event_helpers.h	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
experimental.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc	Allow ingesting overlapping files (#5539)	2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
external_sst_file_ingestion_job.h	Allow ingesting overlapping files (#5539)	2019-09-13 14:49:47 -07:00
external_sst_file_test.cc	Fix a bug in file ingestion (#5760)	2019-08-30 18:29:07 -07:00
fault_injection_test.cc	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
file_indexer.cc	…
file_indexer.h	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
file_indexer_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
filename_test.cc	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
flush_job.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
flush_job.h	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
flush_job_test.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
flush_scheduler.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
flush_scheduler.h	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
forward_iterator.cc	Add more callers for table reader. (#5454)	2019-06-20 14:31:48 -07:00
forward_iterator.h	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
forward_iterator_bench.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
import_column_family_job.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
import_column_family_job.h	Export Import sst files (#5495)	2019-07-17 12:27:14 -07:00
import_column_family_test.cc	Fix tsan and valgrind failures in import_column_family_test	2019-07-19 13:25:36 -07:00
internal_stats.cc	Rename InternalDBStatsType enum names (#5779)	2019-09-06 17:31:10 -07:00
internal_stats.h	Rename InternalDBStatsType enum names (#5779)	2019-09-06 17:31:10 -07:00
job_context.h	WritePrepared: Fix visible key compacted out by compaction (#4883)	2019-01-15 21:34:38 -08:00
listener_test.cc	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
log_format.h	…
log_reader.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
log_reader.h	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
log_test.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
log_writer.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
log_writer.h	Use delete to disable automatic generated methods. (#5009)	2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc	…
logs_with_prep_tracker.h	…
lookup_key.h	Avoid user key copying for Get/Put/Write with user-timestamp (#5502)	2019-07-25 15:27:39 -07:00
malloc_stats.cc	Support jemalloc compiled with `--with-jemalloc-prefix` (#5521)	2019-07-02 12:07:01 -07:00
malloc_stats.h	…
manual_compaction_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
memtable.cc	Add insert hints for each writebatch (#5728)	2019-09-12 17:15:18 -07:00
memtable.h	Add insert hints for each writebatch (#5728)	2019-09-12 17:15:18 -07:00
memtable_list.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
memtable_list.h	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
memtable_list_test.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
merge_context.h	Introduce a new MultiGet batching implementation (#5011)	2019-04-11 14:28:26 -07:00
merge_helper.cc	Fix merging range tombstone covering put during flush/compaction (#5406)	2019-06-04 10:24:14 -07:00
merge_helper.h	Remove v1 RangeDelAggregator (#4778)	2018-12-17 17:33:46 -08:00
merge_helper_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
merge_operator.cc	…
merge_test.cc	Make format	2019-05-31 15:24:43 -07:00
obsolete_files_test.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
options_file_test.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
perf_context_test.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h	…
plain_table_db_test.cc	Fix Appveyor build due to signed/unsigned comparison	2019-09-10 14:34:37 -07:00
pre_release_callback.h	WritePrepared: reduce prepared_mutex_ overhead (#5420)	2019-06-10 11:53:31 -07:00
prefix_test.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
range_del_aggregator.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
range_del_aggregator.h	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
range_del_aggregator_bench.cc	Move some memory related files from util/ to memory/ (#5382)	2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc	simplify include directive involving inttypes (#5402)	2019-06-06 13:56:07 -07:00
range_tombstone_fragmenter.h	Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720)	2019-09-05 17:30:29 -07:00
range_tombstone_fragmenter_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
read_callback.h	WriteUnPrepared: improve read your own write functionality (#5573)	2019-07-23 08:08:19 -07:00
repair.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
repair_test.cc	Organizing rocksdb/db directory	2019-05-31 11:57:01 -07:00
snapshot_checker.h	WritePrepared: fix issue with snapshot released during compaction (#4858)	2019-01-16 09:55:32 -08:00
snapshot_impl.cc	…
snapshot_impl.h	Refresh snapshot list during long compactions (2nd attempt) (#5278)	2019-05-03 17:30:22 -07:00
table_cache.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
table_cache.h	Support row cache with batched MultiGet (#5706)	2019-08-28 16:11:56 -07:00
table_properties_collector.cc	Feature for sampling and reporting compressibility (#4842)	2019-03-18 12:15:34 -07:00
table_properties_collector.h	Feature for sampling and reporting compressibility (#4842)	2019-03-18 12:15:34 -07:00
table_properties_collector_test.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
transaction_log_impl.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
transaction_log_impl.h	reuse scratch buffer in transaction_log_reader (#5702)	2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
trim_history_scheduler.h	Refactor trimming logic for immutable memtables (#5022)	2019-08-23 13:55:34 -07:00
version_builder.cc	Lower the risk for users to run options.force_consistency_checks = true (#5744)	2019-08-29 14:07:37 -07:00
version_builder.h	Lower the risk for users to run options.force_consistency_checks = true (#5744)	2019-08-29 14:07:37 -07:00
version_builder_test.cc	Move some logging related files to logging/ (#5387)	2019-05-31 17:23:59 -07:00
version_edit.cc	Persistent globally unique DB ID in manifest (#5725)	2019-09-03 08:52:24 -07:00
version_edit.h	Persistent globally unique DB ID in manifest (#5725)	2019-09-03 08:52:24 -07:00
version_edit_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
version_set.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
version_set.h	Use delete to disable automatic generated methods. (#5009)	2019-09-11 18:09:00 -07:00
version_set_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
wal_manager.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
wal_manager.h	Adding DB::GetCurrentWalFile() API as a repliction/backup helper (#5765)	2019-09-04 12:10:17 -07:00
wal_manager_test.cc	Divide file_reader_writer.h and .cc (#5803)	2019-09-16 10:33:51 -07:00
write_batch.cc	Add insert hints for each writebatch (#5728)	2019-09-12 17:15:18 -07:00
write_batch_base.cc	…
write_batch_internal.h	Add insert hints for each writebatch (#5728)	2019-09-12 17:15:18 -07:00
write_batch_test.cc	upgrade gtest 1.7.0 => 1.8.1 for json result writing	2019-09-09 11:24:11 -07:00
write_callback.h	…
write_callback_test.cc	WritePrepared: reduce prepared_mutex_ overhead (#5420)	2019-06-10 11:53:31 -07:00
write_controller.cc	…
write_controller.h	…
write_controller_test.cc	Move test related files under util/ to test_util/ (#5377)	2019-05-30 11:25:51 -07:00
write_thread.cc	Option to make write group size configurable (#5759)	2019-09-11 18:28:33 -07:00
write_thread.h	Option to make write group size configurable (#5759)	2019-09-11 18:28:33 -07:00