mirror of
https://github.com/facebook/rocksdb.git
synced 2024-11-25 22:44:05 +00:00
3182beeffc
Summary: Background. One of the core risks of chosing HyperClockCache is ending up with degraded performance if estimated_entry_charge is very significantly wrong. Too low leads to under-utilized hash table, which wastes a bit of (tracked) memory and likely increases access times due to larger working set size (more TLB misses). Too high leads to fully populated hash table (at some limit with reasonable lookup performance) and not being able to cache as many objects as the memory limit would allow. In either case, performance degradation is graceful/continuous but can be quite significant. For example, cutting block size in half without updating estimated_entry_charge could lead to a large portion of configured block cache memory (up to roughly 1/3) going unused. Fix. This change adds a mechanism through which the DB periodically probes the block cache(s) for "problems" to report, and adds diagnostics to the HyperClockCache for bad estimated_entry_charge. The periodic probing is currently done with DumpStats / stats_dump_period_sec, and diagnostics reported to info_log (normally LOG file). Pull Request resolved: https://github.com/facebook/rocksdb/pull/10965 Test Plan: unit test included. Doesn't cover all the implemented subtleties of reporting, but ensures basics of when to report or not. Also manual testing with db_bench. Create db with ``` ./db_bench --benchmarks=fillrandom,flush --num=3000000 --disable_wal=1 ``` Use and check LOG file for HyperClockCache for various block sizes (used as estimated_entry_charge) ``` ./db_bench --use_existing_db --benchmarks=readrandom --num=3000000 --duration=20 --stats_dump_period_sec=8 --cache_type=hyper_clock_cache -block_size=XXXX ``` Seeing warnings / errors or not as expected. Reviewed By: anand1976 Differential Revision: D41406932 Pulled By: pdillinger fbshipit-source-id: 4ca56162b73017e4b9cec2cad74466f49c27a0a7
63 lines
2.7 KiB
C
63 lines
2.7 KiB
C
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
//
|
|
// Must not be included from any .h files to avoid polluting the namespace
|
|
// with macros.
|
|
|
|
#pragma once
|
|
|
|
// Helper macros that include information about file name and line number
|
|
#define ROCKS_LOG_STRINGIFY(x) #x
|
|
#define ROCKS_LOG_TOSTRING(x) ROCKS_LOG_STRINGIFY(x)
|
|
#define ROCKS_LOG_PREPEND_FILE_LINE(FMT) \
|
|
("[%s:" ROCKS_LOG_TOSTRING(__LINE__) "] " FMT)
|
|
|
|
inline const char* RocksLogShorterFileName(const char* file) {
|
|
// 18 is the length of "logging/logging.h".
|
|
// If the name of this file changed, please change this number, too.
|
|
return file + (sizeof(__FILE__) > 18 ? sizeof(__FILE__) - 18 : 0);
|
|
}
|
|
|
|
// Don't inclide file/line info in HEADER level
|
|
#define ROCKS_LOG_HEADER(LGR, FMT, ...) \
|
|
ROCKSDB_NAMESPACE::Log(InfoLogLevel::HEADER_LEVEL, LGR, FMT, ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_AT_LEVEL(LGR, LVL, FMT, ...) \
|
|
ROCKSDB_NAMESPACE::Log((LVL), (LGR), ROCKS_LOG_PREPEND_FILE_LINE(FMT), \
|
|
RocksLogShorterFileName(__FILE__), ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_DEBUG(LGR, FMT, ...) \
|
|
ROCKS_LOG_AT_LEVEL((LGR), InfoLogLevel::DEBUG_LEVEL, FMT, ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_INFO(LGR, FMT, ...) \
|
|
ROCKS_LOG_AT_LEVEL((LGR), InfoLogLevel::INFO_LEVEL, FMT, ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_WARN(LGR, FMT, ...) \
|
|
ROCKS_LOG_AT_LEVEL((LGR), InfoLogLevel::WARN_LEVEL, FMT, ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_ERROR(LGR, FMT, ...) \
|
|
ROCKS_LOG_AT_LEVEL((LGR), InfoLogLevel::ERROR_LEVEL, FMT, ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_FATAL(LGR, FMT, ...) \
|
|
ROCKS_LOG_AT_LEVEL((LGR), InfoLogLevel::FATAL_LEVEL, FMT, ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_BUFFER(LOG_BUF, FMT, ...) \
|
|
ROCKSDB_NAMESPACE::LogToBuffer(LOG_BUF, ROCKS_LOG_PREPEND_FILE_LINE(FMT), \
|
|
RocksLogShorterFileName(__FILE__), \
|
|
##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_BUFFER_MAX_SZ(LOG_BUF, MAX_LOG_SIZE, FMT, ...) \
|
|
ROCKSDB_NAMESPACE::LogToBuffer( \
|
|
LOG_BUF, MAX_LOG_SIZE, ROCKS_LOG_PREPEND_FILE_LINE(FMT), \
|
|
RocksLogShorterFileName(__FILE__), ##__VA_ARGS__)
|
|
|
|
#define ROCKS_LOG_DETAILS(LGR, FMT, ...) \
|
|
; // due to overhead by default skip such lines
|
|
// ROCKS_LOG_DEBUG(LGR, FMT, ##__VA_ARGS__)
|