mirror of
https://github.com/facebook/rocksdb.git
synced 2024-11-29 18:33:58 +00:00
311a544c2a
Summary: This change gathers and publishes statistics about the kinds of items in block cache. This is especially important for profiling relative usage of cache by index vs. filter vs. data blocks. It works by iterating over the cache during periodic stats dump (InternalStats, stats_dump_period_sec) or on demand when DB::Get(Map)Property(kBlockCacheEntryStats), except that for efficiency and sharing among column families, saved data from the last scan is used when the data is not considered too old. The new information can be seen in info LOG, for example: Block cache LRUCache@0x7fca62229330 capacity: 95.37 MB collections: 8 last_copies: 0 last_secs: 0.00178 secs_since: 0 Block cache entry stats(count,size,portion): DataBlock(7092,28.24 MB,29.6136%) FilterBlock(215,867.90 KB,0.888728%) FilterMetaBlock(2,5.31 KB,0.00544%) IndexBlock(217,180.11 KB,0.184432%) WriteBuffer(1,256.00 KB,0.262144%) Misc(1,0.00 KB,0%) And also through DB::GetProperty and GetMapProperty (here using ldb just for demonstration): $ ./ldb --db=/dev/shm/dbbench/ get_property rocksdb.block-cache-entry-stats rocksdb.block-cache-entry-stats.bytes.data-block: 0 rocksdb.block-cache-entry-stats.bytes.deprecated-filter-block: 0 rocksdb.block-cache-entry-stats.bytes.filter-block: 0 rocksdb.block-cache-entry-stats.bytes.filter-meta-block: 0 rocksdb.block-cache-entry-stats.bytes.index-block: 178992 rocksdb.block-cache-entry-stats.bytes.misc: 0 rocksdb.block-cache-entry-stats.bytes.other-block: 0 rocksdb.block-cache-entry-stats.bytes.write-buffer: 0 rocksdb.block-cache-entry-stats.capacity: 8388608 rocksdb.block-cache-entry-stats.count.data-block: 0 rocksdb.block-cache-entry-stats.count.deprecated-filter-block: 0 rocksdb.block-cache-entry-stats.count.filter-block: 0 rocksdb.block-cache-entry-stats.count.filter-meta-block: 0 rocksdb.block-cache-entry-stats.count.index-block: 215 rocksdb.block-cache-entry-stats.count.misc: 1 rocksdb.block-cache-entry-stats.count.other-block: 0 rocksdb.block-cache-entry-stats.count.write-buffer: 0 rocksdb.block-cache-entry-stats.id: LRUCache@0x7f3636661290 rocksdb.block-cache-entry-stats.percent.data-block: 0.000000 rocksdb.block-cache-entry-stats.percent.deprecated-filter-block: 0.000000 rocksdb.block-cache-entry-stats.percent.filter-block: 0.000000 rocksdb.block-cache-entry-stats.percent.filter-meta-block: 0.000000 rocksdb.block-cache-entry-stats.percent.index-block: 2.133751 rocksdb.block-cache-entry-stats.percent.misc: 0.000000 rocksdb.block-cache-entry-stats.percent.other-block: 0.000000 rocksdb.block-cache-entry-stats.percent.write-buffer: 0.000000 rocksdb.block-cache-entry-stats.secs_for_last_collection: 0.000052 rocksdb.block-cache-entry-stats.secs_since_last_collection: 0 Solution detail - We need some way to flag what kind of blocks each entry belongs to, preferably without changing the Cache API. One of the complications is that Cache is a general interface that could have other users that don't adhere to whichever convention we decide on for keys and values. Or we would pay for an extra field in the Handle that would only be used for this purpose. This change uses a back-door approach, the deleter, to indicate the "role" of a Cache entry (in addition to the value type, implicitly). This has the added benefit of ensuring proper code origin whenever we recognize a particular role for a cache entry; if the entry came from some other part of the code, it will use an unrecognized deleter, which we simply attribute to the "Misc" role. An internal API makes for simple instantiation and automatic registration of Cache deleters for a given value type and "role". Another internal API, CacheEntryStatsCollector, solves the problem of caching the results of a scan and sharing them, to ensure scans are neither excessive nor redundant so as not to harm Cache performance. Because code is added to BlocklikeTraits, it is pulled out of block_based_table_reader.cc into its own file. This is a reformulation of https://github.com/facebook/rocksdb/issues/8276, without the type checking option (could still be added), and with actual stat gathering. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8297 Test Plan: manual testing with db_bench, and a couple of basic unit tests Reviewed By: ltamasi Differential Revision: D28488721 Pulled By: pdillinger fbshipit-source-id: 472f524a9691b5afb107934be2d41d84f2b129fb
126 lines
3.3 KiB
C++
126 lines
3.3 KiB
C++
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
#pragma once
|
|
|
|
#include <cassert>
|
|
|
|
#include "rocksdb/cache.h"
|
|
#include "rocksdb/rocksdb_namespace.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
// Returns the cached value given a cache handle.
|
|
template <typename T>
|
|
T* GetFromCacheHandle(Cache* cache, Cache::Handle* handle) {
|
|
assert(cache);
|
|
assert(handle);
|
|
|
|
return static_cast<T*>(cache->Value(handle));
|
|
}
|
|
|
|
// Simple generic deleter for Cache (to be used with Cache::Insert).
|
|
template <typename T>
|
|
void DeleteCacheEntry(const Slice& /* key */, void* value) {
|
|
delete static_cast<T*>(value);
|
|
}
|
|
|
|
// Turns a T* into a Slice so it can be used as a key with Cache.
|
|
template <typename T>
|
|
Slice GetSlice(const T* t) {
|
|
return Slice(reinterpret_cast<const char*>(t), sizeof(T));
|
|
}
|
|
|
|
// Generic resource management object for cache handles that releases the handle
|
|
// when destroyed. Has unique ownership of the handle, so copying it is not
|
|
// allowed, while moving it transfers ownership.
|
|
template <typename T>
|
|
class CacheHandleGuard {
|
|
public:
|
|
CacheHandleGuard() = default;
|
|
|
|
CacheHandleGuard(Cache* cache, Cache::Handle* handle)
|
|
: cache_(cache),
|
|
handle_(handle),
|
|
value_(GetFromCacheHandle<T>(cache, handle)) {
|
|
assert(cache_ && handle_ && value_);
|
|
}
|
|
|
|
CacheHandleGuard(const CacheHandleGuard&) = delete;
|
|
CacheHandleGuard& operator=(const CacheHandleGuard&) = delete;
|
|
|
|
CacheHandleGuard(CacheHandleGuard&& rhs) noexcept
|
|
: cache_(rhs.cache_), handle_(rhs.handle_), value_(rhs.value_) {
|
|
assert((!cache_ && !handle_ && !value_) || (cache_ && handle_ && value_));
|
|
|
|
rhs.ResetFields();
|
|
}
|
|
|
|
CacheHandleGuard& operator=(CacheHandleGuard&& rhs) noexcept {
|
|
if (this == &rhs) {
|
|
return *this;
|
|
}
|
|
|
|
ReleaseHandle();
|
|
|
|
cache_ = rhs.cache_;
|
|
handle_ = rhs.handle_;
|
|
value_ = rhs.value_;
|
|
|
|
assert((!cache_ && !handle_ && !value_) || (cache_ && handle_ && value_));
|
|
|
|
rhs.ResetFields();
|
|
|
|
return *this;
|
|
}
|
|
|
|
~CacheHandleGuard() { ReleaseHandle(); }
|
|
|
|
bool IsEmpty() const { return !handle_; }
|
|
|
|
Cache* GetCache() const { return cache_; }
|
|
Cache::Handle* GetCacheHandle() const { return handle_; }
|
|
T* GetValue() const { return value_; }
|
|
|
|
void Reset() {
|
|
ReleaseHandle();
|
|
ResetFields();
|
|
}
|
|
|
|
private:
|
|
void ReleaseHandle() {
|
|
if (IsEmpty()) {
|
|
return;
|
|
}
|
|
|
|
assert(cache_);
|
|
cache_->Release(handle_);
|
|
}
|
|
|
|
void ResetFields() {
|
|
cache_ = nullptr;
|
|
handle_ = nullptr;
|
|
value_ = nullptr;
|
|
}
|
|
|
|
private:
|
|
Cache* cache_ = nullptr;
|
|
Cache::Handle* handle_ = nullptr;
|
|
T* value_ = nullptr;
|
|
};
|
|
|
|
// Build an aliasing shared_ptr that keeps `handle` in cache while there
|
|
// are references, but the pointer is to the value for that cache entry,
|
|
// which must be of type T. This is copyable, unlike CacheHandleGuard, but
|
|
// does not provide access to caching details.
|
|
template <typename T>
|
|
std::shared_ptr<T> MakeSharedCacheHandleGuard(Cache* cache,
|
|
Cache::Handle* handle) {
|
|
auto wrapper = std::make_shared<CacheHandleGuard<T>>(cache, handle);
|
|
return std::shared_ptr<T>(wrapper, static_cast<T*>(cache->Value(handle)));
|
|
}
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|