mirror of
https://github.com/facebook/rocksdb.git
synced 2024-11-25 14:31:35 +00:00
8515bd50c9
Summary: Added rate limiter and read rate-limiting support to SequentialFileReader. I've updated call sites to SequentialFileReader::Read with appropriate IO priority (or left a TODO and specified IO_TOTAL for now). The PR is separated into four commits: the first one added the rate-limiting support, but with some fixes in the unit test since the number of request bytes from rate limiter in SequentialFileReader are not accurate (there is overcharge at EOF). The second commit fixed this by allowing SequentialFileReader to check file size and determine how many bytes are left in the file to read. The third commit added benchmark related code. The fourth commit moved the logic of using file size to avoid overcharging the rate limiter into backup engine (the main user of SequentialFileReader). Pull Request resolved: https://github.com/facebook/rocksdb/pull/9973 Test Plan: - `make check`, backup_engine_test covers usage of SequentialFileReader with rate limiter. - Run db_bench to check if rate limiting is throttling as expected: Verified that reads and writes are together throttled at 2MB/s, and at 0.2MB chunks that are 100ms apart. - Set up: `./db_bench --benchmarks=fillrandom -db=/dev/shm/test_rocksdb` - Benchmark: ``` strace -ttfe read,write ./db_bench --benchmarks=backup -db=/dev/shm/test_rocksdb --backup_rate_limit=2097152 --use_existing_db strace -ttfe read,write ./db_bench --benchmarks=restore -db=/dev/shm/test_rocksdb --restore_rate_limit=2097152 --use_existing_db ``` - db bench on backup and restore to ensure no performance regression. - backup (avg over 50 runs): pre-change: 1.90443e+06 micros/op; post-change: 1.8993e+06 micros/op (improve by 0.2%) - restore (avg over 50 runs): pre-change: 1.79105e+06 micros/op; post-change: 1.78192e+06 micros/op (improve by 0.5%) ``` # Set up ./db_bench --benchmarks=fillrandom -db=/tmp/test_rocksdb -num=10000000 # benchmark TEST_TMPDIR=/tmp/test_rocksdb NUM_RUN=50 for ((j=0;j<$NUM_RUN;j++)) do ./db_bench -db=$TEST_TMPDIR -num=10000000 -benchmarks=backup -use_existing_db | egrep 'backup' # Restore #./db_bench -db=$TEST_TMPDIR -num=10000000 -benchmarks=restore -use_existing_db done > rate_limit.txt && awk -v NUM_RUN=$NUM_RUN '{sum+=$3;sum_sqrt+=$3^2}END{print sum/NUM_RUN, sqrt(sum_sqrt/NUM_RUN-(sum/NUM_RUN)^2)}' rate_limit.txt >> rate_limit_2.txt ``` Reviewed By: hx235 Differential Revision: D36327418 Pulled By: cbi42 fbshipit-source-id: e75d4307cff815945482df5ba630c1e88d064691
61 lines
2.4 KiB
C++
61 lines
2.4 KiB
C++
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
#pragma once
|
|
#include <array>
|
|
|
|
#include "file/sequence_file_reader.h"
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
// A wrapper on top of Env::SequentialFile for reading text lines from a file.
|
|
// Lines are delimited by '\n'. The last line may or may not include a
|
|
// trailing newline. Uses SequentialFileReader internally.
|
|
class LineFileReader {
|
|
private:
|
|
std::array<char, 8192> buf_;
|
|
SequentialFileReader sfr_;
|
|
IOStatus io_status_;
|
|
const char* buf_begin_ = buf_.data();
|
|
const char* buf_end_ = buf_.data();
|
|
size_t line_number_ = 0;
|
|
bool at_eof_ = false;
|
|
|
|
public:
|
|
// See SequentialFileReader constructors
|
|
template <typename... Args>
|
|
explicit LineFileReader(Args&&... args)
|
|
: sfr_(std::forward<Args&&>(args)...) {}
|
|
|
|
static IOStatus Create(const std::shared_ptr<FileSystem>& fs,
|
|
const std::string& fname, const FileOptions& file_opts,
|
|
std::unique_ptr<LineFileReader>* reader,
|
|
IODebugContext* dbg, RateLimiter* rate_limiter);
|
|
|
|
LineFileReader(const LineFileReader&) = delete;
|
|
LineFileReader& operator=(const LineFileReader&) = delete;
|
|
|
|
// Reads another line from the file, returning true on success and saving
|
|
// the line to `out`, without delimiter, or returning false on failure. You
|
|
// must check GetStatus() to determine whether the failure was just
|
|
// end-of-file (OK status) or an I/O error (another status).
|
|
// The internal rate limiter will be charged at the specified priority.
|
|
bool ReadLine(std::string* out, Env::IOPriority rate_limiter_priority);
|
|
|
|
// Returns the number of the line most recently returned from ReadLine.
|
|
// Return value is unspecified if ReadLine has returned false due to
|
|
// I/O error. After ReadLine returns false due to end-of-file, return
|
|
// value is the last returned line number, or equivalently the total
|
|
// number of lines returned.
|
|
size_t GetLineNumber() const { return line_number_; }
|
|
|
|
// Returns any error encountered during read. The error is considered
|
|
// permanent and no retry or recovery is attempted with the same
|
|
// LineFileReader.
|
|
const IOStatus& GetStatus() const { return io_status_; }
|
|
};
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|