2016-02-09 23:12:00 +00:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
2017-07-15 23:03:42 +00:00
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
//
|
2016-05-06 23:09:09 +00:00
|
|
|
#include "rocksdb/utilities/ldb_cmd.h"
|
2020-07-09 21:33:42 +00:00
|
|
|
|
2022-03-18 23:35:51 +00:00
|
|
|
#include <cinttypes>
|
|
|
|
|
|
|
|
#include "db/db_test_util.h"
|
2020-02-10 23:42:46 +00:00
|
|
|
#include "db/version_edit.h"
|
|
|
|
#include "db/version_set.h"
|
Introduce a new storage specific Env API (#5761)
Summary:
The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc.
This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO.
The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before.
This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection.
The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761
Differential Revision: D18868376
Pulled By: anand1976
fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f
2019-12-13 22:47:08 +00:00
|
|
|
#include "env/composite_env_wrapper.h"
|
2020-02-10 23:42:46 +00:00
|
|
|
#include "file/filename.h"
|
2019-10-09 02:17:39 +00:00
|
|
|
#include "port/stack_trace.h"
|
2022-03-18 23:35:51 +00:00
|
|
|
#include "rocksdb/advanced_options.h"
|
2023-12-21 02:04:08 +00:00
|
|
|
#include "rocksdb/comparator.h"
|
2021-06-15 10:42:52 +00:00
|
|
|
#include "rocksdb/convenience.h"
|
2022-03-18 23:35:51 +00:00
|
|
|
#include "rocksdb/db.h"
|
2020-02-10 23:42:46 +00:00
|
|
|
#include "rocksdb/file_checksum.h"
|
2022-03-18 23:35:51 +00:00
|
|
|
#include "rocksdb/file_system.h"
|
2021-01-12 04:54:58 +00:00
|
|
|
#include "rocksdb/utilities/options_util.h"
|
2019-08-15 23:59:42 +00:00
|
|
|
#include "test_util/sync_point.h"
|
2019-05-30 18:21:38 +00:00
|
|
|
#include "test_util/testharness.h"
|
2020-02-10 23:42:46 +00:00
|
|
|
#include "test_util/testutil.h"
|
|
|
|
#include "util/file_checksum_helper.h"
|
2020-07-09 21:33:42 +00:00
|
|
|
#include "util/random.h"
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
|
2022-10-25 21:29:41 +00:00
|
|
|
using std::map;
|
2016-05-06 23:09:09 +00:00
|
|
|
using std::string;
|
|
|
|
using std::vector;
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
namespace ROCKSDB_NAMESPACE {
|
2019-01-03 19:11:09 +00:00
|
|
|
|
2019-10-09 02:17:39 +00:00
|
|
|
class LdbCmdTest : public testing::Test {
|
|
|
|
public:
|
|
|
|
LdbCmdTest() : testing::Test() {}
|
|
|
|
|
|
|
|
Env* TryLoadCustomOrDefaultEnv() {
|
|
|
|
Env* env = Env::Default();
|
2021-06-15 10:42:52 +00:00
|
|
|
EXPECT_OK(test::CreateEnvFromSystem(ConfigOptions(), &env, &env_guard_));
|
2019-10-09 02:17:39 +00:00
|
|
|
return env;
|
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
|
|
|
std::shared_ptr<Env> env_guard_;
|
|
|
|
};
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
|
2020-06-09 17:01:12 +00:00
|
|
|
TEST_F(LdbCmdTest, HelpAndVersion) {
|
|
|
|
Options o;
|
|
|
|
o.env = TryLoadCustomOrDefaultEnv();
|
|
|
|
LDBOptions lo;
|
|
|
|
static const char* help[] = {"./ldb", "--help"};
|
|
|
|
ASSERT_EQ(0, LDBCommandRunner::RunCommand(2, help, o, lo, nullptr));
|
|
|
|
static const char* version[] = {"./ldb", "--version"};
|
|
|
|
ASSERT_EQ(0, LDBCommandRunner::RunCommand(2, version, o, lo, nullptr));
|
|
|
|
static const char* bad[] = {"./ldb", "--not_an_option"};
|
|
|
|
ASSERT_NE(0, LDBCommandRunner::RunCommand(2, bad, o, lo, nullptr));
|
|
|
|
}
|
|
|
|
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
TEST_F(LdbCmdTest, HexToString) {
|
|
|
|
// map input to expected outputs.
|
2016-03-30 04:25:12 +00:00
|
|
|
// odd number of "hex" half bytes doesn't make sense
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
map<string, vector<int>> inputMap = {
|
2016-03-30 04:25:12 +00:00
|
|
|
{"0x07", {7}}, {"0x5050", {80, 80}}, {"0xFF", {-1}},
|
|
|
|
{"0x1234", {18, 52}}, {"0xaaAbAC", {-86, -85, -84}}, {"0x1203", {18, 3}},
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
for (const auto& inPair : inputMap) {
|
2020-02-20 20:07:53 +00:00
|
|
|
auto actual = ROCKSDB_NAMESPACE::LDBCommand::HexToString(inPair.first);
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
auto expected = inPair.second;
|
|
|
|
for (unsigned int i = 0; i < actual.length(); i++) {
|
2022-10-25 21:29:41 +00:00
|
|
|
EXPECT_EQ(expected[i], static_cast<int>((signed char)actual[i]));
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
}
|
2020-02-20 20:07:53 +00:00
|
|
|
auto reverse = ROCKSDB_NAMESPACE::LDBCommand::StringToHex(actual);
|
2016-04-12 02:21:00 +00:00
|
|
|
EXPECT_STRCASEEQ(inPair.first.c_str(), reverse.c_str());
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
TEST_F(LdbCmdTest, HexToStringBadInputs) {
|
|
|
|
const vector<string> badInputs = {
|
2016-03-30 04:25:12 +00:00
|
|
|
"0xZZ", "123", "0xx5", "0x111G", "0x123", "Ox12", "0xT", "0x1Q1",
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
};
|
2020-06-24 23:20:55 +00:00
|
|
|
for (const auto& badInput : badInputs) {
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
try {
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::LDBCommand::HexToString(badInput);
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
std::cerr << "Should fail on bad hex value: " << badInput << "\n";
|
|
|
|
FAIL();
|
|
|
|
} catch (...) {
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-01-03 19:11:09 +00:00
|
|
|
TEST_F(LdbCmdTest, MemEnv) {
|
2019-10-09 02:17:39 +00:00
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
2019-01-03 19:11:09 +00:00
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
2020-06-25 19:07:47 +00:00
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2019-01-03 19:11:09 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
for (int i = 0; i < 100; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
|
|
|
ASSERT_OK(db->Put(wopts, buf, buf));
|
|
|
|
}
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
|
|
|
|
delete db;
|
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "dump_live_files";
|
|
|
|
char* argv[] = {arg1, arg2, arg3};
|
|
|
|
|
2019-08-15 23:59:42 +00:00
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(3, argv, opts, LDBOptions(), nullptr));
|
2019-01-03 19:11:09 +00:00
|
|
|
}
|
|
|
|
|
2020-02-10 23:42:46 +00:00
|
|
|
class FileChecksumTestHelper {
|
|
|
|
private:
|
|
|
|
Options options_;
|
|
|
|
DB* db_;
|
|
|
|
std::string dbname_;
|
|
|
|
|
|
|
|
Status VerifyChecksum(LiveFileMetaData& file_meta) {
|
|
|
|
std::string cur_checksum;
|
|
|
|
std::string checksum_func_name;
|
|
|
|
|
|
|
|
Status s;
|
|
|
|
EnvOptions soptions;
|
|
|
|
std::unique_ptr<SequentialFile> file_reader;
|
|
|
|
std::string file_path = dbname_ + "/" + file_meta.name;
|
|
|
|
s = options_.env->NewSequentialFile(file_path, &file_reader, soptions);
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
std::unique_ptr<char[]> scratch(new char[2048]);
|
|
|
|
Slice result;
|
2020-03-29 22:57:02 +00:00
|
|
|
FileChecksumGenFactory* file_checksum_gen_factory =
|
|
|
|
options_.file_checksum_gen_factory.get();
|
|
|
|
if (file_checksum_gen_factory == nullptr) {
|
2020-02-10 23:42:46 +00:00
|
|
|
cur_checksum = kUnknownFileChecksum;
|
|
|
|
checksum_func_name = kUnknownFileChecksumFuncName;
|
|
|
|
} else {
|
2020-03-29 22:57:02 +00:00
|
|
|
FileChecksumGenContext gen_context;
|
|
|
|
gen_context.file_name = file_meta.name;
|
|
|
|
std::unique_ptr<FileChecksumGenerator> file_checksum_gen =
|
|
|
|
file_checksum_gen_factory->CreateFileChecksumGenerator(gen_context);
|
|
|
|
checksum_func_name = file_checksum_gen->Name();
|
2020-02-10 23:42:46 +00:00
|
|
|
s = file_reader->Read(2048, &result, scratch.get());
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
while (result.size() != 0) {
|
2020-03-29 22:57:02 +00:00
|
|
|
file_checksum_gen->Update(scratch.get(), result.size());
|
2020-02-10 23:42:46 +00:00
|
|
|
s = file_reader->Read(2048, &result, scratch.get());
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
}
|
2020-03-29 22:57:02 +00:00
|
|
|
file_checksum_gen->Finalize();
|
|
|
|
cur_checksum = file_checksum_gen->GetChecksum();
|
2020-02-10 23:42:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
std::string stored_checksum = file_meta.file_checksum;
|
|
|
|
std::string stored_checksum_func_name = file_meta.file_checksum_func_name;
|
|
|
|
if ((cur_checksum != stored_checksum) ||
|
|
|
|
(checksum_func_name != stored_checksum_func_name)) {
|
|
|
|
return Status::Corruption(
|
|
|
|
"Checksum does not match! The file: " + file_meta.name +
|
|
|
|
", checksum name: " + stored_checksum_func_name + " and checksum " +
|
|
|
|
stored_checksum + ". However, expected checksum name: " +
|
|
|
|
checksum_func_name + " and checksum " + cur_checksum);
|
|
|
|
}
|
|
|
|
return Status::OK();
|
|
|
|
}
|
|
|
|
|
|
|
|
public:
|
|
|
|
FileChecksumTestHelper(Options& options, DB* db, std::string db_name)
|
|
|
|
: options_(options), db_(db), dbname_(db_name) {}
|
2023-12-01 19:15:17 +00:00
|
|
|
~FileChecksumTestHelper() = default;
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
// Verify the checksum information in Manifest.
|
|
|
|
Status VerifyChecksumInManifest(
|
|
|
|
const std::vector<LiveFileMetaData>& live_files) {
|
|
|
|
// Step 1: verify if the dbname_ is correct
|
2022-01-27 19:34:20 +00:00
|
|
|
if (dbname_.back() != '/') {
|
2020-02-10 23:42:46 +00:00
|
|
|
dbname_.append("/");
|
|
|
|
}
|
|
|
|
|
|
|
|
// Step 2, get the the checksum information by recovering the VersionSet
|
|
|
|
// from Manifest.
|
|
|
|
std::unique_ptr<FileChecksumList> checksum_list(NewFileChecksumList());
|
|
|
|
EnvOptions sopt;
|
|
|
|
std::shared_ptr<Cache> tc(NewLRUCache(options_.max_open_files - 10,
|
|
|
|
options_.table_cache_numshardbits));
|
|
|
|
options_.db_paths.emplace_back(dbname_, 0);
|
|
|
|
options_.num_levels = 64;
|
|
|
|
WriteController wc(options_.delayed_write_rate);
|
|
|
|
WriteBufferManager wb(options_.db_write_buffer_size);
|
|
|
|
ImmutableDBOptions immutable_db_options(options_);
|
|
|
|
VersionSet versions(dbname_, &immutable_db_options, sopt, tc.get(), &wb,
|
2023-10-27 22:56:48 +00:00
|
|
|
&wc, nullptr, nullptr, "", "",
|
2023-12-29 02:25:29 +00:00
|
|
|
options_.daily_offpeak_time_utc, nullptr,
|
|
|
|
/*read_only=*/false);
|
2020-02-10 23:42:46 +00:00
|
|
|
std::vector<std::string> cf_name_list;
|
|
|
|
Status s;
|
|
|
|
s = versions.ListColumnFamilies(&cf_name_list, dbname_,
|
2020-03-24 04:50:42 +00:00
|
|
|
immutable_db_options.fs.get());
|
2020-02-10 23:42:46 +00:00
|
|
|
if (s.ok()) {
|
|
|
|
std::vector<ColumnFamilyDescriptor> cf_list;
|
|
|
|
for (const auto& name : cf_name_list) {
|
|
|
|
fprintf(stdout, "cf_name: %s", name.c_str());
|
|
|
|
cf_list.emplace_back(name, ColumnFamilyOptions(options_));
|
|
|
|
}
|
|
|
|
s = versions.Recover(cf_list, true);
|
|
|
|
}
|
|
|
|
if (s.ok()) {
|
|
|
|
s = versions.GetLiveFilesChecksumInfo(checksum_list.get());
|
|
|
|
}
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Step 3 verify the checksum
|
|
|
|
if (live_files.size() != checksum_list->size()) {
|
|
|
|
return Status::Corruption("The number of files does not match!");
|
|
|
|
}
|
|
|
|
for (size_t i = 0; i < live_files.size(); i++) {
|
2023-12-01 19:15:17 +00:00
|
|
|
std::string stored_checksum;
|
|
|
|
std::string stored_func_name;
|
2020-02-10 23:42:46 +00:00
|
|
|
s = checksum_list->SearchOneFileChecksum(
|
|
|
|
live_files[i].file_number, &stored_checksum, &stored_func_name);
|
|
|
|
if (s.IsNotFound()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
if (live_files[i].file_checksum != stored_checksum ||
|
|
|
|
live_files[i].file_checksum_func_name != stored_func_name) {
|
|
|
|
return Status::Corruption(
|
|
|
|
"Checksum does not match! The file: " +
|
2022-05-06 20:03:58 +00:00
|
|
|
std::to_string(live_files[i].file_number) +
|
2020-02-10 23:42:46 +00:00
|
|
|
". In Manifest, checksum name: " + stored_func_name +
|
|
|
|
" and checksum " + stored_checksum +
|
|
|
|
". However, expected checksum name: " +
|
|
|
|
live_files[i].file_checksum_func_name + " and checksum " +
|
|
|
|
live_files[i].file_checksum);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return Status::OK();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Verify the checksum of each file by recalculting the checksum and
|
|
|
|
// comparing it with the one being generated when a SST file is created.
|
|
|
|
Status VerifyEachFileChecksum() {
|
|
|
|
assert(db_ != nullptr);
|
2021-09-21 18:25:41 +00:00
|
|
|
EXPECT_OK(db_->DisableFileDeletions());
|
2020-02-10 23:42:46 +00:00
|
|
|
std::vector<LiveFileMetaData> live_files;
|
|
|
|
db_->GetLiveFilesMetaData(&live_files);
|
2021-09-21 18:25:41 +00:00
|
|
|
Status cs;
|
2020-02-10 23:42:46 +00:00
|
|
|
for (auto a_file : live_files) {
|
2021-09-21 18:25:41 +00:00
|
|
|
cs = VerifyChecksum(a_file);
|
2020-02-10 23:42:46 +00:00
|
|
|
if (!cs.ok()) {
|
2021-09-21 18:25:41 +00:00
|
|
|
break;
|
2020-02-10 23:42:46 +00:00
|
|
|
}
|
|
|
|
}
|
2024-02-14 02:36:25 +00:00
|
|
|
EXPECT_OK(db_->EnableFileDeletions());
|
2021-09-21 18:25:41 +00:00
|
|
|
return cs;
|
2020-02-10 23:42:46 +00:00
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
TEST_F(LdbCmdTest, DumpFileChecksumNoChecksum) {
|
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
2020-06-25 19:07:47 +00:00
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
Random rnd(test::RandomSeed());
|
|
|
|
for (int i = 0; i < 200; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 100; i < 300; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 200; i < 400; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 300; i < 400; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "file_checksum_dump";
|
2021-09-14 00:06:05 +00:00
|
|
|
char arg4[] = "--hex";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
2021-09-14 00:06:05 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
// Verify each sst file checksum value and checksum name
|
|
|
|
FileChecksumTestHelper fct_helper(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper.VerifyEachFileChecksum());
|
|
|
|
|
|
|
|
// Manually trigger compaction
|
|
|
|
char b_buf[16];
|
|
|
|
snprintf(b_buf, sizeof(b_buf), "%08d", 0);
|
|
|
|
char e_buf[16];
|
|
|
|
snprintf(e_buf, sizeof(e_buf), "%08d", 399);
|
|
|
|
Slice begin(b_buf);
|
|
|
|
Slice end(e_buf);
|
|
|
|
CompactRangeOptions options;
|
|
|
|
ASSERT_OK(db->CompactRange(options, &begin, &end));
|
|
|
|
// Verify each sst file checksum after compaction
|
|
|
|
FileChecksumTestHelper fct_helper_ac(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper_ac.VerifyEachFileChecksum());
|
|
|
|
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
|
|
|
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_EQ(0,
|
2021-09-14 00:06:05 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
// Verify the checksum information in memory is the same as that in Manifest;
|
|
|
|
std::vector<LiveFileMetaData> live_files;
|
|
|
|
db->GetLiveFilesMetaData(&live_files);
|
|
|
|
delete db;
|
|
|
|
ASSERT_OK(fct_helper_ac.VerifyChecksumInManifest(live_files));
|
|
|
|
}
|
|
|
|
|
2021-04-15 16:36:57 +00:00
|
|
|
TEST_F(LdbCmdTest, BlobDBDumpFileChecksumNoChecksum) {
|
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
opts.enable_blob_files = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
Random rnd(test::RandomSeed());
|
|
|
|
for (int i = 0; i < 200; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 100; i < 300; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 200; i < 400; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 300; i < 400; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
2021-04-15 16:36:57 +00:00
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
std::string arg2_str = "--db=" + dbname;
|
|
|
|
char arg3[] = "file_checksum_dump";
|
2021-09-14 00:06:05 +00:00
|
|
|
char arg4[] = "--hex";
|
|
|
|
char* argv[] = {arg1, const_cast<char*>(arg2_str.c_str()), arg3, arg4};
|
2021-04-15 16:36:57 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
2021-09-14 00:06:05 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
2021-04-15 16:36:57 +00:00
|
|
|
|
|
|
|
// Verify each sst and blob file checksum value and checksum name
|
|
|
|
FileChecksumTestHelper fct_helper(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper.VerifyEachFileChecksum());
|
|
|
|
|
|
|
|
// Manually trigger compaction
|
|
|
|
std::ostringstream oss_b_buf;
|
|
|
|
oss_b_buf << std::setfill('0') << std::setw(8) << std::fixed << 0;
|
|
|
|
std::ostringstream oss_e_buf;
|
|
|
|
oss_e_buf << std::setfill('0') << std::setw(8) << std::fixed << 399;
|
|
|
|
std::string b_buf = oss_b_buf.str();
|
|
|
|
std::string e_buf = oss_e_buf.str();
|
|
|
|
Slice begin(b_buf);
|
|
|
|
Slice end(e_buf);
|
|
|
|
|
|
|
|
CompactRangeOptions options;
|
|
|
|
ASSERT_OK(db->CompactRange(options, &begin, &end));
|
|
|
|
// Verify each sst file checksum after compaction
|
|
|
|
FileChecksumTestHelper fct_helper_ac(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper_ac.VerifyEachFileChecksum());
|
|
|
|
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
2021-04-15 16:36:57 +00:00
|
|
|
delete db;
|
2021-09-14 00:06:05 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
2021-04-15 16:36:57 +00:00
|
|
|
}
|
|
|
|
|
2020-02-10 23:42:46 +00:00
|
|
|
TEST_F(LdbCmdTest, DumpFileChecksumCRC32) {
|
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
2020-04-14 02:11:19 +00:00
|
|
|
opts.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
DB* db = nullptr;
|
2020-06-25 19:07:47 +00:00
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
Random rnd(test::RandomSeed());
|
|
|
|
for (int i = 0; i < 100; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 50; i < 150; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 100; i < 200; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 150; i < 250; i++) {
|
|
|
|
char buf[16];
|
|
|
|
snprintf(buf, sizeof(buf), "%08d", i);
|
2020-07-09 21:33:42 +00:00
|
|
|
std::string v = rnd.RandomString(100);
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_OK(db->Put(wopts, buf, v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "file_checksum_dump";
|
2021-09-14 00:06:05 +00:00
|
|
|
char arg4[] = "--hex";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
2021-09-14 00:06:05 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
// Verify each sst file checksum value and checksum name
|
|
|
|
FileChecksumTestHelper fct_helper(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper.VerifyEachFileChecksum());
|
|
|
|
|
|
|
|
// Manually trigger compaction
|
|
|
|
char b_buf[16];
|
|
|
|
snprintf(b_buf, sizeof(b_buf), "%08d", 0);
|
|
|
|
char e_buf[16];
|
|
|
|
snprintf(e_buf, sizeof(e_buf), "%08d", 249);
|
|
|
|
Slice begin(b_buf);
|
|
|
|
Slice end(e_buf);
|
|
|
|
CompactRangeOptions options;
|
|
|
|
ASSERT_OK(db->CompactRange(options, &begin, &end));
|
|
|
|
// Verify each sst file checksum after compaction
|
|
|
|
FileChecksumTestHelper fct_helper_ac(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper_ac.VerifyEachFileChecksum());
|
|
|
|
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
|
|
|
|
2020-02-10 23:42:46 +00:00
|
|
|
ASSERT_EQ(0,
|
2021-09-14 00:06:05 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
2020-02-10 23:42:46 +00:00
|
|
|
|
|
|
|
// Verify the checksum information in memory is the same as that in Manifest;
|
|
|
|
std::vector<LiveFileMetaData> live_files;
|
|
|
|
db->GetLiveFilesMetaData(&live_files);
|
|
|
|
ASSERT_OK(fct_helper_ac.VerifyChecksumInManifest(live_files));
|
2021-09-14 00:06:05 +00:00
|
|
|
|
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
2020-02-10 23:42:46 +00:00
|
|
|
}
|
|
|
|
|
2021-04-15 16:36:57 +00:00
|
|
|
TEST_F(LdbCmdTest, BlobDBDumpFileChecksumCRC32) {
|
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
opts.file_checksum_gen_factory = GetFileChecksumGenCrc32cFactory();
|
|
|
|
opts.enable_blob_files = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
Random rnd(test::RandomSeed());
|
|
|
|
for (int i = 0; i < 100; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 50; i < 150; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 100; i < 200; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
for (int i = 150; i < 250; i++) {
|
|
|
|
std::ostringstream oss;
|
|
|
|
oss << std::setfill('0') << std::setw(8) << std::fixed << i;
|
|
|
|
std::string v = rnd.RandomString(100);
|
|
|
|
ASSERT_OK(db->Put(wopts, oss.str(), v));
|
|
|
|
}
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
|
|
|
delete db;
|
2021-04-15 16:36:57 +00:00
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
std::string arg2_str = "--db=" + dbname;
|
|
|
|
char arg3[] = "file_checksum_dump";
|
2021-09-14 00:06:05 +00:00
|
|
|
char arg4[] = "--hex";
|
|
|
|
char* argv[] = {arg1, const_cast<char*>(arg2_str.c_str()), arg3, arg4};
|
2021-04-15 16:36:57 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
2021-09-14 00:06:05 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
2021-04-15 16:36:57 +00:00
|
|
|
|
|
|
|
// Verify each sst and blob file checksum value and checksum name
|
|
|
|
FileChecksumTestHelper fct_helper(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper.VerifyEachFileChecksum());
|
|
|
|
|
|
|
|
// Manually trigger compaction
|
|
|
|
std::ostringstream oss_b_buf;
|
|
|
|
oss_b_buf << std::setfill('0') << std::setw(8) << std::fixed << 0;
|
|
|
|
std::ostringstream oss_e_buf;
|
|
|
|
oss_e_buf << std::setfill('0') << std::setw(8) << std::fixed << 249;
|
|
|
|
std::string b_buf = oss_b_buf.str();
|
|
|
|
std::string e_buf = oss_e_buf.str();
|
|
|
|
Slice begin(b_buf);
|
|
|
|
Slice end(e_buf);
|
|
|
|
|
|
|
|
CompactRangeOptions options;
|
|
|
|
ASSERT_OK(db->CompactRange(options, &begin, &end));
|
|
|
|
// Verify each sst file checksum after compaction
|
|
|
|
FileChecksumTestHelper fct_helper_ac(opts, db, dbname);
|
|
|
|
ASSERT_OK(fct_helper_ac.VerifyEachFileChecksum());
|
|
|
|
|
2021-09-14 00:06:05 +00:00
|
|
|
ASSERT_OK(db->Close());
|
2021-04-15 16:36:57 +00:00
|
|
|
delete db;
|
2021-09-14 00:06:05 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
2021-04-15 16:36:57 +00:00
|
|
|
}
|
|
|
|
|
2019-03-25 20:19:25 +00:00
|
|
|
TEST_F(LdbCmdTest, OptionParsing) {
|
|
|
|
// test parsing flags
|
2019-10-09 02:17:39 +00:00
|
|
|
Options opts;
|
|
|
|
opts.env = TryLoadCustomOrDefaultEnv();
|
2019-03-25 20:19:25 +00:00
|
|
|
{
|
|
|
|
std::vector<std::string> args;
|
2023-12-01 19:15:17 +00:00
|
|
|
args.emplace_back("scan");
|
|
|
|
args.emplace_back("--ttl");
|
|
|
|
args.emplace_back("--timestamp");
|
2020-02-20 20:07:53 +00:00
|
|
|
LDBCommand* command = ROCKSDB_NAMESPACE::LDBCommand::InitFromCmdLineArgs(
|
2019-10-09 02:17:39 +00:00
|
|
|
args, opts, LDBOptions(), nullptr);
|
2019-03-25 20:19:25 +00:00
|
|
|
const std::vector<std::string> flags = command->TEST_GetFlags();
|
|
|
|
EXPECT_EQ(flags.size(), 2);
|
|
|
|
EXPECT_EQ(flags[0], "ttl");
|
|
|
|
EXPECT_EQ(flags[1], "timestamp");
|
|
|
|
delete command;
|
|
|
|
}
|
|
|
|
// test parsing options which contains equal sign in the option value
|
|
|
|
{
|
|
|
|
std::vector<std::string> args;
|
2023-12-01 19:15:17 +00:00
|
|
|
args.emplace_back("scan");
|
|
|
|
args.emplace_back("--db=/dev/shm/ldbtest/");
|
|
|
|
args.emplace_back(
|
2019-03-25 20:19:25 +00:00
|
|
|
"--from='abcd/efg/hijk/lmn/"
|
|
|
|
"opq:__rst.uvw.xyz?a=3+4+bcd+efghi&jk=lm_no&pq=rst-0&uv=wx-8&yz=a&bcd_"
|
|
|
|
"ef=gh.ijk'");
|
2020-02-20 20:07:53 +00:00
|
|
|
LDBCommand* command = ROCKSDB_NAMESPACE::LDBCommand::InitFromCmdLineArgs(
|
2019-10-09 02:17:39 +00:00
|
|
|
args, opts, LDBOptions(), nullptr);
|
2019-03-25 20:19:25 +00:00
|
|
|
const std::map<std::string, std::string> option_map =
|
|
|
|
command->TEST_GetOptionMap();
|
|
|
|
EXPECT_EQ(option_map.at("db"), "/dev/shm/ldbtest/");
|
|
|
|
EXPECT_EQ(option_map.at("from"),
|
|
|
|
"'abcd/efg/hijk/lmn/"
|
|
|
|
"opq:__rst.uvw.xyz?a=3+4+bcd+efghi&jk=lm_no&pq=rst-0&uv=wx-8&yz="
|
|
|
|
"a&bcd_ef=gh.ijk'");
|
|
|
|
delete command;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-08-15 23:59:42 +00:00
|
|
|
TEST_F(LdbCmdTest, ListFileTombstone) {
|
2019-10-09 02:17:39 +00:00
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
2019-08-15 23:59:42 +00:00
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
2020-06-25 19:07:47 +00:00
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2019-08-15 23:59:42 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
ASSERT_OK(db->Put(wopts, "foo", "1"));
|
|
|
|
ASSERT_OK(db->Put(wopts, "bar", "2"));
|
|
|
|
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
|
|
|
|
ASSERT_OK(db->DeleteRange(wopts, db->DefaultColumnFamily(), "foo", "foo2"));
|
|
|
|
ASSERT_OK(db->DeleteRange(wopts, db->DefaultColumnFamily(), "bar", "foo2"));
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
|
|
|
|
delete db;
|
|
|
|
|
|
|
|
{
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "list_file_range_deletes";
|
|
|
|
char* argv[] = {arg1, arg2, arg3};
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
|
2019-08-15 23:59:42 +00:00
|
|
|
"ListFileRangeDeletesCommand::DoCommand:BeforePrint", [&](void* arg) {
|
Prefer static_cast in place of most reinterpret_cast (#12308)
Summary:
The following are risks associated with pointer-to-pointer reinterpret_cast:
* Can produce the "wrong result" (crash or memory corruption). IIRC, in theory this can happen for any up-cast or down-cast for a non-standard-layout type, though in practice would only happen for multiple inheritance cases (where the base class pointer might be "inside" the derived object). We don't use multiple inheritance a lot, but we do.
* Can mask useful compiler errors upon code change, including converting between unrelated pointer types that you are expecting to be related, and converting between pointer and scalar types unintentionally.
I can only think of some obscure cases where static_cast could be troublesome when it compiles as a replacement:
* Going through `void*` could plausibly cause unnecessary or broken pointer arithmetic. Suppose we have
`struct Derived: public Base1, public Base2`. If we have `Derived*` -> `void*` -> `Base2*` -> `Derived*` through reinterpret casts, this could plausibly work (though technical UB) assuming the `Base2*` is not dereferenced. Changing to static cast could introduce breaking pointer arithmetic.
* Unnecessary (but safe) pointer arithmetic could arise in a case like `Derived*` -> `Base2*` -> `Derived*` where before the Base2 pointer might not have been dereferenced. This could potentially affect performance.
With some light scripting, I tried replacing pointer-to-pointer reinterpret_casts with static_cast and kept the cases that still compile. Most occurrences of reinterpret_cast have successfully been changed (except for java/ and third-party/). 294 changed, 257 remain.
A couple of related interventions included here:
* Previously Cache::Handle was not actually derived from in the implementations and just used as a `void*` stand-in with reinterpret_cast. Now there is a relationship to allow static_cast. In theory, this could introduce pointer arithmetic (as described above) but is unlikely without multiple inheritance AND non-empty Cache::Handle.
* Remove some unnecessary casts to void* as this is allowed to be implicit (for better or worse).
Most of the remaining reinterpret_casts are for converting to/from raw bytes of objects. We could consider better idioms for these patterns in follow-up work.
I wish there were a way to implement a template variant of static_cast that would only compile if no pointer arithmetic is generated, but best I can tell, this is not possible. AFAIK the best you could do is a dynamic check that the void* conversion after the static cast is unchanged.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12308
Test Plan: existing tests, CI
Reviewed By: ltamasi
Differential Revision: D53204947
Pulled By: pdillinger
fbshipit-source-id: 9de23e618263b0d5b9820f4e15966876888a16e2
2024-02-07 18:44:11 +00:00
|
|
|
std::string* out_str = static_cast<std::string*>(arg);
|
2019-08-15 23:59:42 +00:00
|
|
|
|
|
|
|
// Count number of tombstones printed
|
|
|
|
int num_tb = 0;
|
|
|
|
const std::string kFingerprintStr = "start: ";
|
|
|
|
auto offset = out_str->find(kFingerprintStr);
|
|
|
|
while (offset != std::string::npos) {
|
|
|
|
num_tb++;
|
|
|
|
offset =
|
|
|
|
out_str->find(kFingerprintStr, offset + kFingerprintStr.size());
|
|
|
|
}
|
|
|
|
EXPECT_EQ(2, num_tb);
|
|
|
|
});
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
|
2019-08-15 23:59:42 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(3, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
|
2019-08-15 23:59:42 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Test the case of limiting tombstones
|
|
|
|
{
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "list_file_range_deletes";
|
|
|
|
char arg4[] = "--max_keys=1";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->SetCallBack(
|
2019-08-15 23:59:42 +00:00
|
|
|
"ListFileRangeDeletesCommand::DoCommand:BeforePrint", [&](void* arg) {
|
Prefer static_cast in place of most reinterpret_cast (#12308)
Summary:
The following are risks associated with pointer-to-pointer reinterpret_cast:
* Can produce the "wrong result" (crash or memory corruption). IIRC, in theory this can happen for any up-cast or down-cast for a non-standard-layout type, though in practice would only happen for multiple inheritance cases (where the base class pointer might be "inside" the derived object). We don't use multiple inheritance a lot, but we do.
* Can mask useful compiler errors upon code change, including converting between unrelated pointer types that you are expecting to be related, and converting between pointer and scalar types unintentionally.
I can only think of some obscure cases where static_cast could be troublesome when it compiles as a replacement:
* Going through `void*` could plausibly cause unnecessary or broken pointer arithmetic. Suppose we have
`struct Derived: public Base1, public Base2`. If we have `Derived*` -> `void*` -> `Base2*` -> `Derived*` through reinterpret casts, this could plausibly work (though technical UB) assuming the `Base2*` is not dereferenced. Changing to static cast could introduce breaking pointer arithmetic.
* Unnecessary (but safe) pointer arithmetic could arise in a case like `Derived*` -> `Base2*` -> `Derived*` where before the Base2 pointer might not have been dereferenced. This could potentially affect performance.
With some light scripting, I tried replacing pointer-to-pointer reinterpret_casts with static_cast and kept the cases that still compile. Most occurrences of reinterpret_cast have successfully been changed (except for java/ and third-party/). 294 changed, 257 remain.
A couple of related interventions included here:
* Previously Cache::Handle was not actually derived from in the implementations and just used as a `void*` stand-in with reinterpret_cast. Now there is a relationship to allow static_cast. In theory, this could introduce pointer arithmetic (as described above) but is unlikely without multiple inheritance AND non-empty Cache::Handle.
* Remove some unnecessary casts to void* as this is allowed to be implicit (for better or worse).
Most of the remaining reinterpret_casts are for converting to/from raw bytes of objects. We could consider better idioms for these patterns in follow-up work.
I wish there were a way to implement a template variant of static_cast that would only compile if no pointer arithmetic is generated, but best I can tell, this is not possible. AFAIK the best you could do is a dynamic check that the void* conversion after the static cast is unchanged.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12308
Test Plan: existing tests, CI
Reviewed By: ltamasi
Differential Revision: D53204947
Pulled By: pdillinger
fbshipit-source-id: 9de23e618263b0d5b9820f4e15966876888a16e2
2024-02-07 18:44:11 +00:00
|
|
|
std::string* out_str = static_cast<std::string*>(arg);
|
2019-08-15 23:59:42 +00:00
|
|
|
|
|
|
|
// Count number of tombstones printed
|
|
|
|
int num_tb = 0;
|
|
|
|
const std::string kFingerprintStr = "start: ";
|
|
|
|
auto offset = out_str->find(kFingerprintStr);
|
|
|
|
while (offset != std::string::npos) {
|
|
|
|
num_tb++;
|
|
|
|
offset =
|
|
|
|
out_str->find(kFingerprintStr, offset + kFingerprintStr.size());
|
|
|
|
}
|
|
|
|
EXPECT_EQ(1, num_tb);
|
|
|
|
});
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->EnableProcessing();
|
2019-08-15 23:59:42 +00:00
|
|
|
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->DisableProcessing();
|
2019-08-15 23:59:42 +00:00
|
|
|
}
|
|
|
|
}
|
2020-05-08 21:12:18 +00:00
|
|
|
|
|
|
|
TEST_F(LdbCmdTest, DisableConsistencyChecks) {
|
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
2020-06-25 19:07:47 +00:00
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2020-05-08 21:12:18 +00:00
|
|
|
|
|
|
|
{
|
|
|
|
DB* db = nullptr;
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
WriteOptions wopts;
|
|
|
|
FlushOptions fopts;
|
|
|
|
fopts.wait = true;
|
|
|
|
|
|
|
|
ASSERT_OK(db->Put(wopts, "foo1", "1"));
|
|
|
|
ASSERT_OK(db->Put(wopts, "bar1", "2"));
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
|
|
|
|
ASSERT_OK(db->Put(wopts, "foo2", "3"));
|
|
|
|
ASSERT_OK(db->Put(wopts, "bar2", "4"));
|
|
|
|
ASSERT_OK(db->Flush(fopts));
|
|
|
|
|
|
|
|
delete db;
|
|
|
|
}
|
|
|
|
|
|
|
|
{
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "checkconsistency";
|
|
|
|
char* argv[] = {arg1, arg2, arg3};
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
2022-02-04 16:18:18 +00:00
|
|
|
"Version::PrepareAppend:forced_check", [&](void* arg) {
|
Prefer static_cast in place of most reinterpret_cast (#12308)
Summary:
The following are risks associated with pointer-to-pointer reinterpret_cast:
* Can produce the "wrong result" (crash or memory corruption). IIRC, in theory this can happen for any up-cast or down-cast for a non-standard-layout type, though in practice would only happen for multiple inheritance cases (where the base class pointer might be "inside" the derived object). We don't use multiple inheritance a lot, but we do.
* Can mask useful compiler errors upon code change, including converting between unrelated pointer types that you are expecting to be related, and converting between pointer and scalar types unintentionally.
I can only think of some obscure cases where static_cast could be troublesome when it compiles as a replacement:
* Going through `void*` could plausibly cause unnecessary or broken pointer arithmetic. Suppose we have
`struct Derived: public Base1, public Base2`. If we have `Derived*` -> `void*` -> `Base2*` -> `Derived*` through reinterpret casts, this could plausibly work (though technical UB) assuming the `Base2*` is not dereferenced. Changing to static cast could introduce breaking pointer arithmetic.
* Unnecessary (but safe) pointer arithmetic could arise in a case like `Derived*` -> `Base2*` -> `Derived*` where before the Base2 pointer might not have been dereferenced. This could potentially affect performance.
With some light scripting, I tried replacing pointer-to-pointer reinterpret_casts with static_cast and kept the cases that still compile. Most occurrences of reinterpret_cast have successfully been changed (except for java/ and third-party/). 294 changed, 257 remain.
A couple of related interventions included here:
* Previously Cache::Handle was not actually derived from in the implementations and just used as a `void*` stand-in with reinterpret_cast. Now there is a relationship to allow static_cast. In theory, this could introduce pointer arithmetic (as described above) but is unlikely without multiple inheritance AND non-empty Cache::Handle.
* Remove some unnecessary casts to void* as this is allowed to be implicit (for better or worse).
Most of the remaining reinterpret_casts are for converting to/from raw bytes of objects. We could consider better idioms for these patterns in follow-up work.
I wish there were a way to implement a template variant of static_cast that would only compile if no pointer arithmetic is generated, but best I can tell, this is not possible. AFAIK the best you could do is a dynamic check that the void* conversion after the static cast is unchanged.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12308
Test Plan: existing tests, CI
Reviewed By: ltamasi
Differential Revision: D53204947
Pulled By: pdillinger
fbshipit-source-id: 9de23e618263b0d5b9820f4e15966876888a16e2
2024-02-07 18:44:11 +00:00
|
|
|
bool* forced = static_cast<bool*>(arg);
|
2020-05-08 21:12:18 +00:00
|
|
|
ASSERT_TRUE(*forced);
|
|
|
|
});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(3, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
}
|
|
|
|
{
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "scan";
|
|
|
|
char* argv[] = {arg1, arg2, arg3};
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
2022-02-04 16:18:18 +00:00
|
|
|
"Version::PrepareAppend:forced_check", [&](void* arg) {
|
Prefer static_cast in place of most reinterpret_cast (#12308)
Summary:
The following are risks associated with pointer-to-pointer reinterpret_cast:
* Can produce the "wrong result" (crash or memory corruption). IIRC, in theory this can happen for any up-cast or down-cast for a non-standard-layout type, though in practice would only happen for multiple inheritance cases (where the base class pointer might be "inside" the derived object). We don't use multiple inheritance a lot, but we do.
* Can mask useful compiler errors upon code change, including converting between unrelated pointer types that you are expecting to be related, and converting between pointer and scalar types unintentionally.
I can only think of some obscure cases where static_cast could be troublesome when it compiles as a replacement:
* Going through `void*` could plausibly cause unnecessary or broken pointer arithmetic. Suppose we have
`struct Derived: public Base1, public Base2`. If we have `Derived*` -> `void*` -> `Base2*` -> `Derived*` through reinterpret casts, this could plausibly work (though technical UB) assuming the `Base2*` is not dereferenced. Changing to static cast could introduce breaking pointer arithmetic.
* Unnecessary (but safe) pointer arithmetic could arise in a case like `Derived*` -> `Base2*` -> `Derived*` where before the Base2 pointer might not have been dereferenced. This could potentially affect performance.
With some light scripting, I tried replacing pointer-to-pointer reinterpret_casts with static_cast and kept the cases that still compile. Most occurrences of reinterpret_cast have successfully been changed (except for java/ and third-party/). 294 changed, 257 remain.
A couple of related interventions included here:
* Previously Cache::Handle was not actually derived from in the implementations and just used as a `void*` stand-in with reinterpret_cast. Now there is a relationship to allow static_cast. In theory, this could introduce pointer arithmetic (as described above) but is unlikely without multiple inheritance AND non-empty Cache::Handle.
* Remove some unnecessary casts to void* as this is allowed to be implicit (for better or worse).
Most of the remaining reinterpret_casts are for converting to/from raw bytes of objects. We could consider better idioms for these patterns in follow-up work.
I wish there were a way to implement a template variant of static_cast that would only compile if no pointer arithmetic is generated, but best I can tell, this is not possible. AFAIK the best you could do is a dynamic check that the void* conversion after the static cast is unchanged.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12308
Test Plan: existing tests, CI
Reviewed By: ltamasi
Differential Revision: D53204947
Pulled By: pdillinger
fbshipit-source-id: 9de23e618263b0d5b9820f4e15966876888a16e2
2024-02-07 18:44:11 +00:00
|
|
|
bool* forced = static_cast<bool*>(arg);
|
2020-05-08 21:12:18 +00:00
|
|
|
ASSERT_TRUE(*forced);
|
|
|
|
});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(3, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
}
|
|
|
|
{
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "scan";
|
|
|
|
char arg4[] = "--disable_consistency_checks";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->SetCallBack(
|
|
|
|
"ColumnFamilyData::ColumnFamilyData", [&](void* arg) {
|
Prefer static_cast in place of most reinterpret_cast (#12308)
Summary:
The following are risks associated with pointer-to-pointer reinterpret_cast:
* Can produce the "wrong result" (crash or memory corruption). IIRC, in theory this can happen for any up-cast or down-cast for a non-standard-layout type, though in practice would only happen for multiple inheritance cases (where the base class pointer might be "inside" the derived object). We don't use multiple inheritance a lot, but we do.
* Can mask useful compiler errors upon code change, including converting between unrelated pointer types that you are expecting to be related, and converting between pointer and scalar types unintentionally.
I can only think of some obscure cases where static_cast could be troublesome when it compiles as a replacement:
* Going through `void*` could plausibly cause unnecessary or broken pointer arithmetic. Suppose we have
`struct Derived: public Base1, public Base2`. If we have `Derived*` -> `void*` -> `Base2*` -> `Derived*` through reinterpret casts, this could plausibly work (though technical UB) assuming the `Base2*` is not dereferenced. Changing to static cast could introduce breaking pointer arithmetic.
* Unnecessary (but safe) pointer arithmetic could arise in a case like `Derived*` -> `Base2*` -> `Derived*` where before the Base2 pointer might not have been dereferenced. This could potentially affect performance.
With some light scripting, I tried replacing pointer-to-pointer reinterpret_casts with static_cast and kept the cases that still compile. Most occurrences of reinterpret_cast have successfully been changed (except for java/ and third-party/). 294 changed, 257 remain.
A couple of related interventions included here:
* Previously Cache::Handle was not actually derived from in the implementations and just used as a `void*` stand-in with reinterpret_cast. Now there is a relationship to allow static_cast. In theory, this could introduce pointer arithmetic (as described above) but is unlikely without multiple inheritance AND non-empty Cache::Handle.
* Remove some unnecessary casts to void* as this is allowed to be implicit (for better or worse).
Most of the remaining reinterpret_casts are for converting to/from raw bytes of objects. We could consider better idioms for these patterns in follow-up work.
I wish there were a way to implement a template variant of static_cast that would only compile if no pointer arithmetic is generated, but best I can tell, this is not possible. AFAIK the best you could do is a dynamic check that the void* conversion after the static cast is unchanged.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12308
Test Plan: existing tests, CI
Reviewed By: ltamasi
Differential Revision: D53204947
Pulled By: pdillinger
fbshipit-source-id: 9de23e618263b0d5b9820f4e15966876888a16e2
2024-02-07 18:44:11 +00:00
|
|
|
ColumnFamilyOptions* cfo = static_cast<ColumnFamilyOptions*>(arg);
|
2020-05-08 21:12:18 +00:00
|
|
|
ASSERT_FALSE(cfo->force_consistency_checks);
|
|
|
|
});
|
|
|
|
SyncPoint::GetInstance()->EnableProcessing();
|
|
|
|
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
SyncPoint::GetInstance()->ClearAllCallBacks();
|
|
|
|
SyncPoint::GetInstance()->DisableProcessing();
|
|
|
|
}
|
|
|
|
}
|
2020-06-04 18:38:34 +00:00
|
|
|
|
|
|
|
TEST_F(LdbCmdTest, TestBadDbPath) {
|
|
|
|
Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
std::unique_ptr<Env> env(NewMemEnv(base_env));
|
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
2020-06-25 19:07:47 +00:00
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2020-06-04 18:38:34 +00:00
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s/.no_such_dir", dbname.c_str());
|
|
|
|
char arg3[1024];
|
|
|
|
snprintf(arg3, sizeof(arg3), "create_column_family");
|
|
|
|
char arg4[] = "bad cf";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
|
|
|
|
|
|
|
ASSERT_EQ(1,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
snprintf(arg3, sizeof(arg3), "drop_column_family");
|
|
|
|
ASSERT_EQ(1,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
}
|
2022-01-05 00:44:54 +00:00
|
|
|
namespace {
|
|
|
|
class WrappedEnv : public EnvWrapper {
|
|
|
|
public:
|
|
|
|
explicit WrappedEnv(Env* t) : EnvWrapper(t) {}
|
|
|
|
static const char* kClassName() { return "WrappedEnv"; }
|
|
|
|
const char* Name() const override { return kClassName(); }
|
|
|
|
};
|
|
|
|
} // namespace
|
2021-01-12 04:54:58 +00:00
|
|
|
TEST_F(LdbCmdTest, LoadCFOptionsAndOverride) {
|
|
|
|
// Env* base_env = TryLoadCustomOrDefaultEnv();
|
|
|
|
// std::unique_ptr<Env> env(NewMemEnv(base_env));
|
2022-01-05 00:44:54 +00:00
|
|
|
std::unique_ptr<Env> env(new WrappedEnv(Env::Default()));
|
2021-01-12 04:54:58 +00:00
|
|
|
Options opts;
|
|
|
|
opts.env = env.get();
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2022-08-11 00:34:38 +00:00
|
|
|
ASSERT_OK(DestroyDB(dbname, opts));
|
2021-01-12 04:54:58 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
ColumnFamilyHandle* cf_handle;
|
|
|
|
ColumnFamilyOptions cf_opts;
|
|
|
|
cf_opts.num_levels = 20;
|
|
|
|
ASSERT_OK(db->CreateColumnFamily(cf_opts, "cf1", &cf_handle));
|
|
|
|
|
|
|
|
delete cf_handle;
|
|
|
|
delete db;
|
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "put";
|
|
|
|
char arg4[] = "key1";
|
|
|
|
char arg5[] = "value1";
|
|
|
|
char arg6[] = "--try_load_options";
|
|
|
|
char arg7[] = "--column_family=cf1";
|
|
|
|
char arg8[] = "--write_buffer_size=268435456";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8};
|
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(8, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ConfigOptions config_opts;
|
|
|
|
Options options;
|
|
|
|
std::vector<ColumnFamilyDescriptor> column_families;
|
|
|
|
config_opts.env = env.get();
|
|
|
|
ASSERT_OK(LoadLatestOptions(config_opts, dbname, &options, &column_families));
|
|
|
|
ASSERT_EQ(column_families.size(), 2);
|
|
|
|
ASSERT_EQ(options.num_levels, opts.num_levels);
|
|
|
|
ASSERT_EQ(column_families[1].options.num_levels, cf_opts.num_levels);
|
|
|
|
ASSERT_EQ(column_families[1].options.write_buffer_size, 268435456);
|
|
|
|
}
|
2021-07-30 19:15:04 +00:00
|
|
|
|
2022-03-18 23:35:51 +00:00
|
|
|
TEST_F(LdbCmdTest, UnsafeRemoveSstFile) {
|
|
|
|
Options opts;
|
|
|
|
opts.level0_file_num_compaction_trigger = 10;
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
|
|
|
std::string dbname = test::PerThreadDBPath(Env::Default(), "ldb_cmd_test");
|
2022-08-11 00:34:38 +00:00
|
|
|
ASSERT_OK(DestroyDB(dbname, opts));
|
2022-03-18 23:35:51 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
// Create three SST files
|
|
|
|
for (size_t i = 0; i < 3; ++i) {
|
2022-05-06 20:03:58 +00:00
|
|
|
ASSERT_OK(db->Put(WriteOptions(), std::to_string(i), std::to_string(i)));
|
2022-03-18 23:35:51 +00:00
|
|
|
ASSERT_OK(db->Flush(FlushOptions()));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Determine which is the "middle" one
|
|
|
|
std::vector<LiveFileMetaData> sst_files;
|
|
|
|
db->GetLiveFilesMetaData(&sst_files);
|
|
|
|
|
|
|
|
std::vector<uint64_t> numbers;
|
|
|
|
for (auto& f : sst_files) {
|
|
|
|
numbers.push_back(f.file_number);
|
|
|
|
}
|
|
|
|
ASSERT_EQ(numbers.size(), 3);
|
|
|
|
std::sort(numbers.begin(), numbers.end());
|
|
|
|
uint64_t to_remove = numbers[1];
|
|
|
|
|
|
|
|
// Close for unsafe_remove_sst_file
|
|
|
|
delete db;
|
|
|
|
db = nullptr;
|
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "unsafe_remove_sst_file";
|
|
|
|
char arg4[20];
|
|
|
|
snprintf(arg4, sizeof(arg4), "%" PRIu64, to_remove);
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
// Re-open, and verify with Get that middle file is gone
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
std::string val;
|
|
|
|
ASSERT_OK(db->Get(ReadOptions(), "0", &val));
|
|
|
|
ASSERT_EQ(val, "0");
|
|
|
|
|
|
|
|
ASSERT_OK(db->Get(ReadOptions(), "2", &val));
|
|
|
|
ASSERT_EQ(val, "2");
|
|
|
|
|
|
|
|
ASSERT_TRUE(db->Get(ReadOptions(), "1", &val).IsNotFound());
|
|
|
|
|
|
|
|
// Now with extra CF, two more files
|
|
|
|
ColumnFamilyHandle* cf_handle;
|
|
|
|
ColumnFamilyOptions cf_opts;
|
|
|
|
ASSERT_OK(db->CreateColumnFamily(cf_opts, "cf1", &cf_handle));
|
|
|
|
for (size_t i = 3; i < 5; ++i) {
|
2022-05-06 20:03:58 +00:00
|
|
|
ASSERT_OK(db->Put(WriteOptions(), cf_handle, std::to_string(i),
|
|
|
|
std::to_string(i)));
|
2022-03-18 23:35:51 +00:00
|
|
|
ASSERT_OK(db->Flush(FlushOptions(), cf_handle));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Determine which is the "last" one
|
|
|
|
sst_files.clear();
|
|
|
|
db->GetLiveFilesMetaData(&sst_files);
|
|
|
|
|
|
|
|
numbers.clear();
|
|
|
|
for (auto& f : sst_files) {
|
|
|
|
numbers.push_back(f.file_number);
|
|
|
|
}
|
|
|
|
ASSERT_EQ(numbers.size(), 4);
|
|
|
|
std::sort(numbers.begin(), numbers.end());
|
|
|
|
to_remove = numbers.back();
|
|
|
|
|
|
|
|
// Close for unsafe_remove_sst_file
|
|
|
|
delete cf_handle;
|
|
|
|
delete db;
|
|
|
|
db = nullptr;
|
|
|
|
|
|
|
|
snprintf(arg4, sizeof(arg4), "%" PRIu64, to_remove);
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
std::vector<ColumnFamilyDescriptor> cfds = {{kDefaultColumnFamilyName, opts},
|
|
|
|
{"cf1", cf_opts}};
|
|
|
|
std::vector<ColumnFamilyHandle*> handles;
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, cfds, &handles, &db));
|
|
|
|
|
|
|
|
ASSERT_OK(db->Get(ReadOptions(), handles[1], "3", &val));
|
|
|
|
ASSERT_EQ(val, "3");
|
|
|
|
|
|
|
|
ASSERT_TRUE(db->Get(ReadOptions(), handles[1], "4", &val).IsNotFound());
|
|
|
|
|
|
|
|
ASSERT_OK(db->Get(ReadOptions(), handles[0], "0", &val));
|
|
|
|
ASSERT_EQ(val, "0");
|
|
|
|
|
2022-10-11 00:59:17 +00:00
|
|
|
// Determine which is the "first" one (most likely to be opened in recovery)
|
|
|
|
sst_files.clear();
|
|
|
|
db->GetLiveFilesMetaData(&sst_files);
|
|
|
|
|
|
|
|
numbers.clear();
|
|
|
|
for (auto& f : sst_files) {
|
|
|
|
numbers.push_back(f.file_number);
|
|
|
|
}
|
|
|
|
ASSERT_EQ(numbers.size(), 3);
|
|
|
|
std::sort(numbers.begin(), numbers.end());
|
|
|
|
to_remove = numbers.front();
|
|
|
|
|
|
|
|
// This time physically delete the file before unsafe_remove
|
|
|
|
{
|
|
|
|
std::string f = dbname + "/" + MakeTableFileName(to_remove);
|
|
|
|
ASSERT_OK(Env::Default()->DeleteFile(f));
|
|
|
|
}
|
|
|
|
|
|
|
|
// Close for unsafe_remove_sst_file
|
|
|
|
for (auto& h : handles) {
|
|
|
|
delete h;
|
|
|
|
}
|
|
|
|
delete db;
|
|
|
|
db = nullptr;
|
|
|
|
|
|
|
|
snprintf(arg4, sizeof(arg4), "%" PRIu64, to_remove);
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, cfds, &handles, &db));
|
|
|
|
|
|
|
|
ASSERT_OK(db->Get(ReadOptions(), handles[1], "3", &val));
|
|
|
|
ASSERT_EQ(val, "3");
|
|
|
|
|
|
|
|
ASSERT_TRUE(db->Get(ReadOptions(), handles[0], "0", &val).IsNotFound());
|
|
|
|
|
2022-03-18 23:35:51 +00:00
|
|
|
for (auto& h : handles) {
|
|
|
|
delete h;
|
|
|
|
}
|
|
|
|
delete db;
|
|
|
|
}
|
|
|
|
|
|
|
|
TEST_F(LdbCmdTest, FileTemperatureUpdateManifest) {
|
|
|
|
auto test_fs = std::make_shared<FileTemperatureTestFS>(FileSystem::Default());
|
|
|
|
std::unique_ptr<Env> env(new CompositeEnvWrapper(Env::Default(), test_fs));
|
|
|
|
Options opts;
|
2024-02-27 22:48:00 +00:00
|
|
|
opts.last_level_temperature = Temperature::kWarm;
|
2022-03-18 23:35:51 +00:00
|
|
|
opts.level0_file_num_compaction_trigger = 10;
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
opts.env = env.get();
|
|
|
|
|
|
|
|
DB* db = nullptr;
|
|
|
|
std::string dbname = test::PerThreadDBPath(env.get(), "ldb_cmd_test");
|
2022-08-11 00:34:38 +00:00
|
|
|
ASSERT_OK(DestroyDB(dbname, opts));
|
2022-03-18 23:35:51 +00:00
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
std::array<Temperature, 5> kTestTemps = {
|
|
|
|
Temperature::kCold, Temperature::kWarm, Temperature::kHot,
|
|
|
|
Temperature::kWarm, Temperature::kCold};
|
|
|
|
std::map<uint64_t, Temperature> number_to_temp;
|
|
|
|
for (size_t i = 0; i < kTestTemps.size(); ++i) {
|
2022-05-06 20:03:58 +00:00
|
|
|
ASSERT_OK(db->Put(WriteOptions(), std::to_string(i), std::to_string(i)));
|
2022-03-18 23:35:51 +00:00
|
|
|
ASSERT_OK(db->Flush(FlushOptions()));
|
|
|
|
|
|
|
|
std::map<uint64_t, Temperature> current_temps;
|
|
|
|
test_fs->CopyCurrentSstFileTemperatures(¤t_temps);
|
|
|
|
for (auto e : current_temps) {
|
|
|
|
if (e.second == Temperature::kUnknown) {
|
|
|
|
test_fs->OverrideSstFileTemperature(e.first, kTestTemps[i]);
|
|
|
|
number_to_temp[e.first] = kTestTemps[i];
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Close & reopen
|
|
|
|
delete db;
|
|
|
|
db = nullptr;
|
|
|
|
test_fs->PopRequestedSstFileTemperatures();
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
for (size_t i = 0; i < kTestTemps.size(); ++i) {
|
|
|
|
std::string val;
|
2022-05-06 20:03:58 +00:00
|
|
|
ASSERT_OK(db->Get(ReadOptions(), std::to_string(i), &val));
|
|
|
|
ASSERT_EQ(val, std::to_string(i));
|
2022-03-18 23:35:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Still all unknown
|
|
|
|
std::vector<std::pair<uint64_t, Temperature>> requests;
|
|
|
|
test_fs->PopRequestedSstFileTemperatures(&requests);
|
|
|
|
ASSERT_EQ(requests.size(), kTestTemps.size());
|
|
|
|
for (auto& r : requests) {
|
|
|
|
ASSERT_EQ(r.second, Temperature::kUnknown);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Close for update_manifest
|
|
|
|
delete db;
|
|
|
|
db = nullptr;
|
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
char arg2[1024];
|
|
|
|
snprintf(arg2, sizeof(arg2), "--db=%s", dbname.c_str());
|
|
|
|
char arg3[] = "update_manifest";
|
|
|
|
char arg4[] = "--update_temperatures";
|
|
|
|
char* argv[] = {arg1, arg2, arg3, arg4};
|
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), nullptr));
|
|
|
|
|
|
|
|
// Re-open, get, and verify manifest temps (based on request)
|
|
|
|
test_fs->PopRequestedSstFileTemperatures();
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, &db));
|
|
|
|
|
|
|
|
for (size_t i = 0; i < kTestTemps.size(); ++i) {
|
|
|
|
std::string val;
|
2022-05-06 20:03:58 +00:00
|
|
|
ASSERT_OK(db->Get(ReadOptions(), std::to_string(i), &val));
|
|
|
|
ASSERT_EQ(val, std::to_string(i));
|
2022-03-18 23:35:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
requests.clear();
|
|
|
|
test_fs->PopRequestedSstFileTemperatures(&requests);
|
|
|
|
ASSERT_EQ(requests.size(), kTestTemps.size());
|
|
|
|
for (auto& r : requests) {
|
|
|
|
ASSERT_EQ(r.second, number_to_temp[r.first]);
|
|
|
|
}
|
|
|
|
delete db;
|
|
|
|
}
|
|
|
|
|
2021-07-30 19:15:04 +00:00
|
|
|
TEST_F(LdbCmdTest, RenameDbAndLoadOptions) {
|
|
|
|
Env* env = TryLoadCustomOrDefaultEnv();
|
|
|
|
Options opts;
|
|
|
|
opts.env = env;
|
|
|
|
opts.create_if_missing = false;
|
|
|
|
|
|
|
|
std::string old_dbname = test::PerThreadDBPath(env, "ldb_cmd_test");
|
|
|
|
std::string new_dbname = old_dbname + "_2";
|
2022-08-11 00:34:38 +00:00
|
|
|
ASSERT_OK(DestroyDB(old_dbname, opts));
|
|
|
|
ASSERT_OK(DestroyDB(new_dbname, opts));
|
2021-07-30 19:15:04 +00:00
|
|
|
|
|
|
|
char old_arg[1024];
|
|
|
|
snprintf(old_arg, sizeof(old_arg), "--db=%s", old_dbname.c_str());
|
|
|
|
char new_arg[1024];
|
|
|
|
snprintf(new_arg, sizeof(old_arg), "--db=%s", new_dbname.c_str());
|
|
|
|
const char* argv1[] = {"./ldb",
|
|
|
|
old_arg,
|
|
|
|
"put",
|
|
|
|
"key1",
|
|
|
|
"value1",
|
|
|
|
"--try_load_options",
|
|
|
|
"--create_if_missing"};
|
|
|
|
|
|
|
|
const char* argv2[] = {"./ldb", old_arg, "get", "key1", "--try_load_options"};
|
|
|
|
const char* argv3[] = {"./ldb", new_arg, "put",
|
|
|
|
"key2", "value2", "--try_load_options"};
|
|
|
|
|
|
|
|
const char* argv4[] = {"./ldb", new_arg, "get", "key1", "--try_load_options"};
|
|
|
|
const char* argv5[] = {"./ldb", new_arg, "get", "key2", "--try_load_options"};
|
|
|
|
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(7, argv1, opts, LDBOptions(), nullptr));
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(5, argv2, opts, LDBOptions(), nullptr));
|
|
|
|
ConfigOptions config_opts;
|
|
|
|
Options options;
|
|
|
|
std::vector<ColumnFamilyDescriptor> column_families;
|
|
|
|
config_opts.env = env;
|
|
|
|
ASSERT_OK(
|
|
|
|
LoadLatestOptions(config_opts, old_dbname, &options, &column_families));
|
|
|
|
ASSERT_EQ(options.wal_dir, "");
|
|
|
|
|
|
|
|
ASSERT_OK(env->RenameFile(old_dbname, new_dbname));
|
|
|
|
ASSERT_NE(
|
|
|
|
0, LDBCommandRunner::RunCommand(6, argv1, opts, LDBOptions(), nullptr));
|
|
|
|
ASSERT_NE(
|
|
|
|
0, LDBCommandRunner::RunCommand(5, argv2, opts, LDBOptions(), nullptr));
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(6, argv3, opts, LDBOptions(), nullptr));
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(5, argv4, opts, LDBOptions(), nullptr));
|
|
|
|
ASSERT_EQ(
|
|
|
|
0, LDBCommandRunner::RunCommand(5, argv5, opts, LDBOptions(), nullptr));
|
2022-08-11 00:34:38 +00:00
|
|
|
ASSERT_OK(DestroyDB(new_dbname, opts));
|
2021-07-30 19:15:04 +00:00
|
|
|
}
|
2022-03-18 23:35:51 +00:00
|
|
|
|
2023-12-21 02:04:08 +00:00
|
|
|
class MyComparator : public Comparator {
|
|
|
|
public:
|
2023-12-21 20:22:02 +00:00
|
|
|
int Compare(const Slice& a, const Slice& b) const override {
|
2023-12-21 02:04:08 +00:00
|
|
|
return a.compare(b);
|
|
|
|
}
|
|
|
|
void FindShortSuccessor(std::string* /*key*/) const override {}
|
|
|
|
void FindShortestSeparator(std::string* /*start*/,
|
|
|
|
const Slice& /*limit*/) const override {}
|
|
|
|
const char* Name() const override { return "my_comparator"; }
|
|
|
|
};
|
|
|
|
|
|
|
|
TEST_F(LdbCmdTest, CustomComparator) {
|
|
|
|
Env* env = TryLoadCustomOrDefaultEnv();
|
|
|
|
MyComparator my_comparator;
|
|
|
|
Options opts;
|
|
|
|
opts.env = env;
|
|
|
|
opts.create_if_missing = true;
|
|
|
|
opts.create_missing_column_families = true;
|
|
|
|
opts.comparator = &my_comparator;
|
|
|
|
|
|
|
|
std::string dbname = test::PerThreadDBPath(env, "ldb_cmd_test");
|
|
|
|
DB* db = nullptr;
|
|
|
|
|
2024-01-25 00:16:18 +00:00
|
|
|
std::vector<ColumnFamilyDescriptor> cfds = {
|
|
|
|
{kDefaultColumnFamilyName, opts}, {"cf1", opts}, {"cf2", opts}};
|
2023-12-21 02:04:08 +00:00
|
|
|
std::vector<ColumnFamilyHandle*> handles;
|
|
|
|
ASSERT_OK(DestroyDB(dbname, opts));
|
|
|
|
ASSERT_OK(DB::Open(opts, dbname, cfds, &handles, &db));
|
|
|
|
ASSERT_OK(db->Put(WriteOptions(), "k1", "v1"));
|
|
|
|
|
|
|
|
for (auto& h : handles) {
|
|
|
|
ASSERT_OK(db->DestroyColumnFamilyHandle(h));
|
|
|
|
}
|
|
|
|
delete db;
|
|
|
|
|
|
|
|
char arg1[] = "./ldb";
|
|
|
|
std::string arg2 = "--db=" + dbname;
|
|
|
|
char arg3[] = "get";
|
|
|
|
char arg4[] = "k1";
|
|
|
|
char* argv[] = {arg1, const_cast<char*>(arg2.c_str()), arg3, arg4};
|
|
|
|
|
|
|
|
ASSERT_EQ(0,
|
2024-01-25 00:16:18 +00:00
|
|
|
LDBCommandRunner::RunCommand(4, argv, opts, LDBOptions(), &cfds));
|
2023-12-21 02:04:08 +00:00
|
|
|
}
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
} // namespace ROCKSDB_NAMESPACE
|
2019-01-03 19:11:09 +00:00
|
|
|
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
int main(int argc, char** argv) {
|
2020-02-20 20:07:53 +00:00
|
|
|
ROCKSDB_NAMESPACE::port::InstallStackTraceHandler();
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
::testing::InitGoogleTest(&argc, argv);
|
2019-10-09 02:17:39 +00:00
|
|
|
RegisterCustomObjects(argc, argv);
|
Remove ldb HexToString method's usage of sscanf
Summary:
Fix hex2String performance issues by removing sscanf dependency.
Also fixed some edge case handling (odd length, bad input).
Test Plan: Created a test file which called old and new implementation, and validated results are the same. I'll paste results in the phabricator diff.
Reviewers: igor, rven, anthony, IslamAbdelRahman, kradhakrishnan, yhchiang, sdong
Reviewed By: sdong
Subscribers: thatsafunnyname, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D46785
2015-09-23 21:25:46 +00:00
|
|
|
return RUN_ALL_TESTS();
|
|
|
|
}
|