2019-04-18 17:51:19 +00:00
|
|
|
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
|
2015-07-20 18:24:54 +00:00
|
|
|
#ifndef ROCKSDB_LITE
|
|
|
|
|
2017-04-06 21:49:13 +00:00
|
|
|
#include <map>
|
2013-08-13 17:56:20 +00:00
|
|
|
#include <memory>
|
2013-08-23 15:38:13 +00:00
|
|
|
#include "rocksdb/compaction_filter.h"
|
2014-07-23 14:21:38 +00:00
|
|
|
#include "rocksdb/utilities/db_ttl.h"
|
2019-05-30 18:21:38 +00:00
|
|
|
#include "test_util/testharness.h"
|
2019-05-31 00:39:43 +00:00
|
|
|
#include "util/string_util.h"
|
2015-07-01 23:13:49 +00:00
|
|
|
#ifndef OS_WIN
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
#include <unistd.h>
|
2015-07-01 23:13:49 +00:00
|
|
|
#endif
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
namespace ROCKSDB_NAMESPACE {
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
namespace {
|
2013-08-03 07:40:22 +00:00
|
|
|
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
typedef std::map<std::string, std::string> KVMap;
|
2013-08-03 07:40:22 +00:00
|
|
|
|
2015-07-13 19:11:05 +00:00
|
|
|
enum BatchOperation { OP_PUT = 0, OP_DELETE = 1 };
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
2014-05-02 14:13:51 +00:00
|
|
|
class SpecialTimeEnv : public EnvWrapper {
|
|
|
|
public:
|
|
|
|
explicit SpecialTimeEnv(Env* base) : EnvWrapper(base) {
|
2020-12-23 07:44:44 +00:00
|
|
|
EXPECT_OK(base->GetCurrentTime(¤t_time_));
|
2014-05-02 14:13:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void Sleep(int64_t sleep_time) { current_time_ += sleep_time; }
|
2019-02-14 21:52:47 +00:00
|
|
|
Status GetCurrentTime(int64_t* current_time) override {
|
2014-05-02 14:13:51 +00:00
|
|
|
*current_time = current_time_;
|
|
|
|
return Status::OK();
|
|
|
|
}
|
|
|
|
|
|
|
|
private:
|
2017-07-05 19:02:00 +00:00
|
|
|
int64_t current_time_ = 0;
|
2014-05-02 14:13:51 +00:00
|
|
|
};
|
|
|
|
|
2015-03-17 21:08:00 +00:00
|
|
|
class TtlTest : public testing::Test {
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
public:
|
|
|
|
TtlTest() {
|
2014-05-02 14:13:51 +00:00
|
|
|
env_.reset(new SpecialTimeEnv(Env::Default()));
|
2018-07-14 00:18:39 +00:00
|
|
|
dbname_ = test::PerThreadDBPath("db_ttl");
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
options_.create_if_missing = true;
|
2014-05-02 14:13:51 +00:00
|
|
|
options_.env = env_.get();
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// ensure that compaction is kicked in to always strip timestamp from kvs
|
2016-06-16 23:02:52 +00:00
|
|
|
options_.max_compaction_bytes = 1;
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// compaction should take place always from level0 for determinism
|
|
|
|
db_ttl_ = nullptr;
|
|
|
|
DestroyDB(dbname_, Options());
|
|
|
|
}
|
|
|
|
|
2019-02-14 21:52:47 +00:00
|
|
|
~TtlTest() override {
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
DestroyDB(dbname_, Options());
|
|
|
|
}
|
|
|
|
|
|
|
|
// Open database with TTL support when TTL not provided with db_ttl_ pointer
|
|
|
|
void OpenTtl() {
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_TRUE(db_ttl_ ==
|
|
|
|
nullptr); // db should be closed before opening again
|
2014-04-29 03:34:20 +00:00
|
|
|
ASSERT_OK(DBWithTTL::Open(options_, dbname_, &db_ttl_));
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Open database with TTL support when TTL provided with db_ttl_ pointer
|
|
|
|
void OpenTtl(int32_t ttl) {
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_TRUE(db_ttl_ == nullptr);
|
2014-04-29 03:34:20 +00:00
|
|
|
ASSERT_OK(DBWithTTL::Open(options_, dbname_, &db_ttl_, ttl));
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
2013-08-03 07:40:22 +00:00
|
|
|
// Open with TestFilter compaction filter
|
|
|
|
void OpenTtlWithTestCompaction(int32_t ttl) {
|
2013-08-13 17:56:20 +00:00
|
|
|
options_.compaction_filter_factory =
|
|
|
|
std::shared_ptr<CompactionFilterFactory>(
|
|
|
|
new TestFilterFactory(kSampleSize_, kNewValue_));
|
2013-08-03 07:40:22 +00:00
|
|
|
OpenTtl(ttl);
|
|
|
|
}
|
|
|
|
|
2013-05-10 21:19:47 +00:00
|
|
|
// Open database with TTL support in read_only mode
|
|
|
|
void OpenReadOnlyTtl(int32_t ttl) {
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_TRUE(db_ttl_ == nullptr);
|
2014-04-29 03:34:20 +00:00
|
|
|
ASSERT_OK(DBWithTTL::Open(options_, dbname_, &db_ttl_, ttl, true));
|
2013-05-10 21:19:47 +00:00
|
|
|
}
|
|
|
|
|
2019-06-20 20:04:13 +00:00
|
|
|
// Call db_ttl_->Close() before delete db_ttl_
|
2019-09-20 19:00:55 +00:00
|
|
|
void CloseTtl() { CloseTtlHelper(true); }
|
2019-06-20 20:04:13 +00:00
|
|
|
|
|
|
|
// No db_ttl_->Close() before delete db_ttl_
|
2019-09-20 19:00:55 +00:00
|
|
|
void CloseTtlNoDBClose() { CloseTtlHelper(false); }
|
2019-06-20 20:04:13 +00:00
|
|
|
|
|
|
|
void CloseTtlHelper(bool close_db) {
|
|
|
|
if (db_ttl_ != nullptr) {
|
|
|
|
if (close_db) {
|
|
|
|
db_ttl_->Close();
|
|
|
|
}
|
|
|
|
delete db_ttl_;
|
|
|
|
db_ttl_ = nullptr;
|
|
|
|
}
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Populates and returns a kv-map
|
|
|
|
void MakeKVMap(int64_t num_entries) {
|
|
|
|
kvmap_.clear();
|
2013-08-03 07:40:22 +00:00
|
|
|
int digits = 1;
|
2014-11-11 21:47:22 +00:00
|
|
|
for (int64_t dummy = num_entries; dummy /= 10; ++digits) {
|
|
|
|
}
|
2013-08-03 07:40:22 +00:00
|
|
|
int digits_in_i = 1;
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
for (int64_t i = 0; i < num_entries; i++) {
|
|
|
|
std::string key = "key";
|
|
|
|
std::string value = "value";
|
2013-08-03 07:40:22 +00:00
|
|
|
if (i % 10 == 0) {
|
|
|
|
digits_in_i++;
|
|
|
|
}
|
|
|
|
for(int j = digits_in_i; j < digits; j++) {
|
|
|
|
key.append("0");
|
|
|
|
value.append("0");
|
|
|
|
}
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
AppendNumberTo(&key, i);
|
|
|
|
AppendNumberTo(&value, i);
|
|
|
|
kvmap_[key] = value;
|
|
|
|
}
|
2014-11-11 21:47:22 +00:00
|
|
|
ASSERT_EQ(static_cast<int64_t>(kvmap_.size()),
|
|
|
|
num_entries); // check all insertions done
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
2013-05-15 22:00:57 +00:00
|
|
|
// Makes a write-batch with key-vals from kvmap_ and 'Write''s it
|
2014-11-11 21:47:22 +00:00
|
|
|
void MakePutWriteBatch(const BatchOperation* batch_ops, int64_t num_ops) {
|
|
|
|
ASSERT_LE(num_ops, static_cast<int64_t>(kvmap_.size()));
|
2013-05-15 22:00:57 +00:00
|
|
|
static WriteOptions wopts;
|
|
|
|
static FlushOptions flush_opts;
|
|
|
|
WriteBatch batch;
|
|
|
|
kv_it_ = kvmap_.begin();
|
2014-11-11 21:47:22 +00:00
|
|
|
for (int64_t i = 0; i < num_ops && kv_it_ != kvmap_.end(); i++, ++kv_it_) {
|
2013-05-15 22:00:57 +00:00
|
|
|
switch (batch_ops[i]) {
|
2015-07-01 23:13:49 +00:00
|
|
|
case OP_PUT:
|
2013-05-15 22:00:57 +00:00
|
|
|
batch.Put(kv_it_->first, kv_it_->second);
|
|
|
|
break;
|
2015-07-01 23:13:49 +00:00
|
|
|
case OP_DELETE:
|
2013-05-15 22:00:57 +00:00
|
|
|
batch.Delete(kv_it_->first);
|
|
|
|
break;
|
|
|
|
default:
|
2017-07-17 04:23:33 +00:00
|
|
|
FAIL();
|
2013-05-15 22:00:57 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
db_ttl_->Write(wopts, &batch);
|
|
|
|
db_ttl_->Flush(flush_opts);
|
|
|
|
}
|
|
|
|
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// Puts num_entries starting from start_pos_map from kvmap_ into the database
|
2014-11-11 21:47:22 +00:00
|
|
|
void PutValues(int64_t start_pos_map, int64_t num_entries, bool flush = true,
|
2014-04-22 18:27:33 +00:00
|
|
|
ColumnFamilyHandle* cf = nullptr) {
|
|
|
|
ASSERT_TRUE(db_ttl_);
|
2014-11-11 21:47:22 +00:00
|
|
|
ASSERT_LE(start_pos_map + num_entries, static_cast<int64_t>(kvmap_.size()));
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
static WriteOptions wopts;
|
|
|
|
static FlushOptions flush_opts;
|
|
|
|
kv_it_ = kvmap_.begin();
|
|
|
|
advance(kv_it_, start_pos_map);
|
2014-11-11 21:47:22 +00:00
|
|
|
for (int64_t i = 0; kv_it_ != kvmap_.end() && i < num_entries;
|
|
|
|
i++, ++kv_it_) {
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_OK(cf == nullptr
|
|
|
|
? db_ttl_->Put(wopts, kv_it_->first, kv_it_->second)
|
|
|
|
: db_ttl_->Put(wopts, cf, kv_it_->first, kv_it_->second));
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
// Put a mock kv at the end because CompactionFilter doesn't delete last key
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_OK(cf == nullptr ? db_ttl_->Put(wopts, "keymock", "valuemock")
|
|
|
|
: db_ttl_->Put(wopts, cf, "keymock", "valuemock"));
|
2013-09-02 04:04:20 +00:00
|
|
|
if (flush) {
|
2014-04-22 18:27:33 +00:00
|
|
|
if (cf == nullptr) {
|
|
|
|
db_ttl_->Flush(flush_opts);
|
|
|
|
} else {
|
|
|
|
db_ttl_->Flush(flush_opts, cf);
|
|
|
|
}
|
2013-09-02 04:04:20 +00:00
|
|
|
}
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Runs a manual compaction
|
2014-04-22 18:27:33 +00:00
|
|
|
void ManualCompact(ColumnFamilyHandle* cf = nullptr) {
|
|
|
|
if (cf == nullptr) {
|
2015-06-17 21:36:14 +00:00
|
|
|
db_ttl_->CompactRange(CompactRangeOptions(), nullptr, nullptr);
|
2014-04-22 18:27:33 +00:00
|
|
|
} else {
|
2015-06-17 21:36:14 +00:00
|
|
|
db_ttl_->CompactRange(CompactRangeOptions(), cf, nullptr, nullptr);
|
2014-04-22 18:27:33 +00:00
|
|
|
}
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
|
2013-08-24 00:36:00 +00:00
|
|
|
// checks the whole kvmap_ to return correct values using KeyMayExist
|
|
|
|
void SimpleKeyMayExistCheck() {
|
|
|
|
static ReadOptions ropts;
|
|
|
|
bool value_found;
|
|
|
|
std::string val;
|
|
|
|
for(auto &kv : kvmap_) {
|
|
|
|
bool ret = db_ttl_->KeyMayExist(ropts, kv.first, &val, &value_found);
|
|
|
|
if (ret == false || value_found == false) {
|
|
|
|
fprintf(stderr, "KeyMayExist could not find key=%s in the database but"
|
|
|
|
" should have\n", kv.first.c_str());
|
2017-07-17 04:23:33 +00:00
|
|
|
FAIL();
|
2013-08-24 00:36:00 +00:00
|
|
|
} else if (val.compare(kv.second) != 0) {
|
|
|
|
fprintf(stderr, " value for key=%s present in database is %s but"
|
|
|
|
" should be %s\n", kv.first.c_str(), val.c_str(),
|
|
|
|
kv.second.c_str());
|
2017-07-17 04:23:33 +00:00
|
|
|
FAIL();
|
2013-08-24 00:36:00 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-12-20 11:46:37 +00:00
|
|
|
// checks the whole kvmap_ to return correct values using MultiGet
|
|
|
|
void SimpleMultiGetTest() {
|
|
|
|
static ReadOptions ropts;
|
|
|
|
std::vector<Slice> keys;
|
|
|
|
std::vector<std::string> values;
|
|
|
|
|
|
|
|
for (auto& kv : kvmap_) {
|
|
|
|
keys.emplace_back(kv.first);
|
|
|
|
}
|
|
|
|
|
|
|
|
auto statuses = db_ttl_->MultiGet(ropts, keys, &values);
|
|
|
|
size_t i = 0;
|
|
|
|
for (auto& kv : kvmap_) {
|
|
|
|
ASSERT_OK(statuses[i]);
|
|
|
|
ASSERT_EQ(values[i], kv.second);
|
|
|
|
++i;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// Sleeps for slp_tim then runs a manual compaction
|
|
|
|
// Checks span starting from st_pos from kvmap_ in the db and
|
|
|
|
// Gets should return true if check is true and false otherwise
|
2013-08-03 07:40:22 +00:00
|
|
|
// Also checks that value that we got is the same as inserted; and =kNewValue
|
|
|
|
// if test_compaction_change is true
|
2014-11-11 21:47:22 +00:00
|
|
|
void SleepCompactCheck(int slp_tim, int64_t st_pos, int64_t span,
|
|
|
|
bool check = true, bool test_compaction_change = false,
|
2014-04-22 18:27:33 +00:00
|
|
|
ColumnFamilyHandle* cf = nullptr) {
|
|
|
|
ASSERT_TRUE(db_ttl_);
|
2014-05-02 14:13:51 +00:00
|
|
|
|
|
|
|
env_->Sleep(slp_tim);
|
2014-04-22 18:27:33 +00:00
|
|
|
ManualCompact(cf);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
static ReadOptions ropts;
|
|
|
|
kv_it_ = kvmap_.begin();
|
|
|
|
advance(kv_it_, st_pos);
|
|
|
|
std::string v;
|
2014-11-11 21:47:22 +00:00
|
|
|
for (int64_t i = 0; kv_it_ != kvmap_.end() && i < span; i++, ++kv_it_) {
|
2014-04-22 18:27:33 +00:00
|
|
|
Status s = (cf == nullptr) ? db_ttl_->Get(ropts, kv_it_->first, &v)
|
|
|
|
: db_ttl_->Get(ropts, cf, kv_it_->first, &v);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
if (s.ok() != check) {
|
2013-08-03 07:40:22 +00:00
|
|
|
fprintf(stderr, "key=%s ", kv_it_->first.c_str());
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
if (!s.ok()) {
|
|
|
|
fprintf(stderr, "is absent from db but was expected to be present\n");
|
|
|
|
} else {
|
|
|
|
fprintf(stderr, "is present in db but was expected to be absent\n");
|
|
|
|
}
|
2017-07-17 04:23:33 +00:00
|
|
|
FAIL();
|
2013-08-03 07:40:22 +00:00
|
|
|
} else if (s.ok()) {
|
|
|
|
if (test_compaction_change && v.compare(kNewValue_) != 0) {
|
|
|
|
fprintf(stderr, " value for key=%s present in database is %s but "
|
|
|
|
" should be %s\n", kv_it_->first.c_str(), v.c_str(),
|
|
|
|
kNewValue_.c_str());
|
2017-07-17 04:23:33 +00:00
|
|
|
FAIL();
|
2013-08-03 07:40:22 +00:00
|
|
|
} else if (!test_compaction_change && v.compare(kv_it_->second) !=0) {
|
|
|
|
fprintf(stderr, " value for key=%s present in database is %s but "
|
|
|
|
" should be %s\n", kv_it_->first.c_str(), v.c_str(),
|
|
|
|
kv_it_->second.c_str());
|
2017-07-17 04:23:33 +00:00
|
|
|
FAIL();
|
2013-08-03 07:40:22 +00:00
|
|
|
}
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// Similar as SleepCompactCheck but uses TtlIterator to read from db
|
2014-11-11 21:47:22 +00:00
|
|
|
void SleepCompactCheckIter(int slp, int st_pos, int64_t span,
|
|
|
|
bool check = true) {
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_TRUE(db_ttl_);
|
2014-05-02 14:13:51 +00:00
|
|
|
env_->Sleep(slp);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
ManualCompact();
|
|
|
|
static ReadOptions ropts;
|
|
|
|
Iterator *dbiter = db_ttl_->NewIterator(ropts);
|
|
|
|
kv_it_ = kvmap_.begin();
|
|
|
|
advance(kv_it_, st_pos);
|
|
|
|
|
|
|
|
dbiter->Seek(kv_it_->first);
|
|
|
|
if (!check) {
|
|
|
|
if (dbiter->Valid()) {
|
|
|
|
ASSERT_NE(dbiter->value().compare(kv_it_->second), 0);
|
|
|
|
}
|
|
|
|
} else { // dbiter should have found out kvmap_[st_pos]
|
2014-11-11 21:47:22 +00:00
|
|
|
for (int64_t i = st_pos; kv_it_ != kvmap_.end() && i < st_pos + span;
|
|
|
|
i++, ++kv_it_) {
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
ASSERT_TRUE(dbiter->Valid());
|
|
|
|
ASSERT_EQ(dbiter->value().compare(kv_it_->second), 0);
|
|
|
|
dbiter->Next();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
delete dbiter;
|
|
|
|
}
|
|
|
|
|
2018-01-18 18:41:26 +00:00
|
|
|
// Set ttl on open db
|
|
|
|
void SetTtl(int32_t ttl, ColumnFamilyHandle* cf = nullptr) {
|
|
|
|
ASSERT_TRUE(db_ttl_);
|
|
|
|
cf == nullptr ? db_ttl_->SetTtl(ttl) : db_ttl_->SetTtl(cf, ttl);
|
|
|
|
}
|
|
|
|
|
2013-08-03 07:40:22 +00:00
|
|
|
class TestFilter : public CompactionFilter {
|
|
|
|
public:
|
2014-10-01 09:53:00 +00:00
|
|
|
TestFilter(const int64_t kSampleSize, const std::string& kNewValue)
|
2013-08-03 07:40:22 +00:00
|
|
|
: kSampleSize_(kSampleSize),
|
|
|
|
kNewValue_(kNewValue) {
|
|
|
|
}
|
|
|
|
|
|
|
|
// Works on keys of the form "key<number>"
|
|
|
|
// Drops key if number at the end of key is in [0, kSampleSize_/3),
|
|
|
|
// Keeps key if it is in [kSampleSize_/3, 2*kSampleSize_/3),
|
|
|
|
// Change value if it is in [2*kSampleSize_/3, kSampleSize_)
|
|
|
|
// Eg. kSampleSize_=6. Drop:key0-1...Keep:key2-3...Change:key4-5...
|
2019-02-14 21:52:47 +00:00
|
|
|
bool Filter(int /*level*/, const Slice& key, const Slice& /*value*/,
|
|
|
|
std::string* new_value, bool* value_changed) const override {
|
2013-08-03 07:40:22 +00:00
|
|
|
assert(new_value != nullptr);
|
|
|
|
|
|
|
|
std::string search_str = "0123456789";
|
|
|
|
std::string key_string = key.ToString();
|
|
|
|
size_t pos = key_string.find_first_of(search_str);
|
|
|
|
int num_key_end;
|
|
|
|
if (pos != std::string::npos) {
|
2015-06-08 18:43:55 +00:00
|
|
|
auto key_substr = key_string.substr(pos, key.size() - pos);
|
|
|
|
#ifndef CYGWIN
|
|
|
|
num_key_end = std::stoi(key_substr);
|
|
|
|
#else
|
|
|
|
num_key_end = std::strtol(key_substr.c_str(), 0, 10);
|
|
|
|
#endif
|
|
|
|
|
2013-08-03 07:40:22 +00:00
|
|
|
} else {
|
|
|
|
return false; // Keep keys not matching the format "key<NUMBER>"
|
|
|
|
}
|
|
|
|
|
2014-11-11 21:47:22 +00:00
|
|
|
int64_t partition = kSampleSize_ / 3;
|
2013-08-03 07:40:22 +00:00
|
|
|
if (num_key_end < partition) {
|
|
|
|
return true;
|
|
|
|
} else if (num_key_end < partition * 2) {
|
|
|
|
return false;
|
|
|
|
} else {
|
|
|
|
*new_value = kNewValue_;
|
|
|
|
*value_changed = true;
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-02-14 21:52:47 +00:00
|
|
|
const char* Name() const override { return "TestFilter"; }
|
2013-08-03 07:40:22 +00:00
|
|
|
|
|
|
|
private:
|
|
|
|
const int64_t kSampleSize_;
|
|
|
|
const std::string kNewValue_;
|
|
|
|
};
|
|
|
|
|
2013-08-13 17:56:20 +00:00
|
|
|
class TestFilterFactory : public CompactionFilterFactory {
|
|
|
|
public:
|
2014-10-01 09:53:00 +00:00
|
|
|
TestFilterFactory(const int64_t kSampleSize, const std::string& kNewValue)
|
2013-08-13 17:56:20 +00:00
|
|
|
: kSampleSize_(kSampleSize),
|
|
|
|
kNewValue_(kNewValue) {
|
|
|
|
}
|
|
|
|
|
2019-02-14 21:52:47 +00:00
|
|
|
std::unique_ptr<CompactionFilter> CreateCompactionFilter(
|
2018-03-05 21:08:17 +00:00
|
|
|
const CompactionFilter::Context& /*context*/) override {
|
2013-08-13 17:56:20 +00:00
|
|
|
return std::unique_ptr<CompactionFilter>(
|
|
|
|
new TestFilter(kSampleSize_, kNewValue_));
|
|
|
|
}
|
|
|
|
|
2019-02-14 21:52:47 +00:00
|
|
|
const char* Name() const override { return "TestFilterFactory"; }
|
2013-08-13 17:56:20 +00:00
|
|
|
|
2019-02-14 21:52:47 +00:00
|
|
|
private:
|
2013-08-13 17:56:20 +00:00
|
|
|
const int64_t kSampleSize_;
|
|
|
|
const std::string kNewValue_;
|
|
|
|
};
|
|
|
|
|
|
|
|
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// Choose carefully so that Put, Gets & Compaction complete in 1 second buffer
|
2015-07-01 23:13:49 +00:00
|
|
|
static const int64_t kSampleSize_ = 100;
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
std::string dbname_;
|
2014-04-29 03:34:20 +00:00
|
|
|
DBWithTTL* db_ttl_;
|
2018-11-09 19:17:34 +00:00
|
|
|
std::unique_ptr<SpecialTimeEnv> env_;
|
2014-04-22 18:27:33 +00:00
|
|
|
|
|
|
|
private:
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
Options options_;
|
|
|
|
KVMap kvmap_;
|
|
|
|
KVMap::iterator kv_it_;
|
2013-08-03 07:40:22 +00:00
|
|
|
const std::string kNewValue_ = "new_value";
|
2018-11-09 19:17:34 +00:00
|
|
|
std::unique_ptr<CompactionFilter> test_comp_filter_;
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}; // class TtlTest
|
|
|
|
|
|
|
|
// If TTL is non positive or not provided, the behaviour is TTL = infinity
|
|
|
|
// This test opens the db 3 times with such default behavior and inserts a
|
2013-11-16 11:21:34 +00:00
|
|
|
// bunch of kvs each time. All kvs should accumulate in the db till the end
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// Partitions the sample-size provided into 3 sets over boundary1 and boundary2
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, NoEffect) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
2014-11-11 21:47:22 +00:00
|
|
|
int64_t boundary1 = kSampleSize_ / 3;
|
|
|
|
int64_t boundary2 = 2 * boundary1;
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl();
|
|
|
|
PutValues(0, boundary1); //T=0: Set1 never deleted
|
|
|
|
SleepCompactCheck(1, 0, boundary1); //T=1: Set1 still there
|
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenTtl(0);
|
|
|
|
PutValues(boundary1, boundary2 - boundary1); //T=1: Set2 never deleted
|
|
|
|
SleepCompactCheck(1, 0, boundary2); //T=2: Sets1 & 2 still there
|
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenTtl(-1);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(boundary2, kSampleSize_ - boundary2); //T=3: Set3 never deleted
|
|
|
|
SleepCompactCheck(1, 0, kSampleSize_, true); //T=4: Sets 1,2,3 still there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
2019-06-20 20:04:13 +00:00
|
|
|
// Rerun the NoEffect test with a different version of CloseTtl
|
|
|
|
// function, where db is directly deleted without close.
|
|
|
|
TEST_F(TtlTest, DestructWithoutClose) {
|
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
int64_t boundary1 = kSampleSize_ / 3;
|
|
|
|
int64_t boundary2 = 2 * boundary1;
|
|
|
|
|
|
|
|
OpenTtl();
|
2019-09-20 19:00:55 +00:00
|
|
|
PutValues(0, boundary1); // T=0: Set1 never deleted
|
|
|
|
SleepCompactCheck(1, 0, boundary1); // T=1: Set1 still there
|
2019-06-20 20:04:13 +00:00
|
|
|
CloseTtlNoDBClose();
|
|
|
|
|
|
|
|
OpenTtl(0);
|
2019-09-20 19:00:55 +00:00
|
|
|
PutValues(boundary1, boundary2 - boundary1); // T=1: Set2 never deleted
|
|
|
|
SleepCompactCheck(1, 0, boundary2); // T=2: Sets1 & 2 still there
|
2019-06-20 20:04:13 +00:00
|
|
|
CloseTtlNoDBClose();
|
|
|
|
|
|
|
|
OpenTtl(-1);
|
2019-09-20 19:00:55 +00:00
|
|
|
PutValues(boundary2, kSampleSize_ - boundary2); // T=3: Set3 never deleted
|
|
|
|
SleepCompactCheck(1, 0, kSampleSize_, true); // T=4: Sets 1,2,3 still there
|
2019-06-20 20:04:13 +00:00
|
|
|
CloseTtlNoDBClose();
|
|
|
|
}
|
|
|
|
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
// Puts a set of values and checks its presence using Get during ttl
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, PresentDuringTTL) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(2); // T=0:Open the db with ttl = 2
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0:Insert Set1. Delete at t=2
|
|
|
|
SleepCompactCheck(1, 0, kSampleSize_, true); // T=1:Set1 should still be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Puts a set of values and checks its absence using Get after ttl
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, AbsentAfterTTL) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(1); // T=0:Open the db with ttl = 2
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0:Insert Set1. Delete at t=2
|
|
|
|
SleepCompactCheck(2, 0, kSampleSize_, false); // T=2:Set1 should not be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Resets the timestamp of a set of kvs by updating them and checks that they
|
|
|
|
// are not deleted according to the old timestamp
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, ResetTimestamp) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(3);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0: Insert Set1. Delete at t=3
|
2014-05-02 14:13:51 +00:00
|
|
|
env_->Sleep(2); // T=2
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=2: Insert Set1. Delete at t=5
|
|
|
|
SleepCompactCheck(2, 0, kSampleSize_); // T=4: Set1 should still be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Similar to PresentDuringTTL but uses Iterator
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, IterPresentDuringTTL) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(2);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0: Insert. Delete at t=2
|
|
|
|
SleepCompactCheckIter(1, 0, kSampleSize_); // T=1: Set should be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Similar to AbsentAfterTTL but uses Iterator
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, IterAbsentAfterTTL) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(1);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0: Insert. Delete at t=1
|
|
|
|
SleepCompactCheckIter(2, 0, kSampleSize_, false); // T=2: Should not be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Checks presence while opening the same db more than once with the same ttl
|
|
|
|
// Note: The second open will open the same db
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, MultiOpenSamePresent) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(2);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0: Insert. Delete at t=2
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenTtl(2); // T=0. Delete at t=2
|
2013-08-03 07:40:22 +00:00
|
|
|
SleepCompactCheck(1, 0, kSampleSize_); // T=1: Set should be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Checks absence while opening the same db more than once with the same ttl
|
|
|
|
// Note: The second open will open the same db
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, MultiOpenSameAbsent) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(1);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0: Insert. Delete at t=1
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenTtl(1); // T=0.Delete at t=1
|
2013-08-03 07:40:22 +00:00
|
|
|
SleepCompactCheck(2, 0, kSampleSize_, false); // T=2: Set should not be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Checks presence while opening the same db more than once with bigger ttl
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, MultiOpenDifferent) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
OpenTtl(1);
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0: Insert. Delete at t=1
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenTtl(3); // T=0: Set deleted at t=3
|
2013-08-03 07:40:22 +00:00
|
|
|
SleepCompactCheck(2, 0, kSampleSize_); // T=2: Set should be there
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
2013-05-10 21:19:47 +00:00
|
|
|
// Checks presence during ttl in read_only mode
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, ReadOnlyPresentForever) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
2013-05-10 21:19:47 +00:00
|
|
|
|
|
|
|
OpenTtl(1); // T=0:Open the db normally
|
2013-08-03 07:40:22 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0:Insert Set1. Delete at t=1
|
2013-05-10 21:19:47 +00:00
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenReadOnlyTtl(1);
|
2013-08-03 07:40:22 +00:00
|
|
|
SleepCompactCheck(2, 0, kSampleSize_); // T=2:Set1 should still be there
|
2013-05-15 22:00:57 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Checks whether WriteBatch works well with TTL
|
|
|
|
// Puts all kvs in kvmap_ in a batch and writes first, then deletes first half
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, WriteBatchTest) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
BatchOperation batch_ops[kSampleSize_];
|
|
|
|
for (int i = 0; i < kSampleSize_; i++) {
|
2015-07-01 23:13:49 +00:00
|
|
|
batch_ops[i] = OP_PUT;
|
2013-05-15 22:00:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
OpenTtl(2);
|
2013-08-03 07:40:22 +00:00
|
|
|
MakePutWriteBatch(batch_ops, kSampleSize_);
|
|
|
|
for (int i = 0; i < kSampleSize_ / 2; i++) {
|
2015-07-01 23:13:49 +00:00
|
|
|
batch_ops[i] = OP_DELETE;
|
2013-05-15 22:00:57 +00:00
|
|
|
}
|
2013-08-03 07:40:22 +00:00
|
|
|
MakePutWriteBatch(batch_ops, kSampleSize_ / 2);
|
|
|
|
SleepCompactCheck(0, 0, kSampleSize_ / 2, false);
|
|
|
|
SleepCompactCheck(0, kSampleSize_ / 2, kSampleSize_ - kSampleSize_ / 2);
|
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
|
|
|
// Checks user's compaction filter for correctness with TTL logic
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, CompactionFilter) {
|
2013-08-03 07:40:22 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
|
|
|
|
OpenTtlWithTestCompaction(1);
|
|
|
|
PutValues(0, kSampleSize_); // T=0:Insert Set1. Delete at t=1
|
|
|
|
// T=2: TTL logic takes precedence over TestFilter:-Set1 should not be there
|
|
|
|
SleepCompactCheck(2, 0, kSampleSize_, false);
|
|
|
|
CloseTtl();
|
|
|
|
|
|
|
|
OpenTtlWithTestCompaction(3);
|
|
|
|
PutValues(0, kSampleSize_); // T=0:Insert Set1.
|
2014-11-11 21:47:22 +00:00
|
|
|
int64_t partition = kSampleSize_ / 3;
|
|
|
|
SleepCompactCheck(1, 0, partition, false); // Part dropped
|
|
|
|
SleepCompactCheck(0, partition, partition); // Part kept
|
2013-08-03 07:40:22 +00:00
|
|
|
SleepCompactCheck(0, 2 * partition, partition, true, true); // Part changed
|
2013-05-10 21:19:47 +00:00
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
2013-08-24 00:36:00 +00:00
|
|
|
// Insert some key-values which KeyMayExist should be able to get and check that
|
|
|
|
// values returned are fine
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, KeyMayExist) {
|
2013-08-24 00:36:00 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
|
|
|
|
OpenTtl();
|
2013-09-02 04:04:20 +00:00
|
|
|
PutValues(0, kSampleSize_, false);
|
2013-08-24 00:36:00 +00:00
|
|
|
|
|
|
|
SimpleKeyMayExistCheck();
|
|
|
|
|
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, MultiGetTest) {
|
2014-12-20 11:46:37 +00:00
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
|
|
|
|
OpenTtl();
|
|
|
|
PutValues(0, kSampleSize_, false);
|
|
|
|
|
|
|
|
SimpleMultiGetTest();
|
|
|
|
|
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
2015-03-17 21:08:00 +00:00
|
|
|
TEST_F(TtlTest, ColumnFamiliesTest) {
|
2014-04-22 18:27:33 +00:00
|
|
|
DB* db;
|
|
|
|
Options options;
|
|
|
|
options.create_if_missing = true;
|
2014-05-02 14:13:51 +00:00
|
|
|
options.env = env_.get();
|
2014-04-22 18:27:33 +00:00
|
|
|
|
|
|
|
DB::Open(options, dbname_, &db);
|
|
|
|
ColumnFamilyHandle* handle;
|
|
|
|
ASSERT_OK(db->CreateColumnFamily(ColumnFamilyOptions(options),
|
|
|
|
"ttl_column_family", &handle));
|
|
|
|
|
|
|
|
delete handle;
|
|
|
|
delete db;
|
|
|
|
|
|
|
|
std::vector<ColumnFamilyDescriptor> column_families;
|
|
|
|
column_families.push_back(ColumnFamilyDescriptor(
|
|
|
|
kDefaultColumnFamilyName, ColumnFamilyOptions(options)));
|
|
|
|
column_families.push_back(ColumnFamilyDescriptor(
|
|
|
|
"ttl_column_family", ColumnFamilyOptions(options)));
|
|
|
|
|
|
|
|
std::vector<ColumnFamilyHandle*> handles;
|
|
|
|
|
2014-04-29 03:34:20 +00:00
|
|
|
ASSERT_OK(DBWithTTL::Open(DBOptions(options), dbname_, column_families,
|
2014-04-29 19:19:54 +00:00
|
|
|
&handles, &db_ttl_, {3, 5}, false));
|
2014-04-22 18:27:33 +00:00
|
|
|
ASSERT_EQ(handles.size(), 2U);
|
2014-04-29 03:34:20 +00:00
|
|
|
ColumnFamilyHandle* new_handle;
|
|
|
|
ASSERT_OK(db_ttl_->CreateColumnFamilyWithTtl(options, "ttl_column_family_2",
|
|
|
|
&new_handle, 2));
|
|
|
|
handles.push_back(new_handle);
|
2014-04-22 18:27:33 +00:00
|
|
|
|
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
PutValues(0, kSampleSize_, false, handles[0]);
|
|
|
|
PutValues(0, kSampleSize_, false, handles[1]);
|
2014-04-29 03:34:20 +00:00
|
|
|
PutValues(0, kSampleSize_, false, handles[2]);
|
2014-04-22 18:27:33 +00:00
|
|
|
|
|
|
|
// everything should be there after 1 second
|
|
|
|
SleepCompactCheck(1, 0, kSampleSize_, true, false, handles[0]);
|
|
|
|
SleepCompactCheck(0, 0, kSampleSize_, true, false, handles[1]);
|
2014-04-29 03:34:20 +00:00
|
|
|
SleepCompactCheck(0, 0, kSampleSize_, true, false, handles[2]);
|
2014-04-22 18:27:33 +00:00
|
|
|
|
2014-04-29 19:19:54 +00:00
|
|
|
// only column family 1 should be alive after 4 seconds
|
|
|
|
SleepCompactCheck(3, 0, kSampleSize_, false, false, handles[0]);
|
2014-04-22 18:27:33 +00:00
|
|
|
SleepCompactCheck(0, 0, kSampleSize_, true, false, handles[1]);
|
2014-04-29 03:34:20 +00:00
|
|
|
SleepCompactCheck(0, 0, kSampleSize_, false, false, handles[2]);
|
2014-04-22 18:27:33 +00:00
|
|
|
|
2014-04-29 19:19:54 +00:00
|
|
|
// nothing should be there after 6 seconds
|
2014-04-22 18:27:33 +00:00
|
|
|
SleepCompactCheck(2, 0, kSampleSize_, false, false, handles[0]);
|
|
|
|
SleepCompactCheck(0, 0, kSampleSize_, false, false, handles[1]);
|
2014-04-29 03:34:20 +00:00
|
|
|
SleepCompactCheck(0, 0, kSampleSize_, false, false, handles[2]);
|
2014-04-22 18:27:33 +00:00
|
|
|
|
|
|
|
for (auto h : handles) {
|
|
|
|
delete h;
|
|
|
|
}
|
|
|
|
delete db_ttl_;
|
|
|
|
db_ttl_ = nullptr;
|
|
|
|
}
|
|
|
|
|
2018-01-18 18:41:26 +00:00
|
|
|
// Puts a set of values and checks its absence using Get after ttl
|
|
|
|
TEST_F(TtlTest, ChangeTtlOnOpenDb) {
|
|
|
|
MakeKVMap(kSampleSize_);
|
|
|
|
|
|
|
|
OpenTtl(1); // T=0:Open the db with ttl = 2
|
|
|
|
SetTtl(3);
|
2020-04-20 18:37:36 +00:00
|
|
|
PutValues(0, kSampleSize_); // T=0:Insert Set1. Delete at t=2
|
2018-01-18 18:41:26 +00:00
|
|
|
SleepCompactCheck(2, 0, kSampleSize_, true); // T=2:Set1 should be there
|
|
|
|
CloseTtl();
|
|
|
|
}
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
} // namespace ROCKSDB_NAMESPACE
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
|
|
|
|
// A black-box test for the ttl wrapper around rocksdb
|
|
|
|
int main(int argc, char** argv) {
|
2015-03-17 21:08:00 +00:00
|
|
|
::testing::InitGoogleTest(&argc, argv);
|
|
|
|
return RUN_ALL_TESTS();
|
Timestamp and TTL Wrapper for rocksdb
Summary:
When opened with DBTimestamp::Open call, timestamps are prepended to and stripped from the value during subsequent Put and Get calls respectively. The Timestamp is used to discard values in Get and custom compaction filter which have exceeded their TTL which is specified during Open.
Have made a temporary change to Makefile to let us test with the temporary file TestTime.cc. Have also changed the private members of db_impl.h to protected to let them be inherited by the new class DBTimestamp
Test Plan: make db_timestamp; TestTime.cc(will not check it in) shows how to use the apis currently, but I will write unit-tests shortly
Reviewers: dhruba, vamsi, haobo, sheki, heyongqiang, vkrest
Reviewed By: vamsi
CC: zshao, xjin, vkrest, MarkCallaghan
Differential Revision: https://reviews.facebook.net/D10311
2013-04-15 20:33:13 +00:00
|
|
|
}
|
2015-07-20 18:24:54 +00:00
|
|
|
|
|
|
|
#else
|
|
|
|
#include <stdio.h>
|
|
|
|
|
2018-04-16 00:19:57 +00:00
|
|
|
int main(int /*argc*/, char** /*argv*/) {
|
2015-07-20 18:24:54 +00:00
|
|
|
fprintf(stderr, "SKIPPED as DBWithTTL is not supported in ROCKSDB_LITE\n");
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif // !ROCKSDB_LITE
|