2013-10-16 21:59:46 +00:00
|
|
|
// Copyright (c) 2013, Facebook, Inc. All rights reserved.
|
|
|
|
// This source code is licensed under the BSD-style license found in the
|
|
|
|
// LICENSE file in the root directory of this source tree. An additional grant
|
|
|
|
// of patent rights can be found in the PATENTS file in the same directory.
|
|
|
|
//
|
2011-03-18 22:37:00 +00:00
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
|
2013-10-05 05:32:05 +00:00
|
|
|
#pragma once
|
2011-03-18 22:37:00 +00:00
|
|
|
#include <string>
|
2013-07-23 21:42:27 +00:00
|
|
|
#include <memory>
|
[RocksDB] [MergeOperator] The new Merge Interface! Uses merge sequences.
Summary:
Here are the major changes to the Merge Interface. It has been expanded
to handle cases where the MergeOperator is not associative. It does so by stacking
up merge operations while scanning through the key history (i.e.: during Get() or
Compaction), until a valid Put/Delete/end-of-history is encountered; it then
applies all of the merge operations in the correct sequence starting with the
base/sentinel value.
I have also introduced an "AssociativeMerge" function which allows the user to
take advantage of associative merge operations (such as in the case of counters).
The implementation will always attempt to merge the operations/operands themselves
together when they are encountered, and will resort to the "stacking" method if
and only if the "associative-merge" fails.
This implementation is conjectured to allow MergeOperator to handle the general
case, while still providing the user with the ability to take advantage of certain
efficiencies in their own merge-operator / data-structure.
NOTE: This is a preliminary diff. This must still go through a lot of review,
revision, and testing. Feedback welcome!
Test Plan:
-This is a preliminary diff. I have only just begun testing/debugging it.
-I will be testing this with the existing MergeOperator use-cases and unit-tests
(counters, string-append, and redis-lists)
-I will be "desk-checking" and walking through the code with the help gdb.
-I will find a way of stress-testing the new interface / implementation using
db_bench, db_test, merge_test, and/or db_stress.
-I will ensure that my tests cover all cases: Get-Memtable,
Get-Immutable-Memtable, Get-from-Disk, Iterator-Range-Scan, Flush-Memtable-to-L0,
Compaction-L0-L1, Compaction-Ln-L(n+1), Put/Delete found, Put/Delete not-found,
end-of-history, end-of-file, etc.
-A lot of feedback from the reviewers.
Reviewers: haobo, dhruba, zshao, emayanke
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D11499
2013-08-06 03:14:32 +00:00
|
|
|
#include <deque>
|
2011-03-18 22:37:00 +00:00
|
|
|
#include "db/dbformat.h"
|
|
|
|
#include "db/skiplist.h"
|
2012-10-19 21:00:53 +00:00
|
|
|
#include "db/version_set.h"
|
2013-08-23 15:38:13 +00:00
|
|
|
#include "rocksdb/db.h"
|
|
|
|
#include "rocksdb/memtablerep.h"
|
Make arena block size configurable
Summary:
Add an option for arena block size, default value 4096 bytes. Arena will allocate blocks with such size.
I am not sure about passing parameter to skiplist in the new virtualized framework, though I talked to Jim a bit. So add Jim as reviewer.
Test Plan:
new unit test, I am running db_test.
For passing paramter from configured option to Arena, I tried tests like:
TEST(DBTest, Arena_Option) {
std::string dbname = test::TmpDir() + "/db_arena_option_test";
DestroyDB(dbname, Options());
DB* db = nullptr;
Options opts;
opts.create_if_missing = true;
opts.arena_block_size = 1000000; // tested 99, 999999
Status s = DB::Open(opts, dbname, &db);
db->Put(WriteOptions(), "a", "123");
}
and printed some debug info. The results look good. Any suggestion for such a unit-test?
Reviewers: haobo, dhruba, emayanke, jpaton
Reviewed By: dhruba
CC: leveldb, zshao
Differential Revision: https://reviews.facebook.net/D11799
2013-07-31 19:42:23 +00:00
|
|
|
#include "util/arena_impl.h"
|
2011-03-18 22:37:00 +00:00
|
|
|
|
2013-10-04 04:49:15 +00:00
|
|
|
namespace rocksdb {
|
2011-03-18 22:37:00 +00:00
|
|
|
|
|
|
|
class Mutex;
|
|
|
|
class MemTableIterator;
|
|
|
|
|
|
|
|
class MemTable {
|
|
|
|
public:
|
2013-07-23 21:42:27 +00:00
|
|
|
struct KeyComparator : public MemTableRep::KeyComparator {
|
|
|
|
const InternalKeyComparator comparator;
|
|
|
|
explicit KeyComparator(const InternalKeyComparator& c) : comparator(c) { }
|
|
|
|
virtual int operator()(const char* a, const char* b) const;
|
|
|
|
};
|
|
|
|
|
2011-05-21 02:17:43 +00:00
|
|
|
// MemTables are reference counted. The initial reference count
|
|
|
|
// is zero and the caller must call Ref() at least once.
|
Make arena block size configurable
Summary:
Add an option for arena block size, default value 4096 bytes. Arena will allocate blocks with such size.
I am not sure about passing parameter to skiplist in the new virtualized framework, though I talked to Jim a bit. So add Jim as reviewer.
Test Plan:
new unit test, I am running db_test.
For passing paramter from configured option to Arena, I tried tests like:
TEST(DBTest, Arena_Option) {
std::string dbname = test::TmpDir() + "/db_arena_option_test";
DestroyDB(dbname, Options());
DB* db = nullptr;
Options opts;
opts.create_if_missing = true;
opts.arena_block_size = 1000000; // tested 99, 999999
Status s = DB::Open(opts, dbname, &db);
db->Put(WriteOptions(), "a", "123");
}
and printed some debug info. The results look good. Any suggestion for such a unit-test?
Reviewers: haobo, dhruba, emayanke, jpaton
Reviewed By: dhruba
CC: leveldb, zshao
Differential Revision: https://reviews.facebook.net/D11799
2013-07-31 19:42:23 +00:00
|
|
|
explicit MemTable(
|
|
|
|
const InternalKeyComparator& comparator,
|
|
|
|
std::shared_ptr<MemTableRepFactory> table_factory,
|
|
|
|
int numlevel = 7,
|
|
|
|
const Options& options = Options());
|
2011-05-21 02:17:43 +00:00
|
|
|
|
|
|
|
// Increase reference count.
|
|
|
|
void Ref() { ++refs_; }
|
|
|
|
|
|
|
|
// Drop reference count. Delete if no more references exist.
|
|
|
|
void Unref() {
|
|
|
|
--refs_;
|
|
|
|
assert(refs_ >= 0);
|
|
|
|
if (refs_ <= 0) {
|
|
|
|
delete this;
|
|
|
|
}
|
|
|
|
}
|
2011-03-18 22:37:00 +00:00
|
|
|
|
|
|
|
// Returns an estimate of the number of bytes of data in use by this
|
|
|
|
// data structure.
|
|
|
|
//
|
|
|
|
// REQUIRES: external synchronization to prevent simultaneous
|
|
|
|
// operations on the same MemTable.
|
|
|
|
size_t ApproximateMemoryUsage();
|
|
|
|
|
|
|
|
// Return an iterator that yields the contents of the memtable.
|
|
|
|
//
|
|
|
|
// The caller must ensure that the underlying MemTable remains live
|
|
|
|
// while the returned iterator is live. The keys returned by this
|
|
|
|
// iterator are internal keys encoded by AppendInternalKey in the
|
2013-01-04 01:13:56 +00:00
|
|
|
// db/dbformat.{h,cc} module.
|
2013-08-23 06:10:02 +00:00
|
|
|
//
|
|
|
|
// If a prefix is supplied, it is passed to the underlying MemTableRep as a
|
|
|
|
// hint that the iterator only need to support access to keys with that
|
|
|
|
// prefix.
|
|
|
|
Iterator* NewIterator(const Slice* prefix = nullptr);
|
2011-03-18 22:37:00 +00:00
|
|
|
|
|
|
|
// Add an entry into memtable that maps key to value at the
|
|
|
|
// specified sequence number and with the specified type.
|
|
|
|
// Typically value will be empty if type==kTypeDeletion.
|
|
|
|
void Add(SequenceNumber seq, ValueType type,
|
|
|
|
const Slice& key,
|
|
|
|
const Slice& value);
|
|
|
|
|
2011-06-22 02:36:45 +00:00
|
|
|
// If memtable contains a value for key, store it in *value and return true.
|
|
|
|
// If memtable contains a deletion for key, store a NotFound() error
|
|
|
|
// in *status and return true.
|
2013-03-21 22:59:47 +00:00
|
|
|
// If memtable contains Merge operation as the most recent entry for a key,
|
2013-07-26 19:57:01 +00:00
|
|
|
// and the merge process does not stop (not reaching a value or delete),
|
[RocksDB] [MergeOperator] The new Merge Interface! Uses merge sequences.
Summary:
Here are the major changes to the Merge Interface. It has been expanded
to handle cases where the MergeOperator is not associative. It does so by stacking
up merge operations while scanning through the key history (i.e.: during Get() or
Compaction), until a valid Put/Delete/end-of-history is encountered; it then
applies all of the merge operations in the correct sequence starting with the
base/sentinel value.
I have also introduced an "AssociativeMerge" function which allows the user to
take advantage of associative merge operations (such as in the case of counters).
The implementation will always attempt to merge the operations/operands themselves
together when they are encountered, and will resort to the "stacking" method if
and only if the "associative-merge" fails.
This implementation is conjectured to allow MergeOperator to handle the general
case, while still providing the user with the ability to take advantage of certain
efficiencies in their own merge-operator / data-structure.
NOTE: This is a preliminary diff. This must still go through a lot of review,
revision, and testing. Feedback welcome!
Test Plan:
-This is a preliminary diff. I have only just begun testing/debugging it.
-I will be testing this with the existing MergeOperator use-cases and unit-tests
(counters, string-append, and redis-lists)
-I will be "desk-checking" and walking through the code with the help gdb.
-I will find a way of stress-testing the new interface / implementation using
db_bench, db_test, merge_test, and/or db_stress.
-I will ensure that my tests cover all cases: Get-Memtable,
Get-Immutable-Memtable, Get-from-Disk, Iterator-Range-Scan, Flush-Memtable-to-L0,
Compaction-L0-L1, Compaction-Ln-L(n+1), Put/Delete found, Put/Delete not-found,
end-of-history, end-of-file, etc.
-A lot of feedback from the reviewers.
Reviewers: haobo, dhruba, zshao, emayanke
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D11499
2013-08-06 03:14:32 +00:00
|
|
|
// prepend the current merge operand to *operands.
|
|
|
|
// store MergeInProgress in s, and return false.
|
2011-06-22 02:36:45 +00:00
|
|
|
// Else, return false.
|
2013-03-21 22:59:47 +00:00
|
|
|
bool Get(const LookupKey& key, std::string* value, Status* s,
|
[RocksDB] [MergeOperator] The new Merge Interface! Uses merge sequences.
Summary:
Here are the major changes to the Merge Interface. It has been expanded
to handle cases where the MergeOperator is not associative. It does so by stacking
up merge operations while scanning through the key history (i.e.: during Get() or
Compaction), until a valid Put/Delete/end-of-history is encountered; it then
applies all of the merge operations in the correct sequence starting with the
base/sentinel value.
I have also introduced an "AssociativeMerge" function which allows the user to
take advantage of associative merge operations (such as in the case of counters).
The implementation will always attempt to merge the operations/operands themselves
together when they are encountered, and will resort to the "stacking" method if
and only if the "associative-merge" fails.
This implementation is conjectured to allow MergeOperator to handle the general
case, while still providing the user with the ability to take advantage of certain
efficiencies in their own merge-operator / data-structure.
NOTE: This is a preliminary diff. This must still go through a lot of review,
revision, and testing. Feedback welcome!
Test Plan:
-This is a preliminary diff. I have only just begun testing/debugging it.
-I will be testing this with the existing MergeOperator use-cases and unit-tests
(counters, string-append, and redis-lists)
-I will be "desk-checking" and walking through the code with the help gdb.
-I will find a way of stress-testing the new interface / implementation using
db_bench, db_test, merge_test, and/or db_stress.
-I will ensure that my tests cover all cases: Get-Memtable,
Get-Immutable-Memtable, Get-from-Disk, Iterator-Range-Scan, Flush-Memtable-to-L0,
Compaction-L0-L1, Compaction-Ln-L(n+1), Put/Delete found, Put/Delete not-found,
end-of-history, end-of-file, etc.
-A lot of feedback from the reviewers.
Reviewers: haobo, dhruba, zshao, emayanke
Reviewed By: haobo
CC: leveldb
Differential Revision: https://reviews.facebook.net/D11499
2013-08-06 03:14:32 +00:00
|
|
|
std::deque<std::string>* operands, const Options& options);
|
2011-06-22 02:36:45 +00:00
|
|
|
|
2012-10-19 21:00:53 +00:00
|
|
|
// Returns the edits area that is needed for flushing the memtable
|
|
|
|
VersionEdit* GetEdits() { return &edit_; }
|
|
|
|
|
2013-02-28 22:09:30 +00:00
|
|
|
// Returns the sequence number of the first element that was inserted
|
|
|
|
// into the memtable
|
|
|
|
SequenceNumber GetFirstSequenceNumber() { return first_seqno_; }
|
|
|
|
|
2013-07-16 18:56:46 +00:00
|
|
|
// Returns the next active logfile number when this memtable is about to
|
|
|
|
// be flushed to storage
|
|
|
|
uint64_t GetNextLogNumber() { return mem_next_logfile_number_; }
|
|
|
|
|
|
|
|
// Sets the next active logfile number when this memtable is about to
|
|
|
|
// be flushed to storage
|
|
|
|
void SetNextLogNumber(uint64_t num) { mem_next_logfile_number_ = num; }
|
|
|
|
|
2013-06-11 21:23:58 +00:00
|
|
|
// Returns the logfile number that can be safely deleted when this
|
|
|
|
// memstore is flushed to storage
|
|
|
|
uint64_t GetLogNumber() { return mem_logfile_number_; }
|
|
|
|
|
|
|
|
// Sets the logfile number that can be safely deleted when this
|
|
|
|
// memstore is flushed to storage
|
|
|
|
void SetLogNumber(uint64_t num) { mem_logfile_number_ = num; }
|
|
|
|
|
2013-08-23 06:10:02 +00:00
|
|
|
// Notify the underlying storage that no more items will be added
|
|
|
|
void MarkImmutable() { table_->MarkReadOnly(); }
|
|
|
|
|
2011-03-18 22:37:00 +00:00
|
|
|
private:
|
2011-05-21 02:17:43 +00:00
|
|
|
~MemTable(); // Private since only Unref() should be used to delete it
|
2011-03-18 22:37:00 +00:00
|
|
|
friend class MemTableIterator;
|
|
|
|
friend class MemTableBackwardIterator;
|
2012-10-19 21:00:53 +00:00
|
|
|
friend class MemTableList;
|
2011-03-18 22:37:00 +00:00
|
|
|
|
|
|
|
KeyComparator comparator_;
|
2011-05-21 02:17:43 +00:00
|
|
|
int refs_;
|
Make arena block size configurable
Summary:
Add an option for arena block size, default value 4096 bytes. Arena will allocate blocks with such size.
I am not sure about passing parameter to skiplist in the new virtualized framework, though I talked to Jim a bit. So add Jim as reviewer.
Test Plan:
new unit test, I am running db_test.
For passing paramter from configured option to Arena, I tried tests like:
TEST(DBTest, Arena_Option) {
std::string dbname = test::TmpDir() + "/db_arena_option_test";
DestroyDB(dbname, Options());
DB* db = nullptr;
Options opts;
opts.create_if_missing = true;
opts.arena_block_size = 1000000; // tested 99, 999999
Status s = DB::Open(opts, dbname, &db);
db->Put(WriteOptions(), "a", "123");
}
and printed some debug info. The results look good. Any suggestion for such a unit-test?
Reviewers: haobo, dhruba, emayanke, jpaton
Reviewed By: dhruba
CC: leveldb, zshao
Differential Revision: https://reviews.facebook.net/D11799
2013-07-31 19:42:23 +00:00
|
|
|
ArenaImpl arena_impl_;
|
2013-07-23 21:42:27 +00:00
|
|
|
shared_ptr<MemTableRep> table_;
|
2011-03-18 22:37:00 +00:00
|
|
|
|
2012-10-19 21:00:53 +00:00
|
|
|
// These are used to manage memtable flushes to storage
|
2012-11-29 00:42:36 +00:00
|
|
|
bool flush_in_progress_; // started the flush
|
2012-10-19 21:00:53 +00:00
|
|
|
bool flush_completed_; // finished the flush
|
|
|
|
uint64_t file_number_; // filled up after flush is complete
|
|
|
|
|
|
|
|
// The udpates to be applied to the transaction log when this
|
|
|
|
// memtable is flushed to storage.
|
|
|
|
VersionEdit edit_;
|
|
|
|
|
2013-02-28 22:09:30 +00:00
|
|
|
// The sequence number of the kv that was inserted first
|
|
|
|
SequenceNumber first_seqno_;
|
|
|
|
|
2013-06-11 21:23:58 +00:00
|
|
|
// The log files earlier than this number can be deleted.
|
2013-07-16 18:56:46 +00:00
|
|
|
uint64_t mem_next_logfile_number_;
|
|
|
|
|
|
|
|
// The log file that backs this memtable (to be deleted when
|
|
|
|
// memtable flush is done)
|
2013-06-11 21:23:58 +00:00
|
|
|
uint64_t mem_logfile_number_;
|
|
|
|
|
2011-03-18 22:37:00 +00:00
|
|
|
// No copying allowed
|
|
|
|
MemTable(const MemTable&);
|
|
|
|
void operator=(const MemTable&);
|
|
|
|
};
|
|
|
|
|
2013-10-04 04:49:15 +00:00
|
|
|
} // namespace rocksdb
|