De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
|
|
//
|
|
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#include "table/block_based/partitioned_index_iterator.h"
|
|
|
|
|
|
|
|
namespace ROCKSDB_NAMESPACE {
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::Seek(const Slice& target) { SeekImpl(&target); }
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::SeekToFirst() { SeekImpl(nullptr); }
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::SeekImpl(const Slice* target) {
|
2020-03-17 19:28:46 +00:00
|
|
|
SavePrevIndexValue();
|
|
|
|
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
if (target) {
|
|
|
|
index_iter_->Seek(*target);
|
|
|
|
} else {
|
|
|
|
index_iter_->SeekToFirst();
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!index_iter_->Valid()) {
|
|
|
|
ResetPartitionedIndexIter();
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2020-03-17 19:28:46 +00:00
|
|
|
InitPartitionedIndexBlock();
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
|
|
|
|
if (target) {
|
|
|
|
block_iter_.Seek(*target);
|
|
|
|
} else {
|
|
|
|
block_iter_.SeekToFirst();
|
|
|
|
}
|
|
|
|
FindKeyForward();
|
|
|
|
|
|
|
|
// We could check upper bound here, but that would be too complicated
|
|
|
|
// and checking index upper bound is less useful than for data blocks.
|
|
|
|
|
|
|
|
if (target) {
|
|
|
|
assert(!Valid() || (table_->get_rep()->index_key_includes_seq
|
|
|
|
? (icomp_.Compare(*target, key()) <= 0)
|
|
|
|
: (user_comparator_.Compare(ExtractUserKey(*target),
|
|
|
|
key()) <= 0)));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::SeekToLast() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
SavePrevIndexValue();
|
|
|
|
index_iter_->SeekToLast();
|
|
|
|
if (!index_iter_->Valid()) {
|
|
|
|
ResetPartitionedIndexIter();
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
InitPartitionedIndexBlock();
|
|
|
|
block_iter_.SeekToLast();
|
|
|
|
FindKeyBackward();
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::Next() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
assert(block_iter_points_to_real_block_);
|
|
|
|
block_iter_.Next();
|
|
|
|
FindKeyForward();
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::Prev() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
assert(block_iter_points_to_real_block_);
|
|
|
|
block_iter_.Prev();
|
|
|
|
|
|
|
|
FindKeyBackward();
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::InitPartitionedIndexBlock() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
BlockHandle partitioned_index_handle = index_iter_->value().handle;
|
|
|
|
if (!block_iter_points_to_real_block_ ||
|
|
|
|
partitioned_index_handle.offset() != prev_block_offset_ ||
|
|
|
|
// if previous attempt of reading the block missed cache, try again
|
|
|
|
block_iter_.status().IsIncomplete()) {
|
|
|
|
if (block_iter_points_to_real_block_) {
|
|
|
|
ResetPartitionedIndexIter();
|
|
|
|
}
|
|
|
|
auto* rep = table_->get_rep();
|
|
|
|
bool is_for_compaction =
|
|
|
|
lookup_context_.caller == TableReaderCaller::kCompaction;
|
|
|
|
// Prefetch additional data for range scans (iterators).
|
|
|
|
// Implicit auto readahead:
|
|
|
|
// Enabled after 2 sequential IOs when ReadOptions.readahead_size == 0.
|
|
|
|
// Explicit user requested readahead:
|
|
|
|
// Enabled from the very first IO when ReadOptions.readahead_size is set.
|
|
|
|
block_prefetcher_.PrefetchIfNeeded(rep, partitioned_index_handle,
|
|
|
|
read_options_.readahead_size,
|
|
|
|
is_for_compaction);
|
|
|
|
|
|
|
|
Status s;
|
|
|
|
table_->NewDataBlockIterator<IndexBlockIter>(
|
|
|
|
read_options_, partitioned_index_handle, &block_iter_,
|
|
|
|
BlockType::kIndex,
|
|
|
|
/*get_context=*/nullptr, &lookup_context_, s,
|
|
|
|
block_prefetcher_.prefetch_buffer(),
|
|
|
|
/*for_compaction=*/is_for_compaction);
|
|
|
|
block_iter_points_to_real_block_ = true;
|
|
|
|
// We could check upper bound here but it is complicated to reason about
|
|
|
|
// upper bound in index iterator. On the other than, in large scans, index
|
|
|
|
// iterators are moved much less frequently compared to data blocks. So
|
|
|
|
// the upper bound check is skipped for simplicity.
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::FindKeyForward() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
// This method's code is kept short to make it likely to be inlined.
|
|
|
|
|
|
|
|
assert(block_iter_points_to_real_block_);
|
|
|
|
|
|
|
|
if (!block_iter_.Valid()) {
|
|
|
|
// This is the only call site of FindBlockForward(), but it's extracted into
|
|
|
|
// a separate method to keep FindKeyForward() short and likely to be
|
|
|
|
// inlined. When transitioning to a different block, we call
|
|
|
|
// FindBlockForward(), which is much longer and is probably not inlined.
|
|
|
|
FindBlockForward();
|
|
|
|
} else {
|
|
|
|
// This is the fast path that avoids a function call.
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::FindBlockForward() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
// TODO the while loop inherits from two-level-iterator. We don't know
|
|
|
|
// whether a block can be empty so it can be replaced by an "if".
|
|
|
|
do {
|
|
|
|
if (!block_iter_.status().ok()) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
ResetPartitionedIndexIter();
|
|
|
|
index_iter_->Next();
|
|
|
|
|
|
|
|
if (!index_iter_->Valid()) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
InitPartitionedIndexBlock();
|
|
|
|
block_iter_.SeekToFirst();
|
|
|
|
} while (!block_iter_.Valid());
|
|
|
|
}
|
|
|
|
|
2020-09-29 23:26:58 +00:00
|
|
|
void PartitionedIndexIterator::FindKeyBackward() {
|
De-template block based table iterator (#6531)
Summary:
Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done:
1. Copy the block based iterator code into partitioned index iterator, and de-template them.
2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks.
3. Separate out the prefetch logic to a helper class and both classes call them.
This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531
Test Plan: build using make and cmake. And build release
Differential Revision: D20473108
fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f
2020-03-16 19:17:34 +00:00
|
|
|
while (!block_iter_.Valid()) {
|
|
|
|
if (!block_iter_.status().ok()) {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
ResetPartitionedIndexIter();
|
|
|
|
index_iter_->Prev();
|
|
|
|
|
|
|
|
if (index_iter_->Valid()) {
|
|
|
|
InitPartitionedIndexBlock();
|
|
|
|
block_iter_.SeekToLast();
|
|
|
|
} else {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|