2016-02-09 23:12:00 +00:00
|
|
|
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
|
2017-07-15 23:03:42 +00:00
|
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
|
|
// (found in the LICENSE.Apache file in the root directory).
|
2013-12-05 21:09:13 +00:00
|
|
|
#include "table/meta_blocks.h"
|
|
|
|
|
|
|
|
#include <map>
|
2014-02-05 00:21:47 +00:00
|
|
|
#include <string>
|
2013-12-05 21:09:13 +00:00
|
|
|
|
2019-05-30 21:47:29 +00:00
|
|
|
#include "block_fetcher.h"
|
A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
Summary:
Currently users have no idea a key is add, delete or merge from TablePropertiesCollector call back. Add a new function to add it.
Also refactor the codes so that
(1) make table property collector and internal table property collector two separate data structures with the later one now exposed
(2) table builders only receive internal table properties
Test Plan: Add cases in table_properties_collector_test to cover both of old and new ways of using TablePropertiesCollector.
Reviewers: yhchiang, igor.sugak, rven, igor
Reviewed By: rven, igor
Subscribers: meyering, yoshinorim, maykov, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D35373
2015-04-06 17:04:30 +00:00
|
|
|
#include "db/table_properties_collector.h"
|
2019-09-16 17:31:27 +00:00
|
|
|
#include "file/random_access_file_reader.h"
|
2021-09-29 11:01:57 +00:00
|
|
|
#include "logging/logging.h"
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
#include "rocksdb/options.h"
|
2014-01-28 05:58:46 +00:00
|
|
|
#include "rocksdb/table.h"
|
2014-02-05 00:21:47 +00:00
|
|
|
#include "rocksdb/table_properties.h"
|
2019-05-30 21:47:29 +00:00
|
|
|
#include "table/block_based/block.h"
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
#include "table/block_based/reader_common.h"
|
2013-12-05 21:09:13 +00:00
|
|
|
#include "table/format.h"
|
2015-10-12 22:06:38 +00:00
|
|
|
#include "table/internal_iterator.h"
|
2015-12-16 02:20:10 +00:00
|
|
|
#include "table/persistent_cache_helper.h"
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
#include "table/sst_file_writer_collectors.h"
|
2014-11-13 19:39:30 +00:00
|
|
|
#include "table/table_properties_internal.h"
|
2019-05-31 00:39:43 +00:00
|
|
|
#include "test_util/sync_point.h"
|
2013-12-05 21:09:13 +00:00
|
|
|
#include "util/coding.h"
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
namespace ROCKSDB_NAMESPACE {
|
2013-12-05 21:09:13 +00:00
|
|
|
|
2021-12-10 16:12:09 +00:00
|
|
|
const std::string kPropertiesBlockName = "rocksdb.properties";
|
format_version=6 and context-aware block checksums (#9058)
Summary:
## Context checksum
All RocksDB checksums currently use 32 bits of checking
power, which should be 1 in 4 billion false negative (FN) probability (failing to
detect corruption). This is true for random corruptions, and in some cases
small corruptions are guaranteed to be detected. But some possible
corruptions, such as in storage metadata rather than storage payload data,
would have a much higher FN rate. For example:
* Data larger than one SST block is replaced by data from elsewhere in
the same or another SST file. Especially with block_align=true, the
probability of exact block size match is probably around 1 in 100, making
the FN probability around that same. Without `block_align=true` the
probability of same block start location is probably around 1 in 10,000,
for FN probability around 1 in a million.
To solve this problem in new format_version=6, we add "context awareness"
to block checksum checks. The stored and expected checksum value is
modified based on the block's position in the file and which file it is in. The
modifications are cleverly chosen so that, for example
* blocks within about 4GB of each other are guaranteed to use different context
* blocks that are offset by exactly some multiple of 4GiB are guaranteed to use
different context
* files generated by the same process are guaranteed to use different context
for the same offsets, until wrap-around after 2^32 - 1 files
Thus, with format_version=6, if a valid SST block and checksum is misplaced,
its checksum FN probability should be essentially ideal, 1 in 4B.
## Footer checksum
This change also adds checksum protection to the SST footer (with
format_version=6), for the first time without relying on whole file checksum.
To prevent a corruption of the format_version in the footer (e.g. 6 -> 5) to
defeat the footer checksum, we change much of the footer data format
including an "extended magic number" in format_version 6 that would be
interpreted as empty index and metaindex block handles in older footer
versions. We also change the encoding of handles to free up space for
other new data in footer.
## More detail: making space in footer
In order to keep footer the same size in format_version=6 (avoid change to IO
patterns), we have to free up some space for new data. We do this two ways:
* Metaindex block handle is encoded down to 4 bytes (from 10) by assuming
it immediately precedes the footer, and by assuming it is < 4GB.
* Index block handle is moved into metaindex. (I don't know why it was
in footer to begin with.)
## Performance
In case of small performance penalty, I've made a "pay as you go" optimization
to compensate: replace `MutableCFOptions` in BlockBasedTableBuilder::Rep
with the only field used in that structure after construction: `prefix_extractor`.
This makes the PR an overall performance improvement (results below).
Nevertheless I'm seeing essentially no difference going from fv=5 to fv=6,
even including that improvement for both. That's based on extreme case table
write performance testing, many files with many blocks. This is relatively
checksum intensive (small blocks) and salt generation intensive (small files).
```
(for I in `seq 1 100`; do TEST_TMPDIR=/dev/shm/dbbench2 ./db_bench -benchmarks=fillseq -memtablerep=vector -disable_wal=1 -allow_concurrent_memtable_write=false -num=3000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=0 -write_buffer_size=100000 -compression_type=none -block_size=1000; done) 2>&1 | grep micros/op | tee out
awk '{ tot += $5; n += 1; } END { print int(1.0 * tot / n) }' < out
```
Each value below is ops/s averaged over 100 runs, run simultaneously with competing
configuration for load fairness
Before -> after (both fv=5): 483530 -> 483673 (negligible)
Re-run 1: 480733 -> 485427 (1.0% faster)
Re-run 2: 483821 -> 484541 (0.1% faster)
Before (fv=5) -> after (fv=6): 482006 -> 485100 (0.6% faster)
Re-run 1: 482212 -> 485075 (0.6% faster)
Re-run 2: 483590 -> 484073 (0.1% faster)
After fv=5 -> after fv=6: 483878 -> 485542 (0.3% faster)
Re-run 1: 485331 -> 483385 (0.4% slower)
Re-run 2: 485283 -> 483435 (0.4% slower)
Re-run 3: 483647 -> 486109 (0.5% faster)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9058
Test Plan:
unit tests included (table_test, db_properties_test, salt in env_test). General DB tests
and crash test updated to test new format_version.
Also temporarily updated the default format version to 6 and saw some test failures. Almost all
were due to an inadvertent additional read in VerifyChecksum to verify the index block checksum,
though it's arguably a bug that VerifyChecksum does not appear to (re-)verify the index block
checksum, just assuming it was verified in opening the index reader (probably *usually* true but
probably not always true). Some other concerns about VerifyChecksum are left in FIXME
comments. The only remaining test failure on change of default (in block_fetcher_test) now
has a comment about how to upgrade the test.
The format compatibility test does not need updating because we have not updated the default
format_version.
Reviewed By: ajkr, mrambacher
Differential Revision: D33100915
Pulled By: pdillinger
fbshipit-source-id: 8679e3e572fa580181a737fd6d113ed53c5422ee
2023-07-30 23:40:01 +00:00
|
|
|
// NB: only used with format_version >= 6
|
|
|
|
const std::string kIndexBlockName = "rocksdb.index";
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
// Old property block name for backward compatibility
|
|
|
|
const std::string kPropertiesBlockOldName = "rocksdb.stats";
|
2021-12-10 16:12:09 +00:00
|
|
|
const std::string kCompressionDictBlockName = "rocksdb.compression_dict";
|
|
|
|
const std::string kRangeDelBlockName = "rocksdb.range_del";
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
|
2013-12-05 21:09:13 +00:00
|
|
|
MetaIndexBuilder::MetaIndexBuilder()
|
2017-11-21 00:31:04 +00:00
|
|
|
: meta_index_block_(new BlockBuilder(1 /* restart interval */)) {}
|
2013-12-05 21:09:13 +00:00
|
|
|
|
2022-10-25 18:50:38 +00:00
|
|
|
void MetaIndexBuilder::Add(const std::string& key, const BlockHandle& handle) {
|
2013-12-05 21:09:13 +00:00
|
|
|
std::string handle_encoding;
|
|
|
|
handle.EncodeTo(&handle_encoding);
|
|
|
|
meta_block_handles_.insert({key, handle_encoding});
|
|
|
|
}
|
|
|
|
|
|
|
|
Slice MetaIndexBuilder::Finish() {
|
|
|
|
for (const auto& metablock : meta_block_handles_) {
|
|
|
|
meta_index_block_->Add(metablock.first, metablock.second);
|
|
|
|
}
|
|
|
|
return meta_index_block_->Finish();
|
|
|
|
}
|
|
|
|
|
2018-06-12 19:46:48 +00:00
|
|
|
// Property block will be read sequentially and cached in a heap located
|
|
|
|
// object, so there's no need for restart points. Thus we set the restart
|
|
|
|
// interval to infinity to save space.
|
2013-12-05 21:09:13 +00:00
|
|
|
PropertyBlockBuilder::PropertyBlockBuilder()
|
2022-05-05 20:08:21 +00:00
|
|
|
: properties_block_(new BlockBuilder(
|
|
|
|
std::numeric_limits<int32_t>::max() /* restart interval */)) {}
|
2013-12-05 21:09:13 +00:00
|
|
|
|
|
|
|
void PropertyBlockBuilder::Add(const std::string& name,
|
|
|
|
const std::string& val) {
|
2024-07-31 20:28:17 +00:00
|
|
|
assert(props_.find(name) == props_.end());
|
2013-12-05 21:09:13 +00:00
|
|
|
props_.insert({name, val});
|
|
|
|
}
|
|
|
|
|
|
|
|
void PropertyBlockBuilder::Add(const std::string& name, uint64_t val) {
|
|
|
|
std::string dst;
|
|
|
|
PutVarint64(&dst, val);
|
|
|
|
|
|
|
|
Add(name, dst);
|
|
|
|
}
|
|
|
|
|
|
|
|
void PropertyBlockBuilder::Add(
|
|
|
|
const UserCollectedProperties& user_collected_properties) {
|
|
|
|
for (const auto& prop : user_collected_properties) {
|
|
|
|
Add(prop.first, prop.second);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void PropertyBlockBuilder::AddTableProperty(const TableProperties& props) {
|
2019-04-19 05:36:32 +00:00
|
|
|
TEST_SYNC_POINT_CALLBACK("PropertyBlockBuilder::AddTableProperty:Start",
|
|
|
|
const_cast<TableProperties*>(&props));
|
|
|
|
|
2021-08-21 03:39:52 +00:00
|
|
|
Add(TablePropertiesNames::kOriginalFileNumber, props.orig_file_number);
|
2013-12-05 21:09:13 +00:00
|
|
|
Add(TablePropertiesNames::kRawKeySize, props.raw_key_size);
|
|
|
|
Add(TablePropertiesNames::kRawValueSize, props.raw_value_size);
|
|
|
|
Add(TablePropertiesNames::kDataSize, props.data_size);
|
|
|
|
Add(TablePropertiesNames::kIndexSize, props.index_size);
|
2017-06-13 17:59:22 +00:00
|
|
|
if (props.index_partitions != 0) {
|
|
|
|
Add(TablePropertiesNames::kIndexPartitions, props.index_partitions);
|
|
|
|
Add(TablePropertiesNames::kTopLevelIndexSize, props.top_level_index_size);
|
|
|
|
}
|
2018-05-26 01:41:31 +00:00
|
|
|
Add(TablePropertiesNames::kIndexKeyIsUserKey, props.index_key_is_user_key);
|
2018-08-09 23:49:45 +00:00
|
|
|
Add(TablePropertiesNames::kIndexValueIsDeltaEncoded,
|
|
|
|
props.index_value_is_delta_encoded);
|
2013-12-05 21:09:13 +00:00
|
|
|
Add(TablePropertiesNames::kNumEntries, props.num_entries);
|
2021-05-22 00:10:29 +00:00
|
|
|
Add(TablePropertiesNames::kNumFilterEntries, props.num_filter_entries);
|
2018-10-30 22:29:58 +00:00
|
|
|
Add(TablePropertiesNames::kDeletedKeys, props.num_deletions);
|
|
|
|
Add(TablePropertiesNames::kMergeOperands, props.num_merge_operands);
|
2018-06-27 03:18:43 +00:00
|
|
|
Add(TablePropertiesNames::kNumRangeDeletions, props.num_range_deletions);
|
2013-12-05 21:09:13 +00:00
|
|
|
Add(TablePropertiesNames::kNumDataBlocks, props.num_data_blocks);
|
|
|
|
Add(TablePropertiesNames::kFilterSize, props.filter_size);
|
2013-12-20 17:35:24 +00:00
|
|
|
Add(TablePropertiesNames::kFormatVersion, props.format_version);
|
|
|
|
Add(TablePropertiesNames::kFixedKeyLen, props.fixed_key_len);
|
2016-04-07 06:10:32 +00:00
|
|
|
Add(TablePropertiesNames::kColumnFamilyId, props.column_family_id);
|
2017-06-28 00:02:20 +00:00
|
|
|
Add(TablePropertiesNames::kCreationTime, props.creation_time);
|
2017-10-23 22:22:05 +00:00
|
|
|
Add(TablePropertiesNames::kOldestKeyTime, props.oldest_key_time);
|
Periodic Compactions (#5166)
Summary:
Introducing Periodic Compactions.
This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted. And also, of course, it helps to cleanup data older than certain threshold.
- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).
This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166
Differential Revision: D14884441
Pulled By: sagar0
fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
2019-04-11 02:24:25 +00:00
|
|
|
if (props.file_creation_time > 0) {
|
|
|
|
Add(TablePropertiesNames::kFileCreationTime, props.file_creation_time);
|
|
|
|
}
|
2021-04-01 01:20:44 +00:00
|
|
|
if (props.slow_compression_estimated_data_size > 0) {
|
|
|
|
Add(TablePropertiesNames::kSlowCompressionEstimatedDataSize,
|
|
|
|
props.slow_compression_estimated_data_size);
|
|
|
|
}
|
|
|
|
if (props.fast_compression_estimated_data_size > 0) {
|
|
|
|
Add(TablePropertiesNames::kFastCompressionEstimatedDataSize,
|
|
|
|
props.fast_compression_estimated_data_size);
|
|
|
|
}
|
Record and use the tail size to prefetch table tail (#11406)
Summary:
**Context:**
We prefetch the tail part of a SST file (i.e, the blocks after data blocks till the end of the file) during each SST file open in hope to prefetch all the stuff at once ahead of time for later read e.g, footer, meta index, filter/index etc. The existing approach to estimate the tail size to prefetch is through `TailPrefetchStats` heuristics introduced in https://github.com/facebook/rocksdb/pull/4156, which has caused small reads in unlucky case (e.g, small read into the tail buffer during table open in thread 1 under the same BlockBasedTableFactory object can make thread 2's tail prefetching use a small size that it shouldn't) and is hard to debug. Therefore we decide to record the exact tail size and use it directly to prefetch tail of the SST instead of relying heuristics.
**Summary:**
- Obtain and record in manifest the tail size in `BlockBasedTableBuilder::Finish()`
- For backward compatibility, we fall back to TailPrefetchStats and last to simple heuristics that the tail size is a linear portion of the file size - see PR conversation for more.
- Make`tail_start_offset` part of the table properties and deduct tail size to record in manifest for external files (e.g, file ingestion, import CF) and db repair (with no access to manifest).
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11406
Test Plan:
1. New UT
2. db bench
Note: db bench on /tmp/ where direct read is supported is too slow to finish and the default pinning setting in db bench is not helpful to profile # sst read of Get. Therefore I hacked the following to obtain the following comparison.
```
diff --git a/table/block_based/block_based_table_reader.cc b/table/block_based/block_based_table_reader.cc
index bd5669f0f..791484c1f 100644
--- a/table/block_based/block_based_table_reader.cc
+++ b/table/block_based/block_based_table_reader.cc
@@ -838,7 +838,7 @@ Status BlockBasedTable::PrefetchTail(
&tail_prefetch_size);
// Try file system prefetch
- if (!file->use_direct_io() && !force_direct_prefetch) {
+ if (false && !file->use_direct_io() && !force_direct_prefetch) {
if (!file->Prefetch(prefetch_off, prefetch_len, ro.rate_limiter_priority)
.IsNotSupported()) {
prefetch_buffer->reset(new FilePrefetchBuffer(
diff --git a/tools/db_bench_tool.cc b/tools/db_bench_tool.cc
index ea40f5fa0..39a0ac385 100644
--- a/tools/db_bench_tool.cc
+++ b/tools/db_bench_tool.cc
@@ -4191,6 +4191,8 @@ class Benchmark {
std::shared_ptr<TableFactory>(NewCuckooTableFactory(table_options));
} else {
BlockBasedTableOptions block_based_options;
+ block_based_options.metadata_cache_options.partition_pinning =
+ PinningTier::kAll;
block_based_options.checksum =
static_cast<ChecksumType>(FLAGS_checksum_type);
if (FLAGS_use_hash_search) {
```
Create DB
```
./db_bench --bloom_bits=3 --use_existing_db=1 --seed=1682546046158958 --partition_index_and_filters=1 --statistics=1 -db=/dev/shm/testdb/ -benchmarks=readrandom -key_size=3200 -value_size=512 -num=1000000 -write_buffer_size=6550000 -disable_auto_compactions=false -target_file_size_base=6550000 -compression_type=none
```
ReadRandom
```
./db_bench --bloom_bits=3 --use_existing_db=1 --seed=1682546046158958 --partition_index_and_filters=1 --statistics=1 -db=/dev/shm/testdb/ -benchmarks=readrandom -key_size=3200 -value_size=512 -num=1000000 -write_buffer_size=6550000 -disable_auto_compactions=false -target_file_size_base=6550000 -compression_type=none
```
(a) Existing (Use TailPrefetchStats for tail size + use seperate prefetch buffer in PartitionedFilter/IndexReader::CacheDependencies())
```
rocksdb.table.open.prefetch.tail.hit COUNT : 3395
rocksdb.sst.read.micros P50 : 5.655570 P95 : 9.931396 P99 : 14.845454 P100 : 585.000000 COUNT : 999905 SUM : 6590614
```
(b) This PR (Record tail size + use the same tail buffer in PartitionedFilter/IndexReader::CacheDependencies())
```
rocksdb.table.open.prefetch.tail.hit COUNT : 14257
rocksdb.sst.read.micros P50 : 5.173347 P95 : 9.015017 P99 : 12.912610 P100 : 228.000000 COUNT : 998547 SUM : 5976540
```
As we can see, we increase the prefetch tail hit count and decrease SST read count with this PR
3. Test backward compatibility by stepping through reading with post-PR code on a db generated pre-PR.
Reviewed By: pdillinger
Differential Revision: D45413346
Pulled By: hx235
fbshipit-source-id: 7d5e36a60a72477218f79905168d688452a4c064
2023-05-08 20:14:28 +00:00
|
|
|
Add(TablePropertiesNames::kTailStartOffset, props.tail_start_offset);
|
2023-06-22 04:49:01 +00:00
|
|
|
if (props.user_defined_timestamps_persisted == 0) {
|
|
|
|
Add(TablePropertiesNames::kUserDefinedTimestampsPersisted,
|
|
|
|
props.user_defined_timestamps_persisted);
|
|
|
|
}
|
Restore file size in backup table file names (and other cleanup) (#7400)
Summary:
Prior to 6.12, backup files using share_files_with_checksum had
the file size encoded in the file name, after the last '\_' and before
the last '.'. We considered this an implementation detail subject to
change, and indeed removed this information from the file name (with an
option to use old behavior) because it was considered
ineffective/inefficient for file name uniqueness. However, some
downstream RocksDB users were relying on this information since the file
size is not explicitly in the backup manifest file.
This primary purpose of this change is "retrofitting" the 6.12 release
(not yet a public release) to simultaneously support the benefits of the
new naming scheme (I/O performance and data correctness at scale) and
preserve the file size information, both as default behaviors. With this
change, we are essentially making the file size information encoded in
the file name an official, though obscure, extension of the backup meta
file format.
We preserve an option (kLegacyCrc32cAndFileSize) to use the original
"legacy" naming scheme, with its caveats, and make it easy to omit the
file size information (no kFlagIncludeFileSize), for more compact file
names. But note that changing the naming scheme used on an existing db
and backup directory can lead to transient space amplification, as some
files will be stored under two names in the shared_checksum directory.
Because some backups were saved using the original 6.12 naming scheme,
we offer two ways of dealing with those files: SST files generated by
older 6.12 versions can either use the default naming scheme in effect
when the SST files were generated (kFlagMatchInterimNaming, default, no
transient space amplification) or can use a new naming scheme (no
kFlagMatchInterimNaming, potential space amplification because some
already stored files getting a new name).
We don't have a natural way to detect which files were generated by
previous 6.12 versions, but this change hacks one in by changing DB
session ids to now use a more concise encoding, reducing file name
length, saving ~dozen bytes from SST files, and making them visually
distinct from DB ids so that they are less likely to be mixed up.
Two final auxiliary notes:
Recognizing that the backup file names have become a de facto part of
the backup meta schema, this change makes them easier to parse and
extend by putting a distinct marker, 's', before DB session ids embedded
in the name. When we extend this to allow custom checksums in the name,
they can get their own marker to ensure safe parsing. For backward
compatibility, file size does not get a marker but is assumed for
`_[0-9]+[.]`
Another change from initial 6.12 default behavior is never including
file custom checksum in the file name. Looking ahead to 6.13, we do not
want the default behavior to cause backup space amplification for
someone turning on file custom checksum checking in BackupEngine; we
want that to be an easy decision. When implemented, including file
custom checksums in backup file names will be a non-default option.
Actual file name patterns and priorities, as regexes:
kLegacyCrc32cAndFileSize OR pre-6.12 SST file ->
[0-9]+_[0-9]+_[0-9]+[.]sst
kFlagMatchInterimNaming set (default) AND early 6.12 SST file ->
[0-9]+_[0-9a-fA-F-]+[.]sst
kUseDbSessionId AND NOT kFlagIncludeFileSize ->
[0-9]+_s[0-9A-Z]{20}[.]sst
kUseDbSessionId AND kFlagIncludeFileSize (default) ->
[0-9]+_s[0-9A-Z]{20}_[0-9]+[.]sst
We might add opt-in options for more '\_' separated data in the name,
but embedded file size, if present, will always be after last '\_' and
before '.sst'.
This change was originally applied to version 6.12. (See https://github.com/facebook/rocksdb/issues/7390)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/7400
Test Plan:
unit tests included. Sync point callbacks are used to mimic
previous version SST files.
Reviewed By: ajkr
Differential Revision: D23759587
Pulled By: pdillinger
fbshipit-source-id: f62d8af4e0978de0a34f26288cfbe66049b70025
2020-09-17 17:22:56 +00:00
|
|
|
if (!props.db_id.empty()) {
|
|
|
|
Add(TablePropertiesNames::kDbId, props.db_id);
|
|
|
|
}
|
|
|
|
if (!props.db_session_id.empty()) {
|
|
|
|
Add(TablePropertiesNames::kDbSessionId, props.db_session_id);
|
|
|
|
}
|
2020-10-19 18:37:05 +00:00
|
|
|
if (!props.db_host_id.empty()) {
|
|
|
|
Add(TablePropertiesNames::kDbHostId, props.db_host_id);
|
|
|
|
}
|
2013-12-05 21:09:13 +00:00
|
|
|
|
|
|
|
if (!props.filter_policy_name.empty()) {
|
2016-04-21 17:16:28 +00:00
|
|
|
Add(TablePropertiesNames::kFilterPolicy, props.filter_policy_name);
|
|
|
|
}
|
|
|
|
if (!props.comparator_name.empty()) {
|
|
|
|
Add(TablePropertiesNames::kComparator, props.comparator_name);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!props.merge_operator_name.empty()) {
|
|
|
|
Add(TablePropertiesNames::kMergeOperator, props.merge_operator_name);
|
|
|
|
}
|
2016-08-26 18:46:32 +00:00
|
|
|
if (!props.prefix_extractor_name.empty()) {
|
|
|
|
Add(TablePropertiesNames::kPrefixExtractorName,
|
|
|
|
props.prefix_extractor_name);
|
|
|
|
}
|
2016-04-21 17:16:28 +00:00
|
|
|
if (!props.property_collectors_names.empty()) {
|
|
|
|
Add(TablePropertiesNames::kPropertyCollectors,
|
|
|
|
props.property_collectors_names);
|
2013-12-05 21:09:13 +00:00
|
|
|
}
|
2016-04-07 06:10:32 +00:00
|
|
|
if (!props.column_family_name.empty()) {
|
|
|
|
Add(TablePropertiesNames::kColumnFamilyName, props.column_family_name);
|
|
|
|
}
|
2016-05-12 16:47:16 +00:00
|
|
|
|
|
|
|
if (!props.compression_name.empty()) {
|
|
|
|
Add(TablePropertiesNames::kCompression, props.compression_name);
|
|
|
|
}
|
2019-04-02 21:48:52 +00:00
|
|
|
if (!props.compression_options.empty()) {
|
|
|
|
Add(TablePropertiesNames::kCompressionOptions, props.compression_options);
|
|
|
|
}
|
2022-07-15 04:49:34 +00:00
|
|
|
if (!props.seqno_to_time_mapping.empty()) {
|
|
|
|
Add(TablePropertiesNames::kSequenceNumberTimeMapping,
|
|
|
|
props.seqno_to_time_mapping);
|
|
|
|
}
|
2024-08-21 23:24:18 +00:00
|
|
|
if (props.key_largest_seqno != UINT64_MAX) {
|
|
|
|
Add(TablePropertiesNames::kKeyLargestSeqno, props.key_largest_seqno);
|
|
|
|
}
|
2013-12-05 21:09:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
Slice PropertyBlockBuilder::Finish() {
|
|
|
|
for (const auto& prop : props_) {
|
2024-07-31 20:28:17 +00:00
|
|
|
assert(last_prop_added_to_block_.empty() ||
|
|
|
|
comparator_->Compare(prop.first, last_prop_added_to_block_) > 0);
|
2013-12-05 21:09:13 +00:00
|
|
|
properties_block_->Add(prop.first, prop.second);
|
2024-07-31 20:28:17 +00:00
|
|
|
#ifndef NDEBUG
|
|
|
|
last_prop_added_to_block_ = prop.first;
|
|
|
|
#endif /* !NDEBUG */
|
2013-12-05 21:09:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return properties_block_->Finish();
|
|
|
|
}
|
|
|
|
|
2021-04-23 03:42:50 +00:00
|
|
|
void LogPropertiesCollectionError(Logger* info_log, const std::string& method,
|
|
|
|
const std::string& name) {
|
2013-12-05 21:09:13 +00:00
|
|
|
assert(method == "Add" || method == "Finish");
|
|
|
|
|
|
|
|
std::string msg =
|
2022-10-25 18:50:38 +00:00
|
|
|
"Encountered error when calling TablePropertiesCollector::" + method +
|
|
|
|
"() with collector name: " + name;
|
2017-03-16 02:22:52 +00:00
|
|
|
ROCKS_LOG_ERROR(info_log, "%s", msg.c_str());
|
2013-12-05 21:09:13 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
bool NotifyCollectTableCollectorsOnAdd(
|
A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
Summary:
Currently users have no idea a key is add, delete or merge from TablePropertiesCollector call back. Add a new function to add it.
Also refactor the codes so that
(1) make table property collector and internal table property collector two separate data structures with the later one now exposed
(2) table builders only receive internal table properties
Test Plan: Add cases in table_properties_collector_test to cover both of old and new ways of using TablePropertiesCollector.
Reviewers: yhchiang, igor.sugak, rven, igor
Reviewed By: rven, igor
Subscribers: meyering, yoshinorim, maykov, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D35373
2015-04-06 17:04:30 +00:00
|
|
|
const Slice& key, const Slice& value, uint64_t file_size,
|
2024-02-02 22:14:43 +00:00
|
|
|
const std::vector<std::unique_ptr<InternalTblPropColl>>& collectors,
|
2013-12-05 21:09:13 +00:00
|
|
|
Logger* info_log) {
|
|
|
|
bool all_succeeded = true;
|
TablePropertiesCollectorFactory
Summary:
This diff addresses task #4296714 and rethinks how users provide us with TablePropertiesCollectors as part of Options.
Here's description of task #4296714:
I'm debugging #4295529 and noticed that our count of user properties kDeletedKeys is wrong. We're sharing one single InternalKeyPropertiesCollector with all Table Builders. In LOG Files, we're outputting number of kDeletedKeys as connected with a single table, while it's actually the total count of deleted keys since creation of the DB.
For example, this table has 3155 entries and 1391828 deleted keys.
The problem with current approach that we call methods on a single TablePropertiesCollector for all the tables we create. Even worse, we could do it from multiple threads at the same time and TablePropertiesCollector has no way of knowing which table we're calling it for.
Good part: Looks like nobody inside Facebook is using Options::table_properties_collectors. This means we should be able to painfully change the API.
In this change, I introduce TablePropertiesCollectorFactory. For every table we create, we call `CreateTablePropertiesCollector`, which creates a TablePropertiesCollector for a single table. We then use it sequentially from a single thread, which means it doesn't have to be thread-safe.
Test Plan:
Added a test in table_properties_collector_test that fails on master (build two tables, assert that kDeletedKeys count is correct for the second one).
Also, all other tests
Reviewers: sdong, dhruba, haobo, kailiu
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18579
2014-05-13 19:30:55 +00:00
|
|
|
for (auto& collector : collectors) {
|
A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge
Summary:
Currently users have no idea a key is add, delete or merge from TablePropertiesCollector call back. Add a new function to add it.
Also refactor the codes so that
(1) make table property collector and internal table property collector two separate data structures with the later one now exposed
(2) table builders only receive internal table properties
Test Plan: Add cases in table_properties_collector_test to cover both of old and new ways of using TablePropertiesCollector.
Reviewers: yhchiang, igor.sugak, rven, igor
Reviewed By: rven, igor
Subscribers: meyering, yoshinorim, maykov, leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D35373
2015-04-06 17:04:30 +00:00
|
|
|
Status s = collector->InternalAdd(key, value, file_size);
|
2013-12-05 21:09:13 +00:00
|
|
|
all_succeeded = all_succeeded && s.ok();
|
|
|
|
if (!s.ok()) {
|
2014-02-05 00:21:47 +00:00
|
|
|
LogPropertiesCollectionError(info_log, "Add" /* method */,
|
|
|
|
collector->Name());
|
2013-12-05 21:09:13 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
return all_succeeded;
|
|
|
|
}
|
|
|
|
|
2019-03-18 19:07:35 +00:00
|
|
|
void NotifyCollectTableCollectorsOnBlockAdd(
|
2024-02-02 22:14:43 +00:00
|
|
|
const std::vector<std::unique_ptr<InternalTblPropColl>>& collectors,
|
Refactor to avoid confusing "raw block" (#10408)
Summary:
We have a lot of confusing code because of mixed, sometimes
completely opposite uses of of the term "raw block" or "raw contents",
sometimes within the same source file. For example, in `BlockBasedTableBuilder`,
`raw_block_contents` and `raw_size` generally referred to uncompressed block
contents and size, while `WriteRawBlock` referred to writing a block that
is already compressed if it is going to be. Meanwhile, in
`BlockBasedTable`, `raw_block_contents` either referred to a (maybe
compressed) block with trailer, or a maybe compressed block maybe
without trailer. (Note: left as follow-up work to use C++ typing to
better sort out the various kinds of BlockContents.)
This change primarily tries to apply some consistent terminology around
the kinds of block representations, avoiding the unclear "raw". (Any
meaning of "raw" assumes some bias toward the storage layer or toward
the logical data layer.) Preferred terminology:
* **Serialized block** - bytes that go into storage. For block-based table
(usually the case) this includes the block trailer. WART: block `size` may or
may not include the trailer; need to be clear about whether it does or not.
* **Maybe compressed block** - like a serialized block, but without the
trailer (or no promise of including a trailer). Must be accompanied by a
CompressionType.
* **Uncompressed block** - "payload" bytes that are either stored with no
compression, used as input to compression function, or result of
decompression function.
* **Parsed block** - an in-memory form of a block in block cache, as it is
used by the table reader. Different C++ types are used depending on the
block type (see block_like_traits.h).
Other refactorings:
* Misc corrections/improvements of internal API comments
* Remove a few misleading / unhelpful / redundant comments.
* Use move semantics in some places to simplify contracts
* Use better parameter names to indicate which parameters are used for
outputs
* Remove some extraneous `extern`
* Various clean-ups to `CacheDumperImpl` (mostly unnecessary code)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10408
Test Plan: existing tests
Reviewed By: akankshamahajan15
Differential Revision: D38172617
Pulled By: pdillinger
fbshipit-source-id: ccb99299f324ac5ca46996d34c5089621a4f260c
2022-09-22 18:25:32 +00:00
|
|
|
const uint64_t block_uncomp_bytes,
|
|
|
|
const uint64_t block_compressed_bytes_fast,
|
2021-03-25 21:58:23 +00:00
|
|
|
const uint64_t block_compressed_bytes_slow) {
|
2019-03-18 19:07:35 +00:00
|
|
|
for (auto& collector : collectors) {
|
Refactor to avoid confusing "raw block" (#10408)
Summary:
We have a lot of confusing code because of mixed, sometimes
completely opposite uses of of the term "raw block" or "raw contents",
sometimes within the same source file. For example, in `BlockBasedTableBuilder`,
`raw_block_contents` and `raw_size` generally referred to uncompressed block
contents and size, while `WriteRawBlock` referred to writing a block that
is already compressed if it is going to be. Meanwhile, in
`BlockBasedTable`, `raw_block_contents` either referred to a (maybe
compressed) block with trailer, or a maybe compressed block maybe
without trailer. (Note: left as follow-up work to use C++ typing to
better sort out the various kinds of BlockContents.)
This change primarily tries to apply some consistent terminology around
the kinds of block representations, avoiding the unclear "raw". (Any
meaning of "raw" assumes some bias toward the storage layer or toward
the logical data layer.) Preferred terminology:
* **Serialized block** - bytes that go into storage. For block-based table
(usually the case) this includes the block trailer. WART: block `size` may or
may not include the trailer; need to be clear about whether it does or not.
* **Maybe compressed block** - like a serialized block, but without the
trailer (or no promise of including a trailer). Must be accompanied by a
CompressionType.
* **Uncompressed block** - "payload" bytes that are either stored with no
compression, used as input to compression function, or result of
decompression function.
* **Parsed block** - an in-memory form of a block in block cache, as it is
used by the table reader. Different C++ types are used depending on the
block type (see block_like_traits.h).
Other refactorings:
* Misc corrections/improvements of internal API comments
* Remove a few misleading / unhelpful / redundant comments.
* Use move semantics in some places to simplify contracts
* Use better parameter names to indicate which parameters are used for
outputs
* Remove some extraneous `extern`
* Various clean-ups to `CacheDumperImpl` (mostly unnecessary code)
Pull Request resolved: https://github.com/facebook/rocksdb/pull/10408
Test Plan: existing tests
Reviewed By: akankshamahajan15
Differential Revision: D38172617
Pulled By: pdillinger
fbshipit-source-id: ccb99299f324ac5ca46996d34c5089621a4f260c
2022-09-22 18:25:32 +00:00
|
|
|
collector->BlockAdd(block_uncomp_bytes, block_compressed_bytes_fast,
|
2021-03-25 21:58:23 +00:00
|
|
|
block_compressed_bytes_slow);
|
2019-03-18 19:07:35 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-12-05 21:09:13 +00:00
|
|
|
bool NotifyCollectTableCollectorsOnFinish(
|
2024-02-02 22:14:43 +00:00
|
|
|
const std::vector<std::unique_ptr<InternalTblPropColl>>& collectors,
|
2023-11-08 22:00:36 +00:00
|
|
|
Logger* info_log, PropertyBlockBuilder* builder,
|
|
|
|
UserCollectedProperties& user_collected_properties,
|
|
|
|
UserCollectedProperties& readable_properties) {
|
2013-12-05 21:09:13 +00:00
|
|
|
bool all_succeeded = true;
|
TablePropertiesCollectorFactory
Summary:
This diff addresses task #4296714 and rethinks how users provide us with TablePropertiesCollectors as part of Options.
Here's description of task #4296714:
I'm debugging #4295529 and noticed that our count of user properties kDeletedKeys is wrong. We're sharing one single InternalKeyPropertiesCollector with all Table Builders. In LOG Files, we're outputting number of kDeletedKeys as connected with a single table, while it's actually the total count of deleted keys since creation of the DB.
For example, this table has 3155 entries and 1391828 deleted keys.
The problem with current approach that we call methods on a single TablePropertiesCollector for all the tables we create. Even worse, we could do it from multiple threads at the same time and TablePropertiesCollector has no way of knowing which table we're calling it for.
Good part: Looks like nobody inside Facebook is using Options::table_properties_collectors. This means we should be able to painfully change the API.
In this change, I introduce TablePropertiesCollectorFactory. For every table we create, we call `CreateTablePropertiesCollector`, which creates a TablePropertiesCollector for a single table. We then use it sequentially from a single thread, which means it doesn't have to be thread-safe.
Test Plan:
Added a test in table_properties_collector_test that fails on master (build two tables, assert that kDeletedKeys count is correct for the second one).
Also, all other tests
Reviewers: sdong, dhruba, haobo, kailiu
Reviewed By: kailiu
CC: leveldb
Differential Revision: https://reviews.facebook.net/D18579
2014-05-13 19:30:55 +00:00
|
|
|
for (auto& collector : collectors) {
|
2024-07-31 20:28:17 +00:00
|
|
|
UserCollectedProperties user_properties;
|
|
|
|
Status s = collector->Finish(&user_properties);
|
2023-11-08 22:00:36 +00:00
|
|
|
if (s.ok()) {
|
|
|
|
for (const auto& prop : collector->GetReadableProperties()) {
|
|
|
|
readable_properties.insert(prop);
|
|
|
|
}
|
2024-07-31 20:28:17 +00:00
|
|
|
#ifndef NDEBUG
|
|
|
|
// Check different user properties collectors are not adding properties of
|
|
|
|
// the same name.
|
|
|
|
for (const auto& pair : user_properties) {
|
|
|
|
assert(user_collected_properties.find(pair.first) ==
|
|
|
|
user_collected_properties.end());
|
|
|
|
}
|
|
|
|
#endif /* !NDEBUG */
|
|
|
|
user_collected_properties.merge(user_properties);
|
2023-11-08 22:00:36 +00:00
|
|
|
} else {
|
2014-02-05 00:21:47 +00:00
|
|
|
LogPropertiesCollectionError(info_log, "Finish" /* method */,
|
|
|
|
collector->Name());
|
2023-11-08 22:00:36 +00:00
|
|
|
if (all_succeeded) {
|
|
|
|
all_succeeded = false;
|
|
|
|
}
|
2013-12-05 21:09:13 +00:00
|
|
|
}
|
|
|
|
}
|
2024-07-31 20:28:17 +00:00
|
|
|
builder->Add(user_collected_properties);
|
2013-12-05 21:09:13 +00:00
|
|
|
return all_succeeded;
|
|
|
|
}
|
|
|
|
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
// FIXME: should be a parameter for reading table properties to use persistent
|
|
|
|
// cache?
|
|
|
|
Status ReadTablePropertiesHelper(
|
|
|
|
const ReadOptions& ro, const BlockHandle& handle,
|
|
|
|
RandomAccessFileReader* file, FilePrefetchBuffer* prefetch_buffer,
|
|
|
|
const Footer& footer, const ImmutableOptions& ioptions,
|
|
|
|
std::unique_ptr<TableProperties>* table_properties,
|
|
|
|
MemoryAllocator* memory_allocator) {
|
2013-12-06 00:51:26 +00:00
|
|
|
assert(table_properties);
|
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
Status s;
|
|
|
|
bool retry = false;
|
|
|
|
while (true) {
|
|
|
|
BlockContents block_contents;
|
|
|
|
size_t len = handle.size() + footer.GetBlockTrailerSize();
|
|
|
|
// If this is an external SST file ingested with write_global_seqno set to
|
|
|
|
// true, then we expect the checksum mismatch because checksum was written
|
|
|
|
// by SstFileWriter, but its global seqno in the properties block may have
|
|
|
|
// been changed during ingestion. For this reason, we initially read
|
|
|
|
// and process without checksum verification, then later try checksum
|
|
|
|
// verification so that if it fails, we can copy to a temporary buffer with
|
|
|
|
// global seqno set to its original value, i.e. 0, and attempt checksum
|
|
|
|
// verification again.
|
|
|
|
if (!retry) {
|
|
|
|
ReadOptions modified_ro = ro;
|
|
|
|
modified_ro.verify_checksums = false;
|
|
|
|
BlockFetcher block_fetcher(
|
|
|
|
file, prefetch_buffer, footer, modified_ro, handle, &block_contents,
|
|
|
|
ioptions, false /* decompress */, false /*maybe_compressed*/,
|
|
|
|
BlockType::kProperties, UncompressionDict::GetEmptyDict(),
|
|
|
|
PersistentCacheOptions::kEmpty, memory_allocator);
|
|
|
|
s = block_fetcher.ReadBlockContents();
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
assert(block_fetcher.GetBlockSizeWithTrailer() == len);
|
|
|
|
TEST_SYNC_POINT_CALLBACK("ReadTablePropertiesHelper:0",
|
|
|
|
&block_contents.data);
|
|
|
|
} else {
|
|
|
|
assert(s.IsCorruption());
|
|
|
|
// If retrying, use a stronger file system read to check and correct
|
|
|
|
// data corruption
|
|
|
|
IOOptions opts;
|
|
|
|
if (PrepareIOFromReadOptions(ro, ioptions.clock, opts) !=
|
|
|
|
IOStatus::OK()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
opts.verify_and_reconstruct_read = true;
|
|
|
|
std::unique_ptr<char[]> data(new char[len]);
|
|
|
|
Slice result;
|
|
|
|
IOStatus io_s =
|
|
|
|
file->Read(opts, handle.offset(), len, &result, data.get(), nullptr);
|
|
|
|
RecordTick(ioptions.stats, FILE_READ_CORRUPTION_RETRY_COUNT);
|
|
|
|
if (!io_s.ok()) {
|
|
|
|
ROCKS_LOG_INFO(ioptions.info_log,
|
|
|
|
"Reading properties block failed - %s",
|
|
|
|
io_s.ToString().c_str());
|
|
|
|
// Return the original corruption error as that's more serious
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
if (result.size() < len) {
|
|
|
|
return Status::Corruption("Reading properties block failed - " +
|
|
|
|
std::to_string(result.size()) +
|
|
|
|
" bytes read");
|
|
|
|
}
|
|
|
|
RecordTick(ioptions.stats, FILE_READ_CORRUPTION_RETRY_SUCCESS_COUNT);
|
|
|
|
block_contents = BlockContents(std::move(data), handle.size());
|
2013-12-06 00:51:26 +00:00
|
|
|
}
|
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
uint64_t block_size = block_contents.data.size();
|
|
|
|
Block properties_block(std::move(block_contents));
|
|
|
|
// Unfortunately, Block::size() might not equal block_contents.data.size(),
|
|
|
|
// and Block hides block_contents
|
|
|
|
std::unique_ptr<MetaBlockIter> iter(properties_block.NewMetaIterator());
|
|
|
|
|
|
|
|
std::unique_ptr<TableProperties> new_table_properties{new TableProperties};
|
|
|
|
// All pre-defined properties of type uint64_t
|
|
|
|
std::unordered_map<std::string, uint64_t*> predefined_uint64_properties = {
|
|
|
|
{TablePropertiesNames::kOriginalFileNumber,
|
|
|
|
&new_table_properties->orig_file_number},
|
|
|
|
{TablePropertiesNames::kDataSize, &new_table_properties->data_size},
|
|
|
|
{TablePropertiesNames::kIndexSize, &new_table_properties->index_size},
|
|
|
|
{TablePropertiesNames::kIndexPartitions,
|
|
|
|
&new_table_properties->index_partitions},
|
|
|
|
{TablePropertiesNames::kTopLevelIndexSize,
|
|
|
|
&new_table_properties->top_level_index_size},
|
|
|
|
{TablePropertiesNames::kIndexKeyIsUserKey,
|
|
|
|
&new_table_properties->index_key_is_user_key},
|
|
|
|
{TablePropertiesNames::kIndexValueIsDeltaEncoded,
|
|
|
|
&new_table_properties->index_value_is_delta_encoded},
|
|
|
|
{TablePropertiesNames::kFilterSize, &new_table_properties->filter_size},
|
|
|
|
{TablePropertiesNames::kRawKeySize,
|
|
|
|
&new_table_properties->raw_key_size},
|
|
|
|
{TablePropertiesNames::kRawValueSize,
|
|
|
|
&new_table_properties->raw_value_size},
|
|
|
|
{TablePropertiesNames::kNumDataBlocks,
|
|
|
|
&new_table_properties->num_data_blocks},
|
|
|
|
{TablePropertiesNames::kNumEntries, &new_table_properties->num_entries},
|
|
|
|
{TablePropertiesNames::kNumFilterEntries,
|
|
|
|
&new_table_properties->num_filter_entries},
|
|
|
|
{TablePropertiesNames::kDeletedKeys,
|
|
|
|
&new_table_properties->num_deletions},
|
|
|
|
{TablePropertiesNames::kMergeOperands,
|
|
|
|
&new_table_properties->num_merge_operands},
|
|
|
|
{TablePropertiesNames::kNumRangeDeletions,
|
|
|
|
&new_table_properties->num_range_deletions},
|
|
|
|
{TablePropertiesNames::kFormatVersion,
|
|
|
|
&new_table_properties->format_version},
|
|
|
|
{TablePropertiesNames::kFixedKeyLen,
|
|
|
|
&new_table_properties->fixed_key_len},
|
|
|
|
{TablePropertiesNames::kColumnFamilyId,
|
|
|
|
&new_table_properties->column_family_id},
|
|
|
|
{TablePropertiesNames::kCreationTime,
|
|
|
|
&new_table_properties->creation_time},
|
|
|
|
{TablePropertiesNames::kOldestKeyTime,
|
|
|
|
&new_table_properties->oldest_key_time},
|
|
|
|
{TablePropertiesNames::kFileCreationTime,
|
|
|
|
&new_table_properties->file_creation_time},
|
|
|
|
{TablePropertiesNames::kSlowCompressionEstimatedDataSize,
|
|
|
|
&new_table_properties->slow_compression_estimated_data_size},
|
|
|
|
{TablePropertiesNames::kFastCompressionEstimatedDataSize,
|
|
|
|
&new_table_properties->fast_compression_estimated_data_size},
|
|
|
|
{TablePropertiesNames::kTailStartOffset,
|
|
|
|
&new_table_properties->tail_start_offset},
|
|
|
|
{TablePropertiesNames::kUserDefinedTimestampsPersisted,
|
|
|
|
&new_table_properties->user_defined_timestamps_persisted},
|
|
|
|
{TablePropertiesNames::kKeyLargestSeqno,
|
|
|
|
&new_table_properties->key_largest_seqno},
|
|
|
|
};
|
|
|
|
|
|
|
|
std::string last_key;
|
|
|
|
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
|
|
|
|
s = iter->status();
|
|
|
|
if (!s.ok()) {
|
|
|
|
break;
|
|
|
|
}
|
2013-12-06 00:51:26 +00:00
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
auto key = iter->key().ToString();
|
|
|
|
// properties block should be strictly sorted with no duplicate key.
|
|
|
|
if (!last_key.empty() &&
|
|
|
|
BytewiseComparator()->Compare(key, last_key) <= 0) {
|
|
|
|
s = Status::Corruption("properties unsorted");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
last_key = key;
|
2013-12-06 00:51:26 +00:00
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
auto raw_val = iter->value();
|
|
|
|
auto pos = predefined_uint64_properties.find(key);
|
2016-10-18 23:59:37 +00:00
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
if (key == ExternalSstFilePropertyNames::kGlobalSeqno) {
|
|
|
|
new_table_properties->external_sst_file_global_seqno_offset =
|
|
|
|
handle.offset() + iter->ValueOffset();
|
|
|
|
}
|
|
|
|
|
|
|
|
if (pos != predefined_uint64_properties.end()) {
|
|
|
|
if (key == TablePropertiesNames::kDeletedKeys ||
|
|
|
|
key == TablePropertiesNames::kMergeOperands) {
|
|
|
|
// Insert in user-collected properties for API backwards compatibility
|
|
|
|
new_table_properties->user_collected_properties.insert(
|
|
|
|
{key, raw_val.ToString()});
|
|
|
|
}
|
|
|
|
// handle predefined rocksdb properties
|
|
|
|
uint64_t val;
|
|
|
|
if (!GetVarint64(&raw_val, &val)) {
|
|
|
|
// skip malformed value
|
|
|
|
auto error_msg =
|
|
|
|
"Detect malformed value in properties meta-block:"
|
|
|
|
"\tkey: " +
|
|
|
|
key + "\tval: " + raw_val.ToString();
|
|
|
|
ROCKS_LOG_ERROR(ioptions.logger, "%s", error_msg.c_str());
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
*(pos->second) = val;
|
|
|
|
} else if (key == TablePropertiesNames::kDbId) {
|
|
|
|
new_table_properties->db_id = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kDbSessionId) {
|
|
|
|
new_table_properties->db_session_id = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kDbHostId) {
|
|
|
|
new_table_properties->db_host_id = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kFilterPolicy) {
|
|
|
|
new_table_properties->filter_policy_name = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kColumnFamilyName) {
|
|
|
|
new_table_properties->column_family_name = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kComparator) {
|
|
|
|
new_table_properties->comparator_name = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kMergeOperator) {
|
|
|
|
new_table_properties->merge_operator_name = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kPrefixExtractorName) {
|
|
|
|
new_table_properties->prefix_extractor_name = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kPropertyCollectors) {
|
|
|
|
new_table_properties->property_collectors_names = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kCompression) {
|
|
|
|
new_table_properties->compression_name = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kCompressionOptions) {
|
|
|
|
new_table_properties->compression_options = raw_val.ToString();
|
|
|
|
} else if (key == TablePropertiesNames::kSequenceNumberTimeMapping) {
|
|
|
|
new_table_properties->seqno_to_time_mapping = raw_val.ToString();
|
|
|
|
} else {
|
|
|
|
// handle user-collected properties
|
2018-10-30 22:29:58 +00:00
|
|
|
new_table_properties->user_collected_properties.insert(
|
|
|
|
{key, raw_val.ToString()});
|
|
|
|
}
|
2013-12-06 00:51:26 +00:00
|
|
|
}
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
// Modified version of BlockFetcher checksum verification
|
|
|
|
// (See write_global_seqno comment above)
|
|
|
|
if (s.ok() && footer.GetBlockTrailerSize() > 0) {
|
|
|
|
s = VerifyBlockChecksum(footer, properties_block.data(), block_size,
|
|
|
|
file->file_name(), handle.offset());
|
|
|
|
if (s.IsCorruption()) {
|
|
|
|
if (new_table_properties->external_sst_file_global_seqno_offset != 0) {
|
|
|
|
std::string tmp_buf(properties_block.data(), len);
|
|
|
|
uint64_t global_seqno_offset =
|
|
|
|
new_table_properties->external_sst_file_global_seqno_offset -
|
|
|
|
handle.offset();
|
|
|
|
EncodeFixed64(&tmp_buf[static_cast<size_t>(global_seqno_offset)], 0);
|
|
|
|
s = VerifyBlockChecksum(footer, tmp_buf.data(), block_size,
|
|
|
|
file->file_name(), handle.offset());
|
|
|
|
}
|
2019-02-11 19:37:07 +00:00
|
|
|
}
|
|
|
|
}
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
|
2024-09-13 20:56:49 +00:00
|
|
|
// If we detected a corruption and the file system supports verification
|
|
|
|
// and reconstruction, retry the read
|
|
|
|
if (s.IsCorruption() && !retry &&
|
|
|
|
CheckFSFeatureSupport(ioptions.fs.get(),
|
|
|
|
FSSupportedOps::kVerifyAndReconstructRead)) {
|
|
|
|
retry = true;
|
|
|
|
} else {
|
|
|
|
if (s.ok()) {
|
|
|
|
*table_properties = std::move(new_table_properties);
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
2014-02-08 03:26:49 +00:00
|
|
|
}
|
2013-12-06 00:51:26 +00:00
|
|
|
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env
Summary: We want to keep Env a think layer for better portability. Less platform dependent codes should be moved out of Env. In this patch, I create a wrapper of file readers and writers, and put rate limiting, write buffering, as well as most perf context instrumentation and random kill out of Env. It will make it easier to maintain multiple Env in the future.
Test Plan: Run all existing unit tests.
Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, igor
Reviewed By: igor
Subscribers: leveldb, dhruba
Differential Revision: https://reviews.facebook.net/D42321
2015-07-17 23:16:11 +00:00
|
|
|
Status ReadTableProperties(RandomAccessFileReader* file, uint64_t file_size,
|
2016-07-19 16:44:03 +00:00
|
|
|
uint64_t table_magic_number,
|
2021-05-05 20:59:21 +00:00
|
|
|
const ImmutableOptions& ioptions,
|
2023-04-21 16:07:18 +00:00
|
|
|
const ReadOptions& read_options,
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
std::unique_ptr<TableProperties>* properties,
|
2020-05-13 01:21:32 +00:00
|
|
|
MemoryAllocator* memory_allocator,
|
|
|
|
FilePrefetchBuffer* prefetch_buffer) {
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
BlockHandle block_handle;
|
2015-01-13 22:33:04 +00:00
|
|
|
Footer footer;
|
2023-04-21 16:07:18 +00:00
|
|
|
Status s =
|
|
|
|
FindMetaBlockInFile(file, file_size, table_magic_number, ioptions,
|
|
|
|
read_options, kPropertiesBlockName, &block_handle,
|
|
|
|
memory_allocator, prefetch_buffer, &footer);
|
2013-12-06 00:51:26 +00:00
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
if (!block_handle.IsNull()) {
|
2023-04-21 16:07:18 +00:00
|
|
|
s = ReadTablePropertiesHelper(read_options, block_handle, file,
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
prefetch_buffer, footer, ioptions, properties,
|
|
|
|
memory_allocator);
|
2013-12-06 00:51:26 +00:00
|
|
|
} else {
|
2014-08-15 19:17:44 +00:00
|
|
|
s = Status::NotFound();
|
2013-12-06 00:51:26 +00:00
|
|
|
}
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
Status FindOptionalMetaBlock(InternalIterator* meta_index_iter,
|
|
|
|
const std::string& meta_block_name,
|
|
|
|
BlockHandle* block_handle) {
|
|
|
|
assert(block_handle != nullptr);
|
|
|
|
meta_index_iter->Seek(meta_block_name);
|
|
|
|
if (meta_index_iter->status().ok()) {
|
|
|
|
if (meta_index_iter->Valid() && meta_index_iter->key() == meta_block_name) {
|
|
|
|
Slice v = meta_index_iter->value();
|
|
|
|
return block_handle->DecodeFrom(&v);
|
2021-12-10 16:12:09 +00:00
|
|
|
} else if (meta_block_name == kPropertiesBlockName) {
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
// Have to try old name for compatibility
|
|
|
|
meta_index_iter->Seek(kPropertiesBlockOldName);
|
|
|
|
if (meta_index_iter->status().ok() && meta_index_iter->Valid() &&
|
|
|
|
meta_index_iter->key() == kPropertiesBlockOldName) {
|
|
|
|
Slice v = meta_index_iter->value();
|
|
|
|
return block_handle->DecodeFrom(&v);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
// else
|
|
|
|
*block_handle = BlockHandle::NullBlockHandle();
|
|
|
|
return meta_index_iter->status();
|
|
|
|
}
|
|
|
|
|
2015-10-12 22:06:38 +00:00
|
|
|
Status FindMetaBlock(InternalIterator* meta_index_iter,
|
2014-05-15 21:09:03 +00:00
|
|
|
const std::string& meta_block_name,
|
|
|
|
BlockHandle* block_handle) {
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
Status s =
|
|
|
|
FindOptionalMetaBlock(meta_index_iter, meta_block_name, block_handle);
|
|
|
|
if (s.ok() && block_handle->IsNull()) {
|
2014-05-15 21:09:03 +00:00
|
|
|
return Status::Corruption("Cannot find the meta block", meta_block_name);
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
} else {
|
|
|
|
return s;
|
2014-05-15 21:09:03 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-04-06 22:54:40 +00:00
|
|
|
Status ReadMetaIndexBlockInFile(RandomAccessFileReader* file,
|
|
|
|
uint64_t file_size, uint64_t table_magic_number,
|
|
|
|
const ImmutableOptions& ioptions,
|
2023-04-21 16:07:18 +00:00
|
|
|
const ReadOptions& read_options,
|
2022-04-06 22:54:40 +00:00
|
|
|
BlockContents* metaindex_contents,
|
|
|
|
MemoryAllocator* memory_allocator,
|
|
|
|
FilePrefetchBuffer* prefetch_buffer,
|
|
|
|
Footer* footer_out) {
|
2015-01-13 22:33:04 +00:00
|
|
|
Footer footer;
|
2020-06-29 21:51:57 +00:00
|
|
|
IOOptions opts;
|
2023-04-21 16:07:18 +00:00
|
|
|
Status s;
|
|
|
|
s = file->PrepareIOOptions(read_options, opts);
|
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
|
|
|
s = ReadFooterFromFile(opts, file, *ioptions.fs, prefetch_buffer, file_size,
|
|
|
|
&footer, table_magic_number);
|
2014-07-18 23:58:13 +00:00
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
if (footer_out) {
|
|
|
|
*footer_out = footer;
|
|
|
|
}
|
2014-07-18 23:58:13 +00:00
|
|
|
|
|
|
|
auto metaindex_handle = footer.metaindex_handle();
|
2023-04-21 16:07:18 +00:00
|
|
|
return BlockFetcher(file, prefetch_buffer, footer, read_options,
|
2022-04-06 22:54:40 +00:00
|
|
|
metaindex_handle, metaindex_contents, ioptions,
|
|
|
|
false /* do decompression */, false /*maybe_compressed*/,
|
|
|
|
BlockType::kMetaIndex, UncompressionDict::GetEmptyDict(),
|
|
|
|
PersistentCacheOptions::kEmpty, memory_allocator)
|
|
|
|
.ReadBlockContents();
|
|
|
|
}
|
|
|
|
|
2023-04-21 16:07:18 +00:00
|
|
|
Status FindMetaBlockInFile(
|
|
|
|
RandomAccessFileReader* file, uint64_t file_size,
|
|
|
|
uint64_t table_magic_number, const ImmutableOptions& ioptions,
|
|
|
|
const ReadOptions& read_options, const std::string& meta_block_name,
|
|
|
|
BlockHandle* block_handle, MemoryAllocator* memory_allocator,
|
|
|
|
FilePrefetchBuffer* prefetch_buffer, Footer* footer_out) {
|
2014-07-18 23:58:13 +00:00
|
|
|
BlockContents metaindex_contents;
|
2022-04-06 22:54:40 +00:00
|
|
|
auto s = ReadMetaIndexBlockInFile(
|
2023-04-21 16:07:18 +00:00
|
|
|
file, file_size, table_magic_number, ioptions, read_options,
|
|
|
|
&metaindex_contents, memory_allocator, prefetch_buffer, footer_out);
|
2014-07-18 23:58:13 +00:00
|
|
|
if (!s.ok()) {
|
|
|
|
return s;
|
|
|
|
}
|
2018-11-14 01:00:49 +00:00
|
|
|
// meta blocks are never compressed. Need to add uncompress logic if we are to
|
|
|
|
// compress it.
|
2020-02-25 23:29:17 +00:00
|
|
|
Block metaindex_block(std::move(metaindex_contents));
|
2014-07-18 23:58:13 +00:00
|
|
|
|
2015-10-12 22:06:38 +00:00
|
|
|
std::unique_ptr<InternalIterator> meta_iter;
|
2021-12-21 19:29:52 +00:00
|
|
|
meta_iter.reset(metaindex_block.NewMetaIterator());
|
2014-07-18 23:58:13 +00:00
|
|
|
|
|
|
|
return FindMetaBlock(meta_iter.get(), meta_block_name, block_handle);
|
|
|
|
}
|
|
|
|
|
2017-08-11 18:59:13 +00:00
|
|
|
Status ReadMetaBlock(RandomAccessFileReader* file,
|
|
|
|
FilePrefetchBuffer* prefetch_buffer, uint64_t file_size,
|
2016-07-19 16:44:03 +00:00
|
|
|
uint64_t table_magic_number,
|
2021-05-05 20:59:21 +00:00
|
|
|
const ImmutableOptions& ioptions,
|
2023-04-21 16:07:18 +00:00
|
|
|
const ReadOptions& read_options,
|
2019-06-19 02:00:03 +00:00
|
|
|
const std::string& meta_block_name, BlockType block_type,
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
BlockContents* contents,
|
2018-11-29 01:58:08 +00:00
|
|
|
MemoryAllocator* memory_allocator) {
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
// TableProperties requires special handling because of checksum issues.
|
|
|
|
// Call ReadTableProperties instead for that case.
|
|
|
|
assert(block_type != BlockType::kProperties);
|
2014-07-18 23:58:13 +00:00
|
|
|
|
|
|
|
BlockHandle block_handle;
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
Footer footer;
|
2023-04-21 16:07:18 +00:00
|
|
|
Status status =
|
|
|
|
FindMetaBlockInFile(file, file_size, table_magic_number, ioptions,
|
|
|
|
read_options, meta_block_name, &block_handle,
|
|
|
|
memory_allocator, prefetch_buffer, &footer);
|
2014-08-15 22:05:09 +00:00
|
|
|
if (!status.ok()) {
|
|
|
|
return status;
|
2014-07-18 23:58:13 +00:00
|
|
|
}
|
|
|
|
|
2023-04-21 16:07:18 +00:00
|
|
|
return BlockFetcher(file, prefetch_buffer, footer, read_options, block_handle,
|
|
|
|
contents, ioptions, false /* decompress */,
|
Improve / clean up meta block code & integrity (#9163)
Summary:
* Checksums are now checked on meta blocks unless specifically
suppressed or not applicable (e.g. plain table). (Was other way around.)
This means a number of cases that were not checking checksums now are,
including direct read TableProperties in Version::GetTableProperties
(fixed in meta_blocks ReadTableProperties), reading any block from
PersistentCache (fixed in BlockFetcher), read TableProperties in
SstFileDumper (ldb/sst_dump/BackupEngine) before table reader open,
maybe more.
* For that to work, I moved the global_seqno+TableProperties checksum
logic to the shared table/ code, because that is used by many utilies
such as SstFileDumper.
* Also for that to work, we have to know when we're dealing with a block
that has a checksum (trailer), so added that capability to Footer based
on magic number, and from there BlockFetcher.
* Knowledge of trailer presence has also fixed a problem where other
table formats were reading blocks including bytes for a non-existant
trailer--and awkwardly kind-of not using them, e.g. no shared code
checking checksums. (BlockFetcher compression type was populated
incorrectly.) Now we only read what is needed.
* Minimized code duplication and differing/incompatible/awkward
abstractions in meta_blocks.{cc,h} (e.g. SeekTo in metaindex block
without parsing block handle)
* Moved some meta block handling code from table_properties*.*
* Moved some code specific to block-based table from shared table/ code
to BlockBasedTable class. The checksum stuff means we can't completely
separate it, but things that don't need to be in shared table/ code
should not be.
* Use unique_ptr rather than raw ptr in more places. (Note: you can
std::move from unique_ptr to shared_ptr.)
Without enhancements to GetPropertiesOfAllTablesTest (see below),
net reduction of roughly 100 lines of code.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/9163
Test Plan:
existing tests and
* Enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to verify that
checksums are now checked on direct read of table properties by TableCache
(new test would fail before this change)
* Also enhanced DBTablePropertiesTest.GetPropertiesOfAllTablesTest to test
putting table properties under old meta name
* Also generally enhanced that same test to actually test what it was
supposed to be testing already, by kicking things out of table cache when
we don't want them there.
Reviewed By: ajkr, mrambacher
Differential Revision: D32514757
Pulled By: pdillinger
fbshipit-source-id: 507964b9311d186ae8d1131182290cbd97a99fa9
2021-11-18 19:42:12 +00:00
|
|
|
false /*maybe_compressed*/, block_type,
|
|
|
|
UncompressionDict::GetEmptyDict(),
|
|
|
|
PersistentCacheOptions::kEmpty, memory_allocator)
|
|
|
|
.ReadBlockContents();
|
2014-07-18 23:58:13 +00:00
|
|
|
}
|
|
|
|
|
2020-02-20 20:07:53 +00:00
|
|
|
} // namespace ROCKSDB_NAMESPACE
|