rocksdb/table
Hui Xiao 7d83b4e3e5 Fix file checksum mismatch due to padded bytes when block_align=true (#12542)
Summary:
**Context/Summary:**
When `BlockBasedTableOptions::block_align=true`, we pad bytes to align blocks d41e568b1c/table/block_based/block_based_table_builder.cc (L1415-L1421).
Those bytes are not included in generating the file checksum upon file creation. But `VerifyFileChecksums()` includes those bytes in generating the file check to compare against the checksum generating upon file creation. Therefore a file checksum mismatch is returned in `VerifyFileChecksums()`.

We decided to include those padded bytes in generating the checksum upon file creation.

Bonus: also fix surrounding code to use actual padded bytes for verification - see https://github.com/facebook/rocksdb/pull/12542#discussion_r1571429163

Pull Request resolved: https://github.com/facebook/rocksdb/pull/12542

Test Plan:
- New UT
- Benchmark
```
TEST_TMPDIR=/dev/shm  ./db_bench --benchmarks=fillseq[-X300] --num=100000 --block_align=1 --compression_type=none
```
Pre-PR:
fillseq [AVG    300 runs] : 422857 (± 3942) ops/sec;   46.8 (± 0.4) MB/sec
Post-PR:
fillseq [AVG    300 runs] : 424707 (± 3799) ops/sec;   47.0 (± 0.4) MB/sec

Reviewed By: ajkr

Differential Revision: D56168447

Pulled By: hx235

fbshipit-source-id: 96209ef950d42943d336f11968ae3fcf9872fc2c
2024-04-22 14:07:34 -07:00
..
adaptive Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
block_based Don't swallow errors in BlockBasedTable::MultiGet (#12486) 2024-03-28 13:56:28 -07:00
cuckoo Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
plain Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
block_fetcher.cc Fix heap use after free error on retry after checksum mismatch (#12464) 2024-03-21 16:19:09 -07:00
block_fetcher.h Fix heap use after free error on retry after checksum mismatch (#12464) 2024-03-21 16:19:09 -07:00
block_fetcher_test.cc Use format_version=6 by default (#12352) 2024-02-15 11:23:48 -08:00
cleanable_test.cc
compaction_merging_iterator.cc Fix a bug where iterator status is not checked (#11782) 2023-09-01 09:34:08 -07:00
compaction_merging_iterator.h
format.cc Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
format.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
get_context.cc Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
get_context.h Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
internal_iterator.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
iter_heap.h
iterator.cc
iterator_wrapper.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
merger_test.cc internal_repo_rocksdb (435146444452818992) (#12115) 2023-12-01 11:15:17 -08:00
merging_iterator.cc Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
merging_iterator.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
meta_blocks.cc Rename IntTblPropCollector -> InternalTblPropColl (#12320) 2024-02-02 14:14:43 -08:00
meta_blocks.h Rename IntTblPropCollector -> InternalTblPropColl (#12320) 2024-02-02 14:14:43 -08:00
mock_table.cc Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
mock_table.h Remove 'virtual' when implied by 'override' (#12319) 2024-01-31 13:14:42 -08:00
multiget_context.h
persistent_cache_helper.cc Remove extra semi colon from internal_repo_rocksdb/repo/db/table_cache_sync_and_async.h 2024-03-04 06:34:44 -08:00
persistent_cache_helper.h Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
persistent_cache_options.h Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
sst_file_dumper.cc Add initial support for TimedPut API (#12419) 2024-03-14 15:44:55 -07:00
sst_file_dumper.h Augment sst_dump tool to verify num_entries in table property (#12322) 2024-02-01 14:35:03 -08:00
sst_file_reader.cc Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
sst_file_reader_test.cc Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
sst_file_writer.cc Fix/improve temperature handling for file ingestion (#12402) 2024-03-05 16:56:08 -08:00
sst_file_writer_collectors.h Rename IntTblPropCollector -> InternalTblPropColl (#12320) 2024-02-02 14:14:43 -08:00
table_builder.h Remove extra semi colon from internal_repo_rocksdb/repo/db/table_cache_sync_and_async.h 2024-03-04 06:34:44 -08:00
table_factory.cc
table_iterator.h Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
table_properties.cc Fix/cleanup SeqnoToTimeMapping (#12253) 2024-01-19 21:50:38 -08:00
table_properties_internal.h
table_reader.h Add new Iterator API Refresh(const snapshot*) (#10594) 2023-09-15 10:44:43 -07:00
table_reader_bench.cc Rename IntTblPropCollector -> InternalTblPropColl (#12320) 2024-02-02 14:14:43 -08:00
table_test.cc Fix file checksum mismatch due to padded bytes when block_align=true (#12542) 2024-04-22 14:07:34 -07:00
two_level_iterator.cc internal_repo_rocksdb (435146444452818992) (#12115) 2023-12-01 11:15:17 -08:00
two_level_iterator.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
unique_id.cc internal_repo_rocksdb (435146444452818992) (#12115) 2023-12-01 11:15:17 -08:00
unique_id_impl.h