rocksdb/table
Jay Huh 57a8e69d4e Include TableProperties in the CompactionServiceResult (#13089)
Summary:
In Remote Compactions, the primary host receives the serialized compaction result from the remote worker and deserializes it to build the output. Unlike Local Compactions, where table properties are built by TableBuilder, in Remote Compactions, these properties were not included in the serialized compaction result. This was likely done intentionally since the table properties are already available in the SST files.

Because TableProperties are not populated as part of CompactionOutputs for remote compactions, we were unable to log the table properties in OnCompactionComplete and use them for verification. We are adding the TableProperties as part of the CompactionServiceOutputFile in this PR. By including the TableProperties in the serialized compaction result, the primary host will be able to access them and verify that they match the values read from the actual SST files.

We are also adding the populating `format_version` in table_properties of in TableBuilder.  This has not been a big issue because the `format_version` is written to the SST files directly from `TableOptions.format_version`. When loaded from the SST files, it's populated directly by reading from the MetaBlock. This info has only been missing in the TableBuilder's Rep.props.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13089

Test Plan:
```
./compaction_job_test
```
```
./compaction_service_test
```

Reviewed By: pdillinger

Differential Revision: D64878740

Pulled By: jaykorean

fbshipit-source-id: b6f2fdce851e6477ecb4dd5a87cdc62e176b746b
2024-10-25 13:13:12 -07:00
..
adaptive Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
block_based Include TableProperties in the CompactionServiceResult (#13089) 2024-10-25 13:13:12 -07:00
cuckoo Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
plain Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
block_fetcher.cc Add ticker stats for read corruption retries (#12923) 2024-08-12 15:32:07 -07:00
block_fetcher.h Fix stale memory access with FSBuffer and tiered sec cache (#12712) 2024-05-30 12:33:58 -07:00
block_fetcher_test.cc Use format_version=6 by default (#12352) 2024-02-15 11:23:48 -08:00
cleanable_test.cc Eliminate unnecessary (slow) block cache Ref()ing in MultiGet (#9899) 2022-04-26 21:59:24 -07:00
compaction_merging_iterator.cc Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
compaction_merging_iterator.h Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
format.cc Fix a couple of missing cases of retry on corruption (#13007) 2024-09-13 13:56:49 -07:00
format.h Fix a couple of missing cases of retry on corruption (#13007) 2024-09-13 13:56:49 -07:00
get_context.cc Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
get_context.h Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
internal_iterator.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
iter_heap.h Format files under table/ by clang-format (#10852) 2022-10-25 11:50:38 -07:00
iterator.cc Fix a bug for surfacing write unix time (#13057) 2024-10-08 11:31:51 -07:00
iterator_wrapper.h Support returning write unix time in iterator property (#12428) 2024-03-15 15:37:37 -07:00
merger_test.cc internal_repo_rocksdb (435146444452818992) (#12115) 2023-12-01 11:15:17 -08:00
merging_iterator.cc Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
merging_iterator.h Fix possible double-free on TruncatedRangeDelIterator (#12805) 2024-06-24 11:51:16 -07:00
meta_blocks.cc Fix a couple of missing cases of retry on corruption (#13007) 2024-09-13 13:56:49 -07:00
meta_blocks.h Add some checks at property block creation side (#12898) 2024-07-31 13:28:17 -07:00
mock_table.cc Fix kBlockCacheTier read when merge-chain base value is in a blob file (#12462) 2024-03-21 12:38:53 -07:00
mock_table.h Fix race to make BlockBasedTableOptions effectively mutable (#13082) 2024-10-25 10:24:54 -07:00
multiget_context.h Add a new MultiGetEntity API (#11222) 2023-02-15 09:34:17 -08:00
persistent_cache_helper.cc Remove extra semi colon from internal_repo_rocksdb/repo/db/table_cache_sync_and_async.h 2024-03-04 06:34:44 -08:00
persistent_cache_helper.h Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
persistent_cache_options.h Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
sst_file_dumper.cc Support read timestamp in ldb (#12641) 2024-05-13 15:43:12 -07:00
sst_file_dumper.h Augment sst_dump tool to verify num_entries in table property (#12322) 2024-02-01 14:35:03 -08:00
sst_file_reader.cc Remove extra semi colon from internal_repo_rocksdb/repo/table/sst_file_reader.cc 2024-05-22 07:14:52 -07:00
sst_file_reader_test.cc Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
sst_file_writer.cc Refactor `table_factory` into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
sst_file_writer_collectors.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
table_builder.h Add CompactForTieringCollector to support automatically trigger compaction for tiering use case (#12760) 2024-06-18 10:51:29 -07:00
table_factory.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
table_iterator.h Add support in SstFileReader to get a raw table iterator (#12385) 2024-04-02 21:23:06 -07:00
table_properties.cc Record largest seqno in table properties and verify in file ingestion (#12951) 2024-08-21 16:24:18 -07:00
table_properties_internal.h Improve / clean up meta block code & integrity (#9163) 2021-11-18 11:43:44 -08:00
table_reader.h Deshim coro in fbcode/internal_repo_rocksdb 2024-09-14 09:48:21 -07:00
table_reader_bench.cc Rename IntTblPropCollector -> InternalTblPropColl (#12320) 2024-02-02 14:14:43 -08:00
table_test.cc Refactor `table_factory` into MutableCFOptions (#13077) 2024-10-17 14:13:20 -07:00
two_level_iterator.cc internal_repo_rocksdb (435146444452818992) (#12115) 2023-12-01 11:15:17 -08:00
two_level_iterator.h Remove unnecessary, confusing 'extern' (#12300) 2024-01-29 10:38:08 -08:00
unique_id.cc internal_repo_rocksdb (435146444452818992) (#12115) 2023-12-01 11:15:17 -08:00
unique_id_impl.h Derive cache keys from SST unique IDs (#10394) 2022-08-12 13:49:49 -07:00