rocksdb/file
Zhichao Cao 4369f2c7bb Checksum for each SST file and stores in MANIFEST (#6216)
Summary:
In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata.

Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string).  A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216

Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check.

Differential Revision: D19171461

Pulled By: zhichao-cao

fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87
2020-02-10 15:52:52 -08:00
..
delete_scheduler.cc Fix use-after-free and double-deleting files in BackgroundCallPurge() (#6193) 2019-12-17 20:08:56 -08:00
delete_scheduler.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
delete_scheduler_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
file_prefetch_buffer.cc Prevent file prefetch when mmap is enabled. (#6206) 2019-12-18 11:01:29 -08:00
file_prefetch_buffer.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
file_util.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
file_util.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
filename.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
filename.h Persistent globally unique DB ID in manifest (#5725) 2019-09-03 08:52:24 -07:00
random_access_file_reader.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
random_access_file_reader.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
read_write_util.cc Allow readahead when reading option files. (#6372) 2020-02-07 15:18:26 -08:00
read_write_util.h Allow readahead when reading option files. (#6372) 2020-02-07 15:18:26 -08:00
readahead_raf.cc Dedup IsFileSectorAligned() to fix unity build. (#5812) 2019-09-16 20:42:07 -07:00
readahead_raf.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
sequence_file_reader.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
sequence_file_reader.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
sst_file_manager_impl.cc Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363) 2020-02-04 13:41:53 -08:00
sst_file_manager_impl.h Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363) 2020-02-04 13:41:53 -08:00
writable_file_writer.cc Checksum for each SST file and stores in MANIFEST (#6216) 2020-02-10 15:52:52 -08:00
writable_file_writer.h Checksum for each SST file and stores in MANIFEST (#6216) 2020-02-10 15:52:52 -08:00