dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
rocksdb/utilities
History
Changyu Bi c2aad555c3 Add `CompressionOptions::checksum` for enabling ZSTD checksum (#11666) 
Summary:
Optionally enable zstd checksum flag (d857369028/lib/zstd.h (L428)) to detect corruption during decompression. Main changes are in compression.h:
* User can set CompressionOptions::checksum to true to enable this feature.
* We enable this feature in ZSTD by setting the checksum flag in ZSTD compression context: `ZSTD_CCtx`.
* Uses `ZSTD_compress2()` to do compression since it supports frame parameter like the checksum flag. Compression level is also set in compression context as a flag.
* Error handling during decompression to propagate error message from ZSTD.
* Updated microbench to test read performance impact.

About compatibility, the current compression decoders should continue to work with the data created by the new compression API `ZSTD_compress2()`: https://github.com/facebook/zstd/issues/3711.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11666

Test Plan:
* Existing unit tests for zstd compression
* Add unit test `DBTest2.ZSTDChecksum` to test the corruption case
* Manually tested that compression levels, parallel compression, dictionary compression, index compression all work with the new ZSTD_compress2() API.
* Manually tested with `sst_dump --command=recompress` that different compression levels and dictionary compression settings all work.
* Manually tested compiling with older versions of ZSTD: v1.3.8, v1.1.0, v0.6.2.
* Perf impact: from public benchmark data: http://fastcompression.blogspot.com/2019/03/presenting-xxh3.html for checksum and https://github.com/facebook/zstd#benchmarks, if decompression is 1700MB/s and checksum computation is 70000MB/s, checksum computation is an additional ~2.4% time for decompression. Compression is slower and checksumming should be less noticeable.
* Microbench:
```
TEST_TMPDIR=/dev/shm ./branch_db_basic_bench --benchmark_filter=DBGet/comp_style:0/max_data:1048576/per_key_size:256/enable_statistics:0/negative_query:0/enable_filter:0/mmap:0/compression_type:7/compression_checksum:1/no_blockcache:1/iterations:10000/threads:1 --benchmark_repetitions=100

Min out of 100 runs:
Main:
10390 10436 10456 10484 10499 10535 10544 10545 10565 10568

After this PR, checksum=false
10285 10397 10503 10508 10515 10557 10562 10635 10640 10660

After this PR, checksum=true
10827 10876 10925 10949 10971 11052 11061 11063 11100 11109
```
* db_bench:
```
Write perf
TEST_TMPDIR=/dev/shm/ ./db_bench_ichecksum --benchmarks=fillseq[-X10] --compression_type=zstd --num=10000000 --compression_checksum=..

[FillSeq checksum=0]
fillseq [AVG    10 runs] : 281635 (± 31711) ops/sec;   31.2 (± 3.5) MB/sec
fillseq [MEDIAN 10 runs] : 294027 ops/sec;   32.5 MB/sec

[FillSeq checksum=1]
fillseq [AVG    10 runs] : 286961 (± 34700) ops/sec;   31.7 (± 3.8) MB/sec
fillseq [MEDIAN 10 runs] : 283278 ops/sec;   31.3 MB/sec

Read perf
TEST_TMPDIR=/dev/shm ./db_bench_ichecksum --benchmarks=readrandom[-X20] --num=100000000 --reads=1000000 --use_existing_db=true --readonly=1

[Readrandom checksum=1]
readrandom [AVG    20 runs] : 360928 (± 3579) ops/sec;    4.0 (± 0.0) MB/sec
readrandom [MEDIAN 20 runs] : 362468 ops/sec;    4.0 MB/sec

[Readrandom checksum=0]
readrandom [AVG    20 runs] : 380365 (± 2384) ops/sec;    4.2 (± 0.0) MB/sec
readrandom [MEDIAN 20 runs] : 379800 ops/sec;    4.2 MB/sec

Compression
TEST_TMPDIR=/dev/shm ./db_bench_ichecksum --benchmarks=compress[-X20] --compression_type=zstd --num=100000000 --compression_checksum=1

checksum=1
compress [AVG    20 runs] : 54074 (± 634) ops/sec;  211.2 (± 2.5) MB/sec
compress [MEDIAN 20 runs] : 54396 ops/sec;  212.5 MB/sec

checksum=0
compress [AVG    20 runs] : 54598 (± 393) ops/sec;  213.3 (± 1.5) MB/sec
compress [MEDIAN 20 runs] : 54592 ops/sec;  213.3 MB/sec

Decompression:
TEST_TMPDIR=/dev/shm ./db_bench_ichecksum --benchmarks=uncompress[-X20] --compression_type=zstd --compression_checksum=1

checksum = 0
uncompress [AVG    20 runs] : 167499 (± 962) ops/sec;  654.3 (± 3.8) MB/sec
uncompress [MEDIAN 20 runs] : 167210 ops/sec;  653.2 MB/sec
checksum = 1
uncompress [AVG    20 runs] : 167980 (± 924) ops/sec;  656.2 (± 3.6) MB/sec
uncompress [MEDIAN 20 runs] : 168465 ops/sec;  658.1 MB/sec
```

Reviewed By: ajkr

Differential Revision: D48019378

Pulled By: cbi42

fbshipit-source-id: 674120c6e1853c2ced1436ac8138559d0204feba
 		2023-08-18 15:01:59 -07:00
..
agg_merge Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
backup Add missing status check when compiling with `ASSERT_STATUS_CHECKED=1` (#11686) 2023-08-09 15:46:44 -07:00
blob_db Add `CompressionOptions::checksum` for enabling ZSTD checksum (#11666) 2023-08-18 15:01:59 -07:00
cassandra Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
checkpoint Add missing status check when compiling with `ASSERT_STATUS_CHECKED=1` (#11686) 2023-08-09 15:46:44 -07:00
compaction_filters Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
convenience Run clang-format on utilities/ (except utilities/transactions/) (#10853) 2022-10-24 16:38:09 -07:00
leveldb_options Put Cache and CacheWrapper in new public header (#11192) 2023-02-09 12:12:02 -08:00
memory Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
merge_operators Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
option_change_migration Add missing status check when compiling with `ASSERT_STATUS_CHECKED=1` (#11686) 2023-08-09 15:46:44 -07:00
options Initial add UDT in memtable only option (#11362) 2023-04-11 17:50:34 -07:00
persistent_cache Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
simulator_cache Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
table_properties_collectors Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
trace Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
transactions Add missing status check when compiling with `ASSERT_STATUS_CHECKED=1` (#11686) 2023-08-09 15:46:44 -07:00
ttl Add missing status check when compiling with `ASSERT_STATUS_CHECKED=1` (#11686) 2023-08-09 15:46:44 -07:00
write_batch_with_index Add missing status check when compiling with `ASSERT_STATUS_CHECKED=1` (#11686) 2023-08-09 15:46:44 -07:00
cache_dump_load.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
cache_dump_load_impl.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
cache_dump_load_impl.h Group rocksdb.sst.read.micros stat by different user read IOActivity + misc (#11444) 2023-08-08 17:26:50 -07:00
compaction_filters.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
counted_fs.cc Fix serious FSDirectory use-after-Close bug (missing fsync) (#10460) 2022-08-02 10:54:32 -07:00
counted_fs.h Explicitly closing all directory file descriptors (#10049) 2022-06-01 18:03:34 -07:00
debug.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_mirror.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_mirror_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_timed.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_timed.h Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
env_timed_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
fault_injection_env.cc Explicitly closing all directory file descriptors (#10049) 2022-06-01 18:03:34 -07:00
fault_injection_env.h Run clang-format on utilities/ (except utilities/transactions/) (#10853) 2022-10-24 16:38:09 -07:00
fault_injection_fs.cc Use FaultInjectionTestFS in transaction_test, clarify Close() APIs (#11499) 2023-06-21 23:38:54 -07:00
fault_injection_fs.h Enable ReadAsync testing and fault injection in db_stress (#11037) 2022-12-15 15:48:50 -08:00
fault_injection_secondary_cache.cc Implement a allow cache hits admission policy for the compressed secondary cache (#11713) 2023-08-18 11:19:48 -07:00
fault_injection_secondary_cache.h Implement a allow cache hits admission policy for the compressed secondary cache (#11713) 2023-08-18 11:19:48 -07:00
memory_allocators.h Major Cache refactoring, CPU efficiency improvement (#10975) 2023-01-11 14:20:40 -08:00
merge_operators.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00
merge_operators.h Run clang-format on utilities/ (except utilities/transactions/) (#10853) 2022-10-24 16:38:09 -07:00
object_registry.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
object_registry_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
util_merge_operators_test.cc Print stack traces on frozen tests in CI (#10828) 2022-10-18 00:35:35 -07:00
wal_filter.cc Remove FactoryFunc from LoadXXXObject (#11203) 2023-02-17 12:54:07 -08:00