mirror of
https://github.com/facebook/rocksdb.git
synced 2024-11-30 04:41:49 +00:00
6c40806e51
Summary: In RocksDB, for a given SST file, all data blocks are compressed with the same dictionary. When we compress a block using the dictionary's raw bytes, the compression library first has to digest the dictionary to get it into a usable form. This digestion work is redundant and ideally should be done once per file. ZSTD offers APIs for the caller to create and reuse a digested dictionary object (`ZSTD_CDict`). In this PR, we call `ZSTD_createCDict` once per file to digest the raw bytes. Then we use `ZSTD_compress_usingCDict` to compress each data block using the pre-digested dictionary. Once the file's created `ZSTD_freeCDict` releases the resources held by the digested dictionary. There are a couple other changes included in this PR: - Changed the parameter object for (un)compression functions from `CompressionContext`/`UncompressionContext` to `CompressionInfo`/`UncompressionInfo`. This avoids the previous pattern, where `CompressionContext`/`UncompressionContext` had to be mutated before calling a (un)compression function depending on whether dictionary should be used. I felt that mutation was error-prone so eliminated it. - Added support for digested uncompression dictionaries (`ZSTD_DDict`) as well. However, this PR does not support reusing them across uncompression calls for the same file. That work is deferred to a later PR when we will store the `ZSTD_DDict` objects in block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4251 Differential Revision: D9257078 Pulled By: ajkr fbshipit-source-id: 21b8cb6bbdd48e459f1c62343780ab66c0a64438 |
||
|---|---|---|
| .. | ||
| backupable | ||
| blob_db | ||
| cassandra | ||
| checkpoint | ||
| compaction_filters | ||
| convenience | ||
| date_tiered | ||
| document | ||
| geodb | ||
| leveldb_options | ||
| lua | ||
| memory | ||
| merge_operators | ||
| option_change_migration | ||
| options | ||
| persistent_cache | ||
| redis | ||
| simulator_cache | ||
| spatialdb | ||
| table_properties_collectors | ||
| trace | ||
| transactions | ||
| ttl | ||
| write_batch_with_index | ||
| col_buf_decoder.cc | ||
| col_buf_decoder.h | ||
| col_buf_encoder.cc | ||
| col_buf_encoder.h | ||
| column_aware_encoding_exp.cc | ||
| column_aware_encoding_test.cc | ||
| column_aware_encoding_util.cc | ||
| column_aware_encoding_util.h | ||
| debug.cc | ||
| env_librados.cc | ||
| env_librados.md | ||
| env_librados_test.cc | ||
| env_mirror.cc | ||
| env_mirror_test.cc | ||
| env_timed.cc | ||
| env_timed_test.cc | ||
| merge_operators.h | ||
| object_registry_test.cc | ||
| util_merge_operators_test.cc | ||