dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-30 22:41:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/util
History
Andrew Kryczka 6c40806e51 Digest ZSTD compression dictionary once per SST file (#4251) 
Summary:
In RocksDB, for a given SST file, all data blocks are compressed with the same dictionary. When we compress a block using the dictionary's raw bytes, the compression library first has to digest the dictionary to get it into a usable form. This digestion work is redundant and ideally should be done once per file.

ZSTD offers APIs for the caller to create and reuse a digested dictionary object (`ZSTD_CDict`). In this PR, we call `ZSTD_createCDict` once per file to digest the raw bytes. Then we use `ZSTD_compress_usingCDict` to compress each data block using the pre-digested dictionary. Once the file's created `ZSTD_freeCDict` releases the resources held by the digested dictionary.

There are a couple other changes included in this PR:

- Changed the parameter object for (un)compression functions from `CompressionContext`/`UncompressionContext` to `CompressionInfo`/`UncompressionInfo`. This avoids the previous pattern, where `CompressionContext`/`UncompressionContext` had to be mutated before calling a (un)compression function depending on whether dictionary should be used. I felt that mutation was error-prone so eliminated it.
- Added support for digested uncompression dictionaries (`ZSTD_DDict`) as well. However, this PR does not support reusing them across uncompression calls for the same file. That work is deferred to a later PR when we will store the `ZSTD_DDict` objects in block cache.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/4251

Differential Revision: D9257078

Pulled By: ajkr

fbshipit-source-id: 21b8cb6bbdd48e459f1c62343780ab66c0a64438
2018-08-23 19:28:18 -07:00
..
aligned_buffer.h Fix Copying of data between buffers in FilePrefetchBuffer (#4100) 2018-07-11 12:28:13 -07:00
allocator.h Change RocksDB License 2017-07-15 16:11:23 -07:00
arena.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
arena.h arena: derive alignment unit from std::max_align_t 2017-10-17 11:13:19 -07:00
arena_test.cc arena: derive alignment unit from std::max_align_t 2017-10-17 11:13:19 -07:00
auto_roll_logger.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
auto_roll_logger.h Fix the Logger::Close() and DBImpl::Close() design pattern 2018-02-23 13:57:26 -08:00
auto_roll_logger_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
autovector.h Change RocksDB License 2017-07-15 16:11:23 -07:00
autovector_test.cc fix memory leak in two_level_iterator 2018-04-15 17:26:26 -07:00
bloom.cc Improve FullFilterBitsReader::HashMayMatch's doc (#4202) 2018-08-06 11:13:18 -07:00
bloom_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
build_version.cc.in Makefile: generate util/build_version.cc from .in file (#1384) 2016-10-25 11:31:39 -07:00
build_version.h Change RocksDB License 2017-07-15 16:11:23 -07:00
cast_util.h Add a missing "once" in .h 2017-07-31 12:12:03 -07:00
channel.h Change RocksDB License 2017-07-15 16:11:23 -07:00
coding.cc Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
coding.h Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
coding_test.cc Coding.h: Added Fixed16 support (#4142) 2018-07-16 23:43:41 -07:00
compaction_job_stats_impl.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
comparator.cc Improve point-lookup performance using a data block hash index (#4174) 2018-08-15 14:30:03 -07:00
compression.h Digest ZSTD compression dictionary once per SST file (#4251) 2018-08-23 19:28:18 -07:00
compression_context_cache.cc run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
compression_context_cache.h run make format for PR 3838 (#3954) 2018-06-05 12:58:02 -07:00
concurrent_arena.cc Cap concurrent arena's shard block size to 128KB (#4147) 2018-07-18 10:43:54 -07:00
concurrent_arena.h util: Fix coverity issues 2017-11-03 14:42:08 -07:00
core_local.h Change RocksDB License 2017-07-15 16:11:23 -07:00
crc32c.cc Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
crc32c.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_ppc.c Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_ppc.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_ppc_asm.S Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
crc32c_ppc_constants.h Make it explicit blob db doesn't support CF 2017-09-08 11:11:04 -07:00
crc32c_test.cc Build and tests fixes for Solaris Sparc (#4000) 2018-06-15 12:42:53 -07:00
delete_scheduler.cc In delete scheduler, before ftruncate file for slow delete, check whether there is other hard links (#4093) 2018-07-09 15:28:12 -07:00
delete_scheduler.h In delete scheduler, before ftruncate file for slow delete, check whether there is other hard links (#4093) 2018-07-09 15:28:12 -07:00
delete_scheduler_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
duplicate_detector.h Improve db_stress with transactions 2018-04-18 16:32:35 -07:00
dynamic_bloom.cc Use nullptr instead of NULL / 0 more consistently. 2018-03-07 12:42:12 -08:00
dynamic_bloom.h Enable MSVC W4 with a few exceptions. Fix warnings and bugs 2017-10-19 10:57:12 -07:00
dynamic_bloom_test.cc Two code changes to make "clang analyze" happy (#4292) 2018-08-20 17:43:41 -07:00
event_logger.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
event_logger_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
fault_injection_test_env.cc Allow DB resume after background errors (#3997) 2018-06-28 12:34:40 -07:00
fault_injection_test_env.h Allow DB resume after background errors (#3997) 2018-06-28 12:34:40 -07:00
file_reader_writer.cc RocksDB Trace Analyzer (#4091) 2018-08-13 11:44:02 -07:00
file_reader_writer.h Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
file_reader_writer_test.cc Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
file_util.cc Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
file_util.h Sync parent directory after deleting a file in delete scheduler 2018-04-26 13:58:20 -07:00
filelock_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
filename.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
filename.h Change RocksDB License 2017-07-15 16:11:23 -07:00
filter_policy.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
gflags_compat.h fix gflags namespace 2017-12-01 10:42:05 -08:00
hash.cc Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
hash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
hash_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
heap.h Change RocksDB License 2017-07-15 16:11:23 -07:00
heap_test.cc fix gflags namespace 2017-12-01 10:42:05 -08:00
kv_map.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_buffer.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
log_buffer.h Change RocksDB License 2017-07-15 16:11:23 -07:00
log_write_bench.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
logging.h Change RocksDB License 2017-07-15 16:11:23 -07:00
memory_usage.h Change RocksDB License 2017-07-15 16:11:23 -07:00
murmurhash.cc Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
murmurhash.h Change RocksDB License 2017-07-15 16:11:23 -07:00
mutexlock.h Change RocksDB License 2017-07-15 16:11:23 -07:00
ppc-opcode.h Updated CRC32 Power Optimization Changes 2017-08-31 14:16:30 -07:00
random.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
random.h Change RocksDB License 2017-07-15 16:11:23 -07:00
rate_limiter.cc rate limit auto-tuning 2017-10-04 19:15:01 -07:00
rate_limiter.h rate limit auto-tuning 2017-10-04 19:15:01 -07:00
rate_limiter_test.cc comment unused parameters to turn on -Wunused-parameter flag 2018-04-12 17:59:16 -07:00
set_comparator.h WritePrepared Txn: Move DuplicateDetector to util 2018-03-05 23:57:12 -08:00
slice.cc use user_key and iterate_upper_bound to determine compatibility of bloom filters (#3899) 2018-06-26 15:57:26 -07:00
slice_transform_test.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
sst_file_manager_impl.cc Expose GetTotalTrashSize in SstFileManager interface (#4206) 2018-08-04 17:57:48 -07:00
sst_file_manager_impl.h Expose GetTotalTrashSize in SstFileManager interface (#4206) 2018-08-04 17:57:48 -07:00
status.cc move static msgs out of Status class (#4144) 2018-07-23 15:44:16 -07:00
stderr_logger.h Change RocksDB License 2017-07-15 16:11:23 -07:00
stop_watch.h Exclude time waiting for rate limiter from rocksdb.sst.read.micros (#4102) 2018-07-13 18:44:14 -07:00
string_util.cc Several small "fixes" 2018-02-15 16:57:37 -08:00
string_util.h Change RocksDB License 2017-07-15 16:11:23 -07:00
sync_point.cc Refactor sync_point to make implementation either customizable or replaceable 2018-03-23 12:56:52 -07:00
sync_point.h Fix singleton destruction order of PosixEnv and SyncPoint (#3951) 2018-06-04 15:58:46 -07:00
sync_point_impl.cc Avoid acquiring SyncPoint mutex when it is disabled (#3991) 2018-06-13 13:13:18 -07:00
sync_point_impl.h Avoid acquiring SyncPoint mutex when it is disabled (#3991) 2018-06-13 13:13:18 -07:00
testharness.cc Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
testharness.h Per-thread unique test db names (#4135) 2018-07-13 17:27:39 -07:00
testutil.cc Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
testutil.h Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
thread_list_test.cc Use nullptr instead of NULL / 0 more consistently. 2018-03-07 12:42:12 -08:00
thread_local.cc Reformatting some recent changes (#4161) 2018-07-20 14:43:38 -07:00
thread_local.h Provide a way to override windows memory allocator with jemalloc for ZSTD 2018-06-04 12:12:48 -07:00
thread_local_test.cc Comment out unused variables 2018-03-05 13:13:41 -08:00
thread_operation.h Change RocksDB License 2017-07-15 16:11:23 -07:00
threadpool_imp.cc Suppress clang analyzer error (#4299) 2018-08-21 16:43:05 -07:00
threadpool_imp.h Support lowering CPU priority of background threads 2018-04-24 08:41:51 -07:00
timer_queue.h Change RocksDB License 2017-07-15 16:11:23 -07:00
timer_queue_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
trace_replay.cc Add tracing function of Seek() and SeekForPrev() to trace_replay (#4228) 2018-08-10 17:57:40 -07:00
trace_replay.h Add tracing function of Seek() and SeekForPrev() to trace_replay (#4228) 2018-08-10 17:57:40 -07:00
transaction_test_util.cc WriteUnPrepared: Implement unprepared batches for transactions (#4104) 2018-07-24 00:13:18 -07:00
transaction_test_util.h WritePrepared Txn: stress test 2017-12-06 09:42:28 -08:00
util.h Add GCC 8 to Travis (#3433) 2018-07-13 10:58:06 -07:00
xxhash.cc fixed typo 2017-06-05 11:27:34 -07:00
xxhash.h Prevent xxhash symbols from polluting global namespace 2015-03-12 12:07:10 -07:00