dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-25 22:44:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Flag CompressionOptions::parallel_threads to be experimental (#6781)

Browse source 
Summary:
The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781

Reviewed By: pdillinger

Differential Revision: D21330678

fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13
This commit is contained in:
sdong 2020-04-30 15:19:31 -07:00 committed by Facebook GitHub Bot
parent ab13d43e1d
commit 6277e28039
2 changed files with 6 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
HISTORY.md
View file

@ -16,7 +16,7 @@
* Add IsDirectory to Env and FS to indicate if a path is a directory. * Add IsDirectory to Env and FS to indicate if a path is a directory.
### New Features ### New Features
* Added support for pipelined & parallel compression optimization for `BlockBasedTableBuilder`. This optimization makes block building, block compression and block appending a pipeline, and uses multiple threads to accelerate block compression. Users can set `CompressionOptions::parallel_threads` greater than 1 to enable compression parallelism. * Added support for pipelined & parallel compression optimization for `BlockBasedTableBuilder`. This optimization makes block building, block compression and block appending a pipeline, and uses multiple threads to accelerate block compression. Users can set `CompressionOptions::parallel_threads` greater than 1 to enable compression parallelism. This feature is experimental for now.
* Provide an allocator for memkind to be used with block cache. This is to work with memory technologies (Intel DCPMM is one such technology currently available) that require different libraries for allocation and management (such as PMDK and memkind). The high capacities available make it possible to provision large caches (up to several TBs in size) beyond what is achievable with DRAM. * Provide an allocator for memkind to be used with block cache. This is to work with memory technologies (Intel DCPMM is one such technology currently available) that require different libraries for allocation and management (such as PMDK and memkind). The high capacities available make it possible to provision large caches (up to several TBs in size) beyond what is achievable with DRAM.
* Option `max_background_flushes` can be set dynamically using DB::SetDBOptions(). * Option `max_background_flushes` can be set dynamically using DB::SetDBOptions().
* Added functionality in sst_dump tool to check the compressed file size for different compression levels and print the time spent on compressing files with each compression type. Added arguments `--compression_level_from` and `--compression_level_to` to report size of all compression levels and one compression_type must be specified with it so that it will report compressed sizes of one compression type with different levels. * Added functionality in sst_dump tool to check the compressed file size for different compression levels and print the time spent on compressing files with each compression type. Added arguments `--compression_level_from` and `--compression_level_to` to report size of all compression levels and one compression_type must be specified with it so that it will report compressed sizes of one compression type with different levels.

11
include/rocksdb/advanced_options.h
View file

@ -119,15 +119,14 @@ struct CompressionOptions {
// Number of threads for parallel compression. // Number of threads for parallel compression.
// Parallel compression is enabled only if threads > 1. // Parallel compression is enabled only if threads > 1.
// THE FEATURE IS STILL EXPERIMENTAL
// //
// This option is valid only when BlockBasedTable is used. // This option is valid only when BlockBasedTable is used.
// //
// When parallel compression is enabled, SST size estimation becomes less // When parallel compression is enabled, SST size file sizes might be
// accurate, because block building and compression are pipelined, and there // more inflated compared to the target size, because more data of unknown
// might be inflight blocks being compressed and not finally written, when // compressed size is in flight when compression is parallelized. To be
// current SST size is fetched. This brings inflation of final output file // reasonably accurate, this inflation is also estimated by using historical
// size.
// To be more accurate, this inflation is also estimated by using historical
// compression ratio and current bytes inflight. // compression ratio and current bytes inflight.
// //
// Default: 1. // Default: 1.