dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-27 02:44:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/table
History
Zhongyi Xie 26015f3b48 add compression options to table properties (#5081) 
Summary:
Since we are planning to use dictionary compression and to use different compression level, it is quite useful to add compression options to TableProperties. For example, in MyRocks, if the feature is available, we can query from information_schema.rocksdb_sst_props to see if all sst files are converted to ZSTD dictionary compressions. Resolves https://github.com/facebook/rocksdb/issues/4992

With this PR, user can query table properties through `GetPropertiesOfAllTables` API and get compression options as std::string:
`window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0;`
or table_properties->ToString() will also contain it
`# data blocks=1; # entries=13; # deletions=0; # merge operands=0; # range deletions=0; raw key size=143; raw average key size=11.000000; raw value size=39; raw average value size=3.000000; data block size=120; index block size (user-key? 0, delta-value? 0)=27; filter block size=0; (estimated) table size=147; filter policy name=N/A; prefix extractor name=nullptr; column family ID=0; column family name=default; comparator name=leveldb.BytewiseComparator; merge operator name=nullptr; property collectors names=[]; SST file compression algo=Snappy; SST file compression options=window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ; creation time=1552946632; time stamp of earliest key=1552946632;`
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5081

Differential Revision: D14716692

Pulled By: miasantreble

fbshipit-source-id: 7d2f2cf84e052bff876e71b4212cfdebf5be32dd
2019-04-02 14:52:34 -07:00
..
adaptive_table_factory.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
adaptive_table_factory.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
block.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_filter_block_test.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_based_table_builder.cc add compression options to table properties (#5081) 2019-04-02 14:52:34 -07:00
block_based_table_builder.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
block_based_table_factory.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
block_based_table_factory.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
block_based_table_reader.cc Revert "Avoid per-key upper bound check in BlockBasedTableIterator (#5101)" (#5132) 2019-04-02 10:00:38 -07:00
block_based_table_reader.h Revert "Avoid per-key upper bound check in BlockBasedTableIterator (#5101)" (#5132) 2019-04-02 10:00:38 -07:00
block_builder.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_builder.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_fetcher.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_fetcher.h Cache dictionary used for decompressing data blocks (#4881) 2019-01-23 18:15:47 -08:00
block_prefix_index.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_prefix_index.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
block_test.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
bloom_block.cc
bloom_block.h Disallow customized hash function in DynamicBloom (#4915) 2019-01-24 10:34:30 -08:00
cleanable_test.cc
cuckoo_table_builder.cc Promote rocksdb.{deleted.keys,merge.operands} to main table properties (#4594) 2018-10-30 15:34:27 -07:00
cuckoo_table_builder.h
cuckoo_table_builder_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_factory.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_factory.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
cuckoo_table_reader.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
cuckoo_table_reader.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
cuckoo_table_reader_test.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
data_block_footer.cc Add db_bench options of data block hash index (#4281) 2018-08-16 18:42:46 -07:00
data_block_footer.h Add db_bench options of data block hash index (#4281) 2018-08-16 18:42:46 -07:00
data_block_hash_index.cc DataBlockHashIndex: Remove the division from EstimateSize() (#4293) 2018-08-20 23:13:50 -07:00
data_block_hash_index.h DataBlockHashIndex: Remove the division from EstimateSize() (#4293) 2018-08-20 23:13:50 -07:00
data_block_hash_index_test.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
filter_block.h
flush_block_policy.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
format.cc Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
format.h Apply automatic formatting to some files (#5114) 2019-03-27 16:24:45 -07:00
full_filter_bits_builder.h
full_filter_block.cc
full_filter_block.h
full_filter_block_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
get_context.cc PlainTable should avoid copying Get() results from immortal source. (#4924) 2019-01-25 17:12:19 -08:00
get_context.h Cache dictionary used for decompressing data blocks (#4881) 2019-01-23 18:15:47 -08:00
index_builder.cc Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
index_builder.h Add path to WritableFileWriter. (#4039) 2018-08-23 10:12:58 -07:00
internal_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
iter_heap.h
iterator.cc Fix arena allocation size in NewEmptyInternalIterator (#4905) 2019-03-29 15:09:35 -07:00
iterator_wrapper.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
merger_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
merging_iterator.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
merging_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
meta_blocks.cc add compression options to table properties (#5081) 2019-04-02 14:52:34 -07:00
meta_blocks.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
mock_table.cc Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
mock_table.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
partitioned_filter_block.cc Fix bug in partition filters with format_version=4 (#4381) 2018-09-17 17:28:15 -07:00
partitioned_filter_block.h Fix bug in partition filters with format_version=4 (#4381) 2018-09-17 17:28:15 -07:00
partitioned_filter_block_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
persistent_cache_helper.cc Remove two variables from BlockContents class and don't use class Block for compressed block (#4650) 2018-11-13 17:02:55 -08:00
persistent_cache_helper.h
persistent_cache_options.h
plain_table_builder.cc Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_builder.h Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_factory.cc Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) 2019-03-01 15:45:45 -08:00
plain_table_factory.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
plain_table_index.cc
plain_table_index.h
plain_table_key_coding.cc
plain_table_key_coding.h Update all unique/shared_ptr instances to be qualified with namespace std (#4638) 2018-11-09 11:19:58 -08:00
plain_table_reader.cc Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
plain_table_reader.h Remove some "using std::..." from header files. (#5113) 2019-03-27 10:28:21 -07:00
scoped_arena_iterator.h
sst_file_reader.cc Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_reader_test.cc Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
sst_file_writer.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
sst_file_writer_collectors.h Fix SstFileReader not able to open ingested file (#5097) 2019-03-26 10:25:18 -07:00
table_builder.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties.cc add compression options to table properties (#5081) 2019-04-02 14:52:34 -07:00
table_properties_internal.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00
table_reader.h Clean up FragmentedRangeTombstoneList (#4692) 2018-11-28 15:29:02 -08:00
table_reader_bench.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_test.cc Revert "Avoid per-key upper bound check in BlockBasedTableIterator (#5101)" (#5132) 2019-04-02 10:00:38 -07:00
two_level_iterator.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
two_level_iterator.h Index value delta encoding (#3983) 2018-08-09 16:58:40 -07:00