mirror of
https://github.com/facebook/rocksdb.git
synced 2024-11-26 16:30:56 +00:00
e6432dfd4c
Summary: Currently, if blob files are enabled (i.e. `enable_blob_files` is true), large values are extracted both during flush/recovery (when SST files are written into level 0 of the LSM tree) and during compaction into any LSM tree level. For certain use cases that have a mix of short-lived and long-lived values, it might make sense to support extracting large values only during compactions whose output level is greater than or equal to a specified LSM tree level (e.g. compactions into L1/L2/... or above). This could reduce the space amplification caused by large values that are turned into garbage shortly after being written at the price of some write amplification incurred by long-lived values whose extraction to blob files is delayed. In order to achieve this, we would like to do the following: - Add a new configuration option `blob_file_starting_level` (default: 0) to `AdvancedColumnFamilyOptions` (and `MutableCFOptions` and extend the related logic) - Instantiate `BlobFileBuilder` in `BuildTable` (used during flush and recovery, where the LSM tree level is L0) and `CompactionJob` iff `enable_blob_files` is set and the LSM tree level is `>= blob_file_starting_level` - Add unit tests for the new functionality, and add the new option to our stress tests (`db_stress` and `db_crashtest.py` ) - Add the new option to our benchmarking tool `db_bench` and the BlobDB benchmark script `run_blob_bench.sh` - Add the new option to the `ldb` tool (see https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool) - Ideally extend the C and Java bindings with the new option - Update the BlobDB wiki to document the new option. Pull Request resolved: https://github.com/facebook/rocksdb/pull/10077 Reviewed By: ltamasi Differential Revision: D36884156 Pulled By: gangliao fbshipit-source-id: 942bab025f04633edca8564ed64791cb5e31627d |
||
|---|---|---|
| .. | ||
| blob_constants.h | ||
| blob_counting_iterator.h | ||
| blob_counting_iterator_test.cc | ||
| blob_fetcher.cc | ||
| blob_fetcher.h | ||
| blob_file_addition.cc | ||
| blob_file_addition.h | ||
| blob_file_addition_test.cc | ||
| blob_file_builder.cc | ||
| blob_file_builder.h | ||
| blob_file_builder_test.cc | ||
| blob_file_cache.cc | ||
| blob_file_cache.h | ||
| blob_file_cache_test.cc | ||
| blob_file_completion_callback.h | ||
| blob_file_garbage.cc | ||
| blob_file_garbage.h | ||
| blob_file_garbage_test.cc | ||
| blob_file_meta.cc | ||
| blob_file_meta.h | ||
| blob_file_reader.cc | ||
| blob_file_reader.h | ||
| blob_file_reader_test.cc | ||
| blob_garbage_meter.cc | ||
| blob_garbage_meter.h | ||
| blob_garbage_meter_test.cc | ||
| blob_index.h | ||
| blob_log_format.cc | ||
| blob_log_format.h | ||
| blob_log_sequential_reader.cc | ||
| blob_log_sequential_reader.h | ||
| blob_log_writer.cc | ||
| blob_log_writer.h | ||
| db_blob_basic_test.cc | ||
| db_blob_compaction_test.cc | ||
| db_blob_corruption_test.cc | ||
| db_blob_index_test.cc | ||
| prefetch_buffer_collection.cc | ||
| prefetch_buffer_collection.h | ||