rocksdb/options
Baptiste Lemaire e3a96c4823 Memtable sampling for mempurge heuristic. (#8628)
Summary:
Changes the API of the MemPurge process: the `bool experimental_allow_mempurge` and `experimental_mempurge_policy` flags have been replaced by a `double experimental_mempurge_threshold` option.
This change of API reflects another major change introduced in this PR: the MemPurgeDecider() function now works by sampling the memtables being flushed to estimate the overall amount of useful payload (payload minus the garbage), and then compare this useful payload estimate with the `double experimental_mempurge_threshold` value.
Therefore, when the value of this flag is `0.0` (default value), mempurge is simply deactivated. On the other hand, a value of `DBL_MAX` would be equivalent to always going through a mempurge regardless of the garbage ratio estimate.
At the moment, a `double experimental_mempurge_threshold` value else than 0.0 or `DBL_MAX` is opnly supported`with the `SkipList` memtable representation.
Regarding the sampling, this PR includes the introduction of a `MemTable::UniqueRandomSample` function that collects (approximately) random entries from the memtable by using the new `SkipList::Iterator::RandomSeek()` under the hood, or by iterating through each memtable entry, depending on the target sample size and the total number of entries.
The unit tests have been readapted to support this new API.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/8628

Reviewed By: pdillinger

Differential Revision: D30149315

Pulled By: bjlemaire

fbshipit-source-id: 1feef5390c95db6f4480ab4434716533d3947f27
2021-08-10 18:09:03 -07:00
..
cf_options.cc Move old files to warm tier in FIFO compactions (#8310) 2021-08-09 12:51:14 -07:00
cf_options.h Added static methods for simple types to OptionTypeInfo (#8249) 2021-05-11 16:15:47 -07:00
configurable.cc Allow CreateFromString to work on complex URIs (#8547) 2021-07-16 15:05:45 -07:00
configurable_helper.h Add customizable_util.h to the public API (#8301) 2021-06-29 09:08:57 -07:00
configurable_test.cc Move RegisterOptions into the Configurable API (#8223) 2021-04-26 03:13:24 -07:00
configurable_test.h Move RegisterOptions into the Configurable API (#8223) 2021-04-26 03:13:24 -07:00
customizable.cc Make EventListener into a Customizable Class (#8473) 2021-07-27 07:47:02 -07:00
customizable_test.cc Make MergeOperator+CompactionFilter/Factory into Customizable Classes (#8481) 2021-08-06 08:27:25 -07:00
db_options.cc Memtable sampling for mempurge heuristic. (#8628) 2021-08-10 18:09:03 -07:00
db_options.h Memtable sampling for mempurge heuristic. (#8628) 2021-08-10 18:09:03 -07:00
options.cc Limit buffering for collecting samples for compression dictionary (#7970) 2021-02-19 14:09:54 -08:00
options_helper.cc Make MergeOperator+CompactionFilter/Factory into Customizable Classes (#8481) 2021-08-06 08:27:25 -07:00
options_helper.h Added static methods for simple types to OptionTypeInfo (#8249) 2021-05-11 16:15:47 -07:00
options_parser.cc Refactor: add LineFileReader and Status::MustCheck (#8026) 2021-03-09 20:12:38 -08:00
options_parser.h Bring the Configurable options together (#5753) 2020-09-14 17:01:01 -07:00
options_settable_test.cc Move old files to warm tier in FIFO compactions (#8310) 2021-08-09 12:51:14 -07:00
options_test.cc Memtable sampling for mempurge heuristic. (#8628) 2021-08-10 18:09:03 -07:00