dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Start version 9.9.0 (#13093) 

Summary:
Pull in HISTORY for 9.8.0, update version.h for next version, update check_format_compatible.sh

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13093

Test Plan: CI

Reviewed By: jowlyzhang

Differential Revision: D64987257

Pulled By: pdillinger

fbshipit-source-id: a7cec329e3d245e63767760aa0298c08c3281695
This commit is contained in:
Peter Dillinger 2024-10-25 13:47:29 -07:00 committed by Facebook GitHub Bot
parent 57a8e69d4e
commit 9ad772e652
18 changed files with 24 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
HISTORY.md
View File

@ -1,6 +1,28 @@
# Rocksdb Change Log
> NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
## 9.8.0 (10/25/2024)
### New Features
* All "simple" options in `BlockBasedTableOptions` are now mutable with `DB::SetOptions()`. For now, "simple" only includes non-pointer options that are 64 bits or less.
* When using iterators with BlobDB, it is now possible to load large values on an on-demand basis, i.e. only if they are actually needed by the application. This can save I/O in use cases where the values associated with certain keys are not needed. For more details, see the new read option `allow_unprepared_value` and the iterator API `PrepareValue`.
* Add a new file ingestion option `IngestExternalFileOptions::fill_cache` to support not adding blocks from ingested files into block cache during file ingestion.
* The option `allow_unprepared_value` is now also supported for multi-column-family iterators (i.e. `CoalescingIterator` and `AttributeGroupIterator`).
* When a file with just one range deletion (standalone range deletion file) is ingested via bulk loading, it will be marked for compaction. During compaction, this type of files can be used to directly filter out some input files that are not protected by any snapshots and completely deleted by the standalone range deletion file.
### Behavior Changes
* During file ingestion, overlapping files level assignment are done in multiple batches, so that they can potentially be assigned to lower levels other than always land on L0.
* OPTIONS file to be loaded by remote worker is now preserved so that it does not get purged by the primary host. A similar technique as how we are preserving new SST files from getting purged is used for this. min_options_file_numbers_ is tracked like pending_outputs_ is tracked.
* Trim readahead_size during scans so data blocks containing keys that are not in the same prefix as the seek key in `Seek()` are not prefetched when `ReadOptions::auto_readahead_size=true` (default value) and `ReadOptions::prefix_same_as_start = true`
* Assigning levels for external files are done in the same way for universal compaction and leveled compaction. The old behavior tends to assign files to L0 while the new behavior will assign the files to the lowest level possible.
### Bug Fixes
* Fix a longstanding race condition in SetOptions for `block_based_table_factory` options. The fix has some subtle behavior changes because of copying and replacing the TableFactory on a change with SetOptions, including requiring an Iterator::Refresh() for an existing Iterator to use the latest options.
* Fix under counting of allocated memory in the compressed secondary cache due to looking at the compressed block size rather than the actual memory allocated, which could be larger due to internal fragmentation.
* `GetApproximateMemTableStats()` could return disastrously bad estimates 5-25% of the time. The function has been re-engineered to return much better estimates with similar CPU cost.
* Skip insertion of compressed blocks in the secondary cache if the lowest_used_cache_tier DB option is kVolatileTier.
* Fix an issue in level compaction where a small CF with small compaction debt can cause the DB to allow parallel compactions. (#13054)
* Several DB option settings could be lost through `GetOptionsFromString()`, possibly elsewhere as well. Affected options, now fixed:`background_close_inactive_wals`, `write_dbid_to_manifest`, `write_identity_file`, `prefix_seek_opt_in_only`
## 9.7.0 (09/20/2024)
### New Features
* Make Cache a customizable class that can be instantiated by the object registry.

2
include/rocksdb/version.h
View File

@ -12,7 +12,7 @@
// NOTE: in 'main' development branch, this should be the *next*
// minor or major version number planned for release.
#define ROCKSDB_MAJOR 9
#define ROCKSDB_MINOR 8
#define ROCKSDB_MINOR 9
#define ROCKSDB_PATCH 0
// Do not use these. We made the mistake of declaring macros starting with

2
tools/check_format_compatible.sh
View File

@ -125,7 +125,7 @@ EOF
# To check for DB forward compatibility with loading options (old version
# reading data from new), as well as backward compatibility
declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb")
declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb")
# To check for DB forward compatibility without loading options (in addition
# to the "with loading options" set), as well as backward compatibility
declare -a db_forward_no_options_refs=() # N/A at the moment

1
unreleased_history/behavior_changes/overlapping_files_ingestion.md
View File

@ -1 +0,0 @@
*During file ingestion, overlapping files level assignment are done in multiple batches, so that they can potentially be assigned to lower levels other than always land on L0.

1
unreleased_history/behavior_changes/parallel-compaction.md
View File

@ -1 +0,0 @@
* Fix an issue in level compaction where a small CF with small compaction debt can cause the DB to allow parallel compactions. (#13054)

1
unreleased_history/behavior_changes/remote_compaction_preserve_options.md
View File

@ -1 +0,0 @@
OPTIONS file to be loaded by remote worker is now preserved so that it does not get purged by the primary host. A similar technique as how we are preserving new SST files from getting purged is used for this. min_options_file_numbers_ is tracked like pending_outputs_ is tracked.

1
unreleased_history/behavior_changes/trim_readahead_by_prefix.md
View File

@ -1 +0,0 @@
* Trim readahead_size during scans so data blocks containing keys that are not in the same prefix as the seek key in `Seek()` are not prefetched when `ReadOptions::auto_readahead_size=true` (default value) and `ReadOptions::prefix_same_as_start = true`

1
unreleased_history/behavior_changes/universal_level_assgiment.md
View File

@ -1 +0,0 @@
*Assigning levels for external files are done in the same way for universal compaction and leveled compaction. The old behavior tends to assign files to L0 while the new behavior will assign the files to the lowest level possible.

1
unreleased_history/bug_fixes/build_db_options.md
View File

@ -1 +0,0 @@
* Several DB option settings could be lost through `GetOptionsFromString()`, possibly elsewhere as well. Affected options, now fixed:`background_close_inactive_wals`, `write_dbid_to_manifest`, `write_identity_file`, `prefix_seek_opt_in_only`

1
unreleased_history/bug_fixes/compressed_secondary_cache_account.md
View File

@ -1 +0,0 @@
Fix under counting of allocated memory in the compressed secondary cache due to looking at the compressed block size rather than the actual memory allocated, which could be larger due to internal fragmentation.

1
unreleased_history/bug_fixes/memtable_stats.md
View File

@ -1 +0,0 @@
* `GetApproximateMemTableStats()` could return disastrously bad estimates 5-25% of the time. The function has been re-engineered to return much better estimates with similar CPU cost.

1
unreleased_history/bug_fixes/set_options_race.md
View File

@ -1 +0,0 @@
Fix a longstanding race condition in SetOptions for `block_based_table_factory` options. The fix has some subtle behavior changes because of copying and replacing the TableFactory on a change with SetOptions, including requiring an Iterator::Refresh() for an existing Iterator to use the latest options.

1
unreleased_history/bug_fixes/skip_insertion_tiered_sec_cache.md
View File

@ -1 +0,0 @@
Skip insertion of compressed blocks in the secondary cache if the lowest_used_cache_tier DB option is kVolatileTier.

1
unreleased_history/new_features/blob_db_allow_unprepared_value.md
View File

@ -1 +0,0 @@
When using iterators with BlobDB, it is now possible to load large values on an on-demand basis, i.e. only if they are actually needed by the application. This can save I/O in use cases where the values associated with certain keys are not needed. For more details, see the new read option `allow_unprepared_value` and the iterator API `PrepareValue`.

1
unreleased_history/new_features/ingest-fill-cache.md
View File

@ -1 +0,0 @@
* Add a new file ingestion option `IngestExternalFileOptions::fill_cache` to support not adding blocks from ingested files into block cache during file ingestion.

1
unreleased_history/new_features/multi_cf_allow_unprepared_value.md
View File

@ -1 +0,0 @@
The option `allow_unprepared_value` is now also supported for multi-column-family iterators (i.e. `CoalescingIterator` and `AttributeGroupIterator`).

1
unreleased_history/new_features/mutable_table_options.md
View File

@ -1 +0,0 @@
* All "simple" options in `BlockBasedTableOptions` are now mutable with `DB::SetOptions()`. For now, "simple" only includes non-pointer options that are 64 bits or less.

1
unreleased_history/new_features/standalone_range_deletion_file.md
View File

@ -1 +0,0 @@
*When a file with just one range deletion (standalone range deletion file) is ingested via bulk loading, it will be marked for compaction. During compaction, this type of files can be used to directly filter out some input files that are not protected by any snapshots and completely deleted by the standalone range deletion file.