From 9ad772e652bb3172ed8821ebacf40757bfbb7da1 Mon Sep 17 00:00:00 2001 From: Peter Dillinger Date: Fri, 25 Oct 2024 13:47:29 -0700 Subject: [PATCH] Start version 9.9.0 (#13093) Summary: Pull in HISTORY for 9.8.0, update version.h for next version, update check_format_compatible.sh Pull Request resolved: https://github.com/facebook/rocksdb/pull/13093 Test Plan: CI Reviewed By: jowlyzhang Differential Revision: D64987257 Pulled By: pdillinger fbshipit-source-id: a7cec329e3d245e63767760aa0298c08c3281695 --- HISTORY.md | 22 +++++++++++++++++++ include/rocksdb/version.h | 2 +- tools/check_format_compatible.sh | 2 +- .../overlapping_files_ingestion.md | 1 - .../behavior_changes/parallel-compaction.md | 1 - .../remote_compaction_preserve_options.md | 1 - .../trim_readahead_by_prefix.md | 1 - .../universal_level_assgiment.md | 1 - .../bug_fixes/build_db_options.md | 1 - .../compressed_secondary_cache_account.md | 1 - .../bug_fixes/memtable_stats.md | 1 - .../bug_fixes/set_options_race.md | 1 - .../skip_insertion_tiered_sec_cache.md | 1 - .../blob_db_allow_unprepared_value.md | 1 - .../new_features/ingest-fill-cache.md | 1 - .../multi_cf_allow_unprepared_value.md | 1 - .../new_features/mutable_table_options.md | 1 - .../standalone_range_deletion_file.md | 1 - 18 files changed, 24 insertions(+), 17 deletions(-) delete mode 100644 unreleased_history/behavior_changes/overlapping_files_ingestion.md delete mode 100644 unreleased_history/behavior_changes/parallel-compaction.md delete mode 100644 unreleased_history/behavior_changes/remote_compaction_preserve_options.md delete mode 100644 unreleased_history/behavior_changes/trim_readahead_by_prefix.md delete mode 100644 unreleased_history/behavior_changes/universal_level_assgiment.md delete mode 100644 unreleased_history/bug_fixes/build_db_options.md delete mode 100644 unreleased_history/bug_fixes/compressed_secondary_cache_account.md delete mode 100644 unreleased_history/bug_fixes/memtable_stats.md delete mode 100644 unreleased_history/bug_fixes/set_options_race.md delete mode 100644 unreleased_history/bug_fixes/skip_insertion_tiered_sec_cache.md delete mode 100644 unreleased_history/new_features/blob_db_allow_unprepared_value.md delete mode 100644 unreleased_history/new_features/ingest-fill-cache.md delete mode 100644 unreleased_history/new_features/multi_cf_allow_unprepared_value.md delete mode 100644 unreleased_history/new_features/mutable_table_options.md delete mode 100644 unreleased_history/new_features/standalone_range_deletion_file.md diff --git a/HISTORY.md b/HISTORY.md index d4fc843a2b..c8ee1325ce 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,28 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 9.8.0 (10/25/2024) +### New Features +* All "simple" options in `BlockBasedTableOptions` are now mutable with `DB::SetOptions()`. For now, "simple" only includes non-pointer options that are 64 bits or less. +* When using iterators with BlobDB, it is now possible to load large values on an on-demand basis, i.e. only if they are actually needed by the application. This can save I/O in use cases where the values associated with certain keys are not needed. For more details, see the new read option `allow_unprepared_value` and the iterator API `PrepareValue`. +* Add a new file ingestion option `IngestExternalFileOptions::fill_cache` to support not adding blocks from ingested files into block cache during file ingestion. +* The option `allow_unprepared_value` is now also supported for multi-column-family iterators (i.e. `CoalescingIterator` and `AttributeGroupIterator`). +* When a file with just one range deletion (standalone range deletion file) is ingested via bulk loading, it will be marked for compaction. During compaction, this type of files can be used to directly filter out some input files that are not protected by any snapshots and completely deleted by the standalone range deletion file. + +### Behavior Changes +* During file ingestion, overlapping files level assignment are done in multiple batches, so that they can potentially be assigned to lower levels other than always land on L0. +* OPTIONS file to be loaded by remote worker is now preserved so that it does not get purged by the primary host. A similar technique as how we are preserving new SST files from getting purged is used for this. min_options_file_numbers_ is tracked like pending_outputs_ is tracked. +* Trim readahead_size during scans so data blocks containing keys that are not in the same prefix as the seek key in `Seek()` are not prefetched when `ReadOptions::auto_readahead_size=true` (default value) and `ReadOptions::prefix_same_as_start = true` +* Assigning levels for external files are done in the same way for universal compaction and leveled compaction. The old behavior tends to assign files to L0 while the new behavior will assign the files to the lowest level possible. + +### Bug Fixes +* Fix a longstanding race condition in SetOptions for `block_based_table_factory` options. The fix has some subtle behavior changes because of copying and replacing the TableFactory on a change with SetOptions, including requiring an Iterator::Refresh() for an existing Iterator to use the latest options. +* Fix under counting of allocated memory in the compressed secondary cache due to looking at the compressed block size rather than the actual memory allocated, which could be larger due to internal fragmentation. +* `GetApproximateMemTableStats()` could return disastrously bad estimates 5-25% of the time. The function has been re-engineered to return much better estimates with similar CPU cost. +* Skip insertion of compressed blocks in the secondary cache if the lowest_used_cache_tier DB option is kVolatileTier. +* Fix an issue in level compaction where a small CF with small compaction debt can cause the DB to allow parallel compactions. (#13054) +* Several DB option settings could be lost through `GetOptionsFromString()`, possibly elsewhere as well. Affected options, now fixed:`background_close_inactive_wals`, `write_dbid_to_manifest`, `write_identity_file`, `prefix_seek_opt_in_only` + ## 9.7.0 (09/20/2024) ### New Features * Make Cache a customizable class that can be instantiated by the object registry. diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index 2a9b1aff3a..d414e2fd5b 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -12,7 +12,7 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. #define ROCKSDB_MAJOR 9 -#define ROCKSDB_MINOR 8 +#define ROCKSDB_MINOR 9 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh index 16309f18cf..131e32faaf 100755 --- a/tools/check_format_compatible.sh +++ b/tools/check_format_compatible.sh @@ -125,7 +125,7 @@ EOF # To check for DB forward compatibility with loading options (old version # reading data from new), as well as backward compatibility -declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb") +declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb") # To check for DB forward compatibility without loading options (in addition # to the "with loading options" set), as well as backward compatibility declare -a db_forward_no_options_refs=() # N/A at the moment diff --git a/unreleased_history/behavior_changes/overlapping_files_ingestion.md b/unreleased_history/behavior_changes/overlapping_files_ingestion.md deleted file mode 100644 index c1895f1b0f..0000000000 --- a/unreleased_history/behavior_changes/overlapping_files_ingestion.md +++ /dev/null @@ -1 +0,0 @@ -*During file ingestion, overlapping files level assignment are done in multiple batches, so that they can potentially be assigned to lower levels other than always land on L0. \ No newline at end of file diff --git a/unreleased_history/behavior_changes/parallel-compaction.md b/unreleased_history/behavior_changes/parallel-compaction.md deleted file mode 100644 index 87f5bb76e6..0000000000 --- a/unreleased_history/behavior_changes/parallel-compaction.md +++ /dev/null @@ -1 +0,0 @@ -* Fix an issue in level compaction where a small CF with small compaction debt can cause the DB to allow parallel compactions. (#13054) \ No newline at end of file diff --git a/unreleased_history/behavior_changes/remote_compaction_preserve_options.md b/unreleased_history/behavior_changes/remote_compaction_preserve_options.md deleted file mode 100644 index c5cfff9b09..0000000000 --- a/unreleased_history/behavior_changes/remote_compaction_preserve_options.md +++ /dev/null @@ -1 +0,0 @@ -OPTIONS file to be loaded by remote worker is now preserved so that it does not get purged by the primary host. A similar technique as how we are preserving new SST files from getting purged is used for this. min_options_file_numbers_ is tracked like pending_outputs_ is tracked. diff --git a/unreleased_history/behavior_changes/trim_readahead_by_prefix.md b/unreleased_history/behavior_changes/trim_readahead_by_prefix.md deleted file mode 100644 index c32439024b..0000000000 --- a/unreleased_history/behavior_changes/trim_readahead_by_prefix.md +++ /dev/null @@ -1 +0,0 @@ -* Trim readahead_size during scans so data blocks containing keys that are not in the same prefix as the seek key in `Seek()` are not prefetched when `ReadOptions::auto_readahead_size=true` (default value) and `ReadOptions::prefix_same_as_start = true` diff --git a/unreleased_history/behavior_changes/universal_level_assgiment.md b/unreleased_history/behavior_changes/universal_level_assgiment.md deleted file mode 100644 index 017f2a025c..0000000000 --- a/unreleased_history/behavior_changes/universal_level_assgiment.md +++ /dev/null @@ -1 +0,0 @@ -*Assigning levels for external files are done in the same way for universal compaction and leveled compaction. The old behavior tends to assign files to L0 while the new behavior will assign the files to the lowest level possible. \ No newline at end of file diff --git a/unreleased_history/bug_fixes/build_db_options.md b/unreleased_history/bug_fixes/build_db_options.md deleted file mode 100644 index 6994ea7193..0000000000 --- a/unreleased_history/bug_fixes/build_db_options.md +++ /dev/null @@ -1 +0,0 @@ -* Several DB option settings could be lost through `GetOptionsFromString()`, possibly elsewhere as well. Affected options, now fixed:`background_close_inactive_wals`, `write_dbid_to_manifest`, `write_identity_file`, `prefix_seek_opt_in_only` diff --git a/unreleased_history/bug_fixes/compressed_secondary_cache_account.md b/unreleased_history/bug_fixes/compressed_secondary_cache_account.md deleted file mode 100644 index 07c73b85eb..0000000000 --- a/unreleased_history/bug_fixes/compressed_secondary_cache_account.md +++ /dev/null @@ -1 +0,0 @@ -Fix under counting of allocated memory in the compressed secondary cache due to looking at the compressed block size rather than the actual memory allocated, which could be larger due to internal fragmentation. diff --git a/unreleased_history/bug_fixes/memtable_stats.md b/unreleased_history/bug_fixes/memtable_stats.md deleted file mode 100644 index 047dfbc3d0..0000000000 --- a/unreleased_history/bug_fixes/memtable_stats.md +++ /dev/null @@ -1 +0,0 @@ -* `GetApproximateMemTableStats()` could return disastrously bad estimates 5-25% of the time. The function has been re-engineered to return much better estimates with similar CPU cost. diff --git a/unreleased_history/bug_fixes/set_options_race.md b/unreleased_history/bug_fixes/set_options_race.md deleted file mode 100644 index 05b282bb8b..0000000000 --- a/unreleased_history/bug_fixes/set_options_race.md +++ /dev/null @@ -1 +0,0 @@ -Fix a longstanding race condition in SetOptions for `block_based_table_factory` options. The fix has some subtle behavior changes because of copying and replacing the TableFactory on a change with SetOptions, including requiring an Iterator::Refresh() for an existing Iterator to use the latest options. diff --git a/unreleased_history/bug_fixes/skip_insertion_tiered_sec_cache.md b/unreleased_history/bug_fixes/skip_insertion_tiered_sec_cache.md deleted file mode 100644 index 7dcbe099fd..0000000000 --- a/unreleased_history/bug_fixes/skip_insertion_tiered_sec_cache.md +++ /dev/null @@ -1 +0,0 @@ -Skip insertion of compressed blocks in the secondary cache if the lowest_used_cache_tier DB option is kVolatileTier. diff --git a/unreleased_history/new_features/blob_db_allow_unprepared_value.md b/unreleased_history/new_features/blob_db_allow_unprepared_value.md deleted file mode 100644 index fb655a134f..0000000000 --- a/unreleased_history/new_features/blob_db_allow_unprepared_value.md +++ /dev/null @@ -1 +0,0 @@ -When using iterators with BlobDB, it is now possible to load large values on an on-demand basis, i.e. only if they are actually needed by the application. This can save I/O in use cases where the values associated with certain keys are not needed. For more details, see the new read option `allow_unprepared_value` and the iterator API `PrepareValue`. diff --git a/unreleased_history/new_features/ingest-fill-cache.md b/unreleased_history/new_features/ingest-fill-cache.md deleted file mode 100644 index f20d6c09d2..0000000000 --- a/unreleased_history/new_features/ingest-fill-cache.md +++ /dev/null @@ -1 +0,0 @@ -* Add a new file ingestion option `IngestExternalFileOptions::fill_cache` to support not adding blocks from ingested files into block cache during file ingestion. \ No newline at end of file diff --git a/unreleased_history/new_features/multi_cf_allow_unprepared_value.md b/unreleased_history/new_features/multi_cf_allow_unprepared_value.md deleted file mode 100644 index bfb0049878..0000000000 --- a/unreleased_history/new_features/multi_cf_allow_unprepared_value.md +++ /dev/null @@ -1 +0,0 @@ -The option `allow_unprepared_value` is now also supported for multi-column-family iterators (i.e. `CoalescingIterator` and `AttributeGroupIterator`). diff --git a/unreleased_history/new_features/mutable_table_options.md b/unreleased_history/new_features/mutable_table_options.md deleted file mode 100644 index 4d2b1fdd8d..0000000000 --- a/unreleased_history/new_features/mutable_table_options.md +++ /dev/null @@ -1 +0,0 @@ -* All "simple" options in `BlockBasedTableOptions` are now mutable with `DB::SetOptions()`. For now, "simple" only includes non-pointer options that are 64 bits or less. diff --git a/unreleased_history/new_features/standalone_range_deletion_file.md b/unreleased_history/new_features/standalone_range_deletion_file.md deleted file mode 100644 index 963062cf3c..0000000000 --- a/unreleased_history/new_features/standalone_range_deletion_file.md +++ /dev/null @@ -1 +0,0 @@ -*When a file with just one range deletion (standalone range deletion file) is ingested via bulk loading, it will be marked for compaction. During compaction, this type of files can be used to directly filter out some input files that are not protected by any snapshots and completely deleted by the standalone range deletion file. \ No newline at end of file