From ab15d3356687ea8cfb1bd579a17c92877f26911a Mon Sep 17 00:00:00 2001 From: Hui Xiao Date: Tue, 24 Oct 2023 12:03:07 -0700 Subject: [PATCH] Update history, version and format testing for 8.8 (#12004) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12004 Reviewed By: cbi42 Differential Revision: D50586984 Pulled By: hx235 fbshipit-source-id: 1480a8c2757340ebf83510557104aaa0e437b3ae --- HISTORY.md | 25 +++++++++++++++++++ include/rocksdb/version.h | 2 +- tools/check_format_compatible.sh | 2 +- ...s_prefetch_on_zero_compaction_readahead.md | 1 - .../stopped_writes_wait_for_recovery.md | 1 - .../bug_fixes/auto_readahead.md | 1 - .../bug_fixes/fallback_only_unsupported.md | 1 - ...eDeltaIterator_and_Write(Un)PreparedTxn.md | 1 - .../max_successive_merges_wide_columns.md | 1 - unreleased_history/bug_fixes/options_race.md | 1 - .../sec_cache_reservation_underflow.md | 1 - .../new_features/attribute_group_multiget.md | 1 - .../new_features/fifo_drop_file_new_stats.md | 1 - .../new_features/offpeak_db_option.md | 1 - .../new_features/runtime_burst_bytes.md | 1 - .../options_files_on_open.md | 1 - ...il_if_options_file_error_default_change.md | 1 - 17 files changed, 27 insertions(+), 16 deletions(-) delete mode 100644 unreleased_history/behavior_changes/no_fs_prefetch_on_zero_compaction_readahead.md delete mode 100644 unreleased_history/behavior_changes/stopped_writes_wait_for_recovery.md delete mode 100644 unreleased_history/bug_fixes/auto_readahead.md delete mode 100644 unreleased_history/bug_fixes/fallback_only_unsupported.md delete mode 100644 unreleased_history/bug_fixes/fix_bounds_check_in_BaseDeltaIterator_and_Write(Un)PreparedTxn.md delete mode 100644 unreleased_history/bug_fixes/max_successive_merges_wide_columns.md delete mode 100644 unreleased_history/bug_fixes/options_race.md delete mode 100644 unreleased_history/bug_fixes/sec_cache_reservation_underflow.md delete mode 100644 unreleased_history/new_features/attribute_group_multiget.md delete mode 100644 unreleased_history/new_features/fifo_drop_file_new_stats.md delete mode 100644 unreleased_history/new_features/offpeak_db_option.md delete mode 100644 unreleased_history/new_features/runtime_burst_bytes.md delete mode 100644 unreleased_history/performance_improvements/options_files_on_open.md delete mode 100644 unreleased_history/public_api_changes/fail_if_options_file_error_default_change.md diff --git a/HISTORY.md b/HISTORY.md index a4f3b69da1..b32a3dd7f7 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,31 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 8.8.0 (10/23/2023) +### New Features +* Introduce AttributeGroup by adding the first AttributeGroup support API, MultiGetEntity(). Through the use of Column Families, AttributeGroup enables users to logically group wide-column entities. More APIs to support AttributeGroup will come soon, including GetEntity, PutEntity, and others. +* Added new tickers `rocksdb.fifo.{max.size|ttl}.compactions` to count FIFO compactions that drop files for different reasons +* Add an experimental offpeak duration awareness by setting `DBOptions::daily_offpeak_time_utc` in "HH:mm-HH:mm" format. This information will be used for resource optimization in the future +* Users can now change the max bytes granted in a single refill period (i.e, burst) during runtime by `SetSingleBurstBytes()` for RocksDB rate limiter + +### Public API Changes +* The default value of `DBOptions::fail_if_options_file_error` changed from `false` to `true`. Operations that set in-memory options (e.g., `DB::Open*()`, `DB::SetOptions()`, `DB::CreateColumnFamily*()`, and `DB::DropColumnFamily()`) but fail to persist the change will now return a non-OK `Status` by default. + +### Behavior Changes +* For non direct IO, eliminate the file system prefetching attempt for compaction read when `Options::compaction_readahead_size` is 0 +* During a write stop, writes now block on in-progress recovery attempts + +### Bug Fixes +* Fix a bug in auto_readahead_size where first_internal_key of index blocks wasn't copied properly resulting in corruption error when first_internal_key was used for comparison. +* Fixed a bug where compaction read under non direct IO still falls back to RocksDB internal prefetching after file system's prefetching returns non-OK status other than `Status::NotSupported()` +* Add bounds check in WBWIIteratorImpl and make BaseDeltaIterator, WriteUnpreparedTxn and WritePreparedTxn respect the upper bound and lower bound in ReadOption. See 11680. +* Fixed the handling of wide-column base values in the `max_successive_merges` logic. +* Fixed a rare race bug involving a concurrent combination of Create/DropColumnFamily and/or Set(DB)Options that could lead to inconsistency between (a) the DB's reported options state, (b) the DB options in effect, and (c) the latest persisted OPTIONS file. +* Fixed a possible underflow when computing the compressed secondary cache share of memory reservations while updating the compressed secondary to total block cache ratio. + +### Performance Improvements +* Improved the I/O efficiency of DB::Open a new DB with `create_missing_column_families=true` and many column families. + ## 8.7.0 (09/22/2023) ### New Features * Added an experimental new "automatic" variant of HyperClockCache that does not require a prior estimate of the average size of cache entries. This variant is activated when HyperClockCacheOptions::estimated\_entry\_charge = 0 and has essentially the same concurrency benefits as the existing HyperClockCache. diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index cecbb7c7b0..1390581554 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -12,7 +12,7 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. #define ROCKSDB_MAJOR 8 -#define ROCKSDB_MINOR 8 +#define ROCKSDB_MINOR 9 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh index 5866680965..e4771706fc 100755 --- a/tools/check_format_compatible.sh +++ b/tools/check_format_compatible.sh @@ -125,7 +125,7 @@ EOF # To check for DB forward compatibility with loading options (old version # reading data from new), as well as backward compatibility -declare -a db_forward_with_options_refs=("6.27.fb" "6.28.fb" "6.29.fb" "7.0.fb" "7.1.fb" "7.2.fb" "7.3.fb" "7.4.fb" "7.5.fb" "7.6.fb" "7.7.fb" "7.8.fb" "7.9.fb" "7.10.fb" "8.0.fb" "8.1.fb" "8.2.fb" "8.3.fb" "8.4.fb" "8.5.fb" "8.6.fb" "8.7.fb") +declare -a db_forward_with_options_refs=("6.27.fb" "6.28.fb" "6.29.fb" "7.0.fb" "7.1.fb" "7.2.fb" "7.3.fb" "7.4.fb" "7.5.fb" "7.6.fb" "7.7.fb" "7.8.fb" "7.9.fb" "7.10.fb" "8.0.fb" "8.1.fb" "8.2.fb" "8.3.fb" "8.4.fb" "8.5.fb" "8.6.fb" "8.7.fb" "8.8.fb") # To check for DB forward compatibility without loading options (in addition # to the "with loading options" set), as well as backward compatibility declare -a db_forward_no_options_refs=() # N/A at the moment diff --git a/unreleased_history/behavior_changes/no_fs_prefetch_on_zero_compaction_readahead.md b/unreleased_history/behavior_changes/no_fs_prefetch_on_zero_compaction_readahead.md deleted file mode 100644 index e09f693ef5..0000000000 --- a/unreleased_history/behavior_changes/no_fs_prefetch_on_zero_compaction_readahead.md +++ /dev/null @@ -1 +0,0 @@ -For non direct IO, eliminate the file system prefetching attempt for compaction read when `Options::compaction_readahead_size` is 0 diff --git a/unreleased_history/behavior_changes/stopped_writes_wait_for_recovery.md b/unreleased_history/behavior_changes/stopped_writes_wait_for_recovery.md deleted file mode 100644 index 2c44d55721..0000000000 --- a/unreleased_history/behavior_changes/stopped_writes_wait_for_recovery.md +++ /dev/null @@ -1 +0,0 @@ -* During a write stop, writes now block on in-progress recovery attempts diff --git a/unreleased_history/bug_fixes/auto_readahead.md b/unreleased_history/bug_fixes/auto_readahead.md deleted file mode 100644 index a6b009fded..0000000000 --- a/unreleased_history/bug_fixes/auto_readahead.md +++ /dev/null @@ -1 +0,0 @@ -Fix a bug in auto_readahead_size where first_internal_key of index blocks wasn't copied properly resulting in corruption error when first_internal_key was used for comparison. diff --git a/unreleased_history/bug_fixes/fallback_only_unsupported.md b/unreleased_history/bug_fixes/fallback_only_unsupported.md deleted file mode 100644 index feb02ce3b7..0000000000 --- a/unreleased_history/bug_fixes/fallback_only_unsupported.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a bug where compaction read under non direct IO still falls back to RocksDB internal prefetching after file system's prefetching returns non-OK status other than `Status::NotSupported()` diff --git a/unreleased_history/bug_fixes/fix_bounds_check_in_BaseDeltaIterator_and_Write(Un)PreparedTxn.md b/unreleased_history/bug_fixes/fix_bounds_check_in_BaseDeltaIterator_and_Write(Un)PreparedTxn.md deleted file mode 100644 index 237351f87f..0000000000 --- a/unreleased_history/bug_fixes/fix_bounds_check_in_BaseDeltaIterator_and_Write(Un)PreparedTxn.md +++ /dev/null @@ -1 +0,0 @@ -Add bounds check in WBWIIteratorImpl and make BaseDeltaIterator, WriteUnpreparedTxn and WritePreparedTxn respect the upper bound and lower bound in ReadOption. See 11680. \ No newline at end of file diff --git a/unreleased_history/bug_fixes/max_successive_merges_wide_columns.md b/unreleased_history/bug_fixes/max_successive_merges_wide_columns.md deleted file mode 100644 index d24b6cf308..0000000000 --- a/unreleased_history/bug_fixes/max_successive_merges_wide_columns.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the handling of wide-column base values in the `max_successive_merges` logic. diff --git a/unreleased_history/bug_fixes/options_race.md b/unreleased_history/bug_fixes/options_race.md deleted file mode 100644 index 42b89b8eab..0000000000 --- a/unreleased_history/bug_fixes/options_race.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed a rare race bug involving a concurrent combination of Create/DropColumnFamily and/or Set(DB)Options that could lead to inconsistency between (a) the DB's reported options state, (b) the DB options in effect, and (c) the latest persisted OPTIONS file. diff --git a/unreleased_history/bug_fixes/sec_cache_reservation_underflow.md b/unreleased_history/bug_fixes/sec_cache_reservation_underflow.md deleted file mode 100644 index 571100a3ed..0000000000 --- a/unreleased_history/bug_fixes/sec_cache_reservation_underflow.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a possible underflow when computing the compressed secondary cache share of memory reservations while updating the compressed secondary to total block cache ratio. diff --git a/unreleased_history/new_features/attribute_group_multiget.md b/unreleased_history/new_features/attribute_group_multiget.md deleted file mode 100644 index 4a7ae818fd..0000000000 --- a/unreleased_history/new_features/attribute_group_multiget.md +++ /dev/null @@ -1 +0,0 @@ -Introduce AttributeGroup by adding the first AttributeGroup support API, MultiGetEntity(). Through the use of Column Families, AttributeGroup enables users to logically group wide-column entities. More APIs to support AttributeGroup will come soon, including GetEntity, PutEntity, and others. diff --git a/unreleased_history/new_features/fifo_drop_file_new_stats.md b/unreleased_history/new_features/fifo_drop_file_new_stats.md deleted file mode 100644 index 30134b2c2a..0000000000 --- a/unreleased_history/new_features/fifo_drop_file_new_stats.md +++ /dev/null @@ -1 +0,0 @@ -Added new tickers `rocksdb.fifo.{max.size|ttl}.compactions` to count FIFO compactions that drop files for different reasons diff --git a/unreleased_history/new_features/offpeak_db_option.md b/unreleased_history/new_features/offpeak_db_option.md deleted file mode 100644 index bdc9b2a293..0000000000 --- a/unreleased_history/new_features/offpeak_db_option.md +++ /dev/null @@ -1 +0,0 @@ -Add an experimental offpeak duration awareness by setting `DBOptions::daily_offpeak_time_utc` in "HH:mm-HH:mm" format. This information will be used for resource optimization in the future diff --git a/unreleased_history/new_features/runtime_burst_bytes.md b/unreleased_history/new_features/runtime_burst_bytes.md deleted file mode 100644 index 26f1a8c872..0000000000 --- a/unreleased_history/new_features/runtime_burst_bytes.md +++ /dev/null @@ -1 +0,0 @@ -Users can now change the max bytes granted in a single refill period (i.e, burst) during runtime by `SetSingleBurstBytes()` for RocksDB rate limiter diff --git a/unreleased_history/performance_improvements/options_files_on_open.md b/unreleased_history/performance_improvements/options_files_on_open.md deleted file mode 100644 index e4ecf29496..0000000000 --- a/unreleased_history/performance_improvements/options_files_on_open.md +++ /dev/null @@ -1 +0,0 @@ -Improved the I/O efficiency of DB::Open a new DB with `create_missing_column_families=true` and many column families. diff --git a/unreleased_history/public_api_changes/fail_if_options_file_error_default_change.md b/unreleased_history/public_api_changes/fail_if_options_file_error_default_change.md deleted file mode 100644 index 44e3bb5076..0000000000 --- a/unreleased_history/public_api_changes/fail_if_options_file_error_default_change.md +++ /dev/null @@ -1 +0,0 @@ -* The default value of `DBOptions::fail_if_options_file_error` changed from `false` to `true`. Operations that set in-memory options (e.g., `DB::Open*()`, `DB::SetOptions()`, `DB::CreateColumnFamily*()`, and `DB::DropColumnFamily()`) but fail to persist the change will now return a non-OK `Status` by default.