From 81765866c4bc055122f2408c2604dfcfb27e915e Mon Sep 17 00:00:00 2001 From: Levi Tamasi Date: Fri, 15 Dec 2023 14:44:23 -0800 Subject: [PATCH] Update HISTORY/version/format compatibility script for the 8.10 release (#12154) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/12154 Reviewed By: jaykorean, akankshamahajan15 Differential Revision: D52216271 Pulled By: ltamasi fbshipit-source-id: 13bab72802eeec8f6e3544be9ebcd7f725a64d2e --- HISTORY.md | 38 +++++++++++++++++++ include/rocksdb/version.h | 2 +- tools/check_format_compatible.sh | 2 +- .../behavior_changes/auto_readahead_size.md | 1 - .../behavior_changes/debt_based_speedup.md | 1 - .../behavior_changes/eviction_effort_cap.md | 1 - .../bug_fixes/auto_readahead_size.md | 1 - .../bug_fixes/avoid_destroying_timer.md | 1 - ...ix_bytes_written_ticker_double_counting.md | 1 - .../bug_fixes/fix_stall_counter.md | 1 - ...tiered_cache_low_latency_false_negative.md | 1 - .../async_support_tune_readahead.md | 1 - .../new_features/wbwi_wide_columns.md | 1 - .../java_api_consistency.md | 16 -------- .../public_api_changes/null_collector.md | 1 - 15 files changed, 40 insertions(+), 29 deletions(-) delete mode 100644 unreleased_history/behavior_changes/auto_readahead_size.md delete mode 100644 unreleased_history/behavior_changes/debt_based_speedup.md delete mode 100644 unreleased_history/behavior_changes/eviction_effort_cap.md delete mode 100644 unreleased_history/bug_fixes/auto_readahead_size.md delete mode 100644 unreleased_history/bug_fixes/avoid_destroying_timer.md delete mode 100644 unreleased_history/bug_fixes/fix_bytes_written_ticker_double_counting.md delete mode 100644 unreleased_history/bug_fixes/fix_stall_counter.md delete mode 100644 unreleased_history/bug_fixes/tiered_cache_low_latency_false_negative.md delete mode 100644 unreleased_history/new_features/async_support_tune_readahead.md delete mode 100644 unreleased_history/new_features/wbwi_wide_columns.md delete mode 100644 unreleased_history/performance_improvements/java_api_consistency.md delete mode 100644 unreleased_history/public_api_changes/null_collector.md diff --git a/HISTORY.md b/HISTORY.md index 47adf0040a..0d7c284b2f 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,44 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 8.10.0 (12/15/2023) +### New Features +* Provide support for async_io to trim readahead_size by doing block cache lookup +* Added initial wide-column support in `WriteBatchWithIndex`. This includes the `PutEntity` API and support for wide columns in the existing read APIs (`GetFromBatch`, `GetFromBatchAndDB`, `MultiGetFromBatchAndDB`, and `BaseDeltaIterator`). + +### Public API Changes +* Custom implementations of `TablePropertiesCollectorFactory` may now return a `nullptr` collector to decline processing a file, reducing callback overheads in such cases. + +### Behavior Changes +* Make ReadOptions.auto_readahead_size default true which does prefetching optimizations for forward scans if iterate_upper_bound and block_cache is also specified. +* Compactions can be scheduled in parallel in an additional scenario: high compaction debt relative to the data size +* HyperClockCache now has built-in protection against excessive CPU consumption under the extreme stress condition of no (or very few) evictable cache entries, which can slightly increase memory usage such conditions. New option `HyperClockCacheOptions::eviction_effort_cap` controls the space-time trade-off of the response. The default should be generally well-balanced, with no measurable affect on normal operation. + +### Bug Fixes +* Fix a corner case with auto_readahead_size where Prev Operation returns NOT SUPPORTED error when scans direction is changed from forward to backward. +* Avoid destroying the periodic task scheduler's default timer in order to prevent static destruction order issues. +* Fix double counting of BYTES_WRITTEN ticker when doing writes with transactions. +* Fix a WRITE_STALL counter that was reporting wrong value in few cases. +* A lookup by MultiGet in a TieredCache that goes to the local flash cache and finishes with very low latency, i.e before the subsequent call to WaitAll, is ignored, resulting in a false negative and a memory leak. + +### Performance Improvements +* Java API extensions to improve consistency and completeness of APIs +1 Extended `RocksDB.get([ColumnFamilyHandle columnFamilyHandle,] ReadOptions opt, ByteBuffer key, ByteBuffer value)` which now accepts indirect buffer parameters as well as direct buffer parameters +2 Extended `RocksDB.put( [ColumnFamilyHandle columnFamilyHandle,] WriteOptions writeOpts, final ByteBuffer key, final ByteBuffer value)` which now accepts indirect buffer parameters as well as direct buffer parameters +3 Added `RocksDB.merge([ColumnFamilyHandle columnFamilyHandle,] WriteOptions writeOptions, ByteBuffer key, ByteBuffer value)` methods with the same parameter options as `put(...)` - direct and indirect buffers are supported +4 Added `RocksIterator.key( byte[] key [, int offset, int len])` methods which retrieve the iterator key into the supplied buffer +5 Added `RocksIterator.value( byte[] value [, int offset, int len])` methods which retrieve the iterator value into the supplied buffer +6 Deprecated `get(final ColumnFamilyHandle columnFamilyHandle, final ReadOptions readOptions, byte[])` in favour of `get(final ReadOptions readOptions, final ColumnFamilyHandle columnFamilyHandle, byte[])` which has consistent parameter ordering with other methods in the same class +7 Added `Transaction.get( ReadOptions opt, [ColumnFamilyHandle columnFamilyHandle, ] byte[] key, byte[] value)` methods which retrieve the requested value into the supplied buffer +8 Added `Transaction.get( ReadOptions opt, [ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value)` methods which retrieve the requested value into the supplied buffer +9 Added `Transaction.getForUpdate( ReadOptions readOptions, [ColumnFamilyHandle columnFamilyHandle, ] byte[] key, byte[] value, boolean exclusive [, boolean doValidate])` methods which retrieve the requested value into the supplied buffer +10 Added `Transaction.getForUpdate( ReadOptions readOptions, [ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value, boolean exclusive [, boolean doValidate])` methods which retrieve the requested value into the supplied buffer +11 Added `Transaction.getIterator()` method as a convenience which defaults the `ReadOptions` value supplied to existing `Transaction.iterator()` methods. This mirrors the existing `RocksDB.iterator()` method. +12 Added `Transaction.put([ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value [, boolean assumeTracked])` methods which supply the key, and the value to be written in a `ByteBuffer` parameter +13 Added `Transaction.merge([ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value [, boolean assumeTracked])` methods which supply the key, and the value to be written/merged in a `ByteBuffer` parameter +14 Added `Transaction.mergeUntracked([ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value)` methods which supply the key, and the value to be written/merged in a `ByteBuffer` parameter + + ## 8.9.0 (11/17/2023) ### New Features * Add GetEntity() and PutEntity() API implementation for Attribute Group support. Through the use of Column Families, AttributeGroup enables users to logically group wide-column entities. diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index 530d4fee76..4ff6ab51fc 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -12,7 +12,7 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. #define ROCKSDB_MAJOR 8 -#define ROCKSDB_MINOR 10 +#define ROCKSDB_MINOR 11 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh index 93b51a9b9e..21c17feae0 100755 --- a/tools/check_format_compatible.sh +++ b/tools/check_format_compatible.sh @@ -125,7 +125,7 @@ EOF # To check for DB forward compatibility with loading options (old version # reading data from new), as well as backward compatibility -declare -a db_forward_with_options_refs=("6.27.fb" "6.28.fb" "6.29.fb" "7.0.fb" "7.1.fb" "7.2.fb" "7.3.fb" "7.4.fb" "7.5.fb" "7.6.fb" "7.7.fb" "7.8.fb" "7.9.fb" "7.10.fb" "8.0.fb" "8.1.fb" "8.2.fb" "8.3.fb" "8.4.fb" "8.5.fb" "8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb") +declare -a db_forward_with_options_refs=("6.27.fb" "6.28.fb" "6.29.fb" "7.0.fb" "7.1.fb" "7.2.fb" "7.3.fb" "7.4.fb" "7.5.fb" "7.6.fb" "7.7.fb" "7.8.fb" "7.9.fb" "7.10.fb" "8.0.fb" "8.1.fb" "8.2.fb" "8.3.fb" "8.4.fb" "8.5.fb" "8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb") # To check for DB forward compatibility without loading options (in addition # to the "with loading options" set), as well as backward compatibility declare -a db_forward_no_options_refs=() # N/A at the moment diff --git a/unreleased_history/behavior_changes/auto_readahead_size.md b/unreleased_history/behavior_changes/auto_readahead_size.md deleted file mode 100644 index b1c98dd86f..0000000000 --- a/unreleased_history/behavior_changes/auto_readahead_size.md +++ /dev/null @@ -1 +0,0 @@ -Make ReadOptions.auto_readahead_size default true which does prefetching optimizations for forward scans if iterate_upper_bound and block_cache is also specified. diff --git a/unreleased_history/behavior_changes/debt_based_speedup.md b/unreleased_history/behavior_changes/debt_based_speedup.md deleted file mode 100644 index 8db8b6688e..0000000000 --- a/unreleased_history/behavior_changes/debt_based_speedup.md +++ /dev/null @@ -1 +0,0 @@ -Compactions can be scheduled in parallel in an additional scenario: high compaction debt relative to the data size diff --git a/unreleased_history/behavior_changes/eviction_effort_cap.md b/unreleased_history/behavior_changes/eviction_effort_cap.md deleted file mode 100644 index 7772141e10..0000000000 --- a/unreleased_history/behavior_changes/eviction_effort_cap.md +++ /dev/null @@ -1 +0,0 @@ -HyperClockCache now has built-in protection against excessive CPU consumption under the extreme stress condition of no (or very few) evictable cache entries, which can slightly increase memory usage such conditions. New option `HyperClockCacheOptions::eviction_effort_cap` controls the space-time trade-off of the response. The default should be generally well-balanced, with no measurable affect on normal operation. diff --git a/unreleased_history/bug_fixes/auto_readahead_size.md b/unreleased_history/bug_fixes/auto_readahead_size.md deleted file mode 100644 index e829737711..0000000000 --- a/unreleased_history/bug_fixes/auto_readahead_size.md +++ /dev/null @@ -1 +0,0 @@ -Fix a corner case with auto_readahead_size where Prev Operation returns NOT SUPPORTED error when scans direction is changed from forward to backward. diff --git a/unreleased_history/bug_fixes/avoid_destroying_timer.md b/unreleased_history/bug_fixes/avoid_destroying_timer.md deleted file mode 100644 index 2a70b6b31a..0000000000 --- a/unreleased_history/bug_fixes/avoid_destroying_timer.md +++ /dev/null @@ -1 +0,0 @@ -Avoid destroying the periodic task scheduler's default timer in order to prevent static destruction order issues. diff --git a/unreleased_history/bug_fixes/fix_bytes_written_ticker_double_counting.md b/unreleased_history/bug_fixes/fix_bytes_written_ticker_double_counting.md deleted file mode 100644 index fe68598745..0000000000 --- a/unreleased_history/bug_fixes/fix_bytes_written_ticker_double_counting.md +++ /dev/null @@ -1 +0,0 @@ -Fix double counting of BYTES_WRITTEN ticker when doing writes with transactions. diff --git a/unreleased_history/bug_fixes/fix_stall_counter.md b/unreleased_history/bug_fixes/fix_stall_counter.md deleted file mode 100644 index be9e5122f8..0000000000 --- a/unreleased_history/bug_fixes/fix_stall_counter.md +++ /dev/null @@ -1 +0,0 @@ -Fix a WRITE_STALL counter that was reporting wrong value in few cases. diff --git a/unreleased_history/bug_fixes/tiered_cache_low_latency_false_negative.md b/unreleased_history/bug_fixes/tiered_cache_low_latency_false_negative.md deleted file mode 100644 index c271724adc..0000000000 --- a/unreleased_history/bug_fixes/tiered_cache_low_latency_false_negative.md +++ /dev/null @@ -1 +0,0 @@ -A lookup by MultiGet in a TieredCache that goes to the local flash cache and finishes with very low latency, i.e before the subsequent call to WaitAll, is ignored, resulting in a false negative and a memory leak. diff --git a/unreleased_history/new_features/async_support_tune_readahead.md b/unreleased_history/new_features/async_support_tune_readahead.md deleted file mode 100644 index 8794a493a5..0000000000 --- a/unreleased_history/new_features/async_support_tune_readahead.md +++ /dev/null @@ -1 +0,0 @@ -Provide support for async_io to trim readahead_size by doing block cache lookup diff --git a/unreleased_history/new_features/wbwi_wide_columns.md b/unreleased_history/new_features/wbwi_wide_columns.md deleted file mode 100644 index 908279e1cb..0000000000 --- a/unreleased_history/new_features/wbwi_wide_columns.md +++ /dev/null @@ -1 +0,0 @@ -Added initial wide-column support in `WriteBatchWithIndex`. This includes the `PutEntity` API and support for wide columns in the existing read APIs (`GetFromBatch`, `GetFromBatchAndDB`, `MultiGetFromBatchAndDB`, and `BaseDeltaIterator`). diff --git a/unreleased_history/performance_improvements/java_api_consistency.md b/unreleased_history/performance_improvements/java_api_consistency.md deleted file mode 100644 index 7e29b629c9..0000000000 --- a/unreleased_history/performance_improvements/java_api_consistency.md +++ /dev/null @@ -1,16 +0,0 @@ -* Java API extensions to improve consistency and completeness of APIs - 1 Extended `RocksDB.get([ColumnFamilyHandle columnFamilyHandle,] ReadOptions opt, ByteBuffer key, ByteBuffer value)` which now accepts indirect buffer parameters as well as direct buffer parameters - 2 Extended `RocksDB.put( [ColumnFamilyHandle columnFamilyHandle,] WriteOptions writeOpts, final ByteBuffer key, final ByteBuffer value)` which now accepts indirect buffer parameters as well as direct buffer parameters - 3 Added `RocksDB.merge([ColumnFamilyHandle columnFamilyHandle,] WriteOptions writeOptions, ByteBuffer key, ByteBuffer value)` methods with the same parameter options as `put(...)` - direct and indirect buffers are supported - 4 Added `RocksIterator.key( byte[] key [, int offset, int len])` methods which retrieve the iterator key into the supplied buffer - 5 Added `RocksIterator.value( byte[] value [, int offset, int len])` methods which retrieve the iterator value into the supplied buffer - 6 Deprecated `get(final ColumnFamilyHandle columnFamilyHandle, final ReadOptions readOptions, byte[])` in favour of `get(final ReadOptions readOptions, final ColumnFamilyHandle columnFamilyHandle, byte[])` which has consistent parameter ordering with other methods in the same class - 7 Added `Transaction.get( ReadOptions opt, [ColumnFamilyHandle columnFamilyHandle, ] byte[] key, byte[] value)` methods which retrieve the requested value into the supplied buffer - 8 Added `Transaction.get( ReadOptions opt, [ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value)` methods which retrieve the requested value into the supplied buffer - 9 Added `Transaction.getForUpdate( ReadOptions readOptions, [ColumnFamilyHandle columnFamilyHandle, ] byte[] key, byte[] value, boolean exclusive [, boolean doValidate])` methods which retrieve the requested value into the supplied buffer - 10 Added `Transaction.getForUpdate( ReadOptions readOptions, [ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value, boolean exclusive [, boolean doValidate])` methods which retrieve the requested value into the supplied buffer - 11 Added `Transaction.getIterator()` method as a convenience which defaults the `ReadOptions` value supplied to existing `Transaction.iterator()` methods. This mirrors the existing `RocksDB.iterator()` method. - 12 Added `Transaction.put([ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value [, boolean assumeTracked])` methods which supply the key, and the value to be written in a `ByteBuffer` parameter - 13 Added `Transaction.merge([ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value [, boolean assumeTracked])` methods which supply the key, and the value to be written/merged in a `ByteBuffer` parameter - 14 Added `Transaction.mergeUntracked([ColumnFamilyHandle columnFamilyHandle, ] ByteBuffer key, ByteBuffer value)` methods which supply the key, and the value to be written/merged in a `ByteBuffer` parameter - diff --git a/unreleased_history/public_api_changes/null_collector.md b/unreleased_history/public_api_changes/null_collector.md deleted file mode 100644 index 2af478c832..0000000000 --- a/unreleased_history/public_api_changes/null_collector.md +++ /dev/null @@ -1 +0,0 @@ -Custom implementations of `TablePropertiesCollectorFactory` may now return a `nullptr` collector to decline processing a file, reducing callback overheads in such cases.