diff --git a/HISTORY.md b/HISTORY.md index 058f326c61..feffb34b4c 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,43 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 9.2.0 (05/01/2024) +### New Features +* Added two options `deadline` and `max_size_bytes` for CacheDumper to exit early +* Added a new API `GetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for wide-column point lookups with read-your-own-writes consistency. Similarly to `GetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. +* [Experimental] Introduce two new cross-column-family iterators - CoalescingIterator and AttributeGroupIterator. The CoalescingIterator enables users to iterate over multiple column families and access their values and columns. During this iteration, if the same key exists in more than one column family, the keys in the later column family will overshadow the previous ones. The AttributeGroupIterator allows users to gather wide columns per Column Family and create attribute groups while iterating over keys across all CFs. +* Added a new API `MultiGetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for batched wide-column point lookups with read-your-own-writes consistency. Similarly to `MultiGetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. +* *Adds a `SstFileReader::NewTableIterator` API to support programmatically read a SST file as a raw table file. +* Add an option to `WaitForCompactOptions` - `wait_for_purge` to make `WaitForCompact()` API wait for background purge to complete + +### Public API Changes +* DeleteRange() will return NotSupported() if row_cache is configured since they don't work together in some cases. +* Deprecated `CompactionOptions::compression` since `CompactionOptions`'s API for configuring compression was incomplete, unsafe, and likely unnecessary +* Using `OptionChangeMigration()` to migrate from non-FIFO to FIFO compaction +with `Options::compaction_options_fifo.max_table_files_size` > 0 can cause +the whole DB to be dropped right after migration if the migrated data is larger than +`max_table_files_size` + +### Behavior Changes +* Enabling `BlockBasedTableOptions::block_align` is now incompatible (i.e., APIs will return `Status::InvalidArgument`) with more ways of enabling compression: `CompactionOptions::compression`, `ColumnFamilyOptions::compression_per_level`, and `ColumnFamilyOptions::bottommost_compression`. +* Changed the default value of `CompactionOptions::compression` to `kDisableCompressionOption`, which means the compression type is determined by the `ColumnFamilyOptions`. +* `BlockBasedTableOptions::optimize_filters_for_memory` is now set to true by default. When `partition_filters=false`, this could lead to somewhat increased average RSS memory usage by the block cache, but this "extra" usage is within the allowed memory budget and should make memory usage more consistent (by minimizing internal fragmentation for more kinds of blocks). +* Dump all keys for cache dumper impl if `SetDumpFilter()` is not called +* `CompactRange()` with `CompactRangeOptions::change_level = true` and `CompactRangeOptions::target_level = 0` that ends up moving more than 1 file from non-L0 to L0 will return `Status::Aborted()`. +* On distributed file systems that support file system level checksum verification and reconstruction reads, RocksDB will now retry a file read if the initial read fails RocksDB block level or record level checksum verification. This applies to MANIFEST file reads when the DB is opened, and to SST file reads at all times. + +### Bug Fixes +* Fix a bug causing `VerifyFileChecksums()` to return false-positive corruption under `BlockBasedTableOptions::block_align=true` +* Provide consistent view of the database across the column families for `NewIterators()` API. +* Fixed feature interaction bug for `DeleteRange()` together with `ColumnFamilyOptions::memtable_insert_with_hint_prefix_extractor`. The impact of this bug would likely be corruption or crashing. +* Fixed hang in `DisableManualCompactions()` where compactions waiting to be scheduled due to conflicts would not be canceled promptly +* Fixed a regression when `ColumnFamilyOptions::max_successive_merges > 0` where the CPU overhead for deciding whether to merge could have increased unless the user had set the option `ColumnFamilyOptions::strict_max_successive_merges` +* Fixed a bug in `MultiGet()` and `MultiGetEntity()` together with blob files (`ColumnFamilyOptions::enable_blob_files == true`). An error looking up one of the keys could cause the results to be wrong for other keys for which the statuses were `Status::OK`. +* Fixed a bug where wrong padded bytes are used to generate file checksum and `DataVerificationInfo::checksum` upon file creation +* Correctly implemented the move semantics of `PinnableWideColumns`. +* Fixed a bug when the recycle_log_file_num in DBOptions is changed from 0 to non-zero when a DB is reopened. On a subsequent reopen, if a log file created when recycle_log_file_num==0 was reused previously, is alive and is empty, we could end up inserting stale WAL records into the memtable. +* *Fix a bug where obsolete files' deletion during DB::Open are not rate limited with `SstFilemManager`'s slow deletion feature even if it's configured. + ## 9.1.0 (03/22/2024) ### New Features * Added an option, `GetMergeOperandsOptions::continue_cb`, to give users the ability to end `GetMergeOperands()`'s lookup process before all merge operands were found. diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index 1842690b26..24506703fb 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -12,7 +12,7 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. #define ROCKSDB_MAJOR 9 -#define ROCKSDB_MINOR 2 +#define ROCKSDB_MINOR 3 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh index 9324a27e41..6da1000f4a 100755 --- a/tools/check_format_compatible.sh +++ b/tools/check_format_compatible.sh @@ -125,7 +125,7 @@ EOF # To check for DB forward compatibility with loading options (old version # reading data from new), as well as backward compatibility -declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb") +declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb") # To check for DB forward compatibility without loading options (in addition # to the "with loading options" set), as well as backward compatibility declare -a db_forward_no_options_refs=() # N/A at the moment diff --git a/unreleased_history/behavior_changes/block_align_compression_incompat.md b/unreleased_history/behavior_changes/block_align_compression_incompat.md deleted file mode 100644 index 7068a42679..0000000000 --- a/unreleased_history/behavior_changes/block_align_compression_incompat.md +++ /dev/null @@ -1 +0,0 @@ -* Enabling `BlockBasedTableOptions::block_align` is now incompatible (i.e., APIs will return `Status::InvalidArgument`) with more ways of enabling compression: `CompactionOptions::compression`, `ColumnFamilyOptions::compression_per_level`, and `ColumnFamilyOptions::bottommost_compression`. diff --git a/unreleased_history/behavior_changes/default_compaction_options_compression.md b/unreleased_history/behavior_changes/default_compaction_options_compression.md deleted file mode 100644 index 2e4e87ff18..0000000000 --- a/unreleased_history/behavior_changes/default_compaction_options_compression.md +++ /dev/null @@ -1 +0,0 @@ -* Changed the default value of `CompactionOptions::compression` to `kDisableCompressionOption`, which means the compression type is determined by the `ColumnFamilyOptions`. diff --git a/unreleased_history/behavior_changes/default_optimize_filters_for_memory.md b/unreleased_history/behavior_changes/default_optimize_filters_for_memory.md deleted file mode 100644 index 0cd00120ae..0000000000 --- a/unreleased_history/behavior_changes/default_optimize_filters_for_memory.md +++ /dev/null @@ -1 +0,0 @@ -`BlockBasedTableOptions::optimize_filters_for_memory` is now set to true by default. When `partition_filters=false`, this could lead to somewhat increased average RSS memory usage by the block cache, but this "extra" usage is within the allowed memory budget and should make memory usage more consistent (by minimizing internal fragmentation for more kinds of blocks). diff --git a/unreleased_history/behavior_changes/dump_all_keys.md b/unreleased_history/behavior_changes/dump_all_keys.md deleted file mode 100644 index 913cc04423..0000000000 --- a/unreleased_history/behavior_changes/dump_all_keys.md +++ /dev/null @@ -1 +0,0 @@ -Dump all keys for cache dumper impl if `SetDumpFilter()` is not called diff --git a/unreleased_history/behavior_changes/refit_to_l0.md b/unreleased_history/behavior_changes/refit_to_l0.md deleted file mode 100644 index 1c9813cace..0000000000 --- a/unreleased_history/behavior_changes/refit_to_l0.md +++ /dev/null @@ -1 +0,0 @@ -`CompactRange()` with `CompactRangeOptions::change_level = true` and `CompactRangeOptions::target_level = 0` that ends up moving more than 1 file from non-L0 to L0 will return `Status::Aborted()`. diff --git a/unreleased_history/behavior_changes/retry_on_corruption.md b/unreleased_history/behavior_changes/retry_on_corruption.md deleted file mode 100644 index fe5a385b18..0000000000 --- a/unreleased_history/behavior_changes/retry_on_corruption.md +++ /dev/null @@ -1 +0,0 @@ -On distributed file systems that support file system level checksum verification and reconstruction reads, RocksDB will now retry a file read if the initial read fails RocksDB block level or record level checksum verification. This applies to MANIFEST file reads when the DB is opened, and to SST file reads at all times. diff --git a/unreleased_history/bug_fixes/block_align_checksum_mismatch.md b/unreleased_history/bug_fixes/block_align_checksum_mismatch.md deleted file mode 100644 index b784e32c5c..0000000000 --- a/unreleased_history/bug_fixes/block_align_checksum_mismatch.md +++ /dev/null @@ -1 +0,0 @@ -Fix a bug causing `VerifyFileChecksums()` to return false-positive corruption under `BlockBasedTableOptions::block_align=true` diff --git a/unreleased_history/bug_fixes/consistent_view_for_new_iterators_api.md b/unreleased_history/bug_fixes/consistent_view_for_new_iterators_api.md deleted file mode 100644 index f537e474aa..0000000000 --- a/unreleased_history/bug_fixes/consistent_view_for_new_iterators_api.md +++ /dev/null @@ -1 +0,0 @@ -Provide consistent view of the database across the column families for `NewIterators()` API. diff --git a/unreleased_history/bug_fixes/del_range_and_memtable_insert_with_hint_prefix_extractor.md b/unreleased_history/bug_fixes/del_range_and_memtable_insert_with_hint_prefix_extractor.md deleted file mode 100644 index 7283fb95ac..0000000000 --- a/unreleased_history/bug_fixes/del_range_and_memtable_insert_with_hint_prefix_extractor.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed feature interaction bug for `DeleteRange()` together with `ColumnFamilyOptions::memtable_insert_with_hint_prefix_extractor`. The impact of this bug would likely be corruption or crashing. diff --git a/unreleased_history/bug_fixes/disable_manual_compaction_hang.md b/unreleased_history/bug_fixes/disable_manual_compaction_hang.md deleted file mode 100644 index c140874a1f..0000000000 --- a/unreleased_history/bug_fixes/disable_manual_compaction_hang.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed hang in `DisableManualCompactions()` where compactions waiting to be scheduled due to conflicts would not be canceled promptly diff --git a/unreleased_history/bug_fixes/max_successive_merges_regression.md b/unreleased_history/bug_fixes/max_successive_merges_regression.md deleted file mode 100644 index d00028014b..0000000000 --- a/unreleased_history/bug_fixes/max_successive_merges_regression.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed a regression when `ColumnFamilyOptions::max_successive_merges > 0` where the CPU overhead for deciding whether to merge could have increased unless the user had set the option `ColumnFamilyOptions::strict_max_successive_merges` diff --git a/unreleased_history/bug_fixes/multiget_partial_error_blob_dereference.md b/unreleased_history/bug_fixes/multiget_partial_error_blob_dereference.md deleted file mode 100644 index dfe5e7386b..0000000000 --- a/unreleased_history/bug_fixes/multiget_partial_error_blob_dereference.md +++ /dev/null @@ -1 +0,0 @@ -* Fixed a bug in `MultiGet()` and `MultiGetEntity()` together with blob files (`ColumnFamilyOptions::enable_blob_files == true`). An error looking up one of the keys could cause the results to be wrong for other keys for which the statuses were `Status::OK`. diff --git a/unreleased_history/bug_fixes/pad_bug.md b/unreleased_history/bug_fixes/pad_bug.md deleted file mode 100644 index ef89658510..0000000000 --- a/unreleased_history/bug_fixes/pad_bug.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a bug where wrong padded bytes are used to generate file checksum and `DataVerificationInfo::checksum` upon file creation diff --git a/unreleased_history/bug_fixes/pinnable_wide_columns_move.md b/unreleased_history/bug_fixes/pinnable_wide_columns_move.md deleted file mode 100644 index 8dceb277d3..0000000000 --- a/unreleased_history/bug_fixes/pinnable_wide_columns_move.md +++ /dev/null @@ -1 +0,0 @@ -Correctly implemented the move semantics of `PinnableWideColumns`. diff --git a/unreleased_history/bug_fixes/recycle_logs_toggle_bug.md b/unreleased_history/bug_fixes/recycle_logs_toggle_bug.md deleted file mode 100644 index 2cfd8c4666..0000000000 --- a/unreleased_history/bug_fixes/recycle_logs_toggle_bug.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a bug when the recycle_log_file_num in DBOptions is changed from 0 to non-zero when a DB is reopened. On a subsequent reopen, if a log file created when recycle_log_file_num==0 was reused previously, is alive and is empty, we could end up inserting stale WAL records into the memtable. diff --git a/unreleased_history/bug_fixes/slow_deletion_on_open.md b/unreleased_history/bug_fixes/slow_deletion_on_open.md deleted file mode 100644 index 338f1e6c4f..0000000000 --- a/unreleased_history/bug_fixes/slow_deletion_on_open.md +++ /dev/null @@ -1 +0,0 @@ -*Fix a bug where obsolete files' deletion during DB::Open are not rate limited with `SstFilemManager`'s slow deletion feature even if it's configured. \ No newline at end of file diff --git a/unreleased_history/new_features/cache_dumper.md b/unreleased_history/new_features/cache_dumper.md deleted file mode 100644 index 9df7267fea..0000000000 --- a/unreleased_history/new_features/cache_dumper.md +++ /dev/null @@ -1 +0,0 @@ -Added two options `deadline` and `max_size_bytes` for CacheDumper to exit early diff --git a/unreleased_history/new_features/get_entity_from_batch_and_db.md b/unreleased_history/new_features/get_entity_from_batch_and_db.md deleted file mode 100644 index 34c583512a..0000000000 --- a/unreleased_history/new_features/get_entity_from_batch_and_db.md +++ /dev/null @@ -1 +0,0 @@ -Added a new API `GetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for wide-column point lookups with read-your-own-writes consistency. Similarly to `GetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. diff --git a/unreleased_history/new_features/multi_cf_iterators.md b/unreleased_history/new_features/multi_cf_iterators.md deleted file mode 100644 index 1305d1501d..0000000000 --- a/unreleased_history/new_features/multi_cf_iterators.md +++ /dev/null @@ -1 +0,0 @@ -[Experimental] Introduce two new cross-column-family iterators - CoalescingIterator and AttributeGroupIterator. The CoalescingIterator enables users to iterate over multiple column families and access their values and columns. During this iteration, if the same key exists in more than one column family, the keys in the later column family will overshadow the previous ones. The AttributeGroupIterator allows users to gather wide columns per Column Family and create attribute groups while iterating over keys across all CFs. diff --git a/unreleased_history/new_features/multi_get_entity_from_batch_and_db.md b/unreleased_history/new_features/multi_get_entity_from_batch_and_db.md deleted file mode 100644 index 95efd40e3b..0000000000 --- a/unreleased_history/new_features/multi_get_entity_from_batch_and_db.md +++ /dev/null @@ -1 +0,0 @@ -Added a new API `MultiGetEntityFromBatchAndDB` to `WriteBatchWithIndex` that can be used for batched wide-column point lookups with read-your-own-writes consistency. Similarly to `MultiGetFromBatchAndDB`, the API can combine data from the write batch with data from the underlying database if needed. See the API comments for more details. diff --git a/unreleased_history/new_features/sst_file_reader_raw_table_iterator.md b/unreleased_history/new_features/sst_file_reader_raw_table_iterator.md deleted file mode 100644 index d3f2661598..0000000000 --- a/unreleased_history/new_features/sst_file_reader_raw_table_iterator.md +++ /dev/null @@ -1 +0,0 @@ -*Adds a `SstFileReader::NewTableIterator` API to support programmatically read a SST file as a raw table file. \ No newline at end of file diff --git a/unreleased_history/new_features/wait_for_purge_option.md b/unreleased_history/new_features/wait_for_purge_option.md deleted file mode 100644 index 518e022373..0000000000 --- a/unreleased_history/new_features/wait_for_purge_option.md +++ /dev/null @@ -1,2 +0,0 @@ -Add an option to `WaitForCompactOptions` - `wait_for_purge` to make `WaitForCompact()` API wait for background purge to complete - diff --git a/unreleased_history/public_api_changes/delete_range_rowcache.md b/unreleased_history/public_api_changes/delete_range_rowcache.md deleted file mode 100644 index a2d66d3728..0000000000 --- a/unreleased_history/public_api_changes/delete_range_rowcache.md +++ /dev/null @@ -1 +0,0 @@ -* DeleteRange() will return NotSupported() if row_cache is configured since they don't work together in some cases. \ No newline at end of file diff --git a/unreleased_history/public_api_changes/deprecate_compaction_options_compression.md b/unreleased_history/public_api_changes/deprecate_compaction_options_compression.md deleted file mode 100644 index 65dea56517..0000000000 --- a/unreleased_history/public_api_changes/deprecate_compaction_options_compression.md +++ /dev/null @@ -1 +0,0 @@ -* Deprecated `CompactionOptions::compression` since `CompactionOptions`'s API for configuring compression was incomplete, unsafe, and likely unnecessary diff --git a/unreleased_history/public_api_changes/migrate_fifo.md b/unreleased_history/public_api_changes/migrate_fifo.md deleted file mode 100644 index 93f0b44678..0000000000 --- a/unreleased_history/public_api_changes/migrate_fifo.md +++ /dev/null @@ -1,4 +0,0 @@ -Using `OptionChangeMigration()` to migrate from non-FIFO to FIFO compaction -with `Options::compaction_options_fifo.max_table_files_size` > 0 can cause -the whole DB to be dropped right after migration if the migrated data is larger than -`max_table_files_size`