From 748f74aca3c5a0271cfe9a0f6804ea9484db2eb2 Mon Sep 17 00:00:00 2001 From: Changyu Bi Date: Mon, 24 Jun 2024 11:53:05 -0700 Subject: [PATCH] Update main branch for 9.4 release (#12802) Summary: Main branch cut at e90e9153d5169f9df8ddd3cf0bf693572768a7b5. Updated HISTORY.md, version and format compatibility test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12802 Reviewed By: ajkr Differential Revision: D58956464 Pulled By: cbi42 fbshipit-source-id: 50d786c145cebf93d1dd554b1b0e26baac3cc88c --- HISTORY.md | 25 +++++++++++++++++++ include/rocksdb/version.h | 2 +- tools/check_format_compatible.sh | 2 +- .../aggressive_close_inactive_wals.md | 1 - .../bug_fixes/checkpoint_unsynced_wal_fix.md | 1 - .../bug_fixes/ldb_dump_wal_put_entity.md | 1 - .../bug_fixes/put_entity_rebuild_txn.md | 1 - unreleased_history/bug_fixes/read_options.md | 1 - .../auto_trigger_compaction_for_tiering.md | 1 - .../new_features/get_entity_for_update.md | 1 - .../new_features/ldb_count_command.md | 1 - .../new_features/write_batch_c_api | 3 --- ...ify-one_i_LaunchParallelMemTableWriters.md | 1 - .../public_api_changes/rename_to_wal_file.md | 1 - .../universal_max_read_amp.md | 1 - 15 files changed, 27 insertions(+), 16 deletions(-) delete mode 100644 unreleased_history/behavior_changes/aggressive_close_inactive_wals.md delete mode 100644 unreleased_history/bug_fixes/checkpoint_unsynced_wal_fix.md delete mode 100644 unreleased_history/bug_fixes/ldb_dump_wal_put_entity.md delete mode 100644 unreleased_history/bug_fixes/put_entity_rebuild_txn.md delete mode 100644 unreleased_history/bug_fixes/read_options.md delete mode 100644 unreleased_history/new_features/auto_trigger_compaction_for_tiering.md delete mode 100644 unreleased_history/new_features/get_entity_for_update.md delete mode 100644 unreleased_history/new_features/ldb_count_command.md delete mode 100644 unreleased_history/new_features/write_batch_c_api delete mode 100644 unreleased_history/performance_improvements/parallel_notify-one_i_LaunchParallelMemTableWriters.md delete mode 100644 unreleased_history/public_api_changes/rename_to_wal_file.md delete mode 100644 unreleased_history/public_api_changes/universal_max_read_amp.md diff --git a/HISTORY.md b/HISTORY.md index a1700f993a..234780a740 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,31 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 9.4.0 (06/23/2024) +### New Features +* Added a `CompactForTieringCollectorFactory` to auto trigger compaction for tiering use case. +* Optimistic transactions and pessimistic transactions with the WriteCommitted policy now support the `GetEntityForUpdate` API. +* Added a new "count" command to the ldb repl shell. By default, it prints a count of keys in the database from start to end. The options --from= and/or --to= can be specified to limit the range. +* Add `rocksdb_writebatch_update_timestamps`, `rocksdb_writebatch_wi_update_timestamps` in C API. +* Add `rocksdb_iter_refresh` in C API. +* Add `rocksdb_writebatch_create_with_params`, `rocksdb_writebatch_wi_create_with_params` to create WB and WBWI with all options in C API + +### Public API Changes +* Deprecated names `LogFile` and `VectorLogPtr` in favor of new names `WalFile` and `VectorWalPtr`. +* Introduce a new universal compaction option CompactionOptionsUniversal::max_read_amp which allows user to define the limit on the number of sorted runs separately from the trigger for compaction (`level0_file_num_compaction_trigger`) #12477. + +### Behavior Changes +* Inactive WALs are immediately closed upon being fully sync-ed rather than in a background thread. This is to ensure LinkFile() is not called on files still open for write, which might not be supported by some FileSystem implementations. This should not be a performance issue, but an opt-out is available with with new DB option `background_close_inactive_wals`. + +### Bug Fixes +* Fix a rare case in which a hard-linked WAL in a Checkpoint is not fully synced (so might lose data on power loss). +* Fixed the output of the `ldb dump_wal` command for `PutEntity` records so it prints the key and correctly resets the hexadecimal formatting flag after printing the wide-column entity. +* Fixed an issue where `PutEntity` records were handled incorrectly while rebuilding transactions during recovery. +* Various read operations could ignore various ReadOptions that might be relevant. Fixed many such cases, which can result in behavior change but a better reflection of specified options. + +### Performance Improvements +* Improved write throughput to memtable when there's a large number of concurrent writers and allow_concurrent_memtable_write=true(#12545) + ## 9.3.0 (05/17/2024) ### New Features * Optimistic transactions and pessimistic transactions with the WriteCommitted policy now support the `GetEntity` API. diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index a7fd178040..26c69ca41b 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -12,7 +12,7 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. #define ROCKSDB_MAJOR 9 -#define ROCKSDB_MINOR 4 +#define ROCKSDB_MINOR 5 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh index 07b7cf346b..8bbf3ae580 100755 --- a/tools/check_format_compatible.sh +++ b/tools/check_format_compatible.sh @@ -125,7 +125,7 @@ EOF # To check for DB forward compatibility with loading options (old version # reading data from new), as well as backward compatibility -declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb") +declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb") # To check for DB forward compatibility without loading options (in addition # to the "with loading options" set), as well as backward compatibility declare -a db_forward_no_options_refs=() # N/A at the moment diff --git a/unreleased_history/behavior_changes/aggressive_close_inactive_wals.md b/unreleased_history/behavior_changes/aggressive_close_inactive_wals.md deleted file mode 100644 index c75f598496..0000000000 --- a/unreleased_history/behavior_changes/aggressive_close_inactive_wals.md +++ /dev/null @@ -1 +0,0 @@ -* Inactive WALs are immediately closed upon being fully sync-ed rather than in a background thread. This is to ensure LinkFile() is not called on files still open for write, which might not be supported by some FileSystem implementations. This should not be a performance issue, but an opt-out is available with with new DB option `background_close_inactive_wals`. diff --git a/unreleased_history/bug_fixes/checkpoint_unsynced_wal_fix.md b/unreleased_history/bug_fixes/checkpoint_unsynced_wal_fix.md deleted file mode 100644 index 35b0622feb..0000000000 --- a/unreleased_history/bug_fixes/checkpoint_unsynced_wal_fix.md +++ /dev/null @@ -1 +0,0 @@ -* Fix a rare case in which a hard-linked WAL in a Checkpoint is not fully synced (so might lose data on power loss). diff --git a/unreleased_history/bug_fixes/ldb_dump_wal_put_entity.md b/unreleased_history/bug_fixes/ldb_dump_wal_put_entity.md deleted file mode 100644 index 5a77642c26..0000000000 --- a/unreleased_history/bug_fixes/ldb_dump_wal_put_entity.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the output of the `ldb dump_wal` command for `PutEntity` records so it prints the key and correctly resets the hexadecimal formatting flag after printing the wide-column entity. diff --git a/unreleased_history/bug_fixes/put_entity_rebuild_txn.md b/unreleased_history/bug_fixes/put_entity_rebuild_txn.md deleted file mode 100644 index 8793c0ff71..0000000000 --- a/unreleased_history/bug_fixes/put_entity_rebuild_txn.md +++ /dev/null @@ -1 +0,0 @@ -Fixed an issue where `PutEntity` records were handled incorrectly while rebuilding transactions during recovery. diff --git a/unreleased_history/bug_fixes/read_options.md b/unreleased_history/bug_fixes/read_options.md deleted file mode 100644 index 931748e14a..0000000000 --- a/unreleased_history/bug_fixes/read_options.md +++ /dev/null @@ -1 +0,0 @@ -* Various read operations could ignore various ReadOptions that might be relevant. Fixed many such cases, which can result in behavior change but a better reflection of specified options. diff --git a/unreleased_history/new_features/auto_trigger_compaction_for_tiering.md b/unreleased_history/new_features/auto_trigger_compaction_for_tiering.md deleted file mode 100644 index 973b40bfbc..0000000000 --- a/unreleased_history/new_features/auto_trigger_compaction_for_tiering.md +++ /dev/null @@ -1 +0,0 @@ -Added a `CompactForTieringCollectorFactory` to auto trigger compaction for tiering use case. \ No newline at end of file diff --git a/unreleased_history/new_features/get_entity_for_update.md b/unreleased_history/new_features/get_entity_for_update.md deleted file mode 100644 index 2a9c2afa04..0000000000 --- a/unreleased_history/new_features/get_entity_for_update.md +++ /dev/null @@ -1 +0,0 @@ -Optimistic transactions and pessimistic transactions with the WriteCommitted policy now support the `GetEntityForUpdate` API. diff --git a/unreleased_history/new_features/ldb_count_command.md b/unreleased_history/new_features/ldb_count_command.md deleted file mode 100644 index d631222147..0000000000 --- a/unreleased_history/new_features/ldb_count_command.md +++ /dev/null @@ -1 +0,0 @@ -Added a new "count" command to the ldb repl shell. By default, it prints a count of keys in the database from start to end. The options --from= and/or --to= can be specified to limit the range. diff --git a/unreleased_history/new_features/write_batch_c_api b/unreleased_history/new_features/write_batch_c_api deleted file mode 100644 index 150af09df8..0000000000 --- a/unreleased_history/new_features/write_batch_c_api +++ /dev/null @@ -1,3 +0,0 @@ -* Add `rocksdb_writebatch_update_timestamps`, `rocksdb_writebatch_wi_update_timestamps` in C API. -* Add `rocksdb_iter_refresh` in C API. -* Add `rocksdb_writebatch_create_with_params`, `rocksdb_writebatch_wi_create_with_params` to create WB and WBWI with all options in C API diff --git a/unreleased_history/performance_improvements/parallel_notify-one_i_LaunchParallelMemTableWriters.md b/unreleased_history/performance_improvements/parallel_notify-one_i_LaunchParallelMemTableWriters.md deleted file mode 100644 index 8fbfca124b..0000000000 --- a/unreleased_history/performance_improvements/parallel_notify-one_i_LaunchParallelMemTableWriters.md +++ /dev/null @@ -1 +0,0 @@ -Improved write throughput to memtable when there's a large number of concurrent writers and allow_concurrent_memtable_write=true(#12545) \ No newline at end of file diff --git a/unreleased_history/public_api_changes/rename_to_wal_file.md b/unreleased_history/public_api_changes/rename_to_wal_file.md deleted file mode 100644 index 58df40c473..0000000000 --- a/unreleased_history/public_api_changes/rename_to_wal_file.md +++ /dev/null @@ -1 +0,0 @@ -* Deprecated names `LogFile` and `VectorLogPtr` in favor of new names `WalFile` and `VectorWalPtr`. diff --git a/unreleased_history/public_api_changes/universal_max_read_amp.md b/unreleased_history/public_api_changes/universal_max_read_amp.md deleted file mode 100644 index 0c10a37b97..0000000000 --- a/unreleased_history/public_api_changes/universal_max_read_amp.md +++ /dev/null @@ -1 +0,0 @@ -* Introduce a new universal compaction option CompactionOptionsUniversal::max_read_amp which allows user to define the limit on the number of sorted runs separately from the trigger for compaction (`level0_file_num_compaction_trigger`) #12477. \ No newline at end of file