dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-30 22:41:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/java/rocksjni
History
Nicolas Pepin-Perreault 5b073a7daa Access SST full file checksum via RocksDB#getLiveFilesMetadata (#11770) 
Summary:
**Description**

This PR passes along the native `LiveFileMetaData#file_checksum` field from the C++ class to the Java API as a copied byte array. If there is no file checksum generator factory set beforehand, then the array will empty. Please advise if you'd rather it be null - an empty array means one extra allocation, but it avoids possible null pointer exceptions.

> **Note**
> This functionality complements but does not supersede https://github.com/facebook/rocksdb/issues/11736

It's outside the scope here to add support for Java based `FileChecksumGenFactory` implementations. As a workaround, users can already use the built-in one by creating their initial `DBOptions` via properties:

```java
final Properties props = new Properties();
props.put("file_checksum_gen_factory", "FileChecksumGenCrc32cFactory");

try (final DBOptions dbOptions = DBOptions.getDBOptionsFromProps(props);
     final ColumnFamilyOptions cfOptions = new ColumnFamilyOptions();
     final Options options = new Options(dbOptions, cfOptions).setCreateIfMissing(true)) {
// do stuff
}
```

I wanted to add a better test, but unfortunately there's no available CRC32C implementation available in Java 8 without adding a dependency or adding a JNI helper for RocksDB's own implementation (or bumping the minimum version for tests to Java 9). That said, I understand the test is rather poor, so happy to change it to whatever you'd like.

**Context**

To give some context, we replicate RocksDB checkpoints to other nodes. Part of this is verifying the integrity of each file during replication. With a large enough RocksDB, computing the checksum ourselves is prohibitively expensive. Since SST files comprise the bulk of the data, we'd much rather delegate this to RocksDB on file write, and read it back after to compare.

It's likely we will provide a follow up to read the file checksum list directly from the manifest without having to open the DB, but this was the easiest first step to get it working for us.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11770

Reviewed By: hx235

Differential Revision: D52420729

Pulled By: ajkr

fbshipit-source-id: a873de35a48aaf315e125733091cd221a97b9073
2023-12-26 14:02:36 -08:00
..
backup_engine_options.cc Fix remaining uses of "backupable" (#9792) 2022-04-05 09:52:33 -07:00
backupenginejni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
cache.cc Put Cache and CacheWrapper in new public header (#11192) 2023-02-09 12:12:02 -08:00
cassandra_compactionfilterjni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
cassandra_value_operator.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
checkpoint.cc Add jni Support for API CreateColumnFamilyWithImport (#11646) 2023-11-06 07:38:42 -08:00
clock_cache.cc Revert NewClockCache signature (#10358) 2022-07-13 17:43:39 -07:00
columnfamilyhandle.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
compact_range_options.cc Implement missing compactrangeoptions from Java API (#10880) 2023-05-24 11:04:46 -07:00
compaction_filter.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_filter_factory.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
compaction_filter_factory_jnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_filter_factory_jnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_job_info.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_job_stats.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_options.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_options_fifo.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
compaction_options_universal.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
comparator.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
comparatorjnicallback.cc Fix Java API ComparatorOptions use after delete error (#11176) 2023-02-17 13:03:41 -08:00
comparatorjnicallback.h Fix Java API ComparatorOptions use after delete error (#11176) 2023-02-17 13:03:41 -08:00
compression_options.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
concurrent_task_limiter.cc Add some missing headers (#10519) 2022-08-11 12:45:50 -07:00
config_options.cc Remove deprecated util functions in options_util.h (#11126) 2023-01-27 11:10:53 -08:00
cplusplus_to_java_convert.h Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
env.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
env_options.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
event_listener.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
event_listener_jnicallback.cc Add (& fix) some simple source code checks (#8821) 2021-09-07 21:19:27 -07:00
event_listener_jnicallback.h Add (& fix) some simple source code checks (#8821) 2021-09-07 21:19:27 -07:00
export_import_files_metadatajni.cc Add jni Support for API CreateColumnFamilyWithImport (#11646) 2023-11-06 07:38:42 -08:00
filter.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
hyper_clock_cache.cc Add HyperClockCache Java API. (#12065) 2023-11-16 15:46:31 -08:00
import_column_family_options.cc Add jni Support for API CreateColumnFamilyWithImport (#11646) 2023-11-06 07:38:42 -08:00
ingest_external_file_options.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
iterator.cc Add Iterator::Refresh(Snapshot*) to RocksJava (#12145) 2023-12-20 18:03:42 -08:00
jni_perf_context.cc Add missing copyright headers to files added in PR 11805 (#11942) 2023-10-11 12:56:39 -07:00
jnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
jnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
kv_helper.h Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() (#11019) 2023-12-11 11:03:17 -08:00
loggerjnicallback.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
loggerjnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
lru_cache.cc Use the default metadata charge policy when creating an LRU cache via the Java API (#10577) 2022-08-29 09:42:04 -07:00
memory_util.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
memtablejni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
merge_operator.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
native_comparator_wrapper_test.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
optimistic_transaction_db.cc RocksJava API - fix Transaction.multiGet() size limit, remove bogus EnsureLocalCapacity() calls (#10674) 2022-10-26 17:25:33 -07:00
optimistic_transaction_options.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
options.cc Add deletion-triggered compaction to RocksJava (#12028) 2023-12-18 13:43:01 -08:00
options_util.cc java API - load block based table config (#10826) 2023-10-12 09:39:01 -07:00
persistent_cache.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
portal.h Access SST full file checksum via RocksDB#getLiveFilesMetadata (#11770) 2023-12-26 14:02:36 -08:00
ratelimiterjni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
remove_emptyvalue_compactionfilterjni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
restorejni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
rocks_callback_object.cc Fix tabs and lint-ignores (#6734) 2020-04-20 11:39:31 -07:00
rocksdb_exception_test.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
rocksjni.cc Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() (#11019) 2023-12-11 11:03:17 -08:00
slice.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
snapshot.cc Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) 2020-02-20 12:09:57 -08:00
sst_file_manager.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
sst_file_reader_iterator.cc Add Iterator::Refresh(Snapshot*) to RocksJava (#12145) 2023-12-20 18:03:42 -08:00
sst_file_readerjni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
sst_file_writerjni.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
sst_partitioner.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
statistics.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
statisticsjni.cc Fix tabs and lint-ignores (#6734) 2020-04-20 11:39:31 -07:00
statisticsjni.h Change internal headers with duplicate names (#11408) 2023-05-17 11:27:09 -07:00
table.cc Remove compressed block cache (#11117) 2023-01-24 17:09:19 -08:00
table_filter.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
table_filter_jnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
table_filter_jnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
table_properties_collector_factory.cc Fix header files to meet Open source requirements (#12164) 2023-12-19 13:43:17 -08:00
table_properties_collector_factory.h Fix header files to meet Open source requirements (#12164) 2023-12-19 13:43:17 -08:00
testable_event_listener.cc Update build files for java8 build (#9541) 2022-02-17 13:29:21 -08:00
thread_status.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
trace_writer.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
trace_writer_jnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
trace_writer_jnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
transaction.cc Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() (#11019) 2023-12-11 11:03:17 -08:00
transaction_db.cc RocksJava API - fix Transaction.multiGet() size limit, remove bogus EnsureLocalCapacity() calls (#10674) 2022-10-26 17:25:33 -07:00
transaction_db_options.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
transaction_log.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
transaction_notifier.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
transaction_notifier_jnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
transaction_notifier_jnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
transaction_options.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
ttl.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
wal_filter.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
wal_filter_jnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
wal_filter_jnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
write_batch.cc Fix pointer to jlong conversion in 32 bit OS (#9396) 2022-03-01 09:02:15 -08:00
write_batch_test.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
write_batch_with_index.cc Add Iterator::Refresh(Snapshot*) to RocksJava (#12145) 2023-12-20 18:03:42 -08:00
write_buffer_manager.cc Put Cache and CacheWrapper in new public header (#11192) 2023-02-09 12:12:02 -08:00
writebatchhandlerjnicallback.cc Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00
writebatchhandlerjnicallback.h Run format check for *.h and *.cc files under java/ (#10851) 2022-10-25 09:26:51 -07:00