Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
NATIVE_JAVA_CLASSES = \
|
|
|
|
|
org.rocksdb.AbstractCompactionFilter\
|
2017-10-12 18:06:51 +00:00
|
|
|
org.rocksdb.AbstractCompactionFilterFactory\
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
org.rocksdb.AbstractComparator\
|
2020-10-14 18:32:10 +00:00
|
|
|
org.rocksdb.AbstractEventListener\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.AbstractSlice\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.AbstractTableFilter\
|
|
|
|
|
org.rocksdb.AbstractTraceWriter\
|
2018-03-02 18:22:38 +00:00
|
|
|
org.rocksdb.AbstractTransactionNotifier\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.AbstractWalFilter\
|
2015-08-04 23:07:17 +00:00
|
|
|
org.rocksdb.BackupEngine\
|
2022-01-27 23:44:23 +00:00
|
|
|
org.rocksdb.BackupEngineOptions\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.BlockBasedTableConfig\
|
|
|
|
|
org.rocksdb.BloomFilter\
|
2014-11-21 20:57:32 +00:00
|
|
|
org.rocksdb.Checkpoint\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.ClockCache\
|
2021-03-17 16:28:47 +00:00
|
|
|
org.rocksdb.Cache\
|
2017-07-21 21:42:32 +00:00
|
|
|
org.rocksdb.CassandraCompactionFilter\
|
2017-06-16 21:12:52 +00:00
|
|
|
org.rocksdb.CassandraValueMergeOperator\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.ColumnFamilyHandle\
|
2014-10-31 22:39:14 +00:00
|
|
|
org.rocksdb.ColumnFamilyOptions\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.CompactionJobInfo\
|
|
|
|
|
org.rocksdb.CompactionJobStats\
|
|
|
|
|
org.rocksdb.CompactionOptions\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.CompactionOptionsFIFO\
|
|
|
|
|
org.rocksdb.CompactionOptionsUniversal\
|
2018-08-17 17:52:58 +00:00
|
|
|
org.rocksdb.CompactRangeOptions\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.ComparatorOptions\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.CompressionOptions\
|
2020-04-22 00:35:28 +00:00
|
|
|
org.rocksdb.ConfigOptions\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.DBOptions\
|
|
|
|
|
org.rocksdb.DirectSlice\
|
2015-03-22 17:30:51 +00:00
|
|
|
org.rocksdb.Env\
|
2016-09-29 21:04:41 +00:00
|
|
|
org.rocksdb.EnvOptions\
|
2014-11-09 19:09:39 +00:00
|
|
|
org.rocksdb.FlushOptions\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.Filter\
|
2017-05-22 17:22:49 +00:00
|
|
|
org.rocksdb.IngestExternalFileOptions\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.HashLinkedListMemTableConfig\
|
|
|
|
|
org.rocksdb.HashSkipListMemTableConfig\
|
2020-09-09 19:42:54 +00:00
|
|
|
org.rocksdb.ConcurrentTaskLimiter\
|
|
|
|
|
org.rocksdb.ConcurrentTaskLimiterImpl\
|
2021-10-19 00:19:04 +00:00
|
|
|
org.rocksdb.KeyMayExist\
|
2015-03-10 22:16:21 +00:00
|
|
|
org.rocksdb.Logger\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.LRUCache\
|
2018-10-08 18:03:34 +00:00
|
|
|
org.rocksdb.MemoryUsageType\
|
|
|
|
|
org.rocksdb.MemoryUtil\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.MergeOperator\
|
2018-03-08 19:16:46 +00:00
|
|
|
org.rocksdb.NativeComparatorWrapper\
|
2018-03-02 18:22:38 +00:00
|
|
|
org.rocksdb.OptimisticTransactionDB\
|
|
|
|
|
org.rocksdb.OptimisticTransactionOptions\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.Options\
|
2017-09-22 00:15:27 +00:00
|
|
|
org.rocksdb.OptionsUtil\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.PersistentCache\
|
2023-10-10 18:07:33 +00:00
|
|
|
org.rocksdb.PerfContext\
|
|
|
|
|
org.rocksdb.PerfLevel\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.PlainTableConfig\
|
2016-10-07 19:32:21 +00:00
|
|
|
org.rocksdb.RateLimiter\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.ReadOptions\
|
2015-07-14 16:48:16 +00:00
|
|
|
org.rocksdb.RemoveEmptyValueCompactionFilter\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.RestoreOptions\
|
2017-10-12 18:06:51 +00:00
|
|
|
org.rocksdb.RocksCallbackObject\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.RocksDB\
|
|
|
|
|
org.rocksdb.RocksEnv\
|
|
|
|
|
org.rocksdb.RocksIterator\
|
2015-03-22 17:30:51 +00:00
|
|
|
org.rocksdb.RocksMemEnv\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.SkipListMemTableConfig\
|
|
|
|
|
org.rocksdb.Slice\
|
2018-04-07 04:22:37 +00:00
|
|
|
org.rocksdb.SstFileManager\
|
2016-09-29 21:04:41 +00:00
|
|
|
org.rocksdb.SstFileWriter\
|
2019-09-10 01:11:02 +00:00
|
|
|
org.rocksdb.SstFileReader\
|
|
|
|
|
org.rocksdb.SstFileReaderIterator\
|
2020-07-24 20:43:14 +00:00
|
|
|
org.rocksdb.SstPartitionerFactory\
|
|
|
|
|
org.rocksdb.SstPartitionerFixedPrefixFactory\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.Statistics\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.ThreadStatus\
|
|
|
|
|
org.rocksdb.TimedEnv\
|
2018-03-02 18:22:38 +00:00
|
|
|
org.rocksdb.Transaction\
|
|
|
|
|
org.rocksdb.TransactionDB\
|
|
|
|
|
org.rocksdb.TransactionDBOptions\
|
|
|
|
|
org.rocksdb.TransactionOptions\
|
2015-01-25 21:05:29 +00:00
|
|
|
org.rocksdb.TransactionLogIterator\
|
2014-11-17 18:22:44 +00:00
|
|
|
org.rocksdb.TtlDB\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.VectorMemTableConfig\
|
2015-01-30 20:05:45 +00:00
|
|
|
org.rocksdb.Snapshot\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.StringAppendOperator\
|
2018-10-13 00:33:55 +00:00
|
|
|
org.rocksdb.UInt64AddOperator\
|
2014-11-09 19:08:35 +00:00
|
|
|
org.rocksdb.WriteBatch\
|
|
|
|
|
org.rocksdb.WriteBatch.Handler\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.WriteOptions\
|
2015-01-03 14:52:07 +00:00
|
|
|
org.rocksdb.WriteBatchWithIndex\
|
2018-10-17 18:47:51 +00:00
|
|
|
org.rocksdb.WriteBufferManager\
|
Add native logger support to RocksJava (#12213)
Summary:
## Overview
In this PR, we introduce support for setting the RocksDB native logger through Java. As mentioned in the discussion on the [Google Group discussion](https://groups.google.com/g/rocksdb/c/xYmbEs4sqRM/m/e73E4whJAQAJ), this work is primarily motivated by the JDK 17 [performance regression in JNI thread attach/detach calls](https://bugs.openjdk.org/browse/JDK-8314859): the only existing RocksJava logging configuration call, `setLogger`, invokes the provided logger over the JNI.
## Changes
Specifically, these changes add support for the `devnull` and `stderr` native loggers. For the `stderr` logger, we add the ability to prefix every log with a `logPrefix`, so that it becomes possible know which database a particular log is coming from (if multiple databases are in use). The API looks like the following:
```java
Options opts = new Options();
NativeLogger stderrNativeLogger = NativeLogger.newStderrLogger(
InfoLogLevel.DEBUG_LEVEL, "[my prefix here]");
options.setLogger(stderrNativeLogger);
try (final RocksDB db = RocksDB.open(options, ...)) {...}
// Cleanup
stderrNativeLogger.close()
opts.close();
```
Note that the API to set the logger is the same, via `Options::setLogger` (or `DBOptions::setLogger`). However, it will set the RocksDB logger to be native when the provided logger is an instance of `NativeLogger`.
## Testing
Two tests have been added in `NativeLoggerTest.java`. The first test creates both the `devnull` and `stderr` loggers, and sets them on the associated `Options`. However, to avoid polluting the testing output with logs from `stderr`, only the `devnull` logger is actually used in the test. The second test does the same logic, but for `DBOptions`.
It is possible to manually verify the `stderr` logger by modifying the tests slightly, and observing that the console indeed gets cluttered with logs from `stderr`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12213
Reviewed By: cbi42
Differential Revision: D52772306
Pulled By: ajkr
fbshipit-source-id: 4026895f78f9cc250daf6bfa57427957e2d8b053
2024-01-18 01:51:36 +00:00
|
|
|
org.rocksdb.WBWIRocksIterator\
|
|
|
|
|
org.rocksdb.util.StdErrLogger
|
2014-05-14 05:22:21 +00:00
|
|
|
|
2020-12-16 23:58:56 +00:00
|
|
|
NATIVE_JAVA_TEST_CLASSES = \
|
|
|
|
|
org.rocksdb.RocksDBExceptionTest\
|
|
|
|
|
org.rocksdb.test.TestableEventListener\
|
2018-03-08 19:16:46 +00:00
|
|
|
org.rocksdb.NativeComparatorWrapperTest.NativeStringComparatorWrapper\
|
2016-08-22 18:01:42 +00:00
|
|
|
org.rocksdb.WriteBatchTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.WriteBatchTestInternalHelper
|
|
|
|
|
|
2022-09-22 23:58:21 +00:00
|
|
|
ROCKSDB_MAJOR = $(shell grep -E "ROCKSDB_MAJOR.[0-9]" ../include/rocksdb/version.h | cut -d ' ' -f 3)
|
|
|
|
|
ROCKSDB_MINOR = $(shell grep -E "ROCKSDB_MINOR.[0-9]" ../include/rocksdb/version.h | cut -d ' ' -f 3)
|
|
|
|
|
ROCKSDB_PATCH = $(shell grep -E "ROCKSDB_PATCH.[0-9]" ../include/rocksdb/version.h | cut -d ' ' -f 3)
|
2014-10-02 18:07:45 +00:00
|
|
|
|
Add a jni library for rocksdb which supports Open, Get, Put, and Close.
Summary:
This diff contains a simple jni library for rocksdb which supports open, get,
put and closeusing default options (including Options, ReadOptions, and
WriteOptions.) In the usual case, Java developers can use the c++ rocksdb
library in the way similar to the following:
RocksDB db = RocksDB.open(path_to_db);
...
db.put("hello".getBytes(), "world".getBytes();
byte[] value = db.get("hello".getBytes());
...
db.close();
Specifically, this diff has the following major classes:
* RocksDB: a Java wrapper class which forwards the operations
from the java side to c++ rocksdb library.
* RocksDBException: ncapsulates the error of an operation.
This exception type is used to describe an internal error from
the c++ rocksdb library.
This diff also include a simple java sample code calling c++ rocksdb library.
To build the rocksdb jni library, simply run make jni, and make jtest will try to
build and run the sample code.
Note that if the rocksdb is not built with the default glibc that Java uses,
java will try to load the wrong glibc during the run time. As a result,
the sample code might not work properly during the run time.
Test Plan:
* make jni
* make jtest
Reviewers: haobo, dhruba, sdong, igor, ljin
Reviewed By: dhruba
CC: leveldb, xjin
Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 21:19:21 +00:00
|
|
|
NATIVE_INCLUDE = ./include
|
2014-09-26 20:57:12 +00:00
|
|
|
ARCH := $(shell getconf LONG_BIT)
|
2021-01-13 23:59:36 +00:00
|
|
|
SHA256_CMD ?= sha256sum
|
Add a jni library for rocksdb which supports Open, Get, Put, and Close.
Summary:
This diff contains a simple jni library for rocksdb which supports open, get,
put and closeusing default options (including Options, ReadOptions, and
WriteOptions.) In the usual case, Java developers can use the c++ rocksdb
library in the way similar to the following:
RocksDB db = RocksDB.open(path_to_db);
...
db.put("hello".getBytes(), "world".getBytes();
byte[] value = db.get("hello".getBytes());
...
db.close();
Specifically, this diff has the following major classes:
* RocksDB: a Java wrapper class which forwards the operations
from the java side to c++ rocksdb library.
* RocksDBException: ncapsulates the error of an operation.
This exception type is used to describe an internal error from
the c++ rocksdb library.
This diff also include a simple java sample code calling c++ rocksdb library.
To build the rocksdb jni library, simply run make jni, and make jtest will try to
build and run the sample code.
Note that if the rocksdb is not built with the default glibc that Java uses,
java will try to load the wrong glibc during the run time. As a result,
the sample code might not work properly during the run time.
Test Plan:
* make jni
* make jtest
Reviewers: haobo, dhruba, sdong, igor, ljin
Reviewed By: dhruba
CC: leveldb, xjin
Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 21:19:21 +00:00
|
|
|
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
JAVA_TESTS = \
|
2022-01-27 23:44:23 +00:00
|
|
|
org.rocksdb.BackupEngineOptionsTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.BackupEngineTest\
|
2021-10-19 16:20:56 +00:00
|
|
|
org.rocksdb.BlobOptionsTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.BlockBasedTableConfigTest\
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
org.rocksdb.BuiltinComparatorTest\
|
Fix Java API ComparatorOptions use after delete error (#11176)
Summary:
The problem
-------------
ComparatorOptions is AutoCloseable.
AbstractComparator does not hold a reference to its ComparatorOptions, but the native C++ ComparatorJniCallback holds a reference to the ComparatorOptions’ native C++ options structure. This gets deleted when the ComparatorOptions is closed, either explicitly, or as part of try-with-resources.
Later, the deleted C++ options structure gets used by the callback and the comparator options are effectively random.
The original bug report https://github.com/facebook/rocksdb/issues/8715 was caused by a GC-initiated finalization closing the still-in-use ComparatorOptions. As of 7.0, finalization of RocksDB objects no longer closes them, which worked round the reported bug, but still left ComparatorOptions with a potentially broken lifetime.
In any case, we encourage API clients to use the try-with-resources model, and so we need it to work. And if they don't use it, they leak resources.
The solution
-------------
The solution implemented here is to make a copy of the native C++ options object into the ComparatorJniCallback, rather than a reference. Then the deletion of the native object held by ComparatorOptions is *correctly* deleted when its scope is closed in try/finally.
Testing
-------
We added a regression unit test based on the original test for the reported ticket.
This checkin closes https://github.com/facebook/rocksdb/issues/8715
We expect that there are more instances of "lifecycle" bugs in the Java API. They are a major source of support time/cost, and we note that they could be addressed as a whole using the model proposed/prototyped in https://github.com/facebook/rocksdb/pull/10736
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11176
Reviewed By: cbi42
Differential Revision: D43160885
Pulled By: pdillinger
fbshipit-source-id: 60b54215a02ad9abb17363319650328c00a9ad62
2023-02-17 21:03:41 +00:00
|
|
|
org.rocksdb.ByteBufferUnsupportedOperationTest\
|
2021-11-01 22:05:25 +00:00
|
|
|
org.rocksdb.BytewiseComparatorRegressionTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.util.BytewiseComparatorTest\
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
org.rocksdb.util.BytewiseComparatorIntTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.CheckPointTest\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.ClockCacheTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.ColumnFamilyOptionsTest\
|
|
|
|
|
org.rocksdb.ColumnFamilyTest\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.CompactionFilterFactoryTest\
|
|
|
|
|
org.rocksdb.CompactionJobInfoTest\
|
|
|
|
|
org.rocksdb.CompactionJobStatsTest\
|
|
|
|
|
org.rocksdb.CompactionOptionsTest\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.CompactionOptionsFIFOTest\
|
|
|
|
|
org.rocksdb.CompactionOptionsUniversalTest\
|
|
|
|
|
org.rocksdb.CompactionPriorityTest\
|
|
|
|
|
org.rocksdb.CompactionStopStyleTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.ComparatorOptionsTest\
|
|
|
|
|
org.rocksdb.CompressionOptionsTest\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.CompressionTypesTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.DBOptionsTest\
|
|
|
|
|
org.rocksdb.DirectSliceTest\
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
org.rocksdb.util.EnvironmentTest\
|
2016-09-29 21:04:41 +00:00
|
|
|
org.rocksdb.EnvOptionsTest\
|
2020-10-14 18:32:10 +00:00
|
|
|
org.rocksdb.EventListenerTest\
|
2017-05-22 17:22:49 +00:00
|
|
|
org.rocksdb.IngestExternalFileOptionsTest\
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
org.rocksdb.util.IntComparatorTest\
|
|
|
|
|
org.rocksdb.util.JNIComparatorTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.FilterTest\
|
|
|
|
|
org.rocksdb.FlushTest\
|
2023-11-06 15:38:42 +00:00
|
|
|
org.rocksdb.ImportColumnFamilyTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.InfoLogLevelTest\
|
2023-10-18 19:46:35 +00:00
|
|
|
org.rocksdb.KeyExistsTest \
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.KeyMayExistTest\
|
2020-09-09 19:42:54 +00:00
|
|
|
org.rocksdb.ConcurrentTaskLimiterTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.LoggerTest\
|
2017-09-22 00:15:27 +00:00
|
|
|
org.rocksdb.LRUCacheTest\
|
2018-10-08 18:03:34 +00:00
|
|
|
org.rocksdb.MemoryUtilTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.MemTableTest\
|
Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() (#11019)
Summary:
### Implement new Java API get()/put()/merge() methods, and transactional variants.
The Java API methods are very inconsistent in terms of how they pass parameters (byte[], ByteBuffer), and what variants and defaulted parameters they support. We try to bring some consistency to this.
* All APIs should support calls with ByteBuffer parameters.
* Similar methods (RocksDB.get() vs Transaction.get()) should support as similar as possible sets of parameters for predictability.
* get()-like methods should provide variants where the caller supplies the target buffer, for the sake of efficiency. Allocation costs in Java can be significant when large buffers are repeatedly allocated and freed.
### API Additions
1. RockDB.get implement indirect ByteBuffers. Added indirect ByteBuffers and supporting native methods for get().
2. RocksDB.Iterator implement missing (byte[], offset, length) variants for key() and value() parameters.
3. Transaction.get() implement missing methods, based on RocksDB.get. Added ByteBuffer.get with and without column family. Added byte[]-as-target get.
4. Transaction.iterator() implement a getIterator() which defaults ReadOptions; as per RocksDB.iterator(). Rationalize support API for this and RocksDB.iterator()
5. RocksDB.merge implement ByteBuffer methods; both direct and indirect buffers. Shadow the methods of RocksDB.put; RocksDB.put only offers ByteBuffer API with explicit WriteOptions. Duplicated this with RocksDB.merge
6. Transaction.merge implement methods as per RocksDB.merge methods. Transaction is already constructed with WriteOptions, so no explicit WriteOptions methods required.
7. Transaction.mergeUntracked implement the same API methods as Transaction.merge except the ones that use assumeTracked, because that’s not a feature of merge untracked.
### Support Changes (C++)
The current JNI code in C++ supports multiple variants of methods through a number of helper functions. There are numerous TODO suggestions in the code proposing that the helpers be re-factored/shared.
We have taken a different approach for the new methods; we have created wrapper classes `JDirectBufferSlice`, `JDirectBufferPinnableSlice`, `JByteArraySlice` and `JByteArrayPinnableSlice` RAII classes which construct slices from JNI parameters and can then be passed directly to RocksDB methods. For instance, the `Java_org_rocksdb_Transaction_getDirect` method is implemented like this:
```
try {
ROCKSDB_NAMESPACE::JDirectBufferSlice key(env, jkey_bb, jkey_off,
jkey_part_len);
ROCKSDB_NAMESPACE::JDirectBufferPinnableSlice value(env, jval_bb, jval_off,
jval_part_len);
ROCKSDB_NAMESPACE::KVException::ThrowOnError(
env, txn->Get(*read_options, column_family_handle, key.slice(),
&value.pinnable_slice()));
return value.Fetch();
} catch (const ROCKSDB_NAMESPACE::KVException& e) {
return e.Code();
}
```
Notice the try/catch mechanism with the `KVException` class, which combined with RAII and the wrapper classes means that there is no ad-hoc cleanup necessary in the JNI methods.
We propose to extend this mechanism to existing JNI methods as further work.
### Support Changes (Java)
Where there are multiple parameter-variant versions of the same method, we use fewer or just one supporting native method for all of them. This makes maintenance a bit easier and reduces the opportunity for coding errors mixing up (untyped) object handles.
In order to support this efficiently, some classes need to have default values for column families and read options added and cached so that they are not re-constructed on every method call.
This PR closes https://github.com/facebook/rocksdb/issues/9776
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11019
Reviewed By: ajkr
Differential Revision: D52039446
Pulled By: jowlyzhang
fbshipit-source-id: 45d0140a4887e42134d2e56520e9b8efbd349660
2023-12-11 19:03:17 +00:00
|
|
|
org.rocksdb.MergeCFVariantsTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.MergeTest\
|
Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() (#11019)
Summary:
### Implement new Java API get()/put()/merge() methods, and transactional variants.
The Java API methods are very inconsistent in terms of how they pass parameters (byte[], ByteBuffer), and what variants and defaulted parameters they support. We try to bring some consistency to this.
* All APIs should support calls with ByteBuffer parameters.
* Similar methods (RocksDB.get() vs Transaction.get()) should support as similar as possible sets of parameters for predictability.
* get()-like methods should provide variants where the caller supplies the target buffer, for the sake of efficiency. Allocation costs in Java can be significant when large buffers are repeatedly allocated and freed.
### API Additions
1. RockDB.get implement indirect ByteBuffers. Added indirect ByteBuffers and supporting native methods for get().
2. RocksDB.Iterator implement missing (byte[], offset, length) variants for key() and value() parameters.
3. Transaction.get() implement missing methods, based on RocksDB.get. Added ByteBuffer.get with and without column family. Added byte[]-as-target get.
4. Transaction.iterator() implement a getIterator() which defaults ReadOptions; as per RocksDB.iterator(). Rationalize support API for this and RocksDB.iterator()
5. RocksDB.merge implement ByteBuffer methods; both direct and indirect buffers. Shadow the methods of RocksDB.put; RocksDB.put only offers ByteBuffer API with explicit WriteOptions. Duplicated this with RocksDB.merge
6. Transaction.merge implement methods as per RocksDB.merge methods. Transaction is already constructed with WriteOptions, so no explicit WriteOptions methods required.
7. Transaction.mergeUntracked implement the same API methods as Transaction.merge except the ones that use assumeTracked, because that’s not a feature of merge untracked.
### Support Changes (C++)
The current JNI code in C++ supports multiple variants of methods through a number of helper functions. There are numerous TODO suggestions in the code proposing that the helpers be re-factored/shared.
We have taken a different approach for the new methods; we have created wrapper classes `JDirectBufferSlice`, `JDirectBufferPinnableSlice`, `JByteArraySlice` and `JByteArrayPinnableSlice` RAII classes which construct slices from JNI parameters and can then be passed directly to RocksDB methods. For instance, the `Java_org_rocksdb_Transaction_getDirect` method is implemented like this:
```
try {
ROCKSDB_NAMESPACE::JDirectBufferSlice key(env, jkey_bb, jkey_off,
jkey_part_len);
ROCKSDB_NAMESPACE::JDirectBufferPinnableSlice value(env, jval_bb, jval_off,
jval_part_len);
ROCKSDB_NAMESPACE::KVException::ThrowOnError(
env, txn->Get(*read_options, column_family_handle, key.slice(),
&value.pinnable_slice()));
return value.Fetch();
} catch (const ROCKSDB_NAMESPACE::KVException& e) {
return e.Code();
}
```
Notice the try/catch mechanism with the `KVException` class, which combined with RAII and the wrapper classes means that there is no ad-hoc cleanup necessary in the JNI methods.
We propose to extend this mechanism to existing JNI methods as further work.
### Support Changes (Java)
Where there are multiple parameter-variant versions of the same method, we use fewer or just one supporting native method for all of them. This makes maintenance a bit easier and reduces the opportunity for coding errors mixing up (untyped) object handles.
In order to support this efficiently, some classes need to have default values for column families and read options added and cached so that they are not re-constructed on every method call.
This PR closes https://github.com/facebook/rocksdb/issues/9776
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11019
Reviewed By: ajkr
Differential Revision: D52039446
Pulled By: jowlyzhang
fbshipit-source-id: 45d0140a4887e42134d2e56520e9b8efbd349660
2023-12-11 19:03:17 +00:00
|
|
|
org.rocksdb.MergeVariantsTest\
|
2022-10-27 00:25:33 +00:00
|
|
|
org.rocksdb.MultiColumnRegressionTest \
|
2021-10-14 18:46:59 +00:00
|
|
|
org.rocksdb.MultiGetManyKeysTest\
|
|
|
|
|
org.rocksdb.MultiGetTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.MixedOptionsTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.MutableColumnFamilyOptionsTest\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.MutableDBOptionsTest\
|
2021-10-19 16:20:56 +00:00
|
|
|
org.rocksdb.MutableOptionsGetSetTest \
|
2018-03-08 19:16:46 +00:00
|
|
|
org.rocksdb.NativeComparatorWrapperTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.NativeLibraryLoaderTest\
|
2018-03-02 18:22:38 +00:00
|
|
|
org.rocksdb.OptimisticTransactionTest\
|
|
|
|
|
org.rocksdb.OptimisticTransactionDBTest\
|
|
|
|
|
org.rocksdb.OptimisticTransactionOptionsTest\
|
2017-09-22 00:15:27 +00:00
|
|
|
org.rocksdb.OptionsUtilTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.OptionsTest\
|
2023-10-10 18:07:33 +00:00
|
|
|
org.rocksdb.PerfLevelTest \
|
|
|
|
|
org.rocksdb.PerfContextTest \
|
Java API consistency between RocksDB.put() , .merge() and Transaction.put() , .merge() (#11019)
Summary:
### Implement new Java API get()/put()/merge() methods, and transactional variants.
The Java API methods are very inconsistent in terms of how they pass parameters (byte[], ByteBuffer), and what variants and defaulted parameters they support. We try to bring some consistency to this.
* All APIs should support calls with ByteBuffer parameters.
* Similar methods (RocksDB.get() vs Transaction.get()) should support as similar as possible sets of parameters for predictability.
* get()-like methods should provide variants where the caller supplies the target buffer, for the sake of efficiency. Allocation costs in Java can be significant when large buffers are repeatedly allocated and freed.
### API Additions
1. RockDB.get implement indirect ByteBuffers. Added indirect ByteBuffers and supporting native methods for get().
2. RocksDB.Iterator implement missing (byte[], offset, length) variants for key() and value() parameters.
3. Transaction.get() implement missing methods, based on RocksDB.get. Added ByteBuffer.get with and without column family. Added byte[]-as-target get.
4. Transaction.iterator() implement a getIterator() which defaults ReadOptions; as per RocksDB.iterator(). Rationalize support API for this and RocksDB.iterator()
5. RocksDB.merge implement ByteBuffer methods; both direct and indirect buffers. Shadow the methods of RocksDB.put; RocksDB.put only offers ByteBuffer API with explicit WriteOptions. Duplicated this with RocksDB.merge
6. Transaction.merge implement methods as per RocksDB.merge methods. Transaction is already constructed with WriteOptions, so no explicit WriteOptions methods required.
7. Transaction.mergeUntracked implement the same API methods as Transaction.merge except the ones that use assumeTracked, because that’s not a feature of merge untracked.
### Support Changes (C++)
The current JNI code in C++ supports multiple variants of methods through a number of helper functions. There are numerous TODO suggestions in the code proposing that the helpers be re-factored/shared.
We have taken a different approach for the new methods; we have created wrapper classes `JDirectBufferSlice`, `JDirectBufferPinnableSlice`, `JByteArraySlice` and `JByteArrayPinnableSlice` RAII classes which construct slices from JNI parameters and can then be passed directly to RocksDB methods. For instance, the `Java_org_rocksdb_Transaction_getDirect` method is implemented like this:
```
try {
ROCKSDB_NAMESPACE::JDirectBufferSlice key(env, jkey_bb, jkey_off,
jkey_part_len);
ROCKSDB_NAMESPACE::JDirectBufferPinnableSlice value(env, jval_bb, jval_off,
jval_part_len);
ROCKSDB_NAMESPACE::KVException::ThrowOnError(
env, txn->Get(*read_options, column_family_handle, key.slice(),
&value.pinnable_slice()));
return value.Fetch();
} catch (const ROCKSDB_NAMESPACE::KVException& e) {
return e.Code();
}
```
Notice the try/catch mechanism with the `KVException` class, which combined with RAII and the wrapper classes means that there is no ad-hoc cleanup necessary in the JNI methods.
We propose to extend this mechanism to existing JNI methods as further work.
### Support Changes (Java)
Where there are multiple parameter-variant versions of the same method, we use fewer or just one supporting native method for all of them. This makes maintenance a bit easier and reduces the opportunity for coding errors mixing up (untyped) object handles.
In order to support this efficiently, some classes need to have default values for column families and read options added and cached so that they are not re-constructed on every method call.
This PR closes https://github.com/facebook/rocksdb/issues/9776
Pull Request resolved: https://github.com/facebook/rocksdb/pull/11019
Reviewed By: ajkr
Differential Revision: D52039446
Pulled By: jowlyzhang
fbshipit-source-id: 45d0140a4887e42134d2e56520e9b8efbd349660
2023-12-11 19:03:17 +00:00
|
|
|
org.rocksdb.PutCFVariantsTest\
|
|
|
|
|
org.rocksdb.PutVariantsTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.PlainTableConfigTest\
|
2017-02-28 00:26:12 +00:00
|
|
|
org.rocksdb.RateLimiterTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.ReadOnlyTest\
|
|
|
|
|
org.rocksdb.ReadOptionsTest\
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
org.rocksdb.util.ReverseBytewiseComparatorIntTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.RocksDBTest\
|
2016-08-22 18:01:42 +00:00
|
|
|
org.rocksdb.RocksDBExceptionTest\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.DefaultEnvTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.RocksIteratorTest\
|
2015-03-22 17:30:51 +00:00
|
|
|
org.rocksdb.RocksMemEnvTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.util.SizeUnitTest\
|
2020-07-06 18:47:00 +00:00
|
|
|
org.rocksdb.SecondaryDBTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.SliceTest\
|
|
|
|
|
org.rocksdb.SnapshotTest\
|
2018-04-07 04:22:37 +00:00
|
|
|
org.rocksdb.SstFileManagerTest\
|
2016-09-29 21:04:41 +00:00
|
|
|
org.rocksdb.SstFileWriterTest\
|
2019-09-10 01:11:02 +00:00
|
|
|
org.rocksdb.SstFileReaderTest\
|
2020-07-24 20:43:14 +00:00
|
|
|
org.rocksdb.SstPartitionerTest\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.TableFilterTest\
|
|
|
|
|
org.rocksdb.TimedEnvTest\
|
2018-03-02 18:22:38 +00:00
|
|
|
org.rocksdb.TransactionTest\
|
|
|
|
|
org.rocksdb.TransactionDBTest\
|
|
|
|
|
org.rocksdb.TransactionOptionsTest\
|
|
|
|
|
org.rocksdb.TransactionDBOptionsTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.TransactionLogIteratorTest\
|
|
|
|
|
org.rocksdb.TtlDBTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.StatisticsTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.StatisticsCollectorTest\
|
2022-03-29 18:54:54 +00:00
|
|
|
org.rocksdb.VerifyChecksumsTest\
|
2019-02-22 22:36:38 +00:00
|
|
|
org.rocksdb.WalFilterTest\
|
2017-03-30 19:04:09 +00:00
|
|
|
org.rocksdb.WALRecoveryModeTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.WriteBatchHandlerTest\
|
|
|
|
|
org.rocksdb.WriteBatchTest\
|
2017-07-12 06:19:53 +00:00
|
|
|
org.rocksdb.WriteBatchThreadedTest\
|
2015-01-31 19:43:09 +00:00
|
|
|
org.rocksdb.WriteOptionsTest\
|
Add native logger support to RocksJava (#12213)
Summary:
## Overview
In this PR, we introduce support for setting the RocksDB native logger through Java. As mentioned in the discussion on the [Google Group discussion](https://groups.google.com/g/rocksdb/c/xYmbEs4sqRM/m/e73E4whJAQAJ), this work is primarily motivated by the JDK 17 [performance regression in JNI thread attach/detach calls](https://bugs.openjdk.org/browse/JDK-8314859): the only existing RocksJava logging configuration call, `setLogger`, invokes the provided logger over the JNI.
## Changes
Specifically, these changes add support for the `devnull` and `stderr` native loggers. For the `stderr` logger, we add the ability to prefix every log with a `logPrefix`, so that it becomes possible know which database a particular log is coming from (if multiple databases are in use). The API looks like the following:
```java
Options opts = new Options();
NativeLogger stderrNativeLogger = NativeLogger.newStderrLogger(
InfoLogLevel.DEBUG_LEVEL, "[my prefix here]");
options.setLogger(stderrNativeLogger);
try (final RocksDB db = RocksDB.open(options, ...)) {...}
// Cleanup
stderrNativeLogger.close()
opts.close();
```
Note that the API to set the logger is the same, via `Options::setLogger` (or `DBOptions::setLogger`). However, it will set the RocksDB logger to be native when the provided logger is an instance of `NativeLogger`.
## Testing
Two tests have been added in `NativeLoggerTest.java`. The first test creates both the `devnull` and `stderr` loggers, and sets them on the associated `Options`. However, to avoid polluting the testing output with logs from `stderr`, only the `devnull` logger is actually used in the test. The second test does the same logic, but for `DBOptions`.
It is possible to manually verify the `stderr` logger by modifying the tests slightly, and observing that the console indeed gets cluttered with logs from `stderr`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/12213
Reviewed By: cbi42
Differential Revision: D52772306
Pulled By: ajkr
fbshipit-source-id: 4026895f78f9cc250daf6bfa57427957e2d8b053
2024-01-18 01:51:36 +00:00
|
|
|
org.rocksdb.WriteBatchWithIndexTest\
|
|
|
|
|
org.rocksdb.util.StdErrLoggerTest
|
2015-01-31 19:43:09 +00:00
|
|
|
|
2015-01-31 22:23:59 +00:00
|
|
|
MAIN_SRC = src/main/java
|
2015-01-31 19:43:09 +00:00
|
|
|
TEST_SRC = src/test/java
|
|
|
|
|
OUTPUT = target
|
2015-01-31 22:23:59 +00:00
|
|
|
MAIN_CLASSES = $(OUTPUT)/classes
|
2015-01-31 19:43:09 +00:00
|
|
|
TEST_CLASSES = $(OUTPUT)/test-classes
|
2015-01-31 22:23:59 +00:00
|
|
|
JAVADOC = $(OUTPUT)/apidocs
|
2014-11-02 00:08:41 +00:00
|
|
|
|
2015-01-31 22:42:13 +00:00
|
|
|
BENCHMARK_MAIN_SRC = benchmark/src/main/java
|
|
|
|
|
BENCHMARK_OUTPUT = benchmark/target
|
|
|
|
|
BENCHMARK_MAIN_CLASSES = $(BENCHMARK_OUTPUT)/classes
|
|
|
|
|
|
2015-01-31 23:43:01 +00:00
|
|
|
SAMPLES_MAIN_SRC = samples/src/main/java
|
|
|
|
|
SAMPLES_OUTPUT = samples/target
|
|
|
|
|
SAMPLES_MAIN_CLASSES = $(SAMPLES_OUTPUT)/classes
|
|
|
|
|
|
2015-01-31 23:21:19 +00:00
|
|
|
JAVA_TEST_LIBDIR = test-libs
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_JUNIT_VER = 4.13.1
|
|
|
|
|
JAVA_JUNIT_SHA256 = c30719db974d6452793fe191b3638a5777005485bae145924044530ffa5f6122
|
2020-12-16 23:58:56 +00:00
|
|
|
JAVA_JUNIT_JAR = junit-$(JAVA_JUNIT_VER).jar
|
|
|
|
|
JAVA_JUNIT_JAR_PATH = $(JAVA_TEST_LIBDIR)/$(JAVA_JUNIT_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_HAMCREST_VER = 2.2
|
|
|
|
|
JAVA_HAMCREST_SHA256 = 5e62846a89f05cd78cd9c1a553f340d002458380c320455dd1f8fc5497a8a1c1
|
|
|
|
|
JAVA_HAMCREST_JAR = hamcrest-$(JAVA_HAMCREST_VER).jar
|
2020-12-16 23:58:56 +00:00
|
|
|
JAVA_HAMCREST_JAR_PATH = $(JAVA_TEST_LIBDIR)/$(JAVA_HAMCREST_JAR)
|
|
|
|
|
JAVA_MOCKITO_VER = 1.10.19
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_MOCKITO_SHA256 = d1a7a7ef14b3db5c0fc3e0a63a81b374b510afe85add9f7984b97911f4c70605
|
2020-12-16 23:58:56 +00:00
|
|
|
JAVA_MOCKITO_JAR = mockito-all-$(JAVA_MOCKITO_VER).jar
|
|
|
|
|
JAVA_MOCKITO_JAR_PATH = $(JAVA_TEST_LIBDIR)/$(JAVA_MOCKITO_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_CGLIB_VER = 3.3.0
|
|
|
|
|
JAVA_CGLIB_SHA256 = 9fe0c26d7464140ccdfe019ac687be1fb906122b508ab54beb810db0f09a9212
|
2020-12-16 23:58:56 +00:00
|
|
|
JAVA_CGLIB_JAR = cglib-$(JAVA_CGLIB_VER).jar
|
|
|
|
|
JAVA_CGLIB_JAR_PATH = $(JAVA_TEST_LIBDIR)/$(JAVA_CGLIB_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_ASSERTJ_VER = 2.9.0
|
|
|
|
|
JAVA_ASSERTJ_SHA256 = 5e88ea3ecbe3c48aa1346fec76c84979fa9c8d22499f11479011691230e8babf
|
2020-12-16 23:58:56 +00:00
|
|
|
JAVA_ASSERTJ_JAR = assertj-core-$(JAVA_ASSERTJ_VER).jar
|
|
|
|
|
JAVA_ASSERTJ_JAR_PATH = $(JAVA_TEST_LIBDIR)/$(JAVA_ASSERTJ_JAR)
|
|
|
|
|
JAVA_TESTCLASSPATH = $(JAVA_JUNIT_JAR_PATH):$(JAVA_HAMCREST_JAR_PATH):$(JAVA_MOCKITO_JAR_PATH):$(JAVA_CGLIB_JAR_PATH):$(JAVA_ASSERTJ_JAR_PATH)
|
2014-11-02 00:08:41 +00:00
|
|
|
|
2015-07-14 17:39:22 +00:00
|
|
|
MVN_LOCAL = ~/.m2/repository
|
|
|
|
|
|
2021-12-22 20:56:45 +00:00
|
|
|
# Set the path of the java commands
|
|
|
|
|
ifeq ($(JAVA_CMD),)
|
|
|
|
|
ifneq ($(JAVA_HOME),)
|
|
|
|
|
JAVA_CMD := $(JAVA_HOME)/bin/java
|
|
|
|
|
else
|
|
|
|
|
JAVA_CMD := java
|
|
|
|
|
endif
|
|
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
ifeq ($(JAVAC_CMD),)
|
|
|
|
|
ifneq ($(JAVA_HOME),)
|
|
|
|
|
JAVAC_CMD := $(JAVA_HOME)/bin/javac
|
|
|
|
|
else
|
|
|
|
|
JAVAC_CMD := javac
|
|
|
|
|
endif
|
|
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
ifeq ($(JAVADOC_CMD),)
|
|
|
|
|
ifneq ($(JAVA_HOME),)
|
|
|
|
|
JAVADOC_CMD := $(JAVA_HOME)/bin/javadoc
|
|
|
|
|
else
|
|
|
|
|
JAVADOC_CMD := javadoc
|
|
|
|
|
endif
|
|
|
|
|
endif
|
|
|
|
|
|
2023-10-17 17:04:35 +00:00
|
|
|
MAVEN_CMD := mvn
|
|
|
|
|
|
2022-02-17 21:28:09 +00:00
|
|
|
# Look for the Java version (1.6->6, 1.7->7, 1.8->8, 11.0->11, 13.0->13, 15.0->15 etc..)
|
|
|
|
|
JAVAC_VERSION := $(shell $(JAVAC_CMD) -version 2>&1)
|
|
|
|
|
JAVAC_MAJOR_VERSION := $(word 2,$(subst ., ,$(JAVAC_VERSION)))
|
|
|
|
|
ifeq ($(JAVAC_MAJOR_VERSION),1)
|
|
|
|
|
JAVAC_MAJOR_VERSION := $(word 3,$(subst ., ,$(JAVAC_VERSION)))
|
|
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
# Test whether the version we see meets our minimum
|
|
|
|
|
MIN_JAVAC_MAJOR_VERSION := 8
|
|
|
|
|
JAVAC_VERSION_GE_MIN := $(shell [ $(JAVAC_MAJOR_VERSION) -ge $(MIN_JAVAC_MAJOR_VERSION) ] > /dev/null 2>&1 && echo true)
|
|
|
|
|
|
2017-02-28 00:26:12 +00:00
|
|
|
# Set the default JAVA_ARGS to "" for DEBUG_LEVEL=0
|
2020-03-12 19:22:03 +00:00
|
|
|
JAVA_ARGS ?=
|
2017-02-28 00:26:12 +00:00
|
|
|
|
2020-03-12 19:22:03 +00:00
|
|
|
JAVAC_ARGS ?=
|
2017-07-12 19:04:55 +00:00
|
|
|
|
2022-02-18 03:38:23 +00:00
|
|
|
# Read plugin configuration
|
|
|
|
|
PLUGIN_PATH = ../plugin
|
|
|
|
|
ROCKSDB_PLUGIN_MKS = $(foreach plugin, $(ROCKSDB_PLUGINS), $(PLUGIN_PATH)/$(plugin)/*.mk)
|
|
|
|
|
include $(ROCKSDB_PLUGIN_MKS)
|
|
|
|
|
|
|
|
|
|
# Add paths to Java sources in plugins
|
|
|
|
|
ROCKSDB_PLUGIN_JAVA_ROOTS = $(foreach plugin, $(ROCKSDB_PLUGINS), $(PLUGIN_PATH)/$(plugin)/java)
|
|
|
|
|
PLUGIN_SOURCES = $(foreach root, $(ROCKSDB_PLUGIN_JAVA_ROOTS), $(foreach pkg, org/rocksdb/util org/rocksdb, $(root)/$(MAIN_SRC)/$(pkg)/*.java))
|
|
|
|
|
CORE_SOURCES = $(foreach pkg, org/rocksdb/util org/rocksdb, $(MAIN_SRC)/$(pkg)/*.java)
|
|
|
|
|
SOURCES = $(wildcard $(CORE_SOURCES) $(PLUGIN_SOURCES))
|
|
|
|
|
PLUGIN_TEST_SOURCES = $(foreach root, $(ROCKSDB_PLUGIN_JAVA_ROOTS), $(foreach pkg, org/rocksdb/test org/rocksdb/util org/rocksdb, $(root)/$(TEST_SRC)/$(pkg)/*.java))
|
|
|
|
|
CORE_TEST_SOURCES = $(foreach pkg, org/rocksdb/test org/rocksdb/util org/rocksdb, $(TEST_SRC)/$(pkg)/*.java)
|
|
|
|
|
TEST_SOURCES = $(wildcard $(CORE_TEST_SOURCES) $(PLUGIN_TEST_SOURCES))
|
|
|
|
|
|
|
|
|
|
# Configure the plugin tests and java classes
|
|
|
|
|
ROCKSDB_PLUGIN_NATIVE_JAVA_CLASSES = $(foreach plugin, $(ROCKSDB_PLUGINS), $(foreach class, $($(plugin)_NATIVE_JAVA_CLASSES), $(class)))
|
|
|
|
|
NATIVE_JAVA_CLASSES = $(NATIVE_JAVA_CLASSES) $(ROCKSDB_PLUGIN_NATIVE_JAVA_CLASSES)
|
|
|
|
|
ROCKSDB_PLUGIN_JAVA_TESTS = $(foreach plugin, $(ROCKSDB_PLUGINS), $(foreach testclass, $($(plugin)_JAVA_TESTS), $(testclass)))
|
|
|
|
|
ALL_JAVA_TESTS = $(JAVA_TESTS) $(ROCKSDB_PLUGIN_JAVA_TESTS)
|
|
|
|
|
|
2017-02-28 00:26:12 +00:00
|
|
|
# When debugging add -Xcheck:jni to the java args
|
|
|
|
|
ifneq ($(DEBUG_LEVEL),0)
|
2020-03-12 19:22:03 +00:00
|
|
|
JAVA_ARGS += -ea -Xcheck:jni
|
|
|
|
|
JAVAC_ARGS += -Xlint:deprecation -Xlint:unchecked
|
2017-02-28 00:26:12 +00:00
|
|
|
endif
|
|
|
|
|
|
2020-03-13 20:36:45 +00:00
|
|
|
# Using a Facebook AWS account for S3 storage. (maven.org has a history
|
|
|
|
|
# of failing in Travis builds.)
|
|
|
|
|
DEPS_URL?=https://rocksdb-deps.s3-us-west-2.amazonaws.com/jars
|
2017-04-22 03:41:37 +00:00
|
|
|
|
2022-02-17 21:28:09 +00:00
|
|
|
java-version:
|
|
|
|
|
ifneq ($(JAVAC_VERSION_GE_MIN),true)
|
|
|
|
|
echo 'Java version is $(JAVAC_VERSION), minimum required version is $(MIN_JAVAC_MAJOR_VERSION)'
|
|
|
|
|
exit 1
|
|
|
|
|
endif
|
|
|
|
|
|
2020-01-29 16:00:16 +00:00
|
|
|
clean: clean-not-downloaded clean-downloaded
|
|
|
|
|
|
|
|
|
|
clean-not-downloaded:
|
|
|
|
|
$(AM_V_at)rm -rf $(NATIVE_INCLUDE)
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_at)rm -rf $(OUTPUT)
|
|
|
|
|
$(AM_V_at)rm -rf $(BENCHMARK_OUTPUT)
|
|
|
|
|
$(AM_V_at)rm -rf $(SAMPLES_OUTPUT)
|
2014-11-19 20:15:01 +00:00
|
|
|
|
2020-01-29 16:00:16 +00:00
|
|
|
clean-downloaded:
|
|
|
|
|
$(AM_V_at)rm -rf $(JAVA_TEST_LIBDIR)
|
|
|
|
|
|
Add a jni library for rocksdb which supports Open, Get, Put, and Close.
Summary:
This diff contains a simple jni library for rocksdb which supports open, get,
put and closeusing default options (including Options, ReadOptions, and
WriteOptions.) In the usual case, Java developers can use the c++ rocksdb
library in the way similar to the following:
RocksDB db = RocksDB.open(path_to_db);
...
db.put("hello".getBytes(), "world".getBytes();
byte[] value = db.get("hello".getBytes());
...
db.close();
Specifically, this diff has the following major classes:
* RocksDB: a Java wrapper class which forwards the operations
from the java side to c++ rocksdb library.
* RocksDBException: ncapsulates the error of an operation.
This exception type is used to describe an internal error from
the c++ rocksdb library.
This diff also include a simple java sample code calling c++ rocksdb library.
To build the rocksdb jni library, simply run make jni, and make jtest will try to
build and run the sample code.
Note that if the rocksdb is not built with the default glibc that Java uses,
java will try to load the wrong glibc during the run time. As a result,
the sample code might not work properly during the run time.
Test Plan:
* make jni
* make jtest
Reviewers: haobo, dhruba, sdong, igor, ljin
Reviewed By: dhruba
CC: leveldb, xjin
Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 21:19:21 +00:00
|
|
|
|
2016-09-16 17:54:31 +00:00
|
|
|
javadocs: java
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_GEN)mkdir -p $(JAVADOC)
|
2021-12-22 20:56:45 +00:00
|
|
|
$(AM_V_at)$(JAVADOC_CMD) -d $(JAVADOC) -sourcepath $(MAIN_SRC) -subpackages org
|
2014-10-02 20:57:54 +00:00
|
|
|
|
2015-01-31 19:43:09 +00:00
|
|
|
javalib: java java_test javadocs
|
2014-11-19 20:21:21 +00:00
|
|
|
|
2022-02-17 21:28:09 +00:00
|
|
|
java: java-version
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_GEN)mkdir -p $(MAIN_CLASSES)
|
2022-02-18 03:38:23 +00:00
|
|
|
$(AM_V_at) $(JAVAC_CMD) $(JAVAC_ARGS) -h $(NATIVE_INCLUDE) -d $(MAIN_CLASSES) $(SOURCES)
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_at)@cp ../HISTORY.md ./HISTORY-CPP.md
|
|
|
|
|
$(AM_V_at)@rm -f ./HISTORY-CPP.md
|
Add a jni library for rocksdb which supports Open, Get, Put, and Close.
Summary:
This diff contains a simple jni library for rocksdb which supports open, get,
put and closeusing default options (including Options, ReadOptions, and
WriteOptions.) In the usual case, Java developers can use the c++ rocksdb
library in the way similar to the following:
RocksDB db = RocksDB.open(path_to_db);
...
db.put("hello".getBytes(), "world".getBytes();
byte[] value = db.get("hello".getBytes());
...
db.close();
Specifically, this diff has the following major classes:
* RocksDB: a Java wrapper class which forwards the operations
from the java side to c++ rocksdb library.
* RocksDBException: ncapsulates the error of an operation.
This exception type is used to describe an internal error from
the c++ rocksdb library.
This diff also include a simple java sample code calling c++ rocksdb library.
To build the rocksdb jni library, simply run make jni, and make jtest will try to
build and run the sample code.
Note that if the rocksdb is not built with the default glibc that Java uses,
java will try to load the wrong glibc during the run time. As a result,
the sample code might not work properly during the run time.
Test Plan:
* make jni
* make jtest
Reviewers: haobo, dhruba, sdong, igor, ljin
Reviewed By: dhruba
CC: leveldb, xjin
Differential Revision: https://reviews.facebook.net/D17109
2014-03-28 21:19:21 +00:00
|
|
|
|
2014-04-09 07:48:20 +00:00
|
|
|
sample: java
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_GEN)mkdir -p $(SAMPLES_MAIN_CLASSES)
|
2021-12-22 20:56:45 +00:00
|
|
|
$(AM_V_at)$(JAVAC_CMD) $(JAVAC_ARGS) -cp $(MAIN_CLASSES) -d $(SAMPLES_MAIN_CLASSES) $(SAMPLES_MAIN_SRC)/RocksDBSample.java
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
|
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni_not_found
|
2021-12-22 20:56:45 +00:00
|
|
|
$(JAVA_CMD) $(JAVA_ARGS) -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) RocksDBSample /tmp/rocksdbjni
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
|
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni_not_found
|
2014-04-02 20:14:55 +00:00
|
|
|
|
2014-11-08 17:58:35 +00:00
|
|
|
column_family_sample: java
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_GEN)mkdir -p $(SAMPLES_MAIN_CLASSES)
|
2021-12-22 20:56:45 +00:00
|
|
|
$(AM_V_at)$(JAVAC_CMD) $(JAVAC_ARGS) -cp $(MAIN_CLASSES) -d $(SAMPLES_MAIN_CLASSES) $(SAMPLES_MAIN_SRC)/RocksDBColumnFamilySample.java
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
2021-12-22 20:56:45 +00:00
|
|
|
$(JAVA_CMD) $(JAVA_ARGS) -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) RocksDBColumnFamilySample /tmp/rocksdbjni
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
2014-11-08 17:58:35 +00:00
|
|
|
|
2018-03-02 18:22:38 +00:00
|
|
|
transaction_sample: java
|
|
|
|
|
$(AM_V_GEN)mkdir -p $(SAMPLES_MAIN_CLASSES)
|
2021-12-22 20:56:45 +00:00
|
|
|
$(AM_V_at)$(JAVAC_CMD) -cp $(MAIN_CLASSES) -d $(SAMPLES_MAIN_CLASSES) $(SAMPLES_MAIN_SRC)/TransactionSample.java
|
2018-03-02 18:22:38 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
2021-12-22 20:56:45 +00:00
|
|
|
$(JAVA_CMD) -ea -Xcheck:jni -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) TransactionSample /tmp/rocksdbjni
|
2018-03-02 18:22:38 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
|
|
|
|
|
|
|
|
|
optimistic_transaction_sample: java
|
|
|
|
|
$(AM_V_GEN)mkdir -p $(SAMPLES_MAIN_CLASSES)
|
2021-12-22 20:56:45 +00:00
|
|
|
$(AM_V_at)$(JAVAC_CMD) -cp $(MAIN_CLASSES) -d $(SAMPLES_MAIN_CLASSES) $(SAMPLES_MAIN_SRC)/OptimisticTransactionSample.java
|
2018-03-02 18:22:38 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
2021-12-22 20:56:45 +00:00
|
|
|
$(JAVA_CMD) -ea -Xcheck:jni -Djava.library.path=target -cp $(MAIN_CLASSES):$(SAMPLES_MAIN_CLASSES) OptimisticTransactionSample /tmp/rocksdbjni
|
2018-03-02 18:22:38 +00:00
|
|
|
$(AM_V_at)@rm -rf /tmp/rocksdbjni
|
|
|
|
|
|
2020-12-16 23:58:56 +00:00
|
|
|
$(JAVA_TEST_LIBDIR):
|
|
|
|
|
mkdir -p "$(JAVA_TEST_LIBDIR)"
|
|
|
|
|
|
|
|
|
|
$(JAVA_JUNIT_JAR_PATH): $(JAVA_TEST_LIBDIR)
|
|
|
|
|
ifneq (,$(wildcard $(MVN_LOCAL)/junit/junit/$(JAVA_JUNIT_VER)/$(JAVA_JUNIT_JAR)))
|
|
|
|
|
cp -v $(MVN_LOCAL)/junit/junit/$(JAVA_JUNIT_VER)/$(JAVA_JUNIT_JAR) $(JAVA_TEST_LIBDIR)
|
|
|
|
|
else
|
|
|
|
|
curl --fail --insecure --output $(JAVA_JUNIT_JAR_PATH) --location $(DEPS_URL)/$(JAVA_JUNIT_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_JUNIT_SHA256_ACTUAL=`$(SHA256_CMD) $(JAVA_JUNIT_JAR_PATH) | cut -d ' ' -f 1`; \
|
|
|
|
|
if [ "$(JAVA_JUNIT_SHA256)" != "$$JAVA_JUNIT_SHA256_ACTUAL" ]; then \
|
|
|
|
|
echo $(JAVA_JUNIT_JAR_PATH) checksum mismatch, expected=\"$(JAVA_JUNIT_SHA256)\" actual=\"$$JAVA_JUNIT_SHA256_ACTUAL\"; \
|
|
|
|
|
exit 1; \
|
|
|
|
|
fi
|
2020-12-16 23:58:56 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
$(JAVA_HAMCREST_JAR_PATH): $(JAVA_TEST_LIBDIR)
|
2021-01-13 23:59:36 +00:00
|
|
|
ifneq (,$(wildcard $(MVN_LOCAL)/org/hamcrest/hamcrest/$(JAVA_HAMCREST_VER)/$(JAVA_HAMCREST_JAR)))
|
|
|
|
|
cp -v $(MVN_LOCAL)/org/hamcrest/hamcrest/$(JAVA_HAMCREST_VER)/$(JAVA_HAMCREST_JAR) $(JAVA_TEST_LIBDIR)
|
2020-12-16 23:58:56 +00:00
|
|
|
else
|
|
|
|
|
curl --fail --insecure --output $(JAVA_HAMCREST_JAR_PATH) --location $(DEPS_URL)/$(JAVA_HAMCREST_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_HAMCREST_SHA256_ACTUAL=`$(SHA256_CMD) $(JAVA_HAMCREST_JAR_PATH) | cut -d ' ' -f 1`; \
|
|
|
|
|
if [ "$(JAVA_HAMCREST_SHA256)" != "$$JAVA_HAMCREST_SHA256_ACTUAL" ]; then \
|
|
|
|
|
echo $(JAVA_HAMCREST_JAR_PATH) checksum mismatch, expected=\"$(JAVA_HAMCREST_SHA256)\" actual=\"$$JAVA_HAMCREST_SHA256_ACTUAL\"; \
|
|
|
|
|
exit 1; \
|
|
|
|
|
fi
|
2020-12-16 23:58:56 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
$(JAVA_MOCKITO_JAR_PATH): $(JAVA_TEST_LIBDIR)
|
|
|
|
|
ifneq (,$(wildcard $(MVN_LOCAL)/org/mockito/mockito-all/$(JAVA_MOCKITO_VER)/$(JAVA_MOCKITO_JAR)))
|
|
|
|
|
cp -v $(MVN_LOCAL)/org/mockito/mockito-all/$(JAVA_MOCKITO_VER)/$(JAVA_MOCKITO_JAR) $(JAVA_TEST_LIBDIR)
|
|
|
|
|
else
|
|
|
|
|
curl --fail --insecure --output "$(JAVA_MOCKITO_JAR_PATH)" --location $(DEPS_URL)/$(JAVA_MOCKITO_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_MOCKITO_SHA256_ACTUAL=`$(SHA256_CMD) $(JAVA_MOCKITO_JAR_PATH) | cut -d ' ' -f 1`; \
|
|
|
|
|
if [ "$(JAVA_MOCKITO_SHA256)" != "$$JAVA_MOCKITO_SHA256_ACTUAL" ]; then \
|
|
|
|
|
echo $(JAVA_MOCKITO_JAR_PATH) checksum mismatch, expected=\"$(JAVA_MOCKITO_SHA256)\" actual=\"$$JAVA_MOCKITO_SHA256_ACTUAL\"; \
|
|
|
|
|
exit 1; \
|
|
|
|
|
fi
|
2020-12-16 23:58:56 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
$(JAVA_CGLIB_JAR_PATH): $(JAVA_TEST_LIBDIR)
|
|
|
|
|
ifneq (,$(wildcard $(MVN_LOCAL)/cglib/cglib/$(JAVA_CGLIB_VER)/$(JAVA_CGLIB_JAR)))
|
|
|
|
|
cp -v $(MVN_LOCAL)/cglib/cglib/$(JAVA_CGLIB_VER)/$(JAVA_CGLIB_JAR) $(JAVA_TEST_LIBDIR)
|
|
|
|
|
else
|
|
|
|
|
curl --fail --insecure --output "$(JAVA_CGLIB_JAR_PATH)" --location $(DEPS_URL)/$(JAVA_CGLIB_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_CGLIB_SHA256_ACTUAL=`$(SHA256_CMD) $(JAVA_CGLIB_JAR_PATH) | cut -d ' ' -f 1`; \
|
|
|
|
|
if [ "$(JAVA_CGLIB_SHA256)" != "$$JAVA_CGLIB_SHA256_ACTUAL" ]; then \
|
|
|
|
|
echo $(JAVA_CGLIB_JAR_PATH) checksum mismatch, expected=\"$(JAVA_CGLIB_SHA256)\" actual=\"$$JAVA_CGLIB_SHA256_ACTUAL\"; \
|
|
|
|
|
exit 1; \
|
|
|
|
|
fi
|
2020-12-16 23:58:56 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
$(JAVA_ASSERTJ_JAR_PATH): $(JAVA_TEST_LIBDIR)
|
|
|
|
|
ifneq (,$(wildcard $(MVN_LOCAL)/org/assertj/assertj-core/$(JAVA_ASSERTJ_VER)/$(JAVA_ASSERTJ_JAR)))
|
|
|
|
|
cp -v $(MVN_LOCAL)/org/assertj/assertj-core/$(JAVA_ASSERTJ_VER)/$(JAVA_ASSERTJ_JAR) $(JAVA_TEST_LIBDIR)
|
|
|
|
|
else
|
|
|
|
|
curl --fail --insecure --output "$(JAVA_ASSERTJ_JAR_PATH)" --location $(DEPS_URL)/$(JAVA_ASSERTJ_JAR)
|
2021-01-13 23:59:36 +00:00
|
|
|
JAVA_ASSERTJ_SHA256_ACTUAL=`$(SHA256_CMD) $(JAVA_ASSERTJ_JAR_PATH) | cut -d ' ' -f 1`; \
|
|
|
|
|
if [ "$(JAVA_ASSERTJ_SHA256)" != "$$JAVA_ASSERTJ_SHA256_ACTUAL" ]; then \
|
|
|
|
|
echo $(JAVA_ASSERTJ_JAR_PATH) checksum mismatch, expected=\"$(JAVA_ASSERTJ_SHA256)\" actual=\"$$JAVA_ASSERTJ_SHA256_ACTUAL\"; \
|
|
|
|
|
exit 1; \
|
|
|
|
|
fi
|
2020-12-16 23:58:56 +00:00
|
|
|
endif
|
|
|
|
|
|
|
|
|
|
resolve_test_deps: $(JAVA_JUNIT_JAR_PATH) $(JAVA_HAMCREST_JAR_PATH) $(JAVA_MOCKITO_JAR_PATH) $(JAVA_CGLIB_JAR_PATH) $(JAVA_ASSERTJ_JAR_PATH)
|
2014-11-02 00:08:41 +00:00
|
|
|
|
2016-09-16 17:54:31 +00:00
|
|
|
java_test: java resolve_test_deps
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_GEN)mkdir -p $(TEST_CLASSES)
|
2022-02-18 03:38:23 +00:00
|
|
|
$(AM_V_at) $(JAVAC_CMD) $(JAVAC_ARGS) -cp $(MAIN_CLASSES):$(JAVA_TESTCLASSPATH) -h $(NATIVE_INCLUDE) -d $(TEST_CLASSES)\
|
|
|
|
|
$(TEST_SOURCES)
|
2015-01-31 19:43:09 +00:00
|
|
|
|
2022-05-23 16:56:40 +00:00
|
|
|
test: java java_test
|
|
|
|
|
$(MAKE) run_test
|
2016-07-29 19:55:54 +00:00
|
|
|
|
|
|
|
|
run_test:
|
2022-02-18 03:38:23 +00:00
|
|
|
$(JAVA_CMD) $(JAVA_ARGS) -Djava.library.path=target -cp "$(MAIN_CLASSES):$(TEST_CLASSES):$(JAVA_TESTCLASSPATH):target/*" org.rocksdb.test.RocksJunitRunner $(ALL_JAVA_TESTS)
|
2023-10-18 18:01:04 +00:00
|
|
|
$(JAVA_CMD) $(JAVA_ARGS) -Djava.library.path=target -cp "$(MAIN_CLASSES):$(TEST_CLASSES):$(JAVA_TESTCLASSPATH):target/*" org.rocksdb.test.RocksJunitRunner org.rocksdb.StatisticsTest
|
2022-02-18 03:38:23 +00:00
|
|
|
|
|
|
|
|
run_plugin_test:
|
|
|
|
|
$(JAVA_CMD) $(JAVA_ARGS) -Djava.library.path=target -cp "$(MAIN_CLASSES):$(TEST_CLASSES):$(JAVA_TESTCLASSPATH):target/*" org.rocksdb.test.RocksJunitRunner $(ROCKSDB_PLUGIN_JAVA_TESTS)
|
2014-04-09 07:48:20 +00:00
|
|
|
|
|
|
|
|
db_bench: java
|
2015-09-16 19:38:17 +00:00
|
|
|
$(AM_V_GEN)mkdir -p $(BENCHMARK_MAIN_CLASSES)
|
2021-12-22 20:56:45 +00:00
|
|
|
$(AM_V_at)$(JAVAC_CMD) $(JAVAC_ARGS) -cp $(MAIN_CLASSES) -d $(BENCHMARK_MAIN_CLASSES) $(BENCHMARK_MAIN_SRC)/org/rocksdb/benchmark/*.java
|
2023-10-17 17:04:35 +00:00
|
|
|
|
|
|
|
|
pmd:
|
|
|
|
|
$(MAVEN_CMD) pmd:pmd pmd:cpd pmd:check
|