dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Improve RocksJava Comparator (#6252) 

Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.

**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.

Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).

In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.

In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.

 ---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.

With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).

These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.

 ---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.

```
ComparatorBenchmarks.put                                                native_bytewise  thrpt   25  124483.795 ± 2032.443  ops/s
ComparatorBenchmarks.put                                        native_reverse_bytewise  thrpt   25  114414.536 ± 3486.156  ops/s
ComparatorBenchmarks.put              java_bytewise_non-direct_reused-64_adaptive-mutex  thrpt   25   17228.250 ± 1288.546  ops/s
ComparatorBenchmarks.put          java_bytewise_non-direct_reused-64_non-adaptive-mutex  thrpt   25   16035.865 ± 1248.099  ops/s
ComparatorBenchmarks.put                java_bytewise_non-direct_reused-64_thread-local  thrpt   25   21571.500 ±  871.521  ops/s
ComparatorBenchmarks.put                  java_bytewise_direct_reused-64_adaptive-mutex  thrpt   25   23613.773 ± 8465.660  ops/s
ComparatorBenchmarks.put              java_bytewise_direct_reused-64_non-adaptive-mutex  thrpt   25   16768.172 ± 5618.489  ops/s
ComparatorBenchmarks.put                    java_bytewise_direct_reused-64_thread-local  thrpt   25   23921.164 ± 8734.742  ops/s
ComparatorBenchmarks.put                              java_bytewise_non-direct_no-reuse  thrpt   25   17899.684 ±  839.679  ops/s
ComparatorBenchmarks.put                                  java_bytewise_direct_no-reuse  thrpt   25   22148.316 ± 1215.527  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_non-direct_reused-64_adaptive-mutex  thrpt   25   11311.126 ±  820.602  ops/s
ComparatorBenchmarks.put  java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex  thrpt   25   11421.311 ±  807.210  ops/s
ComparatorBenchmarks.put        java_reverse_bytewise_non-direct_reused-64_thread-local  thrpt   25   11554.005 ±  960.556  ops/s
ComparatorBenchmarks.put          java_reverse_bytewise_direct_reused-64_adaptive-mutex  thrpt   25   22960.523 ± 1673.421  ops/s
ComparatorBenchmarks.put      java_reverse_bytewise_direct_reused-64_non-adaptive-mutex  thrpt   25   18293.317 ± 1434.601  ops/s
ComparatorBenchmarks.put            java_reverse_bytewise_direct_reused-64_thread-local  thrpt   25   24479.361 ± 2157.306  ops/s
ComparatorBenchmarks.put                      java_reverse_bytewise_non-direct_no-reuse  thrpt   25    7942.286 ±  626.170  ops/s
ComparatorBenchmarks.put                          java_reverse_bytewise_direct_no-reuse  thrpt   25   11781.955 ± 1019.843  ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252

Differential Revision: D19331064

Pulled By: pdillinger

fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
This commit is contained in:
Adam Retter 2020-02-03 12:28:25 -08:00 committed by Facebook Github Bot
parent 800d24ddc5
commit 7242dae7fe
44 changed files with 2726 additions and 1151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
HISTORY.md
View File

@ -1,5 +1,8 @@
# Rocksdb Change Log
## Unreleased
### Public API Change
* Major breaking changes to Java comparators, toward standardizing on ByteBuffer for performant, locale-neutral operations on keys (#6252).
### Bug Fixes
* Fix incorrect results while block-based table uses kHashSearch, together with Prev()/SeekForPrev().
* Fix a bug that prevents opening a DB after two consecutive crash with TransactionDB, where the first crash recovers from a corrupted WAL with kPointInTimeRecovery but the second cannot.

8
java/CMakeLists.txt
View File

@ -122,7 +122,6 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/CompactRangeOptions.java
src/main/java/org/rocksdb/CompactionStopStyle.java
src/main/java/org/rocksdb/CompactionStyle.java
src/main/java/org/rocksdb/Comparator.java
src/main/java/org/rocksdb/ComparatorOptions.java
src/main/java/org/rocksdb/ComparatorType.java
src/main/java/org/rocksdb/CompressionOptions.java
@ -131,7 +130,6 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/DBOptionsInterface.java
src/main/java/org/rocksdb/DBOptions.java
src/main/java/org/rocksdb/DbPath.java
src/main/java/org/rocksdb/DirectComparator.java
src/main/java/org/rocksdb/DirectSlice.java
src/main/java/org/rocksdb/EncodingType.java
src/main/java/org/rocksdb/Env.java
@ -181,6 +179,7 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/ReadTier.java
src/main/java/org/rocksdb/RemoveEmptyValueCompactionFilter.java
src/main/java/org/rocksdb/RestoreOptions.java
src/main/java/org/rocksdb/ReusedSynchronisationType.java
src/main/java/org/rocksdb/RocksCallbackObject.java
src/main/java/org/rocksdb/RocksDBException.java
src/main/java/org/rocksdb/RocksDB.java
@ -236,9 +235,10 @@ set(JAVA_MAIN_CLASSES
src/main/java/org/rocksdb/WriteBatchWithIndex.java
src/main/java/org/rocksdb/WriteOptions.java
src/main/java/org/rocksdb/WriteBufferManager.java
src/main/java/org/rocksdb/util/ByteUtil.java
src/main/java/org/rocksdb/util/BytewiseComparator.java
src/main/java/org/rocksdb/util/DirectBytewiseComparator.java
src/main/java/org/rocksdb/util/Environment.java
src/main/java/org/rocksdb/util/IntComparator.java
src/main/java/org/rocksdb/util/ReverseBytewiseComparator.java
src/main/java/org/rocksdb/util/SizeUnit.java
src/main/java/org/rocksdb/UInt64AddOperator.java
@ -400,11 +400,9 @@ if(${CMAKE_VERSION} VERSION_LESS "3.11.4" OR (${Java_VERSION_MINOR} STREQUAL "7"
org.rocksdb.CompactionOptionsFIFO
org.rocksdb.CompactionOptionsUniversal
org.rocksdb.CompactRangeOptions
org.rocksdb.Comparator
org.rocksdb.ComparatorOptions
org.rocksdb.CompressionOptions
org.rocksdb.DBOptions
org.rocksdb.DirectComparator
org.rocksdb.DirectSlice
org.rocksdb.Env
org.rocksdb.EnvOptions

18
java/Makefile
View File

@ -1,5 +1,7 @@
NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
NATIVE_JAVA_CLASSES = \
org.rocksdb.AbstractCompactionFilter\
org.rocksdb.AbstractCompactionFilterFactory\
org.rocksdb.AbstractComparator\
org.rocksdb.AbstractSlice\
org.rocksdb.AbstractTableFilter\
org.rocksdb.AbstractTraceWriter\
@ -21,11 +23,9 @@ NATIVE_JAVA_CLASSES = org.rocksdb.AbstractCompactionFilter\
org.rocksdb.CompactionOptionsFIFO\
org.rocksdb.CompactionOptionsUniversal\
org.rocksdb.CompactRangeOptions\
org.rocksdb.Comparator\
org.rocksdb.ComparatorOptions\
org.rocksdb.CompressionOptions\
org.rocksdb.DBOptions\
org.rocksdb.DirectComparator\
org.rocksdb.DirectSlice\
org.rocksdb.Env\
org.rocksdb.EnvOptions\
@ -98,10 +98,13 @@ ifeq ($(PLATFORM), OS_MACOSX)
ROCKSDB_JAR = rocksdbjni-$(ROCKSDB_MAJOR).$(ROCKSDB_MINOR).$(ROCKSDB_PATCH)-osx.jar
endif
JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
JAVA_TESTS = \
org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.BackupEngineTest\
org.rocksdb.BlockBasedTableConfigTest\
org.rocksdb.BuiltinComparatorTest\
org.rocksdb.util.BytewiseComparatorTest\
org.rocksdb.util.BytewiseComparatorIntTest\
org.rocksdb.CheckPointTest\
org.rocksdb.ClockCacheTest\
org.rocksdb.ColumnFamilyOptionsTest\
@ -115,16 +118,16 @@ JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.CompactionPriorityTest\
org.rocksdb.CompactionStopStyleTest\
org.rocksdb.ComparatorOptionsTest\
org.rocksdb.ComparatorTest\
org.rocksdb.CompressionOptionsTest\
org.rocksdb.CompressionTypesTest\
org.rocksdb.DBOptionsTest\
org.rocksdb.DirectComparatorTest\
org.rocksdb.DirectSliceTest\
org.rocksdb.util.EnvironmentTest\
org.rocksdb.EnvOptionsTest\
org.rocksdb.HdfsEnvTest\
org.rocksdb.IngestExternalFileOptionsTest\
org.rocksdb.util.EnvironmentTest\
org.rocksdb.util.IntComparatorTest\
org.rocksdb.util.JNIComparatorTest\
org.rocksdb.FilterTest\
org.rocksdb.FlushTest\
org.rocksdb.InfoLogLevelTest\
@ -148,6 +151,7 @@ JAVA_TESTS = org.rocksdb.BackupableDBOptionsTest\
org.rocksdb.RateLimiterTest\
org.rocksdb.ReadOnlyTest\
org.rocksdb.ReadOptionsTest\
org.rocksdb.util.ReverseBytewiseComparatorIntTest\
org.rocksdb.RocksDBTest\
org.rocksdb.RocksDBExceptionTest\
org.rocksdb.DefaultEnvTest\

2
java/jmh/pom.xml
View File

@ -50,7 +50,7 @@
<dependency>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
<version>6.4.6</version>
<version>6.6.0-SNAPSHOT</version>
</dependency>
<dependency>

93
java/jmh/src/main/java/org/rocksdb/jmh/ComparatorBenchmarks.java
View File

@ -9,7 +9,6 @@ package org.rocksdb.jmh;
import org.openjdk.jmh.annotations.*;
import org.rocksdb.*;
import org.rocksdb.util.BytewiseComparator;
import org.rocksdb.util.DirectBytewiseComparator;
import org.rocksdb.util.FileUtils;
import org.rocksdb.util.ReverseBytewiseComparator;
@ -26,13 +25,24 @@ public class ComparatorBenchmarks {
@Param({
"native_bytewise",
"native_reverse_bytewise",
"java_bytewise_adaptive_mutex",
"java_bytewise_non-adaptive_mutex",
"java_reverse_bytewise_adaptive_mutex",
"java_reverse_bytewise_non-adaptive_mutex",
"java_direct_bytewise_adaptive_mutex",
"java_direct_bytewise_non-adaptive_mutex"
"java_bytewise_non-direct_reused-64_adaptive-mutex",
"java_bytewise_non-direct_reused-64_non-adaptive-mutex",
"java_bytewise_non-direct_reused-64_thread-local",
"java_bytewise_direct_reused-64_adaptive-mutex",
"java_bytewise_direct_reused-64_non-adaptive-mutex",
"java_bytewise_direct_reused-64_thread-local",
"java_bytewise_non-direct_no-reuse",
"java_bytewise_direct_no-reuse",
"java_reverse_bytewise_non-direct_reused-64_adaptive-mutex",
"java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex",
"java_reverse_bytewise_non-direct_reused-64_thread-local",
"java_reverse_bytewise_direct_reused-64_adaptive-mutex",
"java_reverse_bytewise_direct_reused-64_non-adaptive-mutex",
"java_reverse_bytewise_direct_reused-64_thread-local",
"java_reverse_bytewise_non-direct_no-reuse",
"java_reverse_bytewise_direct_no-reuse"
})
public String comparatorName;
@ -50,42 +60,49 @@ public class ComparatorBenchmarks {
options = new Options()
.setCreateIfMissing(true);
if (comparatorName == null || "native_bytewise".equals(comparatorName)) {
if ("native_bytewise".equals(comparatorName)) {
options.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
} else if ("native_reverse_bytewise".equals(comparatorName)) {
options.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
} else if ("java_bytewise_adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(true);
comparator = new BytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_bytewise_non-adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(false);
comparator = new BytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_reverse_bytewise_adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(true);
comparator = new ReverseBytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_reverse_bytewise_non-adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(false);
comparator = new ReverseBytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_direct_bytewise_adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(true);
comparator = new DirectBytewiseComparator(comparatorOptions);
options.setComparator(comparator);
} else if ("java_direct_bytewise_non-adaptive_mutex".equals(comparatorName)) {
comparatorOptions = new ComparatorOptions()
.setUseAdaptiveMutex(false);
comparator = new DirectBytewiseComparator(comparatorOptions);
} else if (comparatorName.startsWith("java_")) {
comparatorOptions = new ComparatorOptions();
if (comparatorName.indexOf("non-direct") > -1) {
comparatorOptions.setUseDirectBuffer(false);
} else if (comparatorName.indexOf("direct") > -1) {
comparatorOptions.setUseDirectBuffer(true);
}
if (comparatorName.indexOf("no-reuse") > -1) {
comparatorOptions.setMaxReusedBufferSize(-1);
} else if (comparatorName.indexOf("_reused-") > -1) {
final int idx = comparatorName.indexOf("_reused-");
String s = comparatorName.substring(idx + 8);
s = s.substring(0, s.indexOf('_'));
comparatorOptions.setMaxReusedBufferSize(Integer.parseInt(s));
}
if (comparatorName.indexOf("non-adaptive-mutex") > -1) {
comparatorOptions.setReusedSynchronisationType(ReusedSynchronisationType.MUTEX);
} else if (comparatorName.indexOf("adaptive-mutex") > -1) {
comparatorOptions.setReusedSynchronisationType(ReusedSynchronisationType.ADAPTIVE_MUTEX);
} else if (comparatorName.indexOf("thread-local") > -1) {
comparatorOptions.setReusedSynchronisationType(ReusedSynchronisationType.THREAD_LOCAL);
}
if (comparatorName.startsWith("java_bytewise")) {
comparator = new BytewiseComparator(comparatorOptions);
} else if (comparatorName.startsWith("java_reverse_bytewise")) {
comparator = new ReverseBytewiseComparator(comparatorOptions);
}
options.setComparator(comparator);
} else {
throw new IllegalArgumentException("Unknown comparator name: " + comparatorName);
throw new IllegalArgumentException("Unknown comparatorName: " + comparatorName);
}
db = RocksDB.open(options, dbDir.toAbsolutePath().toString());

38
java/rocksjni/comparator.cc
View File

@ -12,42 +12,33 @@
#include <functional>
#include <string>
#include "include/org_rocksdb_Comparator.h"
#include "include/org_rocksdb_DirectComparator.h"
#include "include/org_rocksdb_AbstractComparator.h"
#include "include/org_rocksdb_NativeComparatorWrapper.h"
#include "rocksjni/comparatorjnicallback.h"
#include "rocksjni/portal.h"
// <editor-fold desc="org.rocksdb.Comparator>
/*
* Class: org_rocksdb_Comparator
* Method: createNewComparator0
* Signature: ()J
* Class: org_rocksdb_AbstractComparator
* Method: createNewComparator
* Signature: (J)J
*/
jlong Java_org_rocksdb_Comparator_createNewComparator0(JNIEnv* env,
jobject jobj,
jlong copt_handle) {
jlong Java_org_rocksdb_AbstractComparator_createNewComparator(
JNIEnv* env, jobject jcomparator, jlong copt_handle) {
auto* copt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(copt_handle);
auto* c = new rocksdb::ComparatorJniCallback(env, jobj, copt);
auto* c = new rocksdb::ComparatorJniCallback(env, jcomparator, copt);
return reinterpret_cast<jlong>(c);
}
// </editor-fold>
// <editor-fold desc="org.rocksdb.DirectComparator>
/*
* Class: org_rocksdb_DirectComparator
* Method: createNewDirectComparator0
* Signature: ()J
* Class: org_rocksdb_AbstractComparator
* Method: usingDirectBuffers
* Signature: (J)Z
*/
jlong Java_org_rocksdb_DirectComparator_createNewDirectComparator0(
JNIEnv* env, jobject jobj, jlong copt_handle) {
auto* copt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(copt_handle);
auto* c = new rocksdb::DirectComparatorJniCallback(env, jobj, copt);
return reinterpret_cast<jlong>(c);
jboolean Java_org_rocksdb_AbstractComparator_usingDirectBuffers(
JNIEnv*, jobject, jlong jhandle) {
auto* c = reinterpret_cast<rocksdb::ComparatorJniCallback*>(jhandle);
return static_cast<jboolean>(c->m_options->direct_buffer);
}
/*
@ -60,4 +51,3 @@ void Java_org_rocksdb_NativeComparatorWrapper_disposeInternal(
auto* comparator = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
delete comparator;
}
// </editor-fold>

717
java/rocksjni/comparatorjnicallback.cc
View File

@ -10,108 +10,281 @@
#include "rocksjni/portal.h"
namespace rocksdb {
BaseComparatorJniCallback::BaseComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt)
: JniCallback(env, jComparator),
mtx_compare(new port::Mutex(copt->use_adaptive_mutex)),
mtx_findShortestSeparator(new port::Mutex(copt->use_adaptive_mutex)) {
ComparatorJniCallback::ComparatorJniCallback(
JNIEnv* env, jobject jcomparator,
const ComparatorJniCallbackOptions* options)
: JniCallback(env, jcomparator),
m_options(options) {
// cache the AbstractComparatorJniBridge class as we will reuse it many times for each callback
m_abstract_comparator_jni_bridge_clazz =
static_cast<jclass>(env->NewGlobalRef(AbstractComparatorJniBridge::getJClass(env)));
// Note: The name of a Comparator will not change during it's lifetime,
// so we cache it in a global var
jmethodID jNameMethodId = AbstractComparatorJni::getNameMethodId(env);
if(jNameMethodId == nullptr) {
jmethodID jname_mid = AbstractComparatorJni::getNameMethodId(env);
if (jname_mid == nullptr) {
// exception thrown: NoSuchMethodException or OutOfMemoryError
return;
}
jstring jsName = (jstring)env->CallObjectMethod(m_jcallback_obj, jNameMethodId);
if(env->ExceptionCheck()) {
jstring js_name = (jstring)env->CallObjectMethod(m_jcallback_obj, jname_mid);
if (env->ExceptionCheck()) {
// exception thrown
return;
}
jboolean has_exception = JNI_FALSE;
m_name = JniUtil::copyString(env, jsName,
m_name = JniUtil::copyString(env, js_name,
&has_exception); // also releases jsName
if (has_exception == JNI_TRUE) {
// exception thrown
return;
}
m_jCompareMethodId = AbstractComparatorJni::getCompareMethodId(env);
if(m_jCompareMethodId == nullptr) {
// cache the ByteBuffer class as we will reuse it many times for each callback
m_jbytebuffer_clazz =
static_cast<jclass>(env->NewGlobalRef(ByteBufferJni::getJClass(env)));
m_jcompare_mid = AbstractComparatorJniBridge::getCompareInternalMethodId(
env, m_abstract_comparator_jni_bridge_clazz);
if (m_jcompare_mid == nullptr) {
// exception thrown: NoSuchMethodException or OutOfMemoryError
return;
}
m_jFindShortestSeparatorMethodId =
AbstractComparatorJni::getFindShortestSeparatorMethodId(env);
if(m_jFindShortestSeparatorMethodId == nullptr) {
m_jshortest_mid =
AbstractComparatorJniBridge::getFindShortestSeparatorInternalMethodId(
env, m_abstract_comparator_jni_bridge_clazz);
if (m_jshortest_mid == nullptr) {
// exception thrown: NoSuchMethodException or OutOfMemoryError
return;
}
m_jFindShortSuccessorMethodId =
AbstractComparatorJni::getFindShortSuccessorMethodId(env);
if(m_jFindShortSuccessorMethodId == nullptr) {
m_jshort_mid =
AbstractComparatorJniBridge::getFindShortSuccessorInternalMethodId(env,
m_abstract_comparator_jni_bridge_clazz);
if (m_jshort_mid == nullptr) {
// exception thrown: NoSuchMethodException or OutOfMemoryError
return;
}
// do we need reusable buffers?
if (m_options->max_reused_buffer_size > -1) {
if (m_options->reused_synchronisation_type
== ReusedSynchronisationType::THREAD_LOCAL) {
// buffers reused per thread
UnrefHandler unref = [](void* ptr) {
ThreadLocalBuf* tlb = reinterpret_cast<ThreadLocalBuf*>(ptr);
jboolean attached_thread = JNI_FALSE;
JNIEnv* _env = JniUtil::getJniEnv(tlb->jvm, &attached_thread);
if (_env != nullptr) {
if (tlb->direct_buffer) {
void* buf = _env->GetDirectBufferAddress(tlb->jbuf);
delete[] static_cast<char*>(buf);
}
_env->DeleteGlobalRef(tlb->jbuf);
JniUtil::releaseJniEnv(tlb->jvm, attached_thread);
}
};
m_tl_buf_a = new ThreadLocalPtr(unref);
m_tl_buf_b = new ThreadLocalPtr(unref);
m_jcompare_buf_a = nullptr;
m_jcompare_buf_b = nullptr;
m_jshortest_buf_start = nullptr;
m_jshortest_buf_limit = nullptr;
m_jshort_buf_key = nullptr;
} else {
//buffers reused and shared across threads
const bool adaptive =
m_options->reused_synchronisation_type == ReusedSynchronisationType::ADAPTIVE_MUTEX;
mtx_compare = std::unique_ptr<port::Mutex>(new port::Mutex(adaptive));
mtx_shortest = std::unique_ptr<port::Mutex>(new port::Mutex(adaptive));
mtx_short = std::unique_ptr<port::Mutex>(new port::Mutex(adaptive));
m_jcompare_buf_a = env->NewGlobalRef(ByteBufferJni::construct(
env, m_options->direct_buffer, m_options->max_reused_buffer_size,
m_jbytebuffer_clazz));
if (m_jcompare_buf_a == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jcompare_buf_b = env->NewGlobalRef(ByteBufferJni::construct(
env, m_options->direct_buffer, m_options->max_reused_buffer_size,
m_jbytebuffer_clazz));
if (m_jcompare_buf_b == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jshortest_buf_start = env->NewGlobalRef(ByteBufferJni::construct(
env, m_options->direct_buffer, m_options->max_reused_buffer_size,
m_jbytebuffer_clazz));
if (m_jshortest_buf_start == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jshortest_buf_limit = env->NewGlobalRef(ByteBufferJni::construct(
env, m_options->direct_buffer, m_options->max_reused_buffer_size,
m_jbytebuffer_clazz));
if (m_jshortest_buf_limit == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jshort_buf_key = env->NewGlobalRef(ByteBufferJni::construct(
env, m_options->direct_buffer, m_options->max_reused_buffer_size,
m_jbytebuffer_clazz));
if (m_jshort_buf_key == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_tl_buf_a = nullptr;
m_tl_buf_b = nullptr;
}
} else {
m_jcompare_buf_a = nullptr;
m_jcompare_buf_b = nullptr;
m_jshortest_buf_start = nullptr;
m_jshortest_buf_limit = nullptr;
m_jshort_buf_key = nullptr;
m_tl_buf_a = nullptr;
m_tl_buf_b = nullptr;
}
}
const char* BaseComparatorJniCallback::Name() const {
return m_name.get();
}
int BaseComparatorJniCallback::Compare(const Slice& a, const Slice& b) const {
ComparatorJniCallback::~ComparatorJniCallback() {
jboolean attached_thread = JNI_FALSE;
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
// TODO(adamretter): slice objects can potentially be cached using thread
// local variables to avoid locking. Could make this configurable depending on
// performance.
mtx_compare.get()->Lock();
env->DeleteGlobalRef(m_abstract_comparator_jni_bridge_clazz);
bool pending_exception =
AbstractSliceJni::setHandle(env, m_jSliceA, &a, JNI_FALSE);
if(pending_exception) {
if(env->ExceptionCheck()) {
// exception thrown from setHandle or descendant
env->ExceptionDescribe(); // print out exception to stderr
env->DeleteGlobalRef(m_jbytebuffer_clazz);
if (m_jcompare_buf_a != nullptr) {
if (m_options->direct_buffer) {
void* buf = env->GetDirectBufferAddress(m_jcompare_buf_a);
delete[] static_cast<char*>(buf);
}
env->DeleteGlobalRef(m_jcompare_buf_a);
}
if (m_jcompare_buf_b != nullptr) {
if (m_options->direct_buffer) {
void* buf = env->GetDirectBufferAddress(m_jcompare_buf_b);
delete[] static_cast<char*>(buf);
}
env->DeleteGlobalRef(m_jcompare_buf_b);
}
if (m_jshortest_buf_start != nullptr) {
if (m_options->direct_buffer) {
void* buf = env->GetDirectBufferAddress(m_jshortest_buf_start);
delete[] static_cast<char*>(buf);
}
env->DeleteGlobalRef(m_jshortest_buf_start);
}
if (m_jshortest_buf_limit != nullptr) {
if (m_options->direct_buffer) {
void* buf = env->GetDirectBufferAddress(m_jshortest_buf_limit);
delete[] static_cast<char*>(buf);
}
env->DeleteGlobalRef(m_jshortest_buf_limit);
}
if (m_jshort_buf_key != nullptr) {
if (m_options->direct_buffer) {
void* buf = env->GetDirectBufferAddress(m_jshort_buf_key);
delete[] static_cast<char*>(buf);
}
env->DeleteGlobalRef(m_jshort_buf_key);
}
if (m_tl_buf_a != nullptr) {
delete m_tl_buf_a;
}
if (m_tl_buf_b != nullptr) {
delete m_tl_buf_b;
}
releaseJniEnv(attached_thread);
}
const char* ComparatorJniCallback::Name() const {
return m_name.get();
}
int ComparatorJniCallback::Compare(const Slice& a, const Slice& b) const {
jboolean attached_thread = JNI_FALSE;
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
const bool reuse_jbuf_a =
static_cast<int64_t>(a.size()) <= m_options->max_reused_buffer_size;
const bool reuse_jbuf_b =
static_cast<int64_t>(b.size()) <= m_options->max_reused_buffer_size;
MaybeLockForReuse(mtx_compare, reuse_jbuf_a || reuse_jbuf_b);
jobject jcompare_buf_a = GetBuffer(env, a, reuse_jbuf_a, m_tl_buf_a, m_jcompare_buf_a);
if (jcompare_buf_a == nullptr) {
// exception occurred
MaybeUnlockForReuse(mtx_compare, reuse_jbuf_a || reuse_jbuf_b);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return 0;
}
pending_exception =
AbstractSliceJni::setHandle(env, m_jSliceB, &b, JNI_FALSE);
if(pending_exception) {
if(env->ExceptionCheck()) {
// exception thrown from setHandle or descendant
env->ExceptionDescribe(); // print out exception to stderr
jobject jcompare_buf_b = GetBuffer(env, b, reuse_jbuf_b, m_tl_buf_b, m_jcompare_buf_b);
if (jcompare_buf_b == nullptr) {
// exception occurred
if (!reuse_jbuf_a) {
DeleteBuffer(env, jcompare_buf_a);
}
MaybeUnlockForReuse(mtx_compare, reuse_jbuf_a || reuse_jbuf_b);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return 0;
}
jint result =
env->CallIntMethod(m_jcallback_obj, m_jCompareMethodId, m_jSliceA,
m_jSliceB);
env->CallStaticIntMethod(
m_abstract_comparator_jni_bridge_clazz, m_jcompare_mid,
m_jcallback_obj,
jcompare_buf_a, reuse_jbuf_a ? a.size() : -1,
jcompare_buf_b, reuse_jbuf_b ? b.size() : -1);
mtx_compare.get()->Unlock();
if(env->ExceptionCheck()) {
if (env->ExceptionCheck()) {
// exception thrown from CallIntMethod
env->ExceptionDescribe(); // print out exception to stderr
result = 0; // we could not get a result from java callback so use 0
}
if (!reuse_jbuf_a) {
DeleteBuffer(env, jcompare_buf_a);
}
if (!reuse_jbuf_b) {
DeleteBuffer(env, jcompare_buf_b);
}
MaybeUnlockForReuse(mtx_compare, reuse_jbuf_a || reuse_jbuf_b);
releaseJniEnv(attached_thread);
return result;
}
void BaseComparatorJniCallback::FindShortestSeparator(
void ComparatorJniCallback::FindShortestSeparator(
std::string* start, const Slice& limit) const {
if (start == nullptr) {
return;
@ -121,78 +294,132 @@ void BaseComparatorJniCallback::FindShortestSeparator(
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
const char* startUtf = start->c_str();
jstring jsStart = env->NewStringUTF(startUtf);
if(jsStart == nullptr) {
// unable to construct string
if(env->ExceptionCheck()) {
env->ExceptionDescribe(); // print out exception to stderr
}
releaseJniEnv(attached_thread);
return;
}
if(env->ExceptionCheck()) {
// exception thrown: OutOfMemoryError
const bool reuse_jbuf_start =
static_cast<int64_t>(start->length()) <= m_options->max_reused_buffer_size;
const bool reuse_jbuf_limit =
static_cast<int64_t>(limit.size()) <= m_options->max_reused_buffer_size;
MaybeLockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
Slice sstart(start->data(), start->length());
jobject j_start_buf = GetBuffer(env, sstart, reuse_jbuf_start, m_tl_buf_a, m_jshortest_buf_start);
if (j_start_buf == nullptr) {
// exception occurred
MaybeUnlockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
env->ExceptionDescribe(); // print out exception to stderr
env->DeleteLocalRef(jsStart);
releaseJniEnv(attached_thread);
return;
}
// TODO(adamretter): slice object can potentially be cached using thread local
// variable to avoid locking. Could make this configurable depending on
// performance.
mtx_findShortestSeparator.get()->Lock();
bool pending_exception =
AbstractSliceJni::setHandle(env, m_jSliceLimit, &limit, JNI_FALSE);
if(pending_exception) {
if(env->ExceptionCheck()) {
// exception thrown from setHandle or descendant
env->ExceptionDescribe(); // print out exception to stderr
jobject j_limit_buf = GetBuffer(env, limit, reuse_jbuf_limit, m_tl_buf_b, m_jshortest_buf_limit);
if (j_limit_buf == nullptr) {
// exception occurred
if (!reuse_jbuf_start) {
DeleteBuffer(env, j_start_buf);
}
if(jsStart != nullptr) {
env->DeleteLocalRef(jsStart);
MaybeUnlockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
jint jstart_len = env->CallStaticIntMethod(
m_abstract_comparator_jni_bridge_clazz, m_jshortest_mid,
m_jcallback_obj,
j_start_buf, reuse_jbuf_start ? start->length() : -1,
j_limit_buf, reuse_jbuf_limit ? limit.size() : -1);
if (env->ExceptionCheck()) {
// exception thrown from CallIntMethod
env->ExceptionDescribe(); // print out exception to stderr
} else if (static_cast<size_t>(jstart_len) != start->length()) {
// start buffer has changed in Java, so update `start` with the result
bool copy_from_non_direct = false;
if (reuse_jbuf_start) {
// reused a buffer
if (m_options->direct_buffer) {
// reused direct buffer
void* start_buf = env->GetDirectBufferAddress(j_start_buf);
if (start_buf == nullptr) {
if (!reuse_jbuf_start) {
DeleteBuffer(env, j_start_buf);
}
if (!reuse_jbuf_limit) {
DeleteBuffer(env, j_limit_buf);
}
MaybeUnlockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
rocksdb::RocksDBExceptionJni::ThrowNew(env, "Unable to get Direct Buffer Address");
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
start->assign(static_cast<const char*>(start_buf), jstart_len);
} else {
// reused non-direct buffer
copy_from_non_direct = true;
}
} else {
// there was a new buffer
if (m_options->direct_buffer) {
// it was direct... don't forget to potentially truncate the `start` string
start->resize(jstart_len);
} else {
// it was non-direct
copy_from_non_direct = true;
}
}
releaseJniEnv(attached_thread);
return;
}
jstring jsResultStart =
(jstring)env->CallObjectMethod(m_jcallback_obj,
m_jFindShortestSeparatorMethodId, jsStart, m_jSliceLimit);
mtx_findShortestSeparator.get()->Unlock();
if(env->ExceptionCheck()) {
// exception thrown from CallObjectMethod
env->ExceptionDescribe(); // print out exception to stderr
env->DeleteLocalRef(jsStart);
releaseJniEnv(attached_thread);
return;
}
env->DeleteLocalRef(jsStart);
if (jsResultStart != nullptr) {
// update start with result
jboolean has_exception = JNI_FALSE;
std::unique_ptr<const char[]> result_start = JniUtil::copyString(env, jsResultStart,
&has_exception); // also releases jsResultStart
if (has_exception == JNI_TRUE) {
if (env->ExceptionCheck()) {
if (copy_from_non_direct) {
jbyteArray jarray = ByteBufferJni::array(env, j_start_buf,
m_jbytebuffer_clazz);
if (jarray == nullptr) {
if (!reuse_jbuf_start) {
DeleteBuffer(env, j_start_buf);
}
if (!reuse_jbuf_limit) {
DeleteBuffer(env, j_limit_buf);
}
MaybeUnlockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
jboolean has_exception = JNI_FALSE;
JniUtil::byteString<std::string>(env, jarray, [start, jstart_len](const char* data, const size_t) {
return start->assign(data, static_cast<size_t>(jstart_len));
}, &has_exception);
env->DeleteLocalRef(jarray);
if (has_exception == JNI_TRUE) {
if (!reuse_jbuf_start) {
DeleteBuffer(env, j_start_buf);
}
if (!reuse_jbuf_limit) {
DeleteBuffer(env, j_limit_buf);
}
env->ExceptionDescribe(); // print out exception to stderr
MaybeUnlockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
releaseJniEnv(attached_thread);
return;
}
releaseJniEnv(attached_thread);
return;
}
start->assign(result_start.get());
}
if (!reuse_jbuf_start) {
DeleteBuffer(env, j_start_buf);
}
if (!reuse_jbuf_limit) {
DeleteBuffer(env, j_limit_buf);
}
MaybeUnlockForReuse(mtx_shortest, reuse_jbuf_start || reuse_jbuf_limit);
releaseJniEnv(attached_thread);
}
void BaseComparatorJniCallback::FindShortSuccessor(
void ComparatorJniCallback::FindShortSuccessor(
std::string* key) const {
if (key == nullptr) {
return;
@ -202,139 +429,207 @@ void BaseComparatorJniCallback::FindShortSuccessor(
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
const char* keyUtf = key->c_str();
jstring jsKey = env->NewStringUTF(keyUtf);
if(jsKey == nullptr) {
// unable to construct string
if(env->ExceptionCheck()) {
env->ExceptionDescribe(); // print out exception to stderr
}
releaseJniEnv(attached_thread);
return;
} else if(env->ExceptionCheck()) {
// exception thrown: OutOfMemoryError
const bool reuse_jbuf_key =
static_cast<int64_t>(key->length()) <= m_options->max_reused_buffer_size;
MaybeLockForReuse(mtx_short, reuse_jbuf_key);
Slice skey(key->data(), key->length());
jobject j_key_buf = GetBuffer(env, skey, reuse_jbuf_key, m_tl_buf_a, m_jshort_buf_key);
if (j_key_buf == nullptr) {
// exception occurred
MaybeUnlockForReuse(mtx_short, reuse_jbuf_key);
env->ExceptionDescribe(); // print out exception to stderr
env->DeleteLocalRef(jsKey);
releaseJniEnv(attached_thread);
return;
}
jstring jsResultKey =
(jstring)env->CallObjectMethod(m_jcallback_obj,
m_jFindShortSuccessorMethodId, jsKey);
jint jkey_len = env->CallStaticIntMethod(
m_abstract_comparator_jni_bridge_clazz, m_jshort_mid,
m_jcallback_obj,
j_key_buf, reuse_jbuf_key ? key->length() : -1);
if(env->ExceptionCheck()) {
if (env->ExceptionCheck()) {
// exception thrown from CallObjectMethod
env->ExceptionDescribe(); // print out exception to stderr
env->DeleteLocalRef(jsKey);
if (!reuse_jbuf_key) {
DeleteBuffer(env, j_key_buf);
}
MaybeUnlockForReuse(mtx_short, reuse_jbuf_key);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
env->DeleteLocalRef(jsKey);
if (jsResultKey != nullptr) {
// updates key with result, also releases jsResultKey.
jboolean has_exception = JNI_FALSE;
std::unique_ptr<const char[]> result_key = JniUtil::copyString(env, jsResultKey,
&has_exception); // also releases jsResultKey
if (has_exception == JNI_TRUE) {
if (env->ExceptionCheck()) {
env->ExceptionDescribe(); // print out exception to stderr
}
releaseJniEnv(attached_thread);
return;
if (static_cast<size_t>(jkey_len) != key->length()) {
// key buffer has changed in Java, so update `key` with the result
bool copy_from_non_direct = false;
if (reuse_jbuf_key) {
// reused a buffer
if (m_options->direct_buffer) {
// reused direct buffer
void* key_buf = env->GetDirectBufferAddress(j_key_buf);
if (key_buf == nullptr) {
rocksdb::RocksDBExceptionJni::ThrowNew(env, "Unable to get Direct Buffer Address");
if (!reuse_jbuf_key) {
DeleteBuffer(env, j_key_buf);
}
MaybeUnlockForReuse(mtx_short, reuse_jbuf_key);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
key->assign(static_cast<const char*>(key_buf), jkey_len);
} else {
// reused non-direct buffer
copy_from_non_direct = true;
}
} else {
// there was a new buffer
if (m_options->direct_buffer) {
// it was direct... don't forget to potentially truncate the `key` string
key->resize(jkey_len);
} else {
// it was non-direct
copy_from_non_direct = true;
}
}
key->assign(result_key.get());
if (copy_from_non_direct) {
jbyteArray jarray = ByteBufferJni::array(env, j_key_buf,
m_jbytebuffer_clazz);
if (jarray == nullptr) {
if (!reuse_jbuf_key) {
DeleteBuffer(env, j_key_buf);
}
MaybeUnlockForReuse(mtx_short, reuse_jbuf_key);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
jboolean has_exception = JNI_FALSE;
JniUtil::byteString<std::string>(env, jarray, [key, jkey_len](const char* data, const size_t) {
return key->assign(data, static_cast<size_t>(jkey_len));
}, &has_exception);
env->DeleteLocalRef(jarray);
if (has_exception == JNI_TRUE) {
if (!reuse_jbuf_key) {
DeleteBuffer(env, j_key_buf);
}
MaybeUnlockForReuse(mtx_short, reuse_jbuf_key);
env->ExceptionDescribe(); // print out exception to stderr
releaseJniEnv(attached_thread);
return;
}
}
}
if (!reuse_jbuf_key) {
DeleteBuffer(env, j_key_buf);
}
MaybeUnlockForReuse(mtx_short, reuse_jbuf_key);
releaseJniEnv(attached_thread);
}
ComparatorJniCallback::ComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt) :
BaseComparatorJniCallback(env, jComparator, copt) {
m_jSliceA = env->NewGlobalRef(SliceJni::construct0(env));
if(m_jSliceA == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jSliceB = env->NewGlobalRef(SliceJni::construct0(env));
if(m_jSliceB == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jSliceLimit = env->NewGlobalRef(SliceJni::construct0(env));
if(m_jSliceLimit == nullptr) {
// exception thrown: OutOfMemoryError
return;
inline void ComparatorJniCallback::MaybeLockForReuse(
const std::unique_ptr<port::Mutex>& mutex, const bool cond) const {
// no need to lock if using thread_local
if (m_options->reused_synchronisation_type != ReusedSynchronisationType::THREAD_LOCAL
&& cond) {
mutex.get()->Lock();
}
}
ComparatorJniCallback::~ComparatorJniCallback() {
jboolean attached_thread = JNI_FALSE;
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
if(m_jSliceA != nullptr) {
env->DeleteGlobalRef(m_jSliceA);
}
if(m_jSliceB != nullptr) {
env->DeleteGlobalRef(m_jSliceB);
}
if(m_jSliceLimit != nullptr) {
env->DeleteGlobalRef(m_jSliceLimit);
}
releaseJniEnv(attached_thread);
}
DirectComparatorJniCallback::DirectComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt) :
BaseComparatorJniCallback(env, jComparator, copt) {
m_jSliceA = env->NewGlobalRef(DirectSliceJni::construct0(env));
if(m_jSliceA == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jSliceB = env->NewGlobalRef(DirectSliceJni::construct0(env));
if(m_jSliceB == nullptr) {
// exception thrown: OutOfMemoryError
return;
}
m_jSliceLimit = env->NewGlobalRef(DirectSliceJni::construct0(env));
if(m_jSliceLimit == nullptr) {
// exception thrown: OutOfMemoryError
return;
inline void ComparatorJniCallback::MaybeUnlockForReuse(
const std::unique_ptr<port::Mutex>& mutex, const bool cond) const {
// no need to unlock if using thread_local
if (m_options->reused_synchronisation_type != ReusedSynchronisationType::THREAD_LOCAL
&& cond) {
mutex.get()->Unlock();
}
}
DirectComparatorJniCallback::~DirectComparatorJniCallback() {
jboolean attached_thread = JNI_FALSE;
JNIEnv* env = getJniEnv(&attached_thread);
assert(env != nullptr);
jobject ComparatorJniCallback::GetBuffer(JNIEnv* env, const Slice& src,
bool reuse_buffer, ThreadLocalPtr* tl_buf, jobject jreuse_buffer) const {
if (reuse_buffer) {
if (m_options->reused_synchronisation_type
== ReusedSynchronisationType::THREAD_LOCAL) {
if(m_jSliceA != nullptr) {
env->DeleteGlobalRef(m_jSliceA);
// reuse thread-local bufffer
ThreadLocalBuf* tlb = reinterpret_cast<ThreadLocalBuf*>(tl_buf->Get());
if (tlb == nullptr) {
// thread-local buffer has not yet been created, so create it
jobject jtl_buf = env->NewGlobalRef(ByteBufferJni::construct(
env, m_options->direct_buffer, m_options->max_reused_buffer_size,
m_jbytebuffer_clazz));
if (jtl_buf == nullptr) {
// exception thrown: OutOfMemoryError
return nullptr;
}
tlb = new ThreadLocalBuf(m_jvm, m_options->direct_buffer, jtl_buf);
tl_buf->Reset(tlb);
}
return ReuseBuffer(env, src, tlb->jbuf);
} else {
// reuse class member buffer
return ReuseBuffer(env, src, jreuse_buffer);
}
} else {
// new buffer
return NewBuffer(env, src);
}
if(m_jSliceB != nullptr) {
env->DeleteGlobalRef(m_jSliceB);
}
if(m_jSliceLimit != nullptr) {
env->DeleteGlobalRef(m_jSliceLimit);
}
releaseJniEnv(attached_thread);
}
jobject ComparatorJniCallback::ReuseBuffer(
JNIEnv* env, const Slice& src, jobject jreuse_buffer) const {
// we can reuse the buffer
if (m_options->direct_buffer) {
// copy into direct buffer
void* buf = env->GetDirectBufferAddress(jreuse_buffer);
if (buf == nullptr) {
// either memory region is undefined, given object is not a direct java.nio.Buffer, or JNI access to direct buffers is not supported by this virtual machine.
rocksdb::RocksDBExceptionJni::ThrowNew(env, "Unable to get Direct Buffer Address");
return nullptr;
}
memcpy(buf, src.data(), src.size());
} else {
// copy into non-direct buffer
const jbyteArray jarray = ByteBufferJni::array(env, jreuse_buffer,
m_jbytebuffer_clazz);
if (jarray == nullptr) {
// exception occurred
return nullptr;
}
env->SetByteArrayRegion(jarray, 0, static_cast<jsize>(src.size()),
const_cast<jbyte*>(reinterpret_cast<const jbyte*>(src.data())));
if (env->ExceptionCheck()) {
// exception occurred
env->DeleteLocalRef(jarray);
return nullptr;
}
env->DeleteLocalRef(jarray);
}
return jreuse_buffer;
}
jobject ComparatorJniCallback::NewBuffer(JNIEnv* env, const Slice& src) const {
// we need a new buffer
jobject jbuf = ByteBufferJni::constructWith(env, m_options->direct_buffer,
src.data(), src.size(), m_jbytebuffer_clazz);
if (jbuf == nullptr) {
// exception occurred
return nullptr;
}
return jbuf;
}
void ComparatorJniCallback::DeleteBuffer(JNIEnv* env, jobject jbuffer) const {
env->DeleteLocalRef(jbuffer);
}
} // namespace rocksdb

122
java/rocksjni/comparatorjnicallback.h
View File

@ -4,7 +4,7 @@
// (found in the LICENSE.Apache file in the root directory).
//
// This file implements the callback "bridge" between Java and C++ for
// rocksdb::Comparator and rocksdb::DirectComparator.
// rocksdb::Comparator
#ifndef JAVA_ROCKSJNI_COMPARATORJNICALLBACK_H_
#define JAVA_ROCKSJNI_COMPARATORJNICALLBACK_H_
@ -16,19 +16,54 @@
#include "rocksdb/comparator.h"
#include "rocksdb/slice.h"
#include "port/port.h"
#include "util/thread_local.h"
namespace rocksdb {
struct ComparatorJniCallbackOptions {
// Use adaptive mutex, which spins in the user space before resorting
// to kernel. This could reduce context switch when the mutex is not
// heavily contended. However, if the mutex is hot, we could end up
// wasting spin time.
// Default: false
bool use_adaptive_mutex;
enum ReusedSynchronisationType {
/**
* Standard mutex.
*/
MUTEX,
ComparatorJniCallbackOptions() : use_adaptive_mutex(false) {
}
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
*/
ADAPTIVE_MUTEX,
/**
* There is a reused buffer per-thread.
*/
THREAD_LOCAL
};
struct ComparatorJniCallbackOptions {
// Set the synchronisation type used to guard the reused buffers.
// Only used if max_reused_buffer_size > 0.
// Default: ADAPTIVE_MUTEX
ReusedSynchronisationType reused_synchronisation_type =
ReusedSynchronisationType::ADAPTIVE_MUTEX;
// Indicates if a direct byte buffer (i.e. outside of the normal
// garbage-collected heap) is used for the callbacks to Java,
// as opposed to a non-direct byte buffer which is a wrapper around
// an on-heap byte[].
// Default: true
bool direct_buffer = true;
// Maximum size of a buffer (in bytes) that will be reused.
// Comparators will use 5 of these buffers,
// so the retained memory size will be 5 * max_reused_buffer_size.
// When a buffer is needed for transferring data to a callback,
// if it requires less than max_reused_buffer_size, then an
// existing buffer will be reused, else a new buffer will be
// allocated just for that callback. -1 to disable.
// Default: 64 bytes
int32_t max_reused_buffer_size = 64;
};
/**
@ -46,47 +81,56 @@ struct ComparatorJniCallbackOptions {
* introduce independent locking in regions of each of those methods
* via the mutexs mtx_compare and mtx_findShortestSeparator respectively
*/
class BaseComparatorJniCallback : public JniCallback, public Comparator {
class ComparatorJniCallback : public JniCallback, public Comparator {
public:
BaseComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt);
ComparatorJniCallback(
JNIEnv* env, jobject jcomparator,
const ComparatorJniCallbackOptions* options);
~ComparatorJniCallback();
virtual const char* Name() const;
virtual int Compare(const Slice& a, const Slice& b) const;
virtual void FindShortestSeparator(
std::string* start, const Slice& limit) const;
virtual void FindShortSuccessor(std::string* key) const;
const ComparatorJniCallbackOptions* m_options;
private:
struct ThreadLocalBuf {
ThreadLocalBuf(JavaVM* _jvm, bool _direct_buffer, jobject _jbuf) :
jvm(_jvm), direct_buffer(_direct_buffer), jbuf(_jbuf) {}
JavaVM* jvm;
bool direct_buffer;
jobject jbuf;
};
inline void MaybeLockForReuse(const std::unique_ptr<port::Mutex>& mutex,
const bool cond) const;
inline void MaybeUnlockForReuse(const std::unique_ptr<port::Mutex>& mutex,
const bool cond) const;
jobject GetBuffer(JNIEnv* env, const Slice& src, bool reuse_buffer,
ThreadLocalPtr* tl_buf, jobject jreuse_buffer) const;
jobject ReuseBuffer(JNIEnv* env, const Slice& src,
jobject jreuse_buffer) const;
jobject NewBuffer(JNIEnv* env, const Slice& src) const;
void DeleteBuffer(JNIEnv* env, jobject jbuffer) const;
// used for synchronisation in compare method
std::unique_ptr<port::Mutex> mtx_compare;
// used for synchronisation in findShortestSeparator method
std::unique_ptr<port::Mutex> mtx_findShortestSeparator;
std::unique_ptr<port::Mutex> mtx_shortest;
// used for synchronisation in findShortSuccessor method
std::unique_ptr<port::Mutex> mtx_short;
std::unique_ptr<const char[]> m_name;
jmethodID m_jCompareMethodId;
jmethodID m_jFindShortestSeparatorMethodId;
jmethodID m_jFindShortSuccessorMethodId;
protected:
jobject m_jSliceA;
jobject m_jSliceB;
jobject m_jSliceLimit;
};
class ComparatorJniCallback : public BaseComparatorJniCallback {
public:
ComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt);
~ComparatorJniCallback();
};
class DirectComparatorJniCallback : public BaseComparatorJniCallback {
public:
DirectComparatorJniCallback(
JNIEnv* env, jobject jComparator,
const ComparatorJniCallbackOptions* copt);
~DirectComparatorJniCallback();
jclass m_abstract_comparator_jni_bridge_clazz; // TODO(AR) could we make this static somehow?
jclass m_jbytebuffer_clazz; // TODO(AR) we could cache this globally for the entire VM if we switch more APIs to use ByteBuffer // TODO(AR) could we make this static somehow?
jmethodID m_jcompare_mid; // TODO(AR) could we make this static somehow?
jmethodID m_jshortest_mid; // TODO(AR) could we make this static somehow?
jmethodID m_jshort_mid; // TODO(AR) could we make this static somehow?
jobject m_jcompare_buf_a;
jobject m_jcompare_buf_b;
jobject m_jshortest_buf_start;
jobject m_jshortest_buf_limit;
jobject m_jshort_buf_key;
ThreadLocalPtr* m_tl_buf_a;
ThreadLocalPtr* m_tl_buf_b;
};
} // namespace rocksdb

87
java/rocksjni/options.cc
View File

@ -173,14 +173,8 @@ void Java_org_rocksdb_Options_setComparatorHandle__JJB(
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator = reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(
jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
comparator = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
@ -3301,14 +3295,8 @@ void Java_org_rocksdb_ColumnFamilyOptions_setComparatorHandle__JJB(
reinterpret_cast<rocksdb::ComparatorJniCallback*>(jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator = reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(
jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
comparator = reinterpret_cast<rocksdb::Comparator*>(jcomparator_handle);
break;
}
@ -6869,24 +6857,75 @@ jlong Java_org_rocksdb_ComparatorOptions_newComparatorOptions(
/*
* Class: org_rocksdb_ComparatorOptions
* Method: useAdaptiveMutex
* Signature: (J)Z
* Method: reusedSynchronisationType
* Signature: (J)B
*/
jboolean Java_org_rocksdb_ComparatorOptions_useAdaptiveMutex(
JNIEnv*, jobject, jlong jhandle) {
return reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->use_adaptive_mutex;
jbyte Java_org_rocksdb_ComparatorOptions_reusedSynchronisationType(
JNIEnv *, jobject, jlong jhandle) {
auto* comparator_opt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle);
return rocksdb::ReusedSynchronisationTypeJni::toJavaReusedSynchronisationType(
comparator_opt->reused_synchronisation_type);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: setUseAdaptiveMutex
* Method: setReusedSynchronisationType
* Signature: (JB)V
*/
void Java_org_rocksdb_ComparatorOptions_setReusedSynchronisationType(
JNIEnv*, jobject, jlong jhandle, jbyte jreused_synhcronisation_type) {
auto* comparator_opt =
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle);
comparator_opt->reused_synchronisation_type =
rocksdb::ReusedSynchronisationTypeJni::toCppReusedSynchronisationType(jreused_synhcronisation_type);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: useDirectBuffer
* Signature: (J)Z
*/
jboolean Java_org_rocksdb_ComparatorOptions_useDirectBuffer(
JNIEnv*, jobject, jlong jhandle) {
return static_cast<jboolean>(
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->direct_buffer);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: setUseDirectBuffer
* Signature: (JZ)V
*/
void Java_org_rocksdb_ComparatorOptions_setUseAdaptiveMutex(
JNIEnv*, jobject, jlong jhandle, jboolean juse_adaptive_mutex) {
void Java_org_rocksdb_ComparatorOptions_setUseDirectBuffer(
JNIEnv*, jobject, jlong jhandle, jboolean jdirect_buffer) {
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->use_adaptive_mutex = static_cast<bool>(juse_adaptive_mutex);
->direct_buffer = jdirect_buffer == JNI_TRUE;
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: maxReusedBufferSize
* Signature: (J)I
*/
jint Java_org_rocksdb_ComparatorOptions_maxReusedBufferSize(
JNIEnv*, jobject, jlong jhandle) {
return static_cast<jint>(
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->max_reused_buffer_size);
}
/*
* Class: org_rocksdb_ComparatorOptions
* Method: setMaxReusedBufferSize
* Signature: (JI)V
*/
void Java_org_rocksdb_ComparatorOptions_setMaxReusedBufferSize(
JNIEnv*, jobject, jlong jhandle, jint jmax_reused_buffer_size) {
reinterpret_cast<rocksdb::ComparatorJniCallbackOptions*>(jhandle)
->max_reused_buffer_size
= static_cast<int32_t>(jmax_reused_buffer_size);
}
/*

331
java/rocksjni/portal.h
View File

@ -1207,6 +1207,157 @@ class ByteJni : public JavaClass {
};
// The portal class for java.nio.ByteBuffer
class ByteBufferJni : public JavaClass {
public:
/**
* Get the Java Class java.nio.ByteBuffer
*
* @param env A pointer to the Java environment
*
* @return The Java Class or nullptr if one of the
* ClassFormatError, ClassCircularityError, NoClassDefFoundError,
* OutOfMemoryError or ExceptionInInitializerError exceptions is thrown
*/
static jclass getJClass(JNIEnv* env) {
return JavaClass::getJClass(env, "java/nio/ByteBuffer");
}
/**
* Get the Java Method: ByteBuffer#allocate
*
* @param env A pointer to the Java environment
* @param jbytebuffer_clazz if you have a reference to a ByteBuffer class, or nullptr
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getAllocateMethodId(JNIEnv* env,
jclass jbytebuffer_clazz = nullptr) {
const jclass jclazz =
jbytebuffer_clazz == nullptr ? getJClass(env) : jbytebuffer_clazz;
if (jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid = env->GetStaticMethodID(
jclazz, "allocate", "(I)Ljava/nio/ByteBuffer;");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: ByteBuffer#array
*
* @param env A pointer to the Java environment
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getArrayMethodId(JNIEnv* env,
jclass jbytebuffer_clazz = nullptr) {
const jclass jclazz =
jbytebuffer_clazz == nullptr ? getJClass(env) : jbytebuffer_clazz;
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid = env->GetMethodID(jclazz, "array", "()[B");
assert(mid != nullptr);
return mid;
}
static jobject construct(
JNIEnv* env, const bool direct, const size_t capacity,
jclass jbytebuffer_clazz = nullptr) {
return constructWith(env, direct, nullptr, capacity, jbytebuffer_clazz);
}
static jobject constructWith(
JNIEnv* env, const bool direct, const char* buf, const size_t capacity,
jclass jbytebuffer_clazz = nullptr) {
if (direct) {
bool allocated = false;
if (buf == nullptr) {
buf = new char[capacity];
allocated = true;
}
jobject jbuf = env->NewDirectByteBuffer(const_cast<char*>(buf), static_cast<jlong>(capacity));
if (jbuf == nullptr) {
// exception occurred
if (allocated) {
delete[] static_cast<const char*>(buf);
}
return nullptr;
}
return jbuf;
} else {
const jclass jclazz =
jbytebuffer_clazz == nullptr ? getJClass(env) : jbytebuffer_clazz;
if (jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
const jmethodID jmid_allocate = getAllocateMethodId(env, jbytebuffer_clazz);
if (jmid_allocate == nullptr) {
// exception occurred accessing class, or NoSuchMethodException or OutOfMemoryError
return nullptr;
}
const jobject jbuf = env->CallStaticObjectMethod(
jclazz, jmid_allocate, static_cast<jint>(capacity));
if (env->ExceptionCheck()) {
// exception occurred
return nullptr;
}
// set buffer data?
if (buf != nullptr) {
jbyteArray jarray = array(env, jbuf, jbytebuffer_clazz);
if (jarray == nullptr) {
// exception occurred
env->DeleteLocalRef(jbuf);
return nullptr;
}
jboolean is_copy = JNI_FALSE;
jbyte* ja = reinterpret_cast<jbyte*>(
env->GetPrimitiveArrayCritical(jarray, &is_copy));
if (ja == nullptr) {
// exception occurred
env->DeleteLocalRef(jarray);
env->DeleteLocalRef(jbuf);
return nullptr;
}
memcpy(ja, const_cast<char*>(buf), capacity);
env->ReleasePrimitiveArrayCritical(jarray, ja, 0);
env->DeleteLocalRef(jarray);
}
return jbuf;
}
}
static jbyteArray array(JNIEnv* env, const jobject& jbyte_buffer,
jclass jbytebuffer_clazz = nullptr) {
const jmethodID mid = getArrayMethodId(env, jbytebuffer_clazz);
if (mid == nullptr) {
// exception occurred accessing class, or NoSuchMethodException or OutOfMemoryError
return nullptr;
}
const jobject jarray = env->CallObjectMethod(jbyte_buffer, mid);
if (env->ExceptionCheck()) {
// exception occurred
return nullptr;
}
return static_cast<jbyteArray>(jarray);
}
};
// The portal class for java.lang.Integer
class IntegerJni : public JavaClass {
public:
@ -1405,7 +1556,7 @@ class JniUtil {
JNIEnv *env;
const jint env_rs = jvm->GetEnv(reinterpret_cast<void**>(&env),
JNI_VERSION_1_2);
JNI_VERSION_1_6);
if(env_rs == JNI_OK) {
// current thread is already attached, return the JNIEnv
@ -1423,8 +1574,8 @@ class JniUtil {
return nullptr;
}
} else if(env_rs == JNI_EVERSION) {
// error, JDK does not support JNI_VERSION_1_2+
std::cerr << "JniUtil::getJniEnv - Fatal: JDK does not support JNI_VERSION_1_2" << std::endl;
// error, JDK does not support JNI_VERSION_1_6+
std::cerr << "JniUtil::getJniEnv - Fatal: JDK does not support JNI_VERSION_1_6" << std::endl;
return nullptr;
} else {
std::cerr << "JniUtil::getJniEnv - Fatal: Unknown error: env_rs=" << env_rs << std::endl;
@ -3303,9 +3454,78 @@ class AbstractTransactionNotifierJni : public RocksDBNativeClass<
}
};
// The portal class for org.rocksdb.AbstractComparatorJniBridge
class AbstractComparatorJniBridge : public JavaClass {
public:
/**
* Get the Java Class org.rocksdb.AbstractComparatorJniBridge
*
* @param env A pointer to the Java environment
*
* @return The Java Class or nullptr if one of the
* ClassFormatError, ClassCircularityError, NoClassDefFoundError,
* OutOfMemoryError or ExceptionInInitializerError exceptions is thrown
*/
static jclass getJClass(JNIEnv* env) {
return JavaClass::getJClass(env,
"org/rocksdb/AbstractComparatorJniBridge");
}
/**
* Get the Java Method: Comparator#compareInternal
*
* @param env A pointer to the Java environment
* @param jclazz the AbstractComparatorJniBridge class
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getCompareInternalMethodId(JNIEnv* env, jclass jclazz) {
static jmethodID mid =
env->GetStaticMethodID(jclazz, "compareInternal",
"(Lorg/rocksdb/AbstractComparator;Ljava/nio/ByteBuffer;ILjava/nio/ByteBuffer;I)I");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#findShortestSeparatorInternal
*
* @param env A pointer to the Java environment
* @param jclazz the AbstractComparatorJniBridge class
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getFindShortestSeparatorInternalMethodId(JNIEnv* env, jclass jclazz) {
static jmethodID mid =
env->GetStaticMethodID(jclazz, "findShortestSeparatorInternal",
"(Lorg/rocksdb/AbstractComparator;Ljava/nio/ByteBuffer;ILjava/nio/ByteBuffer;I)I");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#findShortSuccessorInternal
*
* @param env A pointer to the Java environment
* @param jclazz the AbstractComparatorJniBridge class
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getFindShortSuccessorInternalMethodId(JNIEnv* env, jclass jclazz) {
static jmethodID mid =
env->GetStaticMethodID(jclazz, "findShortSuccessorInternal",
"(Lorg/rocksdb/AbstractComparator;Ljava/nio/ByteBuffer;I)I");
assert(mid != nullptr);
return mid;
}
};
// The portal class for org.rocksdb.AbstractComparator
class AbstractComparatorJni : public RocksDBNativeClass<
const rocksdb::BaseComparatorJniCallback*,
const rocksdb::ComparatorJniCallback*,
AbstractComparatorJni> {
public:
/**
@ -3342,72 +3562,6 @@ class AbstractComparatorJni : public RocksDBNativeClass<
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#compare
*
* @param env A pointer to the Java environment
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getCompareMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid =
env->GetMethodID(jclazz, "compare",
"(Lorg/rocksdb/AbstractSlice;Lorg/rocksdb/AbstractSlice;)I");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#findShortestSeparator
*
* @param env A pointer to the Java environment
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getFindShortestSeparatorMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid =
env->GetMethodID(jclazz, "findShortestSeparator",
"(Ljava/lang/String;Lorg/rocksdb/AbstractSlice;)Ljava/lang/String;");
assert(mid != nullptr);
return mid;
}
/**
* Get the Java Method: Comparator#findShortSuccessor
*
* @param env A pointer to the Java environment
*
* @return The Java Method ID or nullptr if the class or method id could not
* be retieved
*/
static jmethodID getFindShortSuccessorMethodId(JNIEnv* env) {
jclass jclazz = getJClass(env);
if(jclazz == nullptr) {
// exception occurred accessing class
return nullptr;
}
static jmethodID mid =
env->GetMethodID(jclazz, "findShortSuccessor",
"(Ljava/lang/String;)Ljava/lang/String;");
assert(mid != nullptr);
return mid;
}
};
// The portal class for org.rocksdb.AbstractSlice
@ -7109,5 +7263,42 @@ class WalProcessingOptionJni {
}
}
};
// The portal class for org.rocksdb.ReusedSynchronisationType
class ReusedSynchronisationTypeJni {
public:
// Returns the equivalent org.rocksdb.ReusedSynchronisationType for the provided
// C++ rocksdb::ReusedSynchronisationType enum
static jbyte toJavaReusedSynchronisationType(
const rocksdb::ReusedSynchronisationType& reused_synchronisation_type) {
switch(reused_synchronisation_type) {
case rocksdb::ReusedSynchronisationType::MUTEX:
return 0x0;
case rocksdb::ReusedSynchronisationType::ADAPTIVE_MUTEX:
return 0x1;
case rocksdb::ReusedSynchronisationType::THREAD_LOCAL:
return 0x2;
default:
return 0x7F; // undefined
}
}
// Returns the equivalent C++ rocksdb::ReusedSynchronisationType enum for the
// provided Java org.rocksdb.ReusedSynchronisationType
static rocksdb::ReusedSynchronisationType toCppReusedSynchronisationType(
jbyte reused_synchronisation_type) {
switch(reused_synchronisation_type) {
case 0x0:
return rocksdb::ReusedSynchronisationType::MUTEX;
case 0x1:
return rocksdb::ReusedSynchronisationType::ADAPTIVE_MUTEX;
case 0x2:
return rocksdb::ReusedSynchronisationType::THREAD_LOCAL;
default:
// undefined/default
return rocksdb::ReusedSynchronisationType::ADAPTIVE_MUTEX;
}
}
};
} // namespace rocksdb
#endif // JAVA_ROCKSJNI_PORTAL_H_

8
java/rocksjni/sst_file_writerjni.cc
View File

@ -33,14 +33,8 @@ jlong Java_org_rocksdb_SstFileWriter_newSstFileWriter__JJJB(
jcomparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
comparator = reinterpret_cast<rocksdb::DirectComparatorJniCallback *>(
jcomparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
comparator = reinterpret_cast<rocksdb::Comparator *>(jcomparator_handle);
break;
}

9
java/rocksjni/write_batch_with_index.cc
View File

@ -51,15 +51,8 @@ jlong Java_org_rocksdb_WriteBatchWithIndex_newWriteBatchWithIndex__JBIZ(
jfallback_index_comparator_handle);
break;
// JAVA_DIRECT_COMPARATOR
case 0x1:
fallback_comparator =
reinterpret_cast<rocksdb::DirectComparatorJniCallback*>(
jfallback_index_comparator_handle);
break;
// JAVA_NATIVE_COMPARATOR_WRAPPER
case 0x2:
case 0x1:
fallback_comparator = reinterpret_cast<rocksdb::Comparator*>(
jfallback_index_comparator_handle);
break;

79
java/src/main/java/org/rocksdb/AbstractComparator.java
View File

@ -5,19 +5,18 @@
package org.rocksdb;
import java.nio.ByteBuffer;
/**
* Comparators are used by RocksDB to determine
* the ordering of keys.
*
* This class is package private, implementers
* should extend either of the public abstract classes:
* @see org.rocksdb.Comparator
* @see org.rocksdb.DirectComparator
* Implementations of Comparators in Java should extend this class.
*/
public abstract class AbstractComparator<T extends AbstractSlice<?>>
public abstract class AbstractComparator
extends RocksCallbackObject {
protected AbstractComparator() {
AbstractComparator() {
super();
}
@ -25,6 +24,11 @@ public abstract class AbstractComparator<T extends AbstractSlice<?>>
super(copt.nativeHandle_);
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewComparator(nativeParameterHandles[0]);
}
/**
* Get the type of this comparator.
*
@ -32,7 +36,9 @@ public abstract class AbstractComparator<T extends AbstractSlice<?>>
*
* @return The type of the comparator.
*/
abstract ComparatorType getComparatorType();
ComparatorType getComparatorType() {
return ComparatorType.JAVA_COMPARATOR;
}
/**
* The name of the comparator. Used to check for comparator
@ -50,54 +56,69 @@ public abstract class AbstractComparator<T extends AbstractSlice<?>>
public abstract String name();
/**
* Three-way key comparison
* Three-way key comparison. Implementations should provide a
* <a href="https://en.wikipedia.org/wiki/Total_order">total order</a>
* on keys that might be passed to it.
*
* @param a Slice access to first key
* @param b Slice access to second key
* The implementation may modify the {@code ByteBuffer}s passed in, though
* it would be unconventional to modify the "limit" or any of the
* underlying bytes. As a callback, RocksJava will ensure that {@code a}
* is a different instance from {@code b}.
*
* @return Should return either:
* @param a buffer containing the first key in its "remaining" elements
* @param b buffer containing the second key in its "remaining" elements
*
* @return Should return either:
* 1) &lt; 0 if "a" &lt; "b"
* 2) == 0 if "a" == "b"
* 3) &gt; 0 if "a" &gt; "b"
*/
public abstract int compare(final T a, final T b);
public abstract int compare(final ByteBuffer a, final ByteBuffer b);
/**
* <p>Used to reduce the space requirements
* for internal data structures like index blocks.</p>
*
* <p>If start &lt; limit, you may return a new start which is a
* <p>If start &lt; limit, you may modify start which is a
* shorter string in [start, limit).</p>
*
* <p>Simple comparator implementations may return null if they
* wish to use start unchanged. i.e., an implementation of
* this method that does nothing is correct.</p>
* If you modify start, it is expected that you set the byte buffer so that
* a subsequent read of start.remaining() bytes from start.position()
* to start.limit() will obtain the new start value.
*
* @param start String
* @param limit of type T
* <p>Simple comparator implementations may return with start unchanged.
* i.e., an implementation of this method that does nothing is correct.</p>
*
* @return a shorter start, or null
* @param start the start
* @param limit the limit
*/
public String findShortestSeparator(final String start, final T limit) {
return null;
public void findShortestSeparator(final ByteBuffer start,
final ByteBuffer limit) {
// no-op
}
/**
* <p>Used to reduce the space requirements
* for internal data structures like index blocks.</p>
*
* <p>You may return a new short key (key1) where
* <p>You may change key to a shorter key (key1) where
* key1 &ge; key.</p>
*
* <p>Simple comparator implementations may return null if they
* wish to leave the key unchanged. i.e., an implementation of
* <p>Simple comparator implementations may return the key unchanged.
* i.e., an implementation of
* this method that does nothing is correct.</p>
*
* @param key String
*
* @return a shorter key, or null
* @param key the key
*/
public String findShortSuccessor(final String key) {
return null;
public void findShortSuccessor(final ByteBuffer key) {
// no-op
}
public final boolean usingDirectBuffers() {
return usingDirectBuffers(nativeHandle_);
}
private native boolean usingDirectBuffers(final long nativeHandle);
private native long createNewComparator(final long comparatorOptionsHandle);
}

125
java/src/main/java/org/rocksdb/AbstractComparatorJniBridge.java Normal file
View File

@ -0,0 +1,125 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import java.nio.ByteBuffer;
/**
* This class is intentionally private,
* it holds methods which are called
* from C++ to interact with a Comparator
* written in Java.
*
* Placing these bridge methods in this
* class keeps the API of the
* {@link org.rocksdb.AbstractComparator} clean.
*/
class AbstractComparatorJniBridge {
/**
* Only called from JNI.
*
* Simply a bridge to calling
* {@link AbstractComparator#compare(ByteBuffer, ByteBuffer)},
* which ensures that the byte buffer lengths are correct
* before and after the call.
*
* @param comparator the comparator object on which to
* call {@link AbstractComparator#compare(ByteBuffer, ByteBuffer)}
* @param a buffer access to first key
* @param aLen the length of the a key,
* may be smaller than the buffer {@code a}
* @param b buffer access to second key
* @param bLen the length of the b key,
* may be smaller than the buffer {@code b}
*
* @return the result of the comparison
*/
private static int compareInternal(
final AbstractComparator comparator,
final ByteBuffer a, final int aLen,
final ByteBuffer b, final int bLen) {
if (aLen != -1) {
a.mark();
a.limit(aLen);
}
if (bLen != -1) {
b.mark();
b.limit(bLen);
}
final int c = comparator.compare(a, b);
if (aLen != -1) {
a.reset();
}
if (bLen != -1) {
b.reset();
}
return c;
}
/**
* Only called from JNI.
*
* Simply a bridge to calling
* {@link AbstractComparator#findShortestSeparator(ByteBuffer, ByteBuffer)},
* which ensures that the byte buffer lengths are correct
* before the call.
*
* @param comparator the comparator object on which to
* call {@link AbstractComparator#findShortestSeparator(ByteBuffer, ByteBuffer)}
* @param start buffer access to the start key
* @param startLen the length of the start key,
* may be smaller than the buffer {@code start}
* @param limit buffer access to the limit key
* @param limitLen the length of the limit key,
* may be smaller than the buffer {@code limit}
*
* @return either {@code startLen} if the start key is unchanged, otherwise
* the new length of the start key
*/
private static int findShortestSeparatorInternal(
final AbstractComparator comparator,
final ByteBuffer start, final int startLen,
final ByteBuffer limit, final int limitLen) {
if (startLen != -1) {
start.limit(startLen);
}
if (limitLen != -1) {
limit.limit(limitLen);
}
comparator.findShortestSeparator(start, limit);
return start.remaining();
}
/**
* Only called from JNI.
*
* Simply a bridge to calling
* {@link AbstractComparator#findShortestSeparator(ByteBuffer, ByteBuffer)},
* which ensures that the byte buffer length is correct
* before the call.
*
* @param comparator the comparator object on which to
* call {@link AbstractComparator#findShortSuccessor(ByteBuffer)}
* @param key buffer access to the key
* @param keyLen the length of the key,
* may be smaller than the buffer {@code key}
*
* @return either keyLen if the key is unchanged, otherwise the new length of the key
*/
private static int findShortSuccessorInternal(
final AbstractComparator comparator,
final ByteBuffer key, final int keyLen) {
if (keyLen != -1) {
key.limit(keyLen);
}
comparator.findShortSuccessor(key);
return key.remaining();
}
}

4
java/src/main/java/org/rocksdb/ColumnFamilyOptions.java
View File

@ -170,7 +170,7 @@ public class ColumnFamilyOptions extends RocksObject
@Override
public ColumnFamilyOptions setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
assert (isOwningHandle());
setComparatorHandle(nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue());
@ -989,7 +989,7 @@ public class ColumnFamilyOptions extends RocksObject
// NOTE: If you add new member variables, please update the copy constructor above!
private MemTableConfig memTableConfig_;
private TableFormatConfig tableFormatConfig_;
private AbstractComparator<? extends AbstractSlice<?>> comparator_;
private AbstractComparator comparator_;
private AbstractCompactionFilter<? extends AbstractSlice<?>> compactionFilter_;
private AbstractCompactionFilterFactory<? extends AbstractCompactionFilter<?>>
compactionFilterFactory_;

2
java/src/main/java/org/rocksdb/ColumnFamilyOptionsInterface.java
View File

@ -121,7 +121,7 @@ public interface ColumnFamilyOptionsInterface<T extends ColumnFamilyOptionsInter
* @return the instance of the current object.
*/
T setComparator(
AbstractComparator<? extends AbstractSlice<?>> comparator);
AbstractComparator comparator);
/**
* <p>Set the merge operator to be used for merging two merge operands

34
java/src/main/java/org/rocksdb/Comparator.java
View File

@ -1,34 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Base class for comparators which will receive
* byte[] based access via org.rocksdb.Slice in their
* compare method implementation.
*
* byte[] based slices perform better when small keys
* are involved. When using larger keys consider
* using @see org.rocksdb.DirectComparator
*/
public abstract class Comparator extends AbstractComparator<Slice> {
public Comparator(final ComparatorOptions copt) {
super(copt);
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewComparator0(nativeParameterHandles[0]);
}
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_COMPARATOR;
}
private native long createNewComparator0(final long comparatorOptionsHandle);
}

121
java/src/main/java/org/rocksdb/ComparatorOptions.java
View File

@ -1,4 +1,7 @@
// Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved.
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
@ -15,38 +18,116 @@ public class ComparatorOptions extends RocksObject {
}
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
* Default: false
* Get the synchronisation type used to guard the reused buffers.
* Only used if {@link #maxReusedBufferSize()} &gt; 0
* Default: {@link ReusedSynchronisationType#ADAPTIVE_MUTEX}
*
* @return true if adaptive mutex is used.
* @return the synchronisation type
*/
public boolean useAdaptiveMutex() {
public ReusedSynchronisationType reusedSynchronisationType() {
assert(isOwningHandle());
return useAdaptiveMutex(nativeHandle_);
return ReusedSynchronisationType.getReusedSynchronisationType(
reusedSynchronisationType(nativeHandle_));
}
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
* Default: false
* Set the synchronisation type used to guard the reused buffers.
* Only used if {@link #maxReusedBufferSize()} &gt; 0
* Default: {@link ReusedSynchronisationType#ADAPTIVE_MUTEX}
*
* @param reusedSynchronisationType the synchronisation type
*
* @param useAdaptiveMutex true if adaptive mutex is used.
* @return the reference to the current comparator options.
*/
public ComparatorOptions setUseAdaptiveMutex(final boolean useAdaptiveMutex) {
public ComparatorOptions setReusedSynchronisationType(
final ReusedSynchronisationType reusedSynchronisationType) {
assert (isOwningHandle());
setUseAdaptiveMutex(nativeHandle_, useAdaptiveMutex);
setReusedSynchronisationType(nativeHandle_,
reusedSynchronisationType.getValue());
return this;
}
/**
* Indicates if a direct byte buffer (i.e. outside of the normal
* garbage-collected heap) is used, as opposed to a non-direct byte buffer
* which is a wrapper around an on-heap byte[].
*
* Default: true
*
* @return true if a direct byte buffer will be used, false otherwise
*/
public boolean useDirectBuffer() {
assert(isOwningHandle());
return useDirectBuffer(nativeHandle_);
}
/**
* Controls whether a direct byte buffer (i.e. outside of the normal
* garbage-collected heap) is used, as opposed to a non-direct byte buffer
* which is a wrapper around an on-heap byte[].
*
* Default: true
*
* @param useDirectBuffer true if a direct byte buffer should be used,
* false otherwise
* @return the reference to the current comparator options.
*/
public ComparatorOptions setUseDirectBuffer(final boolean useDirectBuffer) {
assert(isOwningHandle());
setUseDirectBuffer(nativeHandle_, useDirectBuffer);
return this;
}
/**
* Maximum size of a buffer (in bytes) that will be reused.
* Comparators will use 5 of these buffers,
* so the retained memory size will be 5 * max_reused_buffer_size.
* When a buffer is needed for transferring data to a callback,
* if it requires less than {@code maxReuseBufferSize}, then an
* existing buffer will be reused, else a new buffer will be
* allocated just for that callback.
*
* Default: 64 bytes
*
* @return the maximum size of a buffer which is reused,
* or 0 if reuse is disabled
*/
public int maxReusedBufferSize() {
assert(isOwningHandle());
return maxReusedBufferSize(nativeHandle_);
}
/**
* Sets the maximum size of a buffer (in bytes) that will be reused.
* Comparators will use 5 of these buffers,
* so the retained memory size will be 5 * max_reused_buffer_size.
* When a buffer is needed for transferring data to a callback,
* if it requires less than {@code maxReuseBufferSize}, then an
* existing buffer will be reused, else a new buffer will be
* allocated just for that callback.
*
* Default: 64 bytes
*
* @param maxReusedBufferSize the maximum size for a buffer to reuse, or 0 to
* disable reuse
*
* @return the maximum size of a buffer which is reused
*/
public ComparatorOptions setMaxReusedBufferSize(final int maxReusedBufferSize) {
assert(isOwningHandle());
setMaxReusedBufferSize(nativeHandle_, maxReusedBufferSize);
return this;
}
private native static long newComparatorOptions();
private native boolean useAdaptiveMutex(final long handle);
private native void setUseAdaptiveMutex(final long handle,
final boolean useAdaptiveMutex);
private native byte reusedSynchronisationType(final long handle);
private native void setReusedSynchronisationType(final long handle,
final byte reusedSynchronisationType);
private native boolean useDirectBuffer(final long handle);
private native void setUseDirectBuffer(final long handle,
final boolean useDirectBuffer);
private native int maxReusedBufferSize(final long handle);
private native void setMaxReusedBufferSize(final long handle,
final int maxReuseBufferSize);
@Override protected final native void disposeInternal(final long handle);
}

3
java/src/main/java/org/rocksdb/ComparatorType.java
View File

@ -7,8 +7,7 @@ package org.rocksdb;
enum ComparatorType {
JAVA_COMPARATOR((byte)0x0),
JAVA_DIRECT_COMPARATOR((byte)0x1),
JAVA_NATIVE_COMPARATOR_WRAPPER((byte)0x2);
JAVA_NATIVE_COMPARATOR_WRAPPER((byte)0x1);
private final byte value;

35
java/src/main/java/org/rocksdb/DirectComparator.java
View File

@ -1,35 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Base class for comparators which will receive
* ByteBuffer based access via org.rocksdb.DirectSlice
* in their compare method implementation.
*
* ByteBuffer based slices perform better when large keys
* are involved. When using smaller keys consider
* using @see org.rocksdb.Comparator
*/
public abstract class DirectComparator extends AbstractComparator<DirectSlice> {
public DirectComparator(final ComparatorOptions copt) {
super(copt);
}
@Override
protected long initializeNative(final long... nativeParameterHandles) {
return createNewDirectComparator0(nativeParameterHandles[0]);
}
@Override
final ComparatorType getComparatorType() {
return ComparatorType.JAVA_DIRECT_COMPARATOR;
}
private native long createNewDirectComparator0(
final long comparatorOptionsHandle);
}

12
java/src/main/java/org/rocksdb/NativeComparatorWrapper.java
View File

@ -5,6 +5,8 @@
package org.rocksdb;
import java.nio.ByteBuffer;
/**
* A simple abstraction to allow a Java class to wrap a custom comparator
* implemented in C++.
@ -12,7 +14,7 @@ package org.rocksdb;
* The native comparator must directly extend rocksdb::Comparator.
*/
public abstract class NativeComparatorWrapper
extends AbstractComparator<Slice> {
extends AbstractComparator {
@Override
final ComparatorType getComparatorType() {
@ -26,26 +28,26 @@ public abstract class NativeComparatorWrapper
}
@Override
public final int compare(final Slice s1, final Slice s2) {
public final int compare(final ByteBuffer s1, final ByteBuffer s2) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final String findShortestSeparator(final String start, final Slice limit) {
public final void findShortestSeparator(final ByteBuffer start, final ByteBuffer limit) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
@Override
public final String findShortSuccessor(final String key) {
public final void findShortSuccessor(final ByteBuffer key) {
throw new IllegalStateException("This should not be called. " +
"Implementation is in Native code");
}
/**
* We override {@link RocksCallbackObject#disposeInternal()}
* as disposing of a native rocksd::Comparator extension requires
* as disposing of a native rocksdb::Comparator extension requires
* a slightly different approach as it is not really a RocksCallbackObject
*/
@Override

2
java/src/main/java/org/rocksdb/OptimisticTransactionOptions.java
View File

@ -37,7 +37,7 @@ public class OptimisticTransactionOptions extends RocksObject
* @return this OptimisticTransactionOptions instance
*/
public OptimisticTransactionOptions setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
assert(isOwningHandle());
setComparator(nativeHandle_, comparator.nativeHandle_);
return this;

4
java/src/main/java/org/rocksdb/Options.java
View File

@ -193,7 +193,7 @@ public class Options extends RocksObject
@Override
public Options setComparator(
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
assert(isOwningHandle());
setComparatorHandle(nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue());
@ -2169,7 +2169,7 @@ public class Options extends RocksObject
private MemTableConfig memTableConfig_;
private TableFormatConfig tableFormatConfig_;
private RateLimiter rateLimiter_;
private AbstractComparator<? extends AbstractSlice<?>> comparator_;
private AbstractComparator comparator_;
private AbstractCompactionFilter<? extends AbstractSlice<?>> compactionFilter_;
private AbstractCompactionFilterFactory<? extends AbstractCompactionFilter<?>>
compactionFilterFactory_;

65
java/src/main/java/org/rocksdb/ReusedSynchronisationType.java Normal file
View File

@ -0,0 +1,65 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
/**
* Determines the type of synchronisation primitive used
* in native code.
*/
public enum ReusedSynchronisationType {
/**
* Standard mutex.
*/
MUTEX((byte)0x0),
/**
* Use adaptive mutex, which spins in the user space before resorting
* to kernel. This could reduce context switch when the mutex is not
* heavily contended. However, if the mutex is hot, we could end up
* wasting spin time.
*/
ADAPTIVE_MUTEX((byte)0x1),
/**
* There is a reused buffer per-thread.
*/
THREAD_LOCAL((byte)0x2);
private final byte value;
ReusedSynchronisationType(final byte value) {
this.value = value;
}
/**
* Returns the byte value of the enumerations value
*
* @return byte representation
*/
public byte getValue() {
return value;
}
/**
* Get ReusedSynchronisationType by byte value.
*
* @param value byte representation of ReusedSynchronisationType.
*
* @return {@link org.rocksdb.ReusedSynchronisationType} instance.
* @throws java.lang.IllegalArgumentException if an invalid
* value is provided.
*/
public static ReusedSynchronisationType getReusedSynchronisationType(
final byte value) {
for (final ReusedSynchronisationType reusedSynchronisationType
: ReusedSynchronisationType.values()) {
if (reusedSynchronisationType.getValue() == value) {
return reusedSynchronisationType;
}
}
throw new IllegalArgumentException(
"Illegal value provided for ReusedSynchronisationType.");
}
}

2
java/src/main/java/org/rocksdb/SstFileWriter.java
View File

@ -28,7 +28,7 @@ public class SstFileWriter extends RocksObject {
*/
@Deprecated
public SstFileWriter(final EnvOptions envOptions, final Options options,
final AbstractComparator<? extends AbstractSlice<?>> comparator) {
final AbstractComparator comparator) {
super(newSstFileWriter(
envOptions.nativeHandle_, options.nativeHandle_, comparator.nativeHandle_,
comparator.getComparatorType().getValue()));

2
java/src/main/java/org/rocksdb/WriteBatchWithIndex.java
View File

@ -57,7 +57,7 @@ public class WriteBatchWithIndex extends AbstractWriteBatch {
* show two entries with the same key.
*/
public WriteBatchWithIndex(
final AbstractComparator<? extends AbstractSlice<?>>
final AbstractComparator
fallbackIndexComparator, final int reservedBytes,
final boolean overwriteKey) {
super(newWriteBatchWithIndex(fallbackIndexComparator.nativeHandle_,

46
java/src/main/java/org/rocksdb/util/ByteUtil.java Normal file
View File

@ -0,0 +1,46 @@
package org.rocksdb.util;
import java.nio.ByteBuffer;
import static java.nio.charset.StandardCharsets.UTF_8;
public class ByteUtil {
/**
* Convert a String to a UTF-8 byte array.
*
* @param str the string
*
* @return the byte array.
*/
public static byte[] bytes(final String str) {
return str.getBytes(UTF_8);
}
/**
* Compares the first {@code count} bytes of two areas of memory. Returns
* zero if they are the same, a value less than zero if {@code x} is
* lexically less than {@code y}, or a value greater than zero if {@code x}
* is lexically greater than {@code y}. Note that lexical order is determined
* as if comparing unsigned char arrays.
*
* Similar to <a href="https://github.com/gcc-mirror/gcc/blob/master/libiberty/memcmp.c">memcmp.c</a>.
*
* @param x the first value to compare with
* @param y the second value to compare against
* @param count the number of bytes to compare
*
* @return the result of the comparison
*/
public static int memcmp(final ByteBuffer x, final ByteBuffer y,
final int count) {
for (int idx = 0; idx < count; idx++) {
final int aa = x.get(idx) & 0xff;
final int bb = y.get(idx) & 0xff;
if (aa != bb) {
return aa - bb;
}
}
return 0;
}
}

116
java/src/main/java/org/rocksdb/util/BytewiseComparator.java
View File

@ -9,6 +9,8 @@ import org.rocksdb.*;
import java.nio.ByteBuffer;
import static org.rocksdb.util.ByteUtil.memcmp;
/**
* This is a Java Native implementation of the C++
* equivalent BytewiseComparatorImpl using {@link Slice}
@ -19,7 +21,7 @@ import java.nio.ByteBuffer;
* and you most likely instead wanted
* {@link org.rocksdb.BuiltinComparator#BYTEWISE_COMPARATOR}
*/
public class BytewiseComparator extends Comparator {
public final class BytewiseComparator extends AbstractComparator {
public BytewiseComparator(final ComparatorOptions copt) {
super(copt);
@ -31,61 +33,89 @@ public class BytewiseComparator extends Comparator {
}
@Override
public int compare(final Slice a, final Slice b) {
return compare(a.data(), b.data());
public int compare(final ByteBuffer a, final ByteBuffer b) {
return _compare(a, b);
}
@Override
public String findShortestSeparator(final String start,
final Slice limit) {
final byte[] startBytes = start.getBytes();
final byte[] limitBytes = limit.data();
// Find length of common prefix
final int min_length = Math.min(startBytes.length, limit.size());
int diff_index = 0;
while ((diff_index < min_length) &&
(startBytes[diff_index] == limitBytes[diff_index])) {
diff_index++;
}
if (diff_index >= min_length) {
// Do not shorten if one string is a prefix of the other
} else {
final byte diff_byte = startBytes[diff_index];
if(diff_byte < 0xff && diff_byte + 1 < limitBytes[diff_index]) {
final byte shortest[] = new byte[diff_index + 1];
System.arraycopy(startBytes, 0, shortest, 0, diff_index + 1);
shortest[diff_index]++;
assert(compare(shortest, limitBytes) < 0);
return new String(shortest);
static int _compare(final ByteBuffer a, final ByteBuffer b) {
assert(a != null && b != null);
final int minLen = a.remaining() < b.remaining() ?
a.remaining() : b.remaining();
int r = memcmp(a, b, minLen);
if (r == 0) {
if (a.remaining() < b.remaining()) {
r = -1;
} else if (a.remaining() > b.remaining()) {
r = +1;
}
}
return null;
}
private static int compare(final byte[] a, final byte[] b) {
return ByteBuffer.wrap(a).compareTo(ByteBuffer.wrap(b));
return r;
}
@Override
public String findShortSuccessor(final String key) {
final byte[] keyBytes = key.getBytes();
public void findShortestSeparator(final ByteBuffer start,
final ByteBuffer limit) {
// Find length of common prefix
final int minLength = Math.min(start.remaining(), limit.remaining());
int diffIndex = 0;
while (diffIndex < minLength &&
start.get(diffIndex) == limit.get(diffIndex)) {
diffIndex++;
}
if (diffIndex >= minLength) {
// Do not shorten if one string is a prefix of the other
} else {
final int startByte = start.get(diffIndex) & 0xff;
final int limitByte = limit.get(diffIndex) & 0xff;
if (startByte >= limitByte) {
// Cannot shorten since limit is smaller than start or start is
// already the shortest possible.
return;
}
assert(startByte < limitByte);
if (diffIndex < limit.remaining() - 1 || startByte + 1 < limitByte) {
start.put(diffIndex, (byte)((start.get(diffIndex) & 0xff) + 1));
start.limit(diffIndex + 1);
} else {
// v
// A A 1 A A A
// A A 2
//
// Incrementing the current byte will make start bigger than limit, we
// will skip this byte, and find the first non 0xFF byte in start and
// increment it.
diffIndex++;
while (diffIndex < start.remaining()) {
// Keep moving until we find the first non 0xFF byte to
// increment it
if ((start.get(diffIndex) & 0xff) <
0xff) {
start.put(diffIndex, (byte)((start.get(diffIndex) & 0xff) + 1));
start.limit(diffIndex + 1);
break;
}
diffIndex++;
}
}
assert(compare(start.duplicate(), limit.duplicate()) < 0);
}
}
@Override
public void findShortSuccessor(final ByteBuffer key) {
// Find first character that can be incremented
final int n = keyBytes.length;
final int n = key.remaining();
for (int i = 0; i < n; i++) {
final byte byt = keyBytes[i];
final int byt = key.get(i) & 0xff;
if (byt != 0xff) {
final byte shortSuccessor[] = new byte[i + 1];
System.arraycopy(keyBytes, 0, shortSuccessor, 0, i + 1);
shortSuccessor[i]++;
return new String(shortSuccessor);
key.put(i, (byte)(byt + 1));
key.limit(i+1);
return;
}
}
// *key is a run of 0xffs. Leave it alone.
return null;
}
}

88
java/src/main/java/org/rocksdb/util/DirectBytewiseComparator.java
View File

@ -1,88 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.rocksdb.ComparatorOptions;
import org.rocksdb.DirectComparator;
import org.rocksdb.DirectSlice;
import java.nio.ByteBuffer;
/**
* This is a Java Native implementation of the C++
* equivalent BytewiseComparatorImpl using {@link DirectSlice}
*
* The performance of Comparators implemented in Java is always
* less than their C++ counterparts due to the bridging overhead,
* as such you likely don't want to use this apart from benchmarking
* and you most likely instead wanted
* {@link org.rocksdb.BuiltinComparator#BYTEWISE_COMPARATOR}
*/
public class DirectBytewiseComparator extends DirectComparator {
public DirectBytewiseComparator(final ComparatorOptions copt) {
super(copt);
}
@Override
public String name() {
return "rocksdb.java.DirectBytewiseComparator";
}
@Override
public int compare(final DirectSlice a, final DirectSlice b) {
return a.data().compareTo(b.data());
}
@Override
public String findShortestSeparator(final String start,
final DirectSlice limit) {
final byte[] startBytes = start.getBytes();
// Find length of common prefix
final int min_length = Math.min(startBytes.length, limit.size());
int diff_index = 0;
while ((diff_index < min_length) &&
(startBytes[diff_index] == limit.get(diff_index))) {
diff_index++;
}
if (diff_index >= min_length) {
// Do not shorten if one string is a prefix of the other
} else {
final byte diff_byte = startBytes[diff_index];
if(diff_byte < 0xff && diff_byte + 1 < limit.get(diff_index)) {
final byte shortest[] = new byte[diff_index + 1];
System.arraycopy(startBytes, 0, shortest, 0, diff_index + 1);
shortest[diff_index]++;
assert(ByteBuffer.wrap(shortest).compareTo(limit.data()) < 0);
return new String(shortest);
}
}
return null;
}
@Override
public String findShortSuccessor(final String key) {
final byte[] keyBytes = key.getBytes();
// Find first character that can be incremented
final int n = keyBytes.length;
for (int i = 0; i < n; i++) {
final byte byt = keyBytes[i];
if (byt != 0xff) {
final byte shortSuccessor[] = new byte[i + 1];
System.arraycopy(keyBytes, 0, shortSuccessor, 0, i + 1);
shortSuccessor[i]++;
return new String(shortSuccessor);
}
}
// *key is a run of 0xffs. Leave it alone.
return null;
}
}

67
java/src/main/java/org/rocksdb/util/IntComparator.java Normal file
View File

@ -0,0 +1,67 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.rocksdb.AbstractComparator;
import org.rocksdb.ComparatorOptions;
import java.nio.ByteBuffer;
/**
* This is a Java implementation of a Comparator for Java int
* keys.
*
* This comparator assumes keys are (at least) four bytes, so
* the caller must guarantee that in accessing other APIs in
* combination with this comparator.
*
* The performance of Comparators implemented in Java is always
* less than their C++ counterparts due to the bridging overhead,
* as such you likely don't want to use this apart from benchmarking
* or testing.
*/
public final class IntComparator extends AbstractComparator {
public IntComparator(final ComparatorOptions copt) {
super(copt);
}
@Override
public String name() {
return "rocksdb.java.IntComparator";
}
@Override
public int compare(final ByteBuffer a, final ByteBuffer b) {
return compareIntKeys(a, b);
}
/**
* Compares integer keys
* so that they are in ascending order
*
* @param a 4-bytes representing an integer key
* @param b 4-bytes representing an integer key
*
* @return negative if a &lt; b, 0 if a == b, positive otherwise
*/
private final int compareIntKeys(final ByteBuffer a, final ByteBuffer b) {
final int iA = a.getInt();
final int iB = b.getInt();
// protect against int key calculation overflow
final long diff = (long)iA - iB;
final int result;
if (diff < Integer.MIN_VALUE) {
result = Integer.MIN_VALUE;
} else if(diff > Integer.MAX_VALUE) {
result = Integer.MAX_VALUE;
} else {
result = (int)diff;
}
return result;
}
}

57
java/src/main/java/org/rocksdb/util/ReverseBytewiseComparator.java
View File

@ -5,10 +5,13 @@
package org.rocksdb.util;
import org.rocksdb.AbstractComparator;
import org.rocksdb.BuiltinComparator;
import org.rocksdb.ComparatorOptions;
import org.rocksdb.Slice;
import java.nio.ByteBuffer;
/**
* This is a Java Native implementation of the C++
* equivalent ReverseBytewiseComparatorImpl using {@link Slice}
@ -19,7 +22,7 @@ import org.rocksdb.Slice;
* and you most likely instead wanted
* {@link BuiltinComparator#REVERSE_BYTEWISE_COMPARATOR}
*/
public class ReverseBytewiseComparator extends BytewiseComparator {
public final class ReverseBytewiseComparator extends AbstractComparator {
public ReverseBytewiseComparator(final ComparatorOptions copt) {
super(copt);
@ -31,7 +34,55 @@ public class ReverseBytewiseComparator extends BytewiseComparator {
}
@Override
public int compare(final Slice a, final Slice b) {
return -super.compare(a, b);
public int compare(final ByteBuffer a, final ByteBuffer b) {
return -BytewiseComparator._compare(a, b);
}
@Override
public void findShortestSeparator(final ByteBuffer start,
final ByteBuffer limit) {
// Find length of common prefix
final int minLength = Math.min(start.remaining(), limit.remaining());
int diffIndex = 0;
while (diffIndex < minLength &&
start.get(diffIndex) == limit.get(diffIndex)) {
diffIndex++;
}
assert(diffIndex <= minLength);
if (diffIndex == minLength) {
// Do not shorten if one string is a prefix of the other
//
// We could handle cases like:
// V
// A A 2 X Y
// A A 2
// in a similar way as BytewiseComparator::FindShortestSeparator().
// We keep it simple by not implementing it. We can come back to it
// later when needed.
} else {
final int startByte = start.get(diffIndex) & 0xff;
final int limitByte = limit.get(diffIndex) & 0xff;
if (startByte > limitByte && diffIndex < start.remaining() - 1) {
// Case like
// V
// A A 3 A A
// A A 1 B B
//
// or
// v
// A A 2 A A
// A A 1 B B
// In this case "AA2" will be good.
//#ifndef NDEBUG
// std::string old_start = *start;
//#endif
start.limit(diffIndex + 1);
//#ifndef NDEBUG
// assert(old_start >= *start);
//#endif
assert(BytewiseComparator._compare(start.duplicate(), limit.duplicate()) > 0);
}
}
}
}

199
java/src/test/java/org/rocksdb/AbstractComparatorTest.java
View File

@ -1,199 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import java.io.IOException;
import java.nio.file.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static org.assertj.core.api.Assertions.assertThat;
import static org.rocksdb.Types.byteToInt;
import static org.rocksdb.Types.intToByte;
/**
* Abstract tests for both Comparator and DirectComparator
*/
public abstract class AbstractComparatorTest<T extends AbstractSlice<?>> {
/**
* Get a comparator which will expect Integer keys
* and determine an ascending order
*
* @return An integer ascending order key comparator
*/
public abstract AbstractComparator<T> getAscendingIntKeyComparator();
/**
* Test which stores random keys into the database
* using an @see getAscendingIntKeyComparator
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @throws java.io.IOException if IO error happens.
*/
public void testRoundtrip(final Path db_path) throws IOException,
RocksDBException {
try (final AbstractComparator<T> comparator = getAscendingIntKeyComparator();
final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store 10,000 random integer keys
final int ITERATIONS = 10000;
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
final Random random = new Random();
for (int i = 0; i < ITERATIONS; i++) {
final byte[] key = intToByte(random.nextInt());
// does key already exist (avoid duplicates)
if (i > 0 && db.get(key) != null) {
i--; // generate a different key
} else {
db.put(key, "value".getBytes());
}
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
final int thisKey = byteToInt(it.key());
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(ITERATIONS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an @see getAscendingIntKeyComparator
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @throws java.io.IOException if IO error happens.
*/
public void testRoundtripCf(final Path db_path) throws IOException,
RocksDBException {
try(final AbstractComparator<T> comparator = getAscendingIntKeyComparator()) {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions().setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions().
setCreateIfMissing(true).
setCreateMissingColumnFamilies(true)) {
// store 10,000 random integer keys
final int ITERATIONS = 10000;
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
final Random random = new Random();
for (int i = 0; i < ITERATIONS; i++) {
final byte key[] = intToByte(random.nextInt());
if (i > 0 && db.get(cfHandles.get(1), key) != null) {
// does key already exist (avoid duplicates)
i--; // generate a different key
} else {
db.put(cfHandles.get(1), key, "value".getBytes());
}
}
} finally {
for (final ColumnFamilyHandle handle : cfHandles) {
handle.close();
}
}
cfHandles.clear();
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
final int thisKey = byteToInt(it.key());
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(ITERATIONS);
} finally {
for (final ColumnFamilyHandle handle : cfHandles) {
handle.close();
}
}
cfHandles.clear();
}
}
}
}
/**
* Compares integer keys
* so that they are in ascending order
*
* @param a 4-bytes representing an integer key
* @param b 4-bytes representing an integer key
*
* @return negative if a &lt; b, 0 if a == b, positive otherwise
*/
protected final int compareIntKeys(final byte[] a, final byte[] b) {
final int iA = byteToInt(a);
final int iB = byteToInt(b);
// protect against int key calculation overflow
final double diff = (double)iA - iB;
final int result;
if (diff < Integer.MIN_VALUE) {
result = Integer.MIN_VALUE;
} else if(diff > Integer.MAX_VALUE) {
result = Integer.MAX_VALUE;
} else {
result = (int)diff;
}
return result;
}
}

65
java/src/test/java/org/rocksdb/ComparatorTest.java → java/src/test/java/org/rocksdb/BuiltinComparatorTest.java
View File

@ -10,12 +10,9 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.IOException;
import java.nio.file.FileSystems;
import static org.assertj.core.api.Assertions.assertThat;
public class ComparatorTest {
public class BuiltinComparatorTest {
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
@ -24,64 +21,12 @@ public class ComparatorTest {
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparator() throws IOException, RocksDBException {
final AbstractComparatorTest<Slice> comparatorTest = new AbstractComparatorTest<Slice>() {
@Override
public AbstractComparator<Slice> getAscendingIntKeyComparator() {
return new Comparator(new ComparatorOptions()) {
@Override
public String name() {
return "test.AscendingIntKeyComparator";
}
@Override
public int compare(final Slice a, final Slice b) {
return compareIntKeys(a.data(), b.data());
}
};
}
};
// test the round-tripability of keys written and read with the Comparator
comparatorTest.testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()));
}
@Test
public void javaComparatorCf() throws IOException, RocksDBException {
final AbstractComparatorTest<Slice> comparatorTest = new AbstractComparatorTest<Slice>() {
@Override
public AbstractComparator<Slice> getAscendingIntKeyComparator() {
return new Comparator(new ComparatorOptions()) {
@Override
public String name() {
return "test.AscendingIntKeyComparator";
}
@Override
public int compare(final Slice a, final Slice b) {
return compareIntKeys(a.data(), b.data());
}
};
}
};
// test the round-tripability of keys written and read with the Comparator
comparatorTest.testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()));
}
@Test
public void builtinForwardComparator()
throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.BYTEWISE_COMPARATOR);
final RocksDB rocksDb = RocksDB.open(options,
dbFolder.getRoot().getAbsolutePath())
) {
@ -133,8 +78,8 @@ public class ComparatorTest {
public void builtinReverseComparator()
throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
.setCreateIfMissing(true)
.setComparator(BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR);
final RocksDB rocksDb = RocksDB.open(options,
dbFolder.getRoot().getAbsolutePath())
) {

40
java/src/test/java/org/rocksdb/ComparatorOptionsTest.java
View File

@ -17,16 +17,42 @@ public class ComparatorOptionsTest {
new RocksNativeLibraryResource();
@Test
public void comparatorOptions() {
public void reusedSynchronisationType() {
try(final ComparatorOptions copt = new ComparatorOptions()) {
assertThat(copt).isNotNull();
// UseAdaptiveMutex test
copt.setUseAdaptiveMutex(true);
assertThat(copt.useAdaptiveMutex()).isTrue();
copt.setReusedSynchronisationType(ReusedSynchronisationType.MUTEX);
assertThat(copt.reusedSynchronisationType())
.isEqualTo(ReusedSynchronisationType.MUTEX);
copt.setUseAdaptiveMutex(false);
assertThat(copt.useAdaptiveMutex()).isFalse();
copt.setReusedSynchronisationType(ReusedSynchronisationType.ADAPTIVE_MUTEX);
assertThat(copt.reusedSynchronisationType())
.isEqualTo(ReusedSynchronisationType.ADAPTIVE_MUTEX);
copt.setReusedSynchronisationType(ReusedSynchronisationType.THREAD_LOCAL);
assertThat(copt.reusedSynchronisationType())
.isEqualTo(ReusedSynchronisationType.THREAD_LOCAL);
}
}
@Test
public void useDirectBuffer() {
try(final ComparatorOptions copt = new ComparatorOptions()) {
copt.setUseDirectBuffer(true);
assertThat(copt.useDirectBuffer()).isTrue();
copt.setUseDirectBuffer(false);
assertThat(copt.useDirectBuffer()).isFalse();
}
}
@Test
public void maxReusedBufferSize() {
try(final ComparatorOptions copt = new ComparatorOptions()) {
copt.setMaxReusedBufferSize(12345);
assertThat(copt.maxReusedBufferSize()).isEqualTo(12345);
copt.setMaxReusedBufferSize(-1);
assertThat(copt.maxReusedBufferSize()).isEqualTo(-1);
}
}
}

52
java/src/test/java/org/rocksdb/DirectComparatorTest.java
View File

@ -1,52 +0,0 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.IOException;
import java.nio.file.FileSystems;
public class DirectComparatorTest {
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void directComparator() throws IOException, RocksDBException {
final AbstractComparatorTest<DirectSlice> comparatorTest = new AbstractComparatorTest<DirectSlice>() {
@Override
public AbstractComparator<DirectSlice> getAscendingIntKeyComparator() {
return new DirectComparator(new ComparatorOptions()) {
@Override
public String name() {
return "test.AscendingIntKeyDirectComparator";
}
@Override
public int compare(final DirectSlice a, final DirectSlice b) {
final byte ax[] = new byte[4], bx[] = new byte[4];
a.data().get(ax);
b.data().get(bx);
return compareIntKeys(ax, bx);
}
};
}
};
// test the round-tripability of keys written and read with the DirectComparator
comparatorTest.testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()));
}
}

7
java/src/test/java/org/rocksdb/OptimisticTransactionOptionsTest.java
View File

@ -6,7 +6,7 @@
package org.rocksdb;
import org.junit.Test;
import org.rocksdb.util.DirectBytewiseComparator;
import org.rocksdb.util.BytewiseComparator;
import java.util.Random;
@ -29,8 +29,9 @@ public class OptimisticTransactionOptionsTest {
@Test
public void comparator() {
try (final OptimisticTransactionOptions opt = new OptimisticTransactionOptions();
final ComparatorOptions copt = new ComparatorOptions();
final DirectComparator comparator = new DirectBytewiseComparator(copt)) {
final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator = new BytewiseComparator(copt)) {
opt.setComparator(comparator);
}
}

2
java/src/test/java/org/rocksdb/SstFileWriterTest.java
View File

@ -65,7 +65,7 @@ public class SstFileWriterTest {
ComparatorOptions comparatorOptions = null;
BytewiseComparator comparator = null;
if (useJavaBytewiseComparator) {
comparatorOptions = new ComparatorOptions();
comparatorOptions = new ComparatorOptions().setUseDirectBuffer(false);
comparator = new BytewiseComparator(comparatorOptions);
options.setComparator(comparator);
sstFileWriter = new SstFileWriter(envOptions, options);

15
java/src/test/java/org/rocksdb/WalFilterTest.java
View File

@ -16,6 +16,7 @@ import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.rocksdb.util.ByteUtil.bytes;
import static org.rocksdb.util.TestUtil.*;
public class WalFilterTest {
@ -32,23 +33,23 @@ public class WalFilterTest {
// Create 3 batches with two keys each
final byte[][][] batchKeys = {
new byte[][] {
u("key1"),
u("key2")
bytes("key1"),
bytes("key2")
},
new byte[][] {
u("key3"),
u("key4")
bytes("key3"),
bytes("key4")
},
new byte[][] {
u("key5"),
u("key6")
bytes("key5"),
bytes("key6")
}
};
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor(u("pikachu"))
new ColumnFamilyDescriptor(bytes("pikachu"))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();

267
java/src/test/java/org/rocksdb/util/BytewiseComparatorIntTest.java Normal file
View File

@ -0,0 +1,267 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.nio.ByteBuffer;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Similar to {@link IntComparatorTest}, but uses {@link BytewiseComparator}
* which ensures the correct ordering of positive integers.
*/
@RunWith(Parameterized.class)
public class BytewiseComparatorIntTest {
// test with 500 random positive integer keys
private static final int TOTAL_KEYS = 500;
private static final byte[][] keys = new byte[TOTAL_KEYS][4];
@BeforeClass
public static void prepareKeys() {
final ByteBuffer buf = ByteBuffer.allocate(4);
final Random random = new Random();
for (int i = 0; i < TOTAL_KEYS; i++) {
final int ri = random.nextInt() & Integer.MAX_VALUE; // the & ensures positive integer
buf.putInt(ri);
buf.flip();
final byte[] key = buf.array();
// does key already exist (avoid duplicates)
if (keyExists(key, i)) {
i--; // loop round and generate a different key
} else {
System.arraycopy(key, 0, keys[i], 0, 4);
}
}
}
private static boolean keyExists(final byte[] key, final int limit) {
for (int j = 0; j < limit; j++) {
if (Arrays.equals(key, keys[j])) {
return true;
}
}
return false;
}
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "non-direct_reused64_mutex", false, 64, ReusedSynchronisationType.MUTEX },
{ "direct_reused64_mutex", true, 64, ReusedSynchronisationType.MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "non-direct_reused64_thread-local", false, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "direct_reused64_thread-local", true, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "non-direct_noreuse", false, -1, null },
{ "direct_noreuse", true, -1, null }
});
}
@Parameter(0)
public String name;
@Parameter(1)
public boolean useDirectBuffer;
@Parameter(2)
public int maxReusedBufferSize;
@Parameter(3)
public ReusedSynchronisationType reusedSynchronisationType;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparatorDefaultCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final BytewiseComparator comparator = new BytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
@Test
public void javaComparatorNamedCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final BytewiseComparator comparator = new BytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
/**
* Test which stores random keys into the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtrip(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
try (final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store TOTAL_KEYS into the db
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(keys[i], "value".getBytes(UTF_8));
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by IntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtripCf(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions()
.setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions()
.setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true)) {
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(cfHandles.get(1), keys[i], "value".getBytes(UTF_8));
}
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
for (final ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
cfDescriptor.getOptions().close();
}
}
}
}
}
}

209
java/src/test/java/org/rocksdb/util/BytewiseComparatorTest.java
View File

@ -5,20 +5,20 @@
package org.rocksdb.util;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.rocksdb.*;
import org.rocksdb.Comparator;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.ByteBuffer;
import java.nio.file.*;
import java.util.*;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.Assert.*;
import static org.rocksdb.util.ByteUtil.bytes;
/**
* This is a direct port of various C++
@ -27,6 +27,13 @@ import static org.junit.Assert.*;
*/
public class BytewiseComparatorTest {
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
private List<String> source_strings = Arrays.asList("b", "d", "f", "h", "j", "l");
private List<String> interleaving_strings = Arrays.asList("a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m");
@ -38,13 +45,15 @@ public class BytewiseComparatorTest {
public void java_vs_cpp_bytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final RocksDB db = openDatabase(dbDir,
BuiltinComparator.BYTEWISE_COMPARATOR)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new BytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
@ -53,8 +62,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
@ -67,14 +74,17 @@ public class BytewiseComparatorTest {
public void java_vs_java_bytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
try(final ComparatorOptions copt = new ComparatorOptions();
final Comparator comparator = new BytewiseComparator(copt);
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator = new BytewiseComparator(copt);
final RocksDB db = openDatabase(dbDir, comparator)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new BytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
@ -83,8 +93,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
@ -97,13 +105,15 @@ public class BytewiseComparatorTest {
public void java_vs_cpp_directBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final RocksDB db = openDatabase(dbDir,
BuiltinComparator.BYTEWISE_COMPARATOR)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final DirectComparator comparator2 = new DirectBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
@ -112,8 +122,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
@ -126,14 +134,17 @@ public class BytewiseComparatorTest {
public void java_vs_java_directBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
try (final ComparatorOptions copt = new ComparatorOptions();
final DirectComparator comparator = new DirectBytewiseComparator(copt);
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try (final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator = new BytewiseComparator(copt);
final RocksDB db = openDatabase(dbDir, comparator)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final DirectComparator comparator2 = new DirectBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(true);
final AbstractComparator comparator2 = new BytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
@ -142,8 +153,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
@ -156,13 +165,15 @@ public class BytewiseComparatorTest {
public void java_vs_cpp_reverseBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try(final RocksDB db = openDatabase(dbDir,
BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new ReverseBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new ReverseBytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
@ -171,8 +182,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
@ -185,14 +194,17 @@ public class BytewiseComparatorTest {
public void java_vs_java_reverseBytewiseComparator()
throws IOException, RocksDBException {
for(int rand_seed = 301; rand_seed < 306; rand_seed++) {
final Path dbDir = Files.createTempDirectory("comparator_db_test");
try (final ComparatorOptions copt = new ComparatorOptions();
final Comparator comparator = new ReverseBytewiseComparator(copt);
final Path dbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
try (final ComparatorOptions copt = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator = new ReverseBytewiseComparator(copt);
final RocksDB db = openDatabase(dbDir, comparator)) {
final Random rnd = new Random(rand_seed);
try(final ComparatorOptions copt2 = new ComparatorOptions();
final Comparator comparator2 = new ReverseBytewiseComparator(copt2)) {
try(final ComparatorOptions copt2 = new ComparatorOptions()
.setUseDirectBuffer(false);
final AbstractComparator comparator2 = new ReverseBytewiseComparator(copt2)) {
final java.util.Comparator<String> jComparator = toJavaComparator(comparator2);
doRandomIterationTest(
db,
@ -201,8 +213,6 @@ public class BytewiseComparatorTest {
8, 100, 3
);
}
} finally {
removeData(dbDir);
}
}
}
@ -215,34 +225,38 @@ public class BytewiseComparatorTest {
final TreeMap<String, String> map = new TreeMap<>(javaComparator);
for (int i = 0; i < num_writes; i++) {
if (num_trigger_flush > 0 && i != 0 && i % num_trigger_flush == 0) {
db.flush(new FlushOptions());
}
try (final FlushOptions flushOptions = new FlushOptions();
final WriteOptions writeOptions = new WriteOptions()) {
for (int i = 0; i < num_writes; i++) {
if (num_trigger_flush > 0 && i != 0 && i % num_trigger_flush == 0) {
db.flush(flushOptions);
}
final int type = rnd.nextInt(2);
final int index = rnd.nextInt(source_strings.size());
final String key = source_strings.get(index);
switch (type) {
case 0:
// put
map.put(key, key);
db.put(new WriteOptions(), bytes(key), bytes(key));
break;
case 1:
// delete
if (map.containsKey(key)) {
map.remove(key);
}
db.delete(new WriteOptions(), bytes(key));
break;
final int type = rnd.nextInt(2);
final int index = rnd.nextInt(source_strings.size());
final String key = source_strings.get(index);
switch (type) {
case 0:
// put
map.put(key, key);
db.put(writeOptions, bytes(key), bytes(key));
break;
case 1:
// delete
if (map.containsKey(key)) {
map.remove(key);
}
db.delete(writeOptions, bytes(key));
break;
default:
fail("Should not be able to generate random outside range 1..2");
default:
fail("Should not be able to generate random outside range 1..2");
}
}
}
try(final RocksIterator iter = db.newIterator(new ReadOptions())) {
try (final ReadOptions readOptions = new ReadOptions();
final RocksIterator iter = db.newIterator(readOptions)) {
final KVIter<String, String> result_iter = new KVIter<>(map);
boolean is_valid = false;
@ -300,7 +314,7 @@ public class BytewiseComparatorTest {
assert (type == 6);
final int key_idx = rnd.nextInt(source_strings.size());
final String key = source_strings.get(key_idx);
final byte[] result = db.get(new ReadOptions(), bytes(key));
final byte[] result = db.get(readOptions, bytes(key));
if (!map.containsKey(key)) {
assertNull(result);
} else {
@ -342,7 +356,7 @@ public class BytewiseComparatorTest {
*/
private RocksDB openDatabase(
final Path dbDir,
final AbstractComparator<? extends AbstractSlice<?>> javaComparator)
final AbstractComparator javaComparator)
throws IOException, RocksDBException {
final Options options = new Options()
.setCreateIfMissing(true)
@ -350,50 +364,25 @@ public class BytewiseComparatorTest {
return RocksDB.open(options, dbDir.toAbsolutePath().toString());
}
private void closeDatabase(final RocksDB db) {
db.close();
}
private void removeData(final Path dbDir) throws IOException {
Files.walkFileTree(dbDir, new SimpleFileVisitor<Path>() {
@Override
public FileVisitResult visitFile(
final Path file, final BasicFileAttributes attrs)
throws IOException {
Files.delete(file);
return FileVisitResult.CONTINUE;
}
@Override
public FileVisitResult postVisitDirectory(
final Path dir, final IOException exc) throws IOException {
Files.delete(dir);
return FileVisitResult.CONTINUE;
}
});
}
private byte[] bytes(final String s) {
return s.getBytes(StandardCharsets.UTF_8);
}
private java.util.Comparator<String> toJavaComparator(
final Comparator rocksComparator) {
final AbstractComparator rocksComparator) {
return new java.util.Comparator<String>() {
@Override
public int compare(final String s1, final String s2) {
return rocksComparator.compare(new Slice(s1), new Slice(s2));
}
};
}
private java.util.Comparator<String> toJavaComparator(
final DirectComparator rocksComparator) {
return new java.util.Comparator<String>() {
@Override
public int compare(final String s1, final String s2) {
return rocksComparator.compare(new DirectSlice(s1),
new DirectSlice(s2));
final ByteBuffer bufS1;
final ByteBuffer bufS2;
if (rocksComparator.usingDirectBuffers()) {
bufS1 = ByteBuffer.allocateDirect(s1.length());
bufS2 = ByteBuffer.allocateDirect(s2.length());
} else {
bufS1 = ByteBuffer.allocate(s1.length());
bufS2 = ByteBuffer.allocate(s2.length());
}
bufS1.put(bytes(s1));
bufS1.flip();
bufS2.put(bytes(s2));
bufS2.flip();
return rocksComparator.compare(bufS1, bufS2);
}
};
}
@ -434,7 +423,7 @@ public class BytewiseComparatorTest {
public void seek(final byte[] target) {
for(offset = 0; offset < entries.size(); offset++) {
if(comparator.compare(entries.get(offset).getKey(),
(K)new String(target, StandardCharsets.UTF_8)) >= 0) {
(K)new String(target, UTF_8)) >= 0) {
return;
}
}
@ -445,7 +434,7 @@ public class BytewiseComparatorTest {
public void seekForPrev(final byte[] target) {
for(offset = entries.size()-1; offset >= 0; offset--) {
if(comparator.compare(entries.get(offset).getKey(),
(K)new String(target, StandardCharsets.UTF_8)) <= 0) {
(K)new String(target, UTF_8)) <= 0) {
return;
}
}

266
java/src/test/java/org/rocksdb/util/IntComparatorTest.java Normal file
View File

@ -0,0 +1,266 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.nio.ByteBuffer;
import java.nio.file.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Tests for IntComparator, but more generally
* also for rocksdb::ComparatorJniCallback implementation.
*/
@RunWith(Parameterized.class)
public class IntComparatorTest {
// test with 500 random integer keys
private static final int TOTAL_KEYS = 500;
private static final byte[][] keys = new byte[TOTAL_KEYS][4];
@BeforeClass
public static void prepareKeys() {
final ByteBuffer buf = ByteBuffer.allocate(4);
final Random random = new Random();
for (int i = 0; i < TOTAL_KEYS; i++) {
final int ri = random.nextInt();
buf.putInt(ri);
buf.flip();
final byte[] key = buf.array();
// does key already exist (avoid duplicates)
if (keyExists(key, i)) {
i--; // loop round and generate a different key
} else {
System.arraycopy(key, 0, keys[i], 0, 4);
}
}
}
private static boolean keyExists(final byte[] key, final int limit) {
for (int j = 0; j < limit; j++) {
if (Arrays.equals(key, keys[j])) {
return true;
}
}
return false;
}
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "non-direct_reused64_mutex", false, 64, ReusedSynchronisationType.MUTEX },
{ "direct_reused64_mutex", true, 64, ReusedSynchronisationType.MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "non-direct_reused64_thread-local", false, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "direct_reused64_thread-local", true, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "non-direct_noreuse", false, -1, null },
{ "direct_noreuse", true, -1, null }
});
}
@Parameter(0)
public String name;
@Parameter(1)
public boolean useDirectBuffer;
@Parameter(2)
public int maxReusedBufferSize;
@Parameter(3)
public ReusedSynchronisationType reusedSynchronisationType;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparatorDefaultCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final IntComparator comparator = new IntComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
@Test
public void javaComparatorNamedCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final IntComparator comparator = new IntComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
/**
* Test which stores random keys into the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtrip(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
try (final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store TOTAL_KEYS into the db
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(keys[i], "value".getBytes(UTF_8));
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by IntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtripCf(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions()
.setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions()
.setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true)) {
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(cfHandles.get(1), keys[i], "value".getBytes(UTF_8));
}
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
}
}
// re-open db and read from start to end
// integer keys should be in ascending
// order as defined by SimpleIntComparator
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MIN_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isGreaterThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
for (final ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
cfDescriptor.getOptions().close();
}
}
}
}
}
}

174
java/src/test/java/org/rocksdb/util/JNIComparatorTest.java Normal file
View File

@ -0,0 +1,174 @@
package org.rocksdb.util;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.*;
import java.util.Arrays;
import static org.assertj.core.api.Assertions.assertThat;
@RunWith(Parameterized.class)
public class JNIComparatorTest {
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "bytewise_non-direct", BuiltinComparator.BYTEWISE_COMPARATOR, false },
{ "bytewise_direct", BuiltinComparator.BYTEWISE_COMPARATOR, true },
{ "reverse-bytewise_non-direct", BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR, false },
{ "reverse-bytewise_direct", BuiltinComparator.REVERSE_BYTEWISE_COMPARATOR, true },
});
}
@Parameter(0)
public String name;
@Parameter(1)
public BuiltinComparator builtinComparator;
@Parameter(2)
public boolean useDirectBuffer;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
private static final int MIN = Short.MIN_VALUE - 1;
private static final int MAX = Short.MAX_VALUE + 1;
@Test
public void java_comparator_equals_cpp_comparator() throws RocksDBException, IOException {
final int[] javaKeys;
try (final ComparatorOptions comparatorOptions = new ComparatorOptions();
final AbstractComparator comparator = builtinComparator == BuiltinComparator.BYTEWISE_COMPARATOR
? new BytewiseComparator(comparatorOptions)
: new ReverseBytewiseComparator(comparatorOptions)) {
final Path javaDbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
storeWithJavaComparator(javaDbDir, comparator);
javaKeys = readAllWithJavaComparator(javaDbDir, comparator);
}
final Path cppDbDir =
FileSystems.getDefault().getPath(dbFolder.newFolder().getAbsolutePath());
storeWithCppComparator(cppDbDir, builtinComparator);
final int[] cppKeys =
readAllWithCppComparator(cppDbDir, builtinComparator);
assertThat(javaKeys).isEqualTo(cppKeys);
}
private void storeWithJavaComparator(final Path dir,
final AbstractComparator comparator) throws RocksDBException {
final ByteBuffer buf = ByteBuffer.allocate(4);
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(comparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
for (int i = MIN; i < MAX; i++) {
buf.putInt(i);
buf.flip();
db.put(buf.array(), buf.array());
buf.clear();
}
}
}
private void storeWithCppComparator(final Path dir,
final BuiltinComparator builtinComparator) throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(builtinComparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
final ByteBuffer buf = ByteBuffer.allocate(4);
for (int i = MIN; i < MAX; i++) {
buf.putInt(i);
buf.flip();
db.put(buf.array(), buf.array());
buf.clear();
}
}
}
private int[] readAllWithJavaComparator(final Path dir,
final AbstractComparator comparator) throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(comparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
try (final RocksIterator it = db.newIterator()) {
it.seekToFirst();
final ByteBuffer buf = ByteBuffer.allocate(4);
final int[] keys = new int[MAX - MIN];
int idx = 0;
while (it.isValid()) {
buf.put(it.key());
buf.flip();
final int thisKey = buf.getInt();
keys[idx++] = thisKey;
buf.clear();
it.next();
}
return keys;
}
}
}
private int[] readAllWithCppComparator(final Path dir,
final BuiltinComparator comparator) throws RocksDBException {
try (final Options options = new Options()
.setCreateIfMissing(true)
.setComparator(comparator);
final RocksDB db =
RocksDB.open(options, dir.toAbsolutePath().toString())) {
try (final RocksIterator it = db.newIterator()) {
it.seekToFirst();
final ByteBuffer buf = ByteBuffer.allocate(4);
final int[] keys = new int[MAX - MIN];
int idx = 0;
while (it.isValid()) {
buf.put(it.key());
buf.flip();
final int thisKey = buf.getInt();
keys[idx++] = thisKey;
buf.clear();
it.next();
}
return keys;
}
}
}
}

270
java/src/test/java/org/rocksdb/util/ReverseBytewiseComparatorIntTest.java Normal file
View File

@ -0,0 +1,270 @@
// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
// This source code is licensed under both the GPLv2 (found in the
// COPYING file in the root directory) and Apache 2.0 License
// (found in the LICENSE.Apache file in the root directory).
package org.rocksdb.util;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
import org.rocksdb.*;
import java.nio.ByteBuffer;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.assertj.core.api.Assertions.assertThat;
/**
* Similar to {@link IntComparatorTest}, but uses
* {@link ReverseBytewiseComparator} which ensures the correct reverse
* ordering of positive integers.
*/
@RunWith(Parameterized.class)
public class ReverseBytewiseComparatorIntTest {
// test with 500 random positive integer keys
private static final int TOTAL_KEYS = 500;
private static final byte[][] keys = new byte[TOTAL_KEYS][4];
@BeforeClass
public static void prepareKeys() {
final ByteBuffer buf = ByteBuffer.allocate(4);
final Random random = new Random();
for (int i = 0; i < TOTAL_KEYS; i++) {
final int ri = random.nextInt() & Integer.MAX_VALUE; // the & ensures positive integer
buf.putInt(ri);
buf.flip();
final byte[] key = buf.array();
// does key already exist (avoid duplicates)
if (keyExists(key, i)) {
i--; // loop round and generate a different key
} else {
System.arraycopy(key, 0, keys[i], 0, 4);
}
}
}
private static boolean keyExists(final byte[] key, final int limit) {
for (int j = 0; j < limit; j++) {
if (Arrays.equals(key, keys[j])) {
return true;
}
}
return false;
}
@Parameters(name = "{0}")
public static Iterable<Object[]> parameters() {
return Arrays.asList(new Object[][] {
{ "non-direct_reused64_mutex", false, 64, ReusedSynchronisationType.MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.ADAPTIVE_MUTEX },
{ "non-direct_reused64_adaptive-mutex", false, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "direct_reused64_adaptive-mutex", true, 64, ReusedSynchronisationType.THREAD_LOCAL },
{ "non-direct_noreuse", false, -1, null },
{ "direct_noreuse", true, -1, null }
});
}
@Parameter(0)
public String name;
@Parameter(1)
public boolean useDirectBuffer;
@Parameter(2)
public int maxReusedBufferSize;
@Parameter(3)
public ReusedSynchronisationType reusedSynchronisationType;
@ClassRule
public static final RocksNativeLibraryResource ROCKS_NATIVE_LIBRARY_RESOURCE =
new RocksNativeLibraryResource();
@Rule
public TemporaryFolder dbFolder = new TemporaryFolder();
@Test
public void javaComparatorDefaultCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final ReverseBytewiseComparator comparator =
new ReverseBytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtrip(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
@Test
public void javaComparatorNamedCf() throws RocksDBException {
try (final ComparatorOptions options = new ComparatorOptions()
.setUseDirectBuffer(useDirectBuffer)
.setMaxReusedBufferSize(maxReusedBufferSize)
// if reusedSynchronisationType == null we assume that maxReusedBufferSize <= 0 and so we just set ADAPTIVE_MUTEX, even though it won't be used
.setReusedSynchronisationType(reusedSynchronisationType == null ? ReusedSynchronisationType.ADAPTIVE_MUTEX : reusedSynchronisationType);
final ReverseBytewiseComparator comparator
= new ReverseBytewiseComparator(options)) {
// test the round-tripability of keys written and read with the Comparator
testRoundtripCf(FileSystems.getDefault().getPath(
dbFolder.getRoot().getAbsolutePath()), comparator);
}
}
/**
* Test which stores random keys into the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtrip(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
try (final Options opt = new Options()
.setCreateIfMissing(true)
.setComparator(comparator)) {
// store TOTAL_KEYS into the db
try (final RocksDB db = RocksDB.open(opt, db_path.toString())) {
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(keys[i], "value".getBytes(UTF_8));
}
}
// re-open db and read from start to end
// integer keys should be in descending
// order
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString());
final RocksIterator it = db.newIterator()) {
it.seekToFirst();
int lastKey = Integer.MAX_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isLessThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
}
}
}
/**
* Test which stores random keys into a column family
* in the database
* using an {@link IntComparator}
* it then checks that these keys are read back in
* ascending order
*
* @param db_path A path where we can store database
* files temporarily
*
* @param comparator the comparator
*
* @throws RocksDBException if a database error happens.
*/
private void testRoundtripCf(final Path db_path,
final AbstractComparator comparator) throws RocksDBException {
final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList(
new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY),
new ColumnFamilyDescriptor("new_cf".getBytes(),
new ColumnFamilyOptions()
.setComparator(comparator))
);
final List<ColumnFamilyHandle> cfHandles = new ArrayList<>();
try (final DBOptions opt = new DBOptions()
.setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true)) {
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles)) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
for (int i = 0; i < TOTAL_KEYS; i++) {
db.put(cfHandles.get(1), keys[i], "value".getBytes(UTF_8));
}
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
}
}
// re-open db and read from start to end
// integer keys should be in descending
// order
final ByteBuffer key = ByteBuffer.allocate(4);
try (final RocksDB db = RocksDB.open(opt, db_path.toString(),
cfDescriptors, cfHandles);
final RocksIterator it = db.newIterator(cfHandles.get(1))) {
try {
assertThat(cfDescriptors.size()).isEqualTo(2);
assertThat(cfHandles.size()).isEqualTo(2);
it.seekToFirst();
int lastKey = Integer.MAX_VALUE;
int count = 0;
for (it.seekToFirst(); it.isValid(); it.next()) {
key.put(it.key());
key.flip();
final int thisKey = key.getInt();
key.clear();
assertThat(thisKey).isLessThan(lastKey);
lastKey = thisKey;
count++;
}
assertThat(count).isEqualTo(TOTAL_KEYS);
} finally {
for (final ColumnFamilyHandle cfHandle : cfHandles) {
cfHandle.close();
}
cfHandles.clear();
for (final ColumnFamilyDescriptor cfDescriptor : cfDescriptors) {
cfDescriptor.getOptions().close();
}
}
}
}
}
}

11
java/src/test/java/org/rocksdb/util/TestUtil.java
View File

@ -58,15 +58,4 @@ public class TestUtil {
random.nextBytes(str);
return str;
}
/**
* Convert a UTF-8 String to a byte array.
*
* @param str the string
*
* @return the byte array.
*/
public static byte[] u(final String str) {
return str.getBytes(UTF_8);
}
}