JMH microbenchmarks for RocksJava (#6241)
Summary:
This is the start of some JMH microbenchmarks for RocksJava.
Such benchmarks can help us decide on performance improvements of the Java API.
At the moment, I have only added benchmarks for various Comparator options, as that is one of the first areas where I want to improve performance. I plan to expand this to many more tests.
Details of how to compile and run the benchmarks are in the `README.md`.
A run of these on a XEON 3.5 GHz 4vCPU (QEMU Virtual CPU version 2.5+) / 8GB RAM KVM with Ubuntu 18.04, OpenJDK 1.8.0_232, and gcc 8.3.0 produced the following:
```
# Run complete. Total time: 01:43:17
REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.
Benchmark (comparatorName) Mode Cnt Score Error Units
ComparatorBenchmarks.put native_bytewise thrpt 25 122373.920 ± 2200.538 ops/s
ComparatorBenchmarks.put java_bytewise_adaptive_mutex thrpt 25 17388.201 ± 1444.006 ops/s
ComparatorBenchmarks.put java_bytewise_non-adaptive_mutex thrpt 25 16887.150 ± 1632.204 ops/s
ComparatorBenchmarks.put java_direct_bytewise_adaptive_mutex thrpt 25 15644.572 ± 1791.189 ops/s
ComparatorBenchmarks.put java_direct_bytewise_non-adaptive_mutex thrpt 25 14869.601 ± 2252.135 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 116528.735 ± 4168.797 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_adaptive_mutex thrpt 25 10651.975 ± 545.998 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-adaptive_mutex thrpt 25 10514.224 ± 930.069 ops/s
```
Indicating a ~7x difference between comparators implemented natively (C++) and those implemented in Java. Let's see if we can't improve on that in the near future...
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6241
Differential Revision: D19290410
Pulled By: pdillinger
fbshipit-source-id: 25d44bf3a31de265502ed0c5d8a28cf4c7cb9c0b
2020-01-07 23:44:17 +00:00
|
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
|
|
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
|
|
<modelVersion>4.0.0</modelVersion>
|
|
|
|
|
|
|
|
<groupId>org.rocksdb</groupId>
|
|
|
|
<artifactId>rocksdbjni-jmh</artifactId>
|
|
|
|
<version>1.0-SNAPSHOT</version>
|
|
|
|
|
|
|
|
<url>http://rocksdb.org/</url>
|
|
|
|
|
|
|
|
<name>rocksdbjni-jmh</name>
|
|
|
|
<description>JMH Benchmarks for RocksDB Java API</description>
|
|
|
|
|
|
|
|
<organization>
|
|
|
|
<name>Facebook, Inc.</name>
|
|
|
|
<url>https://www.facebook.com</url>
|
|
|
|
</organization>
|
|
|
|
|
|
|
|
<licenses>
|
|
|
|
<license>
|
|
|
|
<name>Apache License 2.0</name>
|
|
|
|
<url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
|
|
|
|
<distribution>repo</distribution>
|
|
|
|
</license>
|
|
|
|
<license>
|
|
|
|
<name>GNU General Public License, version 2</name>
|
|
|
|
<url>http://www.gnu.org/licenses/gpl-2.0.html</url>
|
|
|
|
<distribution>repo</distribution>
|
|
|
|
</license>
|
|
|
|
</licenses>
|
|
|
|
|
|
|
|
<scm>
|
|
|
|
<connection>scm:git:git://github.com/facebook/rocksdb.git</connection>
|
|
|
|
<developerConnection>scm:git:git@github.com:facebook/rocksdb.git</developerConnection>
|
|
|
|
<url>http://github.com/facebook/rocksdb/</url>
|
|
|
|
</scm>
|
|
|
|
|
|
|
|
<properties>
|
|
|
|
<project.build.source>1.7</project.build.source>
|
|
|
|
<project.build.target>1.7</project.build.target>
|
|
|
|
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
|
|
|
|
|
|
|
<jmh.version>1.22</jmh.version>
|
|
|
|
<uberjar.name>benchmarks</uberjar.name>
|
|
|
|
</properties>
|
|
|
|
|
|
|
|
<dependencies>
|
|
|
|
<dependency>
|
|
|
|
<groupId>org.rocksdb</groupId>
|
|
|
|
<artifactId>rocksdbjni</artifactId>
|
Improve RocksJava Comparator (#6252)
Summary:
This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy.
**NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into.
Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`).
In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`.
In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`.
---
[JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java.
With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425).
These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical.
---
These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x.
```
ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s
ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s
ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s
ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s
```
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252
Differential Revision: D19331064
Pulled By: pdillinger
fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738
2020-02-03 20:28:25 +00:00
|
|
|
<version>6.6.0-SNAPSHOT</version>
|
JMH microbenchmarks for RocksJava (#6241)
Summary:
This is the start of some JMH microbenchmarks for RocksJava.
Such benchmarks can help us decide on performance improvements of the Java API.
At the moment, I have only added benchmarks for various Comparator options, as that is one of the first areas where I want to improve performance. I plan to expand this to many more tests.
Details of how to compile and run the benchmarks are in the `README.md`.
A run of these on a XEON 3.5 GHz 4vCPU (QEMU Virtual CPU version 2.5+) / 8GB RAM KVM with Ubuntu 18.04, OpenJDK 1.8.0_232, and gcc 8.3.0 produced the following:
```
# Run complete. Total time: 01:43:17
REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.
Benchmark (comparatorName) Mode Cnt Score Error Units
ComparatorBenchmarks.put native_bytewise thrpt 25 122373.920 ± 2200.538 ops/s
ComparatorBenchmarks.put java_bytewise_adaptive_mutex thrpt 25 17388.201 ± 1444.006 ops/s
ComparatorBenchmarks.put java_bytewise_non-adaptive_mutex thrpt 25 16887.150 ± 1632.204 ops/s
ComparatorBenchmarks.put java_direct_bytewise_adaptive_mutex thrpt 25 15644.572 ± 1791.189 ops/s
ComparatorBenchmarks.put java_direct_bytewise_non-adaptive_mutex thrpt 25 14869.601 ± 2252.135 ops/s
ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 116528.735 ± 4168.797 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_adaptive_mutex thrpt 25 10651.975 ± 545.998 ops/s
ComparatorBenchmarks.put java_reverse_bytewise_non-adaptive_mutex thrpt 25 10514.224 ± 930.069 ops/s
```
Indicating a ~7x difference between comparators implemented natively (C++) and those implemented in Java. Let's see if we can't improve on that in the near future...
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6241
Differential Revision: D19290410
Pulled By: pdillinger
fbshipit-source-id: 25d44bf3a31de265502ed0c5d8a28cf4c7cb9c0b
2020-01-07 23:44:17 +00:00
|
|
|
</dependency>
|
|
|
|
|
|
|
|
<dependency>
|
|
|
|
<groupId>org.openjdk.jmh</groupId>
|
|
|
|
<artifactId>jmh-core</artifactId>
|
|
|
|
<version>${jmh.version}</version>
|
|
|
|
</dependency>
|
|
|
|
<dependency>
|
|
|
|
<groupId>org.openjdk.jmh</groupId>
|
|
|
|
<artifactId>jmh-generator-annprocess</artifactId>
|
|
|
|
<version>${jmh.version}</version>
|
|
|
|
<scope>provided</scope>
|
|
|
|
</dependency>
|
|
|
|
</dependencies>
|
|
|
|
|
|
|
|
<build>
|
|
|
|
<plugins>
|
|
|
|
<plugin>
|
|
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
|
|
<artifactId>maven-compiler-plugin</artifactId>
|
|
|
|
<version>3.8.1</version>
|
|
|
|
<configuration>
|
|
|
|
<source>${project.build.source}</source>
|
|
|
|
<target>${project.build.target}</target>
|
|
|
|
<encoding>${project.build.sourceEncoding}</encoding>
|
|
|
|
</configuration>
|
|
|
|
</plugin>
|
|
|
|
|
|
|
|
<plugin>
|
|
|
|
<groupId>com.mycila</groupId>
|
|
|
|
<artifactId>license-maven-plugin</artifactId>
|
|
|
|
<version>3.0</version>
|
|
|
|
<inherited>true</inherited>
|
|
|
|
<configuration>
|
|
|
|
<header>LICENSE-HEADER.txt</header>
|
|
|
|
<failIfMissing>true</failIfMissing>
|
|
|
|
<aggregate>true</aggregate>
|
|
|
|
<strictCheck>true</strictCheck>
|
|
|
|
<excludes>
|
|
|
|
<exclude>pom.xml</exclude>
|
|
|
|
</excludes>
|
|
|
|
<encoding>${project.build.sourceEncoding}</encoding>
|
|
|
|
</configuration>
|
|
|
|
</plugin>
|
|
|
|
|
|
|
|
<plugin>
|
|
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
|
|
<artifactId>maven-shade-plugin</artifactId>
|
|
|
|
<version>3.2.1</version>
|
|
|
|
<executions>
|
|
|
|
<execution>
|
|
|
|
<phase>package</phase>
|
|
|
|
<goals>
|
|
|
|
<goal>shade</goal>
|
|
|
|
</goals>
|
|
|
|
<configuration>
|
|
|
|
<finalName>${project.artifactId}-${project.version}-${uberjar.name}</finalName>
|
|
|
|
<transformers>
|
|
|
|
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
|
|
|
|
<mainClass>org.openjdk.jmh.Main</mainClass>
|
|
|
|
</transformer>
|
|
|
|
</transformers>
|
|
|
|
<filters>
|
|
|
|
<filter>
|
|
|
|
<!--
|
|
|
|
Shading signed JARs will fail without this.
|
|
|
|
http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
|
|
|
|
-->
|
|
|
|
<artifact>*:*</artifact>
|
|
|
|
<excludes>
|
|
|
|
<exclude>META-INF/*.SF</exclude>
|
|
|
|
<exclude>META-INF/*.DSA</exclude>
|
|
|
|
<exclude>META-INF/*.RSA</exclude>
|
|
|
|
</excludes>
|
|
|
|
</filter>
|
|
|
|
</filters>
|
|
|
|
</configuration>
|
|
|
|
</execution>
|
|
|
|
</executions>
|
|
|
|
</plugin>
|
|
|
|
|
|
|
|
</plugins>
|
|
|
|
</build>
|
|
|
|
|
|
|
|
</project>
|