LMDB JNI
LMDB JNI provide a Java API to LMDB which is an ultra-fast, ultra-compact key-value embedded data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases. Transactional with full ACID semantics and crash-proof by design. No corruption. No startup time. Zero-config cache tuning. No dependencies. Proven in production applications.
LMDB JNI is available for 64 bit Linux, OSX, Windows and Android.
Documentation
Presentations
- The Lightning Memory-mapped Database, Mar 02, 2016
- Databaseology lecture series at Carnegie-Mellon University, Oct 08, 2015
- LDAP at Lightning Speed, Jul 05, 2015
- The Lightning Memory-Mapped Database, Jun 24, 2013
Benchmarks
-
[In-Memory Microbenchmark] (http://symas.com/mdb/inmem), June 2014
Multithreaded read performance for a purely in-memory database.
-
In-Memory Microbenchmark (Scaling/NUMA), September 2014
Same as above showing performance improvements with
numactl --interleave=allenabled. -
On-Disk Microbenchmark, November 2014
Multithreaded read performance for a database that is over 5 times larger than the size of RAM.
-
[RxLMDB benchmarks] (https://github.com/deephacks/RxLMDB), July 2015
Benchmarks using RxJava and LMDB comparing zero copy, various serialization mechanisms, parallel and skip scans.
-
LMDB JNI Microbenchmark, February 2015 (source)
Row scanning speed per second compared with the Java ports of RocksDB, LevelDB and MapDB. Mongodb is difficult to setup in JMH but de.flapdoodle.embed.mongo indicate that it is around 50x slower than lmdb_zero_copy.
Benchmark Mode Cnt Score Error Units Iteration.leveldb thrpt 10 6965637.351 ± 784589.894 ops/s Iteration.lmdb_buffer_copy thrpt 10 3157796.643 ± 265830.424 ops/s Iteration.lmdb_zero_copy thrpt 10 16372428.882 ± 1812316.504 ops/s Iteration.mapdb thrpt 10 1358748.670 ± 87502.413 ops/s Iteration.rocksdb thrpt 10 1311441.804 ± 176129.883 ops/s
-
LMDB JNI microbenchmark, February 2016
Random gets on a database with 370 million entries of 30GiB on a non-SSD drive. Keys 29 bytes and values 8 bytes. The target machine was busy serving traffic and this was the memory usage before executing the test.
$ free -m total used free shared buffers cached Mem: 32126 31890 235 0 55 9476 -/+ buffers/cache: 22359 9767 Swap: 7627 2350 5277Percentiles measured in nanoseconds using HdrHistogram.
min 0.50 .90 0.99 0.999 0.9999 max 5376 10367 12991 30335 51967 83967 83967 4608 10175 12799 34559 84991 946175 946175 3568 10239 12991 33791 70655 107007 107007
Maven
Windows, Android and OSX support has been discontinued in lmdbjni 0.4.7 (LMDB 0.9.19) and onward. Users can still build releases at their own convenience, but no artifacts will be published to Maven Central.
Please refer to lmdbjava.
<!-- required java classes -->
<dependency>
<groupId>org.deephacks.lmdbjni</groupId>
<artifactId>lmdbjni</artifactId>
<version>${lmdbjni.version}</version>
</dependency>
<!-- prebuilt liblmdb platform packages -->
<dependency>
<groupId>org.deephacks.lmdbjni</groupId>
<artifactId>lmdbjni-linux64</artifactId>
<version>${lmdbjni.version}</version>
</dependency>
<dependency>
<groupId>org.deephacks.lmdbjni</groupId>
<artifactId>lmdbjni-osx64</artifactId>
<version>${lmdbjni.version}</version>
</dependency>
<dependency>
<groupId>org.deephacks.lmdbjni</groupId>
<artifactId>lmdbjni-win64</artifactId>
<version>${lmdbjni.version}</version>
</dependency>
<!-- Android 5.0 (API level 21) 64-bit ARM -->
<dependency>
<groupId>org.deephacks.lmdbjni</groupId>
<artifactId>lmdbjni-android</artifactId>
<version>${lmdbjni.version}</version>
</dependency>
Usage
Recommended package imports.
import org.fusesource.lmdbjni.*;
import static org.fusesource.lmdbjni.Constants.*;
Opening and closing the database.
try (Env env = new Env("/tmp/mydb")) {
try (Database db = env.openDatabase()) {
... // use the db
}
}
Putting, getting, and deleting key/values.
db.put(bytes("Tampa"), bytes("rocks"));
String value = string(db.get(bytes("Tampa")));
db.delete(bytes("Tampa"));
Iterating and seeking key/values forward and backward.
Transaction tx = env.createReadTransaction();
try (EntryIterator it = db.iterate(tx)) {
for (Entry next : it.iterable()) {
}
}
try (EntryIterator it = db.iterateBackward(tx)) {
for (Entry next : it.iterable()) {
}
}
byte[] key = bytes("London");
try (EntryIterator it = db.seek(tx, key)) {
for (Entry next : it.iterable()) {
}
}
try (EntryIterator it = db.seekBackward(tx, key))) {
for (Entry next : it.iterable()) {
}
}
tx.abort();
Performing transactional updates.
try (Transaction tx = env.createWriteTransaction()) {
db.delete(tx, bytes("Denver"));
db.put(tx, bytes("Tampa"), bytes("green"));
db.put(tx, bytes("London"), bytes("red"));
tx.commit(); // if commit is not called, the transaction is aborted
}
Working against a snapshot view of the database using cursors.
// create a read-only transaction...
try (Transaction tx = env.createReadTransaction()) {
// All read operations will now use the same
// consistent view of the data.
... = db.openCursor(tx);
... = db.get(tx, bytes("Tampa"));
}
A cursor in a write-transaction can be closed before its transaction ends, and will otherwise be closed when its transaction ends. A cursor must not be used after its transaction is closed. Both these try blocks are unsafe and may SIGSEGV.
try (Transaction tx = env.createWriteTransaction();
Cursor cursor = db.openCursor(tx)) {
...
tx.commit();
}
try (Transaction tx = env.createWriteTransaction();
EntryIterator it = db.iterate(tx)) {
...
tx.commit();
}
A cursor in a read-only transaction must be closed explicitly, before or after its transaction ends. Both these try blocks are safe.
try (Transaction tx = env.createReadTransaction();
Cursor cursor = db.openCursor(tx)) {
}
try (Transaction tx = env.createReadTransaction();
EntryIterator it = db.iterate(tx)) {
}
Set a custom key comparison function for a database.
db.setComparator(tx, new Comparator<byte[]>() {
@Override
public int compare(byte[] key1, byte[] key2) {
// do compare
}
});
Atomic hot backup.
env.copy(backupPath);
Using a memory pool to make native memory allocations more efficient:
Env.pushMemoryPool(1024 * 512);
try {
// .. work with the DB in here,
} finally {
Env.popMemoryPool();
}
Zero copy usage
The safest (and least efficient) approach for interacting with LMDB JNI is using buffer copy as shown above. BufferCursor is a more efficient, zero copy mode. This mode is not available on Android.
There are also methods that give access to DirectBuffer, but users should avoid interacting directly with these and use the BufferCursor API instead. Otherwise take extra care of buffer memory address+size and byte ordering. Mistakes may lead to SIGSEGV or unpredictable key ordering etc.
// read only
try (Transaction tx = env.createReadTransaction();
BufferCursor cursor = db.bufferCursor(tx)) {
// iterate from first item and forwards
if (cursor.first()) {
do {
// read a position in buffer
cursor.keyByte(0);
cursor.valByte(0);
} while (cursor.next());
}
// iterate from last item and backwards
if (cursor.last()) {
do {
// copy entire buffer
cursor.keyBytes();
cursor.valBytes();
} while (cursor.prev());
}
// find entry matching exactly the provided key
cursor.keyWriteBytes(bytes("Paris"));
if (cursor.seekKey()) {
// read utf-8 string from position until NULL byte
cursor.valUtf8(0);
}
// find first key greater than or equal to specified key.
cursor.keyWriteBytes(bytes("London"));
if (cursor.seekRange()) {
// read utf-8 string from position until NULL byte
cursor.keyUtf8(0);
cursor.valUtf8(0);
}
}
// open for write
try (Transaction tx = env.createWriteTransaction()) {
// cursors must close before write transactions!
try (BufferCursor cursor = db.bufferCursor(tx)) {
if (cursor.first()) {
// write utf-8 ending with NULL byte
cursor.keyWriteUtf8("England");
cursor.valWriteUtf8("London");
// overwrite existing item if any. Data is not written
// into database before this operation is called and
// no updates are visible outside this transaction until
// the transaction is committed
cursor.overwrite();
cursor.first();
// delete current cursor position
cursor.delete();
}
}
// commit changes or try-with-resources will auto-abort
tx.commit();
}
License
This project is licensed under the Apache License, Version 2.0 but the binary jar it produces also includes liblmdb library of the OpenLDAP project which is licensed under the The OpenLDAP Public License.
2.5k Jan 5, 2023
71 Nov 2, 2022
13k Jan 5, 2023
334 Jan 1, 2023
161 Dec 30, 2022
1.3k Dec 29, 2022
2.1k Dec 29, 2022
142 Jan 1, 2023
386 Jan 1, 2023
1.1k Dec 25, 2022
890 Dec 28, 2022
147 Oct 31, 2022
296 Dec 30, 2022
2k Dec 31, 2022
967 Nov 14, 2022
1.4k Dec 30, 2022
4k Dec 29, 2022