Find a file
Dhruba Borthakur 321dfdc3ae Allow having different compression algorithms on different levels.
Summary:
The leveldb API is enhanced to support different compression algorithms at
different levels.

This adds the option min_level_to_compress to db_bench that specifies
the minimum level for which compression should be done when
compression is enabled. This can be used to disable compression for levels
0 and 1 which are likely to suffer from stalls because of the CPU load
for memtable flushes and (L0,L1) compaction.  Level 0 is special as it
gets frequent memtable flushes. Level 1 is special as it frequently
gets all:all file compactions between it and level 0. But all other levels
could be the same. For any level N where N > 1, the rate of sequential
IO for that level should be the same. The last level is the
exception because it might not be full and because files from it are
not read to compact with the next larger level.

The same amount of time will be spent doing compaction at any
level N excluding N=0, 1 or the last level. By this standard all
of those levels should use the same compression. The difference is that
the loss (using more disk space) from a faster compression algorithm
is less significant for N=2 than for N=3. So we might be willing to
trade disk space for faster write rates with no compression
for L0 and L1, snappy for L2, zlib for L3. Using a faster compression
algorithm for the mid levels also allows us to reclaim some cpu
without trading off much loss in disk space overhead.

Also note that little is to be gained by compressing levels 0 and 1. For
a 4-level tree they account for 10% of the data. For a 5-level tree they
account for 1% of the data.

With compression enabled:
* memtable flush rate is ~18MB/second
* (L0,L1) compaction rate is ~30MB/second

With compression enabled but min_level_to_compress=2
* memtable flush rate is ~320MB/second
* (L0,L1) compaction rate is ~560MB/second

This practicaly takes the same code from https://reviews.facebook.net/D6225
but makes the leveldb api more general purpose with a few additional
lines of code.

Test Plan: make check

Differential Revision: https://reviews.facebook.net/D6261
2012-10-29 11:48:09 -07:00
db Allow having different compression algorithms on different levels. 2012-10-29 11:48:09 -07:00
doc merge 1.5 2012-08-28 11:43:33 -07:00
hdfs Allow a configurable number of background threads. 2012-09-19 15:51:08 -07:00
helpers/memenv A number of fixes: 2011-10-31 17:22:06 +00:00
include/leveldb Allow having different compression algorithms on different levels. 2012-10-29 11:48:09 -07:00
java Add LevelDb's JNI wrapper 2012-10-05 13:13:49 -07:00
port Do not enable checksums for zlib compression. 2012-10-19 16:06:33 -07:00
scribe fix db_test error with scribe logger turned on 2012-08-28 11:22:58 -07:00
snappy Build with gcc-4.7.1-glibc-2.14.1. 2012-09-17 10:56:26 -07:00
table Allow having different compression algorithms on different levels. 2012-10-29 11:48:09 -07:00
thrift Implement RowLocks for assoc schema 2012-10-03 23:19:01 -07:00
tools Fix compilation problem with db_stress when using C11 compiler. 2012-10-12 17:00:25 -07:00
util Allow having different compression algorithms on different levels. 2012-10-29 11:48:09 -07:00
.arcconfig Support arcdiff. 2012-05-09 23:35:05 -07:00
.gitignore Added bloom filter support. 2012-04-17 08:36:46 -07:00
AUTHORS reverting disastrous MOE commit, returning to r21 2011-04-19 23:11:15 +00:00
build_detect_platform Keep build_detect_platform portable 2012-10-26 14:20:04 -07:00
build_detect_version Record the version of the source repository that was used to build the leveldb library. 2012-08-24 15:18:43 -07:00
fbcode.gcc471.sh Enable SSE when building with fbcode support. 2012-10-18 08:43:25 -07:00
LICENSE reverting disastrous MOE commit, returning to r21 2011-04-19 23:11:15 +00:00
Makefile [tools] Add a tool to stress test concurrent writing to levelDB 2012-10-10 12:12:55 -07:00
NEWS sync with upstream @ 21409451 2011-05-21 02:17:43 +00:00
README @20776309 2011-04-20 22:48:11 +00:00
README.fb Enable SSE when building with fbcode support. 2012-10-18 08:43:25 -07:00
TODO A number of smaller fixes and performance improvements: 2011-06-22 02:36:45 +00:00

leveldb: A key-value store
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

The code under this directory implements a system for maintaining a
persistent key/value store.

See doc/index.html for more explanation.
See doc/impl.html for a brief overview of the implementation.

The public interface is in include/*.h.  Callers should not include or
rely on the details of any other header files in this package.  Those
internal APIs may be changed without warning.

Guide to header files:

include/db.h
    Main interface to the DB: Start here

include/options.h
    Control over the behavior of an entire database, and also
    control over the behavior of individual reads and writes.

include/comparator.h
    Abstraction for user-specified comparison function.  If you want
    just bytewise comparison of keys, you can use the default comparator,
    but clients can write their own comparator implementations if they
    want custom ordering (e.g. to handle different character
    encodings, etc.)

include/iterator.h
    Interface for iterating over data. You can get an iterator
    from a DB object.

include/write_batch.h
    Interface for atomically applying multiple updates to a database.

include/slice.h
    A simple module for maintaining a pointer and a length into some
    other byte array.

include/status.h
    Status is returned from many of the public interfaces and is used
    to report success and various kinds of errors.

include/env.h
    Abstraction of the OS environment.  A posix implementation of
    this interface is in util/env_posix.cc

include/table.h
include/table_builder.h
    Lower-level modules that most clients probably won't use directly