Find a file
Mark Callaghan e7206f43ee Improve statistics
Summary:
This adds more statistics to be reported by GetProperty("leveldb.stats").
The new stats include time spent waiting on stalls in MakeRoomForWrite.
This also includes the total amplification rate where that is:
    (#bytes of sequential IO during compaction) / (#bytes from Put)
This also includes a lot more data for the per-level compaction report.
* Rn(MB) - MB read from level N during compaction between levels N and N+1
* Rnp1(MB) - MB read from level N+1 during compaction between levels N and N+1
* Wnew(MB) - new data written to the level during compaction
* Amplify - ( Write(MB) + Rnp1(MB) ) / Rn(MB)
* Rn - files read from level N during compaction between levels N and N+1
* Rnp1 - files read from level N+1 during compaction between levels N and N+1
* Wnp1 - files written to level N+1 during compaction between levels N and N+1
* NewW - new files written to level N+1 during compaction
* Count - number of compactions done for this level

This is the new output from DB::GetProperty("leveldb.stats"). The old output stopped at Write(MB)

                               Compactions
Level  Files Size(MB) Time(sec) Read(MB) Write(MB)  Rn(MB) Rnp1(MB) Wnew(MB) Amplify Read(MB/s) Write(MB/s)   Rn Rnp1 Wnp1 NewW Count
-------------------------------------------------------------------------------------------------------------------------------------
  0        3        6        33        0       576       0        0      576    -1.0       0.0         1.3     0    0    0    0   290
  1      127      242       351     5316      5314     570     4747      567    17.0      12.1        12.1   287 2399 2685  286    32
  2      161      328        54      822       824     326      496      328     4.0       1.9         1.9   160  251  411  160   161
Amplification: 22.3 rate, 0.56 GB in, 12.55 GB out
Uptime(secs): 439.8
Stalls(secs): 206.938 level0_slowdown, 0.000 level0_numfiles, 24.129 memtable_compaction

Task ID: #

Blame Rev:

Test Plan:
run db_bench

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -
(cherry picked from commit ecdeead38f86cc02e754d0032600742c4f02fec8)

Reviewers: dhruba

Differential Revision: https://reviews.facebook.net/D6153
2012-10-24 14:21:38 -07:00
db Improve statistics 2012-10-24 14:21:38 -07:00
doc merge 1.5 2012-08-28 11:43:33 -07:00
hdfs Allow a configurable number of background threads. 2012-09-19 15:51:08 -07:00
helpers/memenv A number of fixes: 2011-10-31 17:22:06 +00:00
include/leveldb Fix broken build. Add stdint.h to get uint64_t 2012-10-23 14:58:53 -07:00
java Add LevelDb's JNI wrapper 2012-10-05 13:13:49 -07:00
port Do not enable checksums for zlib compression. 2012-10-19 16:06:33 -07:00
scribe fix db_test error with scribe logger turned on 2012-08-28 11:22:58 -07:00
snappy Build with gcc-4.7.1-glibc-2.14.1. 2012-09-17 10:56:26 -07:00
table Trigger read compaction only if seeks to storage are incurred. 2012-09-28 11:10:52 -07:00
thrift Implement RowLocks for assoc schema 2012-10-03 23:19:01 -07:00
tools Fix compilation problem with db_stress when using C11 compiler. 2012-10-12 17:00:25 -07:00
util db_bench was not correctly initializing the value for delete_obsolete_files_period_micros option. 2012-10-19 15:10:12 -07:00
.arcconfig Support arcdiff. 2012-05-09 23:35:05 -07:00
.gitignore Added bloom filter support. 2012-04-17 08:36:46 -07:00
AUTHORS reverting disastrous MOE commit, returning to r21 2011-04-19 23:11:15 +00:00
build_detect_platform Enable SSE when building with fbcode support. 2012-10-18 08:43:25 -07:00
build_detect_version Record the version of the source repository that was used to build the leveldb library. 2012-08-24 15:18:43 -07:00
fbcode.gcc471.sh Enable SSE when building with fbcode support. 2012-10-18 08:43:25 -07:00
LICENSE reverting disastrous MOE commit, returning to r21 2011-04-19 23:11:15 +00:00
Makefile [tools] Add a tool to stress test concurrent writing to levelDB 2012-10-10 12:12:55 -07:00
NEWS sync with upstream @ 21409451 2011-05-21 02:17:43 +00:00
README @20776309 2011-04-20 22:48:11 +00:00
README.fb Enable SSE when building with fbcode support. 2012-10-18 08:43:25 -07:00
TODO A number of smaller fixes and performance improvements: 2011-06-22 02:36:45 +00:00

leveldb: A key-value store
Authors: Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

The code under this directory implements a system for maintaining a
persistent key/value store.

See doc/index.html for more explanation.
See doc/impl.html for a brief overview of the implementation.

The public interface is in include/*.h.  Callers should not include or
rely on the details of any other header files in this package.  Those
internal APIs may be changed without warning.

Guide to header files:

include/db.h
    Main interface to the DB: Start here

include/options.h
    Control over the behavior of an entire database, and also
    control over the behavior of individual reads and writes.

include/comparator.h
    Abstraction for user-specified comparison function.  If you want
    just bytewise comparison of keys, you can use the default comparator,
    but clients can write their own comparator implementations if they
    want custom ordering (e.g. to handle different character
    encodings, etc.)

include/iterator.h
    Interface for iterating over data. You can get an iterator
    from a DB object.

include/write_batch.h
    Interface for atomically applying multiple updates to a database.

include/slice.h
    A simple module for maintaining a pointer and a length into some
    other byte array.

include/status.h
    Status is returned from many of the public interfaces and is used
    to report success and various kinds of errors.

include/env.h
    Abstraction of the OS environment.  A posix implementation of
    this interface is in util/env_posix.cc

include/table.h
include/table_builder.h
    Lower-level modules that most clients probably won't use directly