Commit graph

2274 commits

Author SHA1 Message Date
sdong ddd41146c4 MergingIterator uses autovector instead of vector
Summary:
Use autovector in MergingIterator so that if there are 4 or less child iterators in it, iterator wrappers are inline, which is more likely to be cache friendly.

Based on one test run with a shadow traffic of one product, it reduces CPU of MergingIterator::Seek() by half.

Test Plan: make all check

Reviewers: haobo, yhchiang, igor, dhruba

Reviewed By: igor

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18531
2014-05-08 15:01:20 -07:00
Igor Canadi af7453a673 autovector::resize
Summary: Resize the autovector!

Test Plan: test

Reviewers: sdong

Reviewed By: sdong

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18543
2014-05-08 13:50:49 -07:00
Igor Canadi 8e37a29bfb Compaction with zero outputs
Summary: We had a hypothesis in https://reviews.facebook.net/D18507 that empty-string internal keys might have been caused by compaction filter deleting all the entries. I added a unit test for that case. Unforutnately, everything works as expected.

Test Plan: this is a test

Reviewers: dhruba, haobo, sdong

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18519
2014-05-08 13:48:39 -07:00
sdong 1c6a027d23 HashLinkedList::Iterator: remove an ununsed class variable
Summary: This variable is not used. Remove it.

Test Plan: build.

Reviewers: haobo, igor, yhchiang

Reviewed By: haobo

CC: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D18525
2014-05-08 13:28:52 -07:00
dallas marlow f41cde3105 remove anon mmap allocation flag MAP_HUGETLB 2014-05-08 11:45:44 -04:00
Igor Canadi b5616dafd1 Fix iOS compile 2014-05-07 17:48:31 -07:00
Igor Canadi 768d424dd9 [fix] SIGSEGV when VersionEdit in MANIFEST is corrupted
Summary:
This was reported by our customers in task #4295529.

Cause:
* MANIFEST file contains a VersionEdit, which contains file entries whose 'smallest' and 'largest' internal keys are empty. String with zero characters. Root cause of corruption was not investigated. We should report corruption when this happens. However, we currently SIGSEGV.

Here's what happens:
* VersionEdit encodes zero-strings happily and stores them in smallest and largest InternalKeys. InternalKey::Encode() does assert when `rep_.empty()`, but we don't assert in production environemnts. Also, we should never assert as a result of DB corruption.
* As part of our ConsistencyCheck, we call GetLiveFilesMetaData()
* GetLiveFilesMetadata() calls `file->largest.user_key().ToString()`
* user_key() function does: 1. assert(size > 8) (ooops, no assert), 2. returns `Slice(internal_key.data(), internal_key.size() - 8)`
* since `internal_key.size()` is unsigned int, this call translates to `Slice(whatever, 1298471928561892576182756)`. Bazinga.

Fix:
* VersionEdit checks if InternalKey is valid in `VersionEdit::GetInternalKey()`. If it's invalid, returns corruption.

Lessons learned:
* Always keep in mind that even if you `assert()`, production code will continue execution even if assert fails.
* Never `assert` based on DB corruption. Assert only if the code should guarantee that assert can't fail.

Test Plan: dumped offending manifest. Before: assert. Now: corruption

Reviewers: dhruba, haobo, sdong

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18507
2014-05-07 16:52:12 -07:00
Igor Canadi 313b2e5da1 Better INSTALL.md and Makefile rules
Summary: We have a lot of problems with gflags. However, when compiling rocksdb static library, we don't need gflags dependency. Reorganize INSTALL.md such that first-time customers don't need any dependency installed to actually build rocksdb static library.

Test Plan: none

Reviewers: dhruba, haobo

Reviewed By: dhruba

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18501
2014-05-07 16:51:30 -07:00
sdong 9efbd85ac9 fsync directory after creating current file in NewDB()
Summary: One of our users reported current file corruption. The machine was rebooted during the time. This is the only think I can think of which could cause current file corruption. Just add this paranoid check.

Test Plan: make all check

Reviewers: haobo, igor

Reviewed By: haobo

CC: yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D18495
2014-05-06 17:51:33 -07:00
sdong 3a171dcb51 Pass logger to memtable rep and TLB page allocation error logged to info logs
Summary:
TLB page allocation errors are now logged to info logs, instead of stderr.
In order to do that, mem table rep's factory functions take a info logger now.

Test Plan: make all check

Reviewers: haobo, igor, yhchiang

Reviewed By: yhchiang

CC: leveldb, yhchiang, dhruba

Differential Revision: https://reviews.facebook.net/D18471
2014-05-05 16:43:37 -07:00
Igor Canadi 044af85847 Update HISTORY.md -- release RocksDB 3.0 2014-05-05 14:44:57 -07:00
Igor Canadi 7984b9bbb0 BackupableDBTest thread-safe
Summary: We need to lock accesses to some TestEnv variables. Otherwise we get failures like http://ci-builds.fb.com/job/rocksdb_asan_check/657/console

Test Plan: make check

Reviewers: dhruba, haobo, sdong

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18489
2014-05-05 14:30:24 -07:00
Igor Canadi 15c3991933 Add comment about ValueType 2014-05-05 12:57:47 -07:00
Igor Canadi 9e7d00d9e7 Make rocksdb work with all versions of lz4
Summary:
There are some projects in fbcode that define lz4 dependency on r108. We, however, defined dependency on r117. That produced some interesting issues and our build system was not happy.

This diff makes rocksdb work with both r108 and r117. Hopefully this will fix our problems.

Test Plan: compiled rocksdb with both r108 and r117 lz4

Reviewers: dhruba, sdong, haobo

Reviewed By: sdong

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18465
2014-05-05 11:35:40 -07:00
Igor Canadi d2569fea47 log_and_apply_bench on a new benchmark framework
Summary:
db_test includes Benchmark for LogAndApply. This diff removes it from db_test and puts it into a separate log_and_apply bench. I just wanted to play around with our new benchmark framework and figure out how it works.

I would also like to show you how great it is! I believe right set of microbenchmarks can speed up our productivity a lot and help catch early regressions.

Test Plan: no

Reviewers: dhruba, haobo, sdong, ljin, yhchiang

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18261
2014-05-05 11:11:48 -07:00
sdong 9b17558311 PlainTableFactory::PlainTableFactory() to have huge TLB turned off by default
Summary: PlainTableFactory::PlainTableFactory() now has Huge TLB page feature turned on by default. Although it is not a public API (which we always turn the feature off now), our unit tests, like db_test sometimes uses it directly, which causes wrong coverage of codes. This patch fix it to allow unit tests to run with the correct setting

Test Plan: Run db_test and make sure this feature is not on any more.

Reviewers: igor, haobo

Reviewed By: igor

CC: yhchiang, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D18483
2014-05-05 11:05:54 -07:00
Igor Canadi 6785a52b1b Temporary remove perror() calls before we can log from inside of arena 2014-05-05 07:13:48 -07:00
sdong 4a7c747064 Revert "Revert "Allow allocating dynamic bloom, plain table indexes and hash linked list from huge page TLB""
And make the default 0 for hash linked list memtable

This reverts commit d69dc64be7.
2014-05-04 13:56:29 -07:00
Yueh-Hsuan Chiang d56959a5fc [Java] Use environmental variable JAVA_HOME in Makefile for RocksJava. 2014-05-04 13:23:21 -07:00
Igor Canadi db1854d78c Declare all DB methods virtual so that StackableDB can override them 2014-05-04 11:39:49 -07:00
Igor Canadi d69dc64be7 Revert "Allow allocating dynamic bloom, plain table indexes and hash linked list from huge page TLB"
This reverts commit 7dafa3a1d7.
2014-05-04 08:37:09 -07:00
Benjamin Renard 41e5cf2392 Add share_files_with_cheksum option to BackupEngine
Summary: added a new option to BackupEngine: if share_files_with_checksum is set to true, sst files are stored in shared_checksum/ and are identified by the triple (file name, checksum, file size) instead of just the file name. This option is targeted at distributed databases that want to backup their primary replica.

Test Plan: unit tests and tested backup and restore on a distributed rocksdb

Reviewers: igor

Reviewed By: igor

Differential Revision: https://reviews.facebook.net/D18393
2014-05-02 17:08:55 -07:00
Igor Canadi 77edbfd642 link with lz4
Summary: link our fbcode build with lz4

Test Plan: compiles

Reviewers: sdong, haobo, dhruba, yhchiang

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18447
2014-05-02 13:55:33 -07:00
Igor Canadi 4ecfbcf865 ApplyToAllCacheEntries
Summary: Added a method that executes a callback on every cache entry.

Test Plan: added a unit test

Reviewers: haobo

Reviewed By: haobo

CC: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D18441
2014-05-02 16:24:04 -04:00
Igor Canadi 31d38a6732 Add TARGETS to gitignore 2014-05-02 11:18:55 -07:00
Igor Canadi 3b5fa6bd9b Improve ttl_test
Summary:
Our valgrind tests are failing because ttl_test is kind of flakey. This diff should fix valgrind issue and make ttl_test less flakey and much faster.

Instead of relying on Env::Default() for getting current time, I expose `Env*` to all TTL functions that are interested in time. That way, I can insert a custom test Env which is then used to provide exactly the times we need. That way, we don't need to sleep anymore -- we control the time.

Test Plan: ttl_test in normal and valgrind run

Reviewers: dhruba, haobo, sdong, yhchiang

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18399
2014-05-02 07:13:51 -07:00
Igor Canadi 82042f451c Include version in options 2014-05-01 19:28:23 -04:00
Igor Canadi d28ed6931f fix release build 2014-05-01 12:42:06 -07:00
Igor Canadi d29e48bb2e fix compile warning 2014-05-01 14:12:35 -04:00
Igor Canadi 0afc8bc29a xxHash
Summary:
Originally: https://github.com/facebook/rocksdb/pull/87/files

I'm taking over to apply some finishing touches

Test Plan: will add tests

Reviewers: dhruba, haobo, sdong, yhchiang, ljin

Reviewed By: yhchiang

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18315
2014-05-01 14:09:32 -04:00
Yueh-Hsuan Chiang 61955a0dda [Java] Add RocksObject, the base class of all java objects with a c++ pointer.
Summary:
Add RocksObject, the base class of all java objects which has a c++ pointer.
While the finalizer of a RocksObject will release its c++ resource, it is
suggested to call its RocksObject.dispose() to manually release its
c++ resource.

Existing RocksDB java classes are now extending RocksObject.

Test Plan:
make rocksdbjava
make jtest
make jdb_bench

Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18411
2014-05-01 01:44:46 -07:00
Igor Canadi 096f5be0ed Put column family information in LiveFileMetaData
Summary: As summary

Test Plan: compiles :)

Reviewers: dhruba, haobo, sdong, yhchiang

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18405
2014-04-30 16:24:52 -04:00
Yueh-Hsuan Chiang a3ab998ef9 [Java] Add static method RocksDB.loadLibrary()
Summary:
Add static method RocksDB.loadLibrary() which loads necessary library
files.

Test Plan:
make rocksdbjava
make jtest
make jdb_bench
java/jdb_bench.sh

Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett

Reviewed By: ankgup87

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18375
2014-04-30 11:54:02 -07:00
Igor Canadi 16f1aa7b2d Fix signed/unsigned compare 2014-04-30 14:38:01 -04:00
Igor Canadi df70047669 Flush stale column families
Summary:
Added a new option `max_total_wal_size`. Once the total WAL size goes over that, we make an attempt to flush all column families that still have data in the earliest WAL file.

By default, I calculate `max_total_wal_size` dynamically, that should be good-enough for non-advanced customers.

Test Plan: Added a test

Reviewers: dhruba, haobo, sdong, ljin, yhchiang

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18345
2014-04-30 14:33:40 -04:00
sdong 7dafa3a1d7 Allow allocating dynamic bloom, plain table indexes and hash linked list from huge page TLB
Summary: Add an option to allocate a piece of memory from huge page TLB. Add options to trigger it in dynamic bloom, plain table indexes andhash linked list hash table.

Test Plan: make all check

Reviewers: haobo, ljin

Reviewed By: haobo

CC: nkg-, dhruba, leveldb, igor, yhchiang

Differential Revision: https://reviews.facebook.net/D18357
2014-04-30 11:02:26 -07:00
Igor Canadi 66f88c43a5 Some fixes as preparation for release 2014-04-30 09:03:24 -07:00
Igor Canadi d6d67c0efe More s/us fixes 2014-04-30 07:04:36 -07:00
Yueh-Hsuan Chiang 2e11e47c7c Merge pull request #131 from ankgup87/master
[Java] Add multiget JNI bindings
2014-04-29 22:05:55 -07:00
Igor Canadi 046a85af96 Merge pull request #132 from kjk/master
Improve building of shared library so that it e.g. works with Go bindings
2014-04-29 21:26:12 -04:00
Yueh-Hsuan Chiang 9d9d2965cb Add a new mem-table representation based on cuckoo hash.
Summary:
= Major Changes =
* Add a new mem-table representation, HashCuckooRep, which is based cuckoo hash.
  Cuckoo hash uses multiple hash functions.  This allows each key to have multiple
  possible locations in the mem-table.

  - Put: When insert a key, it will try to find whether one of its possible
    locations is vacant and store the key.  If none of its possible
    locations are available, then it will kick out a victim key and
    store at that location.  The kicked-out victim key will then be
    stored at a vacant space of its possible locations or kick-out
    another victim.  In this diff, the kick-out path (known as
    cuckoo-path) is found using BFS, which guarantees to be the shortest.

 - Get: Simply tries all possible locations of a key --- this guarantees
   worst-case constant time complexity.

 - Time complexity: O(1) for Get, and average O(1) for Put if the
   fullness of the mem-table is below 80%.

 - Default using two hash functions, the number of hash functions used
   by the cuckoo-hash may dynamically increase if it fails to find a
   short-enough kick-out path.

 - Currently, HashCuckooRep does not support iteration and snapshots,
   as our current main purpose of this is to optimize point access.

= Minor Changes =
* Add IsSnapshotSupported() to DB to indicate whether the current DB
  supports snapshots.  If it returns false, then DB::GetSnapshot() will
  always return nullptr.

Test Plan:
Run existing tests.  Will develop a test specifically for cuckoo hash in
the next diff.

Reviewers: sdong, haobo

Reviewed By: sdong

CC: leveldb, dhruba, igor

Differential Revision: https://reviews.facebook.net/D16155
2014-04-29 17:13:46 -07:00
Krzysztof Kowalczyk 2b7cf03e0d Update Makefile 2014-04-29 14:29:45 -07:00
Igor Canadi f1c9aa6ebe More unsigned/signed compare fixes 2014-04-29 13:01:06 -07:00
Igor Canadi 38693d99c4 Fix more signed/unsigned comparsions 2014-04-29 12:40:18 -07:00
Igor Canadi d0b006ab13 Fix unsigned and signed integer compare 2014-04-29 12:29:22 -07:00
Igor Canadi 76c3dbf5b5 Make ttl_test a bit more robust 2014-04-29 12:19:54 -07:00
Igor Canadi dd9eb7a7d5 Cache result of ReadFirstRecord()
Summary:
ReadFirstRecord() reads the actual log file from disk on every call. This diff introduces a cache layer on top of ReadFirstRecord(), which should significantly speed up repeated calls to GetUpdatesSince().

I also cleaned up some stuff, but the whole TransactionLogIterator could use some refactoring, especially if we see increased usage.

Test Plan: make check

Reviewers: haobo, sdong, dhruba

Reviewed By: haobo

CC: leveldb

Differential Revision: https://reviews.facebook.net/D18387
2014-04-29 13:27:58 -04:00
Igor Canadi de825e7307 Create new branch instead of tag in make_new_version 2014-04-29 13:22:34 -04:00
Igor Canadi e525bb16ea Make kMajorVersion and kMinorVersion take version from version macros 2014-04-29 11:59:48 -04:00
Igor Canadi 6cb0cb300c Add version.h 2014-04-29 11:33:00 -04:00