dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,330 Commits 196 Branches 277 Tags 390 MiB
Commit Graph

2330 Commits

Author SHA1 Message Date
Igor Canadi 49b5f94c54 Merge pull request #306 from Liuchang0812/fix_cast 
Update logging.cc
 2014-09-22 10:17:03 -07:00
liuchang0812 787cb4db29 remove cast, replace %llu with % PRIu64 2014-09-23 01:10:46 +08:00
whu_liuchang a7574d4fa1 Update logging.cc 2014-09-22 23:37:00 +08:00
whu_liuchang 7e0dcb953f Update logging.cc 
fix cast style to cpp static_cast
 2014-09-22 23:24:53 +08:00
Igor Canadi 57fa3cc5b3 Merge pull request #304 from Liuchang0812/fix-check 
fixed #303: replace %ld with % PRId64
 2014-09-21 11:55:45 -07:00
Igor Canadi cd44522a94 Merge pull request #305 from Liuchang0812/fix-logging 
remove unused variable
 2014-09-21 11:54:55 -07:00
liuchang0812 6a031b6a81 remove unused variable 2014-09-21 22:20:00 +08:00
liuchang0812 4436f17bd8 fixed #303: replace %ld with % PRId64 2014-09-21 22:09:48 +08:00
ankgup87 7a1bd057fa Merge pull request #302 from ankgup87/master 
[Java] Add rate limiter
 2014-09-19 16:14:25 -07:00
Ankit Gupta 423e52cd47 Merge branch 'master' of https://github.com/facebook/rocksdb 2014-09-19 16:12:26 -07:00
Ankit Gupta bfeef94d31 Add rate limiter 2014-09-19 16:11:59 -07:00
Mark Callaghan 32f2532a0b Print compression_size_percent as a signed int 
Summary:
compression_size_percent is an int but was printed as
an unsigned int. So the default of -1 is displayed as a big number.

Test Plan: make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23679
 2014-09-19 13:09:25 -07:00
sdong 976caca09b Skip AllocateTest if fallocate() is not supported in the file system 
Summary: To avoid false positive test failures when the file system doesn't support fallocate. In EnvTest.AllocateTest, we first make a simple fallocate call and check the error codes to rule out the possibility that it is not supported. Skip the test if the error code indicates it is not supported.

Test Plan: Run the test and make sure it passes on file systems supporting and not supporting fallocate

Reviewers: yhchiang, ljin, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23667
 2014-09-19 10:55:59 -07:00
Igor Canadi 3b897cddd7 Enable no-fbcode RocksDB build 
Summary: I want to use open source build rather than fbcode one. This enables me to run `ROCKSDB_NO_FBCODE=1 make` and run it with my system g++.

Test Plan:
ROCKSDB_NO_FBCODE=1 make
make

Reviewers: sdong, ljin, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23613
 2014-09-19 09:27:16 -07:00
Venkatesh Radhakrishnan f44594743f RocksDB: Format uint64 using PRIu64 in db_impl.cc 
Summary: Use PRIu64 to format uint64 in a portable manner

Test Plan: Run "make all check"

Reviewers: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23595
 2014-09-18 22:19:41 -07:00
ankgup87 e17bc65c75 Merge pull request #299 from ankgup87/master 
[Java] Fix build
 2014-09-18 22:15:31 -07:00
Ankit Gupta b93797abc4 Fix build 2014-09-18 22:13:52 -07:00
Yueh-Hsuan Chiang adae3ca1fe [Java] Fix JNI link error caused by the removal of options.db_stats_log_interval 
Summary: Fix JNI link error caused by the removal of options.db_stats_log_interval in https://reviews.facebook.net/D21915.

Test Plan:
make rocksdbjava
make jtest

Reviewers: ljin, ankgup87

Reviewed By: ankgup87

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23505
 2014-09-18 21:09:44 -07:00
Igor Canadi 90b8c07b48 Fix unit tests errors 
Summary: Those were introduced with 2fb1fea30f because the flushing behavior changed when max_background_flushes is > 0.

Test Plan: make check

Reviewers: ljin, yhchiang, sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23577
 2014-09-18 13:32:44 -07:00
Lei Jin 51af7c326c CuckooTable: add one option to allow identity function for the first hash function 
Summary:
MurmurHash becomes expensive when we do millions Get() a second in one
thread. Add this option to allow the first hash function to use identity
function as hash function. It results in QPS increase from 3.7M/s to
~4.3M/s. I did not observe improvement for end to end RocksDB
performance. This may be caused by other bottlenecks that I will address
in a separate diff.

Test Plan:
```
[ljin@dev1964 rocksdb] ./cuckoo_table_reader_test --enable_perf --file_dir=/dev/shm --write --identity_as_first_hash=0
==== Test CuckooReaderTest.WhenKeyExists
==== Test CuckooReaderTest.WhenKeyExistsWithUint64Comparator
==== Test CuckooReaderTest.CheckIterator
==== Test CuckooReaderTest.CheckIteratorUint64
==== Test CuckooReaderTest.WhenKeyNotFound
==== Test CuckooReaderTest.TestReadPerformance
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.272us (3.7 Mqps) with batch size of 0, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.138us (7.2 Mqps) with batch size of 10, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.142us (7.1 Mqps) with batch size of 25, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.142us (7.0 Mqps) with batch size of 50, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.144us (6.9 Mqps) with batch size of 100, # of found keys 125829120

With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.201us (5.0 Mqps) with batch size of 0, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.121us (8.3 Mqps) with batch size of 10, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.123us (8.1 Mqps) with batch size of 25, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.121us (8.3 Mqps) with batch size of 50, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.112us (8.9 Mqps) with batch size of 100, # of found keys 104857600

With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.251us (4.0 Mqps) with batch size of 0, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.107us (9.4 Mqps) with batch size of 10, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.099us (10.1 Mqps) with batch size of 25, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.100us (10.0 Mqps) with batch size of 50, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.116us (8.6 Mqps) with batch size of 100, # of found keys 83886080

With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.189us (5.3 Mqps) with batch size of 0, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.095us (10.5 Mqps) with batch size of 10, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.096us (10.4 Mqps) with batch size of 25, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.098us (10.2 Mqps) with batch size of 50, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.105us (9.5 Mqps) with batch size of 100, # of found keys 73400320

[ljin@dev1964 rocksdb] ./cuckoo_table_reader_test --enable_perf --file_dir=/dev/shm --write --identity_as_first_hash=1
==== Test CuckooReaderTest.WhenKeyExists
==== Test CuckooReaderTest.WhenKeyExistsWithUint64Comparator
==== Test CuckooReaderTest.CheckIterator
==== Test CuckooReaderTest.CheckIteratorUint64
==== Test CuckooReaderTest.WhenKeyNotFound
==== Test CuckooReaderTest.TestReadPerformance
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.230us (4.3 Mqps) with batch size of 0, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.086us (11.7 Mqps) with batch size of 10, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.088us (11.3 Mqps) with batch size of 25, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.083us (12.1 Mqps) with batch size of 50, # of found keys 125829120
With 125829120 items, utilization is 93.75%, number of hash functions: 2.
Time taken per op is 0.083us (12.1 Mqps) with batch size of 100, # of found keys 125829120

With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.159us (6.3 Mqps) with batch size of 0, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.078us (12.8 Mqps) with batch size of 10, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.080us (12.6 Mqps) with batch size of 25, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.080us (12.5 Mqps) with batch size of 50, # of found keys 104857600
With 104857600 items, utilization is 78.12%, number of hash functions: 2.
Time taken per op is 0.082us (12.2 Mqps) with batch size of 100, # of found keys 104857600

With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.154us (6.5 Mqps) with batch size of 0, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.077us (13.0 Mqps) with batch size of 10, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.077us (12.9 Mqps) with batch size of 25, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.078us (12.8 Mqps) with batch size of 50, # of found keys 83886080
With 83886080 items, utilization is 62.50%, number of hash functions: 2.
Time taken per op is 0.079us (12.6 Mqps) with batch size of 100, # of found keys 83886080

With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.218us (4.6 Mqps) with batch size of 0, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.083us (12.0 Mqps) with batch size of 10, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.085us (11.7 Mqps) with batch size of 25, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.086us (11.6 Mqps) with batch size of 50, # of found keys 73400320
With 73400320 items, utilization is 54.69%, number of hash functions: 2.
Time taken per op is 0.078us (12.8 Mqps) with batch size of 100, # of found keys 73400320
```

Reviewers: sdong, igor, yhchiang

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23451
 2014-09-18 11:00:48 -07:00
Yueh-Hsuan Chiang 035043559d Fixed a signed-unsigned comparison in spatial_db.cc -- issue #293 
Summary:
Fixed a signed-unsigned comparison in spatial_db.cc

utilities/spatialdb/spatial_db.cc:542:38: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
cc1plus: all warnings being treated as errors
make: *** [utilities/spatialdb/spatial_db.o] Error 1

Test Plan:
make spatial_db_test
./spatial_db_test

Reviewers: ljin, sdong, reddragon, igor

Reviewed By: reddragon

Subscribers: reddragon, leveldb

Differential Revision: https://reviews.facebook.net/D23565
 2014-09-18 10:57:20 -07:00
Igor Canadi 2fb1fea30f Fix syncronization issues 2014-09-18 10:42:54 -07:00
Igor Canadi ff76895614 Remove some unnecessary constructors 
Summary:
This is continuing the work done by 27b22f13a3

It's just cleaning up some unnecessary constructors. The most important change is removing Block::Block(const BlockContents& contents) constructor. It was only used from the unit test.

Test Plan: compiles

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23547
 2014-09-17 16:45:58 -07:00
Lei Jin feadb9df53 fix cuckoo table builder test 
Summary:
as title

Test Plan:
./cuckoo_table_builder_test

Reviewers:igor

CC:leveldb

Task ID: #

Blame Rev:
 2014-09-17 15:48:33 -07:00
Igor Canadi 3c232e1647 Fix mac compile 2014-09-17 15:45:43 -07:00
Igor Canadi 54cada92b1 Run make format on PR #249 2014-09-17 15:08:50 -07:00
Igor Canadi 27b22f13a3 Merge pull request #249 from tdfischer/decompression-refactoring 
Decompression refactoring
 2014-09-17 15:01:40 -07:00
Torrie Fischer fb6456b00d Replace naked calls to operator new and delete (Fixes #222) 
This replaces a mishmash of pointers in the Block and BlockContents classes with
std::unique_ptr. It also changes the semantics of BlockContents to be limited to
use as a constructor parameter for Block objects, as it owns any block buffers
handed to it.
 2014-09-17 13:50:07 -07:00
Lei Jin 5600c8f6e5 cuckoo table: return estimated size - 1 
Summary:
This is to avoid cutting file prematurely and resulting file size to be
half of specified.

Test Plan: db_bench

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23541
 2014-09-17 13:25:29 -07:00
Lei Jin a062e1f2c4 SetOptions() for memtable related options 
Summary: as title

Test Plan:
make all check
I will think a way to set up stress test for this

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23055
 2014-09-17 12:49:13 -07:00
Lei Jin e4eca6a1e5 Options conversion function for convenience 
Summary: as title

Test Plan: options_test

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23283
 2014-09-17 12:46:32 -07:00
Igor Canadi a7c2094526 Merge pull request #292 from saghmrossi/master 
util/cache_bench.cc needs to include inttypes.h
 2014-09-17 12:43:39 -07:00
Saghm Rossi 4d0523452a Merge branch 'master' of github.com:saghmrossi/rocksdb 2014-09-17 15:38:43 -04:00
Igor Canadi 60a4aa175e Test use_mmap_reads 
Summary: We currently don't test mmap reads as part of db_test. Piggyback it on kWalDir test config.

Test Plan: make check

Reviewers: ljin, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23337
 2014-09-17 12:31:53 -07:00
Yueh-Hsuan Chiang 94e43a1dfe [Java] Fixed 32-bit overflowing issue when converting jlong to size_t 
Summary:
Fixed 32-bit overflowing issue when converting jlong to size_t by
capping jlong to std::numeric_limits<size_t>::max().

Test Plan:
make rocksdbjava
make jtest

Reviewers: ankgup87, ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23511
 2014-09-17 12:30:06 -07:00
Saghm Rossi f9eaaa66e6 added include for inttypes.h to fix nonworking printf statements 2014-09-17 15:15:53 -04:00
Yueh-Hsuan Chiang f090575e43 Replaced "built on on earlier work" by "built on earlier work" in README.md 2014-09-17 01:16:17 -07:00
Igor Canadi faad439ac4 Fix #284 
Summary: This work on my compiler, but it turns out some compilers don't implicitly add constness, see: https://github.com/facebook/rocksdb/issues/284. This diff adds constness explicitly.

Test Plan: still compiles

Reviewers: sdong

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23409
 2014-09-16 10:30:32 -07:00
Igor Canadi 49aacd8d2b Fix make install 
Summary: See https://github.com/facebook/rocksdb/issues/283

Test Plan: make install/uninstall

Reviewers: ljin, sdong, yhchiang

Reviewed By: yhchiang

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23373
 2014-09-15 15:30:17 -07:00
Yueh-Hsuan Chiang acb9348ff3 [Java] Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch. 
Summary:
Include WriteBatch into RocksDBSample.java, fix how DbBenchmark.java handles WriteBatch.
Previously DbBenchmark.java does not use WriteBatch when benchmarks is set to fillbatch.

Test Plan:
make rocksdbjava -j32
make jtest
make jdb_bench
cd java
./jdb_bench.sh --benchmarks=fillbatch

Reviewers: naveenatceg, ljin, sdong, ankgup87

Reviewed By: ankgup87

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22983
 2014-09-15 12:34:12 -07:00
Igor Canadi 4a27a2f193 Don't sync manifest when disableDataSync = true 
Summary: As we discussed offline

Test Plan: compiles

Reviewers: yhchiang, sdong, ljin, dhruba

Reviewed By: sdong

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D22989
 2014-09-15 11:32:01 -07:00
Igor Canadi 9b8480d93b Merge pull request #287 from yinqiwen/rate-limiter-crash-fix 
fix rate limiter crash #286
 2014-09-15 10:00:07 -07:00
yinqiwen 28be16b1dc fix rate limiter crash #286 2014-09-15 20:43:38 +08:00
Igor Canadi 04ce1b25f3 Fix #284 2014-09-13 14:14:10 -07:00
Lei Jin add22e3515 standardize scripts to run RocksDB benchmarks 
Summary:
Hope these scripts will allow people to run/repro benchmark easily
I think it is time to re-run flash benchmarks and report results
Please comment if any other benchmark runs are needed

Test Plan: ran it

Reviewers: yhchiang, igor, sdong

Reviewed By: igor

Subscribers: dhruba, MarkCallaghan, leveldb

Differential Revision: https://reviews.facebook.net/D23139
 2014-09-12 16:25:35 -07:00
Igor Canadi dee91c259d WriteThread 
Summary: This diff just moves the write thread control out of the DBImpl. I will need this as I will control column family data concurrency by only accessing some data in the write thread. That way, we won't have to lock our accesses to column family hash table (mappings from IDs to CFDs).

Test Plan: make check

Reviewers: sdong, yhchiang, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23301
 2014-09-12 16:23:58 -07:00
Igor Canadi 540a257f2c Fix WAL synced 
Summary: Uhm...

Test Plan: nope

Reviewers: sdong, yhchiang, tnovak, ljin

Reviewed By: ljin

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23343
 2014-09-12 16:15:29 -07:00
Igor Canadi 24f034bf49 Merge pull request #282 from Chilledheart/develop 
Fix build issue under macosx
 2014-09-12 14:37:45 -07:00
Chilledheart 49fe329e5e Fix build issue under macosx 2014-09-13 05:05:22 +08:00
Yueh-Hsuan Chiang ebb5c65e60 Add make install 
Summary:
Add make install.  If INSTALL_PATH is not set, then rocksdb will be
installed under "/usr/local" directory (/usr/local/include for headers
and /usr/local/lib for library file(s).)

Test Plan:
Develop a simple rocksdb app, called test.cc, and do the followings.

make clean
make static_lib -j32
sudo make install
g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test
./test

sudo make uninstall
make clean
make shared_lib -j32
sudo make install
g++ -std=c++11 test.cc -lrocksdb -lbz2 -lz -o test
./test

make INSTALL_PATH=/tmp/path install
make INSTALL_PATH=/tmp/path uninstall
and make sure things are installed / uninstalled in the specified path.

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: leveldb

Differential Revision: https://reviews.facebook.net/D23211
 2014-09-11 20:41:02 -07:00