Commit Graph

2718 Commits

Author SHA1 Message Date
Yueh-Hsuan Chiang 7fe247080f Update HISTORY.md for RocksJava 2014-11-14 11:23:00 -08:00
Siying Dong 136b8583bf Merge pull request #395 from lalinsky/fix-env-test
Relax the block count check on deallocation in env_test
2014-11-14 10:56:41 -08:00
Igor Canadi 94fa542f82 Update HISTROY.md for 3.8 release 2014-11-14 10:43:12 -08:00
sdong a177742a9b Make db_stress built for ROCKSDB_LITE
Summary:
Make db_stress built for ROCKSDB_LITE.
The test doesn't pass tough. It seg fault quickly. But I took a look and it doesn't seem to be related to lite version. Likely to be a bug inside RocksDB.

Test Plan: make db_stress

Reviewers: yhchiang, rven, ljin, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28797
2014-11-14 10:20:51 -08:00
Lukáš Lalinský 746cfaac58 Relax the block count check on deallocation in env_test
It seems that on some FS we get more blocks than we ask for. This is
already handled when checking the allocated number of blocks, but
after the file is closed it checks for an exact number of blocks,
which fails on my machine.

I changed the test to add one full page to the size, then calculate
the expected number of blocks and check if the actual number of blocks
is less or equal to that.
2014-11-13 16:58:05 -08:00
sdong f822129b32 Add a unit test for behavior when merge operator and compaction filter co-exist.
Summary: Add a unit test in db_test to verify the behavior when both of merge operator and compaction filter apply to a key when merging.

Test Plan: Run the new test

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28455
2014-11-13 15:21:12 -08:00
Yueh-Hsuan Chiang 4161de92a3 Fix SIGSEGV
Summary: As a short-term fix, let's go back to previous way of calculating NeedsCompaction(). SIGSEGV happens because NeedsCompaction() can happen before super_version (and thus MutableCFOptions) is initialized.

Test Plan: make check

Reviewers: ljin, sdong, rven, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28875
2014-11-13 15:21:04 -08:00
Hasnain Lakhani 373c665edf Fix broken test in 31b02d.
Summary:
CorruptionTest for backupable_db_test did not call
GarbageCollect() after deleting a corrupt backup,
which sometimes lead to test failures as the newly created backup
would reuse the same backup ID and files and fail the consistency
check.

Moved around some of the test logic to ensure that GarbageCollect()
is called at the right time.

Test Plan:
Run backupable_db_test eight times and make sure
it passes repeatedly. Also run make check to make sure other
tests don't fail.

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28863
2014-11-13 14:46:30 -08:00
Igor Canadi 772bc97f13 No CompactFiles in ROCKSDB_LITE
Summary: It adds lots of code.

Test Plan: compile for iOS, compile for mac. works.

Reviewers: rven, sdong, ljin, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28857
2014-11-13 16:45:33 -05:00
Yueh-Hsuan Chiang 1d1a64f58a Move NeedsCompaction() from VersionStorageInfo to CompactionPicker
Summary:
Move NeedsCompaction() from VersionStorageInfo to CompactionPicker
to allow different compaction strategy to have their own way to
determine whether doing compaction is necessary.

When compaction style is set to kCompactionStyleNone, then
NeedsCompaction() will always return false.

Test Plan:
export ROCKSDB_TESTS=Compact
./db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28719
2014-11-13 13:41:43 -08:00
Igor Canadi cd0980150b Add concurrency to compacting SpatialDB
Summary: This will speed up our import times

Test Plan: Added simple unit test just to get code coverage

Reviewers: sdong, ljin, yhchiang, rven, mohaps

Reviewed By: mohaps

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28869
2014-11-13 16:34:29 -05:00
Igor Canadi 3c92e52338 Fix include 2014-11-13 14:47:54 -05:00
Igor Canadi 25f273027b Fix iOS compile with -Wshorten-64-to-32
Summary: So iOS size_t is 32-bit, so we need to static_cast<size_t> any uint64_t :(

Test Plan: TARGET_OS=IOS make static_lib

Reviewers: dhruba, ljin, yhchiang, rven, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28743
2014-11-13 14:39:30 -05:00
sdong fa50abb726 Fix bug of reading from empty DB.
Summary: I found that db_stress sometimes segfault on my machine. Fix the bug.

Test Plan: make all check. Run db_stress

Reviewers: ljin, yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D28803
2014-11-13 10:42:18 -08:00
Hasnain Lakhani 31b02dc21d Improve Backup Engine.
Summary:
Improve the backup engine by not deleting the corrupted
backup when it is detected; instead leaving it to the client
to delete the corrupted backup.

Also add a BackupEngine::Open() call.

Test Plan:
Add check to CorruptionTest inside backupable_db_test
to check that the corrupt backups are not deleted. The previous
version of the code failed this test as backups were deleted,
but after the changes in this commit, this test passes.

Run make check to ensure that no other tests fail.

Reviewers: sdong, benj, sanketh, sumeet, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28521
2014-11-13 09:51:41 -08:00
Yueh-Hsuan Chiang 1033db29f9 Merge pull request #390 from fyrz/RocksJava-Cleanup
[RocksJava] Cleanup
2014-11-12 12:26:21 -08:00
Yueh-Hsuan Chiang 9a03da773c Merge pull request #375 from fyrz/RocksJava-ColumnFamilyOptions-Extension-3.6
[RocksJava] ColumnFamilyOptions extension 3.6
2014-11-12 12:25:15 -08:00
fyrz d50c68e3a5 [RocksJava] JavaDoc cleanup warnings with Java8
Java8 is more restrictive than Java7 with generating
JavaDocs. This commit resolves current existing Java8
warnings.
2014-11-12 20:51:04 +01:00
fyrz 079d942ea8 [RocksJava] Code-cleanup + Java7 warnings removed 2014-11-12 20:42:25 +01:00
fyrz 9a255b95f0 [RocksJava] Sample and Default value
- RocksDB ColumnFamilySample adjusted to C++ sample.
- DefaultColumnFamily is available now as constant in RocksDB.
2014-11-12 19:49:13 +01:00
fyrz 9d2ba21361 [RocksJava] Incorporated review comments 2014-11-12 19:28:26 +01:00
fyrz fa9cfc65f3 [RocksJava] Integrated Review comments from yhchiang in D28023 2014-11-12 19:28:25 +01:00
fyrz 75010d2084 [RocksJava] ColumnFamily custom Options API extension
*********************
                   ***************************
                 ******** ************* ********
                ********   ***********   ********
               ********     *********     ********
              *************************************
              *************************************
              *************************************
               ******     ***       ***     ******
                ******    ***  ***  ***    ******
                 ******        ***        ******
                   ***************************
                      *********************
2014-11-12 19:28:24 +01:00
fyrz 0345c2156f [RocksJava] Extend Options with ColumnFamilyOptions implementation ColumnFamilyOptions implementation with tests
[RocksJava] Extended ColumnFamilyTest

Summary: Options Refactoring split part 3

Test Plan:
make rocksdbjava
make jtest

Reviewers: yhchiang, ankgup87

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28023
2014-11-12 19:28:23 +01:00
Yueh-Hsuan Chiang 9759495229 Fixed clang compile error in version_builder_test
Summary: Fixed clang compile error in version_builder_test

Test Plan: ./version_builder_test

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: sdong, dhruba

Differential Revision: https://reviews.facebook.net/D28731
2014-11-11 15:22:06 -08:00
Yueh-Hsuan Chiang 5811419357 Fixed GetEstimatedActiveKeys
Summary:
Fixed a bug in GetEstimatedActiveKeys which does not normalized
the sampled information correctly.

Add a test in version_builder_test.

Test Plan: version_builder_test

Reviewers: ljin, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28707
2014-11-11 14:28:18 -08:00
Igor Canadi 1f621e6abc Fix additional -Wshorten-64-to-32 errros 2014-11-11 14:09:14 -08:00
Igor Canadi 767777c2bd Turn on -Wshorten-64-to-32 and fix all the errors
Summary:
We need to turn on -Wshorten-64-to-32 for mobile. See D1671432 (internal phabricator) for details.

This diff turns on the warning flag and fixes all the errors. There were also some interesting errors that I might call bugs, especially in plain table. Going forward, I think it makes sense to have this flag turned on and be very very careful when converting 64-bit to 32-bit variables.

Test Plan: compiles

Reviewers: ljin, rven, yhchiang, sdong

Reviewed By: yhchiang

Subscribers: bobbaldwin, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28689
2014-11-11 16:47:22 -05:00
Igor Canadi 113796c493 Fix NewFileNumber()
Summary: I mistakenly changed the behavior to ++next_file_number_ instead of next_file_number_++, as it should have been: 344edbb044/db/version_set.h (L539)

Test Plan: none. not sure if this would break anything. It's just different behavior, so I'd rather not risk

Reviewers: ljin, rven, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28557
2014-11-11 06:58:47 -08:00
Yueh-Hsuan Chiang 625e162c69 Merge pull request #393 from fyrz/RocksJava-Flush
[RocksJava] Memtable flush functionality
2014-11-11 01:11:59 -08:00
fyrz fc6fcbab9e [RocksJava] Flush functionality
RocksJava now supports also flush functionality of
RocksDB.
2014-11-11 07:47:38 +01:00
fyrz 8e5547f64f [RocksJava] Makefile restructured 2014-11-11 07:47:38 +01:00
Yueh-Hsuan Chiang 26f0a78b03 Merge pull request #363 from adamretter/write_batch-iterate
Iterator support for Write Batches
2014-11-10 22:12:11 -08:00
Tomislav Novak 35c8c814e8 Make ForwardIterator::status() more efficient
Summary:
In D19581 I made `ForwardIterator::status()` check all child iterators,
including immutable ones. It's, however, not necessary to do it every
time -- it'll suffice to check only when they're used and their status
could change.

This diff:
* introduces `immutable_status_` which is updated by `Seek()` and `Next()`
* removes special handling of `kIncomplete` status in those methods

Test Plan:
* `db_test`
* hacked ReadSequential in db_bench.cc to check `status()` in addition to
  validity:

```
   $ ./db_bench -use_existing_db -benchmarks readseq -disable_auto_compactions \
      -use_tailing_iterator  # without this patch
   Keys:       16 bytes each
   Values:     100 bytes each (50 bytes after compression)
   Entries:    1000000
   [...]
   DB path: [/dev/shm/rocksdbtest/dbbench]
   readseq      :       0.562 micros/op 1778103 ops/sec;   98.4 MB/s
   $ ./db_bench -use_existing_db -benchmarks readseq -disable_auto_compactions \
      -use_tailing_iterator  # with the patch
   readseq      :       0.433 micros/op 2311363 ops/sec;  127.8 MB/s
```

Reviewers: igor, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb, march, lovro

Differential Revision: https://reviews.facebook.net/D24063
2014-11-10 15:44:20 -08:00
Igor Canadi d88568c68d Move -Wnon-virtual-dtor to c++ flags 2014-11-10 17:41:51 -05:00
Igor Canadi c7ee9c3ab7 Fix -Wnon-virtual-dtor errors
Summary: This breaks mongo+rocks build https://mci.10gen.com/task_log_raw/rocksdb_ubuntu1404_rocksdb_c6e8e3d868660dc66b3bbd438cdc135df6356c5a_14_11_10_21_36_10_compile_ubuntu1404_rocksdb/0?type=T

Test Plan: m check + -Wnon-virtual-dtor

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28653
2014-11-10 17:39:38 -05:00
Igor Canadi 746252197b Merge pull request #384 from msb-at-yahoo/compaction-filter-2-empty-changed-values
CompactionFilterV2: eliminate an often unnecessary allocation.
2014-11-10 16:12:15 -05:00
sdong dd726a59ef Bump Version Number to 3.8
Summary: As tittle.

Test Plan: Not needed.

Reviewers: ljin, igor, yhchiang, rven, fpi

Reviewed By: fpi

Subscribers: leveldb, fpi, dhruba

Differential Revision: https://reviews.facebook.net/D28629
2014-11-10 12:10:56 -08:00
Igor Canadi 4a3bd2bad2 Optimize usage of Status in CompactionJob
Summary: Based on @ljin feedback

Test Plan: compiles

Reviewers: ljin, yhchiang, sdong

Reviewed By: sdong

Subscribers: ljin, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28515
2014-11-10 11:57:58 -08:00
Igor Canadi bcdb9671c0 Fix build 2014-11-09 13:01:50 -05:00
Adam Retter d904fbbb0b Addresed comments from code review https://reviews.facebook.net/D27567 2014-11-09 16:10:11 +00:00
Adam Retter eeb9cf6c42 Test for WriteBatchHandler 2014-11-09 16:10:11 +00:00
Adam Retter 8fb4751d50 Iterator support for Write Batches 2014-11-09 16:10:11 +00:00
Igor Canadi 00211f9c5b Fix SIGSEGV in db_stresS 2014-11-08 13:01:31 -08:00
Igor Canadi a4a2bfd6bd Merge pull request #391 from Liuchang0812/fixmake
remove unused target
2014-11-08 15:52:30 -05:00
Igor Canadi 856059094e Merge pull request #389 from Liuchang0812/master
fix "make static_lib" error
2014-11-08 15:52:18 -05:00
liuchang 01a7706379 remove unused target 2014-11-08 11:51:40 -08:00
liuchang dc34104637 Merge branch 'fix-example' 2014-11-08 11:35:36 -08:00
liuchang e7620536cf fix make static_lib error 2014-11-08 11:35:10 -08:00
Federico Piccinini 543df158c5 Expose sst_dump functionality as library call.
Summary:
Refactor sst_dump to follow the same structure as ldb. Introduce a
SSTDump interface.

Test Plan: built sst_dump and tried it manually.

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28485
2014-11-07 17:23:58 -08:00