Commit graph

2866 commits

Author SHA1 Message Date
fyrz a280af2a57 [RocksJava] Sigsegv fix for MergerOperatorByName 2014-11-25 22:26:02 +01:00
fyrz fcc2dfd9f9 [RocksJava] Support for stored snapshots
Summary:
RocksDB supports two ways of saving snapshots. In
memory and on disk. The later was added with this
pull request to RocksJava.

Test Plan:

Reviewers:

Subscribers:
2014-11-25 22:25:22 +01:00
Yueh-Hsuan Chiang 274ba62707 Block internal_stats in ROCKSDB_LITE
Summary: Block internal_stats in ROCKSDB_LITE.

Test Plan: make OPT=-DROCKSDB_LITE shared_lib

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29541
2014-11-25 12:01:27 -08:00
Igor Canadi 4f2e8bab5f Merge pull request #421 from fyrz/RocksJava-PrecisionFix
[RocksJava] Fix precision problem in rocksjni
2014-11-25 14:32:44 -05:00
fyrz c4765dc10b [RocksJava] Fix precision problem in rocksjni 2014-11-25 20:29:52 +01:00
Igor Canadi 14788e1811 Merge pull request #420 from rdallman/add-wal
c api: add max wal total to opts
2014-11-25 09:34:15 -05:00
Reed Allman 88dd8d889b c api: add max wal total to opts 2014-11-24 22:00:29 -08:00
Yueh-Hsuan Chiang 7e608e2fe3 Block plain_table_index.cc in ROCKSDB_LITE
Summary: Block plain_table_index.cc in ROCKSDB_LITE

Test Plan:
make clean
make OPT=-DROCKSDB_LITE shared_lib -j32
make clean
make shared_lib -j32

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29535
2014-11-24 20:47:27 -08:00
Yueh-Hsuan Chiang 13de000f07 Add rocksdb::ToString() to address cases where std::to_string is not available.
Summary:
In some environment such as android, the c++ library does not have
std::to_string.  This path adds rocksdb::ToString(), which wraps std::to_string
when std::to_string is not available, and implements std::to_string
in the other case.

Test Plan:
make dbg -j32
./db_test
make clean
make dbg OPT=-DOS_ANDROID -j32
./db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29181
2014-11-24 20:44:49 -08:00
Yueh-Hsuan Chiang 90ee85f8e1 Improve listener_test to avoid possible false alarm
Summary:
Improve listener_test to avoid possible false alarm

Test Plan:
./listener_test
2014-11-24 18:28:06 -08:00
Lei Jin 2946e37a08 remove unreliable test in db/cuckoo_table_db_test.cc
Summary:
This compaction trigger does not seem to test any thing specific to
cuckoo table. Remove it.

Test Plan: make all check

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29523
2014-11-24 15:18:09 -08:00
Lei Jin 9c7ca65d21 free builders in VersionSet::DumpManifest
Summary:
Reported by bootcamper
This causes ldb tool to fail the assertion in ~ColumnFamilyData()

Test Plan:
./ldb --db=/tmp/test_db1 --create_if_missing put a1 b1
./ldb manifest_dump --path=/tmp/test_db1/MANIFEST-000001

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29517
2014-11-24 15:03:08 -08:00
Igor Canadi 7530c75ab0 Merge pull request #413 from saghmrossi/master
first rdb commit
2014-11-24 16:54:44 -05:00
Igor Canadi d699d70343 Make RocksDB compile without gflags
Summary: We want to make sure people without gflags can compile RocksDB.

Test Plan: remove gflags, make all

Reviewers: sdong, rven, yhchiang, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29469
2014-11-24 15:53:23 -05:00
Venkatesh Radhakrishnan 3257221499 Fixes valgrind error in GetSnapshotLink. Free checkpoint now.
Summary: Free checkpoint after its directory is removed.

Test Plan: Run valgrind with GetSnapshotLink.

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29493
2014-11-24 10:20:50 -08:00
Igor Canadi ada3d78734 Merge pull request #415 from fyrz/RocksJava-Makefile
[RocksJava] Makefile correction
2014-11-22 22:53:01 -05:00
Yueh-Hsuan Chiang 569853ed10 Fix leak when create_missing_column_families=true on ThreadStatus
Summary:
An entry of ConstantColumnFamilyInfo is created when:
1. DB::Open
2. CreateColumnFamily.

However, there are cases that DB::Open could also call CreateColumnFamily
when create_missing_column_families=true.  As a result, it will create
duplicate ConstantColumnFamilyInfo and one of them would be leaked.

Test Plan: ./deletefile_test

Reviewers: igor, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29307
2014-11-22 00:04:41 -08:00
fyrz c4b65f70fb [RocksJava] Makefile correction
Prevent non exist error message while executing
clean twice.
2014-11-21 23:49:31 +01:00
Shaohua Li 1410180167 Make arena use hugepage if possible
Summary:
arena doesn't use huge page by default. This change will make it happen
if possible. A new paramerter is added for Arena(). If it's set, Arena
will use huge page always. If huge page allocation fails, Arena
allocation will fallback to malloc().

Test Plan:
Change util/arena_test to support huge page allocation.
Run below tests:
1. normal regression test:
  make check
2. Check if huge page allocation works
  echo 50 > /proc/sys/vm/nr_hugepages
  make check

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D28647
2014-11-21 14:11:40 -08:00
sdong 3a40c427b9 Fix db_bench on CLANG mode
Summary: "build all" breaks in Clang mode with db_bench. Fix it.

Test Plan: USE_CLANG=1 make all

Reviewers: ljin, rven, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D29379
2014-11-21 11:30:22 -08:00
Yueh-Hsuan Chiang 9222a2d024 Fixed iOS build caused by GetThreadList feature.
Summary:
Fixed iOS build caused by GetThreadList feature.
2014-11-21 11:00:42 -08:00
Yueh-Hsuan Chiang aa31fc5068 Improve listener_test by ensuring flushes are completed before assert.
Summary: Improve listener_test by ensuring flushes are completed before assert.

Test Plan: listener_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29319
2014-11-21 10:22:05 -08:00
Igor Canadi 7ec71f101c Provide default implementation of LinkFile, don't break the build
Summary: By providing default implementation of LinkFile, we don't break other implementations of Env.

Test Plan: none

Reviewers: rven, dhruba

Reviewed By: dhruba

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29355
2014-11-21 11:05:48 -05:00
Igor Canadi cd278584c9 Clean up StringSplit
Summary: stringSplit is not how we name our functions. Also, we had two StringSplit's in the codebase

Test Plan: make check

Reviewers: yhchiang, dhruba

Reviewed By: dhruba

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29361
2014-11-21 11:05:28 -05:00
Igor Canadi d84069995c Fix mac compile 2014-11-21 09:42:45 -05:00
Yueh-Hsuan Chiang 4f882924dd Merge pull request #404 from fyrz/RocksJava-Backup-Restore-3.8
[RocksJava] Update BackupableDB and RestoreBackupableDB to 3.8.0.
2014-11-21 00:07:13 -08:00
Yueh-Hsuan Chiang 4b63fcbff3 Add enable_thread_tracking to DBOptions
Summary:
Add enable_thread_tracking to DBOptions to allow
tracking thread status related to the DB.  Default is off.

Test Plan:
export ROCKSDB_TESTS=ThreadList
./db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29289
2014-11-20 21:13:18 -08:00
Saghm Rossi bafce61979 first rdb commit
Summary: First commit for rdb shell

Test Plan: unit_test.js does simple assertions on most of the main functionality; will update with rest of tests

Reviewers: igor, rven, lijn, yhciang, sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28749
2014-11-20 23:33:00 -05:00
Bryan Rosario 9e285d4238 Added CompatibleOptions for compatibility with LevelDB Options
Summary: Created a CompatibleOptions object that can be used as a LevelDB Options object and then converted to a RocksDB Options object using the ConvertOptions() method.

Test Plan: Unit test included in diff.

Reviewers: ljin

Reviewed By: ljin

Subscribers: sdong, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28893
2014-11-20 19:24:39 -08:00
Yueh-Hsuan Chiang 353307758b Add IOS_CROSS_COMPILE to macro guard for GetThreadList feature. 2014-11-20 16:13:20 -08:00
Yueh-Hsuan Chiang eecdebe65b Fixed the destruction order of static variables in ThreadStatusImpl. 2014-11-20 16:02:03 -08:00
Venkatesh Radhakrishnan 004f416b77 Moved checkpoint to utilities
Summary:
Moved checkpoint to utilities.
Addressed comments by Igor, Siying, Dhruba

Test Plan: db_test/SnapshotLink

Reviewers: dhruba, igor, sdong

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29079
2014-11-20 15:54:47 -08:00
Yueh-Hsuan Chiang beabc6879c Fixed ~ThreadStatusImpl(). 2014-11-20 15:45:56 -08:00
fyrz faa8d32be0 [RocksJava] Integrated changes from D29019. 2014-11-20 23:34:07 +01:00
fyrz 3d78c7a8cf [RocksJava] Lint adjustments 2014-11-20 22:51:31 +01:00
fyrz d7529b2de9 [RocksJava] Cleanup Backupable implementations
- Correct usage of isInitialized()
- Adjusted JavaDoc
2014-11-20 22:51:30 +01:00
fyrz fa703efb28 [RocksJava] Improved BackupableDBTest
- Splitted methods to meaningful tests
- Added tests for additional functionality
- Covered missing parts
2014-11-20 22:51:30 +01:00
fyrz 24fdc47416 [RocksJava] Backupable/Restorable DB update 3.8.0
- GarbageCollectMethod() available.
- GetCorruptedBackups() available.
2014-11-20 22:51:29 +01:00
fyrz 9972f969ee [RocksJava] BackupableDBOptions alginment + 3.8
- Updated the BackupableDBOptions functionality to 3.8.0.
- Aligned Options implementation with remaining source code.
- Invented test-case.
2014-11-20 22:51:29 +01:00
Yueh-Hsuan Chiang fbc42a0933 Fixed -Werror=unused-but-set-variable in thread_status_impl
Summary:
Fixed -Werror=unused-but-set-variable in thread_status_impl
2014-11-20 13:46:04 -08:00
Yueh-Hsuan Chiang a564be715d Fix asan error in thread_status_impl.cc 2014-11-20 11:47:18 -08:00
Yueh-Hsuan Chiang 7165d18869 Fix clang compile error 2014-11-20 11:00:21 -08:00
Yueh-Hsuan Chiang d0c5f28a5c Introduce GetThreadList API
Summary:
Add GetThreadList API, which allows developer to track the
status of each process.  Currently, calling GetThreadList will
only get the list of background threads in RocksDB with their
thread-id and thread-type (priority) set.  Will add more support
on this in the later diffs.

ThreadStatus currently has the following properties:

  // An unique ID for the thread.
  const uint64_t thread_id;

  // The type of the thread, it could be ROCKSDB_HIGH_PRIORITY,
  // ROCKSDB_LOW_PRIORITY, and USER_THREAD
  const ThreadType thread_type;

  // The name of the DB instance where the thread is currently
  // involved with.  It would be set to empty string if the thread
  // does not involve in any DB operation.
  const std::string db_name;

  // The name of the column family where the thread is currently
  // It would be set to empty string if the thread does not involve
  // in any column family.
  const std::string cf_name;

  // The event that the current thread is involved.
  // It would be set to empty string if the information about event
  // is not currently available.

Test Plan:
./thread_list_test
export ROCKSDB_TESTS=GetThreadList
./db_test

Reviewers: rven, igor, sdong, ljin

Reviewed By: ljin

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D25047
2014-11-20 10:49:32 -08:00
Yueh-Hsuan Chiang 1fd1aecb39 Merge pull request #409 from fyrz/RocksJava-Make-cleanup
[RocksJava] Build improvements
2014-11-20 00:01:48 -08:00
fyrz 91c8dcefc3 [RocksJava] Strip library in publish
Currently maven publishing uses the library with debug symbols. What
leads to unnecessary big library sizes. Included strip to remove
unnecessary stuff. 40M -> 2.7M
2014-11-20 07:28:59 +01:00
fyrz e7fcaa4d92 [RocksJava] JavaDoc is executed too often
Previous to this commit too much targets got dependencies
on javadocs target.

Introduced one additional target "javalib" which resolves
that situation. JavaDoc will now be generated once while
executing a task with prefix "rocksdbjava".
2014-11-20 07:28:16 +01:00
fyrz 2cd1794e4f [RocksJava] Make cleanup - Clean Target
- Remove JNI includes on clean
- Remove target folder generated by Maven
- Remove shared object
- Remove jar
2014-11-20 07:21:30 +01:00
Lei Jin be005e17bb fix clang compilation
Summary:
as title
2014-11-18 20:42:32 -08:00
Yueh-Hsuan Chiang 5e69f19c4c Merge pull request #405 from fyrz/RocksJava-Convenient-Options
[RocksJava] Convenience methods for Options
2014-11-18 20:39:51 -08:00
Igor Canadi 55a344872d Merge pull request #408 from fyrz/Missing-include
Missing header in build on CentOS 6.5, GCC4.7
2014-11-18 17:35:34 -08:00