Commit graph

2853 commits

Author SHA1 Message Date
sdong 1f04066cab Add DBProperty to return number of snapshots and time for oldest snapshot
Summary:
Add a counter in SnapshotList to show number of snapshots. Also a unix timestamp in every snapshot.
Add two DB Properties to return number of snapshots and timestamp of the oldest one.

Test Plan: Add unit test checking

Reviewers: yhchiang, rven, igor

Reviewed By: igor

Subscribers: leveldb, dhruba, MarkCallaghan

Differential Revision: https://reviews.facebook.net/D29919
2014-12-05 17:07:49 -08:00
Venkatesh Radhakrishnan 6436ba6b06 Provide mechanism to restart tests from previous error
Summary:
While running rocksdb tests, we sometimes encounter errors and
the test run stops. We now provide a new make target call check_some
which restarts the test run from a specific test and continues from
there depending on the value of the environment variable ROCKSDBTESTS_START

Test Plan:
Run make check_some with different values of
ROCKSDBTESTS_START.

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29913
2014-12-05 16:16:56 -08:00
Yueh-Hsuan Chiang d84b2badeb Replace exception by abort() in dummy HdfsEnv implementation.
Summary: Replace exception by abort() in dummy HdfsEnv implementation.

Test Plan: make dbg -j32

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29895
2014-12-05 13:30:57 -08:00
Igor Canadi 9260e1ad74 Bump version to 3.9 2014-12-05 11:05:24 -08:00
Yueh-Hsuan Chiang 8f4e1c1c9a Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE
Summary:
Remove the compability check on log2 OS_ANDROID as it's already blocked by ROCKSDB_LITE

Test Plan:
make OPT="-DROCKSDB_LITE -DOS_ANDROID" shared_lib -j32
make shared_lib -j32
2014-12-04 13:56:14 -08:00
Yueh-Hsuan Chiang c4a7423c1d Replace runtime_error exception by abort() in thread_local
Summary: Replace runtime_error exception by abort() in thread_local

Test Plan: make dbg -j32

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29853
2014-12-04 13:35:31 -08:00
Yueh-Hsuan Chiang a94d54aa47 Remove the use of exception in WriteBatch::Handler
Summary:
Remove the use of exception in WriteBatch::Handler.  Now the default
implementations of Put, Merge, and Delete in WriteBatch::Handler are no-op.

Test Plan:
Add three test cases in write_batch_test
./write_batch_test

Reviewers: sdong, igor

Reviewed By: sdong, igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29835
2014-12-04 12:01:55 -08:00
Yueh-Hsuan Chiang a5d4fc0a25 Fix compile warning in db_stress
Summary:
Fix compile warning in db_stress

Test Plan:
make db_stress
2014-12-04 11:59:29 -08:00
Yueh-Hsuan Chiang 1a8f4821a7 Replace exception by assertion in autovector
Summary: Replace exception by assertion in autovector

Test Plan: autovector_test

Reviewers: sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29847
2014-12-04 11:41:56 -08:00
Yueh-Hsuan Chiang 97c1940882 Fix compile warning in db_stress.cc on Mac
Summary:
Fix the following compile warning in db_stress.cc on Mac
tools/db_stress.cc:1688:52: error: format specifies type 'unsigned long' but the argument has type '::google::uint64' (aka 'unsigned long long') [-Werror,-Wformat]
    fprintf(stdout, "DB-write-buffer-size: %lu\n", FLAGS_db_write_buffer_size);
                                           ~~~     ^~~~~~~~~~~~~~~~~~~~~~~~~~
                                           %llu

Test Plan:
make
2014-12-04 11:19:12 -08:00
Yueh-Hsuan Chiang 5f719d7202 Replace exception by setting valid_ = false in DBIter::MergeValuesNewToOld()
Summary: Replace exception by setting valid_ = false in DBIter::MergeValuesNewToOld().

Test Plan:
Not sure if I am right at this, but it seems we currently don't have a good
way to test that code path as it requires dynamically set merge_operator = nullptr
at the time while Merge() is calling.

Reviewers: igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29811
2014-12-04 11:11:11 -08:00
Mark Callaghan c0dee851c3 Improve formatting, add missing newlines
Summary:
Improve formatting

Task ID: #

Blame Rev:

Test Plan:
make

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D29829
2014-12-04 10:34:06 -08:00
Igor Canadi 815f638cd0 Fix java build 2014-12-03 19:06:57 -08:00
Mark Callaghan 32a0a03844 Add Moved(GB) to Compaction IO stats
Summary:
Adds counter for bytes moved (files pushed down a level rather than compacted) to compaction
IO stats as Moved(GB). From the output removed these infrequently used columns: RW-Amp, Rn(cnt), Rnp1(cnt),
Wnp1(cnt), Wnew(cnt).
Example old output:
Level   Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s)  Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt)  Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) RecordIn RecordDrop
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0     0/0          0   0.0      0.0     0.0      0.0    2130.8   2130.8    0.0   0.0      0.0    109.1        0         0         0         0      20002     25068    0.798      28.75     182059    0.16       0          0
  L1   142/0        509   1.0   4618.5  2036.5   2582.0    4602.1   2020.2    4.5   2.3     88.5     88.1    24220    701246   1215528    514282      53466      4229   12.643       0.00          0    0.002032745988  300688729

Example new output:
Level   Files   Size(MB) Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms)     RecordIn   RecordDrop
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0     7/0         13   1.8      0.0     0.0      0.0       0.6      0.6       0.0   0.0      0.0     14.7        44       353    0.124       0.03        626    0.05            0            0
  L1     9/0         16   1.6      0.0     0.0      0.0       0.0      0.0       0.6   0.0      0.0      0.0         0         0    0.000       0.00          0    0.00            0            0

Task ID: #

Blame Rev:

Test Plan:
make check, run db_bench --fillseq --stats_per_interval --stats_interval and look at output

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D29787
2014-12-03 18:28:39 -08:00
Jonah Cohen a14b7873ee Enforce write buffer memory limit across column families
Summary:
Introduces a new class for managing write buffer memory across column
families.  We supplement ColumnFamilyOptions::write_buffer_size with
ColumnFamilyOptions::write_buffer, a shared pointer to a WriteBuffer
instance that enforces memory limits before flushing out to disk.

Test Plan: Added SharedWriteBuffer unit test to db_test.cc

Reviewers: sdong, rven, ljin, igor

Reviewed By: igor

Subscribers: tnovak, yhchiang, dhruba, xjin, MarkCallaghan, yoshinorim

Differential Revision: https://reviews.facebook.net/D22581
2014-12-02 12:09:20 -08:00
Igor Canadi 37d73d597e Fix linters
Summary:
Two fixes:
1. if cpplint is not present on the system, don't return a confusing error in the linter
2. Add include_alpha, which means our includes should be sorted lexicographically

Test Plan: Tried unsorting our includes, lint complained

Reviewers: rven, ljin, yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D28845
2014-12-02 13:53:39 -05:00
Igor Canadi 9a632b4a92 Merge pull request #429 from fyrz/RocksJava-MacOSX-strip-fix
[RocksJava] MacOSX strip support
2014-12-01 13:13:02 -05:00
fyrz b426675061 [RocksJava] MacOSX strip support 2014-12-01 19:01:29 +01:00
Igor Canadi e463cb0bcf Merge pull request #424 from eile/master
Tweak Makefile for building on BG/Q
2014-12-01 10:20:12 -05:00
Stefan Eilemann 91d8981639 Tweak Makefile for building on BG/Q 2014-12-01 09:01:54 +01:00
Igor Canadi 703ef66a86 Merge pull request #426 from fyrz/RocksJava-Restore-PrecisionFix
[RocksJava] Fixed MacOS build of RocksJava
2014-11-27 20:39:39 -05:00
Yueh-Hsuan Chiang bcf9086899 Block Universal and FIFO compactions in ROCKSDB_LITE
Summary: Block Universal and FIFO compactions in ROCKSDB_LITE

Test Plan:
make shared_lib -j32
make OPT=-DROCKSDB_LITE shared_lib

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29589
2014-11-26 15:45:11 -08:00
fyrz 67cb7ca758 [RocksJava] Fixed MacOS build of RocksJava
There were still some precision loss problems
remainging in RocksJava. This pull request resolve
these.
2014-11-26 20:53:23 +01:00
Yueh-Hsuan Chiang b8136a7d27 Merge pull request #398 from fyrz/RocksJava-CreateCheckPoint
[RocksJava] Support for stored snapshots
2014-11-26 11:40:41 -08:00
Yueh-Hsuan Chiang 533592a27d Merge pull request #401 from fyrz/RocksJava-Sigsegv-MergeOperatorName
[RocksJava] Fixes in MergeOperatorByName
2014-11-26 11:40:13 -08:00
Yueh-Hsuan Chiang 73d72ed5c7 Block ReadOnlyDB in ROCKSDB_LITE
Summary:
db_imp_readonly.o is one of the big obj file.  If it's not a necessary
feature, we should probably block it in ROCKSDB_LITE.

    1322704 Nov 24 16:55 db/db_impl_readonly.o

Test Plan:
make shared_lib -j32
make ROCKSDB_LITE shared_lib -j32

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29583
2014-11-26 11:37:59 -08:00
Igor Canadi e47f0fa9ed Merge pull request #425 from adamretter/macosx-clean-fix
Mac OS X jclean fix
2014-11-26 13:40:32 -05:00
Adam Retter ff0cb90d1c Do not delete Java Fatal Error Log, developers may still want these for
reference
2014-11-26 18:18:04 +00:00
Adam Retter 2a792cd300 There will also be a librocksdbjni-osx.jnilib.dSYM folder on MacOSX
builds to be deleted
2014-11-26 18:17:54 +00:00
Yueh-Hsuan Chiang beb74c14ca Fix travis-build error
Summary:
Fix travis-build error
2014-11-26 09:42:13 -08:00
Igor Canadi a486352e07 Merge pull request #423 from zerebubuth/c_iterate_upper_bound
C iterate upper bound
2014-11-26 10:14:16 -05:00
Matt Amos 26109d487a Store upper bound Slice with the same lifetime as the ReadOptions so that we can provide a pointer to it. 2014-11-26 11:29:13 +00:00
Yueh-Hsuan Chiang a97314219e Fix compile error in ROCKSDB_LITE 2014-11-25 23:39:52 -08:00
Yueh-Hsuan Chiang 9d5019327b Replace log2 by implementing Log2 in options_builder
Summary:
log2 function is only used in options_builder, and this function
is not available under certain platform such as android.
This patch implements Log2 by log(n) / log(2).

Test Plan:
make
2014-11-25 23:28:36 -08:00
Matt Amos 805bac6d25 Add test for upper bounds on iterators using C interface. 2014-11-25 23:07:40 +00:00
fyrz f193deea31 [RocksJava] Addressed comments in D28971 2014-11-25 23:33:42 +01:00
fyrz 94f70a86b9 [RocksJava] Incoroporated changes for D29013 2014-11-25 22:26:03 +01:00
fyrz a280af2a57 [RocksJava] Sigsegv fix for MergerOperatorByName 2014-11-25 22:26:02 +01:00
fyrz fcc2dfd9f9 [RocksJava] Support for stored snapshots
Summary:
RocksDB supports two ways of saving snapshots. In
memory and on disk. The later was added with this
pull request to RocksJava.

Test Plan:

Reviewers:

Subscribers:
2014-11-25 22:25:22 +01:00
Yueh-Hsuan Chiang 274ba62707 Block internal_stats in ROCKSDB_LITE
Summary: Block internal_stats in ROCKSDB_LITE.

Test Plan: make OPT=-DROCKSDB_LITE shared_lib

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29541
2014-11-25 12:01:27 -08:00
Igor Canadi 4f2e8bab5f Merge pull request #421 from fyrz/RocksJava-PrecisionFix
[RocksJava] Fix precision problem in rocksjni
2014-11-25 14:32:44 -05:00
fyrz c4765dc10b [RocksJava] Fix precision problem in rocksjni 2014-11-25 20:29:52 +01:00
Igor Canadi 14788e1811 Merge pull request #420 from rdallman/add-wal
c api: add max wal total to opts
2014-11-25 09:34:15 -05:00
Reed Allman 88dd8d889b c api: add max wal total to opts 2014-11-24 22:00:29 -08:00
Yueh-Hsuan Chiang 7e608e2fe3 Block plain_table_index.cc in ROCKSDB_LITE
Summary: Block plain_table_index.cc in ROCKSDB_LITE

Test Plan:
make clean
make OPT=-DROCKSDB_LITE shared_lib -j32
make clean
make shared_lib -j32

Reviewers: ljin, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29535
2014-11-24 20:47:27 -08:00
Yueh-Hsuan Chiang 13de000f07 Add rocksdb::ToString() to address cases where std::to_string is not available.
Summary:
In some environment such as android, the c++ library does not have
std::to_string.  This path adds rocksdb::ToString(), which wraps std::to_string
when std::to_string is not available, and implements std::to_string
in the other case.

Test Plan:
make dbg -j32
./db_test
make clean
make dbg OPT=-DOS_ANDROID -j32
./db_test

Reviewers: ljin, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29181
2014-11-24 20:44:49 -08:00
Yueh-Hsuan Chiang 90ee85f8e1 Improve listener_test to avoid possible false alarm
Summary:
Improve listener_test to avoid possible false alarm

Test Plan:
./listener_test
2014-11-24 18:28:06 -08:00
Lei Jin 2946e37a08 remove unreliable test in db/cuckoo_table_db_test.cc
Summary:
This compaction trigger does not seem to test any thing specific to
cuckoo table. Remove it.

Test Plan: make all check

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29523
2014-11-24 15:18:09 -08:00
Lei Jin 9c7ca65d21 free builders in VersionSet::DumpManifest
Summary:
Reported by bootcamper
This causes ldb tool to fail the assertion in ~ColumnFamilyData()

Test Plan:
./ldb --db=/tmp/test_db1 --create_if_missing put a1 b1
./ldb manifest_dump --path=/tmp/test_db1/MANIFEST-000001

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D29517
2014-11-24 15:03:08 -08:00
Igor Canadi 7530c75ab0 Merge pull request #413 from saghmrossi/master
first rdb commit
2014-11-24 16:54:44 -05:00