Commit graph

4268 commits

Author SHA1 Message Date
Alexey Maykov fa4b5b3db8 Fix for the travis build caused by my previous commit
Summary: My previous commit ('Passing table properties to compaction callback') broke the clang build. Here is the fix.

Test Plan: USE_CLANG=1 make all -j

Reviewers: igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D48489
2015-10-09 19:37:51 -07:00
Alexey Maykov 3d07b815f6 Passing table properties to compaction callback
Summary: It would be nice to have and access to table properties in compaction callbacks. In MyRocks project, it will make possible to update optimizer statistics online.

Test Plan: ran the unit test. Ran myrocks with the new way of collecting stats.

Reviewers: igor, rven, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D48267
2015-10-09 18:10:55 -07:00
krad 64546af83e Adding parser to CI jobs
Summary: The parser will help parse the output in order to send meaningful
notifications.

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-10-09 17:58:31 -07:00
agiardullo def74f8763 Deferred snapshot creation in transactions
Summary: Support for Transaction::CreateSnapshotOnNextOperation().  This is to fix a write-conflict race-condition that Yoshinori was running into when testing MyRocks with LinkBench.

Test Plan: New tests

Reviewers: yhchiang, spetrunia, rven, igor, yoshinorim, sdong

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48099
2015-10-09 15:46:16 -07:00
agiardullo c5f3707d42 DisableIndexing() for Transactions
Summary:
MyRocks reported some perfomance issues when inserting many keys into a transaction due to the cost of inserting new keys into WriteBatchWithIndex.  Frequently, they don't even need the keys to be indexed as they don't need to read them back.  DisableIndexing() can be used to avoid the cost of indexing.

I also plan on eventually investigating if we can improve WriteBatchWithIndex performance.  But even if we improved the perf here, it is still beneficial to be able to disable the indexing all together for large transactions.

Test Plan: unit test

Reviewers: igor, rven, yoshinorim, spetrunia, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48471
2015-10-09 15:36:09 -07:00
sdong 776bd8d5eb Pass column family ID to table property collector
Summary: Pass column family ID through TablePropertiesCollectorFactory::CreateTablePropertiesCollector() so that users can identify which column family this file is for and handle it differently.

Test Plan: Add unit test scenarios in tests related to table properties collectors to verify the information passed in is correct.

Reviewers: rven, yhchiang, anthony, kradhakrishnan, igor, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: yoshinorim, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D48411
2015-10-09 14:36:51 -07:00
James Lent 5a7222782a Ensure that the compression libraries are statically linked into dynamic libraries included
in the Java jar.  Also build the linux libraries using the portable flag to fix a problem with
the linux32 build and improve the general portability of the RocksDB dynamic libraries.
==> linux32: util/crc32c.cc:318:39: error: ‘_mm_crc32_u64’ was not declared in this scope
2015-10-09 14:41:40 -04:00
sdong e61d9c1484 Make DBTest.AggregatedTableProperties more deterministic
Summary: Now based on environment, DBTest.AggregatedTableProperties has a possibility of issuing a L0->L1 compaction after reopening and the results are not what we expected. We tune the L0 compaction trigger to make it less likely to happen.

Test Plan: I can't repro the failure but I think the change is better. Just run the test and make sure it passes.

Reviewers: kradhakrishnan, yhchiang, igor

Reviewed By: igor

Subscribers: leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D48423
2015-10-09 09:47:56 -07:00
sdong b77eb16aba New Manifest format to allow customized fields in NewFile.
Summary: With this commit, we add a new format in manifest when adding a new file. Now path ID and need-compaction hint are first two customized fields.

Test Plan: Add a test case in version_edit_test to verify the encoding and decoding logic. Add a unit test in db_test to verify need compaction is persistent after DB restarting.

Reviewers: kradhakrishnan, anthony, IslamAbdelRahman, yhchiang, rven, igor

Reviewed By: igor

Subscribers: javigon, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D48123
2015-10-08 15:51:45 -07:00
Igor Canadi 6732a5765d Merge pull request #756 from viveknelamangala/master
Install snappy headers to standard locations using yum, so that build…
2015-10-08 15:00:05 -07:00
Vivek Nelamangala a52888ed00 Install snappy headers to standard locations using yum, so that build_tools/build_detect_platform sets -DSNAPPY flag to g++ . Current jars of rocksdb do no have snappy compression avaliable . 2015-10-08 14:52:22 -07:00
Igor Canadi 831101b5fa Make it harder for users to run debug builds in production
Summary:
I see a lot of users compiling RocksDB with `make` or `make all` and then using those binaries in production. They end up running debug builds :(

This diff makes it harder for them:
1. I added an explicit warning to INSTALL.md
2. When you compile with `make all`, your resulting library will be librocksdb_debug.a
3. I also print out a warning when you compile in debug mode.

Hopefully should be enough :)

Test Plan: none

Reviewers: rven, yhchiang, kradhakrishnan, anthony, sdong, dhruba, IslamAbdelRahman

Reviewed By: IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48093
2015-10-08 14:11:32 -07:00
sdong 000836a880 CompactionFilter::Context to contain column family ID
Summary: Add the column family ID to compaction filter context, so it is easier for compaction filter to apply different logic for different column families.

Test Plan: Add a unit test to verify the column family ID passed is correct.

Reviewers: rven, yhchiang, igor

Reviewed By: igor

Subscribers: yoshinorim, leveldb, dhruba

Differential Revision: https://reviews.facebook.net/D48357
2015-10-08 11:27:38 -07:00
Yueh-Hsuan Chiang 3a0bf873b5 Change RocksDB version to 4.1
Summary: Change RocksDB version to 4.1

Test Plan: no code change.

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48387
2015-10-08 11:15:18 -07:00
krad 77318ee14b Enable crash CI jobs
Summary: Enabling CI crash test jobs

Test Plan: Manual testing

Reviewers:

CC: leveldb@

Task ID: #8635140

Blame Rev:
2015-10-08 10:14:43 -07:00
Igor Canadi 9803e0d813 compaction_filter.h cleanup
Summary:
Two changes:
1. remove *V2 filter stuff. we deprecated that a while ago
2. clarify what happens when user sets max_subcompactions to bigger than 1

Test Plan: none

Reviewers: yhchiang, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47871
2015-10-08 09:32:50 -07:00
Islam AbdelRahman 51fa7ecec5 Bytes read/written from cache statistics
Summary: Add 2 new counters BLOCK_CACHE_BYTES_WRITE, BLOCK_CACHE_BYTES_READ to keep track of how many bytes were written to the cache and how many bytes that we read from cache

Test Plan: make check

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D48195
2015-10-07 15:17:20 -07:00
Igor Canadi f925208ffc Create Makefile target unity_test
Summary: unity_test will compile db_test with unity.a instead of librocksdb.a. This will test both the compilation and some small amount of runtime.

Test Plan: This is a test :)

Reviewers: kradhakrishnan

Reviewed By: kradhakrishnan

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48297
2015-10-07 14:46:18 -07:00
dyniusz a065cdb388 bloom hit/miss stats for SST and memtable
Summary:
	hit and miss bloom filter stats for memtable and SST
	stats added to perf_context struct
	key matches and prefix matches combined into one stat

Test Plan: unit test veryfing the functionality added, see BloomStatsTest in db_test.cc for details

Reviewers: yhchiang, igor, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47859
2015-10-07 11:23:20 -07:00
Igor Canadi 40cdf797d2 Fix compile error on platforms without fallocate()
Summary:
If a platform doesn't have ROCKSDB_FALLOCATE_PRESENT, then compiler complains:

util/env_posix.cc:354:8: error: private field 'allow_fallocate_' is not used [-Werror,-Wunused-private-field]

This was caught by travis.

Test Plan: compiles with ROCKSDB_FALLOCATE_PRESENT.

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48327
2015-10-07 11:02:23 -07:00
Igor Canadi 77e4ad7ce2 Fix compile failure on Travis
Summary:
Travis is complaining against using {} to initialize KVMap: https://travis-ci.org/facebook/rocksdb/jobs/84132600

      db/compaction_job_test.cc:526:26: error: chosen constructor is explicit in copy-initialization
        RunCompaction({files}, {});

This diff should fix it

Test Plan: travis

Reviewers: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48309
2015-10-07 10:17:47 -07:00
Lakshmi Narayanan 4049bcde39 Added boolean variable to guard fallocate() calls
Summary:
Added boolean variable to guard fallocate() calls.
Set to false to prevent space leaks when tests fail.

Test Plan:
Compliles
Set to false and ran log device tests

Reviewers: sdong, lovro, igor

Reviewed By: igor

Subscribers: dhruba

Differential Revision: https://reviews.facebook.net/D48027
2015-10-07 10:04:05 -07:00
Igor Canadi aadf49fe6b Travis shouldn't fail when clang-format suggests improvements
Summary: This diff avoids failing travis test when clang-format fails. We should keep the test green even if clang-format fails. It would be good to look at travis output for pull requests to make sure they're following the good code style. However, we broke travis test 3 times today because of formatting issues. We should keep travis test green if the only thing wrong is the formatting.

Test Plan: none

Reviewers: sdong, yhchiang, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48291
2015-10-07 09:41:21 -07:00
Igor Canadi d80ce7f99a Compaction filter on merge operands
Summary:
Since Andres' internship is over, I took over https://reviews.facebook.net/D42555 and rebased and simplified it a bit.

The behavior in this diff is a bit simpler than in D42555:
* only merge operators are passed through FilterMergeValue(). If fitler function returns true, the merge operator is ignored
* compaction filter is *not* called on: 1) results of merge operations and 2) base values that are getting merged with merge operands (the second case was also true in previous diff)

Do we also need a compaction filter to get called on merge results?

Test Plan: make && make check

Reviewers: lovro, tnovak, rven, yhchiang, sdong

Reviewed By: sdong

Subscribers: noetzli, kolmike, leveldb, dhruba, sdong

Differential Revision: https://reviews.facebook.net/D47847
2015-10-07 09:30:03 -07:00
krad 726d9ce1a6 Disabling unity
Summary: Unity CI job is broken. Disabling.

Test Plan: Manual test

Reviewers:

CC: leveldb@

Task ID: #8632483

Blame Rev:
2015-10-06 17:51:31 -07:00
dyniusz 0267502655 Support for LevelDB SST with .ldb suffix
Summary:
	Handle SST files with both ".sst" and ".ldb" suffix.
	This enables user to migrate from leveldb to rocksdb.

Test Plan:
        Added unit test with DB operating on SSTs with names schema.
        See db/dc_test.cc:SSTsWithLdbSuffixHandling for details

Reviewers: yhchiang, sdong, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48003
2015-10-06 17:46:22 -07:00
Siying Dong 5855cdb6d0 Merge pull request #750 from yuslepukhin/fixup_build_options
FIx up build options
2015-10-06 16:41:29 -07:00
Igor Canadi 7bbe10c013 Merge pull request #751 from yuslepukhin/return_noerror
Mmap reads should not return error if reading past file
2015-10-06 16:34:43 -07:00
Dmitri Smirnov e95b703b7f Mmap reads should not return error if reading past file
Summary:
  This mirrors  https://reviews.facebook.net/D45645
  Currently, mmap returns IOError when user tries to read
  data past the end of the file. This diff changes the behavior.
  Now, we return just the bytes that we can, and report the size
  we returned via a Slice result. This is consistent with non-mmap
  behavior and also pread() system call.
2015-10-06 16:19:58 -07:00
Dmitri Smirnov 25c58a2040 Add shared_linked DEBUG flag, remove port from among the include directories. 2015-10-06 15:52:09 -07:00
krad 60b1c0a701 Fix to CI job definition
Summary: Sandcastle does not raise a task if assignee is specified. Removing the
assignee will force the system to use the oncall.

Test Plan: Manual

Reviewers:

CC: leveldb@

Task ID: #6968635

Blame Rev:
2015-10-06 14:21:10 -07:00
Islam AbdelRahman 9babaeed16 Update dump_tool and undump_tool to accept Options
Summary:
Refactor dump_tool and undump_tool so that it's possible to use them with customized options
for example setting a specific comparator similar to what Dragon is doing with the LdbTool

https://phabricator.fb.com/diffusion/FBCODE/browse/master/dragon/tools/Ldb.cpp

Test Plan:
compiles
used it to dump / undump a dragon shard

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, adsharma

Differential Revision: https://reviews.facebook.net/D47853
2015-10-05 19:49:48 -07:00
Igor Canadi eb5b637fb0 Fix condition for bottommost level
Summary:
The function GetBoundaryKeys() returns the smallest key from the first file and largest key from the last file. This is good for any level >0, but it's not correct for level 0. In level 0, files can overlap, so we need to check all files for boundary keys. This bug can cause wrong value for bottommost_level in compaction (value of true, although correct is false), which means we can set sequence numbers to 0 even if the key is not the oldest one in the database.

Herman reported corruption while testing MyRocks. Fortunately, the patch that added the bug was not released yet.

Test Plan: added a new test to compaction_picker_test.

Reviewers: hermanlee4, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48201
2015-10-05 17:40:18 -07:00
Igor Canadi 9eaff629e3 Make corruption_test more robust
Summary:
Latest travis failed because of corruption test TableFileIndexData: https://travis-ci.org/facebook/rocksdb/jobs/83732558

This diff makes the test more explicit:
1. create two files
2. corrupt the second's file index
3. expect to get only 5000 keys when range scanning

Test Plan: the test is still passing :)

Reviewers: sdong, rven, yhchiang, kradhakrishnan, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48183
2015-10-05 14:46:28 -07:00
Igor Canadi bf19dbff44 Fix valgrind - Initialize done variable
Summary: Fixes the valgrind warning "Conditional jump or move depends on uninitialised value(s)"

Test Plan: valgrind test, no more warning

Reviewers: sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D48177
2015-10-05 10:10:11 -07:00
Yueh-Hsuan Chiang 5c7bf56d35 [RocksDB Options] Support more options in RocksDBOptionParser for sanity check.
Summary:
RocksDBOptionsParser now supports CompressionType and the following
pointer-typed options in RocksDBOptionParser
for sanity check:
  prefix_extractor
  table_factory
  comparator
  compaction_filter
  compaction_filter_factory
  merge_operator
  memtable_factory

In the RocksDB Options file, only high level information about pointer-typed
options are serialized, and those information is only used for verification
/ sanity check purpose.

Test Plan: added more tests in options_test

Reviewers: igor, IslamAbdelRahman, sdong, anthony

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47925
2015-10-02 15:35:32 -07:00
Igor Canadi 115427ef63 Add APIs PauseBackgroundWork() and ContinueBackgroundWork()
Summary:
To support a new MongoDB capability, we need to make sure that we don't do any IO for a short period of time. For background, see:
* https://jira.mongodb.org/browse/SERVER-20704
* https://jira.mongodb.org/browse/SERVER-18899

To implement that, I add a new API calls PauseBackgroundWork() and ContinueBackgroundWork() which reuse the capability we already have in place for RefitLevel() function.

Test Plan: Added a new test in db_test. Made sure that test fails when PauseBackgroundWork() is commented out.

Reviewers: IslamAbdelRahman, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47901
2015-10-02 13:17:34 -07:00
krad a39897369a Adding features to CI job description
Summary: Added following capability
(1) Ability to do configurable reports using args
(2) Support to generating a task on failure
(3) Ability to configure email trigger condition

Test Plan: Manual testing

Reviewers:

CC:leveldb@

Task ID: #6968635

Blame Rev:
2015-10-02 12:36:58 -07:00
Igor Canadi a47bf325c6 Merge pull request #748 from yuslepukhin/improve_test_runs
Improve test running scripts
2015-10-02 11:42:53 -07:00
Dmitri Smirnov 65324a16dc Improve test running scripts
Introduce proper command line arguments so we can control the script
  Add appveyor support
  Add an ability to run all other (non db_tests) test executables in parallel
  Use .NET HashSet instead of empty valued hashtable to improve the looks
  TODO: Some of the tests do not use GTests and need to improve log parsing
2015-10-02 11:33:35 -07:00
Igor Canadi 7e4ee42315 Merge pull request #743 from edsrzf/amalgamation
New amalgamation target
2015-09-30 14:35:15 -07:00
Evan Shaw 7a23e4d8ca New amalgamation target
This commit adds two new targets to the Makefile: rocksdb.cc and rocksdb.h

These files, when combined with the c.h header, are a self-contained RocksDB
source distribution called an amalgamation. (The name comes from SQLite's, which
is similar in concept.)

The main benefit of an amalgamation is that it's very easy to drop into a
new project. It also compiles faster compared to compiling individual source
files and potentially gives the compiler more opportunity to make optimizations
since it can see all functions at once.

rocksdb.cc and rocksdb.h are generated by a new script, amalgamate.py.
A detailed description of how amalgamate.py works is in a comment at the top of
the file.

There are also some small changes to existing files to enable the amalgamation:
* Use quotes for includes in unity build
* Fix an old header inclusion in util/xfunc.cc
* Move some includes outside ifdef in util/env_hdfs.cc
* Separate out tool sources in Makefile so they won't be included in unity.cc
* Unity build now produces a static library

Closes #733
2015-10-01 08:29:31 +13:00
Igor Canadi e9a6808c4d Merge pull request #745 from yuslepukhin/test_appveyor_baseline
Improve CI build and fix Windows build breakage
2015-09-30 11:45:01 -07:00
Dmitri Smirnov 9320ffd67a Improve CI build and fix Windows build breakage
Is there a way to enforce CMake additions for internal changes that seem to come
  w/o a PR?
2015-09-30 11:20:23 -07:00
agiardullo 03b08ba9a9 Return MergeInProgress when fetching from transactions or WBWI with overwrite_key
Summary:
WriteBatchWithIndex::GetFromBatchAndDB only works correctly for overwrite_key=false.  Transactions use overwrite_key=true (since WriteBatchWithIndex::GetIteratorWithBase only works when overwrite_key=true).  So currently, Transactions could return incorrectly merged results when calling Get/GetForUpdate().

Until a permanent fix can be put in place, Transaction::Get[ForUpdate] and WriteBatchWithIndex::GetFromBatch[AndDB] will now return MergeInProgress if the most recent write to a key in the batch is a Merge.

Test Plan: more tests

Reviewers: sdong, yhchiang, rven, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47817
2015-09-30 11:14:42 -07:00
Islam AbdelRahman c29af48d3e Add max_file_opening_threads to db_bench
Summary: Add an option to db_bench for max_file_opening_threads

Test Plan: compile and run db_bench

Reviewers: sdong, yhchiang, igor

Reviewed By: igor

Subscribers: dhruba, paultuckfield

Differential Revision: https://reviews.facebook.net/D47811
2015-09-30 09:51:31 -07:00
Yueh-Hsuan Chiang da1cf8a9bc Add a missing check for deprecated options in options_helper.cc
Summary: Add a missing check for deprecated options in options_helper.cc

Test Plan: options_test

Reviewers: sdong, anthony, IslamAbdelRahman, igor

Reviewed By: igor

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47793
2015-09-29 17:58:00 -07:00
agiardullo 5a51fa907b Fix accidental object copy in transactions
Summary: Should have used auto& instead of auto.  Also needed to change the code a bit due to const correctness.

Test Plan: unit tests

Reviewers: sdong, igor, yoshinorim, yhchiang

Reviewed By: yhchiang

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47787
2015-09-29 17:36:56 -07:00
Yueh-Hsuan Chiang 1e73b11af7 Better handling of deprecated options in RocksDBOptionsParser
Summary:
Previously, we treat deprecated options as normal options in
RocksDBOptionsParser.  However, these deprecated options should
not be verified and serialized.

Test Plan: options_test

Reviewers: igor, sdong, IslamAbdelRahman, anthony

Reviewed By: anthony

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47775
2015-09-29 17:13:02 -07:00
Yueh-Hsuan Chiang a8b8295d18 Fixed a compile warning in options_test.cc under clang
Summary:
Fixed the following compile warning in options_test.cc under clang
util/options_test.cc:94:12: error: 'Skip' overrides a member function but is not marked 'override' [-Werror,-Winconsistent-missing-override]
    Status Skip(uint64_t n) {
           ^
./include/rocksdb/env.h:368:18: note: overridden virtual function is here
  virtual Status Skip(uint64_t n) = 0;
                 ^

Test Plan: options_test

Reviewers: igor, sdong, anthony, IslamAbdelRahman

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D47763
2015-09-29 14:59:02 -07:00