dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,202 Commits 196 Branches 277 Tags 388 MiB
C++ 84.3%
Java 9.6%
C 1.6%
Python 1.5%
Perl 0.9%
Other 2%
Go to file
Feng Zhu 0af157f9bf Implement full filter for block based table. 
Summary:
1. Make filter_block.h a base class. Derive block_based_filter_block and full_filter_block. The previous one is the traditional filter block. The full_filter_block is newly added. It would generate a filter block that contain all the keys in SST file.

2. When querying a key, table would first check if full_filter is available. If not, it would go to the exact data block and check using block_based filter.

3. User could choose to use full_filter or tradional(block_based_filter). They would be stored in SST file with different meta index name. "filter.filter_policy" or "full_filter.filter_policy". Then, Table reader is able to know the fllter block type.

4. Some optimizations have been done for full_filter_block, thus it requires a different interface compared to the original one in filter_policy.h.

5. Actual implementation of filter bits coding/decoding is placed in util/bloom_impl.cc

Benchmark: base commit 1d23b5c470
Command:
db_bench --db=/dev/shm/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --write_buffer_size=134217728 --max_write_buffer_number=2 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --verify_checksum=false --max_background_compactions=4 --use_plain_table=0 --memtablerep=prefix_hash --open_files=-1 --mmap_read=1 --mmap_write=0 --bloom_bits=10 --bloom_locality=1 --memtable_bloom_bits=500000 --compression_type=lz4 --num=393216000 --use_hash_search=1 --block_size=1024 --block_restart_interval=16 --use_existing_db=1 --threads=1 --benchmarks=readrandom —disable_auto_compactions=1
Read QPS increase for about 30% from 2230002 to 2991411.

Test Plan:
make all check
valgrind db_test
db_stress --use_block_based_filter = 0
./auto_sanity_test.sh

Reviewers: igor, yhchiang, ljin, sdong

Reviewed By: sdong

Subscribers: dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D20979
 		2014-09-08 10:37:05 -07:00
build_tools Add db_bench with lots of column families to regression tests 2014-09-05 14:20:18 -07:00
coverage Disable the html-based coverage report by default 2014-02-06 12:58:13 -08:00
db Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
doc Remove seek compaction 2014-06-20 10:23:02 +02:00
examples Make it easier to start using RocksDB 2014-05-10 10:49:33 -07:00
hdfs hdfs cleanup and compile test against CDH 4.4. 2014-05-20 17:22:12 -04:00
helpers/memenv Expose in memory Env to the world 2014-04-14 12:28:15 -07:00
include Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
java Remove path with arena==nullptr from NewInternalIterator 2014-09-04 17:40:41 -07:00
linters allow lambda function syntax in cpplint 2014-02-20 12:47:05 -08:00
port Avoid off-by-one error when using readlink 2014-09-05 20:50:29 -07:00
table Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
third-party/rapidjson Fix a rapidjson compile error in mac. 2014-06-23 17:09:24 -06:00
tools Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
util Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
utilities Add missing break statement 2014-09-05 20:50:29 -07:00
.arcconfig Improve/fix bugs for the cpp linter 2014-02-13 17:48:11 -08:00
.clang-format A script that automatically reformat affected lines 2014-01-14 12:21:24 -08:00
.gitignore Changes to support unity build: 2014-08-11 13:22:47 -04:00
.travis.yml Fix travis builds 2014-09-04 10:23:45 -07:00
CONTRIBUTING.md facebook accounts are not required for CLA signers 2014-07-08 05:57:54 -04:00
HISTORY.md Add db_bench with lots of column families to regression tests 2014-09-05 14:20:18 -07:00
INSTALL.md specify the command to install build_tools/mac-install-gflags.sh file in doc 2014-06-17 17:03:21 -05:00
LICENSE Fix copyright year 2014-03-12 12:06:58 -07:00
Makefile Implement full filter for block based table. 2014-09-08 10:37:05 -07:00
PATENTS Fix the patent format 2013-10-16 15:37:32 -07:00
README.md Update README.md 2014-06-23 15:58:54 -07:00
ROCKSDB_LITE.md RocksDBLite 2014-04-15 13:39:26 -07:00

README.md

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

Build Status

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on on earlier work on LevelDB by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com)

This code is a library that forms the core building block for a fast key value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it specially suitable for storing multiple terabytes of data in a single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/master/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning.

Design discussions are conducted in https://www.facebook.com/groups/rocksdb.dev/