dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-26 07:30:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Ari Ekmekji 3c37b3cccd Determine boundaries of subcompactions 
Summary:
Up to this point, the subcompactions that make up a compaction
job have been divided based on the key range of the L1 files, and each
subcompaction has handled the key range of only one file. However
DBOption.max_subcompactions allows the user to designate how many
subcompactions at most to perform. This patch updates the
CompactionJob::GetSubcompactionBoundaries() to determine these
divisions accordingly based on that option and other input/system factors.

The current approach orders the starting and/or ending keys of certain
compaction input files and then generates a histogram to approximate the
size covered by the key range between each consecutive pair of keys. Then
it groups these ranges into groups so that the sizes are approximately equal
to one another. The approach has also been adapted to work for universal
compaction as well instead of just for level-based compaction as it was before.

These subcompactions are then executed in parallel by locally spawning
threads, one for each. The results are then aggregated and the compaction
completed.

Test Plan: make all && make check

Reviewers: yhchiang, anthony, igor, noetzli, sdong

Reviewed By: sdong

Subscribers: MarkCallaghan, dhruba, leveldb

Differential Revision: https://reviews.facebook.net/D43269
2015-09-10 13:50:00 -07:00
..
builder.cc Added Equal method to Comparator interface 2015-09-08 15:30:49 -07:00
builder.h Smarter purging during flush 2015-08-24 11:11:12 -07:00
c.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
c_test.c Deprecate CompactionFilterV2 2015-07-17 18:59:11 +02:00
column_family.cc better tuning of arena block size 2015-09-08 20:53:32 -07:00
column_family.h Total SST files size DB Property 2015-08-20 11:47:19 -07:00
column_family_test.cc better tuning of arena block size 2015-09-08 20:53:32 -07:00
compact_files_test.cc Improved FileExists API 2015-07-20 17:20:40 -07:00
compacted_db_impl.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compacted_db_impl.h Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
compaction.cc Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
compaction.h Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
compaction_job.cc Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
compaction_job.h Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
compaction_job_stats_test.cc Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
compaction_job_test.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
compaction_picker.cc Fix compact_files_example 2015-08-25 12:29:44 -07:00
compaction_picker.h Fix compact_files_example 2015-08-25 12:29:44 -07:00
compaction_picker_test.cc Add a counter about estimated pending compaction bytes 2015-08-20 22:17:10 -07:00
comparator_db_test.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
convenience.cc move convenience.h out of utilities 2015-07-15 14:51:51 -07:00
corruption_test.cc Don't let flushes preempt compactions 2015-07-17 12:02:52 -07:00
cuckoo_table_db_test.cc Block cuckoo table tests in ROCKSDB_LITE 2015-07-20 10:50:46 -07:00
db_bench.cc better db_bench options for transactions 2015-08-31 15:56:07 -07:00
db_compaction_filter_test.cc Add Subcompactions to Universal Compaction Unit Tests 2015-08-31 12:59:02 -07:00
db_compaction_test.cc better tuning of arena block size 2015-09-08 20:53:32 -07:00
db_dynamic_level_test.cc Move DynamicLevel related db-tests to db_dynamic_level_test.cc 2015-07-13 19:00:30 -07:00
db_filesnapshot.cc Add wal files to Checkpoint for multiple column families. 2015-06-19 16:08:31 -07:00
db_impl.cc Relaxing consistency detection to include errors while inserting to memtable as WAL recovery error. 2015-09-10 12:56:17 -07:00
db_impl.h Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
db_impl_debug.cc reduce db mutex contention for write batch groups 2015-08-14 10:55:43 -07:00
db_impl_experimental.cc Clean up InstallSuperVersion 2015-06-17 12:37:59 -07:00
db_impl_readonly.cc Remove db_impl_readonly dependency on utilities 2015-07-14 11:32:54 -07:00
db_impl_readonly.h Use CompactRangeOptions for CompactRange 2015-06-17 14:36:14 -07:00
db_inplace_update_test.cc Move in-place-update related tests from db_test.cc to db_inplace_update_test.cc 2015-07-20 16:05:28 -07:00
db_iter.cc Added Equal method to Comparator interface 2015-09-08 15:30:49 -07:00
db_iter.h
db_iter_test.cc DBIter to out extra keys with higher sequence numbers when changing direction from forward to backward 2015-08-26 13:01:39 -07:00
db_log_iter_test.cc Make TransactionLogIterator related tests from db_test.cc to db_log_iter_test.cc 2015-07-14 16:08:21 -07:00
db_tailing_iter_test.cc Fix case when forward iterator misses a new update 2015-09-04 14:28:45 -07:00
db_test.cc Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
db_universal_compaction_test.cc Arena usage to be calculated using malloc_usable_size() 2015-08-31 09:39:27 -07:00
db_wal_test.cc Add two unit tests for SyncWAL() 2015-08-05 14:27:02 -07:00
dbformat.cc Replace %llu with format macros in ParsedInternalKey::DebugString()) 2015-06-17 20:44:26 -07:00
dbformat.h Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
dbformat_test.cc Avoid manipulating const char* arrays 2015-07-14 00:21:41 -07:00
deletefile_test.cc Improved FileExists API 2015-07-20 17:20:40 -07:00
event_helpers.cc Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
event_helpers.h Add EventListener::OnTableFileDeletion() 2015-06-03 19:57:01 -07:00
experimental.cc Implement DB::PromoteL0 method 2015-04-23 12:10:36 -07:00
fault_injection_test.cc Address windows build issues 2015-08-20 14:08:24 -07:00
file_indexer.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
file_indexer.h Fix public API dependency on internal codes and dependency on MAX_INT32 2015-07-11 10:32:11 -07:00
file_indexer_test.cc Fix possible SIGSEGV in CompactRange (github issue #596) 2015-04-29 10:52:31 -07:00
filename.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
filename.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
filename_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
flush_job.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
flush_job.h Smarter purging during flush 2015-08-24 11:11:12 -07:00
flush_job_test.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
flush_scheduler.cc
flush_scheduler.h Fix data race #1 2015-01-26 11:48:07 -08:00
forward_iterator.cc Relaxed assert in forward iterator 2015-09-08 17:15:11 -07:00
forward_iterator.h Fix case when forward iterator misses a new update 2015-09-04 14:28:45 -07:00
internal_stats.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
internal_stats.h Expose per-level aggregated table properties via GetProperty() 2015-08-25 12:03:54 -07:00
job_context.h fixed leaking log::Writers 2015-07-07 12:10:10 -07:00
listener_test.cc Fix listener_test when using ROCKSDB_MALLOC_USABLE_SIZE 2015-08-31 23:11:12 -07:00
log_format.h
log_reader.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_reader.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_test.cc Removing duplicate code 2015-08-05 07:33:27 -07:00
log_writer.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
log_writer.h Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
managed_iterator.cc Windows Port from Microsoft 2015-07-01 16:13:56 -07:00
managed_iterator.h Fixed xfunc related compile errors in ROCKSDB_LITE 2015-04-09 21:05:18 -07:00
memtable.cc Added Equal method to Comparator interface 2015-09-08 15:30:49 -07:00
memtable.h Allow GetApproximateSize() to include mem table size if it is skip list memtable 2015-06-16 18:13:23 -07:00
memtable_allocator.cc Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_allocator.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00
memtable_list.cc Fixing race condition in DBTest.DynamicMemtableOptions 2015-08-24 17:04:18 -07:00
memtable_list.h Introduce GetIntProperty("rocksdb.size-all-mem-tables") 2015-08-19 13:32:09 -07:00
memtable_list_test.cc Removing duplicate code 2015-08-05 07:33:27 -07:00
memtablerep_bench.cc Added Equal method to Comparator interface 2015-09-08 15:30:49 -07:00
merge_context.h API to fetch from both a WriteBatchWithIndex and the db 2015-05-11 14:51:51 -07:00
merge_helper.cc Added Equal method to Comparator interface 2015-09-08 15:30:49 -07:00
merge_helper.h Simplify querying of merge results 2015-08-17 17:34:38 -07:00
merge_helper_test.cc Simplify querying of merge results 2015-08-17 17:34:38 -07:00
merge_operator.cc Call merge operators with empty values 2015-06-26 11:35:46 -07:00
merge_test.cc Make merge_test runnable in ROCKSDB_LITE 2015-07-20 11:17:52 -07:00
perf_context_test.cc Makefile minor cleanup 2015-03-30 16:05:35 -04:00
plain_table_db_test.cc Arena usage to be calculated using malloc_usable_size() 2015-08-31 09:39:27 -07:00
prefix_test.cc Skip unsupported tests in ROCKSDB_LITE 2015-07-20 11:24:54 -07:00
repair.cc Smarter purging during flush 2015-08-24 11:11:12 -07:00
skiplist.h reduce comparisons by skiplist 2015-08-11 11:25:22 -07:00
skiplist_test.cc rocksdb: switch to gtest 2015-03-17 14:08:00 -07:00
slice.cc Create an abstract interface for write batches 2015-03-17 19:23:08 -07:00
snapshot_impl.cc simple ManagedSnapshot wrapper 2015-08-06 17:59:05 -07:00
snapshot_impl.h simple ManagedSnapshot wrapper 2015-08-06 17:59:05 -07:00
table_cache.cc Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported 2015-09-02 12:57:07 -07:00
table_cache.h ReadaheadRandomAccessFile -- userspace readahead 2015-08-26 15:25:59 -07:00
table_properties_collector.cc A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge 2015-04-06 10:27:21 -07:00
table_properties_collector.h Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
table_properties_collector_test.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
transaction_log_impl.cc Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
transaction_log_impl.h Move rate_limiter, write buffering, most perf context instrumentation and most random kill out of Env 2015-07-17 16:58:18 -07:00
version_builder.cc Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported 2015-09-02 12:57:07 -07:00
version_builder.h Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported 2015-09-02 12:57:07 -07:00
version_builder_test.cc Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_edit.cc Added JSON manifest dump option to ldb command 2015-07-17 10:07:40 -07:00
version_edit.h Added JSON manifest dump option to ldb command 2015-07-17 10:07:40 -07:00
version_edit_test.cc Add TablePropertiesCollector::NeedCompact() to suggest DB to further compact output files 2015-06-05 20:18:21 -07:00
version_set.cc Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
version_set.h Determine boundaries of subcompactions 2015-09-10 13:50:00 -07:00
version_set_test.cc Report live data size estimate 2015-07-21 21:33:20 -07:00
wal_manager.cc Improved FileExists API 2015-07-20 17:20:40 -07:00
wal_manager.h Fix -Wnon-virtual-dtor errors 2014-11-10 17:39:38 -05:00
wal_manager_test.cc Skip unsupported tests in ROCKSDB_LITE 2015-07-20 11:24:54 -07:00
write_batch.cc Unified maps with Comparator for sorting, other cleanup 2015-09-02 13:58:22 -07:00
write_batch_base.cc WriteBatch.Merge w/ SliceParts support 2015-05-29 04:30:03 -07:00
write_batch_internal.h WriteBatch Save Points 2015-07-29 16:54:23 -07:00
write_batch_test.cc WriteBatch Save Points 2015-07-29 16:54:23 -07:00
write_callback.h Optimistic Transactions 2015-05-29 14:36:35 -07:00
write_callback_test.cc Fix compile for write_callback_test in ROCKSDB_LITE 2015-07-20 10:54:15 -07:00
write_controller.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_controller.h Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_controller_test.cc Slow down writes by bytes written 2015-06-11 20:42:18 -07:00
write_thread.cc reduce db mutex contention for write batch groups 2015-08-14 10:55:43 -07:00
write_thread.h reduce db mutex contention for write batch groups 2015-08-14 10:55:43 -07:00
writebuffer.h Enforce write buffer memory limit across column families 2014-12-02 12:09:20 -08:00