dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-26 16:30:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db
History
Levi Tamasi f34782a67d Fix the "records dropped" statistics (#6325) 
Summary:
The earlier code used two conflicting definitions for the number of
input records going into a compaction, one based on the
`rocksdb.num.entries` table property and one based on
`CompactionIterationStats`. The first one is correct and in line
with how output records are counted, while the second one incorrectly
ignores input records in various cases when the `CompactionIterator`
advances or reseeks the input iterator (this can happen, amongst other
cases, when dealing with `SingleDelete`s, regular `Delete`s, `Merge`s,
and compaction filters). This can result in the code undercounting the
input records and computing an incorrect value for "records dropped"
during the compaction. The patch fixes this by switching over to the
correct (table property based) input record count for "records dropped".
Pull Request resolved: https://github.com/facebook/rocksdb/pull/6325

Test Plan: Tested using `make check` and `db_bench`.

Differential Revision: D19525491

Pulled By: ltamasi

fbshipit-source-id: 4340b0b2f41546db8e356db70ca02199e48fa636
2020-01-23 15:27:22 -08:00
..
compaction Fix the "records dropped" statistics (#6325) 2020-01-23 15:27:22 -08:00
db_impl Adjust thread pool sizes when setting max_background_jobs dynamically (#6300) 2020-01-16 14:35:10 -08:00
arena_wrapped_db_iter.cc delete superversions in BackgroundCallPurge (#6146) 2019-12-17 13:22:57 -08:00
arena_wrapped_db_iter.h Refactor ArenaWrappedDBIter into separate files (#5801) 2019-09-13 13:50:43 -07:00
blob_index.h Mark BlobIndex::DebugString const 2019-12-11 17:19:43 -08:00
builder.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
builder.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
c.cc Expose atomic flush option in C API (#6307) 2020-01-17 12:57:48 -08:00
c_test.c Fix error message (#6264) 2020-01-07 12:32:20 -08:00
column_family.cc unordered_write incompatible with max_successive_merges (#6284) 2020-01-10 16:53:19 -08:00
column_family.h delete superversions in BackgroundCallPurge (#6146) 2019-12-17 13:22:57 -08:00
column_family_test.cc Add unit tests for concurrent CF iteration and drop (#6180) 2019-12-18 11:54:35 -08:00
compact_files_test.cc Use aggregate initialization for FlushJobInfo/CompactionJobInfo (#5997) 2019-11-01 11:46:19 -07:00
compacted_db_impl.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
compacted_db_impl.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
comparator_db_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
convenience.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
corruption_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
cuckoo_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
db_basic_test.cc Fix compilation under LITE (#6277) 2020-01-09 15:57:39 -08:00
db_blob_index_test.cc Disable blob iterator test with max_sequential_skip_in_iterations==0 in LITE mode (#6052) 2019-11-19 15:02:41 -08:00
db_block_cache_test.cc Small tidy and speed up of the travis build (#6181) 2019-12-17 13:56:45 -08:00
db_bloom_filter_test.cc Expose and elaborate FilterBuildingContext (#6088) 2019-11-26 18:24:10 -08:00
db_compaction_filter_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_compaction_test.cc Make DBCompactionTest.SkipStatsUpdateTest more robust (#6306) 2020-01-21 12:55:55 -08:00
db_dynamic_level_test.cc Fix flaky DBDynamicLevelTest.DynamicLevelMaxBytesBase2 (#4668) 2018-11-12 16:42:16 -08:00
db_encryption_test.cc Fix EncryptedEnv assert (#5735) 2019-09-05 17:21:42 -07:00
db_filesnapshot.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
db_flush_test.cc Fix a potential bug scheduling unnecessary threads (#6104) 2019-11-27 14:48:49 -08:00
db_info_dumper.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
db_info_dumper.h Change RocksDB License 2017-07-15 16:11:23 -07:00
db_inplace_update_test.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
db_io_failure_test.cc Disable DBIOFailureTest.NoSpaceCompactRange in LITE (#4596) 2018-10-29 14:36:31 -07:00
db_iter.cc Fix blob context when db_iter uses seek (#6051) 2019-11-19 11:39:02 -08:00
db_iter.h Apply formatter on recent 45 commits. (#5827) 2019-09-19 12:34:17 -07:00
db_iter_stress_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iter_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
db_iterator_test.cc Add unit tests for concurrent CF iteration and drop (#6180) 2019-12-18 11:54:35 -08:00
db_log_iter_test.cc Apply modernize-use-override (2nd iteration) 2019-02-14 14:41:36 -08:00
db_memtable_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_merge_operand_test.cc New API to get all merge operands for a Key (#5604) 2019-08-06 14:26:44 -07:00
db_merge_operator_test.cc unordered_write incompatible with max_successive_merges (#6284) 2020-01-10 16:53:19 -08:00
db_options_test.cc Adjust thread pool sizes when setting max_background_jobs dynamically (#6300) 2020-01-16 14:35:10 -08:00
db_properties_test.cc Refactor/consolidate legacy Bloom implementation details (#5784) 2019-09-16 16:17:09 -07:00
db_range_del_test.cc Fix RangeDeletion bug (#6062) 2019-12-12 15:18:02 -08:00
db_sst_test.cc Fix bugs in WAL trash file handling (#5520) 2019-07-06 21:07:32 -07:00
db_statistics_test.cc Make statistics's stats_level change thread-safe (#5030) 2019-03-01 10:42:09 -08:00
db_table_properties_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
db_tailing_iter_test.cc Remove managed iterator 2018-07-17 14:43:18 -07:00
db_test.cc Add oldest snapshot sequence property (#6228) 2020-01-07 08:36:44 -08:00
db_test2.cc Fix another bug caused by recent hash index fix (#6305) 2020-01-17 01:41:04 -08:00
db_test_util.cc Add oldest snapshot sequence property (#6228) 2020-01-07 08:36:44 -08:00
db_test_util.h Add oldest snapshot sequence property (#6228) 2020-01-07 08:36:44 -08:00
db_universal_compaction_test.cc Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125) 2019-12-09 09:23:50 -08:00
db_wal_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
db_write_test.cc Fix queue manipulation in WriteThread::BeginWriteStall() (#6322) 2020-01-23 14:01:28 -08:00
dbformat.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
dbformat.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
dbformat_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
deletefile_test.cc Refactor deletefile_test.cc (#5822) 2019-09-18 16:58:21 -07:00
error_handler.cc Make format 2019-05-31 15:24:43 -07:00
error_handler.h Fix typos in comments (#4456) 2018-10-04 20:46:50 -07:00
error_handler_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
event_helpers.cc BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
event_helpers.h BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) 2019-10-14 15:21:01 -07:00
experimental.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
external_sst_file_basic_test.cc Allow ingesting overlapping files (#5539) 2019-09-13 14:49:47 -07:00
external_sst_file_ingestion_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
external_sst_file_ingestion_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
external_sst_file_test.cc wait pending memtable writes on file ingestion or compact range (#6113) 2019-12-12 14:08:02 -08:00
fault_injection_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
file_indexer.cc Bug when multiple files at one level contains the same smallest key (#6285) 2020-01-13 16:27:42 -08:00
file_indexer.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
file_indexer_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
filename_test.cc Move some logging related files to logging/ (#5387) 2019-05-31 17:23:59 -07:00
flush_job.cc Increase max_log_size in FlushJob to 1024 bytes (#6258) 2020-01-06 10:16:52 -08:00
flush_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_job_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
flush_scheduler.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
flush_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
forward_iterator.cc delete superversions in BackgroundCallPurge (#6146) 2019-12-17 13:22:57 -08:00
forward_iterator.h Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
forward_iterator_bench.cc simplify include directive involving inttypes (#5402) 2019-06-06 13:56:07 -07:00
import_column_family_job.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
import_column_family_job.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
import_column_family_test.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
internal_stats.cc Remove inaccurate code comment (#6274) 2020-01-08 17:51:42 -08:00
internal_stats.h Add oldest snapshot sequence property (#6228) 2020-01-07 08:36:44 -08:00
job_context.h WritePrepared: Fix visible key compacted out by compaction (#4883) 2019-01-15 21:34:38 -08:00
listener_test.cc Propagate SST and blob file numbers through the EventListener interface (#5962) 2019-10-24 14:44:15 -07:00
log_format.h Fix an inaccurate comment (#4315) 2018-08-24 18:13:20 -07:00
log_reader.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_reader.h Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
log_writer.cc Divide file_reader_writer.h and .cc (#5803) 2019-09-16 10:33:51 -07:00
log_writer.h Use delete to disable automatic generated methods. (#5009) 2019-09-11 18:09:00 -07:00
logs_with_prep_tracker.cc Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
logs_with_prep_tracker.h Skip deleted WALs during recovery 2018-05-03 15:43:09 -07:00
lookup_key.h Avoid user key copying for Get/Put/Write with user-timestamp (#5502) 2019-07-25 15:27:39 -07:00
malloc_stats.cc Support jemalloc compiled with --with-jemalloc-prefix (#5521) 2019-07-02 12:07:01 -07:00
malloc_stats.h Change RocksDB License 2017-07-15 16:11:23 -07:00
manual_compaction_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
memtable.cc Misc hashing updates / upgrades (#5909) 2019-10-24 17:16:46 -07:00
memtable.h Do not schedule memtable trimming if there is no history (#6177) 2019-12-13 19:11:19 -08:00
memtable_list.cc Fix a data race related to memtable trimming (#6187) 2019-12-16 13:16:31 -08:00
memtable_list.h Fix a data race related to memtable trimming (#6187) 2019-12-16 13:16:31 -08:00
memtable_list_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
merge_context.h Introduce a new MultiGet batching implementation (#5011) 2019-04-11 14:28:26 -07:00
merge_helper.cc Fix merging range tombstone covering put during flush/compaction (#5406) 2019-06-04 10:24:14 -07:00
merge_helper.h Remove v1 RangeDelAggregator (#4778) 2018-12-17 17:33:46 -08:00
merge_helper_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
merge_operator.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
merge_test.cc unordered_write incompatible with max_successive_merges (#6284) 2020-01-10 16:53:19 -08:00
obsolete_files_test.cc Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820) 2019-09-18 11:52:17 -07:00
options_file_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
perf_context_test.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
pinned_iterators_manager.h Change RocksDB License 2017-07-15 16:11:23 -07:00
plain_table_db_test.cc Fix memory leak on error opening PlainTable (#5951) 2019-10-21 16:53:06 -07:00
pre_release_callback.h WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
prefix_test.cc Organizing rocksdb/db directory 2019-05-31 11:57:01 -07:00
range_del_aggregator.cc Correct pragma once problem with Bazel on Windows (#6321) 2020-01-21 16:12:43 -08:00
range_del_aggregator.h Correct pragma once problem with Bazel on Windows (#6321) 2020-01-21 16:12:43 -08:00
range_del_aggregator_bench.cc Move some memory related files from util/ to memory/ (#5382) 2019-05-30 17:44:09 -07:00
range_del_aggregator_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
range_tombstone_fragmenter.cc Apply formatter to recent 200+ commits. (#5830) 2019-09-20 12:04:26 -07:00
range_tombstone_fragmenter.h Initialized pinned_pos_ and pinned_seq_pos_ in FragmentedRangeTombstoneIterator (#5720) 2019-09-05 17:30:29 -07:00
range_tombstone_fragmenter_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
read_callback.h WriteUnPrepared: improve read your own write functionality (#5573) 2019-07-23 08:08:19 -07:00
repair.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
repair_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
snapshot_checker.h WritePrepared: fix issue with snapshot released during compaction (#4858) 2019-01-16 09:55:32 -08:00
snapshot_impl.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
snapshot_impl.h Add oldest snapshot sequence property (#6228) 2020-01-07 08:36:44 -08:00
table_cache.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_cache.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
table_properties_collector.cc Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector.h Feature for sampling and reporting compressibility (#4842) 2019-03-18 12:15:34 -07:00
table_properties_collector_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
transaction_log_impl.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
transaction_log_impl.h reuse scratch buffer in transaction_log_reader (#5702) 2019-08-26 11:26:29 -07:00
trim_history_scheduler.cc Support concurrent CF iteration and drop (#6147) 2019-12-12 19:04:48 -08:00
trim_history_scheduler.h Refactor trimming logic for immutable memtables (#5022) 2019-08-23 13:55:34 -07:00
version_builder.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_builder.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
version_builder_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit.h Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_edit_test.cc Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) 2019-11-26 21:39:56 -08:00
version_set.cc Make DBCompactionTest.SkipStatsUpdateTest more robust (#6306) 2020-01-21 12:55:55 -08:00
version_set.h Fix a data race for cfd->log_number_ (#6249) 2020-01-06 20:09:51 -08:00
version_set_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager.h Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
wal_manager_test.cc Introduce a new storage specific Env API (#5761) 2019-12-13 14:48:41 -08:00
write_batch.cc Fix a data race related to memtable trimming (#6187) 2019-12-16 13:16:31 -08:00
write_batch_base.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_batch_internal.h Add insert hints for each writebatch (#5728) 2019-09-12 17:15:18 -07:00
write_batch_test.cc upgrade gtest 1.7.0 => 1.8.1 for json result writing 2019-09-09 11:24:11 -07:00
write_callback.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_callback_test.cc WritePrepared: reduce prepared_mutex_ overhead (#5420) 2019-06-10 11:53:31 -07:00
write_controller.cc Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller.h Change RocksDB License 2017-07-15 16:11:23 -07:00
write_controller_test.cc Move test related files under util/ to test_util/ (#5377) 2019-05-30 11:25:51 -07:00
write_thread.cc Fix queue manipulation in WriteThread::BeginWriteStall() (#6322) 2020-01-23 14:01:28 -08:00
write_thread.h Option to make write group size configurable (#5759) 2019-09-11 18:28:33 -07:00