dolysis/rocksdb
3
0
Fork 0
mirror of https://github.com/facebook/rocksdb.git synced 2024-11-27 20:43:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
rocksdb/db/compaction
History
Changyu Bi 333abe9c55 Ignore max_compaction_bytes for compaction input that are within output key-range (#10835) 
Summary:
When picking compaction input files, we sometimes stop picking a file that is fully included in the output key-range due to hitting max_compaction_bytes. Including these input files can potentially reduce WA at the expense of larger compactions. Larger compaction should be fine as files from input level are usually 10X smaller than files from output level. This PR adds a mutable CF option `ignore_max_compaction_bytes_for_input` that is enabled by default. We can remove this option once we are sure it is safe.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10835

Test Plan:
- CI, a unit test on max_compaction_bytes fails before turning this flag off.
- Benchmark does not show much difference in WA: `./db_bench --benchmarks=fillrandom,waitforcompaction,stats,levelstats -max_background_jobs=12 -num=2000000000 -target_file_size_base=33554432 --write_buffer_size=33554432`
```
main:
** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      3/0   91.59 MB   0.8     70.9     0.0     70.9     200.8    129.9       0.0   1.5     25.2     71.2   2886.55           2463.45      9725    0.297   1093M   254K       0.0       0.0
  L1      9/0   248.03 MB   1.0    392.0   129.8    262.2     391.7    129.5       0.0   3.0     69.0     68.9   5821.71           5536.90       804    7.241   6029M  5814K       0.0       0.0
  L2     87/0    2.50 GB   1.0    537.0   128.5    408.5     533.8    125.2       0.7   4.2     69.5     69.1   7912.24           7323.70      4417    1.791   8299M    36M       0.0       0.0
  L3    836/0   24.99 GB   1.0    616.9   118.3    498.7     594.5     95.8       5.2   5.0     66.9     64.5   9442.38           8490.28      4204    2.246   9749M   306M       0.0       0.0
  L4   2355/0   62.95 GB   0.3     67.3    37.1     30.2      54.2     24.0      38.9   1.5     72.2     58.2    954.37            821.18       917    1.041   1076M   173M       0.0       0.0
 Sum   3290/0   90.77 GB   0.0   1684.2   413.7   1270.5    1775.0    504.5      44.9  13.7     63.8     67.3  27017.25          24635.52     20067    1.346     26G   522M       0.0       0.0

Cumulative compaction: 1774.96 GB write, 154.29 MB/s write, 1684.19 GB read, 146.40 MB/s read, 27017.3 seconds

This PR:
** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      3/0   45.71 MB   0.8     72.9     0.0     72.9     202.8    129.9       0.0   1.6     25.4     70.7   2938.16           2510.36      9741    0.302   1124M   265K       0.0       0.0
  L1      8/0   234.54 MB   0.9    384.5   129.8    254.7     384.2    129.6       0.0   3.0     69.0     68.9   5708.08           5424.43       791    7.216   5913M  5753K       0.0       0.0
  L2     84/0    2.47 GB   1.0    543.1   128.6    414.5     539.9    125.4       0.7   4.2     69.6     69.2   7989.31           7403.13      4418    1.808   8393M    36M       0.0       0.0
  L3    839/0   24.96 GB   1.0    615.6   118.4    497.2     593.2     96.0       5.1   5.0     66.6     64.1   9471.23           8489.31      4193    2.259   9726M   306M       0.0       0.0
  L4   2360/0   63.04 GB   0.3     67.6    37.3     30.3      54.4     24.1      38.9   1.5     71.5     57.6    967.30            827.99       907    1.066   1080M   173M       0.0       0.0
 Sum   3294/0   90.75 GB   0.0   1683.8   414.2   1269.6    1774.5    504.9      44.8  13.7     63.7     67.1  27074.08          24655.22     20050    1.350     26G   522M       0.0       0.0

Cumulative compaction: 1774.52 GB write, 157.09 MB/s write, 1683.77 GB read, 149.06 MB/s read, 27074.1 seconds
```

Reviewed By: ajkr

Differential Revision: D40518319

Pulled By: cbi42

fbshipit-source-id: f4ea614bc0ebefe007ffaf05bb9aec9a8ca25b60
2022-10-21 10:22:41 -07:00
..
clipping_iterator.h Make InternalKeyComparator not configurable (#10342) 2022-07-14 10:09:31 -07:00
clipping_iterator_test.cc Print stack traces on frozen tests in CI (#10828) 2022-10-18 00:35:35 -07:00
compaction.cc Allow the last level data moving up to penultimate level (#10782) 2022-10-10 22:50:34 -07:00
compaction.h Exclude timestamp when checking compaction boundaries (#10787) 2022-10-07 14:11:23 -07:00
compaction_iteration_stats.h Support readahead during compaction for blob files (#9187) 2021-11-19 17:53:47 -08:00
compaction_iterator.cc Fix no internal time recorded for small preclude_last_level (#10829) 2022-10-20 17:11:38 -07:00
compaction_iterator.h Add option preserve_internal_time_seconds to preserve the time info (#10747) 2022-10-07 18:49:40 -07:00
compaction_iterator_test.cc Print stack traces on frozen tests in CI (#10828) 2022-10-18 00:35:35 -07:00
compaction_job.cc Add option preserve_internal_time_seconds to preserve the time info (#10747) 2022-10-07 18:49:40 -07:00
compaction_job.h Add option preserve_internal_time_seconds to preserve the time info (#10747) 2022-10-07 18:49:40 -07:00
compaction_job_stats_test.cc Remove own ToString() (#9955) 2022-05-06 13:03:58 -07:00
compaction_job_test.cc Print stack traces on frozen tests in CI (#10828) 2022-10-18 00:35:35 -07:00
compaction_outputs.cc Use sstableKeyCompare() for compaction output boundary check (#10763) 2022-10-06 15:54:58 -07:00
compaction_outputs.h Use sstableKeyCompare() for compaction output boundary check (#10763) 2022-10-06 15:54:58 -07:00
compaction_picker.cc Ignore max_compaction_bytes for compaction input that are within output key-range (#10835) 2022-10-21 10:22:41 -07:00
compaction_picker.h Support subcmpct using reserved resources for round-robin priority (#10341) 2022-07-24 11:12:44 -07:00
compaction_picker_fifo.cc Enable a multi-level db to smoothly migrate to FIFO via DB::Open (#10348) 2022-10-18 14:38:13 -07:00
compaction_picker_fifo.h Add OpenAndTrimHistory API to support trimming data with specified timestamp (#9410) 2022-03-11 16:13:23 -08:00
compaction_picker_level.cc RoundRobin TTL compaction (#10725) 2022-10-04 14:53:32 -07:00
compaction_picker_level.h Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
compaction_picker_test.cc Ignore max_compaction_bytes for compaction input that are within output key-range (#10835) 2022-10-21 10:22:41 -07:00
compaction_picker_universal.cc Fix overlapping check by excluding timestamp (#10615) 2022-09-08 13:03:07 -07:00
compaction_picker_universal.h Make ImmutableOptions struct that inherits from ImmutableCFOptions and ImmutableDBOptions (#8262) 2021-05-05 14:00:17 -07:00
compaction_service_job.cc Improve SubCompaction Partitioning (#10393) 2022-07-23 17:38:49 -07:00
compaction_service_test.cc Deflake CompactionServiceTest.BasicCompactions (#10697) 2022-09-19 14:10:05 -07:00
compaction_state.cc Tiered Compaction: per key placement support (#9964) 2022-07-13 20:54:49 -07:00
compaction_state.h Tiered Compaction: per key placement support (#9964) 2022-07-13 20:54:49 -07:00
file_pri.h Try to start TTL earlier with kMinOverlappingRatio is used (#8749) 2021-11-01 14:36:31 -07:00
sst_partitioner.cc Restore Regex support for ObjectLibrary::Register, rename new APIs to allow old one to be deprecated in the future (#9362) 2022-01-11 06:33:48 -08:00
subcompaction_state.cc Refactor Compaction file cut ShouldStopBefore() (#10629) 2022-09-14 22:09:12 -07:00
subcompaction_state.h Refactor Compaction file cut ShouldStopBefore() (#10629) 2022-09-14 22:09:12 -07:00
tiered_compaction_test.cc Fix no internal time recorded for small preclude_last_level (#10829) 2022-10-20 17:11:38 -07:00