dolysis
/
rocksdb
mirror of https://github.com/facebook/rocksdb.git
3
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
rocksdb/memory
History
Andrew Kryczka 925d8252e5 Shard JemallocNodumpAllocator (#11400) 
Summary:
RocksDB's jemalloc no-dump allocator (`NewJemallocNodumpAllocator()`) was using a single manual arena. This arena's lock contention could be very high when thread caching is disabled for RocksDB blocks (e.g., when using `MALLOC_CONF='tcache_max:4096'` and `rocksdb_block_size=16384`).

This PR changes the jemalloc no-dump allocator to use a configurable number of manual arenas. That number is required to be a power of two so we can avoid division. The allocator shards allocation requests randomly across those manual arenas.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/11400

Test Plan:
- mysqld setup
  - Branch: fb-mysql-8.0.28 (653eba2e56)
  - Build: `mysqlbuild.sh --clean --release`
  - Set env var `MALLOC_CONF='tcache_max:$tcache_max'`
  - Added CLI args `--rocksdb_cache_dump=false --rocksdb_block_cache_size=4294967296 --rocksdb_block_size=16384`
  - Ran under /usr/bin/time
- Large database scenario
  - Setup command: `mysqlslap -h 127.0.0.1 -P 13020 --auto-generate-sql=1 --auto-generate-sql-load-type=write --auto-generate-sql-guid-primary=1 --number-char-cols=8 --auto-generate-sql-execute-number=262144 --concurrency=32 --no-drop`
  - Benchmark command: `mysqlslap -h 127.0.0.1 -P 13020 --query='select count(*) from mysqlslap.t1;' --number-of-queries=320 --concurrency=32`
  - Results:

| tcache_max | num_arenas | Peak RSS MB (% change) | Query latency seconds (% change) |
|---|---|---|---|
| 4096 | **(baseline)** | 4541 | 37.1 |
| 4096 | 1 | 4535 (-0.1%) | 36.7 (-1%) |
| 4096 | 8 | 4687 (+3%) | 10.2 (-73%) |
| 16384 | **(baseline)** | 4514 | 8.4 |
| 16384 | 1 | 4526 (+0.3%) | 8.5 (+1%) |
| 16384 | 8 | 4580 (+1%) | 8.5 (+1%) |

Reviewed By: pdillinger

Differential Revision: D45220794

Pulled By: ajkr

fbshipit-source-id: 9a50c9872bdef5d299e52b115a65ee8a5557d58d
 		2023-05-01 10:14:43 -07:00
..
allocator.h Run clang format against files under example/, memory/ and memtable/ folders (#10893) 2022-10-28 13:16:50 -07:00
arena.cc Fix use of make_unique in Arena::AllocateNewBlock (#11012) 2022-12-01 13:18:40 -08:00
arena.h Improve / refactor anonymous mmap capabilities (#10810) 2022-10-17 17:10:16 -07:00
arena_test.cc Fix use of make_unique in Arena::AllocateNewBlock (#11012) 2022-12-01 13:18:40 -08:00
concurrent_arena.cc Run clang format against files under example/, memory/ and memtable/ folders (#10893) 2022-10-28 13:16:50 -07:00
concurrent_arena.h Run clang format against files under example/, memory/ and memtable/ folders (#10893) 2022-10-28 13:16:50 -07:00
jemalloc_nodump_allocator.cc Shard JemallocNodumpAllocator (#11400) 2023-05-01 10:14:43 -07:00
jemalloc_nodump_allocator.h Shard JemallocNodumpAllocator (#11400) 2023-05-01 10:14:43 -07:00
memkind_kmem_allocator.cc Make MemoryAllocator into a Customizable class (#8980) 2021-12-17 04:20:47 -08:00
memkind_kmem_allocator.h Fix compilation error when building static_lib (#9377) 2022-01-12 09:04:01 -08:00
memory_allocator.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
memory_allocator.h Major Cache refactoring, CPU efficiency improvement (#10975) 2023-01-11 14:20:40 -08:00
memory_allocator_test.cc Remove RocksDB LITE (#11147) 2023-01-27 13:14:19 -08:00
memory_usage.h Meta-internal folly integration with F14FastMap (#9546) 2022-04-13 07:34:01 -07:00