Skip to content

Add 2-line write prefetch option in TCMalloc per-CPU slab Pop. - #1942

Draft
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_1020697409
Draft

copybara-service[bot] wants to merge 1 commit into
masterfrom
test_1020697409

Conversation

@copybara-service

Copy link
Copy Markdown

Add 2-line write prefetch option in TCMalloc per-CPU slab Pop.

TcmallocSlab::Pop currently issues a single read-intent prefetch for the first byte of the next object on the per-CPU freelist (PrefetchNextObject). Freshly allocated memory is written before it is read—so a shared-state read prefetch still incurs a store-upgrade stall—and allocations spanning multiple cache lines or straddling a cache-line boundary miss on their second cache line.

Forward the requested allocation size through CpuCache::AllocateFast, AllocateSlow, and AllocateSlowNoHooks to TcmallocSlab::Pop and, when compiled with -DTCMALLOC_INTERNAL_WRITE_PREFETCH=1 (disabled by default for rollout), issue write-intent prefetches (PrefetchWT0) at prefetch_target and branchlessly (CMOV/CSEL) at prefetch_target + std::min<uint32_t>(size, ABSL_CACHELINE_SIZE + 1) - 1. Single-line objects prefetch the same line (or a straddled second line), larger objects prefetch the second cache line, and test-only CpuCache::Allocate passes the size-class size. Add the write_prefetch build and test variant to cover the enabled path.

`TcmallocSlab::Pop` currently issues a single read-intent prefetch for the first byte of the next object on the per-CPU freelist (`PrefetchNextObject`). Freshly allocated memory is written before it is read—so a shared-state read prefetch still incurs a store-upgrade stall—and allocations spanning multiple cache lines or straddling a cache-line boundary miss on their second cache line.

Forward the requested allocation size through `CpuCache::AllocateFast`, `AllocateSlow`, and `AllocateSlowNoHooks` to `TcmallocSlab::Pop` and, when compiled with `-DTCMALLOC_INTERNAL_WRITE_PREFETCH=1` (disabled by default for rollout), issue write-intent prefetches (`PrefetchWT0`) at `prefetch_target` and branchlessly (`CMOV`/`CSEL`) at `prefetch_target + std::min<uint32_t>(size, ABSL_CACHELINE_SIZE + 1) - 1`. Single-line objects prefetch the same line (or a straddled second line), larger objects prefetch the second cache line, and test-only `CpuCache::Allocate` passes the size-class size. Add the `write_prefetch` build and test variant to cover the enabled path.

PiperOrigin-RevId: 1020697409

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant