perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output

[dktue]

perf: 3.4x faster on the full planet, 3-4.5x on extracts, much less RAM, deterministic output

Full planet: 2 h 10 min -> 38 min 38 s (3.4x faster), 4x less CPU time, 96 GB less peak RAM. On extracts: 3-4.5x faster and 11-20x less peak memory. The tiles are the same apart from a few 1-2 unit vertex shifts (see Output).

Full planet

Planet (planet-261005, 89 GB PBF)	Before (parent commit)	After	Improvement
Wall time	2 h 10 min 08 s	38 min 38 s	3.4x faster
CPU time (user + system)	190 h	47.6 h	4.0x less
Peak RSS	270 GB	174 GB	-96 GB (-36%)
Output size	75.470 GB	75.454 GB	-0.02%

Setup: OpenMapTiles profile (resources/config-openmaptiles.json + process-openmaptiles.lua) with coastline and landcover shapefiles, PMTiles output, in memory (no --store), default thread count. Server: AMD EPYC 9454P (48 cores / 96 threads), 503 GB RAM, Ubuntu 22.04, GCC 11.4. Measured with /usr/bin/time -v, one run per version, back to back on an otherwise idle machine. Both runs wrote all 345,907,442 tiles and exited cleanly.

Part of the planet RAM saving comes from closed ways that are only wrongly oriented. These are now kept as reversed references into the way store, instead of being repaired and stored as separate generated polygons: "Generated polygons" in the log falls from 300M to 7.6M.

Extracts

Extract	Wall time before -> after	Speedup	Peak RAM before -> after	RAM reduction
Luxembourg	7.8 s -> 1.8 s	4.4x faster	5.22 GB -> 0.25 GB	20x less
Stuttgart (Regierungsbezirk)	21.3 s -> 6.7 s	3.2x faster	6.19 GB -> 0.54 GB	11x less

Total CPU time also falls 1.5-1.8x, so the speedup is not just better use of cores; the work itself shrank.

Measurement: interleaved A/B runs (6 pairs per extract) against the parent commit, after a discarded warm-up run. Setup: OpenMapTiles profile, gzip MVT to .mbtiles, 22-thread laptop (Core Ultra 7 165H). Times and RAM are medians.

Depends on #938 (MLT output). This branch is built on top of it, so the diff includes the MLT commit until #938 is merged. The performance changes are only the last commit.

Output

• Tile output no longer depends on thread count or scheduling. Previously, runs with different --threads values differed in up to ~200 Luxembourg tiles, because what the ClipCache held depended on scheduling. Tiles are now split into fixed batches, each with its own clip cache.
• Making output deterministic shifts a small number of z9-11 landcover/landuse/water vertices by 1-2 units out of 4096, inside the clip buffer. Different --threads values already caused differences of this kind before.
• Every other change gives identical output. This was checked with a semantic tile diff on Monaco and Luxembourg, plus Monaco with: maxzoom 16, minzoom 13, --materialize-geometries, --store, --shard-stores and --threads 1.

Main changes

• Memory
  • mmap_allocator: per-thread regions use anonymous mmap instead of zero-filled vectors. This is the largest RSS win.
  • UsedWays is a sparse sorted vector, with a dense-bitmap fallback for planet-scale inputs.
  • On glibc, M_TRIM_THRESHOLD is set unless the user tunes malloc through the environment.
  • Subdivided relation blocks are decompressed once and shared between chunks.
  • Per-batch cache resets.
• Tiles
  • The batch partition and per-batch ClipCache (with per-zoom levels) described under Output.
  • A conservative sweep-based "definitely valid" precheck runs before boost::geometry::is_valid. Boost still runs whenever the precheck is unsure.
  • Visvalingam: rings that are wrong only in orientation are reversed instead of dissolved, the heap is indexed, and MVT attribute writing is memoized.
  • Tile coordinate filtering scans words instead of calling test() per tile.
• Ingest
  • Lua: Find/Holds/Attribute*/MinZoom/ZOrder are native lua_CFunctions that keep kaguya's argument-type rules. Each lua_State gets a size-class pool allocator.
  • Ways whose only fault is wrong orientation are reversed instead of being sent through make_valid.
  • SortedNodeStore: a decoded-chunk cache and batched lookups.
  • TagMap uses flat shards, and AttributePairStore has a thread-local lookup table.
  • Shapefiles: records outside the bbox are skipped using the bbox in each record header.
• polylabel: provably no-op cells are pruned, and an SSE2 distance kernel is used on x86-64. Both give bit-identical results.

Fast paths were cross-checked against the original code paths in TMOPT_VERIFY builds (not part of this PR) on several extracts.

Not tested

• MSVC/Windows and macOS builds. Portable fallbacks exist for the SIMD and popcount code.
• A tile-by-tile comparison of the planet output; only extracts were diffed. Planet --store mode (only in-memory planet runs were timed).

🤖 Generated with Claude Code