[fcollman]

Bulk uploads can now take supervoxel and root id columns from the source file, and skip the lookups those columns make unnecessary.

What changes for users

In step 2 of the upload wizard, each bound spatial point (pre_pt, post_pt on a synapse table) gets two optional rows: <point>_supervoxel_id and <point>_root_id. Plain spatial points (ctr_pt) don't get them, because they have no segmentation columns. These rows only appear on deployments that run Ray uploads; a Celery upload with these columns mapped is refused at start.

When a root column is mapped, step 3 asks when the roots were looked up: at one time, sometime between two times, or not known. Step 4's preview checks the first rows' ids against the chunkedgraph and reports how many match, which catches a column mapped to the wrong point.

How it works

• Separate segmentation CSV: processing writes the supplied ids to a second CSV (<processed>.segmentation.csv), and a new load_segmentation stage imports it into the segmentation table with Cloud SQL.

• Exact ids: CSV id columns are read as text, and nullable parquet integers as Python ints, so 64-bit ids aren't rounded through float64. A file that stores ids as large floats is refused.

• Stages per case:

  Supplied	Spatial lookup	resolve_roots
  nothing	as before	skipped
  supervoxels	skipped for those points	looks up their roots
  supervoxels + roots	skipped	checks every root, looks up only the stale ones
  roots only	runs for that point	checks the roots against the new supervoxels

• resolve_roots:

  • Chunking: it works through the segmentation table in ranges of about 20,000 rows. Ranges follow the ids present, so sparse ids don't make empty chunks.
  • Per chunk: it fills blank supervoxels from the point's position, checks supplied roots, and looks up missing and stale roots. Only changed rows are written.
  • Checkpoints: chunks are recorded in a new upload_root_chunks table. It is a new table only, so the schema version stays 1. Chunks also pause for frozen materialization like spatial chunks do.

• Every supplied root is checked at the job's timestamp, whatever time the user gives. The table is published as correct at that timestamp, and the live update only fixes roots that expire after it, so a root already stale then would otherwise stay wrong. A root is current only if it's a top-layer id that existed and had no new parent before the timestamp. That is kvdbclient 0.9's is_latest_roots rule, written out here because kvdbclient ≥0.8 needs Python 3.13. Older kvdbclient counted ids that never existed as current.

• Parallel lookups: chunkedgraph calls go in pieces of 1,000 ids, 8 at a time per task (UPLOAD_ROOT_THREADS). kvdbclient sends one Bigtable request at a time. On a 1-CPU pod, threading took get_roots from 270 to 1,548 supervoxels a second, and is_latest_roots from 1,523 to 15,136 roots a second.

• Also fixed: the root lookup's timestamp parsing failed when the microseconds were 0.

Tested on ltv7

The test files are the 1M synapse file, with ids taken from synapses_1m_inprod6. For the root files, the roots were either current or from 90 days earlier:

Upload	File	Roots checked / stale	Roots looked up	resolve_roots	Submit → publish
synapses_1m_sv3	supervoxels	–	2,028,544	5m18s*	9m
synapses_1m_root_now1	+ current roots	2,028,542 / 161	163	51s	4m
synapses_1m_root_90d1	+ roots from 90 days earlier	2,028,542 / 40,723	40,725	53s	4m
synapses_1m_root_90d2 (final image, stricter root check)	+ roots from 90 days earlier	2,028,542 / 40,740	40,742	2m20s†	7m

† On a cold cluster; 90d1 ran on workers left up by the run before it.

* About 4 minutes of that was the Ray autoscaler adding workers. Once they were up, the last 48 of 51 chunks finished in about 80 seconds. For comparison, the normal path (spatial lookup) for the same file is 9m07s.

In every table:

• the supervoxels match the file;
• every root equals get_roots(supervoxel, job timestamp);
• the indexes match the model;
• no temporary upl… tables are left over.

The 90-day file's stale roots match the count built into the file (40,603), plus newer edits.

Unit and integration tests: 25 new tests in tests/test_upload_supplied_ids.py, and the full suite passes (820 passed, 2 skipped).

Notes

• CLUSTER: a source with its own unsorted ids still gets its segmentation table clustered by finish_tables, exactly like the annotation table.
• Rewrites: a supervoxels-only upload updates nearly every segmentation row once in resolve_roots, so the CLUSTER in finish_tables rewrites the table. With supplied roots, only stale rows change.

🤖 Generated with Claude Code