[fcollman]

Supersedes #266. This branch includes its commit: the GCS transfer and the CLUSTER skip, which the Celery staging path still uses.

Why

A Ray bulk upload today is built in staging, then copied into production. On minnie's 337M-row synapses_300m_ray2 that meant:

• the annotation indexes were built twice (~5.3h each);
• the tables were copied again;
• both tables (~166 GB) were on the instance twice until cleanup.

The copied annotation table also came out unclustered (id correlation −0.0006), and its id sequence was left at 1.

What changes (Ray path only)

• Build in production under temporary names. The tables are upl<hash6>_<table> and its segmentation table, with no metadata rows until publish. The table's metadata travels in the job's params.

• Annotation table:

  1. Cloud SQL import into the temporary table.
  2. Primary key.
  3. CLUSTER by id, only when the source's ids are unsorted (correlation below UPLOAD_CLUSTER_SKIP_CORRELATION, 0.99).
  4. The remaining indexes.

• Spatial lookup: runs against the temporary table. It takes the schema from the job, since there is no metadata row.

• Segmentation table: clustered by id, then indexed, before publish.

• Publish (one transaction):

  • re-checks that the final names are free;
  • renames the tables, indexes, constraints and sequence to the names index_manager expects;
  • setvals the sequence to max(id);
  • inserts both metadata rows.

  Tables, indexes and metadata become visible together. Publish is idempotent on resume (a table comment marks the job).

• Yields to frozen materialization. create_analysis_database records a clone request in upload_state.clone_requests. It waits until no upload holds the database: each driver pauses its new work, lets in-flight statements finish and drops its connections. The wait is bounded by UPLOAD_CLONE_WAIT_SECONDS (default 2h). Clones made mid-upload drop the upl… tables.

• Failure or cancel drops only the job's recorded upl… tables.

• Name-length check: a job is refused at start if any derived temporary or final index name would exceed 63 bytes.

Celery uploads keep the staging path.

Expected effect on a 337M-row upload

About 22h → about 15.5h, because nothing is transferred and no index is built twice. There is one copy on disk instead of two.

Testing

• Full suite: 741 passed, 2 skipped, on a fresh PostGIS.
• New production-build tests:
  • load, lookup, finish and publish, with sources both in id order and shuffled. They assert clustering, the spatial bounds, points returned from the bbox, and ctid order;
  • publish idempotency and a name collision;
  • the clone gate.
• ltv7, 1M-row runs (vBuildInProd3), checked against the earlier staging-path run:
  • 1,020,000 rows, all ids distinct;
  • the same zero supervoxel and root counts;
  • 6 + 3 indexes with the expected names;
  • both tables at id correlation 1.0;
  • sequence = max(id);
  • metadata valid, no upl leftovers.
• Frozen materialization triggered mid-lookup on ltv7:
  • the upload released the database 24s after the request;
  • the clone completed and contained no upl tables;
  • the upload resumed and published normally.
• ltv7, 3-part file (350 MB, 4,795,306 rows): finished in 20.5 min, after the submitter moved to standard-pool (chart 5.33.2).
  • rows equal distinct ids in both tables; every segmentation id is in the annotation table;
  • no null or zero supervoxel ids; 18 pre and 12 post zero root ids;
  • 6 + 3 indexes with the expected names;
  • both tables at id correlation 1.0;
  • sequence = max(id);
  • metadata valid, no upl leftovers.

Before a large run

• UPLOAD_CLONE_WAIT_SECONDS should exceed the longest single statement. A CLUSTER of a 337M-row annotation table can take hours.
• CLUSTER needs about the table's size in free disk.

🤖 Generated with Claude Code