Skip to content

Vector index bloat: no compress_neighbors/max_neighbors tuning - 542 MB shard for ~656 memories (2048-dim embeddings) #277

Closed

Description

@shdowzh888

Environment

  • opencode-mem: 2.22.2 (issue likely affects all versions up to 2.25.0)
  • OS: Windows 11
  • Embedding: remote OpenAI-compatible API (doubao-embedding-vision), 2048 dims

Symptom

With only ~656 memories in a project shard, the shard DB grew to 542 MB. dbstat shows the two DiskANN shadow tables account for 98% of the file:

object size
memories_vec_idx_shadow 267.4 MB
memories_tags_vec_idx_shadow 266.5 MB
memories (all actual text + metadata) 11.0 MB

That is ~850 KB of index overhead per memory, while the raw vectors are only 2048 × 4 B ≈ 8 KB each.

Root cause

initShardDb() creates both vector indexes with 'metric=cosine' only:

CREATE INDEX ... ON memories (libsql_vector_idx(vector, 'metric=cosine'))

libSQL defaults are aggressive at high dimensionality:

  • max_neighbors = 3·√D → ~137 neighbors per node at D=2048
  • compress_neighbors unset → neighbor vectors stored as uncompressed F32

This matches the documented space formula N · (Storage(T1) + M · Storage(T2)) in the Turso docs.

Fix verified locally

Dropping and re-creating the same-named indexes with tuned parameters, then VACUUM, shrinks the whole shard from 542.5 MB → 73.3 MB (~7.4×) with search quality intact:

DROP INDEX IF EXISTS memories_vec_idx;
DROP INDEX IF EXISTS memories_tags_vec_idx;
CREATE INDEX memories_vec_idx ON memories (libsql_vector_idx(vector,
  'metric=cosine', 'compress_neighbors=float8', 'max_neighbors=20'));
CREATE INDEX memories_tags_vec_idx ON memories (libsql_vector_idx(tags_vector,
  'metric=cosine', 'compress_neighbors=float8', 'max_neighbors=20'));
VACUUM;

Post-rebuild verification: row counts unchanged; vector_top_k smoke test returns correct semantic hits; multi-topic recall quality confirmed unaffected. This matches Kin's published case study (The space complexity of vector indexes in LibSQL, 8x reduction with identical search results on small datasets).

Note: because retrieval goes through vector_top_k('<index-name>', ...), rebuilding with the same index name requires no plugin code changes — the plugin picks up the tuned index transparently.

Suggestion

  1. Add 'compress_neighbors=float8', 'max_neighbors=20' to both index definitions in initShardDb() — optionally exposed as config keys (e.g. indexMaxNeighbors, indexNeighborCompression) for advanced users.
  2. Consider documenting libSQL's native REINDEX <idx> as the maintenance path for existing bloated shards (it rebuilds from base table data).

Activity

  1. lindixu6-hash commented on Aug 28, 2026

    @lindixu6-hash
    Collaborator

    Thanks for the unusually complete report. I verified the core claim against current main / @libsql/client:

    • both indexes are still created with only metric=cosine in src/services/turso/shard-manager.ts;
    • the runtime accepts both compress_neighbors=float8 and max_neighbors=20 for the normal and partial vector indexes;
    • inserts and vector_top_k queries continue to work with those options;
    • Turso's documented storage formula confirms that high-dimensional uncompressed neighbor vectors can dominate the file.

    So this is a real storage-amplification bug, especially for remote 1536/2048-dimensional embeddings.

    One migration correction: plain REINDEX memories_vec_idx does not change index options. I verified that sqlite_schema.sql remains the original libsql_vector_idx(vector, 'metric=cosine') definition before and after REINDEX. Existing shards therefore need an explicit, failure-safe migration that drops and recreates both indexes with the new definition; VACUUM is then needed to reclaim the old pages on disk.

    Before changing the default globally, we should cover two things:

    1. benchmark recall/write-cost/size for the dimensions opencode-mem supports (384, 512, 768, 1024, 1536, 2048), rather than assume one max_neighbors value is optimal everywhere;
    2. design an idempotent migration for existing shards that preserves the current failure guarantees (no partially missing index if rebuild fails, and clear progress/error reporting for large databases).

    A practical first patch can add explicit conservative index options for newly created shards plus failure-sensitive SQL-definition/search tests. A follow-up migration can rebuild existing indexes and vacuum them with backup/rollback semantics. I am marking this as a bug; the reported 542 MB → 73 MB result is strong evidence for prioritizing it.

  2. EyJunge1 commented on Oct 1, 2026

    @EyJunge1
    Collaborator

    Closing as resolved by a later architecture change.

    • #279 added compress_neighbors=float8 / max_neighbors=20 for newly created DiskANN indexes (no rebuild migration for existing shards).
    • #340 hard-cutover to @tursodatabase/database removed DiskANN entirely: shards no longer create memories_vec_idx / memories_tags_vec_idx, and search uses exact cosine via vector_distance_cos.
    • Existing DiskANN shards are rewritten on startup by runTursoEngineMigration() (engine migrator), which drops the shadow-table bloat.

    The original storage amplification no longer applies on current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions