Skip to content

Bump oaklib to >=0.7.3 so sqlite:obo: resolves against the CDN (#562) - #565

Merged
realmarcin merged 5 commits into
mainfrom
fix/562-oaklib-cdn
Aug 28, 2026
Merged

realmarcin merged 5 commits into
mainfrom
fix/562-oaklib-cdn

Conversation

@realmarcin

@realmarcin realmarcin commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Closes #562.
Closes #568.
Closes #569.

INCATools is retiring public access to the raw semantic-sql S3 bucket (semantic-sql#112). Below oaklib 0.7.2, sqlite:obo: selectors resolve against https://s3.amazonaws.com/bbop-sqlite; at 0.7.2+ they use SEMSQL_SQLITE_URL_BASE (default https://semanticsql.berkeleybop.io), with OAKLIB_SEMSQL_SQLITE_URL_BASE as a mirror escape hatch.

This repo sat at 0.6.23 — on the raw-S3 path. The exposure is not theoretical: just validate-products is a blocking CI gate that pulls CHEBI/GO/ENVO/PATO/RO through OAK. It would have gone red when the bucket lost AllUsers.

Change

  • pyproject.toml: oaklib>=0.5.0 → oaklib>=0.7.3, and linkml>=1.10.0 → linkml>=1.11.1, each with its reason recorded inline so the floor is not lowered again by accident.
    • The oaklib floor is 0.7.3, not 0.7.2, per oaklib floor of >=0.7.2 is not installable on Python 3.14, which requires-python permits #569: 0.7.2 declares requires_python <3.14 while this project's requires-python is an unbounded >=3.10, so a 0.7.2 floor would name a minimum that cannot be installed on a supported interpreter. 0.7.3 widened it to <3.15. Verified that 0.7.3 still carries the CDN default (SEMSQL_SQLITE_URL_BASE = https://semanticsql.berkeleybop.io), so the higher floor does not permit an install that reintroduces the bug.
  • uv.lock: oaklib 0.6.23 → 0.7.4.

Verified, not assumed

check result
pre-state 0.6.23 has no SEMSQL_SQLITE_URL_BASE (ImportError) — the issue's mechanism, confirmed
post-state SEMSQL_SQLITE_URL_BASE = https://semanticsql.berkeleybop.io
end-to-end download cache moved aside → get_adapter("sqlite:obo:ro") fetched over the network → RO:0002327 → "enables"
CDN availability all six databases this repo requests return HTTP 200 (ro 0.6 MB → ncbitaxon 2.27 GB)
just validate-products (blocking, OAK-backed) exit 0 locally — 361 OK_CANONICAL, 2 OK_SYNONYM, 1 OK_EXCEPTION, 121 SKIPPED_NO_ADAPTER — and dispatched on CI against this branch: success
test suite 854 passed, on all four CI lanes
just qc / validate-history / validate-strict exit 0 / exit 0 / 0 ERROR rows
Python matrix oaklib 0.7.4 is >=3.10,<3.15; covers 3.10–3.13
setuptools<82 pin (eutils/pkg_resources) still satisfied at 81.0.0 — and 0.7.4 now pins it upstream too

The linkml side effect, and why it is unavoidable

The resolve carries linkml 1.10.0 → 1.11.1 plus transitive moves. This is forced, and the mechanism is exact: linkml 1.10.0 requires click<8.2, while oaklib 0.7.2+ hard-requires llm, which pulls sqlite-utils, which requires click>=8.3.1. Direct contradiction — so oaklib>=0.7.2 and linkml==1.10.0 cannot coexist. That is why the old linkml>=1.10.0 floor named a combination that no longer resolves, and why it is raised here.

Risk was tested rather than argued: validate-strict (0 ERROR rows, tracked report byte-identical), gen-pydantic, and the full suite all pass; deprecation warnings drop from 25 to 2.

Also worth knowing: openai becomes a runtime dependency on the 3.10/3.11 lanes (llm requires it unconditionally), where it previously arrived only via the python>=3.12 dev extra. This repo imports none of the added packages directly.

Review findings filed, not fixed here

Adversarial review surfaced two pre-existing CI-coverage gaps that this PR exposes rather than causes:

Sibling repos CultureMech and MediaIngredientMech pin 0.6.23 identically and need the same bump.

🤖 Generated with Claude Code

INCATools is retiring public access to the raw semantic-sql S3 bucket
(INCATools/semantic-sql#112). Below oaklib 0.7.2, `sqlite:obo:`
selectors resolve against `https://s3.amazonaws.com/bbop-sqlite`; at
0.7.2 and above they resolve against SEMSQL_SQLITE_URL_BASE, defaulting
to https://semanticsql.berkeleybop.io, with OAKLIB_SEMSQL_SQLITE_URL_BASE
as the escape hatch for a mirror.

This repo was pinned at 0.6.23, i.e. on the raw-S3 path. That matters
here because `just validate-products` is a BLOCKING CI gate that pulls
CHEBI/GO/ENVO/PATO/RO through OAK, and audit_canonical_examples pulls
sqlite:obo:ncbitaxon; both break when the bucket loses public access.

pyproject.toml now requires >=0.7.2 rather than >=0.5.0, with the reason
recorded inline so the floor is not lowered again by accident. The lock
moves oaklib 0.6.23 -> 0.7.4.

Verified rather than assumed:

- 0.6.23 has no SEMSQL_SQLITE_URL_BASE at all (ImportError), which is
  itself the pre-state the issue describes; 0.7.4 reports
  https://semanticsql.berkeleybop.io.
- End-to-end through the real download path: with ~/.data/oaklib/ro.db
  moved aside, `get_adapter("sqlite:obo:ro")` fetched ro.db.gz over the
  network and resolved RO:0002327 -> "enables".
- The CDN serves the databases this repo actually requests (ro 0.6 MB,
  pato 28 MB, ncbitaxon 2.3 GB compressed all return HTTP 200).
- `just validate-products`, the blocking OAK-backed gate, exits 0:
  361 OK_CANONICAL, 2 OK_SYNONYM, 1 OK_EXCEPTION, 121 SKIPPED_NO_ADAPTER.
- 854 tests pass, `just qc` and `just validate-history` exit 0,
  `validate-strict` reports 0 ERROR rows, and gen-pydantic still emits
  the dataclasses.

Scope note: the resolve also carries linkml 1.10.0 -> 1.11.1 and five
other transitive moves. That is not required by oaklib's own metadata
(it asks only for linkml-runtime>=1.5.3), but pinning linkml back to
1.10.0 does not resolve. Since this repo is LinkML-heavy the risk was
tested directly rather than argued: validate-strict, gen-pydantic and
the full suite all pass, and the deprecation warnings drop from 25 to 2.

Closes #562

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial review of this PR found the adjacent dependency floor names
a combination that no longer resolves, in the same file where I had just
documented the oaklib floor so it would not be lowered by accident.

pyproject declared `linkml>=1.10.0`, but linkml 1.10.0 cannot coexist
with `oaklib>=0.7.2`. Verified rather than inferred: linkml 1.10.0
requires click<8.2, while oaklib pulls llm -> sqlite-utils, which
requires click>=8.3.1 — a direct contradiction, and the actual mechanism
behind "holding linkml at 1.10.0 does not resolve". The floor is now
>=1.11.1 with that reason recorded inline. The lock is unchanged apart
from the requirement line; linkml stays 1.11.1 and oaklib 0.7.4.

Also correcting the previous commit message: it said
audit_canonical_examples "pulls sqlite:obo:ncbitaxon; both break when
the bucket loses public access." In CI that is wrong —
canonical-example-taxonomy.yaml:47 runs the script with `--ncbi-api`,
not the OAK path. The OAK path is used locally, so the concern is real,
but it was never a CI break. `just validate-products` remains the CI
gate that genuinely depends on this fix.

Two CI-coverage gaps the review surfaced are filed rather than fixed
here, since both are pre-existing and out of this PR's scope: #566
(label-correspondence's paths filter excludes pyproject.toml and
uv.lock, so the blocking OAK gate never ran on this PR) and #567
(uv.lock is in no workflow paths filter at all).

Refs #562

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hon range

The floor this branch set, >=0.7.2, names a version that cannot be installed
on part of the interpreter range this project declares support for:

  requires-python = ">=3.10"   (no upper bound)
  oaklib 0.7.2     requires_python >=3.10,<3.14
  oaklib 0.7.3/4   requires_python >=3.10,<3.15

On Python 3.14 -- which ">=3.10" permits -- 0.7.2 is uninstallable, so the
floor names an unreachable minimum. CI does not catch it because the matrix
stops at 3.13, which is why this branch is green with the defect present.

0.7.3 carries the same CDN default as 0.7.2 (semantic-sql#112), so nothing
about the migration changes; only the floor moves. The lock already resolved
to 0.7.4 and is unchanged apart from the recorded specifier.

This also aligns with CultureBotAI/CultureMech, which reached >=0.7.3 by the
same reasoning and has since merged.

Closes #569.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@realmarcin realmarcin changed the title Bump oaklib to >=0.7.2 so sqlite:obo: resolves against the CDN (#562) Bump oaklib to >=0.7.3 so sqlite:obo: resolves against the CDN (#562) Aug 28, 2026
realmarcin and others added 2 commits August 27, 2026 19:08
Final review found the floor this PR exists to establish is unguarded,
and that its justification overstates the case. Both fixed.

The comment claimed ">=0.7.3 is required, not merely preferred: below
0.7.2 ..." — naming one version as the subject and a different one as
the reason. What #562 requires is 0.7.2, which is where
SEMSQL_SQLITE_URL_BASE landed; the step to 0.7.3 is hygiene and fleet
consistency with CultureMech#365, and it changes no install outcome
here. Measured rather than asserted: under uv, this project's actual
installer, `oaklib>=0.7.2` and `oaklib>=0.7.3` both resolve to 0.7.4 on
3.14 and on 3.15. The comment now says that.

tests/test_oaklib_floor.py is new. The floor lives in one line of
pyproject.toml and nothing checked it, so a routine "relax the
constraints" edit would silently put `sqlite:obo:` back on the retiring
S3 bucket without failing any gate. It asserts the DECLARED floor rather
than the resolved version, because the lock can be regenerated while the
declaration is what binds a fresh install, plus a second test that the
lock and the declaration have not drifted. Proven by mutation: it
accepts >=0.7.3 and >=0.7.2, and catches >=0.5.0, a bare `oaklib`, and
an upper-bound-only specifier. CultureMech#365 added the equivalent
guard for the same reason; this closes that inconsistency between the
repos.

Verified 0.7.3 carries the same CDN default as 0.7.2 before raising the
floor onto it, so the higher floor cannot readmit #562.

One review finding is filed rather than fixed, being pre-existing and
out of scope: #570 — `requires-python` is an unbounded >=3.10 while
tests/test_python_support.py asserts support is 3.10-3.13. That
disagreement is what made #569's premise ambiguous.

Refs #562

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guard I added in the previous commit imported `tomllib`, which is
stdlib only from 3.11. It passed locally on 3.13 and failed the 3.10
lane in CI with ModuleNotFoundError — the one lane that could catch it,
doing exactly its job.

Now reads the raw text with a regex, which is how tests/test_python_support.py
already reads pyproject and for the same reason. No new dependency: adding
tomli for one test would be a worse trade than a five-line parse.

Verified it still resolves the right requirements (oaklib>=0.7.3,
linkml>=1.11.1) and that no stdlib-3.11 import remains.

Refs #562

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@realmarcin
realmarcin merged commit a49e60c into main Aug 28, 2026
8 checks passed
@realmarcin
realmarcin deleted the fix/562-oaklib-cdn branch August 28, 2026 02:19
realmarcin added a commit that referenced this pull request Aug 28, 2026
…aims

Adversarial review found the module docstring still described only half
the file, one revert shape slipping through, and — twice — my own
justification being wrong on the facts.

Docstring: the header said this file "asserts the declared floor, not
the resolved version", which the two tests added in this PR do exactly
the opposite of. Rewritten around the two layers the file actually has:
declaration (what we control) and runtime (what we care about), with why
each covers what the other cannot.

Coverage, from the reviewer's own mutation testing:

- The download-path check asserted the constant was MENTIONED. oaklib
  ships the pre-0.7.2 call commented out beside the live one
  (ensure_from_s3(s3_bucket="bbop-sqlite", ...), labelled "Option 2"),
  so the cheapest revert is a two-line comment swap leaving the unused
  import in place — which passed. Now asserts the constant is
  INTERPOLATED, plus that ensure_from_s3 does not appear in live
  (non-comment) code. Verified: the comment-swap revert is now caught,
  as is an inlined host.
- S3_HOST was the single spelling "s3.amazonaws.com", so a
  virtual-hosted region URL such as
  https://bbop-sqlite.s3.us-east-1.amazonaws.com — the same bucket —
  passed both assertions. Widened to the ".amazonaws.com" domain.
- OVERRIDE_ENV was hardcoded with nothing checking oaklib still reads
  it. If upstream renamed it, delenv would become a no-op and the test
  would silently assert the ambient environment instead of the shipped
  default, which is the one thing that helper exists to prevent. Now
  asserts the name appears in oaklib's constants source.

Two claims in the previous commit message and PR body were wrong, and
the decision they defended was right for a different reason:

- "TraitMech has no tomli" is false. tomli IS in uv.lock, installed on
  <3.11 as a transitive dependency of pytest and coverage, so
  CultureMech's guarded try/except import would have worked here. The
  accurate claim is that it is not a DECLARED dependency, and leaning on
  a transitive test-only package to read our own manifest is fragile.
- "that exact import already broke the 3.10 lane" is false. What broke
  d103043 was a bare `import tomllib` with no fallback, not the guarded
  form. The real justification is the one the code already gives:
  consistency with test_python_support.py.

Also: dropped an unused monkeypatch parameter, hoisted `import inspect`
to module level, and noted in the helper that the reload leaves
oaklib.constants un-overridden for the session (inert — nothing here
reads it under pytest, and oaklib's downloader binds its own copy).

Filed rather than fixed, being pre-existing from #565: #572 —
CDN_DEFAULT_FIRST_RELEASE is 0.7.2 while the declared floor is 0.7.3, so
the guard would accept a floor #565 argued is uninstallable.

Refs #568

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
realmarcin added a commit that referenced this pull request Aug 29, 2026
Sixth instance of one defect: a `paths:` filter narrower than what the job
reads. #184, #200, #250, #252, #554 — and now this. Every one was caught
by human review, never by CI.

Measured before changing anything: six PR-triggered workflows install
dependencies with uv and filter on paths. NONE listed uv.lock (#567), and
three did not list pyproject.toml either (#566):

  canonical-example-taxonomy   both missing
  curation-history             both missing
  label-correspondence         both missing
  pytest / qc / validate-strict   uv.lock missing

label-correspondence is the one that bit: it is the blocking gate that
resolves `sqlite:obo:` through OAK, so a PR changing which host OAK
downloads from is exactly the PR it exists to check — and #565, which
changed precisely that, could not trigger it. It had to be dispatched by
hand to get any CI evidence at all.

Not touched, with reasons: claude-code-review.yml has its pull_request
trigger deliberately removed (quota, 2026-08-13); pr-sanity and
vendored-sync use a bare trigger and already run on every PR;
pr-checks-present and pr-shepherd have no pull_request trigger; and
vendored-sync installs nothing with uv.

tests/test_workflow_dependency_paths.py closes the class rather than this
instance. `audit-qc-paths` derives qc.yaml's read-set from its recipe
chain; this asserts the dependency half for every workflow: if a job runs
`uv sync`, the resolved environment is one of its inputs, so both files
belong in the filter that decides whether it runs. A second test keeps the
push and pull_request filters agreeing, since the workflows that repeat
their list by hand can drift where the anchored ones cannot.

Scoped deliberately: only filtered, PR-triggered, uv-installing jobs. A
job with no `paths:` already runs on everything, and one that never
touches uv has no dependency input to miss. The discovery loop asserts it
found at least one such job, so the test cannot pass vacuously.

Mutation-checked: removing `uv.lock` from qc.yaml fails the guard with
"qc.yaml omits uv.lock". Restored, it passes.

Closes #566
Closes #567

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
realmarcin added a commit that referenced this pull request Aug 29, 2026
* Guard the oaklib semsql URL at runtime, not just by version (#568)

#568 asked for three assertions ported from CultureMech's
test_oaklib_semsql_pin.py. #565 shipped two of them — the declared floor
and the lock agreeing with it — and #568 was auto-closed two seconds
after that merge, so the third never landed. It is the one the issue
called out as mattering most: the version number is what we control,
the resolved constant is what we care about.

Adds two runtime assertions:

- the shipped SEMSQL_SQLITE_URL_BASE contains no s3.amazonaws.com and is
  https. This keeps working if upstream renames the constant, changes
  its default again, or ships a version whose number implies a fix it
  does not contain.
- oaklib's sqlite implementation still READS that constant, and does not
  mention the S3 host directly. A correct constant is worthless if the
  downloader stopped using it: oaklib builds the URL as
  f"{SEMSQL_SQLITE_URL_BASE}/{prefix}.db.gz", and an inlined host there
  would leave every version assertion passing while sqlite:obo: went
  back to the retiring bucket.

The env-override caveat #568 raised is handled deliberately rather than
by accident. SEMSQL_SQLITE_URL_BASE is os.environ.get(OVERRIDE, CDN)
read at import time, so on a machine that sets a mirror the naive test
asserts the override and a reverted default passes unnoticed behind it.
The test clears OAKLIB_SEMSQL_SQLITE_URL_BASE and reloads the module, so
it always asserts what oaklib ships. Verified: with the override set to
the S3 bucket, the suite still passes and still reads the CDN default.

Each assertion was mutation-checked rather than assumed: a reverted
default is caught, an inlined host is caught, and the override does not
mask either.

Adapted rather than transcribed, because two things in CultureMech's
copy are false here. It reads pyproject with tomllib and falls back to
tomli; TraitMech has no tomli, and tomllib is stdlib only from 3.11 —
that exact import already broke the 3.10 lane once in #565, so this
stays on the regex parse test_python_support.py uses. And its comment
justifies the floor with "this project supports <3.15 and runs 3.14 in
CI", which is CultureMech's matrix, not ours (3.10-3.13; see #570).

Extends the existing tests/test_oaklib_floor.py instead of adding a
second file, so the four assertions live together rather than split
across two names for one property.

Closes #568

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Harden the runtime guard, fix the docstring, and correct two false claims

Adversarial review found the module docstring still described only half
the file, one revert shape slipping through, and — twice — my own
justification being wrong on the facts.

Docstring: the header said this file "asserts the declared floor, not
the resolved version", which the two tests added in this PR do exactly
the opposite of. Rewritten around the two layers the file actually has:
declaration (what we control) and runtime (what we care about), with why
each covers what the other cannot.

Coverage, from the reviewer's own mutation testing:

- The download-path check asserted the constant was MENTIONED. oaklib
  ships the pre-0.7.2 call commented out beside the live one
  (ensure_from_s3(s3_bucket="bbop-sqlite", ...), labelled "Option 2"),
  so the cheapest revert is a two-line comment swap leaving the unused
  import in place — which passed. Now asserts the constant is
  INTERPOLATED, plus that ensure_from_s3 does not appear in live
  (non-comment) code. Verified: the comment-swap revert is now caught,
  as is an inlined host.
- S3_HOST was the single spelling "s3.amazonaws.com", so a
  virtual-hosted region URL such as
  https://bbop-sqlite.s3.us-east-1.amazonaws.com — the same bucket —
  passed both assertions. Widened to the ".amazonaws.com" domain.
- OVERRIDE_ENV was hardcoded with nothing checking oaklib still reads
  it. If upstream renamed it, delenv would become a no-op and the test
  would silently assert the ambient environment instead of the shipped
  default, which is the one thing that helper exists to prevent. Now
  asserts the name appears in oaklib's constants source.

Two claims in the previous commit message and PR body were wrong, and
the decision they defended was right for a different reason:

- "TraitMech has no tomli" is false. tomli IS in uv.lock, installed on
  <3.11 as a transitive dependency of pytest and coverage, so
  CultureMech's guarded try/except import would have worked here. The
  accurate claim is that it is not a DECLARED dependency, and leaning on
  a transitive test-only package to read our own manifest is fragile.
- "that exact import already broke the 3.10 lane" is false. What broke
  d103043 was a bare `import tomllib` with no fallback, not the guarded
  form. The real justification is the one the code already gives:
  consistency with test_python_support.py.

Also: dropped an unused monkeypatch parameter, hoisted `import inspect`
to module level, and noted in the helper that the reload leaves
oaklib.constants un-overridden for the session (inert — nothing here
reads it under pytest, and oaklib's downloader binds its own copy).

Filed rather than fixed, being pre-existing from #565: #572 —
CDN_DEFAULT_FIRST_RELEASE is 0.7.2 while the declared floor is 0.7.3, so
the guard would accept a floor #565 argued is uninstallable.

Refs #568

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Restore the assertion my fix deleted, and parse instead of grep

Review of 8cc9712 found that commit — titled "correct two false claims" —
had itself introduced a third. Three fixes.

#575, the serious one. 93443c3 asserted the download path never names
the S3 host; 8cc9712 DELETED that assertion while its message described
the change as "widened to the .amazonaws.com domain". One assertion was
widened; the other was dropped, and the PR body still advertised it.

The cost is concrete: keeping the interpolated line and ADDING an inlined
S3 URL below it leaves the constant present but dead, and passed every
remaining assertion. Restored, and the additive mutation is caught again.

#576. Both source checks scanned raw text, so neither could tell live
code from comments:

- The OVERRIDE_ENV guard was a substring test, and oaklib's constants.py
  names the variable twice — once live, once in a prose comment above it.
  Renaming only the live read left the comment satisfying the guard, so
  delenv silently became a no-op and the test would assert the ambient
  environment instead of the shipped default. That is exactly the failure
  the assertion was added to prevent.
- The ensure_from_s3 filter treated every non-# line as live, so an
  upstream docstring merely MENTIONING the call would fail the lane — a
  behaviour-free change turning CI red, which is how a guard gets deleted.

Both now walk ast.parse: comments and string contents disappear, and a
getattr-assembled call that previously slipped through is caught.

#577. The interpolation check is deliberately kept — it is the only thing
catching the comment-swap revert — but its failure message now names the
likely benign cause (urljoin, an intermediate variable) and says to verify
the resolved URL before relaxing anything, rather than asserting sabotage.

Mutation-verified, all five: real source passes; additive S3 URL caught;
getattr call caught; comment-swap revert caught; docstring mention does
NOT fail. My first harness reported two false results because its
"unparseable" sentinel was a truthy string and the mutations had wrong
indentation, so a SyntaxError read as "passes" — rewritten to let parse
errors raise.

Closes #575
Closes #576
Closes #577

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant