Repository navigation
Bump oaklib to >=0.7.3 so sqlite:obo: resolves against the CDN (#562) - #565
Merged
Merged
Conversation
INCATools is retiring public access to the raw semantic-sql S3 bucket (INCATools/semantic-sql#112). Below oaklib 0.7.2, `sqlite:obo:` selectors resolve against `https://s3.amazonaws.com/bbop-sqlite`; at 0.7.2 and above they resolve against SEMSQL_SQLITE_URL_BASE, defaulting to https://semanticsql.berkeleybop.io, with OAKLIB_SEMSQL_SQLITE_URL_BASE as the escape hatch for a mirror. This repo was pinned at 0.6.23, i.e. on the raw-S3 path. That matters here because `just validate-products` is a BLOCKING CI gate that pulls CHEBI/GO/ENVO/PATO/RO through OAK, and audit_canonical_examples pulls sqlite:obo:ncbitaxon; both break when the bucket loses public access. pyproject.toml now requires >=0.7.2 rather than >=0.5.0, with the reason recorded inline so the floor is not lowered again by accident. The lock moves oaklib 0.6.23 -> 0.7.4. Verified rather than assumed: - 0.6.23 has no SEMSQL_SQLITE_URL_BASE at all (ImportError), which is itself the pre-state the issue describes; 0.7.4 reports https://semanticsql.berkeleybop.io. - End-to-end through the real download path: with ~/.data/oaklib/ro.db moved aside, `get_adapter("sqlite:obo:ro")` fetched ro.db.gz over the network and resolved RO:0002327 -> "enables". - The CDN serves the databases this repo actually requests (ro 0.6 MB, pato 28 MB, ncbitaxon 2.3 GB compressed all return HTTP 200). - `just validate-products`, the blocking OAK-backed gate, exits 0: 361 OK_CANONICAL, 2 OK_SYNONYM, 1 OK_EXCEPTION, 121 SKIPPED_NO_ADAPTER. - 854 tests pass, `just qc` and `just validate-history` exit 0, `validate-strict` reports 0 ERROR rows, and gen-pydantic still emits the dataclasses. Scope note: the resolve also carries linkml 1.10.0 -> 1.11.1 and five other transitive moves. That is not required by oaklib's own metadata (it asks only for linkml-runtime>=1.5.3), but pinning linkml back to 1.10.0 does not resolve. Since this repo is LinkML-heavy the risk was tested directly rather than argued: validate-strict, gen-pydantic and the full suite all pass, and the deprecation warnings drop from 25 to 2. Closes #562 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial review of this PR found the adjacent dependency floor names a combination that no longer resolves, in the same file where I had just documented the oaklib floor so it would not be lowered by accident. pyproject declared `linkml>=1.10.0`, but linkml 1.10.0 cannot coexist with `oaklib>=0.7.2`. Verified rather than inferred: linkml 1.10.0 requires click<8.2, while oaklib pulls llm -> sqlite-utils, which requires click>=8.3.1 — a direct contradiction, and the actual mechanism behind "holding linkml at 1.10.0 does not resolve". The floor is now >=1.11.1 with that reason recorded inline. The lock is unchanged apart from the requirement line; linkml stays 1.11.1 and oaklib 0.7.4. Also correcting the previous commit message: it said audit_canonical_examples "pulls sqlite:obo:ncbitaxon; both break when the bucket loses public access." In CI that is wrong — canonical-example-taxonomy.yaml:47 runs the script with `--ncbi-api`, not the OAK path. The OAK path is used locally, so the concern is real, but it was never a CI break. `just validate-products` remains the CI gate that genuinely depends on this fix. Two CI-coverage gaps the review surfaced are filed rather than fixed here, since both are pre-existing and out of this PR's scope: #566 (label-correspondence's paths filter excludes pyproject.toml and uv.lock, so the blocking OAK gate never ran on this PR) and #567 (uv.lock is in no workflow paths filter at all). Refs #562 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 28, 2026
…hon range The floor this branch set, >=0.7.2, names a version that cannot be installed on part of the interpreter range this project declares support for: requires-python = ">=3.10" (no upper bound) oaklib 0.7.2 requires_python >=3.10,<3.14 oaklib 0.7.3/4 requires_python >=3.10,<3.15 On Python 3.14 -- which ">=3.10" permits -- 0.7.2 is uninstallable, so the floor names an unreachable minimum. CI does not catch it because the matrix stops at 3.13, which is why this branch is green with the defect present. 0.7.3 carries the same CDN default as 0.7.2 (semantic-sql#112), so nothing about the migration changes; only the floor moves. The lock already resolved to 0.7.4 and is unchanged apart from the recorded specifier. This also aligns with CultureBotAI/CultureMech, which reached >=0.7.3 by the same reasoning and has since merged. Closes #569. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Final review found the floor this PR exists to establish is unguarded, and that its justification overstates the case. Both fixed. The comment claimed ">=0.7.3 is required, not merely preferred: below 0.7.2 ..." — naming one version as the subject and a different one as the reason. What #562 requires is 0.7.2, which is where SEMSQL_SQLITE_URL_BASE landed; the step to 0.7.3 is hygiene and fleet consistency with CultureMech#365, and it changes no install outcome here. Measured rather than asserted: under uv, this project's actual installer, `oaklib>=0.7.2` and `oaklib>=0.7.3` both resolve to 0.7.4 on 3.14 and on 3.15. The comment now says that. tests/test_oaklib_floor.py is new. The floor lives in one line of pyproject.toml and nothing checked it, so a routine "relax the constraints" edit would silently put `sqlite:obo:` back on the retiring S3 bucket without failing any gate. It asserts the DECLARED floor rather than the resolved version, because the lock can be regenerated while the declaration is what binds a fresh install, plus a second test that the lock and the declaration have not drifted. Proven by mutation: it accepts >=0.7.3 and >=0.7.2, and catches >=0.5.0, a bare `oaklib`, and an upper-bound-only specifier. CultureMech#365 added the equivalent guard for the same reason; this closes that inconsistency between the repos. Verified 0.7.3 carries the same CDN default as 0.7.2 before raising the floor onto it, so the higher floor cannot readmit #562. One review finding is filed rather than fixed, being pre-existing and out of scope: #570 — `requires-python` is an unbounded >=3.10 while tests/test_python_support.py asserts support is 3.10-3.13. That disagreement is what made #569's premise ambiguous. Refs #562 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The guard I added in the previous commit imported `tomllib`, which is stdlib only from 3.11. It passed locally on 3.13 and failed the 3.10 lane in CI with ModuleNotFoundError — the one lane that could catch it, doing exactly its job. Now reads the raw text with a regex, which is how tests/test_python_support.py already reads pyproject and for the same reason. No new dependency: adding tomli for one test would be a worse trade than a five-line parse. Verified it still resolves the right requirements (oaklib>=0.7.3, linkml>=1.11.1) and that no stdlib-3.11 import remains. Refs #562 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
realmarcin
added a commit
that referenced
this pull request
Aug 28, 2026
…aims Adversarial review found the module docstring still described only half the file, one revert shape slipping through, and — twice — my own justification being wrong on the facts. Docstring: the header said this file "asserts the declared floor, not the resolved version", which the two tests added in this PR do exactly the opposite of. Rewritten around the two layers the file actually has: declaration (what we control) and runtime (what we care about), with why each covers what the other cannot. Coverage, from the reviewer's own mutation testing: - The download-path check asserted the constant was MENTIONED. oaklib ships the pre-0.7.2 call commented out beside the live one (ensure_from_s3(s3_bucket="bbop-sqlite", ...), labelled "Option 2"), so the cheapest revert is a two-line comment swap leaving the unused import in place — which passed. Now asserts the constant is INTERPOLATED, plus that ensure_from_s3 does not appear in live (non-comment) code. Verified: the comment-swap revert is now caught, as is an inlined host. - S3_HOST was the single spelling "s3.amazonaws.com", so a virtual-hosted region URL such as https://bbop-sqlite.s3.us-east-1.amazonaws.com — the same bucket — passed both assertions. Widened to the ".amazonaws.com" domain. - OVERRIDE_ENV was hardcoded with nothing checking oaklib still reads it. If upstream renamed it, delenv would become a no-op and the test would silently assert the ambient environment instead of the shipped default, which is the one thing that helper exists to prevent. Now asserts the name appears in oaklib's constants source. Two claims in the previous commit message and PR body were wrong, and the decision they defended was right for a different reason: - "TraitMech has no tomli" is false. tomli IS in uv.lock, installed on <3.11 as a transitive dependency of pytest and coverage, so CultureMech's guarded try/except import would have worked here. The accurate claim is that it is not a DECLARED dependency, and leaning on a transitive test-only package to read our own manifest is fragile. - "that exact import already broke the 3.10 lane" is false. What broke d103043 was a bare `import tomllib` with no fallback, not the guarded form. The real justification is the one the code already gives: consistency with test_python_support.py. Also: dropped an unused monkeypatch parameter, hoisted `import inspect` to module level, and noted in the helper that the reload leaves oaklib.constants un-overridden for the session (inert — nothing here reads it under pytest, and oaklib's downloader binds its own copy). Filed rather than fixed, being pre-existing from #565: #572 — CDN_DEFAULT_FIRST_RELEASE is 0.7.2 while the declared floor is 0.7.3, so the guard would accept a floor #565 argued is uninstallable. Refs #568 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
realmarcin
added a commit
that referenced
this pull request
Aug 29, 2026
Sixth instance of one defect: a `paths:` filter narrower than what the job reads. #184, #200, #250, #252, #554 — and now this. Every one was caught by human review, never by CI. Measured before changing anything: six PR-triggered workflows install dependencies with uv and filter on paths. NONE listed uv.lock (#567), and three did not list pyproject.toml either (#566): canonical-example-taxonomy both missing curation-history both missing label-correspondence both missing pytest / qc / validate-strict uv.lock missing label-correspondence is the one that bit: it is the blocking gate that resolves `sqlite:obo:` through OAK, so a PR changing which host OAK downloads from is exactly the PR it exists to check — and #565, which changed precisely that, could not trigger it. It had to be dispatched by hand to get any CI evidence at all. Not touched, with reasons: claude-code-review.yml has its pull_request trigger deliberately removed (quota, 2026-08-13); pr-sanity and vendored-sync use a bare trigger and already run on every PR; pr-checks-present and pr-shepherd have no pull_request trigger; and vendored-sync installs nothing with uv. tests/test_workflow_dependency_paths.py closes the class rather than this instance. `audit-qc-paths` derives qc.yaml's read-set from its recipe chain; this asserts the dependency half for every workflow: if a job runs `uv sync`, the resolved environment is one of its inputs, so both files belong in the filter that decides whether it runs. A second test keeps the push and pull_request filters agreeing, since the workflows that repeat their list by hand can drift where the anchored ones cannot. Scoped deliberately: only filtered, PR-triggered, uv-installing jobs. A job with no `paths:` already runs on everything, and one that never touches uv has no dependency input to miss. The discovery loop asserts it found at least one such job, so the test cannot pass vacuously. Mutation-checked: removing `uv.lock` from qc.yaml fails the guard with "qc.yaml omits uv.lock". Restored, it passes. Closes #566 Closes #567 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
realmarcin
added a commit
that referenced
this pull request
Aug 29, 2026
* Guard the oaklib semsql URL at runtime, not just by version (#568) #568 asked for three assertions ported from CultureMech's test_oaklib_semsql_pin.py. #565 shipped two of them — the declared floor and the lock agreeing with it — and #568 was auto-closed two seconds after that merge, so the third never landed. It is the one the issue called out as mattering most: the version number is what we control, the resolved constant is what we care about. Adds two runtime assertions: - the shipped SEMSQL_SQLITE_URL_BASE contains no s3.amazonaws.com and is https. This keeps working if upstream renames the constant, changes its default again, or ships a version whose number implies a fix it does not contain. - oaklib's sqlite implementation still READS that constant, and does not mention the S3 host directly. A correct constant is worthless if the downloader stopped using it: oaklib builds the URL as f"{SEMSQL_SQLITE_URL_BASE}/{prefix}.db.gz", and an inlined host there would leave every version assertion passing while sqlite:obo: went back to the retiring bucket. The env-override caveat #568 raised is handled deliberately rather than by accident. SEMSQL_SQLITE_URL_BASE is os.environ.get(OVERRIDE, CDN) read at import time, so on a machine that sets a mirror the naive test asserts the override and a reverted default passes unnoticed behind it. The test clears OAKLIB_SEMSQL_SQLITE_URL_BASE and reloads the module, so it always asserts what oaklib ships. Verified: with the override set to the S3 bucket, the suite still passes and still reads the CDN default. Each assertion was mutation-checked rather than assumed: a reverted default is caught, an inlined host is caught, and the override does not mask either. Adapted rather than transcribed, because two things in CultureMech's copy are false here. It reads pyproject with tomllib and falls back to tomli; TraitMech has no tomli, and tomllib is stdlib only from 3.11 — that exact import already broke the 3.10 lane once in #565, so this stays on the regex parse test_python_support.py uses. And its comment justifies the floor with "this project supports <3.15 and runs 3.14 in CI", which is CultureMech's matrix, not ours (3.10-3.13; see #570). Extends the existing tests/test_oaklib_floor.py instead of adding a second file, so the four assertions live together rather than split across two names for one property. Closes #568 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Harden the runtime guard, fix the docstring, and correct two false claims Adversarial review found the module docstring still described only half the file, one revert shape slipping through, and — twice — my own justification being wrong on the facts. Docstring: the header said this file "asserts the declared floor, not the resolved version", which the two tests added in this PR do exactly the opposite of. Rewritten around the two layers the file actually has: declaration (what we control) and runtime (what we care about), with why each covers what the other cannot. Coverage, from the reviewer's own mutation testing: - The download-path check asserted the constant was MENTIONED. oaklib ships the pre-0.7.2 call commented out beside the live one (ensure_from_s3(s3_bucket="bbop-sqlite", ...), labelled "Option 2"), so the cheapest revert is a two-line comment swap leaving the unused import in place — which passed. Now asserts the constant is INTERPOLATED, plus that ensure_from_s3 does not appear in live (non-comment) code. Verified: the comment-swap revert is now caught, as is an inlined host. - S3_HOST was the single spelling "s3.amazonaws.com", so a virtual-hosted region URL such as https://bbop-sqlite.s3.us-east-1.amazonaws.com — the same bucket — passed both assertions. Widened to the ".amazonaws.com" domain. - OVERRIDE_ENV was hardcoded with nothing checking oaklib still reads it. If upstream renamed it, delenv would become a no-op and the test would silently assert the ambient environment instead of the shipped default, which is the one thing that helper exists to prevent. Now asserts the name appears in oaklib's constants source. Two claims in the previous commit message and PR body were wrong, and the decision they defended was right for a different reason: - "TraitMech has no tomli" is false. tomli IS in uv.lock, installed on <3.11 as a transitive dependency of pytest and coverage, so CultureMech's guarded try/except import would have worked here. The accurate claim is that it is not a DECLARED dependency, and leaning on a transitive test-only package to read our own manifest is fragile. - "that exact import already broke the 3.10 lane" is false. What broke d103043 was a bare `import tomllib` with no fallback, not the guarded form. The real justification is the one the code already gives: consistency with test_python_support.py. Also: dropped an unused monkeypatch parameter, hoisted `import inspect` to module level, and noted in the helper that the reload leaves oaklib.constants un-overridden for the session (inert — nothing here reads it under pytest, and oaklib's downloader binds its own copy). Filed rather than fixed, being pre-existing from #565: #572 — CDN_DEFAULT_FIRST_RELEASE is 0.7.2 while the declared floor is 0.7.3, so the guard would accept a floor #565 argued is uninstallable. Refs #568 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Restore the assertion my fix deleted, and parse instead of grep Review of 8cc9712 found that commit — titled "correct two false claims" — had itself introduced a third. Three fixes. #575, the serious one. 93443c3 asserted the download path never names the S3 host; 8cc9712 DELETED that assertion while its message described the change as "widened to the .amazonaws.com domain". One assertion was widened; the other was dropped, and the PR body still advertised it. The cost is concrete: keeping the interpolated line and ADDING an inlined S3 URL below it leaves the constant present but dead, and passed every remaining assertion. Restored, and the additive mutation is caught again. #576. Both source checks scanned raw text, so neither could tell live code from comments: - The OVERRIDE_ENV guard was a substring test, and oaklib's constants.py names the variable twice — once live, once in a prose comment above it. Renaming only the live read left the comment satisfying the guard, so delenv silently became a no-op and the test would assert the ambient environment instead of the shipped default. That is exactly the failure the assertion was added to prevent. - The ensure_from_s3 filter treated every non-# line as live, so an upstream docstring merely MENTIONING the call would fail the lane — a behaviour-free change turning CI red, which is how a guard gets deleted. Both now walk ast.parse: comments and string contents disappear, and a getattr-assembled call that previously slipped through is caught. #577. The interpolation check is deliberately kept — it is the only thing catching the comment-swap revert — but its failure message now names the likely benign cause (urljoin, an intermediate variable) and says to verify the resolved URL before relaxing anything, rather than asserting sabotage. Mutation-verified, all five: real source passes; additive S3 URL caught; getattr call caught; comment-swap revert caught; docstring mention does NOT fail. My first harness reported two false results because its "unparseable" sentinel was a truthy string and the mutations had wrong indentation, so a SyntaxError read as "passes" — rewritten to let parse errors raise. Closes #575 Closes #576 Closes #577 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #562.
Closes #568.
Closes #569.
INCATools is retiring public access to the raw semantic-sql S3 bucket (semantic-sql#112). Below oaklib 0.7.2,
sqlite:obo:selectors resolve againsthttps://s3.amazonaws.com/bbop-sqlite; at 0.7.2+ they useSEMSQL_SQLITE_URL_BASE(defaulthttps://semanticsql.berkeleybop.io), withOAKLIB_SEMSQL_SQLITE_URL_BASEas a mirror escape hatch.This repo sat at 0.6.23 — on the raw-S3 path. The exposure is not theoretical:
just validate-productsis a blocking CI gate that pulls CHEBI/GO/ENVO/PATO/RO through OAK. It would have gone red when the bucket lostAllUsers.Change
pyproject.toml:oaklib>=0.5.0→oaklib>=0.7.3, andlinkml>=1.10.0→linkml>=1.11.1, each with its reason recorded inline so the floor is not lowered again by accident.requires_python <3.14while this project'srequires-pythonis an unbounded>=3.10, so a 0.7.2 floor would name a minimum that cannot be installed on a supported interpreter. 0.7.3 widened it to<3.15. Verified that 0.7.3 still carries the CDN default (SEMSQL_SQLITE_URL_BASE = https://semanticsql.berkeleybop.io), so the higher floor does not permit an install that reintroduces the bug.uv.lock: oaklib 0.6.23 → 0.7.4.Verified, not assumed
SEMSQL_SQLITE_URL_BASE(ImportError) — the issue's mechanism, confirmedSEMSQL_SQLITE_URL_BASE = https://semanticsql.berkeleybop.ioget_adapter("sqlite:obo:ro")fetched over the network →RO:0002327 → "enables"just validate-products(blocking, OAK-backed)just qc/validate-history/validate-strict>=3.10,<3.15; covers 3.10–3.13setuptools<82pin (eutils/pkg_resources)The linkml side effect, and why it is unavoidable
The resolve carries linkml 1.10.0 → 1.11.1 plus transitive moves. This is forced, and the mechanism is exact: linkml 1.10.0 requires
click<8.2, while oaklib 0.7.2+ hard-requiresllm, which pullssqlite-utils, which requiresclick>=8.3.1. Direct contradiction — sooaklib>=0.7.2andlinkml==1.10.0cannot coexist. That is why the oldlinkml>=1.10.0floor named a combination that no longer resolves, and why it is raised here.Risk was tested rather than argued:
validate-strict(0 ERROR rows, tracked report byte-identical),gen-pydantic, and the full suite all pass; deprecation warnings drop from 25 to 2.Also worth knowing:
openaibecomes a runtime dependency on the 3.10/3.11 lanes (llm requires it unconditionally), where it previously arrived only via thepython>=3.12dev extra. This repo imports none of the added packages directly.Review findings filed, not fixed here
Adversarial review surfaced two pre-existing CI-coverage gaps that this PR exposes rather than causes:
label-correspondence's paths filter excludespyproject.toml/uv.lock, so the blocking OAK gate never ran on this PR. I dispatched it manually instead; it passed. Same defect family as vendored-sync never fires for 5 of the 6 files it checks (paths filter is narrower than the drift list) #184/A PR touching only docs/ or a new workflow file gets zero CI checks #200/qc and pytest do not trigger on research/**, so the new artifact gate cannot fire on a research-only PR #250/Nothing checks that qc.yaml's paths filter covers what the qc chain reads — third recurrence #252/pytest paths filter omits data/traits/**, so corpus tests never run on data-only PRs — 13 records drifted off the round-trip invariant #554.uv.lockis in no workflow paths filter at all, so a lock-only dependency change triggers almost nothing.Sibling repos CultureMech and MediaIngredientMech pin 0.6.23 identically and need the same bump.
🤖 Generated with Claude Code