Skip to content

Standardise the test setup across repos #10

Closed

Description

@nicoloesch

Tests across the stack mark dialects and select CI jobs in different ways. This issue defines one setup, adds the missing workflows and writes the guide that every repo follows.

Target design

  • Routing marks name the infrastructure a test needs and decide which CI job runs it. In most repos they are the dialect marks sqlite, duckdb and postgresql. A repo whose infrastructure depends on something else uses other routing marks, such as omop-emb's backend marks.
  • An unmarked test needs no database.
  • A test carries the routing mark of every job it should run in, and must pass in each. A test that depends on one backend carries one mark. A test meant for several backends is parametrized, with one mark per param.
  • Adding a routing mark requires adding its job. Otherwise those tests never run.
  • Every test gets its database through the isolated_test_database fixtures. No test opens its own engine or connection.
  • Every fixture that calls isolated_test_database passes request and names the routing mark it requires, which defaults to the dialect. A missing mark fails at setup.
  • A test whose database is not configured fails, and the message names the field. There is no skip. A machine without a database deselects its tests by routing mark.
  • A test that needs several infrastructures carries every routing mark. The single-dialect jobs exclude it with and not.
  • Tests do not skip for a missing optional dependency or tool. CI installs every extra, so a missing one fails.
  • Tests that must pass on every dialect use DIALECT_PARAMS, which marks each param.
  • Use the Dialect enum, never string literals.
  • Use pytestmark when every test in a module needs the marks, and a decorator for a single test.
  • Set --strict-markers in every repo.
  • db_dialect is removed. Process isolation is replaced by a fix in the schema creation code (see Dependencies).

CI jobs

  • Add build-test-nodb.yml, which runs the tests that carry no routing mark. An input lists the routing marks to exclude and defaults to sqlite or duckdb or postgresql.
  • Add build-test-sqlite-v2.yml, which runs -m sqlite by default.
  • Add a markers input to build-test-sqlite-v2.yml and build-test-postgres-v3.yml. build-test-duckdb.yml already has one.
  • Every repo's ci.yml carries the nodb job and a job for each routing mark its tests use. A job that collects no tests fails, so a repo only calls the workflows for marks it has tests for.
  • Keep the caller job ids build-test-sqlite, build-test-duckdb and build-test-postgres. Add build-test-nodb.

Guide

  • Add docs/guides/testing-conventions.md, linked from the mkdocs.yml nav under Guides and from the build-test-* workflow pages.
  • Include a minimal example per dialect and the matching ci.yml job.
  • Include the pattern for session-scoped databases with require_routing_mark, the missing-database behaviour and the warning about mixing dialects in one process.
  • Include the ruff TID251 config that bans inline engines in tests.
  • The first draft follows the oa-configurator release and is revised as the repos migrate.

Dependencies

  • oa-configurator: preserve_create_rules, which keeps create_all and drop_all from altering shared metadata, plus removing db_dialect and the pg_db auto-marking, and failing instead of skipping when a database is not configured.
  • orm-loader: use the helper in bootstrap.py.
  • Each repo: align its tests with the guide.

Tracking

Done when the workflows and guide are merged. Each repo's own issue tracks its migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions