catalog.data.gov

Application Repo

a.k.a Catalog is the public-facing dataset discovery and search application for Data.gov. It serves 515,000+ datasets from 120+ federal, state, municipal, university, and tribal publishing organizations. This is the main app of Data.gov and is generally what folks are thinking about when they refer to Data.gov.

This is a custom Python/Flask application that replaced the legacy CKAN-based catalog in 2025. It reads from the shared harvest database managed by datagov-harvester (it does not write to it -- all dataset metadata is written by the harvester) and uses OpenSearch for full-text search.

Environments

Instance	Url
Production	catalog.data.gov
Staging	catalog-stage.data.gov
Development	catalog-dev.data.gov
Legacy catalog (CKAN-based, running through fall 2026)	catalog-old.data.gov

All environments run on cloud.gov, cf space names prod, staging, and development.

Architecture

• Web app: Python/Flask, served via NGINX proxy on cloud.gov
• Database: Shared Postgres instance managed by datagov-harvester (datagov-harvest-db service) -- read-only. The SQLAlchemy models are duplicated locally in app/models.py for isolation; the app interacts with the shared DB through CatalogDBInterface (app/database/interface.py)
• Search: OpenSearch (((app_name))-opensearch cloud.gov service)
• Storage: S3 for sitemaps and static assets
• Monitoring: New Relic
• Logging: Logstack (cloud.gov log drain)

Data flow: datagov-harvester pulls agency metadata and writes it to the shared Postgres DB → datagov-catalog reads from Postgres and keeps an OpenSearch index in sync → the Flask app serves search and browse pages from OpenSearch, and dataset/organization detail pages from Postgres.

Local development

Prerequisites: Docker + Docker Compose, Python 3.x with Poetry, Node.js/npm (static assets + accessibility testing).

cp .env.sample .env  # first-time setup, values can be left as defaults for local dev
make install-static  # install static assets (needs npm)
make up              # start local docker env
make load-test-data  # load fixture data

Then visit the app locally and iterate. Useful commands:

• make test -- full Python test suite
• make test-pa11y -- accessibility (pa11y-ci) tests, requires the app running
• make lint-check / make lint-fix -- ruff, isort, black
• make poetry-update -- keep local Poetry in sync with what CI uses

Install the git pre-commit hooks once per clone (pip install pre-commit && pre-commit install) to run the same formatters on staged files automatically before each commit; CI still runs make lint-check across the whole tree.

See the datagov-catalog README for the full environment variable reference (DB connection, OpenSearch host, New Relic, S3/sitemap credentials, etc).

Search

Search is powered by OpenSearch, kept in sync with Postgres via a daily batch job (flask search sync clears the index and re-indexes every dataset). See docs/opensearch.md for how indexing and cursor-based pagination (search_after/after) work.

Search filters

Dataset search supports filters (keyword, organization, organization type, publisher, geography, spatial data, collection). Each filter is a single FilterDefinition registered in app/search/filters/ that owns its own request parsing, URL serialization, OpenSearch clause building, sidebar rendering, and OpenAPI docs -- adding a new filter means writing one new module and registering it, not touching four different files.

• How To Add Search Filters -- step-by-step guide for contributors
• Search Filters Refactor Walkthrough -- deeper architectural tour with diagrams of the request → OpenSearch → sidebar flow

Performance

See docs/load-test.md for ApacheBench comparisons against the legacy catalog. Summary: search response times are comparable except for very broad queries with many matches (an open area for improvement); page load times are dramatically better across the board (e.g. dataset detail pages saturate at ~120 req/s vs. ~5 req/s on the legacy catalog).

Collections

Some datasets are grouped into collections, where a parent dataset represents a set of related child datasets (e.g. a series of related files from the same agency program). Collections work differently depending on how the child datasets are harvested:

• For WAF sources, datagov-harvester has a dedicated waf-collection source type: each source has a collection_parent_url, and the harvester creates a parent dataset that the child records point back to (see harvester/harvest.py).
• More generally, a dataset's isPartOf field marks it as a member of a collection. The catalog indexes dcat.isPartOf in OpenSearch and exposes a collection search filter (app/search/filters/collection.py) so members of a collection can be queried together.

This replaced the CKAN-era mechanism (collection_package_id, described on the Collections wiki page), which only applies to inventory.data.gov now.

Duplicate datasets

The old CKAN-based system could harvest the same dataset twice under different identifiers, requiring manual cleanup via SQL queries and a standalone datagov-dedupe tool. That failure mode no longer exists in the current pipeline: datagov-harvester's HarvestSource.filter_duplicate_identifiers() (in harvester/harvest.py) checks every incoming record's identifier against the others in the same job before processing. Duplicates are recorded as a harvest_record_error (rather than becoming a dataset) and dropped from the job.

If you're debugging why a dataset didn't show up after a harvest, check for duplicate-identifier errors on the job via the Harvest API rather than looking for duplicate rows in dataset -- see Harvest O&M and Debugging for query examples.

Logs

See New Relic and/or the cloud.gov wiki page for how to review logs.

Automated tasks

GitHub Actions (.github/workflows) run scheduled and on-push jobs against the current catalog:

• Restart -- rolling-restarts datagov-catalog and datagov-catalog-proxy on staging and prod every 15 minutes
• Create Sitemaps -- runs flask sitemap generate and flask sitemap verify as cloud.gov tasks daily (and on demand per environment)
• Snyk Scan -- weekly dependency vulnerability scan across all projects
• Check DB Models -- on every push/PR, diffs app/models.py and shared/constants.py against the copies in datagov-harvester to catch drift between the two repos' duplicated schema

See Automated O&M Tasks for the equivalent automation on the legacy catalog (catalog-old.data.gov) and inventory.data.gov, which still use CKAN-era GitHub Actions.

Deployment

Deployments run automatically via GitHub Actions (deploy.yml) on push to main:

1. Lint -- ruff
2. Deploy to staging -- deploys to the staging cloud.gov space, runs a smoke test
3. Deploy to prod -- deploys to the prod cloud.gov space after staging succeeds, runs a smoke test
4. Lighthouse CI audit -- runs against prod after deploy

For emergency deployments outside of the normal CI/CD pipeline, see Break Glass deployment.

Related resources

• datagov-catalog README -- local dev setup, environment variables, architecture
• harvest.data.gov -- harvest pipeline UI
• datagov-harvester -- harvester source code and shared DB owner
• Updating organization logos on Catalog

Catalog UI Filter Analysis

Occasionally (every ~60-90 days) we want to understand usage of filters in the catalog UI as an audit to determine their usefulness and propose changes (add, remove, rename, etc...) to those filters to better suit the needs of our users. here's filter analysis README which can be used to guide the analysis.

---

This page previously described the legacy CKAN-based catalog (SOLR, ckan-php-manager, apache2/Ansible ops). That system is now catalog-old.data.gov; this page covers the current Flask/OpenSearch app. Updated 2026-07-23.