Skip to content

Add FXMacroData custom data example - #9876

Open
matvt-cell wants to merge 2 commits into
QuantConnect:masterfrom
matvt-cell:feature-9603-fxmacrodata-custom-data
Open

matvt-cell wants to merge 2 commits into
QuantConnect:masterfrom
matvt-cell:feature-9603-fxmacrodata-custom-data

Conversation

@matvt-cell

@matvt-cell matvt-cell commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds a Python custom data example for the public USD FXMacroData endpoints.

The example includes:

  • FXMacroDataMacroIndicator for /v1/announcements/usd/{indicator}
  • FXMacroDataReleaseCalendar for /v1/calendar/usd
  • announcement_datetime as the point-in-time event timestamp for both data types
  • date retained only as the economic reference period
  • release calendar metadata including release, name, market_tier, and source fields
  • pagination using limit=100, offset, and pagination.has_more
  • public no-key downloads through QCAlgorithm.download
  • merged paginated responses stored through LEAN's Object Store
  • committed fixtures for deterministic regression coverage
  • parser unit tests for both committed fixtures

The regression path uses committed fixtures because anonymous USD history is limited to the most recent 90 days and delayed by 15 minutes. The public no-key path can also be exercised with the use-public-source=true parameter.

Related Issue

Closes #9603

Motivation and Context

LEAN has general Python custom data examples, but no example demonstrating how to consume FXMacroData macroeconomic announcements and USD release-calendar data.

This example also preserves the API's point-in-time semantics by using announcement_datetime as the LEAN event time instead of date, which represents the economic reference period.

The contribution is intentionally limited to the public USD/no-key endpoints so it can be run without credentials.

Requires Documentation Change

No documentation changes are required. This PR adds an in-repository Python example, regression coverage, and parser tests.

How Has This Been Tested?

Tested locally on Windows with the LEAN Release configuration.

  • python -m py_compile Algorithm.Python/FXMacroDataCustomDataRegressionAlgorithm.py
  • dotnet build Tests/QuantConnect.Tests.csproj -c Release
  • dotnet test Tests/QuantConnect.Tests.csproj -c Release --no-build --filter "FullyQualifiedName~FXMacroDataCustomDataTests"
    • 2 tests passed
    • 0 tests failed
  • Fixture-backed regression algorithm completed successfully
  • Public USD/no-key path was smoke-tested with use-public-source=true
  • git diff --cached --check passes

The local Python.NET environment emits a GIL shutdown exception after Analysis Complete; the same shutdown exception was reproduced with the existing BybitCustomDataCryptoRegressionAlgorithm, so it is not specific to these changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which improves implementation)
  • Performance (non-breaking change which improves performance. Please add associated performance test and results)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Non-functional change (xml comments/documentation/etc)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

@arhancanli

Copy link
Copy Markdown

Read the diff (not run). Two things in the pagination and reader path.

  1. _collect_paginated_rows pages with limit/offset over a list that grows. The public USD endpoints add rows when a release lands, and the example builds the Object Store file by calling initialize() once and downloading the pages one after another. If a release is published between page 1 and page 2, the offset shifts by one and the last row of page 1 comes back again as the first row of page 2, so the merged file holds a duplicate with the same announcement_id (or calendar_event_id). The reader does not drop it, so history() returns two points at the same timestamp, and any strategy that counts releases or takes the latest value per timestamp sees it. A cheap guard is to de-duplicate on announcement_id / calendar_event_id when merging (keeping the first), and a test with a fake fetch_page that repeats a row across the page boundary would pin it. _verify_pagination_helper currently only covers clean pages [1,2] and [3].

  2. In FXMacroDataMacroIndicator.reader, float(row["val"]) is outside any guard, while announcement_datetime is None is skipped. If the API ever returns a row with a null val (a scheduled but unreleased indicator, say), one row raises and the whole collection for that date is lost, because the reader builds the list inside one call. Skipping rows with a missing val the same way as missing timestamps, or stating in the docstring that the endpoint never returns them, would make the example safer to copy.

@matvt-cell

Copy link
Copy Markdown
Contributor Author

Thanks @arhancanli, good catches. I’ve addressed both points:

  • pagination now de-duplicates merged rows by announcement_id / calendar_event_id, preserving the first occurrence, with coverage for an overlapping page boundary
  • FXMacroDataMacroIndicator.reader now skips rows with a missing val, consistent with the existing missing-announcement_datetime guard
    I also added coverage for the missing-value case and reran the targeted tests and regression successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add FXMacroData macro custom data example

2 participants