You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds a Python custom data example for the public USD FXMacroData endpoints.
The example includes:
FXMacroDataMacroIndicator for /v1/announcements/usd/{indicator}
FXMacroDataReleaseCalendar for /v1/calendar/usd
announcement_datetime as the point-in-time event timestamp for both data types
date retained only as the economic reference period
release calendar metadata including release, name, market_tier, and source fields
pagination using limit=100, offset, and pagination.has_more
public no-key downloads through QCAlgorithm.download
merged paginated responses stored through LEAN's Object Store
committed fixtures for deterministic regression coverage
parser unit tests for both committed fixtures
The regression path uses committed fixtures because anonymous USD history is limited to the most recent 90 days and delayed by 15 minutes. The public no-key path can also be exercised with the use-public-source=true parameter.
LEAN has general Python custom data examples, but no example demonstrating how to consume FXMacroData macroeconomic announcements and USD release-calendar data.
This example also preserves the API's point-in-time semantics by using announcement_datetime as the LEAN event time instead of date, which represents the economic reference period.
The contribution is intentionally limited to the public USD/no-key endpoints so it can be run without credentials.
Requires Documentation Change
No documentation changes are required. This PR adds an in-repository Python example, regression coverage, and parser tests.
How Has This Been Tested?
Tested locally on Windows with the LEAN Release configuration.
Public USD/no-key path was smoke-tested with use-public-source=true
git diff --cached --check passes
The local Python.NET environment emits a GIL shutdown exception after Analysis Complete; the same shutdown exception was reproduced with the existing BybitCustomDataCryptoRegressionAlgorithm, so it is not specific to these changes.
Types of changes
Bug fix (non-breaking change which fixes an issue)
Refactor (non-breaking change which improves implementation)
Performance (non-breaking change which improves performance. Please add associated performance test and results)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Read the diff (not run). Two things in the pagination and reader path.
_collect_paginated_rows pages with limit/offset over a list that grows. The public USD endpoints add rows when a release lands, and the example builds the Object Store file by calling initialize() once and downloading the pages one after another. If a release is published between page 1 and page 2, the offset shifts by one and the last row of page 1 comes back again as the first row of page 2, so the merged file holds a duplicate with the same announcement_id (or calendar_event_id). The reader does not drop it, so history() returns two points at the same timestamp, and any strategy that counts releases or takes the latest value per timestamp sees it. A cheap guard is to de-duplicate on announcement_id / calendar_event_id when merging (keeping the first), and a test with a fake fetch_page that repeats a row across the page boundary would pin it. _verify_pagination_helper currently only covers clean pages [1,2] and [3].
In FXMacroDataMacroIndicator.reader, float(row["val"]) is outside any guard, while announcement_datetime is None is skipped. If the API ever returns a row with a null val (a scheduled but unreleased indicator, say), one row raises and the whole collection for that date is lost, because the reader builds the list inside one call. Skipping rows with a missing val the same way as missing timestamps, or stating in the docstring that the endpoint never returns them, would make the example safer to copy.
Thanks @arhancanli, good catches. I’ve addressed both points:
pagination now de-duplicates merged rows by announcement_id / calendar_event_id, preserving the first occurrence, with coverage for an overlapping page boundary
FXMacroDataMacroIndicator.reader now skips rows with a missing val, consistent with the existing missing-announcement_datetime guard
I also added coverage for the missing-value case and reran the targeted tests and regression successfully.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a Python custom data example for the public USD FXMacroData endpoints.
The example includes:
FXMacroDataMacroIndicatorfor/v1/announcements/usd/{indicator}FXMacroDataReleaseCalendarfor/v1/calendar/usdannouncement_datetimeas the point-in-time event timestamp for both data typesdateretained only as the economic reference periodrelease,name,market_tier, and source fieldslimit=100,offset, andpagination.has_moreQCAlgorithm.downloadThe regression path uses committed fixtures because anonymous USD history is limited to the most recent 90 days and delayed by 15 minutes. The public no-key path can also be exercised with the
use-public-source=trueparameter.Related Issue
Closes #9603
Motivation and Context
LEAN has general Python custom data examples, but no example demonstrating how to consume FXMacroData macroeconomic announcements and USD release-calendar data.
This example also preserves the API's point-in-time semantics by using
announcement_datetimeas the LEAN event time instead ofdate, which represents the economic reference period.The contribution is intentionally limited to the public USD/no-key endpoints so it can be run without credentials.
Requires Documentation Change
No documentation changes are required. This PR adds an in-repository Python example, regression coverage, and parser tests.
How Has This Been Tested?
Tested locally on Windows with the LEAN Release configuration.
python -m py_compile Algorithm.Python/FXMacroDataCustomDataRegressionAlgorithm.pydotnet build Tests/QuantConnect.Tests.csproj -c Releasedotnet test Tests/QuantConnect.Tests.csproj -c Release --no-build --filter "FullyQualifiedName~FXMacroDataCustomDataTests"use-public-source=truegit diff --cached --checkpassesThe local Python.NET environment emits a GIL shutdown exception after
Analysis Complete; the same shutdown exception was reproduced with the existingBybitCustomDataCryptoRegressionAlgorithm, so it is not specific to these changes.Types of changes
Checklist:
bug-<issue#>-<description>orfeature-<issue#>-<description>