Tash Space Data Preprocessor (TSDP)

TSDP 3.2.3 is a Windows desktop application for downloading and processing astronomical FITS data. It supports multi-HDU inputs, WCS-aware mosaics, normalization, alignment, previews, recovery, and optional Telegram control.

Recommended installation

Run TSDP-Setup-3.2.3.exe. The installer uses the current Windows account, supports installing over a previous version, creates Start menu shortcuts, and can optionally create a desktop shortcut. Python is not required for the installed application.

A portable ZIP is also provided. Extract the entire folder before running TSDP.exe; do not run the executable from inside the ZIP.

On the first launch, the setup assistant checks folders, dependencies, available memory and disk space, aria2, and optional integrations. The same checks remain available from Check Setup and from Settings. The bundled aria2c.exe enables managed downloads out of the box.

Running from source

Use Python 3.10 or newer:

cd C:\Users\egeme\Downloads\tsdp
python -m pip install -e ".[all,dev]"
python TSDP2.py

The normal workflow is Download & Process -> Mosaic -> Align & Composite. Recovery can resume or repair interrupted disk-backed mosaic work. Large reprojections use float32 memmaps, resource-adaptive workers, disk preflight checks, chunked calculations, and cancellation checks rather than forcing complete detector frames into RAM.

Resource planning

Before a mosaic starts, TSDP shows estimated peak working memory, final-product size, scratch-disk use, and free space. Disk-backed Turbo worker counts are not limited to three: they adapt to the requested value, tile count, CPU threads, geometry, and currently available RAM.

Output folders

Every organized processing run receives an isolated, collision-safe job folder. Directories are created only when needed:

Job_Name/
|-- final/{aligned,mosaics,normalized}/<filter>/
|-- previews/{png,jpeg}/
|-- diagnostics/{footprints,coverage_maps,masks,seam_maps,weight_maps,rejected_frames}/
|-- intermediate/{reprojected,cropped,normalized,temporary_fits,memmaps}/
|-- recovery/{manifests,checkpoints}/
|-- metadata/{logs,reports,file_lists}/
`-- job_manifest.json

Final FITS products are never removed with intermediates. Existing flat folders are not moved, renamed, or deleted; a newly numbered job folder is created by default. The manifest uses relative product paths so jobs can be moved safely.

Telegram and security

Configure the chat ID and notification choices in Settings. Tokens are read from Windows Credential Manager through keyring, or from JWST_SUITE_TELEGRAM_TOKEN. Legacy plaintext tokens are migrated out of the JSON settings file atomically. Token values and token-bearing Telegram URLs are redacted from logs and excluded from job manifests.

The existing command set is preserved, including guided /newjob, /status, /rerun, /stop, reference selection, output/previews/FITS/log delivery, ZIP delivery, CSV upload, blink GIFs, cleanup, and /flush.

Architecture

• tsdp2/astronomy: FITS, WCS, normalization, coverage, mosaic, and seam algorithms
• tsdp2/downloads: aria2, queue, MAST import, and post-download services
• tsdp2/telegram: client, parser, listener, orchestration, and output delivery
• tsdp2/recovery: manifests, cleanup, repair, and recovery UI
• tsdp2/ui: focused tabs, dialogs, onboarding, installation check, and About
• tsdp2/utilities: resource planning, release checks, and Tk dispatch
• tsdp2/product.py, config.py, output_paths.py, logging_config.py, and models.py: shared infrastructure

TSDP2.py is a small launcher. Runtime monkey-patching is not used.

Tests and release build

Tests use synthetic offline data:

python -m pytest -q

Build the portable app and installer with:

powershell -ExecutionPolicy Bypass -File packaging\build_release.ps1

Release artifacts and SHA-256 hashes are written to release/. The release build also supports a non-GUI packaged FITS/WCS/reprojection smoke check.

Bundled software

The Windows distribution bundles aria2 1.37.0 and its COPYING file. aria2 remains covered by its own license. TSDP's Python astronomy and UI dependencies are included in the packaged application.

Optical filters and automatic mosaic resources

DECam optical bands (u, g, r, i, z, Y), descriptive FITS filter headers, and names such as g.fits, field_r_001.fits, and z.fits.fz are recognized. Headers take precedence over filenames, and custom FILTER names are retained. Truly unlabelled inputs remain separate UNKNOWN groups and can be assigned with the Mosaic tab's group rename/move controls.

The Mosaic tab enables Auto (safe RAM) by default. It selects per-group worker limits from CPU capacity and currently available RAM, leaves two CPU threads free, and budgets at most 60% of free memory for processing. Disable Auto to use the manual thread limit; memory limits still apply. Full-resolution mosaics use float32 blocks and reusable feather-blend workspaces. GPU acceleration is not used.

Coverage preview scale and selection

The WCS coverage window includes an output scale control and live output dimensions for each selected filter. Resize, move, or rotate an area to update its width x height in output pixels. Scale changes are synchronized with the Mosaic tab; per-filter scale overrides continue to apply and each readout shows its effective scale.

SkyView cutouts use a bounded HTTPS FITS request. If the service is unavailable, a celestial WCS grid keeps selection and size estimation usable offline. A ten-percent coverage overlay is composited once per filter, so repeated frames do not hide the background.