Skip to content

Restrict runinfo to the run lists of a dataset; DataSet livetimes and time selections - #246

Merged
theHenks merged 4 commits into
mainfrom
runinfo-dataset-categories
Nov 7, 2026
Merged

theHenks merged 4 commits into
mainfrom
runinfo-dataset-categories

Conversation

@theHenks

@theHenks theHenks commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor
  • I have reviewed all the code in this pull request and deem it of the standard expected for this project.

Summary

runinfo of a dataset

  • For a dataset other than :default, runinfo holds what the run lists of the dataset in datasets/runlists name: one column per listed category, and an entry of a category only for the runs listed for it. Any other entry has missing start and end keys, no DAQ cycle keys, a NaN livetime and is_analysis_run = false. Selecting a category that the dataset does not have throws an ArgumentError. :default keeps every run and category of datasets/runinfo.
  • get_exposure skips the runs without a start key of the category, and with is_analysis_run = true (the default) the runs that are no analysis runs.
  • read_ldata of a period, partition or run table skips the runs whose entry of the category has no DAQ cycle keys, also with search_disk = true, and throws if no run is left. Reading a single run is unchanged.
  • find_filekey(data, ts) searches the DAQ cycle keys of every run in the metadata, so a timestamp in a cycle outside the dataset resolves to its own cycle.

Conveniences

  • filter(f, ds::DataSet) returns a DataSet, e.g. filter(!in(bad_filekeys(l200)), DataSet(l200, runinfo(l200).phy)). The display of a set built from runinfo shows per category the summed runinfo livetime of the runs with keys in the set, and the runtime from the first to the last key.
  • find_filekey.(data, timestamps) broadcasts over timestamps.
  • A PropsDB of a LegendData takes a Unitful.Time as validity selection, e.g. l200.par.ppars.aoe(t), and selects the file key of the DAQ cycle that contains t.
  • read_ldata(data, tier, t) and read_ldata(data, tier, t, det) with a single Unitful.Time read as [t].
  • read_ldata(data, tier, ds::DataSet) and read_ldata(data, tier, ds, det) read all file keys of the set.

Behavior changes

  • Under a named dataset, a run listed only for some of its categories has no keys, livetime or analysis flag in the others, and categories the run lists do not name are no columns.
  • get_exposure with the default keywords counts only analysis runs. Under :default, runs not in the valid run list no longer count.
  • is_lrun and is_analysis_run throw for a category that the dataset does not have.

Since results change, I suggest releasing this as 0.7.0. p_process_skm_phy in JuLeAna needs to skip runs without a phy start key before JuLeAna moves to this version.

Tests

New testsets "runinfo of a dataset", "DataSet" and "selection by a time" (test_dataprod_config.jl), "Exposure of the runs of a dataset" (test_exposure.jl), and "runs of a dataset" plus single-timestamp reads (test_ext_legendhdf5io.jl). Pkg.test() passes locally (732 passed, 1 broken as on main).

Parts of this pull request, including code, tests and this description, were written with the help of generative AI.

For a dataset other than `:default`, `runinfo` has a column only for the
categories that the run lists of the dataset in datasets/runlists name,
and an entry of a category only for the runs listed for it. Any other
entry has missing start and end keys, no DAQ cycle keys, a `NaN`
livetime and `is_analysis_run = false`. Selecting a category that the
dataset does not have throws an ArgumentError. `:default` keeps every
run and category of datasets/runinfo.

The functions built on `runinfo` follow it:
- `get_exposure` skips the runs without a start key of the category,
  and with `is_analysis_run = true` (the default) the runs that are no
  analysis runs. Under `:default` these are the runs that the `valid`
  run list does not name.
- `read_ldata` of a period, partition or run table skips the runs whose
  entry of the category has no DAQ cycle keys, also with `search_disk`,
  and throws if no run is left.
- `find_filekey(data, ts)` searches the cycle keys of every run of the
  metadata, so a timestamp in a cycle outside the dataset resolves to
  its own cycle.
- `filter(f, ds::DataSet)` returns a `DataSet` of the same name. A set
  built from `runinfo` holds the runinfo livetime of each of its runs
  and categories; its display shows per category the sum of these
  livetimes for the runs with keys in the set, and the runtime from
  the first to the last key of the set.
- `LegendData` broadcasts as a scalar, so `find_filekey.(data, ts)`
  finds the key of each timestamp.
- A `PropsDB` of a `LegendData`, e.g. `l200.par.ppars.aoe`, takes a
  `Unitful.Time` as validity selection and selects the file key of the
  DAQ cycle that contains it.
- `read_ldata(data, tier, t)` and `read_ldata(data, tier, t, det)` with
  a single `Unitful.Time` read as `[t]`.
`read_ldata(data, tier, ds::DataSet)` and `read_ldata(data, tier, ds,
det)` read all file keys of the set, as for any vector of file keys.
@codecov

codecov Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.72727% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.75%. Comparing base (d854b60) to head (15606a7).

Files with missing lines Patch % Lines
ext/LegendDataManagementLegendHDF5IOExt.jl 77.77% 2 Missing ⚠️
src/props_db.jl 85.71% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #246      +/-   ##
==========================================
+ Coverage   63.25%   63.75%   +0.50%     
==========================================
  Files          27       27              
  Lines        2814     2842      +28     
==========================================
+ Hits         1780     1812      +32     
+ Misses       1034     1030       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

legenddata.md: the `dataset` keyword names a run list; `runinfo` of a
dataset other than `:default` has the listed categories and runs only.

channel_run_info.md: runinfo of a dataset, the error for a category the
dataset does not have, analysis runs, `DataSet` construction from
`runinfo`, `filter` and the `livetime`/`runtime` display, and
`find_filekey` over every run, also broadcast over timestamps.

exposure.md: `get_exposure` skips runs without a start key and, by
default, runs that are not analysis runs; `is_analysis_run` keyword.

read_data.md: period, partition and run tables read only runs that list
DAQ cycle keys for the category, `read_ldata` with a `DataSet` and with
a single `Unitful.Time`.

metadata_validity.md: a `PropsDB` of a `LegendData` is called with a
`Unitful.Time`; `AnyProps` throws for a time.
@theHenks
theHenks merged commit 1ebfeb3 into main Nov 7, 2026
12 checks passed
@theHenks
theHenks deleted the runinfo-dataset-categories branch November 7, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant