Repository navigation
Restrict runinfo to the run lists of a dataset; DataSet livetimes and time selections - #246
Merged
Merged
Conversation
For a dataset other than `:default`, `runinfo` has a column only for the categories that the run lists of the dataset in datasets/runlists name, and an entry of a category only for the runs listed for it. Any other entry has missing start and end keys, no DAQ cycle keys, a `NaN` livetime and `is_analysis_run = false`. Selecting a category that the dataset does not have throws an ArgumentError. `:default` keeps every run and category of datasets/runinfo. The functions built on `runinfo` follow it: - `get_exposure` skips the runs without a start key of the category, and with `is_analysis_run = true` (the default) the runs that are no analysis runs. Under `:default` these are the runs that the `valid` run list does not name. - `read_ldata` of a period, partition or run table skips the runs whose entry of the category has no DAQ cycle keys, also with `search_disk`, and throws if no run is left. - `find_filekey(data, ts)` searches the cycle keys of every run of the metadata, so a timestamp in a cycle outside the dataset resolves to its own cycle.
- `filter(f, ds::DataSet)` returns a `DataSet` of the same name. A set built from `runinfo` holds the runinfo livetime of each of its runs and categories; its display shows per category the sum of these livetimes for the runs with keys in the set, and the runtime from the first to the last key of the set. - `LegendData` broadcasts as a scalar, so `find_filekey.(data, ts)` finds the key of each timestamp. - A `PropsDB` of a `LegendData`, e.g. `l200.par.ppars.aoe`, takes a `Unitful.Time` as validity selection and selects the file key of the DAQ cycle that contains it. - `read_ldata(data, tier, t)` and `read_ldata(data, tier, t, det)` with a single `Unitful.Time` read as `[t]`.
`read_ldata(data, tier, ds::DataSet)` and `read_ldata(data, tier, ds, det)` read all file keys of the set, as for any vector of file keys.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #246 +/- ##
==========================================
+ Coverage 63.25% 63.75% +0.50%
==========================================
Files 27 27
Lines 2814 2842 +28
==========================================
+ Hits 1780 1812 +32
+ Misses 1034 1030 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 task done
legenddata.md: the `dataset` keyword names a run list; `runinfo` of a dataset other than `:default` has the listed categories and runs only. channel_run_info.md: runinfo of a dataset, the error for a category the dataset does not have, analysis runs, `DataSet` construction from `runinfo`, `filter` and the `livetime`/`runtime` display, and `find_filekey` over every run, also broadcast over timestamps. exposure.md: `get_exposure` skips runs without a start key and, by default, runs that are not analysis runs; `is_analysis_run` keyword. read_data.md: period, partition and run tables read only runs that list DAQ cycle keys for the category, `read_ldata` with a `DataSet` and with a single `Unitful.Time`. metadata_validity.md: a `PropsDB` of a `LegendData` is called with a `Unitful.Time`; `AnyProps` throws for a time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
runinfoof a dataset:default,runinfoholds what the run lists of the dataset indatasets/runlistsname: one column per listed category, and an entry of a category only for the runs listed for it. Any other entry hasmissingstart and end keys, no DAQ cycle keys, aNaNlivetime andis_analysis_run = false. Selecting a category that the dataset does not have throws anArgumentError.:defaultkeeps every run and category ofdatasets/runinfo.get_exposureskips the runs without a start key of the category, and withis_analysis_run = true(the default) the runs that are no analysis runs.read_ldataof a period, partition or run table skips the runs whose entry of the category has no DAQ cycle keys, also withsearch_disk = true, and throws if no run is left. Reading a single run is unchanged.find_filekey(data, ts)searches the DAQ cycle keys of every run in the metadata, so a timestamp in a cycle outside the dataset resolves to its own cycle.Conveniences
filter(f, ds::DataSet)returns aDataSet, e.g.filter(!in(bad_filekeys(l200)), DataSet(l200, runinfo(l200).phy)). The display of a set built fromruninfoshows per category the summedruninfolivetime of the runs with keys in the set, and the runtime from the first to the last key.find_filekey.(data, timestamps)broadcasts over timestamps.PropsDBof aLegendDatatakes aUnitful.Timeas validity selection, e.g.l200.par.ppars.aoe(t), and selects the file key of the DAQ cycle that containst.read_ldata(data, tier, t)andread_ldata(data, tier, t, det)with a singleUnitful.Timeread as[t].read_ldata(data, tier, ds::DataSet)andread_ldata(data, tier, ds, det)read all file keys of the set.Behavior changes
get_exposurewith the default keywords counts only analysis runs. Under:default, runs not in thevalidrun list no longer count.is_lrunandis_analysis_runthrow for a category that the dataset does not have.Since results change, I suggest releasing this as 0.7.0.
p_process_skm_phyin JuLeAna needs to skip runs without aphystart key before JuLeAna moves to this version.Tests
New testsets "runinfo of a dataset", "DataSet" and "selection by a time" (
test_dataprod_config.jl), "Exposure of the runs of a dataset" (test_exposure.jl), and "runs of a dataset" plus single-timestamp reads (test_ext_legendhdf5io.jl).Pkg.test()passes locally (732 passed, 1 broken as onmain).Parts of this pull request, including code, tests and this description, were written with the help of generative AI.