Repository navigation
fix: make read-only AttrsDicts fully read-only - #63
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #63 +/- ##
==========================================
+ Coverage 91.72% 92.46% +0.74%
==========================================
Files 6 6
Lines 568 664 +96
==========================================
+ Hits 521 614 +93
- Misses 47 50 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
TextDB.on() results share nested objects with the database cache (gipert#48) and are flagged read-only, but only __setitem__/__setattr__/|= checked the flag: pop/update/setdefault/del/clear and any list method silently changed the cache, leaking into later on() calls. The flag also landed on cached files only after an on() call, map() and AttrsDict(...) could unfreeze them, and pickling dropped it. - check the flag in every dict mutator - lists inside read-only AttrsDicts become ReadOnlyList (dumps as a list) - freeze files when TextDB loads them, so call order does not matter - constructor and map() only ever propagate True - pickle keeps the flag; copy/deepcopy give writable copies Assisted-by: Generative AI
Assisted-by: Generative AI
- __copy__ reuses the AttrsDict constructor - __setitem__ skips scanning ReadOnlyList values, whose dict elements are already AttrsDicts (Props.add_to hits this on every on() call) - freezing iterates dict items without copying them and builds each ReadOnlyList without an intermediate list Assisted-by: Generative AI
916bcbc to
67b18ca
Compare
|
I rebased this branch on I also pushed 67b18ca, which removes some redundant work without changing behavior: Two larger suggestions I left for you, since they rework the design:
Verified locally: |
The dict branch of __setattr__("__readonly__") and the module helper
called each other through a suppress_warning flag, and textdb.py
imported the private helper to freeze loaded files.
- _set_readonly(d, flag) now holds the dict walk; __setattr__ only adds
the unfreeze warning and calls it. The suppress_warning argument is
gone (no caller outside this module).
- ReadOnlyList freezes its contents on construction, so TextDB freezes
a loaded file with `loaded.__readonly__ = True` or
`ReadOnlyList(loaded)` and needs no private import.
Assisted-by: Generative AI
|
Suggestion 1 from my previous comment is now implemented in 35211c1. The walk over a dict's contents lives only in |
Since #48,
TextDB.on()results share nested objects with the database cache and are flagged read-only. But the flag is only checked by__setitem__,__setattr__and|=. I reproduced these on the LEGEND dataprod config DB:pop,update,setdefault,del,clearand every list method (append,sort, …) skip the check and silently change the cache. Apoporappendondb.on(t1)shows up in a laterdb.on(t2)in the same process. This bites long-lived processes (Snakemake, notebooks, dashboards).on()has run, sodb["file"]...[k] = vsucceeds or raises depending on call order.AttrsDict(x)andTextDB.map()propagateFalsedown, which unfreezes the shared cache.This PR:
AttrsDicts to aReadOnlyList(list). It is still alistforisinstance, JSON and+, and YAML dumpers are registered so it dumps as a plain list;TextDBloads them;map()only ever propagateTrue;copy.copy/copy.deepcopyreturn writable copies;on()docstring.Writable use of
AttrsDictandProps.read_fromis unchanged.Validation.
tests/test_readonly.py: 28 cases. Onmain, 23 fail and only the 3 already-guarded mutators plus the copy/usability checks pass. Full suite: 58 passed.pre-commit run -aandnox -s pylint(10.00/10) pass.TextDB.on()for all 3,643 (timestamp × category) lookups overdataprod/config,dataprod/overrides/*,datasets/statusesandhardware/configuration/channelmapsgives results identical to 1.4.0, and runs faster: 254 s vs 283 s, since freezing skips subtrees that are already frozen.Known limit: plain dicts nested in lists of lists aren't converted to
AttrsDict, so they can't be frozen.Related: #62, legend-exp/legend-dataflow-scripts#59.
🤖 Generated with Claude Code