Repository navigation
CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork) #102
Description
@jeswr
This may help to join effort
The overlay do not appear so high
CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork)
Analysis for cross-branch comparison, prepared to be shared with @jeswr.
Date: 2026-08-31
Scope: the recurring lock-timeout crashes + sustained 100 % CPU observed on a
Community Solid Server (CSS) with a large account count, and how two
independent branches addressed it.
1. Background / problem statement
A CSS v7.2.0 deployment (pivot-test.solidproject.org:3200, ~1000 accounts) exhibited:
- Recurring crashes (
unhandledRejection: Lock expired after 30000ms .../.internal/accounts/data/<uuid>), roughly every 30 minutes, matching the account-cleanup timer. - Sustained 95–100 % CPU with zero traffic, from ~10 minutes after boot.
- Sporadic proper-lockfile race crash (
Cannot read properties of undefined (reading 'updateTimeout')) under CPU starvation.
Root cause chain (validated)
- Full-tree sweep walks. Every expiring-storage sweep
(WrappedExpiringStorage.removeExpiredEntries) callsentries()on a
ContainerPathStorage, which delegates to the underlying
JsonResourceStorage.entries()and walks the entire/.internal/tree
(recursively reading every account data/index/idp file, thousands of files),
then filters by prefix. The 4 sweeps (cookies / forgot-password / OIDC
adapter / tokens, at 10/15/20/25 min) overlap → sustained CPU saturation.
This is a pre-existing upstream CSS inefficiency (shipped configs use
ContainerPathStorageover a whole-treeKeyValueStorage). - Unbounded sweep deletes. The sweep used
Promise.all(expired.map(delete))
— every delete goes through the locked pipeline → lock/EMFILE/memory spikes. - Fire-and-forget cleanup crash.
BaseLoginAccountStorage.createAccountTimeout
(default 30 min) is an unhandledsetTimeoutwithout try/catch; its
storage.delete()lock-timeout rejection crashes the process. - proper-lockfile 4.1.2 race (unmaintained): renewal timer already fired →
in-flight fs callback recursesupdateLockafterdelete locks[file]→
lock.updateTimeouton undefined →uncaughtException.
2. The two branches
| Ours | jeswr's | |
|---|---|---|
| Repo | CommunitySolidServer branch clean-lock + Pivot port branch fix/clear-lock |
jeswr/CommunitySolidServer branch codex/solidcommunity-recovery-alpha-1 + Pivot deployment branch codex/solidcommunity-durable-deployment |
| CSS used | published @solid/community-server 7.2.0 (unmodified) |
@jeswr/community-solid-server 7.1.10-alpha.2 (npm alias → fork) |
| Authors | bourgeoa | jeswr (+ Claude / Copilot co-authors) |
| Basis | upstream 7.2.0 | upstream ~7.1.9 (Mar 2026) |
| Commits | CSS: 8 · Pivot: 2 | fork: ~35 recovery commits (Aug 24–31) · Pivot: 4 |
The two branches are completely disjoint — no shared commits:
> 3687fb0 chore: lock corrected solidcommunity alpha releases (codex)
> 748bdec chore: pin alpha cache release
> d823320 docs: record first-deployment rollback command
> 259735d feat: add durable solidcommunity.net deployment
< d5e6428 fix: move file-locker override out of shared config (fix/clear-lock)
< 4c46845 fix: port CSS lock/sweep hardening
3. jeswr's fork — changes vs latest CSS (analysis)
Compare CommunitySolidServer/main...jeswr:.../codex/solidcommunity-recovery-alpha-1:
58 commits / 84 files, of which ~35 are the Aug 24–31 "solidcommunity
recovery" work. It also contains bourgeoa's own fa4f3fc (pod quota in
subdomain mode, PR #2210) and b48edf8 (Static Asset root expansion).
3.1 Locking ("improved clear lock")
| Commit | Change |
|---|---|
a6dc692 |
maxHoldDuration on WrappedExpiringReadWriteLocker — an absolute cap on total lock hold time, counted from acquisition and never renewed by activity. Default 0 = unlimited (backward compatible, stock configs unchanged). Opt-in config/util/resource-locker/file-capped.json sets 1 hour. Purpose: a stream of trickle-readers can no longer renew a read lock forever and starve a waiting writer. |
a4f44e9 |
Keep write locks alive while request data flows — LockingResourceStore.lockedRepresentationWrite monkey-patches the incoming representation's data.read to call maintainLock() on every chunk (slow uploads no longer drop the lock). If the lock expires mid-write: restore the hook + data.destroy(error) so the source store can't keep writing unprotected. |
0119654 |
Restore the stream read method after lock release — preserve the exact function reference (no .bind() wrappers accumulating), restore it on unlock. |
ded8013, 03c6dc8 |
Remove unsafe stream cast; PR-review finding fix. |
| Redis | 2703fe3 renewed TTL (a crashed holder can't deadlock peers), 7c008e6 don't wipe peer instances' locks on boot, ca24f8c wipe namespace only on single-instance shutdown. |
| Cluster | 117f21b back off + budget cluster worker restarts. |
3.2 Expiring sweeps
| Commit | Change |
|---|---|
02019c1 |
WrappedExpiringStorage now implements Finalizable (finalize() clears the interval, unref() kept) and gains an optional jitter param (default 0.15 × timeout) so the 4 sweep instances don't all fire at the same instant. All four are wired into urn:solid-server:default:Finalizer. |
fbea7e6 |
feat: sweep expired notification channels from storage. |
3.3 Container listing streaming (biggest perf change)
| Commit | Change |
|---|---|
9b214f0 + follow-ups (64c849b, f40d969, 3519acf, 1debc2a, 1824600, cc92f14, a4cd01d, 689427f, 680b85c, 65c9892) |
DataAccessorBasedStore.getRepresentation emits container listings as a lazy Readable<Quad> (streamContainerRepresentation) instead of folding every child into one N3 store. The ldp:contains list now lives in the streamed body (one marker kept in metadata so AllowAcceptHeaderWriter's empty-container check still works); JsonResourceStorage + SingleContainerJsonStorage read members from the body (draining also releases the container read lock before recursing). Measured: a 15k-child container under 20 concurrent reads drops peak live heap from ~3056 MB to ~140 MB (flat). |
3.4 Caching
| Commit | Change |
|---|---|
842d943 |
Reusable promise-cache abstraction. |
86603d4 |
Cache the representation-converter negotiation result. |
3a6278b |
Read each effective ACL only once per request. |
3e31908 |
Cache ACL documents with write-driven invalidation. |
2e21de4, 74adac3 |
Parse/serialize buffered quads synchronously in QuadUtil/readableToQuads. |
86b8eff |
Stream file metadata quads (later reverted in 4c7632d). |
3.5 Identity
| Commit | Change |
|---|---|
ddb2064 |
Reuse known account ID on cookie refresh (drops a redundant read). |
fcc8e93 |
Dedicated OIDC client storage — ExpiringAdapterFactory takes a second clientStorage; the Client model uses it, everything else uses the main storage (falls back if unset). Exposes urn:solid-server:default:IdpAdapterExpiringStorage (transient provider state) and urn:solid-server:default:IdpClientAdapterExpiringStorage (persistent dynamic registrations) as named components. |
⚠️ These two @ids exist only in the fork — not in published CSS 7.2.0.
The Pivotsolidcommunity.netoverrides use them: adapter →MemoryMapStorage,
clients → persistedWrappedExpiringStorage(ContainerPathStorage(/idp/adapter/)).
3.6 Mapping + misc
| Commit | Change |
|---|---|
dee7e9f |
ExtensionBasedMapper "learns" extension lookup order: sample directory (≤128 entries via opendir), rank the ≤24 most frequent $.ext extensions, LRU-cache 1024 directory profiles, dedupe concurrent scans; stat exact filename first, probe extensions by frequency, full readdir fallback + profile refresh. |
4fc6428 |
Skip the log-format pipeline for filtered-out log levels. |
5cec047 |
Skip stack-trace capture for client-error control flow. |
3.7 Packaging
0a61514 release prep · 91d840e package production hotfixes · c892276
canonicalize fork component metadata · 1ea23d4 support installation through
CSS npm alias.
4. Our branch — changes vs latest CSS
CSS clean-lock (8 commits)
| Commit | Change |
|---|---|
53aa83297 |
fix(account): do not crash on lock timeout during account cleanup — try/catch in BaseLoginAccountStorage.createAccountTimeout. |
9bb140418 |
fix(storage): bound delete concurrency in expiring-storage sweep (batch 32 in WrappedExpiringStorage). |
94edf2759 |
feat(storage): configurable sweep batchSize (optional). |
a807630b7 |
fix(storage): destroy representation stream when a read lock expires mid-read (LockingResourceStore, timedOut flag). |
81c8027ed |
fix(locking): bound lock-acquisition retries — FileSystemResourceLocker retryCount: -1 → 600 (~30 s max wait). |
afb8cb181 |
feat(config): give the IDP expiring adapter storage @id urn:solid-server:default:ExpiringAdapterStorage. |
9cb8dd07a |
fix(config): scope expiring-storage sweeps to their own containers — 4 config files. |
fc598140b |
fix(deps): guard proper-lockfile updateLock race via patch-package. |
The core root-cause fix (9cb8dd07a): drop ContainerPathStorage and point
JsonResourceStorage.container at the subpath:
MaxKeyLengthStorage(JsonResourceStorage(
source: ResourceStore_Backend,
baseUrl: variable:baseUrl,
container: "/.internal/<subpath>/"
))
- cookies →
/.internal/accounts/cookies/ - forgot-password →
/.internal/accounts/forgot-password/ - tokens →
/.internal/idp/tokens/ - adapter →
/.internal/idp/adapter/
On-disk paths are unchanged → backward compatible, no migration.
Validated on the box: node CPU 0.2–0.7 %, load ~0.00, flat pings.
Pivot fix/clear-lock (2 commits)
| Commit | Change |
|---|---|
4c46845 |
Port: proper-lockfile patch + CSS adapter-@id patch, postinstall, config/pivot-overrides.json (4 scoped sweeps incl. adapter), SafeBaseLoginAccountStorage (+ src/index.ts export). |
d5e6428 |
Move the file-locker override out of the shared config → config/pivot-file-locker-overrides.json (imported by prod/suffix/dev configs, not test.json). |
Pivot-specific constraints solved along the way:
TS2415privateloggercollision →safeLogger.- componentsjs-generator can't resolve
IndexedStoragegenerics from published
CSS d.ts →storage: any, non-generic class. - nested config params flattened in components.js (
attemptSettings_retryCount, …). test.jsonuses a memory locker → "Missing type for override target" → override
moved to its own file.
5. Direct comparison
Both branches were built for the same symptom set but attack it from
opposite sides:
| Concern | Ours (fix/clear-lock) |
jeswr (codex/...) |
|---|---|---|
| Sweep cost (root cause) | Scope each sweep to its own container — eliminates the full-tree walk | Jitter sweeps + move cookies/adapter to memory in deployment |
| Lock hold time | keep 30 s idle expiry | maxHoldDuration absolute cap (1 h opt-in) |
| Lock acquisition | Bounded retries (retryCount: 600) |
— (memory LockStorage, single-process) |
| Crash guard | proper-lockfile race guard + SafeBaseLoginAccountStorage try/catch |
worker-restart back-off/budget |
| Write-path lock renewal | — (only read-side stream destroy, a807630b7) |
keep write locks alive during data flow + destroy stream on expiry |
| Peak memory (listings) | — | streamed container listings (O(children) → O(1)) |
| Caching | — | promise/ACL/converter caches + @jeswr/css-cached-storage |
| Runs on | unmodified published CSS 7.2.0 | requires the fork (adapter @ids + maxHoldDuration are fork-only) |
Key divergence on the sweep root cause
The fork does not change the sweep's storage architecture. Its on-disk
IdpClientAdapterExpiringStorage is still
WrappedExpiringStorage(ContainerPathStorage(/idp/adapter/, KeyValueStorage)),
so the sweep still walks all of /.internal/ — the exact O(n) full-tree walk
that caused the sustained CPU in our deployment. jeswr's countermeasures are:
jitter (spread the cost), memory-backed hot storages (avoid disk sweeps for
cookies/adapter), plus the caching layer.
Measured numbers
| before | after | |
|---|---|---|
| CPU at 1000 accounts | 95–100 % sustained | 0.2–0.7 % (ours) |
| Response time | volatile 1000–4000 ms | flat ~0 ms (ours) |
| Container listing (15k children, 20 concurrent reads) | ~3056 MB peak heap | ~140 MB (jeswr) |
6. Compatibility & synergy
The two approaches are complementary, not competing:
- Our scoped sweeps would benefit the fork too — it would eliminate the
remaining on-disk client-storage sweep walk (currently jittered but still O(n)). - jeswr's
maxHoldDuration+ write-lock renewal would strengthen our branch
(we only bounded acquisition retries; his work bounds hold time and protects
slow uploads). - Both share a clear principle: sweeps and lock renewals must be bounded —
ours bounds work (scope + batch + retries), jeswr's bounds time
(maxHoldDuration) and memory (streaming + caching).
Hard incompatibility: the codex Pivot branch cannot run on published CSS
without the fork (fork-only @ids IdpAdapterExpiringStorage /
IdpClientAdapterExpiringStorage, and maxHoldDuration). Our branch runs on
unmodified published CSS 7.2.0. Any future merge of the fork's work into
upstream would unblock the codex branch.
7. Suggested next steps (for discussion)
- Decide whether to upstream the fork's "solidcommunity recovery" work
(maxHoldDuration, write-lock renewal, streamed container listings,
promise/ACL caches) as PRs against CSS. - Consider porting our scoped-sweep config change onto the fork, so the
remaining on-disk sweep is O(own container) instead of O(whole tree). - Evaluate
maxHoldDuration+ write-lock renewal for ourclean-lock
branch (they address a real starvation scenario we did not cover). - Cross-check the adapter-@id naming (
ExpiringAdapterStorageours vs
IdpAdapterExpiringStorage/IdpClientAdapterExpiringStoragefork) for a
single canonical convention.
Appendix — useful references
- jeswr fork: https://github.com/jeswr/CommunitySolidServer/tree/codex/solidcommunity-recovery-alpha-1
- Pivot codex deployment branch:
codex/solidcommunity-durable-deployment - Pivot our branch:
fix/clear-lock - CSS our branch:
clean-lock - Related issue: pod quota in subdomain mode fix: find pod quota in subdomain mode, compute once by write not by chunk CommunitySolidServer/CommunitySolidServer#2210
root@DESKTOP-PIRJOCS:/mnt/d/github/solidos/workspaces/pivot# cat notes/css-lock-sweep-comparison-clean-lock-vs-codex.md
CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork)
Analysis for cross-branch comparison, prepared to be shared with @jeswr.
Date: 2026-08-31
Scope: the recurring lock-timeout crashes + sustained 100 % CPU observed on a
Community Solid Server (CSS) with a large account count, and how two
independent branches addressed it.
1. Background / problem statement
A CSS v7.2.0 deployment (pivot-test.solidproject.org:3200, ~1000 accounts) exhibited:
- Recurring crashes (
unhandledRejection: Lock expired after 30000ms .../.internal/accounts/data/<uuid>), roughly every 30 minutes, matching the account-cleanup timer. - Sustained 95–100 % CPU with zero traffic, from ~10 minutes after boot.
- Sporadic proper-lockfile race crash (
Cannot read properties of undefined (reading 'updateTimeout')) under CPU starvation.
Root cause chain (validated)
- Full-tree sweep walks. Every expiring-storage sweep
(WrappedExpiringStorage.removeExpiredEntries) callsentries()on a
ContainerPathStorage, which delegates to the underlying
JsonResourceStorage.entries()and walks the entire/.internal/tree
(recursively reading every account data/index/idp file, thousands of files),
then filters by prefix. The 4 sweeps (cookies / forgot-password / OIDC
adapter / tokens, at 10/15/20/25 min) overlap → sustained CPU saturation.
This is a pre-existing upstream CSS inefficiency (shipped configs use
ContainerPathStorageover a whole-treeKeyValueStorage). - Unbounded sweep deletes. The sweep used
Promise.all(expired.map(delete))
— every delete goes through the locked pipeline → lock/EMFILE/memory spikes. - Fire-and-forget cleanup crash.
BaseLoginAccountStorage.createAccountTimeout
(default 30 min) is an unhandledsetTimeoutwithout try/catch; its
storage.delete()lock-timeout rejection crashes the process. - proper-lockfile 4.1.2 race (unmaintained): renewal timer already fired →
in-flight fs callback recursesupdateLockafterdelete locks[file]→
lock.updateTimeouton undefined →uncaughtException.
2. The two branches
| Ours | jeswr's | |
|---|---|---|
| Repo | CommunitySolidServer branch clear-lock (PR #2235) + Pivot port branch fix/clear-lock (PR #144) |
jeswr/CommunitySolidServer branch codex/solidcommunity-recovery-alpha-1 + Pivot deployment branch codex/solidcommunity-durable-deployment |
| CSS used | published @solid/community-server 7.2.0 (unmodified) |
@jeswr/community-solid-server 7.1.10-alpha.2 (npm alias → fork) |
| Authors | bourgeoa | jeswr (+ Claude / Copilot co-authors) |
| Basis | upstream 7.2.0 | upstream ~7.1.9 (Mar 2026) |
| Commits | CSS: 8 · Pivot: 2 | fork: ~35 recovery commits (Aug 24–31) · Pivot: 4 |
The two branches are completely disjoint — no shared commits:
> 3687fb0 chore: lock corrected solidcommunity alpha releases (codex)
> 748bdec chore: pin alpha cache release
> d823320 docs: record first-deployment rollback command
> 259735d feat: add durable solidcommunity.net deployment
< d5e6428 fix: move file-locker override out of shared config (fix/clear-lock)
< 4c46845 fix: port CSS lock/sweep hardening
3. jeswr's fork — changes vs latest CSS (analysis)
Compare CommunitySolidServer/main...jeswr:.../codex/solidcommunity-recovery-alpha-1:
58 commits / 84 files, of which ~35 are the Aug 24–31 "solidcommunity
recovery" work. It also contains bourgeoa's own fa4f3fc (pod quota in
subdomain mode, PR #2210) and b48edf8 (Static Asset root expansion).
3.1 Locking ("improved clear lock")
| Commit | Change |
|---|---|
a6dc692 |
maxHoldDuration on WrappedExpiringReadWriteLocker — an absolute cap on total lock hold time, counted from acquisition and never renewed by activity. Default 0 = unlimited (backward compatible, stock configs unchanged). Opt-in config/util/resource-locker/file-capped.json sets 1 hour. Purpose: a stream of trickle-readers can no longer renew a read lock forever and starve a waiting writer. |
a4f44e9 |
Keep write locks alive while request data flows — LockingResourceStore.lockedRepresentationWrite monkey-patches the incoming representation's data.read to call maintainLock() on every chunk (slow uploads no longer drop the lock). If the lock expires mid-write: restore the hook + data.destroy(error) so the source store can't keep writing unprotected. |
0119654 |
Restore the stream read method after lock release — preserve the exact function reference (no .bind() wrappers accumulating), restore it on unlock. |
ded8013, 03c6dc8 |
Remove unsafe stream cast; PR-review finding fix. |
| Redis | 2703fe3 renewed TTL (a crashed holder can't deadlock peers), 7c008e6 don't wipe peer instances' locks on boot, ca24f8c wipe namespace only on single-instance shutdown. |
| Cluster | 117f21b back off + budget cluster worker restarts. |
3.2 Expiring sweeps
| Commit | Change |
|---|---|
02019c1 |
WrappedExpiringStorage now implements Finalizable (finalize() clears the interval, unref() kept) and gains an optional jitter param (default 0.15 × timeout) so the 4 sweep instances don't all fire at the same instant. All four are wired into urn:solid-server:default:Finalizer. |
fbea7e6 |
feat: sweep expired notification channels from storage. |
3.3 Container listing streaming (biggest perf change)
| Commit | Change |
|---|---|
9b214f0 + follow-ups (64c849b, f40d969, 3519acf, 1debc2a, 1824600, cc92f14, a4cd01d, 689427f, 680b85c, 65c9892) |
DataAccessorBasedStore.getRepresentation emits container listings as a lazy Readable<Quad> (streamContainerRepresentation) instead of folding every child into one N3 store. The ldp:contains list now lives in the streamed body (one marker kept in metadata so AllowAcceptHeaderWriter's empty-container check still works); JsonResourceStorage + SingleContainerJsonStorage read members from the body (draining also releases the container read lock before recursing). Measured: a 15k-child container under 20 concurrent reads drops peak live heap from ~3056 MB to ~140 MB (flat). |
3.4 Caching
| Commit | Change |
|---|---|
842d943 |
Reusable promise-cache abstraction. |
86603d4 |
Cache the representation-converter negotiation result. |
3a6278b |
Read each effective ACL only once per request. |
3e31908 |
Cache ACL documents with write-driven invalidation. |
2e21de4, 74adac3 |
Parse/serialize buffered quads synchronously in QuadUtil/readableToQuads. |
86b8eff |
Stream file metadata quads (later reverted in 4c7632d). |
3.5 Identity
| Commit | Change |
|---|---|
ddb2064 |
Reuse known account ID on cookie refresh (drops a redundant read). |
fcc8e93 |
Dedicated OIDC client storage — ExpiringAdapterFactory takes a second clientStorage; the Client model uses it, everything else uses the main storage (falls back if unset). Exposes urn:solid-server:default:IdpAdapterExpiringStorage (transient provider state) and urn:solid-server:default:IdpClientAdapterExpiringStorage (persistent dynamic registrations) as named components. |
⚠️ These two @ids exist only in the fork — not in published CSS 7.2.0.
The Pivotsolidcommunity.netoverrides use them: adapter →MemoryMapStorage,
clients → persistedWrappedExpiringStorage(ContainerPathStorage(/idp/adapter/)).
3.6 Mapping + misc
| Commit | Change |
|---|---|
dee7e9f |
ExtensionBasedMapper "learns" extension lookup order: sample directory (≤128 entries via opendir), rank the ≤24 most frequent $.ext extensions, LRU-cache 1024 directory profiles, dedupe concurrent scans; stat exact filename first, probe extensions by frequency, full readdir fallback + profile refresh. |
4fc6428 |
Skip the log-format pipeline for filtered-out log levels. |
5cec047 |
Skip stack-trace capture for client-error control flow. |
3.7 Packaging
0a61514 release prep · 91d840e package production hotfixes · c892276
canonicalize fork component metadata · 1ea23d4 support installation through
CSS npm alias.
4. Our branch — changes vs latest CSS
CSS clear-lock (8 commits) — PR #2235
| Commit | Change |
|---|---|
53aa83297 |
fix(account): do not crash on lock timeout during account cleanup — try/catch in BaseLoginAccountStorage.createAccountTimeout. |
9bb140418 |
fix(storage): bound delete concurrency in expiring-storage sweep (batch 32 in WrappedExpiringStorage). |
94edf2759 |
feat(storage): configurable sweep batchSize (optional). |
a807630b7 |
fix(storage): destroy representation stream when a read lock expires mid-read (LockingResourceStore, timedOut flag). |
81c8027ed |
fix(locking): bound lock-acquisition retries — FileSystemResourceLocker retryCount: -1 → 600 (~30 s max wait). |
afb8cb181 |
feat(config): give the IDP expiring adapter storage @id urn:solid-server:default:ExpiringAdapterStorage. |
9cb8dd07a |
fix(config): scope expiring-storage sweeps to their own containers — 4 config files. |
fc598140b |
fix(deps): guard proper-lockfile updateLock race via patch-package. |
The core root-cause fix (9cb8dd07a): drop ContainerPathStorage and point
JsonResourceStorage.container at the subpath:
MaxKeyLengthStorage(JsonResourceStorage(
source: ResourceStore_Backend,
baseUrl: variable:baseUrl,
container: "/.internal/<subpath>/"
))
- cookies →
/.internal/accounts/cookies/ - forgot-password →
/.internal/accounts/forgot-password/ - tokens →
/.internal/idp/tokens/ - adapter →
/.internal/idp/adapter/
On-disk paths are unchanged → backward compatible, no migration.
Validated on the box: node CPU 0.2–0.7 %, load ~0.00, flat pings.
Pivot fix/clear-lock (2 commits) — PR #144
| Commit | Change |
|---|---|
4c46845 |
Port: proper-lockfile patch + CSS adapter-@id patch, postinstall, config/pivot-overrides.json (4 scoped sweeps incl. adapter), SafeBaseLoginAccountStorage (+ src/index.ts export). |
d5e6428 |
Move the file-locker override out of the shared config → config/pivot-file-locker-overrides.json (imported by prod/suffix/dev configs, not test.json). |
Pivot-specific constraints solved along the way:
TS2415privateloggercollision →safeLogger.- componentsjs-generator can't resolve
IndexedStoragegenerics from published
CSS d.ts →storage: any, non-generic class. - nested config params flattened in components.js (
attemptSettings_retryCount, …). test.jsonuses a memory locker → "Missing type for override target" → override
moved to its own file.
5. Direct comparison
Both branches were built for the same symptom set but attack it from
opposite sides:
| Concern | Ours (fix/clear-lock) |
jeswr (codex/...) |
|---|---|---|
| Sweep cost (root cause) | Scope each sweep to its own container — eliminates the full-tree walk | Jitter sweeps + move cookies/adapter to memory in deployment |
| Lock hold time | keep 30 s idle expiry | maxHoldDuration absolute cap (1 h opt-in) |
| Lock acquisition | Bounded retries (retryCount: 600) |
— (memory LockStorage, single-process) |
| Crash guard | proper-lockfile race guard + SafeBaseLoginAccountStorage try/catch |
worker-restart back-off/budget |
| Write-path lock renewal | — (only read-side stream destroy, a807630b7) |
keep write locks alive during data flow + destroy stream on expiry |
| Peak memory (listings) | — | streamed container listings (O(children) → O(1)) |
| Caching | — | promise/ACL/converter caches + @jeswr/css-cached-storage |
| Runs on | unmodified published CSS 7.2.0 | requires the fork (adapter @ids + maxHoldDuration are fork-only) |
Key divergence on the sweep root cause
The fork does not change the sweep's storage architecture. Its on-disk
IdpClientAdapterExpiringStorage is still
WrappedExpiringStorage(ContainerPathStorage(/idp/adapter/, KeyValueStorage)),
so the sweep still walks all of /.internal/ — the exact O(n) full-tree walk
that caused the sustained CPU in our deployment. jeswr's countermeasures are:
jitter (spread the cost), memory-backed hot storages (avoid disk sweeps for
cookies/adapter), plus the caching layer.
Measured numbers
| before | after | |
|---|---|---|
| CPU at 1000 accounts | 95–100 % sustained | 0.2–0.7 % (ours) |
| Response time | volatile 1000–4000 ms | flat ~0 ms (ours) |
| Container listing (15k children, 20 concurrent reads) | ~3056 MB peak heap | ~140 MB (jeswr) |
6. Compatibility & synergy
The two approaches are complementary, not competing:
- Our scoped sweeps would benefit the fork too — it would eliminate the
remaining on-disk client-storage sweep walk (currently jittered but still O(n)). - jeswr's
maxHoldDuration+ write-lock renewal would strengthen our branch
(we only bounded acquisition retries; his work bounds hold time and protects
slow uploads). - Both share a clear principle: sweeps and lock renewals must be bounded —
ours bounds work (scope + batch + retries), jeswr's bounds time
(maxHoldDuration) and memory (streaming + caching).
Hard incompatibility: the codex Pivot branch cannot run on published CSS
without the fork (fork-only @ids IdpAdapterExpiringStorage /
IdpClientAdapterExpiringStorage, and maxHoldDuration). Our branch runs on
unmodified published CSS 7.2.0. Any future merge of the fork's work into
upstream would unblock the codex branch.
7. Suggested next steps (for discussion)
- Decide whether to upstream the fork's "solidcommunity recovery" work
(maxHoldDuration, write-lock renewal, streamed container listings,
promise/ACL caches) as PRs against CSS. - Consider porting our scoped-sweep config change onto the fork, so the
remaining on-disk sweep is O(own container) instead of O(whole tree). - Evaluate
maxHoldDuration+ write-lock renewal for ourclear-lock
branch (they address a real starvation scenario we did not cover). - Cross-check the adapter-@id naming (
ExpiringAdapterStorageours vs
IdpAdapterExpiringStorage/IdpClientAdapterExpiringStoragefork) for a
single canonical convention.
Appendix — useful references
- jeswr fork: https://github.com/jeswr/CommunitySolidServer/tree/codex/solidcommunity-recovery-alpha-1
- Pivot codex deployment branch:
codex/solidcommunity-durable-deployment - Pivot our branch (PR):
fix/clear-lock— fix: port CSS lock/sweep hardening solid-contrib/pivot#144 - CSS our branch (PR):
clear-lock— fix memory leak from periodic maintenance clear locks CommunitySolidServer/CommunitySolidServer#2235 - Related issue: pod quota in subdomain mode fix: find pod quota in subdomain mode, compute once by write not by chunk CommunitySolidServer/CommunitySolidServer#2210
Activity