fix(o11y): move the Loki index to an index_ prefix so the compactor can run (DEV-3144) - #402
Merged
Merged
Conversation
…ctor can run
index.prefix "index/" contains a slash, so the compactor cannot map a table back to
its schema period and skips it ("can't find schema for table"), leaving the index
uncompacted. A second schema period from 2026-10-03 uses "index_"; the first stays so
existing data remains readable. The stop check lists both prefixes per day because
records up to 7 days old still land in the old-prefix table after that date.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…time budget Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
3 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Part of DEV-3144. Loki's compactor never compacts because
index.prefix: "index/"contains a slash: reproduced on the filesystem backend withgrafana/loki:3.3.2, where the compactor logsskipping compaction since we can't find schema for tableevery cycle, while the same setup withindex_compacts and removes the source files. This PR appends a second schema period with prefixindex_(same store, schema and 24h period) and keeps the old period so existing data stays readable.supervisor/shutdown.shnow lists bothindex/index/<day>/andindex/index_<day>/per day, still fail-closed on either listing, because records up to 7 days old land in the old-prefix table after the switch. ADR-0041 §B.4 documents the periods, the time budget (worst case about 630 s inside the documented 900 s stop grace) and when the old period can be dropped (not before 90 days afterfrom). The exact S3 error string (invalid prefix index/<n>/browser) was not reproduced because only the filesystem path was run; the failure class is confirmed.Deploy constraint: the new period starts at
from: 2026-10-03(00:00 UTC) and must be live in the deployed image before then. If this deploys afterfrom, index entries already written underindex/index/<day>for days on or afterfrombecome unreachable. If it will slip, changefromin both Loki configs to at least one day after the real deploy date before merging. Compaction only starts working for new-prefix tables afterfrom; old tables stay uncompacted until they age out.Types of changes
runner/) changeHow was this verified?
node --testono11y-shutdown-snapshot.test.mjsando11y-box-config.test.mjsis 46 tests, 46 pass, 0 fail, 0 skipped; with the oldshutdown.shand Loki configs restored, 8 of them fail (new schema-period tests, new dual-prefix tests, and existing tests whose call counts changed).loki -verify-configreports the config valid for both Loki configs. The fullstop-roundtrip.mjsdocker compose run exited 0 with all checks passing (it seeds uploader-named keys under both prefixes). Not run: the rest of the pipeline suite, and anything against R2 or MinIO's S3 path for the compactor. Becausefromis in the future, local runs and production write the old prefix until then, so the new prefix is covered by the seeded keys and unit tests rather than real traffic. One reasoned, not staged, risk: a compaction in flight could remove the freshly uploaded final file before the post-exit listing, which would refuse the stop as unclean (the safe direction, a replay that dedupes).Checklist
runner/config/frameworks.json(see CONTRIBUTING.md); otherwise it won't appear on demos.handsontable.compnpm build(andpnpm dev) in the affected example/server-example locallyNone of the example checklist items apply; this PR changes only
runner/.Related issue(s):
Note
Medium Risk
Changes SIGTERM clean-shutdown index confirmation and introduces a time-bound schema cutover; mis-timed deploy past
fromcould make new index entries unreachable, though tests and fail-closed listing reduce marker false positives.Overview
Adds a second Loki schema period (
from: 2026-10-03) withindex.prefix: "index_"(no slash) in both S3 and filesystem configs, while keeping the originalindex/period so existing data stays readable. A slash in the old prefix prevented the compactor from mapping tables, so compaction never ran.Shutdown / clean-stop checks now list both table layouts per day (
index/index/<day>/andindex/index_<day>/), still fail-closed if any listing fails. ADR-0041 documents the migration, deploy-before-fromrequirement, and the ~240s worst-case listing budget per snapshot.Tests and local harnesses follow: config pins for two periods, snapshot tests with doubled curl call counts, stub
curlkeys under the requested prefix, andstop-roundtrip.mjsseeds pre-existing index objects under both prefixes for the C1 negative control.Reviewed by Cursor Bugbot for commit 682ffb1. Bugbot is set up for automated code reviews on this repo. Configure here.