feat(storage): see what a module is holding, and drop what outlived it #46
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/issue-39-storage-enumeration"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #39.
ctx.storagecapped each value at 8 MB and bounded nothing else: no enumeration, no way to remove rows for a module that was deleted or renamed, andupdated_at_mswritten on everysetand read by nothing.Store
ModuleKvgains three methods:list(module, limit?){ key, bytes, updatedAtMs }[]— never values.limitbecomes a SQLLIMIT, so it bounds the read itself, not just the response — the query runs on DuckDB's one serialized queueusage(){ keys, bytes, updatedAtMs }, every module holding rowsdeleteAll(module)DELETE … RETURNINGso the number and the delete are one statementSizes come from
strlen, notlength: DuckDB'slengthcounts characters and the cap insetis in UTF-8 bytes, and two units for one number is how a 3 MB value reads as 1 MB. There is a test pinning that.usageanddeleteAllare unscoped by design and never appear onctx. They are reached only throughModuleHost.moduleStorage, which is typedPick<ModuleKv, "list" | "usage" | "deleteAll">(OperatorModuleKv) — so "operator surfaces only" is structural, not a comment:get/setcannot leak along with them.Module-facing:
ctx.storage.list()Its own keys, with sizes and write times, never the values — asking what you hold must not cost the whole footprint to answer. It passes no
limit: a module's contract is every key; the bound exists for the operator route below.It takes no module argument. Like
get/set/delete, the facade inhost.tscloses over the module's own name, solistcan only ever answer for the caller. A test proves that end to end — a real module whoseregister(ctx)exposesctx.storage.list()on a route, with a neighbour module's rows in the same table:Confirmed it has teeth: pointing the facade at the neighbour's name fails it.
The use it exists for is expiring your own entries — a cache keyed per image or per remote object can otherwise only delete the keys it still remembers, which are not the ones that accumulate.
Operator-facing:
/systemDropping a whole module's rows is deliberately not on
ctx: the rows most in need of dropping belong to a module that no longer loads and therefore has noctxat all./systempayload gainsmoduleStorage: { modules, totalBytes, unidentified }, each row carryingconfigured— three-valued.truewhen the deployment still asks for the module;falsewhen nothing does — the orphan case the issue is about;nullwhen the answer is not knowable this boot. The last state exists because a failure can record only the repo slug while the rows live under the manifest name: a slug can never match them, so while any external module failed before its manifest was readable, an unmatched row must not be called orphaned.ModuleFailurecarriesnameFromManifestto make that decidable — and the recovery keys on the manifest being READABLE, not acceptable:readManifestNameparses the schema without the SDK gate, so a module requiring a newer SDK major — the one failure mode that never resolves itself — still names its failure, from whatever checkout is on disk (the previous boot's for a clone failure, the fresh one for a manifest the gate refused).nullis therefore reserved for a failure with no readable manifest anywhere.unidentifiedlists those failures by name (still slugs, redacted), so the page can say which module is unaccounted for instead of a deployment-wide flag.usage()that throws is a thrown/system, not an empty section: an empty store answers[], so a catch there could only dress a database failure up as "Nothing stored" on the page whose job is finding rows.GET|DELETE /api/core/system/module-storage/:module, bothadminOnly, both under the app-wide CSRF guard, with the delete logged (module,removed,by). The GET caps at 500 keys in the SQL and reports the true total fromusage()'s row count; both routes have a non-admin 403 test run against real sessions rather than the auth-disabled static admin./systemsnapshot and theRETURNINGcount is the actual confirmation. Arming Clear on a still-configured module says the module may be using these rows right now; theconfigured: falserow says clearing is the only cleanup; theconfigured: nullrow says the opposite — leave the rows alone unless the module is known gone for good — and names the unidentified module.Never automatic on a load failure, per the review: a repo that could not clone this boot is absent for a reason that resolves itself, and deleting its state would turn a network blip into permanent data loss.
Not included
The optional per-module total bound. It wants a number, and the enumeration this adds is what will supply one from a real deployment instead of a guess. Deferring it also keeps this change from making a previously-succeeding
set()start throwing.Verification
Beyond the suite: built the app and ran it against a data dir seeded with rows for a loaded module (
docker) and an unmatched module, with an unreachable external repo configured so the unidentified path is live, then drove the page in headless Edge:The two-click clear removes exactly one module's rows, the notice carries the server's count, and the totals refresh with the database agreeing. A delete that lands while the follow-up
/systemread fails surfaces a section-level message naming what happened, removed count included.The branch now carries a merge of
main(#45'sstage: "retry"rescue and #49's git-test isolation landed there since the last round). Two merge consequences owned here: the rescue record gainsnameFromManifest, carried from the failed attempt — whether the name is the manifest's is a fact about the name, and the rescue does not change the name — andseedRepospawns git throughTEST_GIT_SPAWN/tools/test-gitconfiglike every other git-shelling test, which supersedes the inline gpgsign override this branch had (and with it, a failing call reports the git verb again, since no flags precede it).Gates, on the merged tree:
deno task check,deno lint(203 files),deno fmt --check(321 files),deno task check:svelte(0 errors),deno task build, andpackages/server/tests/— 131 passed: the 129 the third review measured, plus a regression pinning that an SDK-incompatible manifest still names its failure (sdk: "^999", clone succeeds, stagemanifest,nameFromManifest: true) — the round-3 blocker: the recovery calledreadManifest, which throws on the SDK gate after parsing, so the failure that lasts forever was exactly the one that recorded the slug — plus the retry-rescue test that arrived with the merge.The rate limiter's on-request sweep is throttled to once a minute above the threshold: the map key is client-supplied (leftmost
x-forwarded-forbehind an appending proxy), so the map's size — and with it a per-request scan's cost — is a number an attacker rotating addresses chooses; the comment now says so, and no longer claims the old interval was unref'd (it was not — and the leak's discriminator is the CI-pinned Deno image, not "some machines").Docs:
docs/modules.mdgainslist()and what it is for;CLAUDE.mdgains the split between what is onctxand what is on/system, and why.Re-reviewed the whole diff against the PR head (
2d71242) in a clean worktree, and re-ran every gate rather than taking the description's word for them.Gates, independently reproduced
deno task checkdeno lintdeno fmt --checkdeno task check:sveltepackages/server/tests/Two notes on the suite. The description says 127; I get 126 on the same commit — not a real problem, but the number in the body is not the number the tree produces. And
external_modules_test.tsfails outright on any machine withcommit.gpgsign=true, becauseseedReposhells out togit commitwithout-c commit.gpgsign=false(gpg: signing failed, thenfatal: failed to write commit object). That is pre-existing, not yours, but it is why a clean run here neededGIT_CONFIG_KEY_0=commit.gpgsign.Claims I checked rather than accepted
strlenvslength. Correct, and worth the comment:SELECT length('☃☃☃'), strlen('☃☃☃')returns3, 9. The unit really does matchset'sTextEncodercap.deleteAllhas exactly one caller, the DELETE route.list(). Holds structurally — the facade closes overname, there is no argument to pass, and the end-to-end test through a realregister(ctx)is the right shape for proving it.So the core of this is sound. What follows is what did not survive.
Blocking
1.
configured: falseis wrong for the exact case this PR says it protects — and the UI turns that into an invitation to delete.knownis built fromconfig.modules+ loaded names + failed names. ButModuleFailure.nameis documented, athost.ts:130, as "manifest name once known, otherwise the configured name/slug", andprepareExternalModuleonly assignsname = manifest.nameaftersyncReposucceeds (external.ts:139-141). A clone-stage failure therefore records the slug —git.imhof.cloud-OpsDeck-mod-teamspeak— while the rows are under the manifest name,teamspeak.I reproduced it end to end against
buildSystemInfo, with one external module configured, onerecordFailure({ stage: "clone", name: <slug> }), and one row stored under the manifest name:The page then renders
not configuredplus "Nothing configured under this name — rows left by a module that was removed or renamed. Clearing is the only thing that removes them." — for a module the deployment is still asking for, which will come back on the next boot. That is precisely the "network blip becomes permanent data loss" the storage doc comment, the CLAUDE.md paragraph and the PR body all say this design avoids; it is just moved from an automatic deletion to a UI that tells the operator the deletion is safe. The strongest wording is insystem.tsitself: "A repo that could not clone this boot has not stopped being ours, and its rows must not read as orphaned." The code does not deliver that.Two ways out, either acceptable:
srcDiron disk with a readableopsdeck.module.json, soprepareExternalModule's catch can fall back toreadManifest(srcDir)before settling for the slug. That fixes the failure record for/system's module list too, not just this section.configuredhonest about not knowing: when any configured external module failed before its manifest was readable, the answer for an unmatched row is unknown, not orphaned — and the copy has to stop saying "removed or renamed" and stop implying clearing is the correct next step.A regression test belongs with it: slug ≠ manifest name,
stage: "clone", rows under the manifest name. Everything currently tested aboutconfigureduses a built-in whose name matches, which is the case that cannot fail.Should fix before merge
2.
deps.kv.usage().catch(() => [])(system.ts:131) makes a database error indistinguishable from an empty store. The section rendersEmptyState— "Nothing stored", "modules keep derived state here throughctx.storage" — and the operator who came here to find orphans concludes there are none. ThegetMetacalls above it catch for a real reason (the meta row may be absent);usage()has no such case — I confirmed an empty table returns[], not a throw. So the catch only ever hides a genuine failure. Given you added a section-level message for "the delete landed but the refresh did not", swallowing the read is inconsistent with the standard this PR sets for itself.3.
ModuleHost.moduleStoragehands out the whole unscopedModuleKv(host.ts:253),getandsetincluded.storage.ts:13states the principle: "Scoping is the host's job, not the caller's promise." This getter converts it back into a promise — the comment says "for operator surfaces only", and nothing enforces it.Pick<ModuleKv, "list" | "usage" | "deleteAll">costs nothing and keeps the guarantee structural, which is the whole reason the facade exists.4.
clearStoragedoes not guard the DELETE against rejection (SystemPage.svelte:185).toggleKeys, fourteen lines above, does.catch(() => null). Here a dropped connection rejects inside thetry,finallyclearsbusyandconfirming, and the operator sees the armed button revert with no message at all — the one outcome worse than a wrong total is a destructive action whose failure is invisible. TheclearErrorpath you added for the refresh case is the right pattern; extend it to the request itself.Smaller
5.
GET /system/module-storage/:moduleis unbounded (app.ts:347). The comment justifying the per-module fetch says a module "may hold thousands" — but once expanded there is no cap, no pagination, and no truncation notice. For a change whose subject is unbounded storage, a limit with a "showing N of M" line is cheap.6.
deleteAll's count is not atomic with its delete (storage.ts:181). The comment calls the number "the whole confirmation the operator gets", and it can be wrong:Promise.all([kv.deleteAll("m"), kv.set("m", "c", "z")])with two rows present reportedremoved = 2while three rows were actually removed and the module'ssetresolved successfully with its row already gone. Rare, and arguably inherent to "clear everything", but the comment overstates what the number guarantees.DELETE … RETURNINGcounted in one statement, if DuckDB obliges, removes the gap; otherwise soften the claim.7. Misplaced JSDoc (
host.ts:243-253). The new getter was inserted between/** Record a module that never reached load() at all (clone/build failure). */andrecordFailure, so that doc now sits onmoduleStorageandrecordFailurehas none.8. Dead nullable.
ModuleStorageUsage.updatedAtMs: number | nullis documented "null for a module holding nothing", butGROUP BY modulecannot produce a group with no rows andupdated_at_msisBIGINT NOT NULL(schema.ts:210). Confirmed: an empty table yields[]. The type and the branch atstorage.ts:163describe a state that cannot exist, and it propagates into the SDK-facing shape and the Svelte interface.9.
storage.ts:13now lies. "every method takes the module name" —usage()is the first that does not. It is a load-bearing comment; it should say which methods are unscoped and who is allowed to call them.10.
SystemDeps.kvduplicatesSystemDeps.host.buildSystemInfoalready has the host and could readhost.moduleStorage; instead every caller must remember to passkv: host.moduleStorage, and one that passes aModuleKvover a differentDbgets totals describing a database nobody is serving.11. Test gap: neither new route has a non-admin case.
withStackrunsOPSDECK_AUTH=disabled, which injects a static admin, so both HTTP assertions pass on a session that is admin by construction. The CSRF negative is tested and good; the role gate the PR body advertises twice is not exercised at all.12.
class="keys"on thedlhas no rule in the style block — the baredlselector is what styles it. Harmless, but the attribute reads as if it does something.Happy to re-review quickly once 1-4 are addressed; the rest can ride along.
@ -342,0 +344,4 @@// never the values: this is a diagnostics page, and a module's stored state// can be anything it derived, including data its own UI gates on a role.api.get("/system/module-storage/:module",Unbounded. The comment on the frontend explains the per-module fetch by "a module may hold thousands" — but the expanded list has no cap, no pagination and no truncation notice, so the thousands arrive in one response and one array. A limit plus a "showing N of M" line matches what the rest of this change is arguing for.
Also: neither this route nor the DELETE has a non-admin test.
withStackrunsOPSDECK_AUTH=disabled, so the HTTP tests are all admin by construction — the CSRF negative is covered, the role gate is not.@ -244,0 +250,4 @@* module's own name (see `contextFor`), and that closure is the whole* mechanism keeping one module out of another's rows.*/get moduleStorage(): ModuleKv {Two things.
The getter returns the full unscoped
ModuleKv,get/setincluded, so "for operator surfaces only" is enforced by the comment alone — whilestorage.ts:13insists scoping is "the host's job, not the caller's promise".Pick<ModuleKv, "list" | "usage" | "deleteAll">keeps that structural at zero cost.And it landed between
recordFailure's doc comment andrecordFailure, so/** Record a module that never reached load() at all … */now documents this getter.@ -40,0 +51,4 @@keys: number;bytes: number;/** most recent write, or null for a module holding nothing */updatedAtMs: number | null;Unreachable.
GROUP BY moduleyields no group without rows, andupdated_at_msisBIGINT NOT NULL(schema.ts:210) — an empty table gives[], not a row with a null. The nullability propagates into the SDK shape and the Svelte interface and the branch at line 163 can never be taken.@ -95,0 +178,4 @@// counted first: DuckDB's DELETE reports nothing back through `run`, and// "removed 0 rows" versus "removed 40" is the whole confirmation the// operator getsconst [row] = await this.db.query<{ keys: number }>(The count and the delete are separate trips through the serialized queue, so the number the comment calls "the whole confirmation the operator gets" can be wrong.
Promise.all([kv.deleteAll("m"), kv.set("m", "c", "z")])over two existing rows reportedremoved = 2while three rows went, and the module'ssetresolved fine with its row already deleted.DELETE … RETURNINGcounted in one statement if DuckDB allows it; otherwise weaken the comment.@ -105,0 +123,4 @@// URL rather than by module name, so the name it stores under is only// knowable from a load or a failure. A repo that could not clone this boot// has not stopped being ours, and its rows must not read as orphaned.const known = new Set([Blocking.
host.failed.map(f => f.name)cannot supply the manifest name for a clone-stage failure:ModuleFailure.nameis "manifest name once known, otherwise the configured name/slug" (host.ts:130), andprepareExternalModulesetsname = manifest.nameonly aftersyncReporeturns (external.ts:139-141).So for
https://git.imhof.cloud/OpsDeck/mod-teamspeak.gitwhose manifest name isteamspeak,knowngets the sluggit.imhof.cloud-OpsDeck-mod-teamspeakand the rows underteamspeakcome outconfigured: false. Reproduced against this exact function:The comment right here says "its rows must not read as orphaned" — this is the code that makes them read that way, and the UI copy then tells the operator clearing is the only fix. Either resolve the manifest name on a clone failure (the previous boot's
srcDirstill has a readableopsdeck.module.json), or make the unmatched case unknown rather than orphaned.@ -105,0 +128,4 @@...host.loaded.map((m) => m.name),...host.failed.map((f) => f.name),]);const usage = await deps.kv.usage().catch(() => []);.catch(() => [])renders a database failure as "Nothing stored" — on the one surface whose purpose is finding rows nobody else can see. ThegetMetacatches above have a real absent-row case;usage()does not (an empty table returns[], I checked), so this only ever hides a genuine error. Let it throw, or carry the failure into the payload the wayclearErrorcarries the refresh failure.@ -128,0 +182,4 @@async function clearStorage(module: string) {busy = true;try {const res = await coreFetch(Unguarded, unlike
toggleKeysfourteen lines up which does.catch(() => null). A dropped connection rejects here,finallyresetsbusyandconfirming, and the operator watches the armed button revert with no message — a destructive action whose failure is silent. Route it intoclearErrorlike the refresh failure below.@ -325,0 +475,4 @@</div>{#if !row.configured}<div class="source">Nothing configured under this name — rows left by a module thatThis copy is the sharp edge on the
configuredbug. For a still-configured external module whose clone failed this boot, the row says "removed or renamed" and "clearing is the only thing that removes them" — so the page actively recommends destroying state that is coming back. Untilconfiguredcan tell orphaned from could not be identified this boot, the wording must not assert the first.Rework for the review is up as
8f39595(plus7847fcafor the sanitizer leak your gpgsign note led me to). All 12 items addressed, by your numbering:1 (blocking) — both ways out, taken together.
prepareExternalModule's catch now falls back toreadManifest(srcDir)when the manifest was never read, so a clone failure after any previous success records the manifest name — fixing the/systemmodule list as well as this section. For the case with nothing on disk to read,ModuleFailuregains a requirednameFromManifest, andbuildSystemInfomakesconfiguredthree-valued: while any external failure has no manifest name, an unmatched row isnull— "not knowable this boot" — neverfalse. The regression test you asked for is there (slug ≠ manifest name,stage: "clone", rows under the manifest name →null; same failure with the manifest name →true), plus anexternal_modules_testproving the on-disk recovery through a real seeded checkout whose fetch fails.2 — the
catch(() => [])is gone; ausage()throw is now a failed/systemrequest, with a comment saying why no catch belongs there.3 —
moduleStoragereturnsOperatorModuleKv = Pick<ModuleKv, "list" | "usage" | "deleteAll">. Tests that seed rows construct a rawModuleKvover the sameDb, which is itself a nice proof the surface lostset.4 — the DELETE is guarded like
toggleKeys; a rejection lands inclearError("the request never reached the server") instead of a silently disarmed button.5 — the GET caps at 500 and returns
{ keys, total }; the pane shows "showing the first N of M keys" when cut.6 —
DELETE … RETURNING key— DuckDB obliges, so the count and the delete are one statement and the comment's claim is now true.7 —
recordFailurehas its doc back; the getter has its own.8 —
updatedAtMsisnumberend to end (store,/systempayload, Svelte interface); the unreachable branch is gone.9 — the scoping comment now names
get/set/delete/listas the facade-scoped set andusage/deleteAllas operator-only viamoduleStorage.10 —
SystemDeps.kvremoved;buildSystemInforeadshost.moduleStorageoff the host it already has.11 — both routes have a non-admin case: the stack runs in real OIDC mode with sessions seeded through
SessionStore, viewer gets 403 on GET and on DELETE (rows untouched), admin passes the same gate.12 — the dead
class="keys"is gone.Also from your gate notes:
seedRepocommits with-c commit.gpgsign=false, so a signing machine runs the suite clean. And chasing why the suite failed here turned up the cause worth keeping:rateLimit()started a sweep interval nothing ever cleared — two leaked timers percreateApp, which some Deno setups' sanitizer counts even unref'd. The sweep now runs on the request path when the bucket map crosses a size threshold, so there is no background timer at all (7847fca).Gates re-run on the new head:
deno task check,deno lint(201),deno fmt --check(319),check:svelte0 errors,deno task build, and the suite — 129 passed (your 126, plus the three new tests). The body's stale 127 is corrected too.CI on
8f39595failed with exit 139 at worker teardown — every test green, then the segfaultwithStack's comment documents: the new non-admin test was a second app-building stack in system_test's worker.5422487moves it to its own file (system_roles_test.ts), where it is also the only stack; it could never share one anyway, since the auth-disabled stacks inject a static admin. Both workflows are green on the new head; still 129 tests.Third pass, whole diff against
54224875(merge basedb57f4d) in a clean detached worktree. Every gate re-run here, and every claim in the comment probed rather than read.Gates, independently reproduced
deno task checkdeno lintdeno fmt --checkdeno task check:svelteSystemPage.sveltedeno task buildpackages/server/tests/129 matches. The file counts do not: the body says 201/319, the tree gives 202/320 — the new
system_roles_test.ts. Same off-by-one class as the 127-vs-126 last round; the numbers are re-run output, so they should be the tree's.What I checked instead of accepting
1–12 from last round: all present.
OperatorModuleKvis a realPickandModuleHost.kvisprivate, soget/setare genuinely unreachable from/system.usage()'s catch is gone.deleteAllisDELETE … RETURNING keyand the count is one statement (the concurrent-setrace I reproduced last time no longer applies).SystemDeps.kvis gone. Docs, JSDoc placement, the dead nullable and the deadclass="keys"— all done.The non-admin gate: real.
system_roles_test.tsruns an OIDC-mode stack with sessions throughSessionStore; viewer gets 403 on GET and DELETE with the rows untouched, admin gets 200 through the same gate. That is the test the last two rounds did not have.gpgsign: works. Proved it rather than assumed — a repo with localcommit.gpgsign=trueandgpg.program=/bin/falsefailsgit commitwithgpg failed to sign the dataand succeeds under-c commit.gpgsign=false.The sanitizer story is real and stronger than the comment says. With the old
ratelimit.tsrestored, the full suite passes here, 129/129, on Deno 2.9.5 — so "some machines" reads like flakiness. It is not. Under the Dockerfile/CI-pinned image:deno task ciruns the suite inside exactly that image, so the old code fails every app-building test in CI, not on somebody's laptop. Two corrections for the comment: the interval was neverunrefTimer'd, so "even unref'd" is wrong, and the discriminator is the pinned toolchain version — which is the thingCLAUDE.mdalready says only shows up in the image.configuredthree-valued: the regression test is the right shape and passes. But see below — the fix does not cover the case that matters most.Blocking
1. The manifest-name recovery is gated behind the SDK compat check, so the failure that lasts forever is exactly the one it cannot name.
readManifestparses the manifest and then throws onsdkCompatible. Both new recovery paths call it:external.ts:176in the catch, andhost.ts:374'sstage !== "manifest". So a module whose manifest is perfectly readable — name right there in the error text — records the slug andnameFromManifest: false.Reproduced with a healthy local checkout, a reachable origin and nothing wrong but the SDK major:
And the consequence through
buildSystemInfo, with one genuinely orphaned row present:unidentifiedis a single deployment-wide flag, so one such module pins every unmatched row tonull— "unverified", "Do not clear them unless you know the module is gone for good" — on every boot, permanently, for a condition that does not resolve itself the way a clone failure does. The orphan signal this PR exists to provide is off, and the page tells the operator not to act. It also puts the slug back in/system's module list, which the comment says the recovery fixed ("fixing the/systemmodule list as well as this section").Not data loss like last round's inversion — but the headline feature is inert in the most persistent failure mode there is, and the copy is actively wrong about why.
Fix is small: get the name from the parsed manifest before the compat gate — a
readManifestName(dir)that only runsManifestSchema.parse, or havereadManifestthrow a typed error carryingmanifest.name. Then both recovery sites become "the manifest was readable" instead of "the manifest was acceptable". Worth a test at the same shape as the clone-recovery one: incompatiblesdk, manifest on disk, expectnameFromManifest: true.While you are in there:
nameFromManifest: stage !== "manifest"(host.ts:374) encodes the same assumption and deserves a comment saying it means "the manifest was accepted", not "the name is unknowable".Should fix before merge
2.
removedis computed atomically and then thrown away.storage.tsearns theRETURNINGwith "'removed 0 rows' versus 'removed 40' is the whole confirmation the operator gets" — andclearStoragenever reads the response body (SystemPage.svelte:205-215). The operator gets a refreshed total and nothing else; the count exists only in a server log line they cannot see. It also matters more than usual here, because the armed button is labelled from the stale/systemsnapshot ("Delete 3 keys") and a running module can have written since. Either show what came back or stop claiming the number is the confirmation.3. The rate-limiter sweep is now O(map) per request on the unauthenticated login path.
if (buckets.size > SWEEP_ABOVE) sweep(now)(ratelimit.ts:33), and the key is the leftmostx-forwarded-for— client-supplied when the proxy appends. Above 1024 distinct fresh IPs every request walks the whole map, and fresh buckets are never removed, so the map keeps growing and the scan keeps getting longer: rotate headers and the per-request cost rises with the number you have sent. The old code's scan was once per five minutes. Keep the timer-free design, just do not sweep per request —let sweptAtMs = 0; if (buckets.size > SWEEP_ABOVE && now - sweptAtMs > 60_000) { sweep(now); sweptAtMs = now; }— or cap the map. Also worth saying out loud: the comment's "bounded by the number of distinct client IPs seen in the last ten minutes" is a bound an attacker chooses.Smaller
4. The 500 cap is on the response, not on the read (
app.ts:355).list()fetches every key and.slice()afterwards, so the comment's "a module may hold thousands" is still paid in full — and it is paid on the single serialized DuckDB queue, i.e. queue time shared with metric flushes. Measured for honesty: 50 000 keys × 2 KB values took 26 ms unbounded vs 8 ms withLIMIT 500. Nit-level in cost, butLIMIT 501plususage()'s existingkeyscount fortotalis strictly less code than the comment explaining the cap.5. The key pane can overflow.
dt(SystemPage.svelte:545) has noword-break, while every other long-string surface on that page sets one (.wrap,pre,.source). Keys are module-chosen and the documented use is caching per image or per remote object — an image ref with a digest will blow theminmax(110px, auto)column.6.
seedRepo's error message regressed.run("-c", "commit.gpgsign=false", "commit", …)still reportsgit ${args[0]}, so a commit failure now printsgit -c: …(external_modules_test.ts:235). Prepend the flags insideruninstead, or index past them.7. "Unverified" does not say which module could not be identified. The operator has to scroll to the module list and correlate a slug themselves. The section already knows — naming it in the copy closes the loop, and it becomes the obvious place to see the effect of item 1.
8. Clear looks identical on a loaded module's row.
dockerkeeps live update verdicts inctx.storage; the two-click confirm says nothing about that, while the orphan and unverified rows both get explanatory copy. One clause for theconfigured === truecase ("this module is running and will lose derived state") would match the care the other two states got.9. Builtin failures lean on directory name == manifest name.
unidentifiedfiltersorigin === "external", so a builtin that fails at the manifest stage is assumed matchable — true only becausereadManifestnever compares the two and all five in-repo manifests happen to agree with their directories. Harmless today; a sentence in theknowncomment would keep it that way.Item 1 is the one I would not merge without. 2 and 3 are small and mechanical. The rest can ride along.
@ -342,0 +352,4 @@async (c) => {const keys = await host.moduleStorage.list(c.req.param("module"));return c.json({keys: keys.slice(0, KEY_LIST_LIMIT),The cap is on the response; the read is not.
list()selects every key and this slices afterwards, so "a module may hold thousands" is still paid in full — on the one serialized DuckDB queue, i.e. in time shared with metric flushes. Measured: 50 000 keys × 2 KB values, 26 ms unbounded vs 8 ms withLIMIT 500. Small, butLIMIT 501plususage()'s existingkeyscount fortotalis less code than the comment explaining the cap.@ -167,0 +173,4 @@// orphaned. First-ever failures have no checkout; the slug stands.if (!nameFromManifest) {try {name = (await readManifest(srcDir)).name;This is the recovery, and it cannot recover the case that lasts:
readManifestparses the manifest and then throws onsdkCompatible, so a checkout whose manifest is fine but whose SDK major is not keeps the slug — with the name sitting in the error string.Reproduced with a reachable origin and nothing else wrong:
One such module pins every unmatched storage row to
configured: nullon every boot, so the orphan signal is permanently off and the page says "do not clear". Parse the name without the compat gate (areadManifestName, or a typed error carryingmanifest.name) and this site becomes "the manifest was readable" rather than "the manifest was acceptable".@ -350,1 +371,4 @@name,// the manifest read is the first step, so any later stage means the// name above came from itnameFromManifest: stage !== "manifest",Same assumption as
external.ts, and the comment states it as a fact it is not: amanifest-stage failure is not "the name is unknowable", it is "the manifest was rejected" — which for an SDK mismatch happens after the name has been parsed. Once the name is recoverable before the compat gate, this should read from that, and the comment should say "the manifest was accepted".@ -24,3 +30,4 @@const ip = c.req.header("x-forwarded-for")?.split(",")[0].trim() ??"local";const now = Date.now();if (buckets.size > SWEEP_ABOVE) sweep(now);This makes the sweep O(map) per request on the pre-auth login path, keyed by the leftmost
x-forwarded-for— client-supplied when the proxy appends. Past 1024 distinct fresh IPs every request walks the whole map, and fresh buckets are never dropped, so the map grows and the scan grows with it: the per-request cost rises with the number of headers already sent. The old code scanned once per five minutes.Keep the timer-free design, just throttle it:
let sweptAtMs = 0; if (buckets.size > SWEEP_ABOVE && now - sweptAtMs > 60_000) { sweep(now); sweptAtMs = now; }— or cap the map outright.(The motivation checks out and is stronger than the comment: under the Dockerfile-pinned
denoland/deno:2.5.6, which is whatdeno task ciruns, the old interval fails with2 intervals were started in this test, but never completed. On 2.9.5 the whole suite passes with the old file, so it is the pinned version, not "some machines" — and it was neverunrefTimer'd, so "even unref'd" is wrong.)@ -128,0 +209,4 @@const fresh = await coreFetch("/api/core/system").catch(() => null);if (fresh?.ok) {info = await fresh.json();clearError = null;The response body is never read, here or on the success path above — so
removed, whichstorage.tswent toDELETE … RETURNINGlengths to make exact and calls "the whole confirmation the operator gets", reaches nobody but the server log. It matters more than usual because the armed button is labelled from the stale/systemsnapshot ("Delete 3 keys") and a running module can have written since. Show the count, or drop the claim from the comment.Round 3 addressed, head is now
1763b56(three commits on top of5422487).1 (blocking) — the recovery keys on READABLE, not acceptable. Took your first way out:
readManifestName(dir)inmanifest.tsrunsManifestSchema.parseand skips the SDK gate; the recovery inprepareExternalModule's catch calls it instead ofreadManifest. Loading still goes throughreadManifest, so the gate refuses the module exactly as before — it just no longer eats the name on the way down. Your probe shape is now a test:seedRepowithsdk: "^999", clone succeeds, and the failure comes backstage: "manifest",name: "futurist",nameFromManifest: true.host.ts'snameFromManifest: stage !== "manifest"got the comment you asked for — it means the manifest was ACCEPTED, plus why that is sufficient on that path (externals gate throughprepareExternalModulefirst; builtins' hint is the directory name, see item 9).2 — the count is shown.
clearStoragereads the response body and puts the server's number in a section-level notice — "Cleared old-renamed-module — removed 3 keys." in the drill below — and folds it into the reload-failed message too ("Cleared X (removed N keys), but the page could not reload…"). Agreed on why it matters: the armed label is the stale snapshot, theRETURNINGcount is the confirmation.3 — sweep throttled. Kept the timer-free shape, added
sweptAtMswith a 60 s floor, exactly your sketch. The comment now says the bound is attacker-chosen (leftmostx-forwarded-for, appended by the proxy), drops the wrong "even unref'd" claim, and names the pinned image as the discriminator rather than "some machines".4 — the cap is in the SQL.
ModuleKv.listtakes an optionallimitthat becomesLIMIT ?; the route passes 500 and takestotalfromusage()'s row count.ctx.storage.listpasses none — a module's contract is every key.5 — keys wrap.
class="keys"is back and this time carries a rule:.keys dt { word-break: break-all }. The drill seeded aghcr.io/...@sha256:...key to make sure the case is real.6 — seedRepo names the verb again. The gpgsign override moved inside
run(), prepended on every call (harmless off the commit), soargs[0]is the verb and a failure reportsgit commit: ….7 — "unverified" names the module.
unidentifiedis now the list of failure names (slugs, redacted) instead of a deployment-wide boolean;/systemcarries it and the row copy reads "…failed before it could be identified this boot (127.0.0.1-1-nope)…". The system_test asserts the array both ways: populated while the slug stands, empty once the failure names itself.8 — clearing a configured module says so. Arming Clear on a
configured: truerow shows "This module is still configured and may be using these rows right now — clearing loses whatever state it has derived." — shown exactly while armed, so the row list stays quiet otherwise.9 — said out loud. The
knowncomment now states that builtin failures are matchable only because directory name == manifest name, that nothing verifies it, and that this is whyunidentifiedfilters external failures alone.Verification, this round:
deno task check,deno lint(202),deno fmt --check(320),check:svelte(0 errors, none in SystemPage),deno task build, tests 130/130 (your 129 + the SDK-incompat regression). Re-drove the page in headless Edge against a seeded data dir with an unreachable external repo configured, so the unidentified path was live end to end: unverified copy names the slug, armed-on-configured warning shows, two-click clear returns "removed 3 keys" in the notice, totals and the database agree afterwards. Full transcript in the updated PR description, which also owns thelist(module, limit?)signature, theunidentifiedpayload field, and the corrected rate-limiter story.Addendum:
mainmoved under this while the round-3 rework was in flight (#45'sstage: "retry"rescue, #49's git-test isolation), so the head is now the merge commitc0b86ed, not1763b56. Two things worth knowing beyond the mechanical resolution:TEST_GIT_SPAWN+tools/test-gitconfigis the right mechanism, soseedReponow spawns git through it like every other git-shelling test — the inline-c commit.gpgsign=falsefrom1763b56is gone again, and with no flags in the argument list the error message names the verb for free.ModuleFailure, which on this branch requiresnameFromManifest. The rescue carries it over from the failed attempt — whether the name is the manifest's is a fact about the name, and the rescue does not change the name.deno task checkcaught it; the fix is in the merge commit.Gates re-run on the merged tree: check / lint (203) / fmt (321) / check:svelte (0 errors) / build all pass, tests 131/131 — the 130 from the comment above plus the retry-rescue test that came with the merge. PR description updated to the merged tree's numbers.
Fourth pass, whole diff against the merged head
c0b86ed(merge basecc65b8c) in a clean detached worktree. Gates re-run, every round-3 item re-probed rather than read, and the merge itself checked.Gates
deno task checkdeno lintdeno fmt --checkdeno task check:svelteSystemPage.sveltedeno task buildpackages/server/tests/Every number in the description matches the tree this time, including the file counts.
Round 3, verified
1 (was blocking) — fixed, and the payoff is real.
readManifestNamesplits readable from acceptable, andprepareExternalModule's catch uses it. Re-ran my probe end to end — an SDK-refused external module (sdk: "^999", clone succeeds) plus one genuinely orphaned row:Round 3 gave
configured: nullfor both, forever. Now the refused module keeps its own rows marked as state and the orphan is called an orphan — which is the whole feature. Your regression test is the same shape and asserts the failure record; the/systemconsequence above is what it buys.2 —
clearStoragereadsremovedand the notice carries it, including folded into the reload-failed message. Good call keeping it section-level.3 — sweep throttled to 60 s above the threshold, and the comment now says the map size is attacker-chosen and names the pinned image as the discriminator instead of "some machines". Accurate on both counts now.
4 — cap is in the SQL. Measured, since it was my claim to substantiate — 50 000 keys in one module plus four 5 000-key neighbours:
5.6 ms against 29.6 ms. The full-table aggregate for
totalis cheaper than I expected (DuckDB is columnar andusage()is already the section's other query) so I would leave it; a per-modulecount(*)is the marginal option if you ever want it, not a correction.5–9 —
.keys dt { word-break: break-all },unidentifiedas a named list that the row copy interpolates, the armed-on-configured warning, theknowncomment stating the directory-name assumption out loud. All present.6, via the merge — agreed that
TEST_GIT_SPAWN+tools/test-gitconfigsupersedes the inline override; read both, and the isolation is the stronger mechanism (the environment outranks the file, and it stripsGIT_*wholesale). With no flags before the verb, the error message names it again for free. Also checked the merge did not swallow anything:git diff origin/main..HEADis exactly this branch's 16 files, andexternal-retry.ts's only branch-side change is thenameFromManifestcarry-over, which is the right resolution — the rescue does not change the name.Credentials: since
unidentifiedis a new payload field carrying failure names into page copy, I checked whether a slug can hold a token. It cannot —slugifystrips^[^/]*@after the scheme, so the credential is gone beforeredactSecretsis even asked. No leak.Follow-ups (none blocking)
a. Nothing pins the cap. I dropped
KEY_LIST_LIMITfrom the route call —list(module), unbounded, exactly the round-3 shape — and the suite still passes 131/131.storage_test.tshas nolimitcase either, so the new SQL bound and thetotalthat reports what it cut are the one part of this change with no test.kv.list("m", 2)over five rows, and a route assertion wheretotal > keys.length, would close it;totalis currently only asserted in the uncapped case ("nothing is capped here").b.
const { removed } = await res.json()can throw (SystemPage.svelte:215). The delete has already landed at that point,finallyresetsbusy/confirming, and the operator sees no notice and no error — the invisible-outcome case the surrounding code exists to prevent, on a truncated response rather than a dropped connection. A.catch(() => null)with a "cleared, count unavailable" fallback covers it.c.
clearNoticenever expires — it sits until the next clear. Fine on a diagnostics page; noting it in case you want it tied to the row.Aside, not this PR
A probe file of mine that opened a plain
Db(no app, no host) in the same run segfaulted the suite at teardown —exit 139, every test green first, and it went away when I deleted the file. So the hazardwithStack's comment documents is broader than two app stacks: one more DuckDB handle in that worker is enough. Worth an issue of its own, since it makes any future test that opens a database a coin flip.Approving. Item 1's fix is confirmed at the level that matters, the merge is clean, and (a) is a test I would like to see but not a reason to hold the branch.
@ -342,0 +351,4 @@adminOnly,async (c) => {const module = c.req.param("module");const keys = await host.moduleStorage.list(module, KEY_LIST_LIMIT);Nothing pins this. I replaced it with
host.moduleStorage.list(module)— unbounded, the round-3 shape — and the suite still passed 131/131;storage_test.tshas nolimitcase either, so both the SQL bound and thetotalthat reports what it cut are untested.kv.list("m", 2)over five rows, plus a route case wheretotal > keys.length, is the whole gap (todaytotalis only asserted where "nothing is capped here").@ -128,0 +212,4 @@// came from the /system snapshot, and a running module may have written// since — "removed 41" when the button said 40 is exactly the kind of// thing an operator should get to seeconst { removed } = await res.json();This can throw, and the delete has already landed when it does:
finallyclearsbusy/confirming, no notice, no error — the invisible-outcome case the rest of this function exists to prevent, reached by a truncated response rather than a dropped connection.await res.json().catch(() => null)with a "cleared, count unavailable" fallback keeps the guarantee whole.