An external module whose git URL doesn't resolve at startup is dropped for the container's lifetime, with no retry and nothing that tells the operator #26

New issue

Closed

opened 2026-08-11 09:30:31 +02:00 by julian · 0 comments

julian commented

2026-08-11 09:30:31 +02:00

Owner

A DNS failure at container start (fatal: Could not resolve host: git.imhof.cloud) makes an external module disappear. Not crash, not degrade — disappear: no sidebar entry, no overview widget, nothing on the home page that differs from a module that was never configured. The failure is recoverable seconds later and nothing recovers it.

What happens now

prepareExternalModules (packages/server/src/modules/external.ts:27) loops over config.externalModules, and the whole body is one try (external.ts:47-87). syncRepo shells out to git clone (external.ts:101); on a resolver that isn't up yet, git exits non-zero, git() throws (external.ts:143), and the catch records a ModuleFailure with stage: "clone" and logs external module skipped. That is the entire handling. The comment above the function says failures "never abort startup" — correct and deliberate, but "never abort" was implemented as "never mention again".

Three things compound:

One attempt, ever. prepareExternalModules is called once, at packages/server/main.ts:137. Nothing re-runs it. git clone has no internal retry, and the AbortSignal.timeout(120_000) at external.ts:139 is a deadline, not a retry. A resolver that comes up two seconds later changes nothing until someone restarts the container.

Nothing notifies. NotificationStore is constructed at main.ts:76, ahead of module prep, and is not module-scoped by construction — create(module, n) takes the name as an argument. But nothing in core ever writes one; notify is only handed out to modules through ctx (host.ts:328). So the notification bell, which is exactly the surface this belongs on, stays empty.

The one place it does show is unlisted on purpose. main.ts:140 feeds the failures to host.recordFailure, and /system renders them with the stage (SystemPage.svelte:309). That page is deliberately kept out of the sidebar and out of the palette's default list. It answers "why did this break" for someone who already knows something broke. It does not tell anyone that something broke.

Net effect: the signal is one line in container stdout and a page nobody visits unprompted.

The worse case: a blip demotes a working module

Once a repo has been cloned successfully, later boots take the fetch branch (external.ts:103) with a valid checkout on disk and, usually, a warm frontend build cached by commit under /data/modules/dist/<name>/.commit. A fetch that fails on DNS still throws, still propagates out of syncRepo, still skips the module entirely.

So a transient network failure takes a module that could have run its last known commit and unloads it instead. The code needed to serve it is sitting in the data dir. That's the part I'd fix first regardless of what happens with retries.

Why the transient case is the common one

The container comes up with the rest of the compose stack. A forge on the same host that hasn't finished starting, a resolver not ready yet, a WireGuard link still handshaking — all resolve on their own within seconds, and all currently cost a module until the next manual restart.

Not every failure deserves a retry

stage already distinguishes clone / manifest / duplicate / frontend-build / backend, and only the clone stage is network-shaped. But stage: "clone" covers both "cannot resolve host" and "the remote said 403", and the two want opposite treatment — retrying an auth rejection just adds a multiple of the backoff to every startup. What actually separates them is git's stderr, which today only survives as a message string on the failure record (external.ts:78).

Options

Not prescribing one; they're independent and trade differently.

Fall back to the existing checkout when a fetch fails. Only applies once a repo has cloned successfully, which is every boot after the first. No startup delay, no policy question, and it turns the common case from "module gone" into "module running a possibly stale commit". Needs the staleness to be visible somewhere — a failure record that doesn't disable the module is a shape ModuleFailure doesn't currently have.
Bounded retry with backoff inside syncRepo, gated on the error looking like a network failure. Cheap, but it runs on the startup path — Deno.serve is at main.ts:163, after module prep — so every second of backoff is a second the whole UI is unreachable. Keep it small (a few attempts, single-digit seconds) or move module prep off the startup path first.
Raise a notification per recorded failure. The store is already live at that point, so this is close to a one-liner: notifications.create("core", …) for each entry in external.failed, keyed by module so a restart loop doesn't stack duplicates. Two things to decide: whether "core" is an acceptable module value on a table whose rows have so far always been a real module, and what link holds — /system is the right target, but the field is documented as a module-relative page path.
Retry in the background after serve, and load the module when it succeeds. The one that actually fixes it, and the expensive one. createApp mounts module routers in a loop over host.loaded at construction (app.ts:314) and registers the /api/* 404 catch-all immediately after (app.ts:325), so a module loaded after that point gets no API route — its requests hit the 404. Making this work means mounting module routers through an indirection that can gain entries later. Worth doing on its own merits (it's the same missing capability behind reloading a module without a restart), but it's a change to how the app is assembled, not a fix to the module loader.

3 is what stops the failure being silent. 1 is the largest reduction in how often it matters. 4 is the only one that makes it self-healing.

A DNS failure at container start (`fatal: Could not resolve host: git.imhof.cloud`) makes an external module disappear. Not crash, not degrade — disappear: no sidebar entry, no overview widget, nothing on the home page that differs from a module that was never configured. The failure is recoverable seconds later and nothing recovers it. ## What happens now `prepareExternalModules` (`packages/server/src/modules/external.ts:27`) loops over `config.externalModules`, and the whole body is one `try` (`external.ts:47-87`). `syncRepo` shells out to `git clone` (`external.ts:101`); on a resolver that isn't up yet, git exits non-zero, `git()` throws (`external.ts:143`), and the catch records a `ModuleFailure` with `stage: "clone"` and logs `external module skipped`. That is the entire handling. The comment above the function says failures "never abort startup" — correct and deliberate, but "never abort" was implemented as "never mention again". Three things compound: **One attempt, ever.** `prepareExternalModules` is called once, at `packages/server/main.ts:137`. Nothing re-runs it. `git clone` has no internal retry, and the `AbortSignal.timeout(120_000)` at `external.ts:139` is a deadline, not a retry. A resolver that comes up two seconds later changes nothing until someone restarts the container. **Nothing notifies.** `NotificationStore` is constructed at `main.ts:76`, ahead of module prep, and is not module-scoped by construction — `create(module, n)` takes the name as an argument. But nothing in core ever writes one; `notify` is only handed out to modules through `ctx` (`host.ts:328`). So the notification bell, which is exactly the surface this belongs on, stays empty. **The one place it does show is unlisted on purpose.** `main.ts:140` feeds the failures to `host.recordFailure`, and `/system` renders them with the stage (`SystemPage.svelte:309`). That page is deliberately kept out of the sidebar and out of the palette's default list. It answers "why did this break" for someone who already knows something broke. It does not tell anyone that something broke. Net effect: the signal is one line in container stdout and a page nobody visits unprompted. ## The worse case: a blip demotes a working module Once a repo has been cloned successfully, later boots take the `fetch` branch (`external.ts:103`) with a valid checkout on disk and, usually, a warm frontend build cached by commit under `/data/modules/dist/<name>/.commit`. A fetch that fails on DNS still throws, still propagates out of `syncRepo`, still skips the module entirely. So a transient network failure takes a module that could have run its last known commit and unloads it instead. The code needed to serve it is sitting in the data dir. That's the part I'd fix first regardless of what happens with retries. ## Why the transient case is the common one The container comes up with the rest of the compose stack. A forge on the same host that hasn't finished starting, a resolver not ready yet, a WireGuard link still handshaking — all resolve on their own within seconds, and all currently cost a module until the next manual restart. ## Not every failure deserves a retry `stage` already distinguishes clone / manifest / duplicate / frontend-build / backend, and only the clone stage is network-shaped. But `stage: "clone"` covers both "cannot resolve host" and "the remote said 403", and the two want opposite treatment — retrying an auth rejection just adds a multiple of the backoff to every startup. What actually separates them is git's stderr, which today only survives as a message string on the failure record (`external.ts:78`). ## Options Not prescribing one; they're independent and trade differently. 1. **Fall back to the existing checkout when a fetch fails.** Only applies once a repo has cloned successfully, which is every boot after the first. No startup delay, no policy question, and it turns the common case from "module gone" into "module running a possibly stale commit". Needs the staleness to be visible somewhere — a failure record that doesn't disable the module is a shape `ModuleFailure` doesn't currently have. 2. **Bounded retry with backoff inside `syncRepo`,** gated on the error looking like a network failure. Cheap, but it runs on the startup path — `Deno.serve` is at `main.ts:163`, after module prep — so every second of backoff is a second the whole UI is unreachable. Keep it small (a few attempts, single-digit seconds) or move module prep off the startup path first. 3. **Raise a notification per recorded failure.** The store is already live at that point, so this is close to a one-liner: `notifications.create("core", …)` for each entry in `external.failed`, keyed by module so a restart loop doesn't stack duplicates. Two things to decide: whether `"core"` is an acceptable `module` value on a table whose rows have so far always been a real module, and what `link` holds — `/system` is the right target, but the field is documented as a module-relative page path. 4. **Retry in the background after serve, and load the module when it succeeds.** The one that actually fixes it, and the expensive one. `createApp` mounts module routers in a loop over `host.loaded` at construction (`app.ts:314`) and registers the `/api/*` 404 catch-all immediately after (`app.ts:325`), so a module loaded after that point gets no API route — its requests hit the 404. Making this work means mounting module routers through an indirection that can gain entries later. Worth doing on its own merits (it's the same missing capability behind reloading a module without a restart), but it's a change to how the app is assembled, not a fix to the module loader. 3 is what stops the failure being silent. 1 is the largest reduction in how often it matters. 4 is the only one that makes it self-healing.

julian referenced this issue from a commit

2026-08-11 11:44:59 +02:00

feat(modules): alert on disabled modules, and retry unreachable repos

julian referenced this issue from a pull request that will close it,

2026-08-11 11:45:30 +02:00

Alert on disabled modules, and retry repos that were unreachable at startup #27

julian self-assigned this

2026-08-11 11:55:20 +02:00

julian closed this issue

2026-08-11 15:44:43 +02:00

julian referenced this issue from a commit

2026-08-11 15:44:45 +02:00