A crash-killed clone wedges the module work tree permanently #28

New issue

Closed

opened 2026-08-11 15:44:12 +02:00 by julian · 0 comments

julian commented

2026-08-11 15:44:12 +02:00

Owner

Follow-up from the review of #27 (approved; not a blocker there).

syncRepo (packages/server/src/modules/external.ts:186-199) picks clone-vs-fetch from the presence of <srcDir>/.git, never from its validity:

const exists = await Deno.stat(`${dir}/.git`).then(() => true).catch(() => false);
if (!exists) await git(["clone", "--quiet", ref.url, dir]);
else await git(["-C", dir, "fetch", "--quiet", "--tags", "origin"]);

Measured behaviour on an interrupted clone:

kill	leftover in `srcDir`
SIGTERM — what the `AbortController` in `git()` sends	none; git's own handler removes the partial directory
SIGKILL	`. .. .git`

So the 120 s self-kill added in #27 is clean. SIGKILL is the other case: an OOM kill, docker stop past the grace period, a host reset mid-clone. With a partial .git on disk, every git command in syncRepo answers

fatal: not a git repository (or any parent up to mount point /)

That matches no fragment in TRANSIENT_GIT_ERRORS, so isRetryableFailure is false, the retry loop never takes the repo, and the module stays disabled on that boot and every boot after it — with no action named in the notification that would clear it. Manual rm -rf /data/modules/src/<slug> is the only recovery, and nothing tells the operator that.

The wedge predates #27. It is worth closing anyway because #27 is what makes "the container heals itself" the contract, and this is the single clone-stage failure it can never heal.

Suggested fix

Probe with git -C <dir> rev-parse --git-dir instead of stat(.git).
On a non-zero answer, remove srcDir and fall through to a fresh clone.
Test: create <srcDir>/.git as a directory containing nothing (or a regular file), assert prepareExternalModule still ends up at the right commit.

Follow-up from the review of #27 (approved; not a blocker there). `syncRepo` (`packages/server/src/modules/external.ts:186-199`) picks clone-vs-fetch from the *presence* of `<srcDir>/.git`, never from its validity: ```ts const exists = await Deno.stat(`${dir}/.git`).then(() => true).catch(() => false); if (!exists) await git(["clone", "--quiet", ref.url, dir]); else await git(["-C", dir, "fetch", "--quiet", "--tags", "origin"]); ``` Measured behaviour on an interrupted clone: | kill | leftover in `srcDir` | |---|---| | SIGTERM — what the `AbortController` in `git()` sends | none; git's own handler removes the partial directory | | SIGKILL | `. .. .git` | So the 120 s self-kill added in #27 is clean. SIGKILL is the other case: an OOM kill, `docker stop` past the grace period, a host reset mid-clone. With a partial `.git` on disk, every git command in `syncRepo` answers ``` fatal: not a git repository (or any parent up to mount point /) ``` That matches no fragment in `TRANSIENT_GIT_ERRORS`, so `isRetryableFailure` is false, the retry loop never takes the repo, and the module stays disabled on that boot and every boot after it — with no action named in the notification that would clear it. Manual `rm -rf /data/modules/src/<slug>` is the only recovery, and nothing tells the operator that. The wedge predates #27. It is worth closing anyway because #27 is what makes "the container heals itself" the contract, and this is the single clone-stage failure it can never heal. ### Suggested fix - Probe with `git -C <dir> rev-parse --git-dir` instead of `stat(.git)`. - On a non-zero answer, remove `srcDir` and fall through to a fresh clone. - Test: create `<srcDir>/.git` as a directory containing nothing (or a regular file), assert `prepareExternalModule` still ends up at the right commit.

julian referenced this issue from a commit

2026-08-11 15:44:45 +02:00

Alert on disabled modules, and retry repos that were unreachable at startup (#27)

julian referenced this issue from a commit

2026-08-16 13:31:09 +02:00

fix(modules): re-clone a work tree git cannot use, instead of wedging on it

julian referenced this issue from a pull request that will close it,

2026-08-16 13:31:24 +02:00

fix(modules): re-clone a work tree git cannot use, instead of wedging on it #40

julian referenced this issue from a commit

2026-08-21 23:50:56 +02:00

fix(modules): recover a wedged work tree by re-cloning on failure, not by probing harder