WIP: build(dev-rig): a disposable server to test the compose path against

thisilike commented

2026-08-09 23:01:42 +02:00

Owner

First of four. Stands up a throwaway server with its own dockerd and OpsDeck nested inside it using the production mounts, so the compose path and the --ansi never progress parsing in compose_progress.ts are actually exercised. The nesting is the point: hostexec only considers the chroot strategy when the host root is set and is not /, so running OpsDeck as the rig's own PID 1 drops the probe to ["direct"] and silently tests the engine-API fallback instead.

No app code. docker/dev-rig/ and one paragraph in CLAUDE.md.

Three seeded stacks in a git work tree (web, jobs, broken), plus a stub IdP under RIG_OIDC=1 so role-dependent paths are testable.

Also carries the fixes from the review on the first round: the exported image no longer lands in the production build context, a failed IdP start is fatal instead of booting against a dead issuer, the stub implements the refresh grant, SIGTERM shuts the nested engine down instead of letting docker SIGKILL a mid-write DuckDB, login.sh fails loudly at whichever leg broke, and down.sh removes the rig and every volume it owns.

Verified on a cold rig: overlay2, hostExec chroot with hostRoot=/host/root, all three stacks up, graceful stop in ~3s, and under RIG_OIDC a viewer 403 / admin 200 split with the refresh grant surviving repeated revalidation.

First of four. Stands up a throwaway server with its own dockerd and OpsDeck nested inside it using the production mounts, so the compose path and the `--ansi never` progress parsing in `compose_progress.ts` are actually exercised. The nesting is the point: `hostexec` only considers the `chroot` strategy when the host root is set and is not `/`, so running OpsDeck as the rig's own PID 1 drops the probe to `["direct"]` and silently tests the engine-API fallback instead. No app code. `docker/dev-rig/` and one paragraph in CLAUDE.md. Three seeded stacks in a git work tree (`web`, `jobs`, `broken`), plus a stub IdP under `RIG_OIDC=1` so role-dependent paths are testable. Also carries the fixes from the review on the first round: the exported image no longer lands in the production build context, a failed IdP start is fatal instead of booting against a dead issuer, the stub implements the refresh grant, SIGTERM shuts the nested engine down instead of letting docker SIGKILL a mid-write DuckDB, `login.sh` fails loudly at whichever leg broke, and `down.sh` removes the rig and every volume it owns. Verified on a cold rig: overlay2, `hostExec chroot` with `hostRoot=/host/root`, all three stacks up, graceful stop in ~3s, and under `RIG_OIDC` a viewer 403 / admin 200 split with the refresh grant surviving repeated revalidation.

thisilike added 16 commits

2026-08-09 23:01:42 +02:00

build(dev-rig): a disposable server to test the compose path against 10a7f987f7

`deno task check` and the unit tests cannot see the defects that matter most
in the docker module — a progress bar that walks backwards, a spinner that
never stops, a compose file rewritten wrongly. Those need a real engine, real
compose files and a real pull.

This is one privileged throwaway container running its own dockerd, with
OpsDeck nested inside it using the production mounts. The nesting is the
point: only then does `chroot /host/root` land somewhere that is not our own
mount namespace, and only then is the compose path taken at all — otherwise
hostexec's probe refuses every strategy and the engine-API fallback runs
instead, leaving `--progress json` and the pin route untested.

Two consequences of standing a container up as a host, both handled in the
entrypoint: `/` is remounted rshared so the nested bind has something to
propagate from, and the container markers are deleted so the probe believes
the chroot left the container.

Includes a hardcoded OIDC provider, because OPSDECK_AUTH=disabled injects a
static admin and leaves every role-dependent path untestable.

fix(ui): let Enter reach the focused button in ConfirmDialog 6732538d31

The keydown handler was bound to window and confirmed on Enter whatever
had focus, calling preventDefault() so the focused button never saw the
key itself. That defeated the danger variant's focus-on-Cancel: the
comment there says the dangerous button should never be one stray Enter
away, and it was exactly one.

Behind a queue of prompts it is worse than one. Each answered prompt
mounts the next, so a held Enter drains the queue at the key repeat
rate — one destructive action per repeat, none of them read.

Drop the branch. A focused button activates on Enter natively, which is
the documented behaviour without the reach: Enter confirms when Confirm
has focus, and cancels on the danger variant, where Cancel has it.

fix(docker): gate queued confirmations on the compose project lock 0e50ec3d39

Answering a depends_on prompt started the update without waiting for it
and presented the next queued prompt in the same microtask. But the
server reserves the whole compose project for the length of an update,
and that check runs before the dependents check — so the second
acknowledgement came back 409 "update already running", with no
needsConfirmation field to route it. The human read a destructive
warning, accepted it, and got a raw HTTP 409 in the row's error line:
the exact outcome the queue was added to prevent.

Queue entries now carry their project, and only a prompt whose project
is free is shown. The ones waiting say so in their row instead of going
quiet, and the dialog holds a busy state while the update it just
acknowledged is being started.

The same reservation explains two more. stackLocked only tracked stack
updates although /container/:id/update takes the identical lock, so
sibling buttons stayed enabled and 409'd; it becomes projectBusy(), which
also counts per-container updates. And subscribeLive("docker:update")
registered no onResync, so a dropped channel — routine, the socket closes
after a minute hidden — lost the terminal event that is the only thing
clearing the in-flight flags, wedging every button in the project until
a reload.

fix(dev-rig): stop the rig misreporting the build, the tester and itself c045f7f0ae

Nine defects, all of them cases where the rig said one thing and did
another.

The exported app image landed in the production build context and
.dockerignore did not exclude it (docker/dev-rig/.gitignore has no
bearing on a build context). Every run baked the previous run's image
into the image being built, and its always-different bytes meant `COPY
. .` never cache-hit, so deno install, build and check ran cold on every
start. The Dockerfile's claim that what runs here is the artifact we
ship was false: it shipped a copy of itself.

The stub IdP readiness loop logged on timeout and carried on. OpsDeck
then started against a dead issuer, /api/core/health answered 200
because it is registered before the auth middleware, and up.sh printed
"ready" for a rig where every login 503s. It exits now, like the dockerd
loop next to it.

The stub only implemented authorization_code while advertising and
issuing refresh tokens. OpsDeck re-validates every session against the
IdP each OPSDECK_SESSION_REVALIDATE_MINUTES, so at 15 minutes the grant
answered invalid_grant, the session was deleted, and a role test died
looking like an OpsDeck auth bug. Adds the grant with rotation, and
RIG_REVALIDATE_MINUTES so the path is testable in seconds rather than
quarter-hours.

`exec docker logs -f` made the log follower PID 1, so docker stop
SIGKILLed dockerd and the nested DuckDB mid-write — and the retained
volume carried that torn database into the next run. There is a trap
now. The budget is shaped by measurement: OpsDeck closes its database
and exits 0 in about 150ms, several seed containers ignore SIGTERM
outright, and a full dockerd teardown does not fit in the outer ten
seconds at all. Waiting on it is what got the rig killed at 137; a
bounded wait brings the whole shutdown in at ~3s.

login.sh used no -f and checked nothing, so a failed login exited 0 with
an empty jar and every follow-up "viewer" check measured an anonymous
request instead. Each leg is checked, and the jar has to hold a session
cookie.

The nested engine ran on vfs, which has no copy-on-write and materialises
~1.6GB just to load the app image; it is overlay2 now, falling back to
vfs, and keeping whichever driver an existing volume was written with
rather than silently orphaning its images. python:3.12-slim — the image
the web stack exists to make progress bars watchable with — was missing
from the pre-pull list and was fetched inline on the readiness path.
Pulls and compose runs are concurrent, the wait is 420s and configurable,
and it fails immediately if the rig has exited.

/srv/stacks gets a volume, so it no longer resets while the engine that
built its containers persists, and --reset drops the set together.
`docker load` re-tagged rig-base every run and orphaned the previous
image with nothing to prune it.

The rig image's OPSDECK_* block was dead — the entrypoint runs OpsDeck
as a nested container with its own -e flags — and its comment was
inverted: hostexec only considers chroot when the host root is not "/",
so HOST_ROOT=/ is precisely what reduces the probe to ["direct"].

Docs last, same class of defect. CLAUDE.md and the README described
`--progress json`, a pin route and /api/mod/docker/jobs, none of which
exist on main; the flag is `--ansi never` and the 403 example now uses
/stack/:project/files, which is genuinely admin-only. The seed-stack
table no longer claims rendering that is not implemented — exited maps
to serious, restarting to unknown, and there is no stack-level rollup.

fix(dev-rig): run the scripts through their failure paths fbd896cb84

Addresses the review on #2. Four of its points were already covered by
the previous commit (IdP exit 1, the dead OPSDECK_* ENV block, the
python:3.12-slim pre-pull, the volumes surviving `docker rm -f`); this
is the rest.

`mount --make-rshared /` was a warning. It is the one precondition the
whole rig rests on — without the propagation the nested rslave bind
quietly shows the wrong thing and every conclusion drawn from the rig is
worthless — so it exits.

The nested run passed OPSDECK_AUTH twice, betting on docker's
last-flag-wins for as long as that holds, mounted all of /certs
including the CA's private key when OpsDeck needs exactly ca.pem, and
left OPSDECK_HOST_ROOT riding on an image default while setting its two
siblings explicitly. One argument array now, built once, with AUTH set
in exactly one branch. The host-root bind still exposes /host/root/certs
— a real host has its secrets on it too — but nothing hands the key over
a second time.

The log follower reattaches instead of ending with the nested container,
so `docker exec opsdeck-rig docker restart opsdeck` no longer takes the
rig down with it. The README documented that wart; now it documents that
it works.

up.sh rejects unknown arguments instead of treating `-rerun` as a
ten-minute full build, skips re-exporting a tar that --rerun cannot have
changed, and publishes 9443 only under RIG_OIDC. down.sh removes the
container and every volume it owns, which is what makes "disposable"
true; --reset is the same cleanup followed by a build.

The stub's /role mutated on GET, so pasting the URL into a browser
decided who the next login was; it is POST to write, GET to read. Its
doc comment now says which of the things discovery advertises are
deliberately unchecked (client secret, client_id, redirect_uri), since
the comment already inventoried what IS checked.

login.sh checks its arguments and finishes on /api/core/me — behind the
auth middleware and open to every role, unlike /api/core/health, which
answers 200 to anybody and would have proved nothing.

Dockerfile: the apt line hardcoded bookworm while the base is whatever
production ships; it reads VERSION_CODENAME, as the same line already
does for the architecture.

Verified on a cold rig (no volumes): overlay2, only 8080 published,
hostExec chroot with hostRoot=/host/root, all three stacks up. Under
RIG_OIDC: one OPSDECK_AUTH, ca.pem the only file in the nested /certs,
GET /role read-only, login.sh reporting mapped roles, viewer 403 and
admin 200 on /stack/web/files, and the rig surviving a restart of the
nested container.

fix(server): type the timers by what setTimeout actually returns 64b2338a45

`deno task check` failed on any Deno newer than the 2.5.6 the image pins:
two TS2322s for `Type 'Timeout' is not assignable to type 'number'`. With
npm dependencies in the graph the Node typings win, so setInterval and
setTimeout return a Timeout, not a number — and the annotations said
number.

It passed in CI only because the Dockerfile builds on the pinned version,
which meant the documented `deno task check` failed for a contributor on
current Deno, in files they had not touched. ReturnType<typeof setInterval>
follows the call instead of asserting a shape, so it is right on both.

fix(server): type the websocket wrapper instead of casting it away 399f886d6f

`deno lint` reported 23 problems and nothing enforced it, so they had
accumulated. Seven of them were one function: safeUpgradeWebSocket, which
took `any` for the handler, the context, the options, the events object
and every callback argument, and was then cast to `any` again at the call
site behind a deno-lint-ignore.

It is typed now: hono's own WSEvents for the shape it must produce, the
SDK's ModuleContext["upgradeWebSocket"] for the contract it fills, and a
single documented assertion where those two meet — which is the only
place they genuinely differ, because the SDK describes the callbacks in
its own terms so modules never import hono.

The rest: three require-await middlewares now `return await next()`,
which is what makes them honestly async (the MiddlewareHandler contract
needs the promise, so dropping `async` was not an option); a regex with
two literal spaces says ` {2}`; DuckDBConnection is a type-only import;
and an unused type import is gone.

build(deps): pin the versions that were resolving implicitly 42f9221008

Three things were held in place by the lockfile rather than by anything
that says so.

Six test files imported `jsr:@std/assert` with no version at all (four
others pinned @1). Whatever the lock captured is what they got, and a
fresh resolve takes the next major. It is a workspace import now, pinned
^1, imported by bare specifier — which is what deno lint's
no-unversioned-import and no-import-prefix were both pointing at.

`deno task dev`, `build:shell` and `build:shared` ran `npm:vite` with no
range. Inside the workspace that resolves to 7.3.6 through the "vite"
import-map entry; from anywhere else it resolves to 8.2.1, and CLAUDE.md
records that Vite 8 fails under Deno for want of node:util parseEnv. The
pin was real but it lived in an entry nothing imports by bare specifier,
so it reads as dead weight. The tasks name the range themselves now.

`deno install` in the image is `--frozen`: the container is supposed to
be the lockfile made real, and without it a stale lock is resolved past
instead of failing the build.

Also here, since they are the same file: `start` and `dev:server` had
drifted from the CMD the image runs. They lacked --allow-sys=hostname
and write access to the docker socket, both of which the shipped command
has. Deno.hostname() is try/caught at both call sites, so it degraded
silently rather than crashing — the hardware module reported the host as
literally "server", and docker's self-container detection returned null.

And `deno fmt` now skips html/css/svg. Deno's formatting of those is not
stable across versions: 2.5.6 (the image, and now CI) and 2.9.x disagree
on three files in opposite directions, one wanting `<!doctype html>` and
the other `<!DOCTYPE html>`. Left in, `deno fmt --check` could only ever
pass on whichever version last ran `deno fmt`. TypeScript and Svelte
formatting agree across both.

fix(ui): show a failed load instead of a skeleton that never resolves 3a53dde0ee

Every hand-written page fetched its data as `apiFetch(...).then(...)`
with no rejection path. A 500, a dropped connection, a restarting server
— `data` stays null, the `{#if !data}` branch keeps rendering its
skeleton for as long as the page is open, and the only trace is an
unhandled rejection in the console. Nothing on screen says anything is
wrong, and nothing ever will.

Three pages were worse than having no catch: ContainerPage,
StackDetailPage and PoolDetailPage caught, handled 404 by navigating
away, then `throw e` for everything else — a rethrow inside .catch is a
rejected promise nobody awaits, so it produced the same dead skeleton
plus a warning.

The schema renderer already did this correctly (SchemaPage catches and
renders EmptyState with the error as its hint), so the mobile path
degraded properly while the desktop pages did not. Pages now match it.
The three overview widgets get one line instead: they render into a
fixed ~260px slot on the home page where an EmptyState does not fit.
StoragePage reports its two fetches separately, because smartctl can be
unavailable while the mount table is perfectly fine.

Each of these is nine or so lines of the same shape, which wants a
shared helper — but the .then bodies do page-specific work (StacksPage
prunes its confirmation queue, ContainerPage drops a stale prompt), so
a generic loader needs callbacks, and there is no browser test
framework here to prove that refactor did not change behaviour. Left as
the local pattern SchemaPage already established.

ci: verify formatting, lint, types and tests before building e19877592c

The workflow ran checkout, build, push, deploy — and nothing else. No
deno fmt --check, no deno lint, no deno test. Typecheck happened only as
a side effect of the image build, on main, after a full install and
build. So nothing was enforced, and it showed: 85 of 220 files were
unformatted and deno lint reported 23 problems.

It also only ran on push to main, meaning a pull request got no signal
at all until after it was merged and deployed.

A verify job now runs the four checks, and build depends on it but is
gated to push-on-main, so a PR is verified and stops there. Verify runs
in denoland/deno:2.5.6 — the same version docker/Dockerfile builds with,
deliberately: a toolchain difference between where code is written and
where it is validated is its own class of failure, and this repo already
had two of them (the setInterval typings, and deno fmt disagreeing with
itself on html and css).

style: format the repo with deno fmt c90221c43a

Mechanical, no behaviour. 85 of 220 files had drifted because nothing
ever checked — the CI job added in the previous commit is what stops it
happening again.

fix(zfs): declare the loadError the pools page renders 5bfd1e6065

Introduced two commits ago and never declared: the page assigned
`loadError` in .then and .catch and branched on it in the template, but
there was no `let loadError = $state(...)`. Svelte compiles a free
identifier straight through as a global reference, so `/m/zfs` threw
ReferenceError on first render and displayed nothing at all — on the
page the change was meant to make more robust.

Nothing in the pipeline sees this. The module build succeeds, and
deno check, deno lint and deno fmt --check do not read Svelte template
scope, so the verify job added in this branch would have passed it too.
A free identifier does survive minification (the minifier cannot rename
a global), so `grep loadError packages/shell/dist/modules/*/index.js`
returning nothing is a real, if narrow, check — it is empty now.

svelte-check would catch the whole class properly, but it reports 203
errors and 21 warnings on this repo today, nearly all of them module
resolution it cannot follow through Deno's import map. That is a
separate piece of work, not a drive-by.

fix(ui): clear the load error when a reload succeeds 00fbb6f952

The three detail pages set `loadError` in .catch but only ever assigned
`data` in .then, so the error was sticky. All three reload on a timer
(liveRefresh at 30 s) and on live-channel resync, which makes the
failure mode worse than a one-off: one transient 502 from a proxy
restart pins "failed to load" on screen permanently while fresh data
arrives underneath it every 30 seconds, until the user hard-reloads.

GpuPage and DemoPage fetch exactly once today so they cannot hit it, but
they get the same reset rather than leaving the trap set for whoever
adds a poll.

fix(ui): focus the safe choice when a dialog mounts while busy 482706faab

The focus effect read `open`, `variant` and the two element bindings,
but not `busy` — and both buttons are `disabled={busy}`, so .focus() on
them does nothing. It never re-ran when `busy` cleared either.

That is reachable in the queued-confirmation flow this branch added:
StacksPage renders DependentsConfirm inside `{#key confirmUpdate?.id}`
with `busy={confirming}`, so when the answer to one prompt is still in
flight and the next queued prompt becomes current, the new dialog is
created disabled, focuses nothing, and stays that way. Escape still
worked, Enter did nothing, and the Tab trap's swap sent the first Tab to
"Update anyway" — the exact invariant ("focus lands on the safe choice
for destructive actions") that justified removing the Enter handler in
the first place.

The focus effect is now separate from the keydown effect and tracks
`busy`, so it fires as soon as the buttons can take focus; splitting
them also keeps the focus-restore cleanup from running on every re-run.
The Tab trap no longer assumes focus is on one of the two buttons —
when it is on neither it goes to the safe one instead of falling
through to the dangerous one.

fix(docker): key the project lock off the id the update started with adbac1baa5

projectBusy() resolved the in-flight flag through the current container
list: does any container in this project have updatingC[c.id] set. But
updatingC is keyed by the pre-update container id and a successful
recreate mints a new one, while load() runs on the 30 s poll and on
every docker:stacks push — including the ones the update itself
provokes. Once that refresh lands, the old id is gone from `data` and
projectBusy returns false even though the server still holds
updating.has(project).

So the gate opened mid-update: sibling update buttons re-enabled, and a
queued depends_on prompt for another container in the same project was
released. The user read the destructive warning, accepted it, and got
409 "update already running" with nothing done — which is the outcome
the queue exists to prevent.

The project is captured in updatingProjectOf when the update starts,
which is the last moment the association is resolvable, and projectBusy
reads that instead of the live list. The confirmation queue uses the
same helper rather than repeating the lookup.

test(notifications): stop the ordering assertion racing the clock

Build and Deploy / verify (pull_request) Failing after 14s

Details

Build and Deploy / build (pull_request) Has been skipped

Details

cbf044965e

"create, list, dismiss round-trip" asserts newest-first, but created_at
is Date.now() at millisecond resolution and both creates routinely land
in the same millisecond, leaving ORDER BY created_at DESC to break the
tie however DuckDB feels. Measured at one failed run in six across the
full suite — enough to make the verify job added in this branch
intermittently red, which is how a CI signal gets ignored.

Pre-existing, and not something the review turned up; it surfaced while
re-running the suite. Fixed in the test rather than the query because
which of two same-millisecond notifications is "newer" is genuinely
undefined, and inventing a tiebreaker in SQL would assert a fact the
data does not carry.

55 consecutive full-suite runs since. One earlier failure in that window
went unattributed — its output was discarded — and did not reproduce, so
this closes the measured flake, not necessarily every flake.

thisilike referenced this pull request

2026-08-09 23:32:44 +02:00

feat(docker): the docker overhaul, integrated #10

thisilike commented

2026-08-09 23:33:04 +02:00

Author

Owner

Superseded by #10, which merges this branch together with the rest of the stack and resolves the collisions between them. Closing rather than merging in place: replaying these merges one PR at a time would reproduce the same conflicts and lose the resolutions. The branch is untouched.

thisilike closed this pull request

2026-08-09 23:33:04 +02:00