feat(ui): a language is a table row, not a switch statement #68

Merged

julian merged 2 commits from feat/highlight-query-driven into main 2026-09-23 13:51:22 +02:00

Conversation 3 Commits 2 Files changed 24 +2038 -119

thisilike commented 2026-09-21 18:49:35 +02:00

Owner

• Copy link
• Copy Markdown

Syntax highlighting had two languages — yaml and env — because adding one meant
writing another classify() switch by hand, mapping that grammar's node types
to colours. The grammar authors already ship that mapping as
queries/highlights.scm, so highlight.ts is now a table and nothing else:

python: { wasm: pythonWasmUrl, query: () => import("./queries/python.ts") },

Twelve languages where there were two: yaml, toml, json, ini, env, bash,
python, javascript, typescript, css, go, rust.

Where the pieces come from

WASM from @vscode/tree-sitter-wasm, already a dependency, which ships
grammars and nothing else. Two small packages join it for languages it lacks:
tree-sitter-json (592 KB) and @tree-sitter-grammars/tree-sitter-toml
(852 KB).

Queries are vendored into packages/ui/src/queries/*.ts by
deno task queries (tools/refresh-queries.ts), pinned by the version in that
tool's table, MIT throughout, provenance in each generated header. Not
imported, because a grammar's own package is mostly generated C —
tree-sitter-typescript is 38 MB for 8 KB of query text — and
docker/Dockerfile copies the build stage's /app, node_modules and all,
into the runtime image. Roughly 100 MB of parser sources would have ridden
along for nothing.

typescript concatenates JavaScript's query with TypeScript's, because
upstream's TypeScript query is only the delta over it.

What stays ours

highlight-spans.ts, which is the half a query cannot decide:

• capture name → token class by longest prefix, so string.special.path
  lands on string and an unmapped name is left unpainted. variable is
  deliberately unmapped: a JS query captures nearly every identifier, and
  colouring them all is confetti.
• captures nest rather than compete, so the innermost wins, and two
  captures of one range keep the later pattern — the precedence convention the
  query files are written against (general first, specific after).
• output is built only from escaped slices plus class-carrying spans, which
  is what keeps {@html} safe over a file nobody vetted. Classes come from a
  closed map, never from the query text.

It is pure, so it is tested. highlight.ts cannot be — its imports are Vite
instructions — which is why the split exists at all.

Tests

highlight_spans_test.ts (10) covers precedence, nesting, crossing spans,
out-of-range spans, and the invariant that highlighting never changes the
text it highlights.

highlight_grammars_test.ts (14) is the tripwire: it parses the GRAMMARS
table out of highlight.ts rather than restating it, then loads every grammar,
compiles every query against the WASM actually shipped, and asserts a sample of
each language comes back painted. A grammar bump that renames a node type fails
the suite instead of quietly turning one language back into plain text. It
caught its own regression during development, when the table's declaration line
changed shape.

Also

CodeBlock takes lang: null for plain text, and the schema's codeList
resolves through isHighlightLang — it used to render anything that was not
"env" as YAML, which with twelve grammars would be actively wrong rather than
merely unhelpful.

Verification

deno task check, check:svelte:shell (0 errors), check:svelte:modules
(0 errors), deno lint clean, deno task build green, deno task test for
packages/ui, packages/sdk, packages/shell and packages/server green.

Built output: 12 grammar WASM under /shared/assets/ (fetched only when a file
of that language is opened) and one lazy query chunk per language under
/shared/chunks/. Served locally with the right content types
(application/wasm, text/javascript).

Rendered by hand against a running server: YAML keys keep the colour they had
(no regression on compose files), TypeScript gets keywords, types, template
holes and comments, and a systemd unit gets section names, directive names and
values.

Downstream

module_filebrowser uses isHighlightLang for its file preview. That module
must not be re-pinned on a deployment until this is merged and the image is
built — a missing export fails the whole module graph, which is a blank page
rather than a degraded one.

🤖 Generated with Claude Code

Syntax highlighting had two languages — yaml and env — because adding one meant writing another `classify()` switch by hand, mapping that grammar's node types to colours. The grammar authors already ship that mapping as `queries/highlights.scm`, so `highlight.ts` is now a table and nothing else: ```ts python: { wasm: pythonWasmUrl, query: () => import("./queries/python.ts") }, ``` Twelve languages where there were two: **yaml, toml, json, ini, env, bash, python, javascript, typescript, css, go, rust**. ## Where the pieces come from **WASM** from `@vscode/tree-sitter-wasm`, already a dependency, which ships grammars and nothing else. Two small packages join it for languages it lacks: `tree-sitter-json` (592 KB) and `@tree-sitter-grammars/tree-sitter-toml` (852 KB). **Queries** are vendored into `packages/ui/src/queries/*.ts` by `deno task queries` (`tools/refresh-queries.ts`), pinned by the version in that tool's table, MIT throughout, provenance in each generated header. Not imported, because a grammar's own package is mostly generated C — `tree-sitter-typescript` is 38 MB for 8 KB of query text — and `docker/Dockerfile` copies the build stage's `/app`, `node_modules` and all, into the runtime image. Roughly 100 MB of parser sources would have ridden along for nothing. `typescript` concatenates JavaScript's query with TypeScript's, because upstream's TypeScript query is only the delta over it. ## What stays ours `highlight-spans.ts`, which is the half a query cannot decide: - capture name → token class by **longest prefix**, so `string.special.path` lands on `string` and an unmapped name is left unpainted. `variable` is deliberately unmapped: a JS query captures nearly every identifier, and colouring them all is confetti. - captures **nest** rather than compete, so the innermost wins, and two captures of one range keep the later pattern — the precedence convention the query files are written against (general first, specific after). - output is built **only** from escaped slices plus class-carrying spans, which is what keeps `{@html}` safe over a file nobody vetted. Classes come from a closed map, never from the query text. It is pure, so it is tested. `highlight.ts` cannot be — its imports are Vite instructions — which is why the split exists at all. ## Tests `highlight_spans_test.ts` (10) covers precedence, nesting, crossing spans, out-of-range spans, and the invariant that **highlighting never changes the text it highlights**. `highlight_grammars_test.ts` (14) is the tripwire: it parses the `GRAMMARS` table out of `highlight.ts` rather than restating it, then loads every grammar, compiles every query against the WASM actually shipped, and asserts a sample of each language comes back painted. A grammar bump that renames a node type fails the suite instead of quietly turning one language back into plain text. It caught its own regression during development, when the table's declaration line changed shape. ## Also `CodeBlock` takes `lang: null` for plain text, and the schema's `codeList` resolves through `isHighlightLang` — it used to render anything that was not `"env"` as YAML, which with twelve grammars would be actively wrong rather than merely unhelpful. ## Verification `deno task check`, `check:svelte:shell` (0 errors), `check:svelte:modules` (0 errors), `deno lint` clean, `deno task build` green, `deno task test` for `packages/ui`, `packages/sdk`, `packages/shell` and `packages/server` green. Built output: 12 grammar WASM under `/shared/assets/` (fetched only when a file of that language is opened) and one lazy query chunk per language under `/shared/chunks/`. Served locally with the right content types (`application/wasm`, `text/javascript`). Rendered by hand against a running server: YAML keys keep the colour they had (no regression on compose files), TypeScript gets keywords, types, template holes and comments, and a systemd unit gets section names, directive names and values. ## Downstream `module_filebrowser` uses `isHighlightLang` for its file preview. That module must not be re-pinned on a deployment until this is merged and the image is built — a missing export fails the whole module graph, which is a blank page rather than a degraded one. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

thisilike added 1 commit 2026-09-21 18:49:35 +02:00

feat(ui): a language is a table row, not a switch statement ... 4823c806d2

The kit classified tree-sitter node types by hand, which is why it had two
languages: each new one meant another switch nobody but its author could
review. A grammar already ships that mapping — `highlights.scm`, written by
the people who wrote the grammar — so highlight.ts is now a table of (WASM,
query), and twelve languages fit where two did.

The queries are vendored by tools/refresh-queries.ts rather than imported. A
grammar's npm package is mostly generated C — tree-sitter-typescript is 38 MB
for 8 KB of query text — and docker/Dockerfile copies the build stage's /app,
node_modules included, into the runtime image. The WASM does come from
packages, because @vscode/tree-sitter-wasm ships grammars and nothing else;
json and toml are the two small additions for languages it lacks.

What a capture NAME means to this design system stays ours, in
highlight-spans.ts: prefix-matched to a token class, unmapped names left
unpainted (a query captures nearly every identifier, and colouring them all
is confetti), captures nested so the innermost wins, and output built only
from escaped slices plus class-carrying spans — which is what keeps {@html}
safe over a file nobody vetted. That file is pure, so it is tested;
highlight.ts cannot be, its imports being bundler instructions.
highlight_grammars_test.ts closes the gap by parsing the table out of the
source and compiling every query against the grammar actually shipped, which
is the only thing that catches a WASM bump renaming a node type.

CodeBlock takes lang: null for plain text, and the schema's codeList now
recognises every language the kit ships instead of rendering anything that
was not "env" as YAML.

thisilike requested review from julian 2026-09-21 18:49:44 +02:00

julian was assigned by thisilike 2026-09-21 18:49:53 +02:00

julian requested changes 2026-09-22 12:14:07 +02:00

Dismissed

julian left a comment

• Copy link
• Copy Markdown

Ran both new test files on the branch (24/24 green) and probed query.captures() ordering on Python Foo() / MAX: captures arrive in ascending pattern index, so "later pattern wins" in renderSpans is correct. The highlighting itself looks right. One real problem, and it undoes the PR's own stated reason for vendoring.

Bug — tools/refresh-queries.ts, ensureCached: deno info npm:<spec> writes every query-source package into deno.lock. deno install --frozen (docker/Dockerfile L19) then installs all of them into node_modules, and COPY --from=build /app ships them in the runtime image. Measured on a fresh checkout of this branch:

node_modules/.deno/tree-sitter-typescript@0.23.2                 38M
node_modules/.deno/@tree-sitter-grammars+tree-sitter-markdown@0.3.2  7.9M
tree-sitter-{bash,css,go,javascript,python,rust,typescript} + tree-sitter@0.21.1 — 107M total

Fix: deno info --no-lock npm:<spec> in ensureCached, then scrub deno.lock of the tree-sitter-{bash,css,go,javascript,python,rust,typescript} and tree-sitter@0.21.1 entries (json/yaml/toml stay — those are real deps), and verify ls node_modules/.deno after deno install --frozen shows only the WASM packages.

Risk — deno.lock: stray @tree-sitter-grammars/tree-sitter-markdown@0.3.2 and tree-sitter-javascript@0.23.1, neither in the LANGS table. Same scrub.

Nit — tools/refresh-queries.ts L134: the deno fmt result is unchecked; a failed fmt silently leaves CI-failing files. Check .success and throw.

Nit — tools/refresh-queries.ts L114: cache path hardcodes registry.npmjs.org; breaks under NPM_CONFIG_REGISTRY. Fine for a dev tool, worth a comment.

Everything else LGTM: renderSpans nesting and escaping are sound, Object.hasOwn guard is right, the cached rejection in load() is a good call, and the grammar tripwire test does what it claims.

Ran both new test files on the branch (24/24 green) and probed `query.captures()` ordering on Python `Foo()` / `MAX`: captures arrive in ascending pattern index, so "later pattern wins" in `renderSpans` is correct. The highlighting itself looks right. One real problem, and it undoes the PR's own stated reason for vendoring. **Bug — `tools/refresh-queries.ts`, `ensureCached`:** `deno info npm:<spec>` writes every query-source package into `deno.lock`. `deno install --frozen` (docker/Dockerfile L19) then installs all of them into `node_modules`, and `COPY --from=build /app` ships them in the runtime image. Measured on a fresh checkout of this branch: ``` node_modules/.deno/tree-sitter-typescript@0.23.2 38M node_modules/.deno/@tree-sitter-grammars+tree-sitter-markdown@0.3.2 7.9M tree-sitter-{bash,css,go,javascript,python,rust,typescript} + tree-sitter@0.21.1 — 107M total ``` Fix: `deno info --no-lock npm:<spec>` in `ensureCached`, then scrub `deno.lock` of the `tree-sitter-{bash,css,go,javascript,python,rust,typescript}` and `tree-sitter@0.21.1` entries (json/yaml/toml stay — those are real deps), and verify `ls node_modules/.deno` after `deno install --frozen` shows only the WASM packages. **Risk — `deno.lock`:** stray `@tree-sitter-grammars/tree-sitter-markdown@0.3.2` and `tree-sitter-javascript@0.23.1`, neither in the `LANGS` table. Same scrub. **Nit — `tools/refresh-queries.ts` L134:** the `deno fmt` result is unchecked; a failed fmt silently leaves CI-failing files. Check `.success` and throw. **Nit — `tools/refresh-queries.ts` L114:** cache path hardcodes `registry.npmjs.org`; breaks under `NPM_CONFIG_REGISTRY`. Fine for a dev tool, worth a comment. Everything else LGTM: `renderSpans` nesting and escaping are sound, `Object.hasOwn` guard is right, the cached rejection in `load()` is a good call, and the grammar tripwire test does what it claims.

thisilike added 1 commit 2026-09-23 10:44:19 +02:00

fix(ui): fetching a query source no longer makes it a dependency ... 555f3ee507

`deno info npm:<spec>` records every package it resolves in deno.lock,
and `deno install --frozen` in the image installs whatever the lock
names. The seven C grammars the generator only reads a text file from,
the `tree-sitter` native binding they pull in as a peer, and two strays
(`tree-sitter-markdown`, `tree-sitter-javascript@0.23.1`) added 106 MB to
node_modules, which the runtime stage copies wholesale — the exact cost
vendoring the queries was meant to avoid.

The generator now fetches with --no-lock, and the lock is scrubbed back
to the three grammars whose WASM is actually imported (yaml, toml,
json), resolved without the optional `tree-sitter` peer as on main.
Fresh `deno install --frozen`: 348 MB -> 242 MB, node_modules/.deno
holds only the WASM packages; a non-frozen install leaves the lock
byte-identical.

Also: a failed `deno fmt` of the generated files now fails the run
instead of leaving output CI rejects, and the registry path assumption
is noted.

thisilike commented 2026-09-23 10:44:29 +02:00

Author

Owner

• Copy link
• Copy Markdown

Thanks. I checked each point before changing anything. All four hold, and they're fixed in 555f3ee.

Bug: ensureCached writing to the lock. Confirmed. On a fresh git archive HEAD of 4823c80, deno install --frozen gave a 348M node_modules, with tree-sitter-typescript@0.23.2 at 38M, tree-sitter-bash at 20M, tree-sitter-rust at 15M, and so on. ensureCached now calls deno info --no-lock (the flag exists in 2.9.x). I re-ran deno task queries afterwards: deno.lock came out byte-identical, and so did the generated queries.

Risk: stray lock entries. Scrubbed. One correction to the suggested fix: deleting tree-sitter@0.21.1 on its own leaves the lock corrupt (Could not find '@tree-sitter-grammars/tree-sitter-toml@0.7.0' in the list of packages). json, yaml and toml were resolved with it as their optional peer (0.7.0_tree-sitter@0.21.1). So I re-keyed those three without the peer suffix, which is how origin/main already had yaml. The yaml block is now identical to main's. I edited the text directly so the rest of the lock keeps its formatting and the diff shows only tree-sitter lines (+8/−138).

I checked the result on a fresh copy:

• deno install --frozen passes, and node_modules/.deno contains only tree-sitter-{json,toml,yaml}, @vscode/tree-sitter-wasm and web-tree-sitter from the tree-sitter family.
• node_modules went from 348M to 242M.
• A plain deno install without --frozen leaves the lock byte-identical, so it's in canonical form.
• deno task check passes.

Nit: unchecked deno fmt. Now checks .success and throws with fmt's stderr.

Nit: hardcoded registry. Added a one-line comment. I didn't change the code, since this is a dev tool.

Local results: deno fmt --check and deno lint clean, deno task check clean, deno task test packages/ tools/ 570 passed / 0 failed / 1 ignored.

Thanks. I checked each point before changing anything. All four hold, and they're fixed in 555f3ee. **Bug: `ensureCached` writing to the lock.** Confirmed. On a fresh `git archive HEAD` of 4823c80, `deno install --frozen` gave a 348M `node_modules`, with `tree-sitter-typescript@0.23.2` at 38M, `tree-sitter-bash` at 20M, `tree-sitter-rust` at 15M, and so on. `ensureCached` now calls `deno info --no-lock` (the flag exists in 2.9.x). I re-ran `deno task queries` afterwards: `deno.lock` came out byte-identical, and so did the generated queries. **Risk: stray lock entries.** Scrubbed. One correction to the suggested fix: deleting `tree-sitter@0.21.1` on its own leaves the lock corrupt (`Could not find '@tree-sitter-grammars/tree-sitter-toml@0.7.0' in the list of packages`). json, yaml and toml were resolved *with* it as their optional peer (`0.7.0_tree-sitter@0.21.1`). So I re-keyed those three without the peer suffix, which is how `origin/main` already had yaml. The yaml block is now identical to main's. I edited the text directly so the rest of the lock keeps its formatting and the diff shows only tree-sitter lines (+8/−138). I checked the result on a fresh copy: - `deno install --frozen` passes, and `node_modules/.deno` contains only `tree-sitter-{json,toml,yaml}`, `@vscode/tree-sitter-wasm` and `web-tree-sitter` from the tree-sitter family. - `node_modules` went from 348M to 242M. - A plain `deno install` without `--frozen` leaves the lock byte-identical, so it's in canonical form. - `deno task check` passes. **Nit: unchecked `deno fmt`.** Now checks `.success` and throws with fmt's stderr. **Nit: hardcoded registry.** Added a one-line comment. I didn't change the code, since this is a dev tool. Local results: `deno fmt --check` and `deno lint` clean, `deno task check` clean, `deno task test packages/ tools/` 570 passed / 0 failed / 1 ignored.

thisilike requested review from julian 2026-09-23 10:44:59 +02:00

julian approved these changes 2026-09-23 13:50:04 +02:00

julian left a comment

• Copy link
• Copy Markdown

Re-reviewed at 555f3ee. All four items from the previous review are resolved, and I checked them by running the code, not just reading it:

• --no-lock in ensureCached: ran deno run -A tools/refresh-queries.ts in a clean worktree. It exits 0 and leaves git status empty, so deno.lock is untouched and the vendored queries regenerate byte-identical.
• Lock scrub: the only tree-sitter entries left in deno.lock are the real deps (toml, yaml, json, @vscode/tree-sitter-wasm, web-tree-sitter). The stray markdown and javascript@0.23.1 entries are gone. After deno install --frozen, node_modules/.deno contains only those packages and none of the ~107M of parser sources.
• fmt result checked: it now throws with stderr on failure.
• Registry assumption: it has a comment now.

I also swept the full diff again. {@html} output is built only from escaped slices plus classes from the closed CLASSES map. The CodeBlock callers (CodeListBlock through isHighlightLang, and the docker StackDetailPage with hardcoded langs) are compatible with lang: null.

LGTM. Reminder from the PR body: re-pin module_filebrowser only after this image is built.

Re-reviewed at 555f3ee. All four items from the previous review are resolved, and I checked them by running the code, not just reading it: - **`--no-lock` in `ensureCached`:** ran `deno run -A tools/refresh-queries.ts` in a clean worktree. It exits 0 and leaves `git status` empty, so `deno.lock` is untouched and the vendored queries regenerate byte-identical. - **Lock scrub:** the only tree-sitter entries left in `deno.lock` are the real deps (toml, yaml, json, `@vscode/tree-sitter-wasm`, `web-tree-sitter`). The stray markdown and javascript@0.23.1 entries are gone. After `deno install --frozen`, `node_modules/.deno` contains only those packages and none of the ~107M of parser sources. - **fmt result checked:** it now throws with stderr on failure. - **Registry assumption:** it has a comment now. I also swept the full diff again. `{@html}` output is built only from escaped slices plus classes from the closed `CLASSES` map. The CodeBlock callers (CodeListBlock through `isHighlightLang`, and the docker StackDetailPage with hardcoded langs) are compatible with `lang: null`. LGTM. Reminder from the PR body: re-pin `module_filebrowser` only after this image is built.

julian merged commit ac2ccb3ad7 into main 2026-09-23 13:51:22 +02:00

julian deleted branch feat/highlight-query-driven 2026-09-23 13:51:22 +02:00

julian referenced this pull request from a commit 2026-09-23 13:51:23 +02:00

Merge pull request 'feat(ui): a language is a table row, not a switch statement' (#68) from feat/highlight-query-driven into main

Sign in to join this conversation.

Reviewers

No reviewers

julian

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

julian

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

OpsDeck/core!68

No description provided.