OpsDeck/core

Watch

Fork 0

tests: one extra DuckDB handle in the test worker can segfault the suite at teardown (exit 139) #54

New issue

Closed

opened 2026-08-21 23:21:29 +02:00 by julian · 1 comment

julian commented

2026-08-21 23:21:29 +02:00

Owner

From the aside in the approving review on #46, plus the flake noted in the approving review on #44.

A probe file that opened a plain Db (no app, no host) in the same run segfaulted the suite at teardown — exit 139, every test green first — and the failure went away when the file was deleted. So the hazard withStack's comment documents is broader than "two app stacks": one more DuckDB handle in that worker is enough. Any future test that opens a database directly is a coin flip.

Likely related: the #44 review measured the full suite clean on 20 of 21 runs with one unreproducible failure — same rate across two commits, reads as a pre-existing suite flake, and CI has shown the same exit-139-in-teardown shape before.

Worth either serializing DB-opening tests into their own worker, documenting the constraint where a test author will see it, or finding the teardown ordering bug in the DuckDB binding.

Sources: #46 (comment) (aside), #44 (comment) (determinism note).

From the aside in the approving review on #46, plus the flake noted in the approving review on #44. A probe file that opened a plain `Db` (no app, no host) in the same run segfaulted the suite at teardown — exit 139, every test green first — and the failure went away when the file was deleted. So the hazard `withStack`'s comment documents is broader than "two app stacks": one more DuckDB handle in that worker is enough. Any future test that opens a database directly is a coin flip. Likely related: the #44 review measured the full suite clean on 20 of 21 runs with one unreproducible failure — same rate across two commits, reads as a pre-existing suite flake, and CI has shown the same exit-139-in-teardown shape before. Worth either serializing DB-opening tests into their own worker, documenting the constraint where a test author will see it, or finding the teardown ordering bug in the DuckDB binding. Sources: https://git.imhof.cloud/OpsDeck/core/pulls/46#issuecomment-1034 (aside), https://git.imhof.cloud/OpsDeck/core/pulls/44#issuecomment-1035 (determinism note).

julian referenced this issue from a commit

2026-09-06 22:37:06 +02:00

deps(image): denoland/deno 2.9.5 -> 2.9.6, which is the exit-139 flake

julian closed this issue

2026-09-06 22:37:06 +02:00

julian commented

2026-09-06 22:37:35 +02:00

Author

Owner

Root-caused from a core dump. It is not the handle count, and the fix is the Deno pin — 229af54 bumps denoland/deno 2.9.5 → 2.9.6 (also closing #58 and #59, which had been sitting there with the fix since 2026-08-31).

Every 139 is the same frame:

#1  Napi::details::FinalizeData<_duckdb_data_chunk,
      void (*)(Napi::BasicEnv, _duckdb_data_chunk*), void>::Wrapper(...)
    from @duckdb/node-bindings-linux-x64/duckdb.node
#0  0x00005b68973204be in ?? ()      <- called through a freed pointer

DuckDB data chunks carry no dispose/closeSync — in 1.5.4-r.1 and in the current 1.5.5-r.4 alike, they are freed only by their napi finalizer. Deno 2.9.5 runs the pending ones at env teardown through a hint that is already gone. It happens at process exit: the reduced case prints its last line and then segfaults.

Same script, same machine, same package:

runtime	result
Node 20.19.2	10/10 clean
Deno 2.7.5	5/5 clean
Deno 2.9.5	12/12 crash (with and without explicit `closeSync`)
Deno 2.9.6	5/5 clean

The pin moved 2.5.6 → 2.9.5 on 2026-08-10 in 0eabe16. This issue was filed 2026-08-21; run #640 failed 08-21 and #713 failed 09-06. All after.

The handle-count diagnosis does not hold. Measured on the full suite, pinned image, adding bare Db open/close cycles that have no writer at all:

extra handles	exit 139
0	1/16
+20	3/8
+100	1/8

No dose-response — 100 extra handles crashed less than 20. Roughly 5 crashes in 32 runs regardless of arm. The variable is whether GC left a chunk finalizer pending at exit, not how many handles exist; a probe file "fixing" it by being deleted moved allocation timing, which reads as causation across a handful of runs. Twelve full-suite runs on 2.9.6 are clean, though the reduced case above is the actual proof.

Worth keeping from the withStack comment: one stack per test is still the right shape for other reasons, but it is not load-bearing for this crash.

Also, so it is not rediscovered: f0057db (Db refusing statements after close) was pushed with this flake in mind and is not a fix for it — the finalizer never goes through Db, and the suite still crashed with it in. It stands on its own — TaskService queues a sweep by microtask that stop() cannot cancel, and tasks_test's rig drains it by hand for exactly that reason — but it is a different bug.

Root-caused from a core dump. It is not the handle count, and the fix is the Deno pin — `229af54` bumps `denoland/deno` 2.9.5 → 2.9.6 (also closing #58 and #59, which had been sitting there with the fix since 2026-08-31). Every 139 is the same frame: ``` #1 Napi::details::FinalizeData<_duckdb_data_chunk, void (*)(Napi::BasicEnv, _duckdb_data_chunk*), void>::Wrapper(...) from @duckdb/node-bindings-linux-x64/duckdb.node #0 0x00005b68973204be in ?? () <- called through a freed pointer ``` DuckDB data chunks carry no `dispose`/`closeSync` — in 1.5.4-r.1 and in the current 1.5.5-r.4 alike, they are freed only by their napi finalizer. Deno 2.9.5 runs the pending ones at env teardown through a hint that is already gone. It happens at process *exit*: the reduced case prints its last line and then segfaults. Same script, same machine, same package: | runtime | result | |---|---| | Node 20.19.2 | 10/10 clean | | Deno 2.7.5 | 5/5 clean | | **Deno 2.9.5** | **12/12 crash** (with and without explicit `closeSync`) | | Deno 2.9.6 | 5/5 clean | The pin moved 2.5.6 → 2.9.5 on 2026-08-10 in `0eabe16`. This issue was filed 2026-08-21; run #640 failed 08-21 and #713 failed 09-06. All after. **The handle-count diagnosis does not hold.** Measured on the full suite, pinned image, adding bare `Db` open/close cycles that have no writer at all: | extra handles | exit 139 | |---|---| | 0 | 1/16 | | +20 | 3/8 | | +100 | 1/8 | No dose-response — 100 extra handles crashed *less* than 20. Roughly 5 crashes in 32 runs regardless of arm. The variable is whether GC left a chunk finalizer pending at exit, not how many handles exist; a probe file "fixing" it by being deleted moved allocation timing, which reads as causation across a handful of runs. Twelve full-suite runs on 2.9.6 are clean, though the reduced case above is the actual proof. Worth keeping from the `withStack` comment: one stack per test is still the right shape for other reasons, but it is not load-bearing for this crash. Also, so it is not rediscovered: `f0057db` (Db refusing statements after close) was pushed with this flake in mind and is **not** a fix for it — the finalizer never goes through `Db`, and the suite still crashed with it in. It stands on its own — `TaskService` queues a sweep by microtask that `stop()` cannot cancel, and `tasks_test`'s rig drains it by hand for exactly that reason — but it is a different bug.

No labels

No milestone

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

OpsDeck/core#54

No description provided.

Rows
Columns