fix(download): a final name the filesystem let us claim, not one a check saw free #7
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/download-name-claim"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #5 — the S29 follow-up and the four nits the #4 review trail left behind.
S29 — the claim
uniqueNameoverexists()thenrenameTowas check-then-claim: two transfers really can run at once (one per page on the back stack, each with its own ViewModel and its own guard), both resolve the same free candidate, and the loser's rename silently replaces the winner — the exact loss the numbering exists to prevent.claimFinalNamemakes the claim atomic:createNewFile()(O_CREAT|O_EXCL) is one operation — whoever creates the placeholder owns the name, and every other transfer numbers past it. A lost race comes back asfalse, strikes that name, and asksuniqueNamefor the next. (The named racers are two ViewModels in one process, so an in-memory reserved set would have closed this race too; the filesystem claim costs no more and also holds against writers this process cannot see.)The move — corrected by review
The first cut moved the bytes with
Files.move(…, REPLACE_EXISTING)and argued that was safe because the target was our own placeholder. Review proved it was the bug:REPLACE_EXISTINGis notrename(2)— the library unlinks an existing target first and renames onto the freed name, so for a stretch of every call the just-claimed name did not exist, another transfer'sexists()read it as free, and both landed under one name (507 lost transfers in 1000 rounds of eight concurrent claims). The claim was real; the move handed it back.Now
ATOMIC_MOVE: a barerename(2)that replaces the placeholder in one step and never vacates the name. NotrenameTo, though on Android it is the same syscall — on the desk, where the unit tests run,renameTorefuses an existing target outright, and the concurrency test would be exercising a different move than the one that ships.A fill the filesystem refuses (a scanner briefly holding the fresh placeholder — observed on the desk as
AccessDeniedException) releases the placeholder, strikes the name, and claims the next candidate; the last refusal rides along as the cause if the loop ever exhausts.One loss accepted, said out loud: a process death between claim and move strands a zero-byte file under the final name, and later transfers number past it for good. No sweep takes it back — in the saved-downloads root a zero-byte file is indistinguishable from a genuinely empty download, and deleting a saved file is the loss this code exists to prevent (the same principle as the stray-partials decision below). The operator can see it and delete it; code must not.
The test that catches it
200 rounds of eight transfers finishing the same basename at once behind a
CyclicBarrier: one distinct final name per transfer, every transfer's own bytes on disk under its own name, and nothing else in the directory — no leftover partial, no stranded placeholder. It fails on theREPLACE_EXISTINGmove in roughly half the rounds; the sequential test (kept) never could. The losing threads also walkcreateNewFile() == false, solostand the retry loop are covered.Nits
runActionandsubmitFormrethrow the coroutine's own cancellation out of theirrunCatching, asprobe()already did. The token guard absorbed it — a cancelled action's token is already retired — but that was the guard doing the rule's job by accident; the rule now has zero exceptions.downloadJobnulls itself on completion, same pattern and same===guard asactionJob/submitJob.sweepOrphanParts' KDoc stops claiming Q+ cleans up failures it cannot see: a process death strands anIS_PENDINGrow too, and that row is the platform's to expire (~a week), not ours to sweep.versionName0.31.0 → 0.31.1 (bug fix).compileDebugKotlinandtestDebugUnitTestgreen locally (12/12 inDownloadNamingTest, concurrency test rerun clean).Note for merge order: #6 (spotless) is open; this branch's imports are already in the sorted order that rule enforces, so whichever lands second should rebase clean.
🤖 Generated with Claude Code
The claim is handed back before the bytes land — S29 still loses files
I did not take the reasoning on trust; I built the race and ran it. On the pristine PR head, 507 of 1000 rounds (8 concurrent
claimFinalNamecalls on the samename, one temp file each) end with two transfers under the same final name and one transfer's bytes gone. That is the exact loss the PR says it closes.The cause is one line —
StandardCopyOption.REPLACE_EXISTING.Files.movewithREPLACE_EXISTINGis notrename(2). Insun.nio.fs.UnixCopyFile.move(the ojluni copy Android ships — the samejava.nio.filethatminSdk 26makes available at all) an existing target isunlinked first, and only then is the sourcerenamed over the now-free name. So the winner's own placeholder is deleted before the move, the name goes absent mid-call, and a loser'sexists()→createNewFile()walks straight into that hole and "claims" a name the winner is holding.Measured directly — one thread spinning on
exists()while another moves onto the same target 20 000 times:~17% of the call's wall clock is a window where the claimed name does not exist. The KDoc rejects plain
Files.movebecause "its refusal is astatbefore arenameinside the library — the same two steps, closer together", and then reaches for the one flag that turns those two steps into three, the middle one beingunlinkof the very thing being defended. The claim is genuinely atomic; the move throws it away.Two more symptoms the same probe turned up: a saved file vanishing (
FileNotFoundException: a (6).txt (No such file or directory)on reading back a pathclaimFinalNamehad just returned — the unlink window seen from the other side), and the numbering ending up sparser than the file count.The fix is one word
Swap
REPLACE_EXISTINGforATOMIC_MOVE, or go back totemp.renameTo(target). Both are a barerename(2), which replaces the target atomically and never vacates the name. I ran each at 1000 rounds × 8 threads: zero duplicates, zero lost bodies, zero leftovers.renameTois arguably the better of the two — it is the call that already shipped, so it adds no new exception surface (ATOMIC_MOVEcan raiseAtomicMoveNotSupportedExceptionon a filesystem that will not do it, and that would only ever show up on a device).Worth noting once the mechanics are right: the placeholder is what makes this work, and the placeholder also means a process death between
createNewFileand the rename leaves a zero-byte file under the final name ingetExternalFilesDir(DIRECTORY_DOWNLOADS)— outside.parts/, sosweepPartsnever sees it, indistinguishable from a real empty download, and poisoning that name for every later transfer. Tiny window, but the PR body explicitly declines to write a root-directory cleaner while this change is what creates the litter such a cleaner would exist for. The honest resolution is probably to teach the sweep about zero-byte files indirolder thanPART_ORPHAN_MS, and to stop the KDoc claiming "an unclaimed name must not survive as a zero-byte download" when one can.The test is green on the broken code
That is the second finding, and it is why the first one shipped. Every assertion in
a claimed name lands the bytes and leaves every other file alonepasses verbatim against the olduniqueName+renameTo— including the zero-byte-placeholder case, which is justexists(), exactly what the old check already honoured. Nothing in it can reachcreateNewFile() == false, solost, the retry, and theerror("cannot claim …")backstop have no coverage at all. "Pinned by a test" names three behaviours that all predate the change; the one behaviour the PR exists for is the one nothing exercises. A test that stays green while half the transfers lose their bytes is not a regression net.The concurrency is testable on the host JVM — this is roughly what I ran:
If a threaded test in this suite is unwanted, the cheaper seam is a
create: (File) -> Boolean = File::createNewFileparameter — a stub that returnsfalseonce coverslostand the retry deterministically.One premise worth correcting even after the fix
"The filesystem is the only thing both transfers share" is not true here.
dirisgetExternalFilesDir(DIRECTORY_DOWNLOADS)— app-private — and the two racers the issue and the KDoc name are twoSchemaViewModels in one process. They share a class loader; asynchronizedreserved-name set would have closed this, needed noO_EXCL/FUSE argument, and would have been trivially testable.createNewFileis a fine choice and survives more than that, but the paragraph justifying it is arguing a threat model that does not apply, and the untestability that followed from it is what let the broken move through.Checked and good
Not rubber-stamped — each of these I chased to the point where it could have been wrong:
CancellationExceptionrethrow (runAction1392,submitForm2046) — correct, and safe for a non-obvious reason. The rethrow skipsresult.fold, and neither launch has afinally, so it is load-bearing that a cancellation always arrives with the token already retired. It does: all four cancel sites (start570-573,endRun1893-1896) callsubmitJob?.cancel()/actionJob?.cancel()beforeclaim.release()/releaseRun, and none of those statements suspend, so the release always wins the race to the main thread. I also went looking for aCancellationExceptionfrom something other than the job's own cancellation — awithTimeoutwould have wedged the claim silently — and there is none in the schema or network path (withTimeoutOrNullappears only inAlertsMessagingService). Chaining.onFailureontorunCatching { … }binds where you'd want, andkotlinx.coroutines.CancellationExceptionis already imported at line 54.downloadJobself-nulling — genuinely the same shape asactionJob(1470-1473) andsubmitJob(2122-2124),===guard included.invokeOnCompletionfiring off-thread was my worry; the body's tail resumes onDispatchers.MainafterwithContext(Dispatchers.IO)returns, so completion and the read inopenActionare both on Main. AssigningdownloadJobafterlaunchis fine even underMain.immediate: a body that completed before the assignment gets its handler invoked immediately after it.claimFinalNameruns on IO, not Main — the whole download is insidewithContext(Dispatchers.IO)(1024), so up to 999exists()calls plus a create and a rename are not a StrictMode violation. Worth stating since the function does noticeably more filesystem work than the two calls it replaced.sweepOrphanPartsKDoc — accurate now. MediaStore does setDATE_EXPIRESto ~7 days on a pending insert, and the pre-Q path really does deletetempon every visible failure,claimFinalName's throw included (the outercatch (e: Throwable)at 1284 covers it)..{name}.{random}.partshapes in a directory the file browser legitimately serves from is worse than the litter. (It does sit awkwardly beside the zero-byte corpse this PR introduces, which is the same category of thing.)versionName0.31.1 — single occurrence in the tree, no changelog or README to keep in step.DownloadNamingTest11/11 green locally, as claimed (--offline testDebugUnitTest,tests="11" failures="0"). The claim is true; it just does not mean what it is being asked to mean.Everything above is worth keeping. Blocking only on the one line, and on a test that would have caught it.
@ -321,0 +337,4 @@* from that instant every other transfer's `exists()` sees it and numbers* past it. The move then replaces a placeholder that is ours by* construction — `REPLACE_EXISTING` is not the silent overwrite it looks* like, it is scoped to a name nobody else can hold. (`Files.move` WITHOUTThis is exactly backwards, and it is the sentence that let the bug through.
REPLACE_EXISTINGis a silent overwrite of a name another transfer can hold, because its implementation vacates the name before renaming onto it.And the parenthetical rejects plain
Files.movefor being "astatbefore arenameinside the library" — the flag chosen instead makes it three steps in that same library, the middle one anunlinkof the very placeholder being defended. PlainFiles.movewas too weak; this is actively worse than therenameToit replaced.@ -321,0 +352,4 @@val candidate = uniqueName(name) { it in lost || File(dir, it).exists() }val target = File(dir, candidate)val claimed = try {target.createNewFile()Once the move is fixed, the placeholder is what makes the claim work — and it is also a new class of litter. A process death between this line and the rename leaves a zero-byte file under the final name in
getExternalFilesDir(DIRECTORY_DOWNLOADS): outside.parts/, sosweepPartsnever sees it, indistinguishable from a real empty download, and it poisons that name for every later transfer (backup.tar.gz (2),(3), forever).Microseconds wide, so not blocking — but the PR body declines a root-directory cleaner in the same breath as introducing the litter one would exist for, and line 368's "an unclaimed name must not survive as a zero-byte download" only holds for the
IOExceptionpath. Either teach the sweep about zero-byte files indirolder thanPART_ORPHAN_MS, or say out loud that the corpse is accepted.@ -321,0 +363,4 @@return@repeat}try {Files.move(temp.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING)This line is the bug.
REPLACE_EXISTINGis notrename(2): ojluni'sUnixCopyFile.moveunlinks an existing target first, then renames onto the freed name. So this deletes the placeholder the line above just claimed, and for the duration of the call the name does not exist — another transfer'sexists()reads it as free,createNewFile()succeeds, and both land under the same name.Measured: one thread spinning on
exists()while another moves onto the target 20 000 times reports the target absent in 56106 of 337030 polls withREPLACE_EXISTING, and 0 of 276001 withATOMIC_MOVE. Eight concurrentclaimFinalNamecalls on one name lose a transfer's bytes in 507 of 1000 rounds as written.or
StandardCopyOption.ATOMIC_MOVE. Both are a barerename(2)— atomic replace, no vacating. 1000 rounds x 8 threads clean on either.renameTois the safer pick: it is the call that already shipped, so it adds no new exception surface (ATOMIC_MOVEcan raiseAtomicMoveNotSupportedException, and only ever on a device).@ -108,0 +131,4 @@// this very instant — it counts as taken exactly like a fileassertTrue(File(downloads, "a (3).txt").createNewFile())val third = File(downloads, ".a.txt.3.part").apply { writeText("three") }assertEquals("a (4).txt", claimFinalName(downloads, "a.txt", third).name)Every assertion in this test passes verbatim against the old
uniqueName+renameTo. A zero-byte placeholder counting as taken is justexists(), which the old check already honoured — so all three behaviours the PR body lists as "pinned" predate the change, and nothing here can reachcreateNewFile() == false.lost, the retry loop and theerror("cannot claim …")backstop have zero coverage.Proof that this matters: this test is green right now, while 8 concurrent claims lose a transfer's bytes in 507 of 1000 rounds.
The race is testable on the host JVM —
CyclicBarrier, 8 threads, one temp file each, assert one distinct name per transfer and each transfer's own bytes under its own name (full version in the review body). If a threaded test is unwanted here, add acreate: (File) -> Boolean = File::createNewFileparameter and stub onefalse— that coverslostand the retry deterministically.Reworked in
4d18c66— every finding addressed, PR description updated to match.The move (blocking): you were right, and the KDoc paragraph defending
REPLACE_EXISTINGwas exactly backwards — the flag's unlink-then-rename vacates the claimed name mid-call. Swapped for a barerename(2), with one deviation from your preferred fix:ATOMIC_MOVE, notrenameTo. Reason: the unit tests run on a Windows host, whererenameTorefuses an existing target outright — the first cut of this rework usedrenameToand your concurrency test failed instantly withexpected:<8> but was:<0>, because the test would have been exercising a different move than the one that ships.ATOMIC_MOVEisrename(2)on Android/Linux andMoveFileEx(MOVEFILE_REPLACE_EXISTING)on the desk: replaces in one step on both, never vacates. YourAtomicMoveNotSupportedExceptionconcern is covered — it is anIOException, and a refused fill no longer throws: it deletes the placeholder, strikes the name intolost, and claims the next candidate (the last refusal becomes the cause if the loop ever exhausts). That retry path is not theoretical: on this desk, Defender briefly holding the fresh placeholder madeATOMIC_MOVEfail withAccessDeniedExceptionin the first green-on-Linux-shaped run of the concurrency test.The test (blocking): your test, essentially verbatim — 200 rounds × 8 threads behind a
CyclicBarrier, asserting one distinct final name per transfer and every transfer's own bytes, plus a third assertion that the directory holds exactly the names handed back (no leftover partial, no stranded placeholder; deliberately not asserted dense, since a struck name legitimately leaves a numbering hole). Temp files are written before the threads start so a thread dying short of the barrier cannot hang the other seven. The losing threads walkcreateNewFile() == false, solostand the retry loop are covered without the stub seam. 12/12 green across three runs.The zero-byte corpse: accepted and said out loud rather than swept. A zero-byte file in the saved-downloads root is indistinguishable from a genuinely empty download, and deleting a saved file is the loss this whole function exists to prevent — the same principle that keeps the stray-partials deleter out, now applied consistently. KDoc states the loss, its window, and that it is the operator's to delete; the "an unclaimed name must not survive as a zero-byte download" claim is now scoped to the refusal path, where it is actually true.
The premise: dropped. The KDoc now says the named racers are two ViewModels in one process and a reserved-name set would have closed that race;
createNewFileis kept because it costs no more and also holds against writers this process cannot see.The move is fixed, and I proved it rather than read it
I did not take the rework on trust either. On the PR head (
4d18c66) I flipped the one word back —ATOMIC_MOVE→REPLACE_EXISTING, nothing else — and ran the suite three times:concurrent claims of one name never lose a transfer's bytesfailed 3/3,expected:<8> but was:<7>. Restored the word: 5/5 green. So the test does catch the exact regression it was written for, on a Linux host, reliably — that was the part of the last round that mattered most, and it holds.Then I pushed harder than the shipped test does: 3000 rounds × 16 threads (15× the committed load, ~48 000 threads) — clean, 1.5 s. And I instrumented both retry branches to check the coverage claim is real rather than nominal: one class run hits
createNewFile() == false752 times, solostand the retry loop are genuinely walked, not just reachable in principle.CI parity checked too, since
.forgejo/workflows/build.ymlonly fires on push to main and this PR therefore gets no pre-merge run:lintDebugclean (the only output is the baseline's own meta-warnings plus a pre-existingModifierParameter, nothing from this diff —NewApiin particular is silent,java.nio.fileis API 26 andminSdkis 26), andtestReleaseUnitTest— the variant CI actually runs, not the debug one you quoted — green across every class.Approving. Four follow-ups below, none blocking.
1. The refusal branch has no coverage on the host CI uses
The instrumented run recorded zero refusals on Linux. That is expected — unlink of an open file just works here — but it means
refused,target.delete(), and theIllegalStateException("cannot claim a name …", refused)backstop are exercised only incidentally, on your desk, by Defender. The branch you added in answer to the last review is the one part of the new code nothing pins.It is deterministic in three lines: hand the claim a temp that does not exist, and every fill is refused with
NoSuchFileException. I wrote it and ran it — green, 13/13:That one test covers the strike, the delete, the exhaustion, the cause riding along, and the "no corpse" property — all four, on the host CI runs.
2. The accepted loss is wider than the KDoc scopes it to
Process death is not the only way.
target.delete()'s result is ignored, and the refusal you actually observed — a scanner holding the fresh placeholder — is precisely the state in which a Windows delete also fails. When it does, the placeholder survives as a zero-byte file under the final name with no crash involved, poisoning that number exactly as the process-death case does. Two consequences: the KDoc's "one loss, and it needs a process death" is too narrow, and the third assertion (assertEquals(landed.keys, downloads.list()!!.toSet())) is then a real flake on the one host where the refusal has been seen.The shipping impact is nil — Android unlinks an open file — so the code needs nothing. The sentence does: the loss is a claim this transfer could not release, whatever stopped it.
3. Worth writing down: this only ships to API 26–28
claimFinalNameis reached fromdownloadToAppDownloads, which is theelseofSDK_INT >= Q(line 1051). Everything from Android 10 up takesdownloadViaMediaStore, where the final name is settled by MediaStore's own insert — an atomic dedupe, no check-then-claim anywhere in that path, so there is genuinely nothing to fix there. I checked, because "the PR fixed the legacy path and left the modern one racing" would have been blocking.It does mean the 507/1000 loss I measured last round was only ever reachable on Android 8–9. Fine — but the KDoc argues from "two pages on the back stack", which reads as a statement about any device, and the next person to touch the Q+ path deserves the qualifier where the reasoning lives.
4. "on the desk, where these tests run"
They also run on
cth-ubuntu-latest, twice per push to main (lintDebug,testReleaseUnitTest), whererenameTowould have been fine. The reason forATOMIC_MOVEis sound and I would not change it — a test that must be green on both hosts has to exercise the move that ships — but the clause puts the tests in one place and there are two.Checked and good
Re-verified at the new head, not carried over from the last round:
runAction(1417) andsubmitForm(2068) now throw pastresult.fold, and neither launch has afinally— so it is load-bearing that the release always wins. Both cancel sites hold:start(595-599) andendRun(1918-1921) eachcancel()the jobs and thenclaim.release()/releaseRun(error)with no suspension in between, andreleaseRunclearsuploading,runningAction,submittingForm,uploadPercentin oneupdate. A cancellation dispatched from the IO thread queues behind that. Nothing sticks.downloadJobself-nulling is right againstabandonDownload(), which nulls the field itself — the handler's===then finds a field that is null or a later job and no-ops. Assigning afterlaunchis safe underMain.immediate: a body that finished first gets its handler invoked on registration.createNewFilecannot be steered out of the directory.nameis whatever the server'scontent-dispositionsaid, so I chased it:filenameFromDisposition→sanitiseFilenamestrips through the last/and\and rejects"",.,..before it ever reaches the claim. The oldrenameTohad the same exposure, but this version creates on that name, so it was worth confirming rather than assuming.lostis tested beforeexists()in the predicate —it in lost || File(dir, it).exists(). That ordering is what keeps the retry loop's syscall count bounded by what is in the directory instead of by iteration count; reversed, a thousand refusals would be a stat storm on the IO dispatcher. Easy to lose in a later tidy-up.AtomicMoveNotSupportedExceptionis unreachable here, which settles the concern I raised: on Unix it isEXDEV-only, andtemplives indir/.parts/— same volume by construction. Handling it as a plainIOExceptionis right anyway.versionName0.31.1 is the only occurrence in the tree; no changelog or README to keep in step. Import order in both files already matches what #6 will enforce, so the rebase note holds.createNewFileover a reserved-name set on the "costs no more" ground is the right call — the set would have been process-local and this is not.Everything the last round blocked on is closed, and closed in a way a rerun would catch again. The four items above are follow-ups, not gates.