hardware: GPU page shows frozen samples during an nvidia-smi outage #70

New issue

Open

opened 2026-09-22 11:40:22 +02:00 by julian · 0 comments

julian commented 2026-09-22 11:40:22 +02:00

Owner

• Copy link
• Copy Markdown

Raised in the review of #62.

While the nvidia-smi probe is failing, the last written GPU samples (writeGpu) stay in place and nothing marks them stale. Before #62 that meant frozen values until the container restarted; with the backoff it means frozen values for the length of the outage (a driver upgrade, a module reload, a card falling off the bus), with retries up to a minute apart.

Question to decide: what should the GPU page show during an outage?

• clear the samples on the first backed-off failure (card stays listed, metrics empty)
• keep the samples but attach a staleSince / updatedAt so the UI can grey them out
• leave as is

Deliberately left out of #62; the fix there is about the metrics dying, not about how an outage is displayed.

Raised in the review of #62. While the nvidia-smi probe is failing, the last written GPU samples (`writeGpu`) stay in place and nothing marks them stale. Before #62 that meant frozen values until the container restarted; with the backoff it means frozen values for the length of the outage (a driver upgrade, a module reload, a card falling off the bus), with retries up to a minute apart. Question to decide: what should the GPU page show during an outage? - clear the samples on the first backed-off failure (card stays listed, metrics empty) - keep the samples but attach a `staleSince` / `updatedAt` so the UI can grey them out - leave as is Deliberately left out of #62; the fix there is about the metrics dying, not about how an outage is displayed.

julian referenced this issue 2026-09-22 11:43:39 +02:00

fix(hardware): nvidia-smi probe backs off instead of giving up for good #62

julian referenced this issue 2026-09-22 11:43:40 +02:00

fix(hardware): nvidia-smi probe backs off instead of giving up for good #62

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

feat/nav-badges

fix/docker-buildable-and-autoupdate-ui

feat/docker-storage

fix/issue-25-dep-check-local-image

fix/issue-47-host-path-followups

fix/issue-29-deno-sh-followups

fix/storage-review-nits

feat/issue-39-storage-enumeration

fix/test-repo-no-signing

feat/ui-access-denied-and-rate

feat/schema-parity

ci/dependency-update-issues

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

OpsDeck/core#70

No description provided.