hardware: NVIDIA probe never runs when the nvidia modules are unloaded at startup #69
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Raised in the review of #62. That PR fixes the probe dying after an outage that starts after a successful detection; this is the other half.
startGpuCollectorsgates the nvidia-smi probe on a one-time DRM snapshot (gpu.ts,detectGpusis called exactly once):detectGpusreads/sys/class/drm/card*, where an NVIDIA node exists only whilenvidia-drmis bound. If the process starts while the NVIDIA modules are unloaded and the host has any other DRM card (on a server board typically the ASPEED BMC,0x1a03, bound byast),cards.length !== 0and nothing incardsisnvidia.wantNvidiastays false for the lifetime of the process: nvidia-smi is never invoked, the backoff never ticks, and the GPU metrics never appear. A restart does not fix it unless the module happens to be back first.This is reachable on epyclab: the driver upgrade script stops the opsdeck container for the duration, so whether GPU metrics exist afterwards depends on whether the module is loaded before the container comes back. The same hole applies to a compute-only host that never loads
nvidia-drm.Options, both of which change behaviour on non-NVIDIA hosts and so want a decision rather than a drive-by:
/sys/bus/pci/devices/*/vendor == 0x10dewith PCI class0x0300/0x0302, so a card counts as NVIDIA whether or notnvidia-drmis bounddetectGpuson the backoff's retry schedule so a card that appears later is picked upWith #62's backoff in place, cheap perpetual probing is affordable, which removes the reason the one-time DRM gate existed in the first place.