Skip to content

fix: account for GPU time-slicing replicas in GPU health check - #30

Merged
hlts2 merged 2 commits into
mainfrom
fix/gpu-time-slicing
Nov 4, 2026
Merged

hlts2 merged 2 commits into
mainfrom
fix/gpu-time-slicing

Conversation

@hlts2

@hlts2 hlts2 commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Fix the GPU health check so that nodes with GPU time-slicing are not reported as GPUCountMismatch
  • Compute the expected GPU count as nvidia.com/gpu.count × nvidia.com/gpu.replicas
  • Treat a missing or invalid nvidia.com/gpu.replicas label as 1, so the behaviour for non-shared GPU nodes is unchanged
  • HasGPU is unchanged and still decides GPU nodes from nvidia.com/gpu.count only

Problem

With GPU time-slicing, nvidia.com/gpu.count reports the number of physical GPUs, while allocatable nvidia.com/gpu is physical GPUs × nvidia.com/gpu.replicas (NVIDIA GPU Operator docs).

Labels:
  nvidia.com/gpu.count=1
  nvidia.com/gpu.replicas=4
  nvidia.com/gpu.sharing-strategy=time-slicing
Allocatable:
  nvidia.com/gpu: 4

The checker expected 1 but saw 4, so healthy time-sliced nodes failed the GPU check and became reboot targets after the 10-minute threshold.

With GPU time-slicing, nvidia.com/gpu.count reports physical GPUs while
allocatable nvidia.com/gpu is physical GPUs x nvidia.com/gpu.replicas,
so healthy time-sliced nodes were reported as GPUCountMismatch.

Compute the expected count as gpu.count x gpu.replicas, treating a
missing or invalid replicas label as 1.

Signed-off-by: hlts2 <hiroto.funakoshi.hiroto@gmail.com>
@hlts2 hlts2 self-assigned this Nov 4, 2026
@hlts2
hlts2 merged commit 6dc5ae4 into main Nov 4, 2026
1 check passed
@hlts2
hlts2 deleted the fix/gpu-time-slicing branch November 4, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants