Skip to content

Report GPU power draw in job metrics - #4502

Open
prawalpokharel wants to merge 1 commit into
dstackai:masterfrom
prawalpokharel:issue_4501_gpu_power_metrics
Open

prawalpokharel wants to merge 1 commit into
dstackai:masterfrom
prawalpokharel:issue_4501_gpu_power_metrics

Conversation

@prawalpokharel

Copy link
Copy Markdown

Closes #4501

The runner now asks nvidia-smi for power.draw and enforced.power.limit in the same query it already runs for memory and utilization. The server stores both with each metrics point, returns them from the metrics API as gpu_power_draw_watts_gpu{N} and gpu_power_limit_watts_gpu{N}, and exports a dstack_job_gpu_power_draw_watts gauge to Prometheus. The gauge mostly matters on backends without the DCGM exporter, since DCGM_FI_DEV_POWER_USAGE is already exported where DCGM runs.

dstack metrics gets a POWER column for jobs that report power, drawn against the enforced limit the same way memory is drawn against total memory:

               UTILIZATION                           MEMORY                                        POWER
 job=0  cpu                     no data                               no data
        gpu=0                   ▁▇▇▇▇▇▇▇▇▇▇▇▇▇▇▁ 0%                   ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 660MB/24GB                   ▃▇▇▇▆▇▇▇▇▇▇▇▇▇▇▃ 67W/150W

               18 Nov 23:45 ┄┄┄┄┄┄┄ 19 Nov 00:45     18 Nov 23:45 ┄┄┄┄┄┄┄ 19 Nov 00:45             18 Nov 23:45 ┄┄┄┄┄┄┄ 19 Nov 00:45

This is get_metrics_table run on 30 minutes of the collector's readings from an A10. The readings have no host CPU or memory, so those cells say no data.

The third chart takes space from the other two: for a single job at 120 columns, each chart goes from 41 cells to 23. If the terminal is too narrow for three charts, the column is left out. The example on the metrics docs page still shows two columns.

_FIXED_COLUMNS also goes from 30 to 32. On master, 100% next to a 2 TB host's 1024GB/2048GB doesn't fit, and the labels get cut to … at 140 of the 184 widths from 57 to 240 columns. It's a separate fix, so I can move it to its own PR if you prefer.

Readings nvidia-smi can't give ([N/A], [Not Supported], [Unknown Error]) are left out, and memory and utilization are still reported. Points from AMD, Gaudi and older runners have NULL in the new columns. The migration only adds two nullable columns, and older servers ignore the new runner fields. Power series skip points without a reading, so they can have fewer timestamps than the other metrics in the same response. The UI doesn't show power yet.

The collector was checked on T4, A10, A100 and H100 GPUs on Modal (drivers 610.57.04 and 620.70.06). Its values match NVML readings taken within 100 ms of each nvidia-smi call, to the two decimals nvidia-smi prints.

The runner adds power.draw and enforced.power.limit to the nvidia-smi
query it already runs. The server stores them with each metrics point,
returns them from the metrics API and exports
dstack_job_gpu_power_draw_watts to Prometheus. dstack metrics shows a
POWER column drawn against the enforced limit.

Also raise _FIXED_COLUMNS from 30 to 32, the width that 100% next to a
2 TB host's memory label needs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Show GPU power draw in dstack metrics

1 participant