Repository navigation
Report GPU power draw in job metrics - #4502
Open
prawalpokharel wants to merge 1 commit into
Open
prawalpokharel wants to merge 1 commit into
prawalpokharel wants to merge 1 commit into
Conversation
The runner adds power.draw and enforced.power.limit to the nvidia-smi query it already runs. The server stores them with each metrics point, returns them from the metrics API and exports dstack_job_gpu_power_draw_watts to Prometheus. dstack metrics shows a POWER column drawn against the enforced limit. Also raise _FIXED_COLUMNS from 30 to 32, the width that 100% next to a 2 TB host's memory label needs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4501
The runner now asks
nvidia-smiforpower.drawandenforced.power.limitin the same query it already runs for memory and utilization. The server stores both with each metrics point, returns them from the metrics API asgpu_power_draw_watts_gpu{N}andgpu_power_limit_watts_gpu{N}, and exports adstack_job_gpu_power_draw_wattsgauge to Prometheus. The gauge mostly matters on backends without the DCGM exporter, sinceDCGM_FI_DEV_POWER_USAGEis already exported where DCGM runs.dstack metricsgets a POWER column for jobs that report power, drawn against the enforced limit the same way memory is drawn against total memory:This is
get_metrics_tablerun on 30 minutes of the collector's readings from an A10. The readings have no host CPU or memory, so those cells sayno data.The third chart takes space from the other two: for a single job at 120 columns, each chart goes from 41 cells to 23. If the terminal is too narrow for three charts, the column is left out. The example on the metrics docs page still shows two columns.
_FIXED_COLUMNSalso goes from 30 to 32. On master,100%next to a 2 TB host's1024GB/2048GBdoesn't fit, and the labels get cut to…at 140 of the 184 widths from 57 to 240 columns. It's a separate fix, so I can move it to its own PR if you prefer.Readings
nvidia-smican't give ([N/A],[Not Supported],[Unknown Error]) are left out, and memory and utilization are still reported. Points from AMD, Gaudi and older runners have NULL in the new columns. The migration only adds two nullable columns, and older servers ignore the new runner fields. Power series skip points without a reading, so they can have fewer timestamps than the other metrics in the same response. The UI doesn't show power yet.The collector was checked on T4, A10, A100 and H100 GPUs on Modal (drivers 610.57.04 and 620.70.06). Its values match NVML readings taken within 100 ms of each
nvidia-smicall, to the two decimalsnvidia-smiprints.