nvidia-smi (NVIDIA System Management Interface) is a command-line utility, built on top of the NVIDIA Management Library (NVML), for monitoring and managing NVIDIA GPU devices. It ships with the NVIDIA display driver.
Works on: Tesla, Quadro/RTX, GRID, and most GeForce GPUs.
Platforms: Linux and Windows (64-bit). On Windows it lives at
C:\Windows\System32\nvidia-smi.exe (already on PATH).
Privileges: Read/query commands need no special rights. Configuration
commands (power limits, clocks, ECC, resets) need root (Linux) or an
Administrator shell (Windows).
nvidia-smi -l 1 # Loop, refresh every 1 second
nvidia-smi -l 5 # Refresh every 5 seconds
nvidia-smi -lms 500 # Refresh every 500 milliseconds
On Linux you can also use watch:
watch -n 1 nvidia-smi
On Windows PowerShell there is no watch. Use the built-in loop or:
while ($true) { Clear-Host; nvidia-smi; Start-Sleep -Seconds 1 }
3.2 dmon — scrolling device monitor
One compact line per sample, ideal for logging.
nvidia-smi dmon # Default: all GPUs, 1-second interval
nvidia-smi dmon -i 0 # Only GPU 0
nvidia-smi dmon -d 2 # Sample every 2 seconds
nvidia-smi dmon -c 10 # Print 10 samples then exit
nvidia-smi dmon -s pucvmet # Choose which metric groups to show
nvidia-smi dmon -o DT # Prefix each line with Date and Time
-s metric group flags:
p power+temp · u utilization · c clocks · v power/thermal violations ·
m framebuffer memory · e ECC + PCIe errors · t PCIe throughput.
3.3 pmon — per-process monitor
nvidia-smi pmon # Per-process stats, all GPUs
nvidia-smi pmon -i 0 # GPU 0 only
nvidia-smi pmon -d 5 # 5-second interval (1–10 allowed)
nvidia-smi pmon -c 20 # 20 samples then exit
nvidia-smi pmon -s um # u = util, m = memory
4. Targeted Queries (CSV / Scripting)
The --query-gpu + --format combination is the backbone of any GPU
monitoring script.
# Human-readable CSV with headers and units
nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total \
--format=csv
# Parser-friendly: no header row, no unit suffixes
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.free \
--format=csv,noheader,nounits
--format modifiers:csv (required base) · noheader (drop the title
row) · nounits (strip MiB, %, W, etc.).
Commonly used --query-gpu fields
Field
Description
index
Zero-based GPU index.
name
Product name.
uuid
Globally unique GPU identifier.
serial
Board serial number.
pci.bus_id
PCIe bus address.
driver_version
Installed driver version.
temperature.gpu
Core temperature (°C).
utilization.gpu
GPU compute utilization (%).
utilization.memory
Memory-controller utilization (%).
memory.total
Total framebuffer memory.
memory.used / memory.free
Used / free framebuffer memory.
power.draw
Current power draw (W).
power.limit
Enforced power limit (W).
power.max_limit / power.min_limit
Power-limit bounds.
clocks.sm / clocks.gr
SM / graphics clock (MHz).
clocks.mem
Memory clock (MHz).
fan.speed
Fan speed (%).
pstate
Performance state (P0–P12).
compute_mode
Compute mode.
ecc.errors.uncorrected.volatile.total
Volatile uncorrectable ECC errors.
Run nvidia-smi --help-query-gpu for the full list.
# Windows PowerShell
while ($true) {
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,temperature.gpu,power.draw `
--format=csv,noheader | Add-Content gpu_log.csv
Start-Sleep -Seconds 1
}
5. Process & Memory Inspection
nvidia-smi # Process table at the bottom of the output
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits
nvidia-smi -q -d PIDS # Detailed per-process section
# Linux — find the PID in nvidia-smi, then:
kill -9 <PID>
# Windows
taskkill /PID <PID> /F
If memory stays occupied with no process listed, a process likely
crashed without releasing its CUDA context. On Linux, resetting the GPU
(nvidia-smi --gpu-reset -i <id>) usually clears it; otherwise reboot.
Within the same NUMA node, across PCIe host bridges.
SYS
Across NUMA nodes (slowest; crosses the CPU interconnect).
X
Self.
7. Administration & Configuration
All commands below modify GPU state. Run them as root (sudo) on Linux
or from an Administrator PowerShell/CMD on Windows. Target a specific
GPU with -i <index>; without -i the change applies to all GPUs.
7.1 Persistence mode (Linux only)
Keeps the driver loaded even when no client is active — removes driver
load latency and stabilizes clocks.
nvidia-smi -i 0 # By index
nvidia-smi -i 0,2 # Comma-separated list of indexes
nvidia-smi -i GPU-a1b2c3d4-... # By UUID
nvidia-smi -i 0000:01:00.0 # By PCI bus ID
The environment variable CUDA_VISIBLE_DEVICES affects CUDA applications,
notnvidia-smi — nvidia-smi always sees every physical GPU.
nvidia-smi: command not found / 'nvidia-smi' is not recognized
The driver is not installed or not on PATH. Install the NVIDIA driver. On
Windows the binary is at C:\Windows\System32\nvidia-smi.exe.NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver
The kernel module is not loaded or the driver/kernel versions mismatch
(common after a kernel update on Linux). Reinstall or rebuild the driver
module; a reboot often fixes it.GPU-Util is 0% but a job is running
Utilization is sampled over a short window — it reflects whether a kernel
was active, not how hard it worked. A data-loading-bound job can show low
utilization even while busy.Memory used but no process shown
A crashed process left a dangling CUDA context. Reset the GPU
(nvidia-smi --gpu-reset) or reboot.Insufficient Permissions
A configuration command was run without elevation. Use sudo (Linux) or an
Administrator shell (Windows).Header CUDA version ≠ installed CUDA
Expected. The header shows the maximum CUDA version the driver supports.
Check the installed toolkit with nvcc --version.Reference compiled 2026-05-16. Field names and flags reflect recent NVIDIA
driver branches (R535/R550+); older drivers may omit some options. Always
confirm against nvidia-smi -h on your system.