Monitors GPU performance, VRAM allocation, and inference metrics for Ollama models to optimize AI workloads.
The GPU monitoring skill provides specialized tools for tracking and optimizing hardware performance when running Ollama inference. It enables developers to monitor NVIDIA and AMD GPU status, VRAM usage, and critical inference metrics such as tokens per second and evaluation durations. By providing automated health checks and troubleshooting guides for common issues like CPU fallback or Out-of-Memory (OOM) errors, this skill ensures that local AI models are running at peak efficiency on available hardware.
Key Features
01Inference performance benchmarking (tokens/sec and eval timing)
02Real-time monitoring for NVIDIA (SMI) and AMD (ROCm) hardware
03Detailed VRAM tracking per model via Ollama API integration
04Automated GPU health checks and container troubleshooting
05Active utilization monitoring during live inference sessions
060 GitHub stars
Use Cases
01Debugging Out-of-Memory (OOM) errors during model loading
02Optimizing hardware allocation for concurrent model serving
03Benchmarking LLM performance across different quantization levels
