Manages local Ollama inference servers using Podman Quadlet to provide GPU-accelerated LLM capabilities.
The Ollama skill provides a streamlined interface for deploying and managing local Large Language Model (LLM) inference servers on Bazzite. It utilizes Podman Quadlet for efficient containerization and supports a single-instance design that optimizes shared GPU memory across loaded models. Whether you need to configure hardware acceleration for NVIDIA, AMD, or Intel GPUs, pull the latest open-source models like Llama 3 and Mistral, or integrate a local API into your development workflow, this skill automates the complex setup and management tasks involved in running local AI.
Key Features
01Persistent model storage and workspace volume mounting
02Multi-vendor GPU acceleration support (NVIDIA, AMD, Intel, and CPU)
03Streamlined model lifecycle management (Pull, List, Run, Delete)
04Automated Podman Quadlet container management
05Configurable API port, bind address, and context window size
060 GitHub stars
Use Cases
01Running private LLMs locally to ensure data privacy during development
02Testing and comparing different open-source models without cloud inference costs
03Providing a local backend for AI-integrated coding tools and applications
