OpenVINO Integration with Ollama

Click here to expand/collapse content

Ollama

Get up and running with large language models.

macOS

Download

Windows

Download

Linux

curl -fsSL https://ollama.com/install.sh | sh

Manual install instructions

Docker

The official Ollama Docker image ollama/ollama is available on Docker Hub.

Libraries

Community

Quickstart

To run and chat with Llama 3.2:

ollama run llama3.2

Model library

Ollama supports a list of models available on ollama.com/library

Here are some example models that can be downloaded:

Model	Parameters	Size	Download
DeepSeek-R1	7B	4.7GB	`ollama run deepseek-r1`
DeepSeek-R1	671B	404GB	`ollama run deepseek-r1:671b`
Llama 3.3	70B	43GB	`ollama run llama3.3`
Llama 3.2	3B	2.0GB	`ollama run llama3.2`
Llama 3.2	1B	1.3GB	`ollama run llama3.2:1b`
Llama 3.2 Vision	11B	7.9GB	`ollama run llama3.2-vision`
Llama 3.2 Vision	90B	55GB	`ollama run llama3.2-vision:90b`
Llama 3.1	8B	4.7GB	`ollama run llama3.1`
Llama 3.1	405B	231GB	`ollama run llama3.1:405b`
Phi 4	14B	9.1GB	`ollama run phi4`
Phi 3 Mini	3.8B	2.3GB	`ollama run phi3`
Gemma 2	2B	1.6GB	`ollama run gemma2:2b`
Gemma 2	9B	5.5GB	`ollama run gemma2`
Gemma 2	27B	16GB	`ollama run gemma2:27b`
Mistral	7B	4.1GB	`ollama run mistral`
Moondream 2	1.4B	829MB	`ollama run moondream`
Neural Chat	7B	4.1GB	`ollama run neural-chat`
Starling	7B	4.1GB	`ollama run starling-lm`
Code Llama	7B	3.8GB	`ollama run codellama`
Llama 2 Uncensored	7B	3.8GB	`ollama run llama2-uncensored`
LLaVA	7B	4.5GB	`ollama run llava`
Solar	10.7B	6.1GB	`ollama run solar`

[!NOTE] You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.

Customize a model

Import from GGUF

Ollama supports importing GGUF models in the Modelfile:

Create a file named Modelfile, with a FROM instruction with the local filepath to the model you want to import.
```
FROM ./vicuna-33b.Q4_0.gguf
```
Create the model in Ollama
```
ollama create example -f Modelfile
```
Run the model
```
ollama run example
```

Import from Safetensors

See the guide on importing models for more information.

Customize a prompt

Models from the Ollama library can be customized with a prompt. For example, to customize the llama3.2 model:

ollama pull llama3.2

Create a Modelfile:

FROM llama3.2

# set the temperature to 1 [higher is more creative, lower is more coherent]
PARAMETER temperature 1

# set the system message
SYSTEM """
You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.
"""

Next, create and run the model:

ollama create mario -f ./Modelfile
ollama run mario
>>> hi
Hello! It's your friend Mario.

For more information on working with a Modelfile, see the Modelfile documentation.

CLI Reference

Create a model

ollama create is used to create a model from a Modelfile.

ollama create mymodel -f ./Modelfile

Pull a model

ollama pull llama3.2

This command can also be used to update a local model. Only the diff will be pulled.

Remove a model

ollama rm llama3.2

Copy a model

ollama cp llama3.2 my-model

Multiline input

For multiline input, you can wrap text with """:

>>> """Hello,
... world!
... """
I'm a basic program that prints the famous "Hello, world!" message to the console.

Multimodal models

ollama run llava "What's in this image? /Users/jmorgan/Desktop/smile.png"

Output: The image features a yellow smiley face, which is likely the central focus of the picture.

Pass the prompt as an argument

ollama run llama3.2 "Summarize this file: $(cat README.md)"

Output: Ollama is a lightweight, extensible framework for building and running language models on the local machine. It provides a simple API for creating, running, and managing models, as well as a library of pre-built models that can be easily used in a variety of applications.

Show model information

ollama show llama3.2

List models on your computer

ollama list

List which models are currently loaded

ollama ps

Stop a model which is currently running

ollama stop llama3.2

Start Ollama

ollama serve is used when you want to start ollama without running the desktop application.

Building

See the developer guide

Running local builds

Next, start the server:

./ollama serve

Finally, in a separate shell, run a model:

./ollama run llama3.2

REST API

Ollama has a REST API for running and managing models.

Generate a response

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt":"Why is the sky blue?"
}'

Chat with a model

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "why is the sky blue?" }
  ]
}'

See the API documentation for all endpoints.

Community Integrations

Web & Desktop

Open WebUI
Enchanted (macOS native)
Hollama
Lollms-Webui
LibreChat
Bionic GPT
HTML UI
Saddle
Chatbot UI
Chatbot UI v2
Typescript UI
Minimalistic React UI for Ollama Models
Ollamac
big-AGI
Cheshire Cat assistant framework
Amica
chatd
Ollama-SwiftUI
Dify.AI
MindMac
NextJS Web Interface for Ollama
Msty
Chatbox
WinForm Ollama Copilot
NextChat with Get Started Doc
Alpaca WebUI
OllamaGUI
OpenAOE
Odin Runes
LLM-X (Progressive Web App)
AnythingLLM (Docker + MacOs/Windows/Linux native app)
Ollama Basic Chat: Uses HyperDiv Reactive UI
Ollama-chats RPG
IntelliBar (AI-powered assistant for macOS)
QA-Pilot (Interactive chat tool that can leverage Ollama models for rapid understanding and navigation of GitHub code repositories)
ChatOllama (Open Source Chatbot based on Ollama with Knowledge Bases)
CRAG Ollama Chat (Simple Web Search with Corrective RAG)
RAGFlow (Open-source Retrieval-Augmented Generation engine based on deep document understanding)
StreamDeploy (LLM Application Scaffold)
chat (chat web app for teams)
Lobe Chat with Integrating Doc
Ollama RAG Chatbot (Local Chat with multiple PDFs using Ollama and RAG)
BrainSoup (Flexible native client with RAG & multi-agent automation)
macai (macOS client for Ollama, ChatGPT, and other compatible API back-ends)
RWKV-Runner (RWKV offline LLM deployment tool, also usable as a client for ChatGPT and Ollama)
Ollama Grid Search (app to evaluate and compare models)
Olpaka (User-friendly Flutter Web App for Ollama)
OllamaSpring (Ollama Client for macOS)
LLocal.in (Easy to use Electron Desktop Client for Ollama)
Shinkai Desktop (Two click install Local AI using Ollama + Files + RAG)
AiLama (A Discord User App that allows you to interact with Ollama anywhere in discord )
Ollama with Google Mesop (Mesop Chat Client implementation with Ollama)
R2R (Open-source RAG engine)
Ollama-Kis (A simple easy to use GUI with sample custom LLM for Drivers Education)
OpenGPA (Open-source offline-first Enterprise Agentic Application)
Painting Droid (Painting app with AI integrations)
Kerlig AI (AI writing assistant for macOS)
AI Studio
Sidellama (browser-based LLM client)
LLMStack (No-code multi-agent framework to build LLM agents and workflows)
BoltAI for Mac (AI Chat Client for Mac)
Harbor (Containerized LLM Toolkit with Ollama as default backend)
PyGPT (AI desktop assistant for Linux, Windows and Mac)
Alpaca (An Ollama client application for linux and macos made with GTK4 and Adwaita)
AutoGPT (AutoGPT Ollama integration)
Go-CREW (Powerful Offline RAG in Golang)
PartCAD (CAD model generation with OpenSCAD and CadQuery)
Ollama4j Web UI - Java-based Web UI for Ollama built with Vaadin, Spring Boot and Ollama4j
PyOllaMx - macOS application capable of chatting with both Ollama and Apple MLX models.
Claude Dev - VSCode extension for multi-file/whole-repo coding
Cherry Studio (Desktop client with Ollama support)
ConfiChat (Lightweight, standalone, multi-platform, and privacy focused LLM chat interface with optional encryption)
Archyve (RAG-enabling document library)
crewAI with Mesop (Mesop Web Interface to run crewAI with Ollama)
Tkinter-based client (Python tkinter-based Client for Ollama)
LLMChat (Privacy focused, 100% local, intuitive all-in-one chat interface)
Local Multimodal AI Chat (Ollama-based LLM Chat with support for multiple features, including PDF RAG, voice chat, image-based interactions, and integration with OpenAI.)
ARGO (Locally download and run Ollama and Huggingface models with RAG on Mac/Windows/Linux)
OrionChat - OrionChat is a web interface for chatting with different AI providers
G1 (Prototype of using prompting strategies to improve the LLM's reasoning through o1-like reasoning chains.)
Web management (Web management page)
Promptery (desktop client for Ollama.)
Ollama App (Modern and easy-to-use multi-platform client for Ollama)
chat-ollama (a React Native client for Ollama)
SpaceLlama (Firefox and Chrome extension to quickly summarize web pages with ollama in a sidebar)
YouLama (Webapp to quickly summarize any YouTube video, supporting Invidious as well)
DualMind (Experimental app allowing two models to talk to each other in the terminal or in a web interface)
ollamarama-matrix (Ollama chatbot for the Matrix chat protocol)
ollama-chat-app (Flutter-based chat app)
Perfect Memory AI (Productivity AI assists personalized by what you have seen on your screen, heard and said in the meetings)
Hexabot (A conversational AI builder)
Reddit Rate (Search and Rate Reddit topics with a weighted summation)
OpenTalkGpt (Chrome Extension to manage open-source models supported by Ollama, create custom models, and chat with models from a user-friendly UI)
VT (A minimal multimodal AI chat app, with dynamic conversation routing. Supports local models via Ollama)
Nosia (Easy to install and use RAG platform based on Ollama)
Witsy (An AI Desktop application available for Mac/Windows/Linux)
Abbey (A configurable AI interface server with notebooks, document storage, and YouTube support)
Minima (RAG with on-premises or fully local workflow)
aidful-ollama-model-delete (User interface for simplified model cleanup)
Perplexica (An AI-powered search engine & an open-source alternative to Perplexity AI)
Ollama Chat WebUI for Docker (Support for local docker deployment, lightweight ollama webui)
AI Toolkit for Visual Studio Code (Microsoft-official VSCode extension to chat, test, evaluate models with Ollama support, and use them in your AI applications.)
MinimalNextOllamaChat (Minimal Web UI for Chat and Model Control)
Chipper AI interface for tinkerers (Ollama, Haystack RAG, Python)
ChibiChat (Kotlin-based Android app to chat with Ollama and Koboldcpp API endpoints)
LocalLLM (Minimal Web-App to run ollama models on it with a GUI)
Ollamazing (Web extension to run Ollama models)

Cloud

Terminal

oterm
Ellama Emacs client
Emacs client
neollama UI client for interacting with models from within Neovim
gen.nvim
ollama.nvim
ollero.nvim
ollama-chat.nvim
ogpt.nvim
gptel Emacs client
Oatmeal
cmdh
ooo
shell-pilot(Interact with models via pure shell scripts on Linux or macOS)
tenere
llm-ollama for Datasette's LLM CLI.
typechat-cli
ShellOracle
tlm
podman-ollama
gollama
ParLlama
Ollama eBook Summary
Ollama Mixture of Experts (MOE) in 50 lines of code
vim-intelligence-bridge Simple interaction of "Ollama" with the Vim editor
x-cmd ollama
bb7
SwollamaCLI bundled with the Swollama Swift package. Demo
aichat All-in-one LLM CLI tool featuring Shell Assistant, Chat-REPL, RAG, AI tools & agents, with access to OpenAI, Claude, Gemini, Ollama, Groq, and more.
PowershAI PowerShell module that brings AI to terminal on Windows, including support for Ollama
orbiton Configuration-free text editor and IDE with support for tab completion with Ollama.

Apple Vision Pro

Enchanted

Database

pgai - PostgreSQL as a vector database (Create and search embeddings from Ollama models using pgvector)
- Get started guide
MindsDB (Connects Ollama models with nearly 200 data platforms and apps)
chromem-go with example
Kangaroo (AI-powered SQL client and admin tool for popular databases)

Package managers

Libraries

LangChain and LangChain.js with example
Firebase Genkit
crewAI
Yacana (User-friendly multi-agent framework for brainstorming and executing predetermined flows with built-in tool integration)
Spring AI with reference and example
LangChainGo with example
LangChain4j with example
LangChainRust with example
LangChain for .NET with example
LLPhant
LlamaIndex and LlamaIndexTS
LiteLLM
OllamaFarm for Go
OllamaSharp for .NET
Ollama for Ruby
Ollama-rs for Rust
Ollama-hpp for C++
Ollama4j for Java
ModelFusion Typescript Library
OllamaKit for Swift
Ollama for Dart
Ollama for Laravel
LangChainDart
Semantic Kernel - Python
Haystack
Elixir LangChain
Ollama for R - rollama
Ollama for R - ollama-r
Ollama-ex for Elixir
Ollama Connector for SAP ABAP
Testcontainers
Portkey
PromptingTools.jl with an example
LlamaScript
llm-axe (Python Toolkit for Building LLM Powered Apps)
Gollm
Gollama for Golang
Ollamaclient for Golang
High-level function abstraction in Go
Ollama PHP
Agents-Flex for Java with example
Parakeet is a GoLang library, made to simplify the development of small generative AI applications with Ollama.
Haverscript with examples
Ollama for Swift
Swollama for Swift with DocC
GoLamify
Ollama for Haskell
multi-llm-ts (A Typescript/JavaScript library allowing access to different LLM in unified API)
LlmTornado (C# library providing a unified interface for major FOSS & Commercial inference APIs)
Ollama for Zig
Abso (OpenAI-compatible TypeScript SDK for any LLM provider)

Mobile

Enchanted
Maid
Ollama App (Modern and easy-to-use multi-platform client for Ollama)
ConfiChat (Lightweight, standalone, multi-platform, and privacy focused LLM chat interface with optional encryption)

Extensions & Plugins

Raycast extension
Discollama (Discord bot inside the Ollama discord channel)
Continue
Vibe (Transcribe and analyze meetings with Ollama)
Obsidian Ollama plugin
Logseq Ollama plugin
NotesOllama (Apple Notes Ollama plugin)
Dagger Chatbot
Discord AI Bot
Ollama Telegram Bot
Hass Ollama Conversation
Rivet plugin
Obsidian BMO Chatbot plugin
Cliobot (Telegram bot with Ollama support)
Copilot for Obsidian plugin
Obsidian Local GPT plugin
Open Interpreter
Llama Coder (Copilot alternative using Ollama)
Ollama Copilot (Proxy that allows you to use ollama as a copilot like Github copilot)
twinny (Copilot and Copilot chat alternative using Ollama)
Wingman-AI (Copilot code and chat alternative using Ollama and Hugging Face)
Page Assist (Chrome Extension)
Plasmoid Ollama Control (KDE Plasma extension that allows you to quickly manage/control Ollama model)
AI Telegram Bot (Telegram bot using Ollama in backend)
AI ST Completion (Sublime Text 4 AI assistant plugin with Ollama support)
Discord-Ollama Chat Bot (Generalized TypeScript Discord Bot w/ Tuning Documentation)
ChatGPTBox: All in one browser extension with Integrating Tutorial
Discord AI chat/moderation bot Chat/moderation bot written in python. Uses Ollama to create personalities.
Headless Ollama (Scripts to automatically install ollama client & models on any OS for apps that depends on ollama server)
Terraform AWS Ollama & Open WebUI (A Terraform module to deploy on AWS a ready-to-use Ollama service, together with its front end Open WebUI service.)
node-red-contrib-ollama
Local AI Helper (Chrome and Firefox extensions that enable interactions with the active tab and customisable API endpoints. Includes secure storage for user prompts.)
vnc-lm (Discord bot for messaging with LLMs through Ollama and LiteLLM. Seamlessly move between local and flagship models.)
LSP-AI (Open-source language server for AI-powered functionality)
QodeAssist (AI-powered coding assistant plugin for Qt Creator)
Obsidian Quiz Generator plugin
AI Summmary Helper plugin
TextCraft (Copilot in Word alternative using Ollama)
Alfred Ollama (Alfred Workflow)
TextLLaMA A Chrome Extension that helps you write emails, correct grammar, and translate into any language
Simple-Discord-AI

Supported backends

llama.cpp project founded by Georgi Gerganov.

Observability

Lunary is the leading open-source LLM observability platform. It provides a variety of enterprise-grade features such as real-time analytics, prompt templates management, PII masking, and comprehensive agent tracing.
OpenLIT is an OpenTelemetry-native tool for monitoring Ollama Applications & GPUs using traces and metrics.
HoneyHive is an AI observability and evaluation platform for AI agents. Use HoneyHive to evaluate agent performance, interrogate failures, and monitor quality in production.
Langfuse is an open source LLM observability platform that enables teams to collaboratively monitor, evaluate and debug AI applications.
MLflow Tracing is an open source LLM observability tool with a convenient API to log and visualize traces, making it easy to debug and evaluate GenAI applications.

Ollama-ov

Getting started with large language models and using the GenAI backend.

We provide two ways to download the executable file of Ollama, one is to download it from Google Drive, and the other is to download it from Baidu Drive:

Google Driver

Windows

Download exe + Download OpenVINO GenAI

Linux(Ubuntu 22.04)

Download + Donwload OpenVINO GenAI

百度云盘

Windows

Download exe + Download OpenVINO GenAI

Linux(Ubuntu 22.04)

Download + Donwload OpenVINO GenAI

Docker

Linux

We also prepared a Dockerfile to help developers quickly build Docker images: Dockerfile

docker build -t ollama_openvino:v1 -f Dockerfile_genai .

then

docker run --rm -it ollama_openvino:v1

Model library

The native Ollama only supports models in the GGUF format, the Ollama-OV invoke OpenVINO GenAI which requires models in the OpenVINO format. Therefore, we have enabled support for OpenVINO model files in Ollama. For public LLMs, you can access and download OpenVINO IR model from HuggingFace or ModelScope:

Model	Parameters	Size	Compression	Download	Device
Qwen3-8B-int4-asym-ov	8B	4.9GB	INT4_ASYM_128	ModelScope	CPU, GPU, NPU(base)
Qwen3-4B-int4-ov	4B	2.7GB	INT4_ASYM_128	ModelScope	CPU, GPU, NPU(base)
Qwen3-1.7B-int4-ov	1.7B	1.2GB	INT4_ASYM_128	ModelScope	CPU, GPU, NPU(base)
Qwen3-0.6B-int4-ov	0.6B	0.5GB	INT4_ASYM_128	ModelScope	CPU, GPU, NPU(base)
DeepSeek-R1-Distill-Qwen-1.5B-int4-ov	1.5B	1.4GB	INT4_ASYM_32	ModelScope	CPU, GPU, NPU(base)
DeepSeek-R1-Distill-Qwen-1.5B-int4-ov-npu	1.5B	1.1GB	INT4_SYM_CW	ModelScope	NPU(best)
DeepSeek-R1-Distill-Qwen-7B-int4-ov	7B	4.3GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
DeepSeek-R1-Distill-Qwen-7B-int4-ov-npu	7B	4.1GB	INT4_SYM_CW	ModelScope	NPU(best)
DeepSeek-R1-Distill-Qwen-14B-int4-ov	14B	8.0GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
DeepSeek-R1-Distill-llama-8B-int4-ov	8B	4.5GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
DeepSeek-R1-Distill-llama-8B-int4-ov-npu	8B	4.2GB	INT4_SYM_CW	ModelScope	NPU(best)
llama-3.2-1b-instruct-int4-ov	1B	0.8GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
llama-3.2-3b-instruct-int4-ov	3B	1.9GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
llama-3.2-3b-instruct-int4-ov-npu	3B	1.8GB	INT4_SYM_CW	ModelScope	NPU(best)
Phi-3.5-mini-instruct-int4-ov	3.8B	2.1GB	INT4_ASYM	HF, ModelScope	CPU, GPU
Phi-3-mini-128k-instruct-int4-ov	3.8B	2.5GB	INT4_ASYM	HF, ModelScope	CPU, GPU
Phi-3-mini-4k-instruct-int4-ov	3.8B	2.2GB	INT4_ASYM	HF, ModelScope	CPU, GPU
Phi-3-medium-4k-instruct-int4-ov	14B	7.4GB	INT4_ASYM	HF, ModelScope	CPU, GPU
Qwen2.5-0.5B-Instruct-openvino-ovms-int4	0.5B	0.3GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
Qwen2.5-1.5B-Instruct-openvino-ovms-int4	1.5B	0.9GB	INT4_SYM_128	ModelScope	CPU, GPU, NPU(base)
Qwen2.5-3B-Instruct-gptq-ov	3B	2.7GB	INT4_GPTQ	ModelScope	CPU, GPU
Qwen2.5-7B-Instruct-int4-ov	7B	4.3GB	INT4_ASYM	ModelScope	CPU, GPU
minicpm-1b-sft-int4-ov	1B	0.7GB	INT4_SYM	ModelScope	CPU, GPU, NPU(base)
gemma-2-9b-it-int4-ov	9B	5.3GB	INT4_ASYM	HF, ModelScope	CPU, GPU
gemma-3-1b-it-int4-ov	1B	0.7G	INT4_SYM_128	ModelScope	CPU, GPU
TinyLlama-1.1B-Chat-v1.0-int4-ov	1.1B	0.6GB	INT4_ASYM	HF, ModelScope	CPU, GPU

INT4_SYM_128: INT4 symmetric compression with NNCF, group size 128, all linear layer compressed. Similar to Q4_0 compression.
INT4_SYM_CW: INT4 symmetric compression with NNCF, channel wise compression for NPU best performance.
INT4_ASYM: INT4 asymmetric compression with NNCF, has better accuracy than symmetric, NPU not support asymmetric compression.
INT4_GPTQ: INT4 GPTQ compression by NNCF which aligned with Huggingface.

Just provide above model link as example for part models, for other LLMs, you can check OpenVINO GenAI model support list. If you have customized LLM, please follow model conversion step of GenAI.

Ollama Model File

We added two new parameters to Modelfile based on the original parameters:

Parameter	Description	Value Type	Example Usage
stop_id	Sets the stop ids to use	int	stop_id 151643
max_new_token	The maximum number of tokens generated by genai (Default: 2048)	int	max_new_token 4096

Quick start

Start Ollama

First, set GODEBUG=cgocheck=0 env:

Linux

export GODEBUG=cgocheck=0

Windows

set GODEBUG=cgocheck=0

Next, ollama serve is used when you want to start ollama (you must use the Ollama compiled by ollama_ov to start the serve):
```
ollama serve
```

Import from openVINO IR

How to create an Ollama model based on Openvino IR

Example

Let's take deepseek-ai/DeepSeek-R1-Distill-Qwen-7B as an example.

Download the OpenVINO model

Download from ModelScope: DeepSeek-R1-Distill-Qwen-7B-int4-ov

pip install modelscope
modelscope download --model zhaohb/DeepSeek-R1-Distill-Qwen-7B-int4-ov --local_dir ./DeepSeek-R1-Distill-Qwen-7B-int4-ov

If the OpenVINO model exists in HF, we can also download it from HF. Here we take TinyLlama-1.1B-Chat-v1.0-int4-ov as an example to introduce how to download the model from HF.

If your network access to HuggingFace is unstable, you can try to use a proxy image to pull the model.
```
set HF_ENDPOINT=https://hf-mirror.com
```
```
pip install -U huggingface_hub
huggingface-cli download --resume-download OpenVINO/TinyLlama-1.1B-Chat-v1.0-int4-ov  --local-dir  TinyLlama-1.1B-Chat-v1.0-int4-ov --local-dir-use-symlinks False
```

Package OpenVINO IR into a tar.gz file

tar -zcvf DeepSeek-R1-Distill-Qwen-7B-int4-ov.tar.gz DeepSeek-R1-Distill-Qwen-7B-int4-ov

Create a file named Modelfile, with a FROM instruction with the local filepath to the model you want to import. For convenience, we have put the model file of DeepSeek-R1-Distill-Qwen-7B-int4-ov model under example dir: Modelfile for DeepSeek, we can use it directly.

Note:
1. The ModelType "OpenVINO" parameter is mandatory and must be explicitly set.
2. The InferDevice parameter is optional. If not specified, the system will prioritize using the GPU by default. If no GPU is available, it will automatically fall back to using the CPU. If InferDevice is explicitly set, the system will strictly use the specified device. If the specified device is unavailable, the system will follow the same fallback strategy as when InferDevice is not set (i.e., GPU first, then CPU).
3. For more information on working with a Modelfile, see the Modelfile documentation.

Unzip OpenVINO GenAI package and set environment

cd openvino_genai_windows_2025.2.0.0.dev20250320_x86_64
setupvars.bat

Create the model in Ollama

ollama create DeepSeek-R1-Distill-Qwen-7B-int4-ov:v1 -f Modelfile

You will see output similar to the following:

   gathering model components 
   copying file sha256:77acf6474e4cafb67e57fe899264e9ca39a215ad7bb8f5e6b877dcfa0fabf919 100% 
   using existing layer sha256:77acf6474e4cafb67e57fe899264e9ca39a215ad7bb8f5e6b877dcfa0fabf919 
   creating new layer sha256:9b345e4ef9f87ebc77c918a4a0cee4a83e8ea78049c0ee2dc1ddd2a337cf7179 
   creating new layer sha256:ea49523d744c40bc900b4462c43132d1c8a8de5216fa8436cc0e8b3e89dddbe3 
   creating new layer sha256:b6bf5bcca7a15f0a06e22dcf5f41c6c0925caaab85ec837067ea98b843bf917a 
   writing manifest 
   success

Run the model

ollama run DeepSeek-R1-Distill-Qwen-7B-int4-ov:v1

CLI Reference

Show model information

ollama show DeepSeek-R1-Distill-Qwen-7B-int4-ov:v1

List models on your computer

ollama list

List which models are currently loaded

ollama ps

Stop a model which is currently running

ollama stop DeepSeek-R1-Distill-Qwen-7B-int4-ov:v1

Building from source

Install prerequisites:

Go
C/C++ Compiler e.g. Clang on macOS, TDM-GCC (Windows amd64) or llvm-mingw (Windows arm64), GCC/Clang on Linux.

Then build and run Ollama from the root directory of the repository:

Windows

clone repo

git lfs install
git lfs clone https://github.com/zhaohb/ollama_ov.git

Enable CGO:
```
go env -w CGO_ENABLED=1
```
Initialize the GenAI environment

Download GenAI runtime from GenAI, then extract it to a directory openvino_genai_windows_2025.2.0.0.dev20250320_x86_64 .
```
cd openvino_genai_windows_2025.2.0.0.dev20250320_x86_64
setupvars.bat
```

Setting cgo environment variables

set CGO_LDFLAGS=-L%OpenVINO_DIR%\..\lib\intel64\Release
set CGO_CFLAGS=-I%OpenVINO_DIR%\..\include

building Ollama
```
go build -o ollama.exe
```
If you don't want to recompile ollama, you can choose to directly use the compiled executable file, and then initialize the genai environment in step 3 to run ollama directly ollama.

But if you encounter the error when executing ollama.exe, it is recommended that you recompile from source code.
```
This ver
9D51
sion of ollama.exe is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.'
```

Linux

clone repo

git lfs install
git lfs clone https://github.com/zhaohb/ollama_ov.git

Enable CGO:
```
go env -w CGO_ENABLED=1
```
Initialize the GenAI environment

Download GenAI runtime from GenAI, then extract it to a directory openvino_genai_ubuntu22_2025.2.0.0.dev20250320_x86_64.
```
cd openvino_genai_ubuntu22_2025.2.0.0.dev20250320_x86_64 
source setupvars.sh
```

Setting cgo environment variables

export CGO_LDFLAGS=-L$OpenVINO_DIR/../lib/intel64/
export CGO_CFLAGS=-I$OpenVINO_DIR/../include

building Ollama
```
go build -o ollama
```
If you don't want to recompile ollama, you can choose to directly use the compiled executable file, and then initialize the genai environment in step 3 to run ollama directly ollama.

If you encounter problems during use, it is recommended to rebuild from source.

Running local builds

First, set GODEBUG=cgocheck=0 env:

Linux

export GODEBUG=cgocheck=0

Windows

set GODEBUG=cgocheck=0

Next, start the server:
```
ollama serve
```

Finally, in a separate shell, run a model:

ollama run DeepSeek-R1-Distill-Qwen-7B-int4-ov:v1

Community Integrations

Terminal

oterm

Web & Desktop

AnythingLLM

Future Development Plan

Here are some features and improvements planned for future releases:

Multimodal models: Support for multimodal models that can process both text and image data.

Name		Name	Last commit message	Last commit date
Latest commit History 3,930 Commits
.github		.github
api		api
app		app
auth		auth
cmd		cmd
convert		convert
discover		discover
dist		dist
docs		docs
envconfig		envconfig
examples/modelfile		examples/modelfile
format		format
fs		fs
genai		genai
images		images
integration		integration
kvcache		kvcache
llama		llama
llm		llm
macapp		macapp
ml		ml
model		model
openai		openai
parser		parser
progress		progress
readline		readline
runner		runner
sample		sample
scripts		scripts
server		server
template		template
types		types
version		version
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
.golangci.yaml		.golangci.yaml
CMakeLists.txt		CMakeLists.txt
CMakePresets.json		CMakePresets.json
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
Dockerfile_genai		Dockerfile_genai
LICENSE		LICENSE
Makefile.sync		Makefile.sync
README.md		README.md
SECURITY.md		SECURITY.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go

License

zhaohb/ollama_ov

Folders and files

Latest commit

History

Repository files navigation

OpenVINO Integration with Ollama

Ollama

macOS

Windows

Linux

Docker

Libraries

Community

Quickstart

Model library

Customize a model

Import from GGUF

Import from Safetensors

Customize a prompt

CLI Reference

Create a model

Pull a model

Remove a model

Copy a model

Multiline input

Multimodal models

Pass the prompt as an argument

Show model information

List models on your computer

List which models are currently loaded

Stop a model which is currently running

Start Ollama

Building

Running local builds

REST API

Generate a response

Chat with a model

Community Integrations

Web & Desktop

Cloud

Terminal

Apple Vision Pro

Database

Package managers

Libraries

Mobile

Extensions & Plugins

Supported backends

Observability

Ollama-ov

Google Driver

Windows

Linux(Ubuntu 22.04)

百度云盘

Windows

Linux(Ubuntu 22.04)

Docker

Linux

Model library

Ollama Model File

Quick start

Start Ollama

Linux

Windows

Import from openVINO IR

Example

CLI Reference

Show model information

List models on your computer

List which models are currently loaded

Stop a model which is currently running

Building from source

Windows

Linux

Running local builds

Linux

Windows

Community Integrations

Terminal

Packages