Running open models locally with Ollama

Cloud APIs are the easiest way to use an LLM, but sometimes you want the model on your own machine: no internet, no per-call bill, and private data that never leaves your laptop. Ollama is a free, open-source tool that downloads open-weight models and runs them locally, with a simple command line and an HTTP API.

In this tutorial you will install Ollama, chat with a model, call it from Python, get structured output, and reuse your existing OpenAI-style code against it. Prerequisites: Python 3.10+, a laptop with at least 8 GB of RAM (more is better), and about 2 to 5 GB of free disk space per small model.

A good rule: use local models for learning, prototypes, offline tools and sensitive data, and for simple tasks like classification or short summaries. Use a cloud model when quality matters most. Many real products use both. See *Prompting vs RAG vs fine-tuning* for how to improve a small model's results.

ollama --version
ollama pull llama3.2
ollama run llama3.2
ollama list
ollama ps

llama3.2 is a small model from Meta that suits laptops. Other families are listed in the model library on ollama.com; pick small versions (for example ones marked 1b, 3b or 4b) first. Model names and sizes change, so check the library.

Ollama's service listens on http://localhost:11434. The official Python library is a thin wrapper around that HTTP API. Install it with pip install ollama. Keep the Ollama app running while you do this.

import ollama

MODEL = "llama3.2"

response = ollama.chat(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a friendly assistant for Brew & Bean, a coffee shop. Keep answers under 30 words."},
        {"role": "user", "content": "Which of your drinks has no milk?"},
    ],
)
print(response.message.content)

Run with python chat_local.py. The messages list works like cloud chat APIs: system, user and assistant roles. To continue a conversation, append the model's reply and the next user message to the list and call again.

import ollama

stream = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain what an API is in two sentences."}],
    stream=True,
)
for chunk in stream:
    print(chunk.message.content, end="", flush=True)
print()

With stream=True you get chunks as they are generated, just as in *Streaming responses to a UI*. This matters even more locally, because small machines generate slowly and waiting for the full answer feels bad.

Ollama can force the reply to match a JSON schema by passing the schema in format. Combined with Pydantic you get validated Python objects, the same idea as in *Structured output with JSON schemas*. Small models still make mistakes in the values, so keep temperature low and validate.

import ollama
from pydantic import BaseModel

class OrderLine(BaseModel):
    item: str
    qty: int

class Order(BaseModel):
    lines: list[OrderLine]

response = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Extract the order: two cappuccinos and one banana bread please"}],
    format=Order.model_json_schema(),
    options={"temperature": 0},
)
order = Order.model_validate_json(response.message.content)
print(order)

Expected result: something like lines=[OrderLine(item='cappuccino', qty=2), OrderLine(item='banana bread', qty=1)]. Item names may vary in wording, so map them to your menu ids in code.

Ollama also exposes an OpenAI-compatible endpoint under /v1. If you already have code written for the OpenAI Python library, often only the base URL and model name need to change. This makes it easy to switch between a local model for development and a cloud model in production. Install with pip install openai.

from openai import OpenAI

# The key is required by the library but ignored by Ollama
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

reply = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Give me one tip for a job interview."}],
)
print(reply.choices[0].message.content)

Compatibility covers the common chat features, but not everything, so test any advanced feature you rely on.

You can also call the plain HTTP API from any language. Here is the same chat with curl (use Git Bash, macOS or Linux; in Windows PowerShell write the JSON to a file or use Invoke-RestMethod):

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Say hello in Hindi."}],
  "stream": false
}'

"stream": false returns one JSON object instead of a stream of lines.

A Modelfile lets you save a system prompt and settings as your own named model.

FROM llama3.2
PARAMETER temperature 0.3
SYSTEM """You are the assistant for Brew & Bean, a coffee shop in Bengaluru. Answer briefly and politely. If you do not know, say so."""

Save as a file called Modelfile (no extension), then run ollama create brew-bot -f Modelfile and ollama run brew-bot.

This is prompting, not training: the model's weights do not change.

I am testing a small local model against a cloud model for [task, e.g. summarising customer reviews]. Here are 5 sample inputs:

[paste inputs]

For each input write the ideal answer in 2 sentences, then list 3 specific mistakes a weak model might make on that input. I will use this as a checklist to compare answers.

What to expect: Use the answers as your scoring rubric: run the same 5 inputs through both models and tick off mistakes. Do not just trust a general feeling.

Expect: "When would you use a local model instead of an API?" Give the trade-offs: privacy, cost at scale, offline, latency control versus lower quality and hardware needs. Mention quantisation (compressing weights so models fit smaller memory, with a small quality cost) and tools such as Ollama and llama.cpp. Show that you would decide with an eval, not a hunch.