Models and Fine-tuning

This guide covers the available models in Ollama, how to use them, and techniques for customizing models to suit your specific requirements.

Available Models

Ollama supports a variety of open-source LLMs. Here are some of the most commonly used models:

General-Purpose Models

Model	Size	Description	Command
Llama 2	7B to 70B	Meta's general-purpose model	ollama pull llama2
Mistral	7B	High-quality open-source model	ollama pull mistral
Mixtral	8x7B	Mixture-of-experts model	ollama pull mixtral
Phi-2	2.7B	Microsoft's compact model	ollama pull phi
Neural Chat	7B	Optimized for chat	ollama pull neural-chat
Vicuna	7B to 33B	Fine-tuned LLaMa model	ollama pull vicuna

Code-Specialized Models

Model	Size	Description	Command
CodeLlama	7B to 34B	Code-focused Llama variant	ollama pull codellama
WizardCoder	7B to 34B	Fine-tuned for code tasks	ollama pull wizardcoder
DeepSeek Coder	6.7B to 33B	Specialized for code	ollama pull deepseek-coder

Small/Efficient Models

Model	Size	Description	Command
TinyLlama	1.1B	Compact model for limited resources	ollama pull tinyllama
Gemma	2B to 7B	Google's lightweight model	ollama pull gemma
Phi-2	2.7B	Efficient and compact	ollama pull phi

Multilingual Models

Model	Description	Command
BLOOM	Multilingual capabilities	ollama pull bloom
Qwen	Chinese and English	ollama pull qwen
Japanese Stable LM	Japanese language	ollama pull stablej

Model Management

Listing Models

# List all downloaded models
ollama list

Pulling Models

# Pull a specific model version
ollama pull mistral:7b-v0.1

Removing Models

# Remove a model
ollama rm mistral

Model Parameters

Control model behavior with these parameters:

Parameter	Description	Range
temperature	Controls randomness	0.0 - 2.0
top_p	Nucleus sampling threshold	0.0 - 1.0
top_k	Limits vocabulary to top K tokens	1 - 100+
context_length	Maximum context window size	Model dependent
seed	Random seed for reproducibility	Any integer

Example usage:

# Run a model with specific parameters
ollama run mistral --temperature 0.7 --top_p 0.9

Customizing Models with Modelfiles

Ollama uses Modelfiles (similar to Dockerfiles) to create custom model configurations.

Basic Modelfile Example

FROM mistral:latest
PARAMETER temperature 0.7
SYSTEM You are an expert DevOps engineer specializing in cloud infrastructure.

Save this in a file named Modelfile and create a custom model:

ollama create devops-assistant -f ./Modelfile
ollama run devops-assistant

Advanced Modelfile Example

FROM codellama:latest
PARAMETER temperature 0.3
PARAMETER top_p 0.8
PARAMETER stop "```"
TEMPLATE """
<system>
You are a senior software developer specialized in infrastructure as code, container orchestration, and CI/CD pipelines.
</system>

<user>
{{.Prompt}}
</user>

<assistant>
"""

Modelfile Commands Reference

Command	Description	Example
FROM	Base model	FROM mistral:latest
PARAMETER	Set inference parameter	PARAMETER temperature 0.7
SYSTEM	Set system message	SYSTEM You are a helpful assistant
TEMPLATE	Define chat template	TEMPLATE <s>{{.System}}</s>{{.Prompt}}
ADAPTER	Apply LoRA adapter	ADAPTER ./adapter.bin
MESSAGE	Add example conversation	MESSAGE user "Hi"
PROMPT	Default prompt	PROMPT Answer in bullet points
LICENSE	Specify model license	LICENSE MIT

Fine-tuning with Custom Data

While Ollama doesn't directly support fine-tuning, you can use pre-fine-tuned models and adapt them with Modelfiles.

Using External Fine-tuned Models

1. Fine-tune a model using an external tool like LLaMA Factory
2. Convert the model to GGUF format

3. Import into Ollama:

# Import a fine-tuned model
ollama create custom-model -f Modelfile

With Modelfile:

FROM ./path/to/fine-tuned-model.gguf
PARAMETER temperature 0.7

Behavior Fine-tuning with Examples

You can "soft fine-tune" model behavior by providing examples in the Modelfile:

FROM mistral:latest
SYSTEM You are a DevOps automation expert who explains complex concepts clearly.

# Include examples of desired behavior
MESSAGE user "How can I automate AWS infrastructure provisioning?"
MESSAGE assistant "For AWS infrastructure automation, I recommend these approaches:

1. **Terraform**: Infrastructure as Code (IaC) tool with declarative configuration
   - Maintains state of resources
   - Supports modules for reusability
   - Works with multiple cloud providers

2. **AWS CloudFormation**: Native AWS IaC service
   - JSON/YAML templates
   - Stack management with change sets
   - Deep integration with AWS services

3. **AWS CDK**: Define infrastructure using familiar programming languages
   - TypeScript, Python, Java support
   - Compiles to CloudFormation
   - Object-oriented approach

For most teams, Terraform offers the best balance of flexibility and functionality."

Model Quantization

Ollama supports various quantization levels to balance performance and resource usage:

Quantization	Size Impact	Performance	Command
Q4_K_M	3-4GB (7B model)	Good	ollama pull mistral:7b-q4_k_m
Q5_K_M	4-5GB (7B model)	Better	ollama pull mistral:7b-q5_k_m
Q8_0	7-8GB (7B model)	Best	ollama pull mistral:7b-q8_0

For resource-constrained environments, use more aggressive quantization:

# Pull a highly quantized model
ollama pull tinyllama:1.1b-q4_0

RAG (Retrieval-Augmented Generation)

Enhance models with external knowledge using RAG:

#!/bin/bash

# Simple RAG implementation with Ollama
MODEL="mistral:latest"
QUERY="What are the key components of a Kubernetes cluster?"
CONTEXT_FILE="kubernetes-docs.txt"

# Get context from a document
CONTEXT=$(grep -i "kubernetes components\|control plane\|node components" "$CONTEXT_FILE" | head -n 15)

# Create prompt with context
PROMPT="Based on the following information:\n\n$CONTEXT\n\nPlease answer: $QUERY"

# Send to Ollama
ollama run $MODEL --prompt "$PROMPT"

Practical Model Selection Guide

Use Case	Recommended Model	Why
General chat	mistral:7b	Good balance of size and capability
Code assistance	codellama:7b	Specialized for code understanding/generation
Resource-constrained	tinyllama:1.1b	Small memory footprint
Technical documentation	neural-chat:7b	Clear instruction following
Complex reasoning	mixtral:8x7b or llama2:70b	Sophisticated reasoning capabilities

DevOps-Specific Model Configuration

For DevOps-specific tasks, create a specialized model configuration:

# DevOps Assistant Modelfile
FROM codellama:latest
PARAMETER temperature 0.3
PARAMETER top_p 0.8
SYSTEM You are an expert in DevOps practices, cloud infrastructure, CI/CD pipelines, and infrastructure as code. You provide concise, accurate answers with practical examples when appropriate. You're familiar with AWS, Azure, GCP, Kubernetes, Docker, Terraform, Ansible, GitHub Actions, and other DevOps tools.

# Example prompt for debugging
PROMPT """
I'm encountering the following issue with my CI/CD pipeline or infrastructure:

{{.Input}}

Please help me by:
1. Identifying potential causes
2. Suggesting troubleshooting steps
3. Recommending a solution
4. Providing a brief example if applicable
"""

Create this model:

ollama create devops-assistant -f ./DevOps-Modelfile

Next Steps

Now that you understand Ollama's models:

1. Configure GPU acceleration to speed up model inferencing
2. Set up Open WebUI for a graphical interface
3. Explore DevOps usage examples for practical applications