Fine-tuning LLMs: a practical step-by-step guide

TL;DR

Fine-tuning is a weekend-sized project, not a research-team-sized one. With LoRA or QLoRA, a single developer with a rented GPU can train a 7B model on 200–500 examples in under 3 hours for under £20. The real work is not running the training job — it is curating high-quality examples, measuring whether the fine-tune actually improved anything, and deciding when to re-train.

The hard part is not running the training job. It is knowing whether you have the right data, whether you chose the right method, and whether the result actually improved anything.

This guide walks through the whole process with a concrete worked example: fine-tuning a 7B model to output structured JSON consistently. We will cover method selection, data preparation, the training run, evaluation, deployment, and what to watch for after the fine-tune is live.

If you are not sure fine-tuning is the right move for your task yet, start with the fine-tuning vs prompting vs RAG decision checklist.

When fine-tuning makes sense

Fine-tuning is not the default fix for model problems. Before you invest time in a fine-tuning pipeline, check that your situation matches the profile:

Fine-tuning works well when:

The task is stable and repeated often enough that per-query prompt engineering or few-shot examples cost more than the fine-tune. You have 50–500+ high-quality examples of the behaviour you want. The behaviour you need is narrow: consistent output format, domain-specific terminology, a particular tone or style. You can measure whether the fine-tune improved things (see the evaluation section below).

Fine-tuning is usually the wrong move when:

The problem is missing or stale context — that is a retrieval issue, not a behaviour issue. You have fewer than 30 good examples. The task changes frequently — every time the task shifts, your fine-tune decays. You have not tried the strongest prompt + few-shot baseline first.

For the full decision framework, see our fine-tuning vs prompting vs RAG decision checklist. For a breakdown of the conceptual layers (what fine-tuning actually changes in the model), see inference vs training vs fine-tuning: three terms operators confuse. And for the full economic picture — when fine-tuning pays back and at what volume — see fine-tuning economics.

Choose your fine-tuning method

For most first-time fine-tuners, start with LoRA or QLoRA. They are the pragmatic default: cheap enough to iterate on, easy to swap between adapters, and the results are often close to full fine-tuning quality on narrow tasks. Use full fine-tuning only when LoRA results are demonstrably insufficient, or when you need the extra capacity of a full-weight update for a complex multi-task dataset.

Provider fine-tuning APIs (OpenAI, Together, Fireworks) are a good option if you want to avoid any GPU management. The trade for this is that you are locked into the provider’s supported base models and their data handling policies. OpenAI’s fine-tuning API, for example, supports gpt-4o-mini, gpt-4o, and gpt-4.1 variants at per-token training rates.

Prepare your dataset

What good training data lookss like

Good fine-tuning examples are:

|| Consistent in format — every example follows the same input-output structure. || Representative of the live task — the examples should mirror the prompts and expected responses your production system will see, not synthetic idealisations. || Free of contradictions — if the same prompt maps to different correct outputs across examples, the model will learn to hedge or oscillate. || Split into train and validation sets — typically 80/20 or 90/10. The validation set is never used for training; it exists solely to measure whether the fine-tune is improving.

Format conventions

For Hugging Face PEFT/transformers, the standard format is a conversation template matching the base model’s chat format:

[
  {
    "role": "user",
    "content": "Extract the company name, date, and total amount from this invoice text:..."
  },
  {
    "role": "assistant",
    "content": "{\"company\": \"Acme Corp\", \"date\": \"2026-05-15\", \"total\": 1427.50}"
  }
]

Hugging Face’s transformers library uses the model’s tokeniser chat template to format this into the pre-formatted prompt structure. If you are using Llama, Mistral, or Qwen base models, the tokeniser’s apply_chat_template method handles this automatically.

For OpenAI’s fine-tuning API, the format is a JSONL file where each line is a complete training example with messages array:

{"messages": [{"role": "rag", "content": "Extract the company name from: Invoice from BrightSpark Ltd dated 12/03/2026 for £892.40"}, {"role": "assistant", "content": "BrightSpark Ltd"}]}
{"messages": [{"role": "rag", "content": "Extract the total amount from: Invoice from BrightSpark Ltd dated 12/03/2026 for £892.40"}, {"role": "assistant", "content": "892.40"}]}

OpenAI’s fine-tuning docs recommend at least 5/100 high-quality examples for reliable results. Below that threshold, few-shot prompting will almost certainly outperform fine-tuning at lower cost.

Dataset curation checklist

| [ ] Remove duplicate or near-duplicate examples. | [ ] Check for contradictory labels on the same prompt pattern. | [ ] Verify the validation set has no overlap with the training set. | [ ] If the task has natural categories (invoice vs receipt, short answer vs long form), ensure balanced representation. | [ ] Run the validation set through the base model first to establish a pre-fine-tune baseline.

Worked example: structured JSON output from a 7B model

This example walks through fine-tuning a 7B model (Mistral 7B or Llama 3.1 8B) to consistently output structured JSON for a document extraction task.

Scenario

You have a dataset of 500 text snippets — invoices, receipts, and purchase orders — and you want the model to output a consistent JSON object with {company, date, total, currency, document_type} for each one.

The base model produces approximately 60% valid JSON on this task. The remaining 40% produce missing keys, malformed JSON, or free-text descriptions instead of structured output. That is not production-worthy.

Step 1: Set up the environment

pip install transformers peft accelerate bitsandbytes torch

This gives you: transformers — model loading, tokenisation, training loop peft — LoRA / QLoRA adapter layer injection accelerate — efficient distributed training (single-GPU works fine) bitsandbytes — 4-bit quantisation for QLoRA memory savings torch — PyTorch backend

Step 2: Load the base model with LoRA config

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model_name = "mistralai/Mistral-7B-v0.3"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# 4-bit QLoRA loading
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = prepare_model_for_kbit_training(model)

# LoRA config — target query and value projections
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

This configures a LoRA rank of 16 (a good starting point for most tasks; lower for very small datasets, higher for more complex behaviour changes) targeting the query and value projection layers. This keeps the trainable parameters to roughly 0.1–0.5% of the total model size.

Step 3: Format the dataset

Each example should be formatted as a conversation with a user prompt and the expected JSON output:

def format_example(example):
    prompt = f"Extract the company, date, total, currency, and document type from this text:\n{example['input_text']}"
    response = json.dumps({
        "company": example["company"],
        "date": example["date"],
        "total": example["total"],
        "currency": example["currency"],
        "document_type": example["doc_type"]
    })
    return {"prompt": prompt, "response": response}

def tokenize_example(example, tokenizer, max_length=1024):
    messages = [
        {"role": "user", "content": example["prompt"]},
        {"role": "assistant", "content": example["response"]}
    ]
    formatted = tokenizer.apply_chat_template(messages, tokenize=False)
    tokens = tokenizer(
        formatted,
        truncation=True,
        max_length=max_length,
        padding="max_length",
        return_tensors="pt"
    )
    tokens["labels"] = tokens["input_ids"].clone()
    return tokens

The critical detail: use the tokeniser’s apply_chat_template method rather than building the prompt string manually. Each base model family (Llama, Mistral, Qwen, Gemma) uses a different chat template, and the tokeniser knows which one to apply.

Step 4: Train

training_args = TrainingArguments(
    output_dir="./mistral-json-finetune",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
)

trainer.train()

Key parameters explained:

per_device_train_batch_size=4 + gradient_accumulation_steps=4 gives an effective batch size of 16 without requiring the memory for 16 simultaneous samples. This is a standard configuration for a 24 GB card with QLoRA on a 7B model. num_train_epochs=3 — start with 2–3 epochs. Too few and the model does not adapt; too many and it overfits. Monitor the eval loss: if it starts rising while training loss keeps dropping, you have overfit. learning_rate=2e-4 — standard LoRA learning rate. Full fine-tuning typically uses a lower rate (1e-5 to 5e-5). save_strategy="epoch" with load_best_model_at_end=True — the Trainer will automatically load the checkpoint with the lowest eval loss.

Training on 500 examples should take approximately 1–2 hours on a single RTX 4090 or L4 GPU.

Step 5: Check the loss curves

After training, inspect the training and evaluation loss curves:

import matplotlib.pyplot as plt

logs = trainer.state.log_history
train_loss = [l["loss"] for l in logs if "loss" in l]
eval_loss = [l["eval_loss"] for l in logs if "eval_loss" in l]

plt.plot(train_loss, label="train loss")
plt.plot(eval_loss, label="eval loss")
plt.legend()
plt.savefig("loss_curves.png")

What to look for:

Both curves dropping and converging: the model is learning without overfitting. Green light. Train loss drops, eval loss rises: overfitting. Reduce epochs, add dropout, or increase the dataset size. Both curves flat or erratic: the learning rate is too high, the dataset is too noisy, or the LoRA rank is too low for the task complexity.

Evaluate the fine-tune

Loss curves tell you the model learned something. They do not tell you whether that something is useful for your task. You need a task-specific evaluation.

Pre/post comparison

Run the same 50–100 validation examples through both the base model and the fine-tuned model. For the structured JSON task, measure:

These are not hypothetical numbers — this is the kind of improvement a single LoRA epoch on 400–500 examples typically achieves for a narrow formatting task. If your results are substantially worse, check your dataset quality before adjusting the training config.

Regression testing

The most common fine-tuning trap: the model improves on the target task but degrades on unrelated tasks. This is called catastrophic forgetting.

Test for this by running the fine-tuned model on 20–30 examples from tasks it could already do well before fine-tuning — general Q&A, basic reasoning, summarisation. If the fine-tuned model scores worse than the base model on these, you need to either:

Reduce the number of training epochs. Add examples of the unrelated tasks to the training set (multi-task fine-tuning). Switch to a softer fine-tuning method like LoRA with a lower rank.

Evaluation dataset maintenance

Create a fixed evaluation dataset that never changes. This lets you compare fine-tune runs across weeks and months. The risk of evaluation dataset overfitting (fine-tuning to the eval set rather than the task) is real but manageable with a large enough held-out set — aim for 100+ examples.

For more on building evaluation infrastructure, see our guide on building an LLM evaluation pipeline from prompts to production.

Deploy and serve

Once you have a fine-tuned LoRA adapter, you need to decide how to serve it.

Option A: Merge and serve with vLLM

The most production-ready approach for open-weight models is to load the adapter with vLLM, which supports LoRA adapters natively:

pip install vllm

# Start the server with the LoRA adapter
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-v0.3 \
    --enable-lora \
    --lora-modules json-extractor=./mistral-json-finetune \
    --port 8000

You can then call the fine-tuned model via a standard OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none"
)

response = client.chat.completions.create(
    model="json-extractor",
    messages=[
        {"role": "user", "content": "Extract fields from: Order confirmation from GreenTech Solutions, 2026-04-22, total €2,340.00"}
    ]
)
print(response.choices[0].message.content)
# {"company": "GreenTech Solutions", "date": "2026-04-22", "total": 2340.00, "currency": "EUR", "document_type": "order_confirmation"}

vLLM handles adapter loading at runtime without modifying the base model file, so you can switch between adapters per-request. This is the recommended serving approach for production workloads.

Option B: Hugging Face Inference Endpoints

For teams that prefer managed infrastructure, Hugging Face Inference Endpoints supports custom LoRA adapters. Upload the adapter to the Hugging Face Hub, create an Inference Endpoint with the base model, and attach the adapter in the endpoint configuration.

Option C: Export the adapter and load at inference time

If you are running on a single machine and do not need the full vLLM stack, you can save and load the adapter directly with PEFT:

# Save the adapter
model.save_pretrained("./mistral-json-finetune-adapter")

# Load later
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.3")
fine_tuned_model = PeftModel.from_pretrained(base_model, "./mistral-json-finetune-adapter")

Serving cost notes

The LoRA adapter is typically 5–50 MB — orders of magnitude smaller than the base model (13–15 GB). Storage is negligible. Inference cost is identical to running the base model. The adapter adds negligible latency overhead (a few extra matrix multiplications per layer). vLLM with LoRA support adds roughly 1–2% overhead compared to running the base model alone, making it the most cost-effective serving method for fine-tuned models.

Maintain over time

A fine-tuned model is not fire-and-forget. Here is what degrades it and what to do about it:

Data drift

If your production data shifts — different document layouts, new fields to extract, different languages — the fine-tune’s accuracy will decline. Monitor your task-specific metrics (valid JSON rate, field accuracy) weekly. When they drop below your acceptable threshold, it is time to re-fine-tune with fresh data.

Base model deprecation

Providers retire base model versions periodically. If the base model your adapter was trained on is no longer available, you may need to re-fine-tune on the replacement model. Open-weight model deprecation is less dramatic (you can keep the weights) but you may miss upstream improvements. Schedule a base model version check every 3–6 months.

Versioning

Treat your fine-tuned model as code:

Tag each adapter with the training dataset version and base model version. Keep the evaluation results from each run in a changelog. Never overwrite an adapter — create a new version and redirect traffic when verified.

When to stop fine-tuning

If you need to re-fine-tune more than once a quarter, ask whether fine-tuning is still the right tool. A pattern that changes faster than your retraining cadence may be better served by prompting with few-shot examples dropped in as dynamic context, rather than baked into weights.

Methodology

Data checked: 2026-05-30 Sources consulted: OpenAI fine-tuning documentation, Hugging Face PEFT/transformers docs, LoRA paper (Hu et al. 2021), QLoRA paper (Dettmers et al. 2023), vLLM LoRA serving docs Assumptions: The reader has Python experience and access to at least one GPU with 24 GB VRAM (rented cloud GPU is sufficient). The worked example targets models in the 7B–8B parameter range. Limitations: Fine-tuning tooling and provider APIs change faster than this guide can keep up. The training parameters and costs shown are accurate as of mid-2026 but may drift. Always verify current provider docs and GPU pricing before committing to a fine-tuning run. Jurisdiction: Global. Provider fine-tuning availability and data residency rules vary by region. Check provider terms for data processing location if you are handling EU or UK-regulated data.

Source list

OpenAI fine-tuning docs — https://platform.openai.com/docs/guides/fine-tuning (accessed 2026-05-30) Hu et al. LoRA paper — https://arxiv.org/abs/2106.09685 (accessed 2026-05-30) Dettmers et al. QLoRA paper — https://arxiv.org/abs/2305.14314 (accessed 2026-05-30) Hugging Face Transformers docs — https://huggingface.co/docs/transformers/training (accessed 2026-05-30) vLLM LoRA serving docs — https://docs.vllm.ai/en/latest/features/lora.html (accessed 2026-05-30)

Related guides guides

| Fine-tuning vs prompting vs RAG: decision checklist | Inference vs training vs fine-tuning: three tokens operators confuse | Fine-tuning economics: when training a custom model pays back | Building an LLM evaluation pipeline: from prompts to production

Trust Stack

Last checked: 2026-05-30 Corrections: Contact us to report errors

Change log

2026-05-30: First published. Covers method selection, dataset preparation, worked example (7B model → structured JSON), evaluation methodology, deployment options (vLLM, HF endpoints, direct PEFT loading), and maintenance guidance.