Multi-modal LLMs for production: when to use image, audio and video capabilities

TL;DR

Adding image, audio or video inputs to your LLM product is no longer a novelty — GPT-5o (OpenAI, June 2026), Claude 4 Sonnet and Gemini 3 all handle visual and audio natively at competitive prices. The real question is whether the capability solves a specific problem cheaply enough to justify the added complexity. Use a cost-accuracy-latency matrix before committing: if your primary use case does not beat the baseline text-only or OCR approach, hold off until pricing improves.

When multimodal adds real value vs. when it is a gimmick

The frontier models converged on support for images, audio and video roughly in lock-step during 2025. What separated the useful features from the hype turned out to be use case fit. Image input pays off when users need visual context that text alone cannot carry well enough: product returns (“my item arrived damaged”), real estate listings (“is this kitchen move-in ready?”) and medical imaging workflows (preliminary radiology triage where a model flags urgent cases for human review). Anthropic’s Claude 4 Sonnet benchmarks published in early 2026 showed a 38% improvement over previous versions on visual reasoning tasks, with particular gains on text-heavy images like scanned forms and receipts.

Audio input makes sense when your users are on the move. Dictation workflows, call centre transcription and accessibility tools for the visually impaired all benefit from native audio understanding without an intermediate speech-to-text pipeline. Google Gemini 3’s evaluation paper demonstrated that direct audio tokenisation removes a failure mode common in serialised pipelines: the TTS-then-VLM approach silently drops emphasis and emotional tone because each step compresses information independently.

Video input remains the hardest case to justify in production today. While all three providers support video frames as input, the per-frame pricing adds up fast and latency compounds with duration. A single 30-second clip at 2 fps yields 60 images — at current OpenAI GPT-5o pricing of roughly $1.25 per 1M input image tokens (with a standard 468-token equivalent per image), that alone runs well over $0.75 for one video upload if the model attends to every frame. Video makes sense only when temporal reasoning is essential and no structured or event-stream alternative exists.

A practical test before committing: can you get 80% of the value from OCR + text RAG, server-side event processing (WebRTC frames on changes), or a dedicated specialist model? If yes, skip the multimodal LLM for now.

Cost model: per-image vs per-token pricing

Multimodal input breaks the traditional token-counting model in ways that matter for production budgets. Image and audio inputs are priced as equivalent token counts based on pixel area or sample count, not on linguistic complexity. An image’s cost is proportional to its dimensions squared times the model’s spatial resolution multiplier. GPT-4o and GPT-5o use a grid of 560x560 tokens per image at base resolution; additional tokens are added for each zoom level the model uses to examine fine detail. Claude 4 Sonnet similarly uses a token-area calculation with its Haiku model handling lower-resolution passes before the main model looks deeper.

Audio pricing works differently again. Providers charge per second of audio input, typically converted to an equivalent token count at a fixed ratio (Google uses 1 second = ~16 tokens for direct audio input; OpenAI maps audio samples similarly). For comparison, OpenAI’s GPT-5o charges $2.50 per 1M input image tokens and around $2.40 per million output text tokens as of mid-2026, while Claude 4 Sonnet’s pricing through Anthropic sits at roughly $3.00/1M for image input on the Sonnet tier. Google charges less per second of audio but can be more expensive for longer clips because their model processes every second rather than chunking efficiently.

Provider pricing as of June 2026:

For document-heavy workloads, dedicated OCR services like AWS Textract or Google Document AI can actually be cheaper than sending documents through a multimodal LLM — but only if you have straightforward layouts. Once documents contain handwritten text, mixed languages, or complex structures that require comprehension (not just character extraction), the multimodal LLM often wins on both accuracy and end-to-end simplicity despite higher raw per-request cost.

Latency impact of different modalities

Multimodal input adds latency overhead beyond what you see with text-only prompts, and the amount depends on which modality you are using. For images, providers typically process frames in parallel during encoding before feeding them to the transformer — a typical GPT-5o image+text prompt (per OpenAI API docs, June 2026) completes faster than Claude 4 Sonnet for equivalent queries, around 1.2 to 2 seconds versus 2 to 3.5 seconds from initial upload to first output token on average (measured with medium-resolution images under 1 MB).

Audio input adds a preprocessing step: the model must convert raw audio samples into acoustic embeddings before the language head can begin generation. This is why Claude 4 Sonnet, which processes audio natively rather than routing through Whisper first, shows lower overall latency than workflows that chain external STT models to vision-language models. Average time-to-first-token for audio prompts is typically 2–5 seconds longer than text-only counterparts depending on clip length.

Video remains the heaviest: providers process frames at reduced rates (usually 1–2 fps) and attend across temporal windows during generation. A single query on a 30-second video clip can take anywhere from 8 to 30 seconds end-to-end on current provider APIs as of June 2026, compared to under 2 seconds for an equivalent text prompt. For production systems where users expect near-instant responses, this latency gap is often the deciding factor against video input — even when the output quality justifies it architecturally.

# Rough latency estimate guide for multimodal LLM calls (production estimates)
LATENCY_ESTIMATES = {
  "text_only": {"min_ms": 300, "max_ms": 2000},
  "image_text": {"min_ms": 800, "max_ms": 4000},
  "audio_text": {"min_ms": 1500, "max_ms": 6000},
  "video_text_30s": {"min_ms": 8000, "max_ms": 30000},
}

Accuracy benchmarks by modality

Vision models have improved dramatically since GPT-4V arrived in 2023. Current generations like GPT-5o and Claude 4 Sonnet score above 96% on standard document parsing tasks for clean, printed text and can handle mixed-language documents reasonably well (though accuracy drops sharply below 70% for CJK scripts without explicit fine-tuning or prompting context).

Where vision models still struggle, as of mid-2026:

GPT-4V’s evaluation (June 2023 baseline, still representative of the architecture but with improvements through GPT-5o) showed that while OCR on clean documents was already near-human, tasks requiring spatial reasoning about diagrams and charts sat at only 68% accuracy. The gap between “can read text in an image” and “can understand what that text means in context” is where teams should pay attention.

Anthropic published Claude 4 Sonnet multimodal benchmarks in early 2026 claiming state-of-the-art performance on visual reasoning (89/100 on their internal test set) but also acknowledged a significant drop-off on edge cases: medical imaging, microscopic images and technical schematics still require domain-specific fine-tuning or specialist models that outperform general-purpose multimodal LLMs on targeted benchmarks.

For audio, the accuracy picture is similar. Google’s Gemini 3 evaluation paper (per Google DeepMind, January 2026) reported speech recognition accuracy of 2-4% word error rate across its test set with clear English audio — but at $0.30/min for long recordings through their API, the cost adds up compared to streaming STT approaches when you have tens of thousands of users.

Decision matrix: choosing the right modality

Use this framework to decide whether multimodal input makes sense for your product:

The key pattern: if your task is pure perception (classify, detect, transcribe), a specialist model is likely cheaper and faster. If your product requires comprehension or reasoning over the visual/audio content after extraction, multimodal LLMs are worth the cost.

Practical implementation patterns

The most common production pattern for multi-modal features today is an image-first API path: accept the media through your server, compress and resize it client-side (always send the smallest resolution that preserves information), run a fast specialist check (e.g. is this a receipt or a photo?), then decide whether to route to a specialised extraction service or a multimodal LLM based on confidence from the first pass.

When building with multiple modalities, cache results aggressively — GPT-5o and Claude 4 Sonnet both include prompt caching for repeated visual contexts (documents that appear across multiple queries) at significant discounts: roughly half-price if the document image is identical between calls or within a small hash range. The same principle applies to audio: if your users upload the same recording twice, return the cached parse rather than re-billing for every token.

For latency-sensitive applications, chunk-and-stream both the visual input and the model output. Process images frame-by-frame with a fast classifier first (is this image worth detailed analysis?), only sending high-confidence candidates to the full multimodal LLM. Use the approach from Latency in LLM apps: first-token, total time and user experience to structure your streaming UX so users see partial results while the model processes higher-resolution frames.

Methodology

• Data checked: 2026-06-24
• Sources consulted: Provider pricing pages (OpenAI, Anthropic, Google Cloud), model benchmark papers (Claude 4 Sonnet multimodal benchmarks 2026, Gemini 3 multimodal evaluation paper), industry reports on multimodal production deployments in healthcare and document processing sectors
• Assumptions: Prices quoted reflect public API rates at time of writing (June 2026). Internal model performance may differ for organisations with custom fine-tuning or enterprise agreements. Reader is assumed to have a working knowledge of LLM architecture concepts including context windows and tokenization.
• Limitations: This article focuses on public API-based multimodal input; it does not cover self-hosted multimodal models, local inference deployments, or model training/fine-tuning for specialized visual/audio tasks. Pricing is provided as reference only — actual costs vary by provider contract terms.
• Jurisdiction: Global. The pricing and benchmark data referenced applies to US/EU/US regions unless noted. No regulatory analysis of multimodal AI usage is included beyond noting the EU AI Act’s classification of vision-language systems as high-risk if deployed for biometric identification or remote behavioral analysis.

Source list

• OpenAI GPT-5o pricing — https://openai.com/api/pricing/ (accessed 2026-06-24)
• Anthropic Claude 4 Sonnet multimodal benchmarks — https://www.anthropic.com/research/claude-4-sonnet-multimodal-evaluation (accessed 2026-06-24)
• Google Gemini 3 multimodal evaluation paper — https://storage.googleapis.com/deepmind-media/gemini/google-ai-gemini-3-jan2026.pdf (accessed 2026-06-24)
• AWS Textract pricing and documentation — https://aws.amazon.com/textract/pricing/ (accessed 2026-06-24)
• Google Document AI pricing — https://cloud.google.com/document-ai/pricing (accessed 2026-06-24)
• Chatbot Arena image modality leaderboards — https://lmsys.org/leaderboard/image/ (accessed 2026-06-24)

Trust Stack

• Last checked: 2026-06-24
• Corrections: Contact us to report errors

Change log

• 2026-06-24: First draft published

Related guides

• Multimodal models explained: text, images, audio and video in practical products
• Latency in LLM apps: first-token, total time and user experience
• LLM inference cost per query: a real-world estimator