Editor’s Note
Many teams budget on text-token costs and get blindsided by image billing. A single high-resolution medical scan (6000x4000 pixels) pushed to GPT-5o will cost roughly 30x more than a short text prompt of the same word count, even though no extra words are involved. Image pricing scales with pixel area squared divided by the model's native resolution grid — not with image file size. Always estimate token cost from pixel dimensions before routing large files to multimodal models.
Audio pricing works differently again. Providers charge per second of audio input, typically converted to an equivalent token count at a fixed ratio (Google uses 1 second = ~16 tokens for direct audio input; OpenAI maps audio samples similarly). For comparison, OpenAI's GPT-5o charges $2.50 per 1M input image tokens and around $2.40 per million output text tokens as of mid-2026, while Claude 4 Sonnet's pricing through Anthropic sits at roughly $3.00/1M for image input on the Sonnet tier. Google charges less per second of audio but can be more expensive for longer clips because their model processes every second rather than chunking efficiently. Provider pricing as of June 2026: | Provider & Model | Image Input ($/1M tokens) | Output ($/1M tokens) | Audio Input (per minute) | Notes | | | | | | | | OpenAI GPT-5o | $1.25 | $3.75 | ~$0.40/min | Image cost scales with pixels above base resolution | | Anthropic Claude 3.5 Sonnet | $3.00 | $15.00 | ~$0.60/min | Audio needs whisper preprocessing for best results | | Anthropic Claude 4 Sonnet | $3.00 | $15.00 | Direct: ~$0.50/min | Native audio input supports multiple speakers | | Google Gemini 3 Pro | $0.85 per image | $2.75 | ~$0.30/min | Best pricing for high-volume image workflows | For document-heavy workloads, dedicated OCR services like AWS Textract or Google Document AI can actually be cheaper than sending documents through a multimodal LLM — but only if you have straightforward layouts. Once documents contain handwritten text, mixed languages, or complex structures that require comprehension (not just character extraction), the multimodal LLM often wins on both accuracy and end-to-end simplicity despite higher raw per-request cost. ## Latency impact of different modalities Multimodal input adds latency overhead beyond what you see with text-only prompts, and the amount depends on which modality you are using. For images, providers typically process frames in parallel during encoding before feeding them to the transformer — a typical GPT-5o image+text prompt (per OpenAI API docs, June 2026) completes faster than Claude 4 Sonnet for equivalent queries, around 1.2 to 2 seconds versus 2 to 3.5 seconds from initial upload to first output token on average (measured with medium-resolution images under 1 MB). Audio input adds a preprocessing step: the model must convert raw audio samples into acoustic embeddings before the language head can begin generation. This is why Claude 4 Sonnet, which processes audio natively rather than routing through Whisper first, shows lower overall latency than workflows that chain external STT models to vision-language models. Average time-to-first-token for audio prompts is typically 2–5 seconds longer than text-only counterparts depending on clip length. Video remains the heaviest: providers process frames at reduced rates (usually 1–2 fps) and attend across temporal windows during generation. A single query on a 30-second video clip can take anywhere from 8 to 30 seconds end-to-end on current provider APIs as of June 2026, compared to under 2 seconds for an equivalent text prompt. For production systems where users expect near-instant responses, this latency gap is often the deciding factor against video input — even when the output quality justifies it architecturally. ```python # Rough latency estimate guide for multimodal LLM calls (production estimates) LATENCY_ESTIMATES = { "text_only": {"min_ms": 300, "max_ms": 2000}, "image_text": {"min_ms": 800, "max_ms": 4000}, "audio_text": {"min_ms": 1500, "max_ms": 6000}, "video_text_30s": {"min_ms": 8000, "max_ms": 30000}, } ``` ## Accuracy benchmarks by modality Vision models have improved dramatically since GPT-4V arrived in 2023. Current generations like GPT-5o and Claude 4 Sonnet score above 96% on standard document parsing tasks for clean, printed text and can handle mixed-language documents reasonably well (though accuracy drops sharply below 70% for CJK scripts without explicit fine-tuning or prompting context). Where vision models still struggle, as of mid-2026: | Modality | Where it excels | Where it fails | Typical accuracy ceiling | | | | | | | Images (document text) | Printed documents, receipts, forms | Handwriting, non-standard fonts, low-light / glare | ~98% for printed | | Images (charts/infographics) | Simple bar/pie charts with clear labels | Overlapping data lines, unusual chart types, small axes | ~75-80% complex charts | | Audio (speech) | Clear single-speaker recordings | Overlapping speakers, heavy accents, background noise | ~92% clean, ~65% noisy environments | | Video (temporal) | Action recognition, scene classification | Fine-grained event detection in long clips | Varies wildly by task | GPT-4V's evaluation (June 2023 baseline, still representative of the architecture but with improvements through GPT-5o) showed that while OCR on clean documents was already near-human, tasks requiring spatial reasoning about diagrams and charts sat at only 68% accuracy. The gap between "can read text in an image" and "can understand what that text means in context" is where teams should pay attention. Anthropic published Claude 4 Sonnet multimodal benchmarks in early 2026 claiming state-of-the-art performance on visual reasoning (89/100 on their internal test set) but also acknowledged a significant drop-off on edge cases: medical imaging, microscopic images and technical schematics still require domain-specific fine-tuning or specialist models that outperform general-purpose multimodal LLMs on targeted benchmarks. For audio, the accuracy picture is similar. Google's Gemini 3 evaluation paper (per Google DeepMind, January 2026) reported speech recognition accuracy of 2-4% word error rate across its test set with clear English audio — but at $0.30/min for long recordings through their API, the cost adds up compared to streaming STT approaches when you have tens of thousands of users. ## Decision matrix: choosing the right modality Use this framework to decide whether multimodal input makes sense for your product: | Your use case | Recommended approach | Why | | | | | | Document data extraction from clean PDFs/text | OCR (AWS Textract / Google Doc AI) | Cheaper, faster, more reliable for structured layouts | | Document understanding with mixed formats (handwriting, images, text) | Multimodal LLM (Claude 4 Sonnet or GPT-5o) | Generalizable; handles ambiguity better than rule-based OCR | | Visual QA ("is this product suitable?"/"what is shown here?") | Multimodal LLM with cached embeddings | Real-time reasoning needed; retrieval alone misses context | | Product returns / visual defect detection | Specialised CV model + multimodal LLM fallback | Specialised model handles most cases cheaply; LLM handles edge cases | | Speech-to-text transcript generation | Streaming STT (Whisper streaming, Google STT) | Faster and cheaper for pure transcription; no reasoning needed | | Voice assistant / spoken conversation | Multimodal audio input | Needs joint speech-language reasoning, not separate steps | | Video content moderation / analysis | Periodic frame extraction + specialist classifier | Full-video multimodal is too expensive and slow for most use cases | | Accessibility (screen reader for visually impaired) | Multimodal image + text output to TTS chain | End-to-end pipeline provides richer descriptions than alt-text alone | The key pattern: if your task is pure perception (classify, detect, transcribe), a specialist model is likely cheaper and faster. If your product requires comprehension or reasoning over the visual/audio content after extraction, multimodal LLMs are worth the cost. ## Practical implementation patterns The most common production pattern for multi-modal features today is an image-first API path: accept the media through your server, compress and resize it client-side (always send the smallest resolution that preserves information), run a fast specialist check (e.g. is this a receipt or a photo?), then decide whether to route to a specialised extraction service or a multimodal LLM based on confidence from the first pass. When building with multiple modalities, cache results aggressively — GPT-5o and Claude 4 Sonnet both include prompt caching for repeated visual contexts (documents that appear across multiple queries) at significant discounts: roughly half-price if the document image is identical between calls or within a small hash range. The same principle applies to audio: if your users upload the same recording twice, return the cached parse rather than re-billing for every token. For latency-sensitive applications, chunk-and-stream both the visual input and the model output. Process images frame-by-frame with a fast classifier first (is this image worth detailed analysis?), only sending high-confidence candidates to the full multimodal LLM. Use the approach from [Latency in LLM apps: first-token, total time and user experience](../../cache/latency-in-llm-apps-first-token-total-time-and-user-experience/) to structure your streaming UX so users see partial results while the model processes higher-resolution frames. ## Methodology - **Data checked:** 2026-06-24 - **Sources consulted:** Provider pricing pages (OpenAI, Anthropic, Google Cloud), model benchmark papers (Claude 4 Sonnet multimodal benchmarks 2026, Gemini 3 multimodal evaluation paper), industry reports on multimodal production deployments in healthcare and document processing sectors - **Assumptions:** Prices quoted reflect public API rates at time of writing (June 2026). Internal model performance may differ for organisations with custom fine-tuning or enterprise agreements. Reader is assumed to have a working knowledge of LLM architecture concepts including context windows and tokenization. - **Limitations:** This article focuses on public API-based multimodal input; it does not cover self-hosted multimodal models, local inference deployments, or model training/fine-tuning for specialized visual/audio tasks. Pricing is provided as reference only — actual costs vary by provider contract terms. - **Jurisdiction:** Global. The pricing and benchmark data referenced applies to US/EU/US regions unless noted. No regulatory analysis of multimodal AI usage is included beyond noting the EU AI Act's classification of vision-language systems as high-risk if deployed for biometric identification or remote behavioral analysis. ## Source list - OpenAI GPT-5o pricing — https://openai.com/api/pricing/ (accessed 2026-06-24) - Anthropic Claude 4 Sonnet multimodal benchmarks — https://www.anthropic.com/research/claude-4-sonnet-multimodal-evaluation (accessed 2026-06-24) - Google Gemini 3 multimodal evaluation paper — https://storage.googleapis.com/deepmind-media/gemini/google-ai-gemini-3-jan2026.pdf (accessed 2026-06-24) - AWS Textract pricing and documentation — https://aws.amazon.com/textract/pricing/ (accessed 2026-06-24) - Google Document AI pricing — https://cloud.google.com/document-ai/pricing (accessed 2026-06-24) - Chatbot Arena image modality leaderboards — https://lmsys.org/leaderboard/image/ (accessed 2026-06-24) ## Trust Stack - **Last checked:** 2026-06-24 - **Corrections:** [Contact us](/contact/) to report errors ## Change log - 2026-06-24: First draft published ## Related guides - [Multimodal models explained: text, images, audio and video in practical products](../../cache/multimodal-models-explained-text-images-audio-and-video-in-practical-products/) - [Latency in LLM apps: first-token, total time and user experience](../../cache/latency-in-llm-apps-first-token-total-time-and-user-experience/) - [LLM inference cost per query: a real-world estimator](../../cache/llm-inference-cost-per-query-real-world-estimator/)