AI coding agents: what to measure before trusting them
A practical measurement framework for evaluating AI coding agents on real engineering work: review burden, test pass rate, diff quality, security issues and rollback cost.
Run
Workflows you can actually test
The Run lane is for doing the work: choosing models, setting up evals, pricing a feature, testing retrieval, running agents, and explaining trade-offs without turning the meeting into acronym soup.
24published guides
25guide briefs
48Cache briefs to link from
21diff/context briefs
Published now
Live guides in this prototype
AI incident response: what to do when a model gives harmful or wrong advice
A practical guide to adapting incident response for prompts, outputs, evals, rollbacks and customer-facing AI failures.
AI output monitoring: what to log, sample and review
A practical guide to balancing observability with sensitive-data retention risk in production LLM systems.
Citation quality in AI answers: source-grounded does not mean source-faithful
A practical guide to testing whether cited sources actually support the generated claim, not just whether the answer looks grounded.
PII handling for LLM apps: minimisation before redaction
A practical guide to reducing personal-data exposure in AI features by minimising what you send before you try to redact it.
Prompt injection explained for business users
A practical guide to direct and indirect prompt injection, what can go wrong in retrieval-heavy apps, and what controls actually reduce risk.
Data leakage in LLM apps: logs, prompts, files and vendor retention
A practical guide to where LLM data leaks happen, what to minimise before sending data, and what retention settings to check before launch.
Tool-use safety: stopping agents from taking dangerous actions
A practical guide to approval gates, least privilege, dry runs and audit logs for AI agents with tools.
Jailbreaks vs product safety: what operators can realistically control
A practical guide to separating model-level safety from app-level permissions, tool boundaries and operational controls.
Hallucination testing: how to build a small regression set
A practical guide to creating a small test set for unsupported claims, regression checking and safer LLM updates.
Chat history is not memory: how LLM apps remember users
A plain-English guide to the difference between chat history, profile memory, stored app data and training data, with privacy checks for product teams.
AI agents vs workflows: a plain-English difference for teams
A decision guide for operators who need to know when deterministic automation is enough and when real agent behaviour is worth the operational cost.
Golden datasets for LLM products: how small regression sets prevent regressions
A practical explanation of why a small, stable test set is often more useful than a huge benchmark when you need confidence before shipping.
Eval CI for AI apps: testing prompts before every release
A release-process guide for teams that want automated evaluation to run before changes reach users.
The model release treadmill: how to avoid rebuilding every month
A practical guide to decoupling your product from provider churn so every model update does not become a rewrite.
Refusals and over-refusals: testing whether safety blocks useful work
A plain-English guide to distinguishing sensible safety boundaries from over-refusal that breaks legitimate use cases.
Eval gaming: when models optimise for the test rather than the task
A guide to spotting benchmark overfitting and test-specific behaviour before it turns into product disappointment.
Red teaming an LLM feature: a practical first-week checklist
A practical first-week checklist for finding failure modes in a new LLM feature — with concrete test items, sample prompts, and decision guides.
MCP explained: tools, resources, prompts and the current hype gap
A clear guide to what Model Context Protocol is, what it is not, and why the marketing sometimes runs ahead of the wiring.
Vector databases: when semantic search is enough and when it is not
A practical guide to deciding whether you need a vector database, a search index or something much simpler.
Chunking documents for RAG: size, overlap and metadata choices
A practical guide to breaking source material into retrievable pieces without wrecking meaning or search quality.
Rerankers explained: the quiet quality layer in RAG systems
How rerankers improve retrieval precision — with a worked example, model/provider names, latency numbers, and a decision framework for when the extra step pays for itself.
Inference vs training vs fine-tuning: three terms operators confuse
A plain-English guide to the three phases of model work, what each one changes, and what the difference means for budget, data and risk.
Latency in LLM apps: first token, total time and user experience
A plain-English guide to why AI features feel slow, what to measure, and how to separate queueing, model time, tool calls and network delay.
Editorial rule
Every guide needs a stopping point
- Define the job, input, output, assumptions, and failure modes.
- Show the checks that prove the workflow worked, or say what remains unproved.
- Link back to Cache for concepts instead of re-explaining everything on every page.
- Keep tool/provider claims dated. LLM tooling ages like milk in a warm server rack.
Briefed pipeline
Run queued for drafting, edit and promotion
Chat history is not memory: how LLM apps remember users
Understand AI memory features and their privacy implications.
AI agents vs workflows: a plain-English difference for teams
Decide whether a task needs an “agent” or deterministic automation.
How LLM benchmarks work, and what they miss
Understand benchmark claims in model launches.
Golden datasets for LLM products: how small regression sets prevent regressions
Build a practical evaluation set for an AI feature.
Eval CI for AI apps: testing prompts before every release
Put AI regression tests into a software pipeline.
The model release treadmill: how to avoid rebuilding every month
Reduce disruption from frequent model releases/deprecations.
Hallucination testing: how to build a small regression set
Reduce factual errors in LLM outputs.
Refusals and over-refusals: testing whether safety blocks useful work
Diagnose unwanted model refusals.
Eval gaming: when models optimise for the test rather than the task
Understand why high benchmark scores may not translate.
Red teaming an LLM feature: a practical first-week checklist
Test an AI feature before launch.
Fine-tuning vs prompting vs RAG: decision checklist
Choose an adaptation strategy for an AI product.
Function calling and tool use: where agents actually fail
Build reliable tool-using AI workflows.
MCP explained: tools, resources, prompts and the current hype gap
Understand Model Context Protocol and when it helps.
Vector databases: when semantic search is enough and when it is not
Decide whether to add a vector database.
Chunking documents for RAG: size, overlap and metadata choices
Improve retrieval quality in document AI.
Rerankers explained: the quiet quality layer in RAG systems
Understand reranking after vector search.
Building a minimum viable RAG system without overengineering
Plan a simple document-QA prototype.
AI coding agents: what to measure before trusting them
Evaluate coding agents for real engineering work.
Schema-first AI extraction: making LLMs useful for messy documents
Extract structured data from unstructured text.
LLM observability basics: traces, prompts, evals and feedback loops
Monitor an LLM app in production.
Fallback design: what happens when the AI call fails?
Design resilient AI features.
Human-in-the-loop AI: approval queues that do not become bottlenecks
Add human review to AI workflows.
Prompt versioning: treating prompts like production code
Manage changes to prompts in teams.
Building an internal AI policy bot: safe pattern or risky shortcut?
Use AI over internal policies and procedures.
The evidence-led AI website manifesto: how theLLMs will review claims
Understand the site’s editorial trust model.
Search by idea
Find articles by describing what you need
Try "how much do tokens cost?", "run a model on my own hardware", or "stop prompt injection attacks". Search runs in your browser against our article index.
Describe what you are trying to understand.