hero_image:- “/images/hero/ai-incident-response-what-to-do-when-a-model-gives-harmful-or-wrong-advice.png” layout:- ../../layouts/GuideLayout.astro title:- “AI- incident- response:- what- to- do- when- a- model- gives- harmful- or- wrong- advice” description:- “A- practical- guide- to- adapting- incident- response- for- prompts,- outputs,- evals,- rollbacks- and- customer-facing- AI- failures.” writtenBy:- “gemma4:26b” reviewedBy:- “deepseek-r1:32b” lastChecked:- “2026-05-28” scope:- “Global.- Provider- and- standards- sources- checked- as- of- 2026-05-28.”

#- AI- incident- response:- what- to- do- when- a- model- gives- harmful- or- wrong- advice

When- an- AI- feature- gives- bad- advice,- the- incident- is- not- just- the- answer.- It- is- the- path- that- let- the- answer- reach- a- user,- and- the- path- that- failed- to- stop- it- in- time.

Treat- harmful- outputs- like- product- incidents:- contain,- preserve- evidence,- roll- back- or- disable- the- risky- path,- notify- the- right- people,- and- only- then- ask- how- the- model- got- there.

The- right- playbook- depends- on- severity.- A- wrong- suggestion- is- not- the- same- as- a- safety-critical- mistake,- but- both- need- a- clear- path- for- containment- and- review.

##- TL;DR

Treat- harmful- AI- outputs- as- critical- product- incidents- that- require- immediate- containment- and- investigation.- When- a- model- generates- harmful- or- incorrect- advice,- you- must- first- preserve- all- session- evidence—prompts,- context,- and- logs—before- rolling- back- or- disabling- the- risky- path.- Only- after- securing- the- system- should- you- investigate- whether- the- failure- occurred- in- the- prompt,- retrieval- augmentation,- or- tool- authorization.

When- an- AI- feature- gives- bad- advice,- the- incident- is- not- just- the- answer.- It- is- the- path- that- let- the- answer- reach- a- user,- and- the- path- that- failed- to- stop- it- in- time.

##- What- this- means ##- TL;DR

Treat- harmful- AI- outputs- as- critical- product- incidents- that- require- immediate- containment- and- investigation.- When- a- model- generates- harmful- or- incorrect- advice,- you- must- first- preserve- all- session- evidence—prompts,- context,- and- logs—before- rolling- back- or- disabling- the- risky- path.- Only- after- securing- the- system- should- you- investigate- whether- the- failure- occurred- in- the- prompt,- retrieval- augmentation,- or- tool- authorization.

When- an- AI- feature- gives- bad- advice,- the- incident- is- not- just- the- answer.- It- is- the- path- that- let- the- answer- reach- a- user,- and- the- path- that- failed- to- stop- it- in- time.

##- What- this- means

AI- incident- response- follows- the- same- structure- as- any- product- incident- —- contain,- triage,- fix,- learn- —- but- with- an- extra- step:- you- need- to- separate- the- model- output- from- the- product- path- that- delivered- it.- The- output- was- wrong.- But- was- the- failure- in- the- model,- the- prompt,- the- retrieved- context,- the- tool- permission,- the- approval- gate,- or- the- output- validation?- The- answer- determines- whether- you- fix- the- system- prompt,- add- a- guardrail,- update- the- RAG- source,- or- disable- the- tool.

The- incident- severity- determines- the- response- speed- and- escalation- path.- A- P1- incident- (harmful- output- reaching- a- wide- audience,- safety-critical- error,- data- exposure)- needs- immediate- containment- —- disable- the- feature- or- route.- A- P3- incident- (wrong- advice- that- is- annoying- but- not- harmful,- formatting- error- in- a- low-traffic- surface)- can- be- queued- for- the- next- sprint.- Most- teams- skip- the- severity- classification- and- treat- every- model- mistake- as- urgent,- which- means- nothing- is- actually- urgent.

##- Where- teams- misuse- it

|— No- severity- ladder- for- AI- incidents.- Without- a- severity- classification,- a- model- generating- a- slightly- wrong- product- description- gets- the- same- response- as- a- model- generating- harmful- medical- advice.- Both- get- escalated,- both- get- a- post-mortem,- and- the- team- fatigues- on- the- low-severity- ones- while- the- high-severity- ones- get- lost- in- the- noise. |— Fixing- the- prompt- but- leaving- the- tool- path- open.- A- model- generated- an- incorrect- refund- amount.- The- team- updates- the- system- prompt- to- say- “always- verify- refund- amounts.”- But- the- tool- that- executes- refunds- has- no- approval- gate- and- no- upper- limit- check.- The- prompt- fix- reduces- the- surface- but- does- not- fix- the- product- —- the- same- failure- can- recur- with- a- differently- phrased- user- request. |— Skipping- evidence- preservation- before- rolling- back.- When- a- harmful- output- is- detected,- the- team- rolls- back- the- model- version- or- disables- the- feature- immediately- —- the- right- instinct.- But- they- do- not- preserve- the- prompt,- context,- tool- calls,- and- output- that- caused- the- incident.- When- the- post-mortem- starts,- nobody- has- the- data- to- understand- what- happened. |— Treating- every- incident- as- a- model- problem.- A- model- gave- a- wrong- answer.- The- investigation- stops- at- “the- model- hallucinated.”- But- the- real- failure- was- in- retrieval- (the- wrong- document- was- returned),- tool- permissions- (the- model- had- write- access- it- should- not- have- had),- or- output- validation- (the- check- that- should- have- caught- the- wrong- fact- was- missing).- Model- behaviour- is- the- last- thing- to- blame,- not- the- first.