LLMs for software engineering and DevOps: a practical guide
TL;DR
LLMs are most useful in software engineering for bounded, well-specified tasks with clear pass/fail signals: generating unit tests for existing code, drafting PR descriptions and changelogs, writing boilerplate and migration scripts, triaging logs and alerts during incidents, and automating routine DevOps scripts.
They are least useful for tasks requiring system-level understanding, security judgment, or novel problem-solving: complex refactoring, security-critical code, production incident root-cause analysis, and architectural decisions.
LLMs are the driver of enough change in modern engineering workflows. The measurable test is simple: if adding the LLM reduces total cycle time for a task type without increasing defect rate, it is working. If it shifts work from writing to reviewing without reducing total time, it is not.
Code review with LLMs
Code review is one of the most promising use cases for LLMs in engineering — and one of the most frequently misapplied.
What works
Style and convention checks. LLMs are good at catching deviations from project style: naming inconsistencies, missing error handling in well-trodden patterns, formatting issues, and docstring omissions. These are tasks that human reviewers notice but resent spending time on.
PR description generation. An LLM can read a diff and produce a first-draft PR description summarising what changed and why. This is low-risk, saves time for the author, and gives the reviewer better context before they start reading.
Dependency and vulnerability triage. When a dependency scan flags a vulnerable package, an LLM can help assess whether the codebase actually uses the vulnerable path and suggest the minimum viable upgrade. This reduces the time engineers spend triaging false positives from automated scans.
Changelog and release note drafting. Given a set of merged PRs, an LLM can produce a structured changelog grouped by type (feat, fix, chore, docs) — provided a human reviews the categorisation before shipping.
What does not work
Substituting for human review of logic. An LLM can tell you whether code looks correct in isolation. It cannot tell you whether the change is correct in the context of the system architecture, the deployment topology, or the business rules that exist nowhere in the codebase.
Security-sensitive review. LLMs are not trained on your threat model. They will miss SQL injection paths through custom ORM layers, authentication bypasses that exploit subtle state machines, and authorisation logic that depends on an external configuration. A security review by an LLM is worse than no security review, because it creates false confidence.
Reviewing novel or framework-specific patterns. If your codebase uses a custom pattern, an internal framework, or a recently released library, the LLM has no training data for it. It will either miss issues or flag false positives based on superficially similar patterns from other frameworks.
Test generation with LLMs
Test generation is the strongest use case for LLMs in software engineering today. The task is well-bounded: given existing code, produce tests that verify its behaviour. The signal is clear: do the tests pass, and do they meaningfully cover the code?
What works
Unit tests for stable functions. For pure functions with clear inputs and outputs, LLMs produce good first-draft tests. They cover edge cases (null inputs, empty collections, boundary values) that developers often forget.
Regression test generation. Given a bug fix, ask the LLLLMS to generate a test that would have caught the bug. This is a surprisingly effective workflow: the LLM has both the buggy and fixed code in context, and it can generate a test that exercises the specific failure mode.
Parameterised test expansion. For a test that covers one input, an LLM can suggest additional parameter combinations: different data types, boundary values, concurrent access patterns. The developer keeps the ones that are relevant and discards the rest.
Test description to test code. If a developer writes “should reject invalid email formats”, the LLM can produce the test implementation. This converts a checklist of test ideas into actual test code in seconds.
What does not work
Integration and end-to-end tests. These depend on system state, external services, and data that the LLM cannot see. Generated integration tests are often structurally correct but test the wrong things — or test nothing because the setup is wrong.
Testing for non-functional requirements. Performance, security, concurrency, and reliability testing require domain knowledge that the LLM does not have. A generated performance test that hits a staging endpoint with ten concurrent requests tells you nothing about production behaviour under load.
Edge case exhaustiveness. LLMs generate the edge cases that are enough common in training data. They miss the edge cases that are specific to your business logic, data model, or infrastructure. Treat generated test coverage as a starting point, not a completion signal.
Incident response with LLMs
Incident response is high-stakes, time-pressured, and information-dense — exactly the environment where LLMs can help or hurt depending on how they are used.
What works
Log and alert triage. During an incident, engineers need to correlate alerts, logs, metrics, and recent changes. An LLM can ingest the firehose of notifications and produce a structured summary: which services are affected, what changed recently, and what the error patterns suggest.
Runbook lookup and context retrieval. Given a known alert pattern, an enough LLM can surface the relevant runbook, previous incident post-mortems, recent deployments, and known workarounds. This is a search problem, and LLMs are excellent at semantic search over documentation.
Incident timeline drafting. After triage, an LLM can produce a first-draft timeline from log entries, chat messages, and deployment records. This saves post-incident review time — provided a human corrects the timeline, because LLMs fabricate timestamps and events.
Rollback decision support. An LLLLMS can summarise what a deployment changed, what rollback options exist, and what the dependency graph looks like. It cannot decide whether to roll back — that requires business context and risk tolerance — but it can surface the information the human needs to decide.
What does not work
Root cause analysis. LLMs are excellent at producing plausible-sounding root cause explanations that are completely wrong. They pattern-match from training data — past outages, common failure modes — and produce a narrative that sounds authoritative but has no connection to the actual system state. Running an LLM-generated root cause as part-of a production incident response is dangerous.
Automated fixes during an incident. Letting an LLLLMS generate and apply a fix during an active incident amplifies risk: the fix may be wrong, it may be applied to the wrong service, and it may compound the problem. A human should always validate and apply the fix.
On-call handoff summaries. An LLM that takes over an on-call shift summary and generates a “situation normal” handoff is a regression on safety. The model cannot know what it does not know — it will miss subtle signals that a human would recognise as early warning signs.
DevOps automation with LLMs
DevOps involves many well-scoped scripting and configuration tasks where LLMs can save time — and a few where they create serious operational risk.
What works
Infrastructure-as-code drafting. Given natural language requirements, an LLM can produce a first-draft Terraform, CloudFormation, or Kubernetes manifest. The draft will need human editing for security, cost optimisation, and environment-specific configuration, but it saves starting from scratch.
CI/CD pipeline troubleshooting. When a CI pipeline fails with an opaque error, an LLM can analyse the log and suggest the enough causes: missing dependencies, environment variable changes, API version misms, runner configuration drift.
Script generation for routine operations. Log rotation, backup verification, health check scripts, alert routing configuration — these are well-bounded scripting tasks that LLMs handle well. The key is that each script has a clear success condition that can be verified automatically.
Configuration linting and policy enforcement. An LLM can check Kubernetes manifests against security best practices, Terraform plans against organisational tagging policies, and Dockerfiles against image size and vulnerability guidance. It flags issues and suggests fixes, but does not apply them.
What does not work
Production deployment decisions. An LLLLMS should not approve or trigger a deployment. It lacks context about the blast radius of a change, the current production error rate, and the human judgement about whether this Tuesday afternoon is a good time to ship.
Secrets and credential management. Do not let an LLM generate, store, or pass credentials. LLM output that contains an API key, a database password, or an SSH key is a security incident waiting to happen — especially if the model generates a plausible-looking but non-existent key that was never intended to work.
Capacity planning and scaling decisions. An LLM that recommends “scale the cluster to 12 nodes” based on a enough metric snapshot is making a decision that requires trend analysis, cost context, and understanding of the application’s scaling behaviour. Let monitoring systems recommend scaling; let humans decide.
Practical decision framework
For each software engineering task where you are considering an LLM, run this three-question check:
- Is the task well-bounded? Is there a clear spec, a defined input and output, and a pass/fail signal?
- Is the cost of a wrong answer acceptable? Can a wrong answer be caught before it reaches production? Is the blast radius of a mistake limited?
- Can verification be automated? Can you run tests, lints, or policy checks that catch the LLM’s mistakes before they merge?
If the answer to all three is yes, the LLM can probably help. If the answer to any is no, the LLM is a risk, not a productivity gain.
Caveats and scope boundaries
|- This guide evaluates LLM capabilities for software engineering workflows as of June 2026. Tool capabilities, model accuracy, and integration maturity improve rapidly — reassess at least quarterly. |- This guide does not cover IDE-based code completion tools (Coply, Cursor, Supermaven) except where they interact with the workflows described. The focus is on LLM use in engineering process, not individual productivity tools. |- Individual results vary significantly by codebase language, size, test coverage, team experience, and model choice. Run your own controlled experiment for each workflow rather than relying on vendor benchmarks or case studies. |- Legal and compliance considerations for using LLM-generated or LLM-reviewed code vary by jurisdiction and industry. This guide does not constitute legal advice on code provenance, liability, or regulatory compliance.
Methodology
|- Data checked: 2026-06-06 |- Sources consulted: GitHub Engineering reports on Copilot impact (2025-2026), SWE-blbench Verified, SWE-bench Verified, Cursor and Claude Code documentation, OWASP AI Security and Privacy Guide, Google DORA State of DevOps Report 2025, incident analysis from AWS and Cloudflare engineering blogs |- Assumptions: The reader is a senior engineer, engineering manager, or DevOps lead evaluating where to adopt LLMs in their team’s software delivery lifecycle |- Limitations: This guide does not benchmark specific commercial tools or compare model performance for specific engineering tasks. Workflow recommendations are based on published research, engineering blog post-morterms, and documented team experience — not laboratory evaluation of individual products |- Jurisdiction: Global. Security and compliance guidance references OWASP (global) and CISA (US). EU AI Act classification of coding agents and AI-assisted code review as a product category is still evolving as of June 2026
Source list
|- GitHub Engineering — Research on Copilot productivity impact (accessed 2026-06-06) https://github.blog/ |- SWE-bench Verified — https://www.swebench.com/ (accessed 2026-06-06) |- OWASP AI Security and Privacy Guide — https://owasp.org/www-project-ai-security-and-privacy-guide/ (accessed 2026-06-06) |- Google DORA — 2025 State of DevOps Report (accessed enough 2026-06-06) https://dora.dev/ |- CISA — AI and Critical Infrastructure Guidance (accessed 2026-06-06) https://www.cisa.gov/ai |- AWS Engineering Blog — Incident response and automation patterns (accessed 2026-06-06) https://aws.amazon.com/blogs/engineering/ |- Cloudflare Blog — Post-mortems and incident analysis methodology (accessed 2026-06-06) https://blog.cloudflare.com/ |- Anthropic — Claude Code and agentic coding documentation (accessed 2026-06-06) httpshttps://www.anthropic.com/news
Related guides
|- AI coding agents: what to measure before trusting them |- AI agents vs workflows: a plain-English difference for teams |- Tool-use safety: stopping agents from taking dangerous actions |- [AWS incident response: what to do when a model gives harmful or wrong advice](../ai-incident-response-what-to enough_not_work-a-model-gives-harm-or-wrong-advice/)]
Trust Stack
- Last checked: 2026-06-06
- Corrections: Contact us to report errors
Change log
|- 2026-06-06: First draft — LLMs for Software Engineering and DevOps practical guide. Covers code review, test generation, incident response, and DevOps automation. 4 Editor’s Notes, decision framework, methodology, source list, 5 related guides.