Here’s what makes this genuinely difficult. AI generated code compiles and looks clean, which can lead reviewers to trust it more than they should. Human written code with a subtle SQL injection is sometimes suspicious partly because the surrounding code may look uncertain or improvised. AI output often doesn’t read that way. It can read like a senior developer wrote it quickly. That surface polish can work against you.
Prompt-level controls are an underused lever. Feeding explicit security requirements into your prompts, asking specifically for parameterized queries, requesting input validation, can measurably reduce the rate of vulnerable output. It doesn’t eliminate the risk, but it shifts the model’s output distribution in a useful direction.
Why AI Models Often Produce Vulnerable Code
The control that matters most, though, is the one teams most often skip: mandatory human review with a security lens, treating AI-generated code as equal to any other code in the review queue. A common failure in practice is letting AI output bypass the review path that human-written code goes through. The code looks authoritative. The review is abbreviated. The vulnerability ships. Feeding explicit security requirements into prompts and running automated scans are both valuable, but neither substitutes for a reviewer who understands the system the code is entering and is specifically looking for what the model can’t see.
There’s a second failure mode that gets less attention and deserves more. AI models sometimes invent plausible-sounding package names that don’t exist in any actual registry. This has been called “slopsquatting,” and the attack surface it creates is real and growing. Attackers can register those invented package names on npm or PyPI, two large public package registries for JavaScript and Python respectively, so that when a developer installs an AI-suggested dependency, they may pull malware instead of a legitimate library.
The Confidence Problem Makes Detection Harder
Beyond the training data and the hallucination problem, there’s a third failure mode that is more subtle and arguably more dangerous in production environments. The model doesn’t know your authentication model. It doesn’t know how your data is classified, or where your trust boundaries sit, or which endpoints are exposed to the public internet. A hardcoded secret or a server-side request forgery (SSRF) vulnerability, one that allows an attacker to make the server issue requests to internal systems, hits very differently on an exposed cloud endpoint than it would in a sandboxed test environment.
Start with SAST tools integrated directly into your CI pipeline. Semgrep, Snyk Code, and CodeQL are all capable of catching many of the injection and hardcoded-credentials patterns that AI models reproduce most often. Running these tools only after merge is often too late, by that point, the pattern is already in your history.
The Specific Vulnerability Classes That Appear Most Often
The remediation approach layers several controls, and none of them alone is sufficient.
AI coding assistants have changed how fast software gets written. They have not changed what makes software secure.
Package Hallucination and the Slopsquatting Attack Surface
Enable secret scanning through tools like GitHub secret scanning or gitleaks. Hardcoded keys are a class of failure that AI models can insert with particular frequency, and secret scanning is one of the more reliable ways to catch them at the repository level before they propagate.
You run a security scan on code that looks perfectly fine. It compiles. The logic reads cleanly. A colleague reviewed it and moved on. Then your SAST tool — a static application security testing scanner that analyzes source code for security flaws without executing it which flags a SQL injection pattern that went through three rounds of review without anyone blinking. That scenario is becoming routine, and the reason is structural, not accidental.
The Situational Awareness Gap
Research has found a meaningful share of AI suggested packages were hallucinated. The exact proportion varies by language ecosystem and prompt context, but the pattern is consistent enough that any AI-suggested import statement warrants explicit verification before it gets near a build pipeline.
A study by Neil Perry and colleagues at Stanford found that developers using an AI assistant tended to write less secure code while simultaneously believing their output was more secure than it actually was. That confidence gap, lower actual security, higher perceived security, is a condition that lets vulnerabilities survive review. NYU’s “Asleep at the Keyboard” study put a number on the other side of this: roughly 40% of GitHub Copilot completions in the security-relevant scenarios they tested were vulnerable. Readability and safety are largely unrelated in AI output, and treating them as correlated is an expensive assumption.
How to Catch These Vulnerabilities Before They Ship
Not all AI security failures look the same. The vulnerability classes that tend to surface most often in AI-generated code include: SQL and command injection (SQL injection is catalogued as CWE-89), hardcoded secrets and API keys (CWE-798), path traversal, weak or missing input validation, and deprecated cryptographic implementations. Each of these has a plausible line back to the training data: they appeared in real-world repositories, they appeared frequently, and the model learned them as common patterns.
Injection vulnerabilities are particularly common because the “quick” way to construct a database query, string concatenation, appears more often in public repositories than parameterized queries do. The model doesn’t know that the fast approach is the dangerous one. It knows that the fast approach is the common one.
The result is a system that predicts plausible code rather than secure code. It optimizes for what commonly appears in its training data, not for what passes a threat model. That distinction matters more than it first sounds. A model can produce syntactically correct, logically coherent, yet dangerous output and it has no built-in mechanism to know the difference. SQL injection, hardcoded API keys, weak cryptographic primitives like MD5 or ECB mode: these patterns appeared frequently in training, so the model can reproduce them with the same confidence it uses for everything else.
The training data problem is a root issue, and it doesn’t have an easy fix. AI code models are trained on public repositories, and those repositories contain many insecure and outdated patterns. The model learns from everything: the careful security-reviewed commit and the three year old Stack Overflow snippet with hardcoded credentials. Both appear in the training corpus. Both reinforce what “normal” code looks like.
This is where AI-generated code can fail differently from a junior developer. A junior developer writing the same flawed code at least knows which system they’re working in. They can ask a question. They understand context from the sprint board, the architecture diagram, the conversations that happened before they touched the ticket. The AI model typically has none of that. It has the prompt, and the prompt rarely contains a threat model.
