Large language models are now a core part of the software development lifecycle. The 2025 Stack Overflow Developer Survey found that 82% of developers used OpenAI’s GPT models in their work last year, and Google has reported that AI now writes over 25% of new code committed at the company.
All of that rests on one assumption. The model understands what you asked, and its answer is accurate. When it doesn’t, the model keeps producing confident-looking code drawn from patterns it half-remembers. This is content scarcity at work. In AI terms, it’s called hallucination, and it shows up when the input falls out-of-distribution relative to the training data.
The byproduct of content scarcity is LLMs generating code with bugs. At the volume LLMs now write code, that is the failure mode worth taking seriously.
How LLMs Generate Code?
An LLM generates code by predicting tokens based on statistical patterns learned during training, conditioned on the context supplied at inference time. For widely used technologies like mainstream Python libraries, JavaScript frameworks, standard SQL, and common auth patterns, the model has enormous statistical signal. Those patterns appear repeatedly across GitHub, documentation, tutorials, and forums it was trained on.
Since Vaswani et al.’s Attention Is All You Need (2017), transformer-based LLMs generate output token by token using self-attention over the input context and learned weights. Self-attention lets the model weigh which prior tokens matter most when predicting the next one, but it doesn’t verify correctness against any external ground truth. It’s fundamentally a probability distribution over the next token, sampled from what the model has seen.
When the training distribution is dense for a given pattern, that probability sharpens around the right answer. When representation is thin, the distribution flattens, and the model still has to sample something. So it generates code by analogy, and that’s where it ships broken code that reads as clean. This is a simplified view (real-world behavior is also shaped by RLHF and fine-tuning), but the underlying dynamic holds.
Why Content Scarcity Bites Security Hardest
Security is where scarcity hits hardest, because security-critical code is systematically underrepresented in training data relative to the volume of naive or insecure code the model has seen. Public corpora are dominated by tutorials, quick-fix answers, and toy examples. Mature, production-grade defensive patterns and hardened auth flows do exist in public. But they are outnumbered many times over by insecure or half-correct examples, and a large share of battle-tested code lives in private repos that never enter public datasets at all.
The result: an LLM’s model of “how to write auth” is disproportionately shaped by naive tutorials, instead of battle-tested code. It generates OAuth flows that skip PKCE, input validation that misses second-order injection, and JWT handling that never validates the algorithm claim. Newer models fine-tuned on secure coding practices are closing this gap, but the baseline problem remains.
The bugs LLMs produce range from trivial to critical, but they are overwhelmingly semantic. A 2024 empirical study analyzed 333 real bugs across ChatGPT, CodeGen, and Copilot and found that semantic issues dominated, while pure syntax errors made up under 10%.
Common bugs that get shipped are:
- Wrong parameter order or type
- Incorrect error handling
- Lack of business logic enforcement on the backend
- Server-side API keys hardcoded into frontend code
- Pulling in vulnerable or unmainted libraries
- Version drift (deprecated auth flows, outdated crypto)
- Fabricated dependencies (the USENIX Security 2025 paper “We Have a Package for You!“ tested 16 popular LLMs across 576,000 code samples and found 19.7% of suggested packages were hallucinated, with 43% of hallucinated names reappearing across every re-run)
The API-misuse angle shows up separately too. In Can LLM Replace Stack Overflow? (AAAI 2024), researchers tested LLMs against 1,208 real Stack Overflow questions across 18 Java APIs. 62% of GPT-4’s generated code contained API misuse. That figure reflects the model version tested at the time; newer releases may perform differently, but the underlying pattern of API-contract failures on less-represented libraries remains consistent across follow-up studies.
The pattern repeats everywhere. Methodologies vary, and no single number tells the whole story, but the direction is unambiguous: bad prompts and content scarcity account for a meaningful share of the bugs LLMs ship.
Then What’s the Fix?
We believe the answer is autonomous pentesting and AI-powered DAST sitting inside the CI/CD pipeline. Good prompting, senior engineers reviewing output, and knowing what you’re doing all help too, but those are not scalable.
Autonomous verification is the safety net that scales with the amount of code LLMs are now writing, and it works precisely because content scarcity in LLMs is now a known structural problem rather than an edge case.
You might reasonably ask, if content scarcity is what causes LLMs to write buggy code, how can another AI tool be the fix? The answer is in what the tool is grounded in and what it’s designed to do.
Take our Autonomous Pentesting platform. It’s trained on the insights from 10+ million findings and 5k+ pentests conducted over years. That insight is exactly the kind of production-grade offensive security context general LLMs are starved of. The agent reasons over that grounded knowledge, uses dedicated validator agents to confirm exploitability before flagging anything as critical, and operates inside guardrails designed to prove impact without causing outages.
That is entirely different from an LLM writing by producing something novel from thin priors. Moreover, verification is a fundamentally more constrained problem than generation, and it’s where domain-tuned, well-grounded agents outperform their generative cousins.
Final Thoughts
The takeaway isn’t to ban or stop using LLMs. Content scarcity is structural, and the productivity gains from AI-assisted coding are real. The goal is to make sure being wrong isn’t enough to become a production bug.
