THE SECURITY COST OF VIBE CODING: A STUDY

We built four apps just
to break them.

We gave four AI coding tools the same prompt. The apps shipped, so did 137 vulnerabilities across 5 different ways of getting the same 9 things wrong.

Download the Report
Get the scorecard
Findings surfaced by Astra's autonomous pentesting agents.

DIY STRESS TEST

Patch the app

3 flaws, 30 seconds, zero shortcuts. Spoiler alert: AI coding tools had way longer,
and still missed 137 of these.

HOW TO PLAY

You have 30 seconds - the countdown starts when you click Start

Find the hidden flaw in each card.

Can you find all 3, before the timer hits zero?

Start Game

THE METHODOLOGY

Same build, same pressure, every time

Emergent, Replit, Lovable, and Base44 all received an identical prompt, no advance notice, and
the same bar for what counted as a real finding with Astra's autonomous pentesting platform.

Zero advance notice to ensure unbiased & authentic testing.

No platform knew this was coming, and no vendor relationship affected access, testing, or results.

2 autonomous pentest bots that don't talk to each other.

Structured Pentest mapped the app and built attack chains, while Bug Bounty Hunter Bot went in cold, simulating an unbriefed attacker.

A validator agent tried to break every finding again.

Nothing moved forward until it was independently reproduced.

Then a human security engineer checked it all anyway.

Every finding was reviewed a second time before it counted, and anything unconfirmed was cut before analysis.

Want this run against your own
app instead of a demo build?

Meet Astra autonomous pentest

Talk to us

A black and blue logo with headphones on it.

THE WEAK POINTS

Four builds, one failure mode

Four separate codebases, four separate frameworks, and the same nine weak points gave out anyway.

See what gave out

The four pressure points

Prompt specificity

Vague asks, like “secure login,” instead of an exact threshold.

Finding volume vs. severity

High counts didn’t mean higher risk. Severity did.

Access control enforcement

The most severe failure, when it appeared, almost every time.

Fix complexity

Config-level for the universal patterns, data-layer for the rest.

See exactly where all four
builds gave out, and how to
reinforce yours.

Download the Report

Log in 8 times with the wrong password.

Attempt 6 should return a 429, not a 200.

Check any response's headers in DevTools.

No Content-Security-Policy means this is already a real finding.

Reset a password with a fake email, then a real one.

The response, wording, and timing should be identical either way.

See what gave out

Your next AI-generated app
hasn't been through this yet

Get the full report for every fracture point, every fix, and the prompt
built to hold up before you ship.

Get the PDF

Does this rank the four platforms against each other?

No, this report evaluates what four applications built from the same prompt looked like under test, not which platform is more secure. Section 2 of the report states this explicitly.

Did any of the platforms know this was coming?

No platform received advance notice, and no vendor relationship affected access, testing, or results. One of the four platforms tested is an Astra customer, and that's disclosed in full in the report.

Are these real vulnerabilities, or theoretical ones?

Real vulnerabilities. A finding was only counted if it was actually exploited, with evidence of impact, and independently reconfirmed by a second pass. Nothing unconfirmed appears in any number in this report.

Is this a one-time snapshot, or an ongoing audit?

One prompt, one run per platform, tested in a single quarter. Platforms update constantly, so this reflects a point in time, not a permanent state. The report is explicit about this limitation.

What's actually in the download?

The full severity breakdown, all nine vulnerability patterns with the exact fix for each, the complete copy-paste security prompt, and platform-specific checklists you can run in under 20 minutes.

Do I need a security background to use any of this?

No. The 10-minute checklist and the prompt in Section 9 are both written to be used directly, no security team required.
Click here to update your cookies settings