Automated pentest: what it can prove, what it misses, and how to choose one

What is an automated pentest?

An automated pentest is a penetration test run by software: it attacks your live app from the outside, the way a stranger would, without a person driving each step. A good one reports a weakness only when the attack worked, and shows you the request and the response that prove it.

Proof is what separates it from a scanner. NIST defines penetration testing as security testing in which assessors "mimic real-world attacks to identify methods for circumventing the security features of an application, system, or network". A scanner compares what it sees with a list of known problems and reports what might be wrong. A pentest, automated or not, tries the attack and reports what happened.

The label covers a wide range today. Some automated pentest tools are scanners with a new name, others run fixed attack playbooks, and a newer group lets AI agents choose the next step (see AI penetration testing). One question sorts them all: does each finding come with proof?

How an automated pentest works, step by step

On a modern web app (a React or Next.js frontend talking to Supabase, Firebase or an API), a proof-based automated pentest runs in five stages. The penetration testing methodology guide maps the same stages to the published standards.

1. 1

   Recon: read the app like a browser

   It loads your public pages and reads every JavaScript file the browser downloads. That is where a stranger learns the most: the backend URL, the public keys, the table names in calls like supabase.from('orders'), the API routes, the framework and its version.

2. 2

   Mapping: list what can be attacked

   Each table, database function, storage bucket, collection and API route becomes a target, noted as reachable signed out or only with a session.

3. 3

   Attack: as a stranger, then as two users

   Each target is tried with no account, then with a fresh account, then with a second account against records the first one created. Many access control bugs only show when user B asks for user A's data.

4. 4

   Proof: keep the request that worked

   A finding is the exact request and the answer it got, with personal data redacted. A careful tool replays data findings once more before the report and drops any that the app now refuses.

5. 5

   Fix, then retest

   The report says what to change: the policy, the rule, the header, the version. After the fix, the same request runs again and must fail. Then it runs on every deploy, because the next migration can reopen the hole.

Automated pentest vs vulnerability scanner vs manual pentest

Vendors use the three words loosely. This is how they differ in practice. The pentest vs vulnerability scan guide goes deeper on the first two. To pick a scanner, the website vulnerability scanner guide shows what the free ones check and miss.

NIST notes that application vulnerability scanners "typically have high false positive rates", because they match patterns instead of trying the attack. For a small team, a long list of maybes costs more time than it saves.

What an automated pentest finds well on modern web apps

Apps built with Lovable, Bolt, Replit, v0 or a coding agent tend to break in the same places, and the most common break we see is access control: the backend answers someone it should not. The OWASP Top 10:2025 keeps broken access control at number one and says 100% of the applications tested had some form of it. These are flaws a machine can prove without guessing:

Check the obvious ones yourself first

You do not need a tool for a first pass. Four checks to run on your own app today:

1. 1

   Search your JavaScript for server keys

   Open your live site and the browser's developer tools, then search all loaded scripts (Ctrl+Shift+F in Chrome, Cmd+Option+F on a Mac) for sk_live_ and rk_live_ (Stripe secret and restricted keys) and sb_secret_ (a Supabase key that skips RLS). Stripe's docs say only publishable keys are safe outside your backend. A legacy Supabase key starts with eyJ: paste it in the Supabase key checker to see whether it is the public one or the service role.

2. 2

   Ask your site for files that should not exist

   Request /.env and /.git/config on your own domain with the commands below, and read the body that comes back.

3. 3

   List the tables with RLS off

   In the Supabase SQL editor, run select tablename from pg_tables where schemaname = 'public' and not rowsecurity;. Each table it returns is readable and writable by any role with a grant on it, which on projects that still grant anon means anyone holding your public key. The Supabase RLS guide has the policies to add. On Firebase, read Firebase security rules for the auth != null trap.

4. 4

   Open your own data from a second account

   Sign up twice. As account A, create something private and copy its URL or id. As account B, open it. If B sees it, you have an IDOR vulnerability.

# Your own domain only. Read the body, not just the status code.
curl -s https://your-app.example/.env | head -c 300
curl -s https://your-app.example/.git/config | head -c 300

What an automated pentest cannot prove

We build one, so here is where it stops.

• Business logic. A coupon that applies twice, a refund on an order that was never paid, a checkout step that can be skipped. Each request looks normal, and only someone who knows your business rules can tell the sequence is wrong. The OWASP Web Security Testing Guide says automating business logic abuse cases "is not possible and remains a manual art".
• What it cannot reach. A tool tests what it can sign into. Admin roles it has no account for, features behind an invite or a paid plan, and flows that need a phone or a card stay untested unless you give it access. A good report lists them under "not tested".
• Your people and your office. NIST names physical security testing and social engineering as non-technical methods a pentest can include, such as getting past locks and badge readers, or calling the help desk posing as a user to get a password reset. No web tool covers them.
• New research. A tool covers the classes of mistakes it has attacks for. A new flaw class, or a long chain of small bugs that only makes sense with context, needs a person.
• A report for an audit. The PCI Security Standards Council's guidance calls penetration testing "essentially a manual endeavor" and says judgment is needed for attack vectors that "typically cannot be identified through automated means". If an auditor or an enterprise customer asks for a pentest report, ask them what they accept before you buy anything.

An empty report means the attacks that ran did not work, on that date. It does not mean nothing else could. A report should say which attacks ran, which failed, and when.

Why last quarter's pentest is already out of date

NIST puts it plainly: an assessment "provides a snapshot of security at a given point in time". For a team that ships every day with a coding agent, a significant change can be one prompt. Two things move under a finished pentest:

• Your own code. A new feature arrives with a migration that creates a table. Supabase's documentation warns that a table in an exposed schema without RLS "is readable and writable by any role with a grant on it". Last month's pentest never saw that table.
• Your dependencies. CVE-2025-29927, published on 21 March 2025, let requests skip authorization checks done in Next.js middleware (CVSS 9.1, fixed in 12.3.5, 13.5.9, 14.2.25 and 15.2.3). CVE-2025-55182, a remote code execution flaw in React Server Components 19.0.0, 19.1.0, 19.1.1 and 19.2.0 that needs no login, was published on 3 December 2025 with a CVSS of 10.0, and CISA added it to its catalog of exploited vulnerabilities on 5 December. An affected app that passed a pentest on 1 December faced an actively exploited flaw four days later, without a line of its own code changing.

Continuous penetration testing answers both: the attacks run again on every deploy, and again when the tool adds an attack for a new CVE. It does not replace a deep manual test. It covers the months between two of them.

How to choose an automated pentest tool

Ask for a sample report first. Our pentest report sample shows what a credible finding looks like. Then check these points:

• Proof for every finding: the request, the response and the account it ran as. A severity score with no request behind it is a maybe.
• Two-account testing: access control tried signed out, signed in, and as a second user against the first user's data.
• Safe on a live app: reads by default; any write rolled back or made on a test record the tool creates and then deletes; no denial-of-service traffic; risky probes only when you opt in.
• Ownership checks: the tool refuses to attack an address until you prove it is yours. If it attacks any URL you paste, it will attack yours for anyone who asks.
• CVEs proven by behaviour: a version number in a header is a hint. Proof is the vulnerable behaviour, shown without a destructive payload.
• A "not tested" list: what it could not reach, and why.
• Fixes you can apply: the policy, the rule or the config line, in a form your coding agent can read.
• Retests included: after the fix, on every deploy, and when the attack library grows.
• Care with your data: counts and redacted samples in the report, never a dump of your users.

Price comes after these. A cheap tool that reports maybes costs you the hours to check each one. The penetration testing cost guide explains what drives the price of manual and automated tests.

Automated, manual, or both?

For most teams shipping a web app on Supabase, Firebase or their own API, the order is: close the obvious leaks before launch, run an automated pentest on every deploy, then buy a manual test when someone asks for one.

• Automated alone is a fair start before launch, when nobody has asked you for a pentest report yet.
• Add a manual pentest when a customer, an investor or an auditor asks for one, when money moves through credits or payouts, or when your roles and admin tools go beyond what a tool can model.
• Keep the automated test running afterwards: it tells you whether the fixes stayed fixed.

The guide to pentests for startups walks through that sequence stage by stage, and web application penetration testing covers what a full test of a modern stack includes.

How Keybrum runs an automated pentest

Keybrum is the automated pentest we build for apps made with Lovable, Bolt, Replit, v0, Cursor, Claude Code and Base44. Its library holds more than 100 attacks: Supabase and Firebase access rules, leaked secrets in the bundle, IDOR across two accounts, unprotected AI endpoints, unsigned Stripe webhooks, exposed .env files, security headers, and Next.js and React CVEs.

It only attacks an app after its owner proves ownership, and it proves a CVE by its behaviour, never by guessing from a version number. Your coding agent (Cursor, Claude Code) reads each finding, with its proof and its fix, through Keybrum's MCP server. A one-time report costs €49. Plans, from €29 a month excluding VAT, attack every deploy, keep retests open and add a public trust page that shows what was tested and when.