Pentest vs vulnerability scan: what each one proves, and when you need it

Pentest vs vulnerability scan: the short answer

A vulnerability scan asks what looks wrong. A pentest asks what someone can actually do. The scanner compares what it can see from outside (version numbers, response headers, files it can fetch) with a database of known problems, and lists every match. Nothing is exploited, so nothing is proven. A penetration test is an authorized attack: a tester tries to use the weaknesses, combines them, and reports the ones that worked, with the request as proof.

NIST's testing guide, SP 800-115, defines penetration testing as security testing in which evaluators "mimic real-world attacks in an attempt to identify ways to circumvent the security features of an application, system, or network". It is blunt about scanners: "application vulnerability scanners typically have high false positive rates", and they also suffer from high false negative rates. In plain words, they report problems you do not have and miss some you do.

• Vulnerability scan: an automated check, done in minutes, that matches versions, headers and configuration against known weaknesses. ZAP, a free and open source project, is a common example. To pick one, see the guide to website vulnerability scanners.
• Vulnerability assessment: the scan plus a review. Someone removes the false positives, ranks what is left and plans the fixes. NIST defines it as a systematic examination to "identify security deficiencies". It still attacks nothing.
• Penetration test (pentest): an attack on a live system, within an agreed scope, by people, software or both. A scan is often its first step. What you get is what an attacker could reach, and how.

Scan, manual pentest and automated pentest compared

The PCI Security Standards Council, which writes the card industry's security rules, sums up the split in its penetration testing guidance. A scan's purpose is to "identify, rank, and report vulnerabilities"; a pentest's is to "identify ways to exploit vulnerabilities to circumvent or defeat the security features of system components". A scan takes "several seconds to several minutes per scanned host"; a pentest "may last days or weeks". The last column is the option most founders now meet.

The same app, scanned and then attacked

Take a typical app built with an AI coding tool: a Next.js 15.1.0 frontend with the App Router, hosted on Vercel, Supabase for the database and sign-in, Stripe for payments. The app is made up; each critical finding below is a pattern Keybrum has an attack for. First, what a scanner reports:

Content Security Policy (CSP) Header Not Set
Missing Anti-clickjacking Header
Vulnerable JS Library
next 15.1.0 matches CVE-2025-55182 (critical, CVSS 10.0)
next 15.1.0 matches CVE-2025-29927 (critical, CVSS 9.1)

Every line is accurate, and two are rated critical. Only one of those two is an emergency for this app. Here is what attacking the same app shows, finding by finding:

The scanner caught one emergency, raised a critical that cannot be reached here, and missed the three critical flaws in the app's own code and rules. An open table, an IDOR (insecure direct object reference: one user reaching another's data by changing an id) and an unsigned webhook all answer with a normal 200 OK. Nothing in the response says the data belongs to someone else. To know that, a tester needs two accounts and an idea of who should see what. OWASP's testing guide says it plainly: "Automated tools find it hard to understand context."

A proven finding for the open table looks like this:

Finding   orders: readable by anyone, signed out
Attack    supabase.table-read
Request   GET https://<project>.supabase.co/rest/v1/orders?select=*&limit=10
          apikey: <the public key from the JavaScript bundle>
Response  200 OK, 10 rows returned, more counted
Means     RLS is off on public.orders, or a policy lets anon read it
Fix       enable RLS, add a policy that limits reads to the owner, retest

Vulnerability assessment vs penetration testing

Many vendors sell a vulnerability assessment: a scan with a person in the loop. PCI's guidance describes even its scans that way, as "a variety of automated tools combined with manual verification of identified issues", with a report of "potential risks posed by known vulnerabilities" ranked by CVSS score (the 0 to 10 severity scale used in CVE records). A pentest report goes further: it describes "specific methods how and to what extent" each weakness may be exploited.

Both are worth paying for, as long as you know which one you bought. Signs that a "pentest" is a scan with a new cover:

• The report is a list of CVEs and missing headers sorted by CVSS score, with no requests or responses.
• Nobody asked you for two test accounts, so nothing checked whether one user can reach another's data.
• No finding says what an attacker could read, change or delete.
• The offer includes no retest of your fixes.

The penetration test report sample shows what a finding with proof should contain, from the request to the retest.

When you need a scan, a pentest, or both

If you are not sure which row you are in, start with access control. In the scan behind CVE-2025-48757, 170 of 1,645 apps built with Lovable, about one in ten, had missing or weak RLS that let unauthenticated visitors read or write data (Lovable disputes the CVE). No header or version scan would have flagged them.

For the enterprise row: in an article republished by the Cloud Security Alliance, a CPA calls a pentest "the most common requirement we hear as the last component to close a deal with an enterprise customer". If a questionnaire is waiting in your inbox, the security questionnaire guide covers what you can answer honestly.

What SOC 2, ISO 27001 and PCI DSS ask for

SOC 2 points of focus are examples, not requirements. The same CPA puts it this way: "there are no explicit requirements for penetration testing (or any controls) within a SOC 2 report".

Where proof-based automation fits

Between the scanner and the consultant sits software that attacks like a pentester, at the price and speed of a scan. One rule separates it from a scanner: no attack that worked, no finding.

Keybrum works this way. It runs more than 100 attacks against the live app: Supabase tables, RPC and storage, Firebase rules, secrets in the JavaScript bundle, unprotected AI endpoints, unsigned Stripe webhooks, exposed .env files, and known Next.js and React CVEs. It tests IDOR across two accounts, and it never sends remote code execution or denial-of-service payloads.

It only attacks apps whose owner proved ownership. Each fix goes to your coding agent (Cursor, Claude Code) through Keybrum's MCP server, and Keybrum attacks again after the fix and on every deploy.

What it does not replace: a person who reads your business rules and asks what happens if a coupon is applied twice, or a refund is requested after a plan change. Nor a report signed by a human tester, which an auditor or enterprise buyer may require. The automated pentest guide goes through its limits, and the web app pentest guide covers what a manual engagement adds.

How to check your own app this week

1. 1

   Run a passive scan of your own site

   ZAP's baseline scan crawls your site for a minute and reads the responses without sending attacks: docker run -t ghcr.io/zaproxy/zaproxy:stable zap-baseline.py -t https://your-app.com. For the header findings, the security headers checker gives the exact fix for Next.js, Vercel and Netlify.

2. 2

   Triage each finding with one question

   Can you show the request that abuses it? If not, it is a lead. For a version match, read the advisory's conditions: hosting, configuration, and whether your code reaches the flawed path.

3. 3

   Test what scanners miss, with two accounts

   Sign up twice on your own app. As account A, note the ids in the pages and in the API calls of the browser's network tab. As account B, request the same ids. If B sees A's data, you have an IDOR: the IDOR guide shows the fix.

4. 4

   Check the database rules

   On Supabase, run the query from the RLS checker to list tables with row level security off. On Firebase, paste your rules into the Firebase rules checker.

5. 5

   Retest after every fix

   A fix is done when the attack that proved the finding fails again on the deployed app.