White box pentest: what it finds in an AI-built app, and what it cannot prove

What a white box pentest is

A white box pentest is a penetration test where the tester gets the inside view first: source code, database migrations, security rules, configuration and usually test accounts. NIST defines white box testing as assuming explicit and substantial knowledge of the internal structure and implementation of what is tested. The tester reads the code to decide what to attack, then attacks the running app and proves what worked.

Black, grey and white box describe how much the tester knows on day one:

• Black box. The tester gets the app's address and nothing else, like a stranger. It shows what anyone can reach, but much of the paid time goes into discovery.
• Grey box (or gray box). The tester gets test accounts for each role and sometimes API docs, what NIST calls some knowledge of the internals. For an app with user accounts, it is the sensible default.
• White box (also crystal box or clear box). The tester also gets the code, schema and rules. It finds flaws that are hard to reach from outside, at the price of more specialist time.

For scale, Blaze prices a manual test of a simple SaaS app at $4,999 to $8,999 for 3 to 7 testing days, and a complex one at $20,000 to $30,000 or more for two to three weeks or longer. The penetration testing cost guide explains day rates, and the web app pentest guide covers a full test.

What white box review looks like on an app built with AI

Apps generated with Lovable, Bolt, v0, Cursor or Claude Code keep most of their access control in a few files. On Supabase or Firebase the browser often talks to the database directly, so the policies or rules are the access control. A reviewer reads these first:

A coding agent can do a first pass. Cloudflare's open-source security-audit skill runs it in six phases and gives every candidate to a fresh verifier that tries to disprove it, ending each as confirmed, needs_validation or rejected. Keybrum's code audit includes it; the AI pentest guide explains why that independent check matters.

Three bugs: what the code shows, what the live app proves

Three patterns from generated code: what a reviewer spots in the repository, and what only a request to the running app settles.

1. A Supabase policy using (true)

-- supabase/migrations/20260310120000_profiles.sql
create table public.profiles (
  id uuid primary key references auth.users on delete cascade,
  full_name text,
  email text,
  stripe_customer_id text
);

alter table public.profiles enable row level security;

create policy "Profiles are viewable"
  on public.profiles for select
  using (true);

What white box review spots. The policy lets every row through. With no to clause, a Postgres policy applies to every role, anon included. Supabase says a using (true) read policy for anon gives every unauthenticated visitor every row the role can reach through its grants, so use it only for public data. Emails and Stripe customer ids are not.

What a black box test proves. Whether a stranger gets those rows today, which the migration cannot show: whether anon holds a select grant in production, and whether someone has changed the policy in the dashboard since. Supabase warns that schema changes made on the remote database through the SQL editor or Table Editor bypass the migration history. The proof is one row read with your public key while signed out, against your own project.

The fix limits the read to the owner, in a new migration:

drop policy "Profiles are viewable" on public.profiles;

create policy "Users read their own profile"
  on public.profiles for select
  to authenticated
  using ((select auth.uid()) = id);

Put any field that must be public in a separate table, or in a view created with security_invoker = true so it obeys RLS. The Supabase RLS guide covers the common cases.

2. A Firebase rule that only checks request.auth != null

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /users/{userId} {
      allow read, write: if request.auth != null;
    }
  }
}

What white box review spots. Any signed-in user can read and overwrite every document in users. Firebase's guide to insecure rules names this pattern: if a rule includes auth != null, confirm that you want any logged-in user to have access. The fix: allow read, write: if request.auth != null && request.auth.uid == userId;.

What a black box test proves. Whether a stranger can become "any logged-in user", which depends on the sign-in methods enabled in the Firebase console, not on the rules file. And whether production runs firestore.rules at all: a CLI deploy overwrites rules edited in the console, so a console-only fix disappears at the next deploy. The proof is account B reading account A's document. The Firebase security rules guide has owner-only patterns.

3. A server route that trusts a user id from the body

// app/api/account/route.ts
import { db } from "@/lib/db";

export async function POST(req: Request) {
  const { userId, displayName, plan } = await req.json();
  await db.user.update({
    where: { id: userId },
    data: { displayName, plan },
  });
  return Response.json({ ok: true });
}

What white box review spots. The route reads no session, updates whichever user id the request names, and accepts a plan field the interface never sends: an IDOR and a free upgrade in one handler. The Next.js data security guide says the same of Server Actions: a page-level check does not cover them, a direct POST reaches them, and each must check that the caller may act on that specific record.

What a black box test proves. Whether the route is reachable in production and what it changes. A firewall rule or a missing deploy can block it; a second copy under another path can make it worse. The proof is account B changing account A's record, then reading it back.

// app/api/account/route.ts
import { auth } from "@/lib/auth";
import { db } from "@/lib/db";

export async function POST(req: Request) {
  const session = await auth();
  if (!session?.user) {
    return Response.json({ error: "Unauthorized" }, { status: 401 });
  }
  const { displayName } = await req.json();
  await db.user.update({
    where: { id: session.user.id }, // from the session, never from the body
    data: { displayName },          // plan changes go through billing only
  });
  return Response.json({ ok: true });
}

What a black box test proves that code review cannot

OWASP's Web Security Testing Guide calls source code analysis the technique of choice for technical testing, strong on access control and business logic, and free of black box guesswork. Among its weaknesses: it cannot easily detect runtime errors, and the source code actually deployed might differ from the one being analyzed. On an app built with AI, that gap has three parts:

• Reachability. A dangerous function matters only if a request can reach it. Code review flags dead routes with the same confidence as live ones.
• Real exposure. The code says a policy is broad. The live app says which columns come back, to whom, and whether the table holds ten test rows or every customer.
• Configuration outside the repository. Policies added in the SQL editor, rules edited in the Firebase console, a bucket switched to public, open sign-ups, hosting variables, a deployed build that is not the commit you reviewed. None of it shows in a diff.

To check whether production still matches your migrations, run this in the SQL editor of your own Supabase project:

-- Tables in the public schema, and whether RLS is on
select tablename, rowsecurity
from pg_tables
where schemaname = 'public';

-- Every live policy, with its roles and conditions
select tablename, policyname, roles, cmd, qual, with_check
from pg_policies
where schemaname = 'public'
order by tablename;

The reverse holds too. OWASP notes that dynamic testing struggles to cover a whole code base, especially behind nested conditions. A black box test that finds nothing has only shown that its own attacks failed.

Grey box: test accounts and partial knowledge

Grey box sits between the two: the address plus accounts, sometimes API docs or a list of roles, but no code. Blaze calls it usually the best value for web, API and mobile tests, because testers spend their days on real risk instead of discovery.

The accounts are what matter. One user reading another user's data only shows when user B asks for what belongs to user A. Give the tester two regular users, two admins if you have an admin area, and two teams if your app is multi-tenant, each with some data.

Grey box misses what the interface never exercises: an old API route still deployed, a database function no screen calls, a field the server accepts but the form never sends. A white box pass closes that gap.

Which one to choose for your situation

Combining a code audit with live proof

The two fail in opposite directions. Code review alone gives false positives: using (true) on a table of public blog posts is intended, and a dangerous route may be unreachable. Black box alone leaves blind spots: a failed attack may have lacked the right account, the right role or the route's name. Together:

1. 1

   Read the code for candidates

   Migrations, rules, routes, environment handling, the lockfile. Each suspicion becomes a candidate with a file and a line.

2. 2

   Attack the live app for each candidate

   Signed out, as user A, then as user B asking for A's data. A candidate becomes a finding only with the request and response that proved it.

3. 3

   Keep the unproven ones as leads

   The route may be unreachable, or the test lacked an account. Note why and retry; Cloudflare's skill has a needs_validation verdict for this.

4. 4

   Trace live findings back to the code

   When an attack finds something the code does not explain, such as a dashboard policy, write the fix into a migration or rules file.

5. 5

   Retest after the fix, and on every deploy

   Replay the same request; the finding closes only when it fails. An agent building a new feature can rewrite the route you just fixed.

We built Keybrum on that split. Your coding agent runs the code audit, Cloudflare's skill included, handed over through Keybrum's MCP server. Keybrum's attack library then tries the findings on the live app, signed out and as a second signed-in user. Your agent reads each fix through MCP, and Keybrum attacks again after the fix and, on plans, on every deploy. It only attacks apps whose owner proved ownership. The automated penetration testing guide covers what still needs a human, and the methodology guide the phases of a test.

What to hand a tester for each type

Whichever box you buy, judge the report, ours included, by one question: can you replay each finding yourself?