AI pentest: what an AI agent can prove, and how to test your own AI features

What people mean by an AI pentest

AI pentest is used for two different jobs. The first is a penetration test run by AI: a language model agent reads your app, decides what to try, sends the requests and writes the report. The second is a penetration test of the AI inside your app: the chat route, the summarize button, the assistant that can read your database. Vendors rank for both under the same words, so check which one you are buying before you compare offers.

Autonomous pentesting is the far end of the first meaning: the agent plans and runs each step with no human in between. AI-assisted pentesting keeps a person, or fixed code, in charge of what gets sent and of what counts as a finding.

Where AI agents genuinely help in a pentest

Much of a web pentest is reading: minified JavaScript, route names, API schemas, error messages, source code. Language models read fast, and they do not get tired on the four hundredth route.

• Reading the bundle. The JavaScript your app ships names its backend, its API routes, often its tables, and sometimes a key that should not be there. An agent turns that into a list of things to test.
• Mapping the API. From fetch calls, OpenAPI files and tRPC procedure names, it lists the endpoints and guesses which ones should need a session.
• Reading the code. With the repository, an agent can follow a request from the route to the database query and spot a missing ownership check that a test from outside can only infer.
• Writing the fix. A finding becomes a change in your code, in your framework and your style. A report alone never does that step for you.

Research agrees on both the strengths and the limit. The PentestGPT paper found that language models handle sub-tasks well, such as using testing tools, interpreting their output and proposing next actions, but struggle to keep an integrated understanding of the whole test. In a 2025 study on a university network of about 8,000 hosts, the authors' own agent, ARTEMIS, placed second overall against ten professional pentesters and six other agents, ahead of nine of the ten humans. Some versions of it cost about $18 an hour, against $60 for the professionals.

Where AI pentesting hallucinates

The same study names the cost of that speed: AI agents had higher false-positive rates than the humans and struggled with tasks that needed a graphical interface. ARTEMIS had an 82% valid submission rate, so nearly one report in five was not valid. Reading the report does not tell you which one: a hallucinated finding is written with the same confidence as a real one.

Open source maintainers see this at scale. Daniel Stenberg, curl's lead developer, closed curl's bug bounty on January 31, 2026, citing an explosion of AI slop reports among other trends. In earlier years, more than 15% of submissions turned out to be confirmed vulnerabilities. Starting in 2025, the rate fell below 5%.

Here is how a plausible finding can be false, and what settles it:

• A 200 read as a leak. The status says OK, but the body is the sign-in page, an empty list [], or data you publish on purpose. Only the response body settles it.
• A version read as a CVE. The package version is in the affected range, but the vulnerable feature is not used, or your host blocks the request. Only the behaviour settles it.
• A public key read as a secret. The Supabase anon key and a Firebase web API key are meant to be in the browser. The risk depends on what they can reach.
• A chain that never ran. A confident paragraph on how one flaw leads to the next, with no request showing that any step worked.

Why every finding must carry proof

A finding is a claim about your live app. Proof turns it into a fact you can check yourself, without trusting the tool, the model or the vendor. People who build these tools say the same: Snyk writes that a model asked to certify its own findings has a structural conflict of interest, and Cloudflare's open-source security-audit skill follows the rule that the agent that checks a finding is never the agent that found it.

Before you act on an AI pentest report, from Keybrum or anyone else, ask for this:

• The exact request that worked: method, URL, headers and body, plus the response it got, with personal data redacted.
• A replay command you can run against your own app, and the answer you should get once the fix is live.
• An independent confirmation: the same request sent again on its own, or a second checker that tried to disprove the finding.
• For access control, two accounts: proof that account B received a record that belongs to account A.
• For a CVE, the behaviour of your app, not a version number matched against a list.
• What was not tested, and what the tool refuses to send: writes to real data, remote code execution, denial of service.
• A retest against the live app after you deploy the fix.

A report that cannot give you the first two items is a list of leads. Leads are useful, especially from a code review, but they are not findings yet. The pentest vs vulnerability scan guide draws the same line between a possible flaw and a proven one.

Testing the AI features in your app: what a stranger can do

Now the second meaning. When your app calls OpenAI or Anthropic, the call should go through a server route or an edge function that holds the key, so the key never reaches the browser. The weak point is then the route itself: if it answers anyone, a stranger can skip your interface and use it as a free model proxy, billed to you.

OWASP lists this as Unbounded Consumption (LLM10 in its 2025 Top 10 for LLM applications). It includes what OWASP calls Denial of Wallet: a high volume of operations that exploits the cost-per-use model of cloud-based AI services.

The cost is not theoretical. When Sysdig found attackers running language models on stolen cloud credentials in 2024, it estimated that the abuse could cost the victim over $46,000 a day if left undiscovered. An open AI route needs no stolen credential at all.

One trap is specific to Supabase Edge Functions. The key your frontend sends is public, and Supabase's documentation states that the verify_jwt check on Edge Functions alone does not authenticate a caller who sends only an API key. An AI function that relies on that platform check, with no check in its own code, answers anyone who has opened your site.

How to test your own AI endpoints

Run these against your own app, with your own test accounts. Each call asks the model for one word, so it costs about as much as one short message from a real user.

1. 1

   List every route that calls a model

   Search your repository for the provider SDKs and helpers: openai, @anthropic-ai/sdk, generateText, streamText. If you use Supabase, look in supabase/functions/ too. Note the URL of each route or function.

2. 2

   Call each one with no session

   Use the first command below, with no cookie and no Authorization header. A 401 or 403 is what you want. A 200 with a model's answer means anyone can use the route.

3. 3

   Call it signed in, 30 times in a row

   With a test account's session cookie, use the loop below. If all 30 calls succeed, there is no per-user limit at that rate. Pick the daily limit your users actually need and check that you get a 429 above it.

4. 4

   For Supabase, call each AI function with only the public key

   Send the request with your anon or publishable key and no user session. If the model answers, the function trusts the platform check alone.

5. 5

   Set a hard spend limit at the provider

   On OpenAI, open Project settings, then Limits, enter a monthly spend limit and turn on Enforce a hard limit: calls past it fail with a 429 and the code project_spend_limit_exceeded. OpenAI warns that recorded spend can slightly exceed the limit. On Anthropic, set your own monthly limit on the Billing page of the Claude Console. Anthropic also allows spend limits per workspace, but not on the default workspace.

6. 6

   Search the bundle for provider keys

   Paste your live JavaScript into the API key leak checker. An OpenAI key (sk-proj- for project keys) or an Anthropic key (sk-ant-) in the browser is worse than an open route: rotate it now, then follow what to do when an API key leaks.

# Your own app only. No cookie, no Authorization header.
curl -i -X POST "https://YOUR-APP.com/api/chat" \
  -H "content-type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with the single word OK."}]}'

# Signed in as your own test account: 30 calls, then count the status codes.
for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST "https://YOUR-APP.com/api/chat" \
    -H "content-type: application/json" \
    -H "cookie: YOUR_TEST_SESSION_COOKIE" \
    -d '{"messages":[{"role":"user","content":"Reply with OK."}]}'
done | sort | uniq -c

# A Supabase Edge Function, called with the public key only (your project only).
curl -i -X POST "https://YOUR_PROJECT_REF.supabase.co/functions/v1/YOUR_FUNCTION" \
  -H "apikey: YOUR_ANON_OR_PUBLISHABLE_KEY" \
  -H "Authorization: Bearer YOUR_ANON_KEY" \
  -H "content-type: application/json" \
  -d '{"prompt":"Reply with the single word OK."}'

The fix: a session, a per-user limit and a spend cap

Three changes close most of this, and they go before the model call, so a refused request costs nothing. Here is an open AI route in Next.js:

// app/api/chat/route.ts: answers anyone, with any messages
import OpenAI from "openai";
const openai = new OpenAI();
const MODEL = process.env.OPENAI_MODEL!;

export async function POST(req: Request) {
  const { messages } = await req.json();
  const completion = await openai.chat.completions.create({ model: MODEL, messages });
  return Response.json(completion);
}

And the same route with a session, a daily limit per user, a size limit, and messages built on the server:

// app/api/chat/route.ts
import OpenAI from "openai";
const openai = new OpenAI(); // reads OPENAI_API_KEY, on the server only
const MODEL = process.env.OPENAI_MODEL!;
const SYSTEM_PROMPT = "Summarize the user's text in three sentences.";

export async function POST(req: Request) {
  const session = await getSession(req); // your auth library
  if (!session) return Response.json({ error: "Sign in required" }, { status: 401 });

  // Count calls per user per day in your database or Redis.
  if (await overDailyLimit(session.userId, 50))
    return Response.json({ error: "Daily limit reached" }, { status: 429 });

  const { text } = await req.json();
  if (typeof text !== "string" || text.length > 4000)
    return Response.json({ error: "Message too long" }, { status: 413 });

  const completion = await openai.chat.completions.create({
    model: MODEL,
    max_completion_tokens: 500,
    messages: [
      { role: "system", content: SYSTEM_PROMPT }, // written here, never sent by the client
      { role: "user", content: text },
    ],
  });
  return Response.json({ text: completion.choices[0]?.message?.content ?? "" });
}

For a Supabase Edge Function, let the @supabase/server wrapper require a signed-in user instead of trusting the platform check. With auth: "user", it accepts only a valid user JWT on the Authorization header:

// supabase/functions/chat/index.ts
import { withSupabase } from "npm:@supabase/server@1";

export default {
  fetch: withSupabase({ auth: "user" }, async (req, ctx) => {
    const userId = ctx.userClaims?.id;
    const { text } = await req.json();
    // 1. Count today's calls for userId and return 429 above your limit.
    // 2. Call the model with the key from Deno.env.get("OPENAI_API_KEY").
    const answer = await askModel(text); // your call to the provider
    return Response.json({ text: answer });
  }),
};

Then keep the hard spend limit from step 5, so a bug in your own counter still has a ceiling. The cap stops the feature for every user once reached, so treat it as the last line behind your per-user limit. The unprotected AI endpoint guide goes deeper on limits and on spotting abuse early.

Prompt injection: limit what the model can reach

OWASP ranks prompt injection first in the same list (LLM01): input that changes the model's behaviour in ways you did not intend, typed by the user (direct) or hidden in a page, file or email the model reads (indirect). OWASP also writes that it is unclear whether fool-proof prevention exists. So the most useful test is a review of what a successful injection could reach, and you can do it in your own code:

• The model's tools reach only what the signed-in user could reach anyway, through the same checks your API applies. OWASP's Excessive Agency entry calls this running extensions in the user's context.
• Any write the model proposes, such as sending an email, changing a record or charging a card, waits for a click from the user.
• The system prompt holds no key, connection string or internal rule you would mind seeing in public. OWASP says it should not be treated as a secret, nor used as a security control.
• Authorization lives in your code. A rule like "never show other users' orders" written in a prompt can be argued away by the right input.
• Pages, files and emails the model reads are passed as clearly marked untrusted content, separate from your instructions.

How Keybrum splits the work: attacks prove, your agent fixes

We built Keybrum on that split. The attacks are code, not a model improvising: more than 100 of them, each written in advance with the requests it may send and the response that counts as proof. The AI part is your own coding agent, such as Cursor or Claude Code, doing what agents are good at: reading code and writing the fix. Code review plus live proof is what testers call grey box; the white box, black box and grey box guide explains why the pair finds more than either alone.

For AI features, two attacks do what this page describes. One sends a one-line prompt, asking for the single word OK, with no session, to API routes named like AI endpoints, and counts a finding only when a model answers. The other does the same for Supabase Edge Functions that look like AI. Functions that do not look like AI are never called, since they could send emails or charge cards, and API routes that look like actions are skipped.

Before the report, each data finding whose proof is a read is requested a second time, on its own: if the app now refuses or answers empty, the finding is withdrawn. Leads from your agent's code audit are linked to the live attack that proves them, and the rest are tried in the next run. CVEs are proven by behaviour, never by a version number, and no attack sends remote code execution or denial-of-service payloads. Keybrum only attacks apps whose owner proved ownership.

Where to go from here

If you are comparing tools, read automated penetration testing next. If an AI tool wrote most of your code, the AI-generated code security audit covers the checks for the code itself, and the web app pentest guide covers the app around your AI features. Whatever you pick, judge it by one test: can you replay each finding yourself?