Continuous penetration testing: how to attack every release of your app

Continuous penetration testing attacks a live app from the outside each time it changes, usually on every deploy, plus a scheduled run. A yearly pentest describes the app as it was on the day of the test. Apps built with Lovable, Bolt or Cursor change every week, and one prompt can rewrite a database policy or an API route. The loop that holds is test, fix, retest: a finding closes when the same attack fails again.

The team behind Keybrum's attacksUpdated 9 min read

In short

  • Continuous pentesting repeats outsider attacks on the deployed app whenever it changes, so the findings describe the code that runs today.
  • A vulnerability scan lists possible weaknesses. A pentest tries them and shows what an unauthorized user can actually reach.
  • A finding closes only when a retest of the same attack fails.
  • Start the tests on each deploy, add a weekly run for drift, and track the time to fix and the findings that come back.
On this page

What continuous penetration testing means

A traditional penetration test has a start date, an end date and a report. Someone probes the app for a few days and writes up what they could reach. The report starts to age with the next merged pull request.

Continuous penetration testing removes the fixed window. The test runs again on a schedule, weekly or nightly, and when something happens: a new deployment, a new preview, a dependency update. The tester can be a person, an automated agent or both. What makes it continuous is the link between testing and change.

Three words carry the definition:

  • Outsider. The test starts where an attacker starts: an address, no login, no source code. It asks what a stranger can reach.
  • Live. It runs against the deployed app. Wrong environment variables, open storage buckets and unprotected endpoints often exist only in the running system.
  • Repeated. The same attacks run again after each meaningful change, so a regression is caught the week it appears.

How it differs from scans, yearly pentests and vulnerability management

The terms get mixed up on vendor pages. They answer different questions.

ApproachQuestion it answersUsual rhythmWhat you get
Vulnerability scanWhich known weaknesses might exist?Daily to quarterlyA list of matches, often long and not ranked by real risk
Yearly pentestCan an attacker exploit this app today?Once or twice a yearA detailed report on one snapshot
Continuous pentestCan an attacker exploit it after this change?On every release, plus a scheduleA running record of exposures that are proven and reachable now
Vulnerability managementWhich findings do we fix first, and did we fix them?An ongoing processTracking, priorities and verification

For a small team the line between scanning and testing matters most. A scanner says a library version has a known flaw. A pentest says whether anyone can use that flaw on your app, and which data they get. The second answer decides what to fix this afternoon. What an automated pentest proves and pentest vs vulnerability scan go further on that line.

A red team exercise is a different thing again. It plays a determined adversary against the whole organization, people and processes included. A pentest looks for exploitable flaws in a defined target. Red team vs pentest helps you choose.

Why a yearly test stops working when you ship often

A point-in-time test assumes the app holds still. In a normal week of AI-assisted development, any of these can happen:

  • A prompt regenerates the data layer, and a row level security policy is dropped or loosened.
  • A new API route skips the authentication middleware the other routes use.
  • A secret key is pasted into client code to make something work, and ships to every visitor's browser.
  • A preview deployment goes up with real data and a public address.
  • A framework dependency is bumped, and a new advisory lands for it a month later. A CVE watch for Next.js, React and Supabase follows those.

None of these is rare. Each one appears after the tester has left, so a yearly report cannot see it.

The SANS Institute describes the same shift: penetration testing is moving toward attack surface management and continuous testing, where a change in the environment starts verification and exploitation attempts right away. Exploitability changes over time as new techniques and flaws appear, so a conclusion reached once goes stale.

How continuous testing works in practice

Tools differ. A program that holds over time has five parts.

  1. 1

    A defined target

    List what is exposed: production domains, staging and preview addresses, API endpoints, subdomains. A forgotten subdomain is a classic way in. The attack surface management guide builds that list without an enterprise toolchain, and the subdomain takeover guide covers the most common forgotten asset.

  2. 2

    Triggers

    Start the tests from events: a new production deployment, a new preview, a merge to the main branch as a CI check, and a scheduled run to catch drift when nothing was deployed.

  3. 3

    Proof

    A finding helps when you can see it is real: the request that returned data it should not have, the key readable without login. Proof ends the debate about false positives and shows what is really exposed, which sets the priority.

  4. 4

    Priority

    Rank findings by reach and consequence before a generic severity score. A medium flaw that exposes your users' emails comes before a high one in a component nobody can reach. Ask three questions of each finding: can an outsider reach it, can it be exploited, and what data or action does it give them?

  5. 5

    Retest

    This is the step most teams skip. A fix goes in, it looks right, and the same issue comes back with the next regeneration of that file. A retest runs the original attack again and confirms it fails. Without it, fixed is a guess.

OWASP's DevSecOps Verification Standard takes the same position. It says penetration testing should be scheduled around the delivery process: significant releases and new features are assessed before or shortly after they ship, and at the highest maturity level features are assessed as they are built. It also asks for every finding in one issue tracker, followed to resolution, with the fix retested.

What it finds in apps built with AI coding tools

Apps made with Lovable, Bolt, Replit and similar tools share a few weak points, because the platforms produce similar architectures: a front end, a hosted database, serverless functions and a pile of third-party keys.

  • Exposed keys. API keys, service tokens and database credentials in client code or in public configuration.
  • Unprotected data. Tables or storage buckets that return records to anyone who asks, because access rules are missing or too broad.
  • Broken authorization. One user reads or edits another user's records by changing an id in a request. See IDOR and broken access control.
  • Weak authentication paths. Endpoints that skip the login check, or API keys that can be bypassed. The Better Auth API key bypass shows how a library flaw becomes a live exposure.
  • Framework flaws. Known issues in the Next.js or Vite version you deploy, such as the Vite dev server file read. The Next.js CVE checker reads your version in a few seconds.
  • API weaknesses. Missing rate limits, responses that say too much and routes with no authentication. See what API penetration testing covers.

Authorization flaws deserve the most attention. They live in your business logic, ordinary feature work introduces them, and the next regeneration can bring them back. Repeated testing is built for that kind of bug.

A cadence for a small team

You do not need a security department for this. A founder or a small product team can work like this:

  1. 1

    A baseline before launch

    Run a full outsider test on the production app and fix what it proves. Before launch, pentesting for startups says what to test first.

  2. 2

    Retest the fixes

    Close a ticket on a passing retest.

  3. 3

    Hook testing to deployments

    Each new deployment gets the same attacks, without anyone having to remember.

  4. 4

    Add a scheduled run

    Weekly is a sensible start. It catches changes that did not go through your pipeline, such as a dashboard setting changed by hand.

  5. 5

    Review once a month

    Look at what was found, how long each fix took and what came back. A finding that keeps coming back points to a prompt, a template or a habit worth changing.

A deep manual engagement by a consultant fits with this. The continuous layer covers the weeks between two engagements.

Agencies and teams with many apps

With several apps, the number of targets grows faster than the attention you have. A few habits keep it manageable:

  • One inventory per client. Every domain, preview address and API of each app, updated at handoff.
  • The same baseline everywhere. Run the same outsider test on every app before delivery, so no client asks why one build was tested and another was not.
  • Evidence for the client. A dated report and a retest that shows the finding closed make a stronger handoff than a verbal assurance.
  • One tracker. OWASP's standard asks for findings in one issue tracker, followed to resolution. With ten apps, that is what stops a known issue from staying open for months.
  • Rules for production apps built with AI. If clients expect enterprise standards, the enterprise vibe coding rules give you a checklist.

Measuring whether it works

Four numbers show whether the testing changes anything:

  • Time to fix. Days from a proven finding to a fix that passes the retest. It should go down over time.
  • Findings that come back. A high count means the fixes are patches, or that regenerated code keeps undoing them.
  • Findings per release. A count that rises after one kind of change shows where your process leaks.
  • Exposed data. The findings that give an outsider access to user data or secrets. On production the goal is zero.

The aim is a trend you can show a client, a customer or an investor.

Choosing a continuous testing setup

Whatever you pick, hold it to these six points:

  • It attacks the live deployed app from the outside, on top of any code review.
  • It proves what can be exploited, beyond a list of possibilities.
  • It can start on a deployment, beyond a calendar.
  • It gives fixes your team can apply quickly, ideally written for the coding agent you use.
  • It retests each fix and says whether it holds.
  • It produces a report someone who is not a developer can read.

Where Keybrum fits

Keybrum attacks your deployed app as a stranger, then as a new user, then as a second user who asks for the first one's data. A finding is a request that worked, shown with the response and with personal data masked. Each finding comes with a fix written for your coding agent, which can also read it through the MCP server. After your fix the same attack runs again, and deploy hooks, a CI check and scheduled runs attack each new version.

Prices are public: a one-off Report on one app is €49, Maker is €29 a month or €290 a year, and Team is €99 a month or €990 a year. The details are on the pricing page, and pentest as a service compares the offers that put human testers behind a platform.

Questions

Is continuous penetration testing the same as continuous vulnerability scanning?

No. Scanning looks for known weaknesses and reports possible matches. Continuous penetration testing attempts to exploit your live app from the outside, again and again, and shows what an attacker could actually reach. Many teams use both. Only the pentest tells you whether a weakness matters.

How often should I test?

Tie testing to change first: every production deployment, and every preview that carries real data or real integrations. Then add a scheduled run, weekly for most small teams, to catch drift. If you ship rarely, a test after each release plus a monthly run is enough to start.

Do I still need a yearly pentest if I test continuously?

Some buyers, auditors and enterprise customers ask for a dated pentest report. Continuous testing gives you a series of dated reports and retests that you can share. Check what the customer or the standard asks for, and keep the reports as evidence.

Can continuous testing break my app?

Any test against production carries some risk. Keybrum's attacks are designed to run on a live app, and how attacks stay safe says what each one is allowed to do. With any tool you do not know yet, start on a preview that mirrors production, with test accounts.

Is it legal to pentest an app?

Testing your own app, or one you have written permission to test, is standard practice. Testing someone else's system without authorization is not. An agency should get a written scope from the client before it tests a deployment it does not own.

Sources

  1. 1.Continuous Penetration Testing and the Rise of the Offensive SOC, SANS Institute
  2. 2.DevSecOps Verification Standard, TEST-004 Penetration Testing, OWASP

Your app is public. So are its mistakes.

Check what any attacker can already see in it, free, in ten seconds. The full attack, with proof and fixes: €49 once, or €29 a month on a plan.

or continue with Google or GitHub