In short
- NIST describes penetration testing as security testing in which evaluators mimic real-world attacks to find ways around the security features of an application, system or network.
- A red team, in the US government glossary NIST publishes, is a group authorized to emulate a potential adversary's attack against an enterprise's security posture, to show the impact and what works for the defenders.
- CIS Control 18 puts it in one line: red team exercises exploit vulnerabilities like penetration tests do, but they simulate the tactics of a specific adversary to see how the environment would withstand it.
- None of the five safeguards of CIS Control 18 belongs to Implementation Group 1, the basic set. The CIS Controls start penetration testing at Implementation Group 2.
- A red team tests your detection and response. If nobody watches alerts at your company, there is nothing for it to test. Fix what a stranger can reach in your app first.
On this page
- Red team vs pentest: the difference in one table
- What a penetration test is, with the standard definitions
- What a red team exercise is
- Who needs which
- Why a startup with an AI-built app needs neither first
- What to do instead, this week
- Common mix-ups: purple team, bug bounty, AI red teaming
- Where Keybrum stands: neither a red team nor a human pentest
- Questions
- Sources
Red team vs pentest: the difference in one table
Both are attacks you asked for. They differ in what they try to learn. A penetration test is wide and loud: testers get a scope, a few days and often accounts, and they report every weakness they can prove. A red team exercise is narrow and quiet: a team picks a goal, such as reaching the customer database, and tries to get there the way a real adversary would, without being caught.
| Penetration test | Red team exercise | |
|---|---|---|
| Question it answers | What is broken in this scope? | Could this adversary reach its goal, and would we notice? |
| Scope | Defined in advance: an app, an API, a network range | The organization: people, premises, suppliers, any system that leads to the goal |
| Goal | Find and prove as many flaws as possible | Reach one objective by any allowed path |
| Who knows | The team usually knows the test is running | Often only a few people, so the defenders react as they would to a real attack |
| Methods | Technical testing of the scope | Technical attacks, plus phishing, phone calls or physical entry when agreed |
| Stealth | Not needed | Central: being detected is part of the result |
| What it measures | The flaws in your systems | Your detection and response, often called the blue team |
| Output | A list of findings with proof and fixes | A story of the attack: what worked, what was detected, how long it took to respond |
| What you need first | A deployed system and test accounts | A security team, monitoring, and the basics already fixed |
What a penetration test is, with the standard definitions
The NIST glossary carries several definitions. The one from NIST SP 800-115, the technical guide to security testing, reads: "Security testing in which evaluators mimic real-world attacks in an attempt to identify ways to circumvent the security features of an application, system, or network." It adds that most penetration tests involve looking for combinations of vulnerabilities that give more access than a single one would.
NIST SP 800-53, the catalogue of security controls, uses a shorter one: "A test methodology in which assessors, typically working under specific constraints, attempt to circumvent or defeat the security features of a system." The words that matter are specific constraints. A pentest has a scope, a time box and rules of engagement, and the team being tested usually helps: it provides accounts, documentation, sometimes the source code.
The CIS Controls separate it from scanning. Control 18 says vulnerability testing only checks for the presence of known, insecure assets and stops there, while penetration testing goes further and exploits the weaknesses to see how far an attacker could get. The pentest vs vulnerability scan guide covers that line, and the web app pentest guide shows what a test of one app includes.
What a red team exercise is
The glossary NIST publishes takes its red team definition from the US Committee on National Security Systems: "A group of people authorized and organized to emulate a potential adversary's attack or exploitation capabilities against an enterprise's security posture." Its stated objective is to improve security by demonstrating the impacts of successful attacks and by demonstrating what works for the defenders, the blue team, in an operational environment.
NIST SP 800-53 defines the red team exercise as an exercise reflecting real-world conditions, conducted as a simulated adversarial attempt to compromise an organization's missions or business processes, to give a comprehensive assessment of its security capability.
CIS Control 18 states the relation to pentesting in plain terms. Red team exercises are similar to penetration tests in that vulnerabilities are exploited, but the focus is different: red teams simulate the tactics, techniques and procedures of a specific attacker, to evaluate how the environment would withstand that adversary or that category of adversaries.
The regulated version exists too. The European Central Bank's TIBER-EU framework is a European framework for threat intelligence-based ethical red-teaming, designed for entities that provide core financial infrastructure. The ECB says it can help financial entities meet the threat-led penetration testing requirements of the EU's Digital Operational Resilience Act. That is the kind of organization red teaming was built for.
Who needs which
| Organization | What fits | Why |
|---|---|---|
| Bank, payment infrastructure, critical operator | Red team exercises, on top of regular pentests | A regulator may require them, and there is a defence team to measure |
| Company with a security operations team and monitoring | Pentests on each system, a red team exercise when the basics hold | The exercise tells you whether the monitoring you pay for sees an attack |
| Software company selling to enterprises | A penetration test of the product, once a year or per major release | Customers and auditors ask for the report. See the cost guide |
| Startup with one app, no security staff | Self-checks and automated attacks now, a pentest when a customer asks | There is no blue team to test, and the common flaws are found by simpler means |
The CIS Controls rank this for you. They sort their safeguards into three Implementation Groups, from the basics every organization should do (IG1) to the full set for mature ones (IG3). Control 18 has five safeguards. The first three, a penetration testing program, a yearly external test and fixing its findings, start at IG2. The last two, validating security measures after each test and a yearly internal test, are IG3 only. Nothing in Control 18 is in IG1.
Why a startup with an AI-built app needs neither first
Suppose you shipped an app with Lovable, Cursor or Claude Code on Supabase or Firebase, and you are two people. Work through what each exercise would give you.
- A red team would have nothing to measure. Its result is how well your defenders detect and respond. With no monitoring and nobody on call, the answer is known before the invoice: the attack would not be noticed. You would pay senior consultants to learn it.
- A pentest would spend its first days on the cheap findings. A table with row level security off, a server key in the JavaScript bundle, one user reading another user's records by changing an id. A tester finds these quickly, at a day rate. Software finds them for far less.
- Both describe one moment. An app built by prompts changes every day. A migration or a policy edit can open a table the day after the report.
This is not an argument against human testing. A penetration test becomes the right purchase when a customer contract, an auditor or card rules ask for a report, or when real money moves through logic that only a person can reason about. The pentest guide for startups covers when that moment comes and how to scope it.
What to do instead, this week
Do the equivalent of a pentest's first day yourself. You need your own app, two accounts on it, and half an hour. If the app runs on Supabase, start in the SQL editor:
-- Tables in the public schema with row level security off
select tablename
from pg_tables
where schemaname = 'public' and not rowsecurity;
-- Policies that let everyone through
select tablename, policyname, cmd, roles, qual, with_check
from pg_policies
where schemaname = 'public'
and (qual = 'true' or with_check = 'true');- 1
Close what anyone can read
Enable row level security on every table the first query returns and replace the open policies from the second. The Supabase RLS guide has policies to paste. On Firebase, review your rules with the Firebase rules checker.
- 2
Take server keys out of the browser
Paste your bundle into the API key leak checker. Any secret key it finds must be rotated and moved to server code.
- 3
Run the two-account test
As user B, request a record user A created. If it comes back, you have an IDOR, the flaw at the top of the OWASP lists.
- 4
Check the endpoints that cost money
Call your AI endpoint and your Stripe webhook signed out, on your own app. An AI route that answers a stranger spends your credits. A webhook that accepts an unsigned event can mark an order as paid.
- 5
Repeat on every deploy
The checks describe today's build. Put them on a schedule, or let a tool run them each time you ship.
Common mix-ups: purple team, bug bounty, AI red teaming
- Purple team. The red team and the defenders work together in the open, attack by attack, so the defenders tune their detection as they go. It is a training format more than a test.
- Bug bounty. Outside researchers are paid per valid finding, with no fixed schedule. It suits a company that can triage reports every day.
- AI red teaming. The same words are used for testing an AI model or a chatbot with adversarial prompts. It is a different exercise from the one on this page.
Where Keybrum stands: neither a red team nor a human pentest
Keybrum is automated. It does the part of a penetration test that software can do on an app built with AI tools, and it does it on every deploy. It attacks your live app as a stranger and as two accounts it signs up: Supabase tables, functions, storage and realtime channels, Firebase rules, server keys in the bundle checked against their provider, one account reading or changing another's records, AI endpoints that answer without an account, unsigned Stripe webhooks, exposed .env files, known Next.js and React CVEs. Every finding carries the request that worked, and it closes only when the same attack fails after your fix.
What it does not do is most of this page. It runs no phishing, no phone calls, no physical entry. It does not try to stay hidden and does not measure whether your team detects an attack. It has no human judgment on business logic, and it does not issue the signed report of a pentest firm. If a regulator asks you for a red team exercise or a customer for a third-party penetration test, hire the people who do that. Keybrum only attacks apps whose owner proved ownership.
A one-time report costs €49. Plans start at €29 a month excluding VAT for 5 apps, attack every deploy and keep retests open. Run it before a human test, so the tester's days go to what software cannot find.
Questions
Is red teaming the same as penetration testing?
No. Both exploit weaknesses with permission. A penetration test covers a defined scope and reports every flaw it can prove. A red team exercise imitates a specific adversary across the organization and measures whether the defenders detect and stop it.
Which is more expensive, a red team or a pentest?
Both are quoted per engagement, from the scope and the days of work. A red team exercise covers people and premises as well as systems and runs longer than a test of one app, so expect a larger quote. Published pentest prices are in the penetration testing cost guide.
Should a startup hire a red team?
Rarely. A red team tests detection and response, so it needs a security team and monitoring to measure. The CIS Controls place penetration testing itself in Implementation Group 2 and above. A startup gets more from closing open tables, leaked keys and missing ownership checks.
What comes first, a pentest or a red team exercise?
The pentest. It finds and fixes the flaws in each system. A red team exercise then checks whether an attacker can still reach a goal and whether you notice. Running the exercise first mostly rediscovers what a pentest would have listed.
What is a blue team?
The blue team is the group that defends: the people who watch alerts, investigate and respond. The glossary NIST publishes describes the red team's purpose partly as demonstrating what works for the defenders, the blue team, in an operational environment.
Is Keybrum a red team tool?
No. Keybrum is an automated attack on your own live app that proves each finding with the request that worked. It does no social engineering, does not try to evade detection and does not test your response to an incident.
Sources
- 1.Glossary: penetration testing (NIST SP 800-115, SP 800-53 Rev. 5), NIST
- 2.Glossary: red team (CNSSI 4009-2022), NIST
- 3.Glossary: red team exercise (NIST SP 800-53 Rev. 5), NIST
- 4.SP 800-115: Technical Guide to Information Security Testing and Assessment, NIST
- 5.CIS Critical Security Control 18: Penetration Testing, Center for Internet Security
- 6.CIS Controls Assessment Specification: Control 18, Center for Internet Security
- 7.What is TIBER-EU?, European Central Bank
- 8.CVE-2025-48757, NIST National Vulnerability Database