The short version
- We only test apps whose owner has verified them. No exceptions.
- We test the live app from the outside, the way a user or a stranger reaches it. We don't need your code or your builder login.
- There are two levels:
- The Stranger (Dummy Run) has no account and sees what anyone on the internet sees.
- Two dummies (Full Crash Test) are two test accounts you create. They check whether one user can get at the other's data.
- Every finding comes with evidence. If we can't show it, it doesn't count against you.
- The Safety Rating follows a fixed, published rule. It's the same rule for every app.
Step 0: Verification and scope
Before anything runs:
- You prove ownership in one of three ways: a
<meta name="crashdummy-verify">tag on your site, a/.well-known/crashdummy.txtfile, or a DNS TXT record. - We set the scope. We test your verified address, plus the backend services your own app's code connects to, for example your Supabase project. Requests to anything else are blocked.
- Dummy Run: you accept our testing terms when you order. The Stranger never logs in, so the verification tag and that acceptance are enough.
- Full Crash Test: you also create two test accounts and sign a one-page testing authorization, because the dummies go inside your app.
What we test
Level 1: The Stranger (Dummy Run, and the start of every Full Crash Test)
No account, no password. The Stranger only sees what any visitor could see.
| # | Check | What we actually do | What we record |
|---|---|---|---|
| S1 | Look around | A real browser opens your app, visits its pages and notes which services it talks to (for example Supabase or Stripe) | The services and public keys found. Public keys are expected, so they aren't a finding on their own |
| S2 | Secrets in your public code | We scan the JavaScript your app sends to every visitor for known secret formats: Supabase secret or service_role keys, Stripe live secret keys, AI provider keys and similar. Any token we find is decoded to see what role it carries | Key type plus a masked prefix (sk_live_ab12…), never the full key |
| S3 | Tables anyone can read | Using only your app's public key, we ask each table whether it will answer someone who isn't logged in | Table name and row count, never row contents |
| S4 | Files anyone can reach | We check whether storage buckets are public and whether their contents can be listed | Bucket name, public or private, a file count |
| S5 | Functions anyone can call | We list database functions exposed to logged-out visitors | Function names |
| S6 | Web hygiene | We check security headers and CORS, and whether /.env, /.git or source maps are reachable | Present or absent, plus response codes |
Level 2: Two dummies (Full Crash Test only)
Dummy A and Dummy B are two test accounts you create in your own app.
| # | Check | What we actually do | What we record |
|---|---|---|---|
| D1 | Dummy A vs Dummy B | Dummy A uses the app normally while we record its requests. We then replay the requests that refer to A's records using Dummy B's session, and try to read, change and delete them | The request shape, the status code, and whether the response matched A's data. Matching uses hashes, and we never store the content |
| D2 | Self-upgrades | Dummy B tries to edit its own profile fields that control privileges, like is_pro, plan or role | Whether the change was accepted |
| D3 | Locked doors | Dummy B (a regular user) tries to open admin pages and call admin-only actions | Status codes and screenshots of what B saw |
| D4 | Sign-up and sign-in | We check whether sign-up is open, whether email is confirmed, and what the password rules are. We send a small burst to email-sending features: at most 10 requests | Settings observed, and whether the burst was stopped |
| D5 | Payments | We try to unlock a paid feature from the browser without paying. Stripe test mode only | Whether the server refused |
| D6 | Critical flows | We run your sign-up, sign-in and main action end to end, on a mobile and a desktop screen size | Pass or fail, plus screenshots |
| D7 | Human review | A human tester reviews every finding before you get it, and checks the repair prompts | — |
What we don't test (v0.1)
To be clear about the limits: we don't run full penetration tests, fuzzing or injection testing beyond the basics. We don't test native mobile apps, or anything you haven't put in scope. A crash test is focused on the problems AI-built apps hit most often, as listed above. It isn't a complete security audit.
Evidence: "evidence or it didn't happen"
- Every finding includes evidence you can check: the request we sent, the answer we got, and a timeline of what each dummy did (the Crash Replay).
- Confirmed vs likely: a finding is confirmed when the evidence directly proves it, for example Dummy B received a row that matched Dummy A's data. Anything weaker is marked likely. Only confirmed findings affect your Safety Rating. Likely findings still appear in the report, so you can judge them yourself.
- Accuracy target: we test every check against our own deliberately broken demo apps and their fixed versions. Each check must catch the planted problem, and stay quiet on the fixed version. Our target is under 5% false positives on that test suite.
The Safety Rating
Severity levels
| Level | Meaning | Examples |
|---|---|---|
| Totaled (critical) | Anyone, or any user, can read or change other users' data, or a privileged secret has leaked | A table of user profiles readable without login. Dummy B reads Dummy A's records. A service_role key in public code |
| Crash (high) | Privilege escalation, private files that are public, or payment bypass | A user can set is_pro=true. Invoices in a public bucket. A paid feature unlocked from the browser |
| Dent (medium) | Abuse vectors and weak sign-in flows | An email-sending feature with no limit. No email confirmation where one is needed |
| Scratch (low) | Hygiene issues | Missing security headers. A reachable source map |
Severity is scored as impact × exploitability. In other words: how bad would it be, and how easy is it to do?
The rule
Every app starts at 5 stars. Then, in this order:
- Any Totaled finding caps the rating at 1 star.
- Any Crash finding caps it at 2 stars.
- Two or more Dents cap it at 3 stars.
- One Dent caps it at 4 stars.
- Scratches then lower it in half-star steps: each Scratch removes half a star from the cap, Scratches together remove at most one star, and the rating never drops below 1 star.
Only confirmed findings count. The same findings always produce the same rating, with no judgment calls and no hidden weighting.
Worked examples
| Confirmed findings | Rating | Why |
|---|---|---|
| None | ★★★★★ 5 | Nothing found in the areas we test |
| 1 Crash, 1 Dent | ★★ 2 | A Crash caps the rating at 2 |
| 2 Dents | ★★★ 3 | Two or more Dents cap the rating at 3 |
| 1 Dent | ★★★★ 4 | One Dent caps the rating at 4 |
| 1 Scratch | ★★★★½ 4.5 | Each Scratch removes half a star |
| 4 Scratches | ★★★★ 4 | Scratches together remove at most one star |
| 1 Dent, 1 Scratch | ★★★½ 3.5 | One Dent caps the rating at 4, then the Scratch removes half a star |
| 1 Totaled, 3 Scratches | ★ 1 | Never below 1 star |
What the rating means, and what it doesn't
- It describes the app on the test date, across the checks listed above. It isn't a guarantee, a certification or a promise that your app is secure.
- 5 stars means: in the areas we test, our dummies found nothing confirmed.
- Any AI edit can change it. That's why every report is dated, and why Re-crash and Airbag exist.
What we never do
We never:
- Test an app whose owner hasn't verified it. That includes "just a quick look" at someone else's app, a competitor's, or a well-known app.
- Store other people's data. We keep counts, hashes, structure and redacted snippets. We don't keep content.
- Keep secrets. Any key we find is shown masked, and is never stored or used.
- Try to knock your app over. No load tests, no stress tests, no denial of service. We send at most 5 requests per second, with a cap per test and timeouts on every check.
- Change real users' data. Writes happen only in the test accounts you gave us, and we clean them up afterwards.
- Touch real money. Payment checks run in Stripe test mode only.
- Contact your users or staff, or try phishing or social engineering.
- Test the platforms themselves. We don't probe Lovable, Supabase, Vercel or anyone else's infrastructure, or any other customer's project hosted there.
- Keep access. We don't keep access after the test, install anything, or move beyond the scope you approved.
- Publish your results. Reports are private. You decide whether to share a public summary, and it never includes evidence.
And we always:
- Keep a log of every request we send: method, address, status and which check sent it.
- Can stop instantly: any test can be halted at any moment.
- Delete evidence 30 days after your report, unless you ask in writing for a different period.
- Tell you right away if we find something critical during a Full Crash Test. We don't wait for the report.
Changelog
| Version | Date | Change |
|---|---|---|
| v0.1 | 19 Nov 2026 | First public version |