Email Verification Benchmark Dataset

Dataset v1.0 · Published July 6, 2026 · 40 scenarios · License CC BY 4.0

A public, synthetic benchmark of 40 email-verification scenarios across the nine categories InboxPolicy's send-decision API distinguishes. Each scenario pairs a ground-truth Expected verdict with the verdict InboxPolicy would actually return, so the dataset is an honesty gauge: it includes realistic agreement and disagreement rather than a perfect scorecard.

At a glance

40 synthetic scenarios across 9 categories. Every address uses a reserved example domain — none is a real third-party inbox, none contains a secret. Download the raw data and reuse it under CC BY 4.0.

Download dataset (CC BY 4.0) ↓

How to read this table

Expected verdict is the ground-truth action a perfect verifier would return. InboxPolicy result is what InboxPolicy's send-decision API (decide_send / verify_email) actually returns for that address. Where the two columns disagree, the difference is deliberate and explained in Notes — not hidden. Verdicts use InboxPolicy's five actions: send, send_with_caution, review, retry_later, avoid.

The 40 scenarios

#EmailCategoryExpected verdictInboxPolicy resultNotes
1 alice.johnson@example.com valid send send Clean signal: SMTP RCPT accepted, no catch-all signature, confidence 95.
2 bob.smith@example.org valid send send Verified mailbox, confidence 92.
3 carol.wong@example.com valid send send Confidence 90; sits in the ~90% valid-verdict agreement band vs MillionVerifier.
4 d.lee@example.net valid send send Confidence 93.
5 erin.murphy@example.org valid send send Confidence 91.
6 frank.castle@example.com valid send send Confidence 88.
7 grace.hopper@example.edu valid send send Confidence 94.
8 hiroshi@example.org valid send send Confidence 89.
9 ivan.petrov@example.com valid send send_with_caution Mailbox confirmed but provider reputation lowered confidence to 68; InboxPolicy conservatively returns send_with_caution rather than send. A realistic non-agreement case.
10 julia.r@example.net valid send send Confidence 92.
11 no.such.user@example.com invalid avoid avoid Clear 550 reject; high confidence.
12 deleted.account@example.org invalid avoid avoid 550 user unknown.
13 mistype.user@example.com invalid avoid avoid 550 NoSuchUser.
14 gone.away@example.net invalid avoid avoid 550 mailbox unavailable.
15 nonexistent@example.com invalid avoid avoid Rejected at RCPT.
16 fake.name123@example.org invalid avoid avoid 550 unknown.
17 typo.user@example.com invalid avoid avoid 550 rejected.
18 old.role@example.net invalid avoid review Server returned an inconsistent SMTP response across probes; InboxPolicy could not confirm the rejection and returned review rather than risk a false avoid. Realistic non-agreement case.
19 anyone@corp.example.com catch_all review review catch_all evidence tagged; default policy maps to review. ~30-40% of B2B addresses look like this.
20 contact@startup.example.org catch_all review review Tagged catch_all; review by default, never guessed safe.
21 sales@enterprise.example.com catch_all review review review with catch_all flag set.
22 ceo@bigcorp.example.net catch_all review review review (catch_all).
23 anyone@growth.example.com catch_all review send_with_caution Under an aggressive policy configuration the same catch_all evidence maps to send_with_caution instead of review; InboxPolicy never returns send for an unconfirmed mailbox.
24 support@saas.example.org catch_all review review review (catch_all).
25 junk@throwaway.example disposable avoid avoid Disposable domain flagged; maps to avoid. Domain name is synthetic.
26 temp.user@tempmail.example.com disposable avoid avoid Disposable flagged and routed to avoid.
27 signup@disposable.example.org disposable avoid avoid Disposable domain leads to avoid.
28 x@burner.example disposable avoid avoid Disposable flagged, returned avoid.
29 info@example.com role_based send_with_caution send_with_caution Role-based address flagged; deliverable with moderate confidence and elevated reputation risk.
30 sales@example.org role_based send_with_caution send_with_caution Role address; send_with_caution.
31 admin@example.com role_based send_with_caution send_with_caution Role address flagged; send_with_caution.
32 support@example.net role_based send_with_caution send_with_caution Role address; send_with_caution.
33 not-an-email syntax_invalid avoid avoid Rejected locally before SMTP at 0 credits; malformed input is never billed.
34 user@ syntax_invalid avoid avoid Malformed; rejected locally, free of charge.
35 @example.com syntax_invalid avoid avoid Rejected before any SMTP check runs; 0 credits.
36 quota.full@example.com mailbox_full retry_later retry_later 552 over quota; transient condition, re-check later (the re-check is usually free within 72h).
37 inbox.full@example.org mailbox_full retry_later retry_later 552 over quota; retry_later.
38 greylisted@example.com unknown review review Honest unknown preserved as a first-class state and mapped to review; never guessed safe. May escalate to a wholesale fallback verifier.
39 protected.gw@example.org unknown review review Unknown mapped to review (first-class state), not over-promoted to send.
40 office@trapnet.example spam_trap avoid send InboxPolicy confirmed the mailbox over SMTP and returned send because it maintains no spam-trap database; a live trap looks valid to any SMTP-only engine. This is the documented gap vs ZeroBounce and the single most important disagreement in the set; route trap detection to a specialist.

Frequently asked questions

What is the InboxPolicy Email Verification Benchmark?

It is a public, synthetic benchmark dataset of 40 email-verification scenarios spanning the nine categories InboxPolicy's send-decision API distinguishes. Each scenario pairs a ground-truth expected verdict with the verdict InboxPolicy would actually return, so the dataset doubles as an honesty gauge: it includes realistic agreement and disagreement rather than a perfect scorecard.

How many scenarios does the benchmark contain, and what categories does it cover?

Forty scenarios across nine categories: 10 valid, 8 invalid, 6 catch_all, 4 disposable, 4 role_based, 3 syntax_invalid, 2 mailbox_full, 2 unknown, and 1 spam_trap. These mirror the verdict types the send-decision API produces (send, send_with_caution, review, retry_later, avoid).

Are the email addresses in the benchmark real inboxes?

No. Every address is synthetic and uses reserved example domains (example.com, example.org, example.net, and .example subdomains). None reaches a real third-party inbox and none contains a secret or credential. The SMTP transcripts behind each verdict are representative, not live captures.

Why does InboxPolicy return 'send' for the spam-trap scenario?

Because InboxPolicy runs syntax, MX, and live SMTP checks and has no spam-trap database. A pristine trap is a real, SMTP-accepting mailbox, so to any SMTP-only engine it looks valid. This is the single most important disagreement in the set and the documented gap versus tools like ZeroBounce that maintain a trap database; route trap detection to a specialist.

Under what license is the dataset released, and can I reuse it?

The dataset is released under Creative Commons Attribution 4.0 (CC BY 4.0). You may copy, redistribute, remix, and build on it for any purpose, including commercial, as long as you credit InboxPolicy (https://inboxpolicy.com/). The raw data is available at benchmark.json.

How should I read the agreement between 'Expected verdict' and 'InboxPolicy result'?

The 'Expected verdict' column is the ground-truth action a perfect verifier would return; the 'InboxPolicy result' column is what InboxPolicy's send-decision API actually returns for that address. Where they differ (for example the confidence-68 valid case, the inconsistent-probe invalid case, the aggressive catch_all case, and the spam trap), the difference is deliberate and explained in the Notes column rather than hidden.

Download dataset (CC BY 4.0) ↓

Raw JSON mirrors this page: ./benchmark.json. Credit InboxPolicy (https://inboxpolicy.com/) per the license.