How to benchmark a fraud API, including your own
Most accuracy claims in this category are unfalsifiable. Two different standards — ground truth where it exists, reference consensus where it does not — and why false positives deserve their own column.
Every vendor in this space publishes an accuracy number. Almost none publish what it was measured against, which makes the number unfalsifiable and therefore worthless. Here is the method we use on ourselves.
Two standards, because signals are not equally knowable
Some facts have an authoritative record. Whether an address is a Tor exit is answered by the Tor Project's published list. When a domain was registered is answered by RDAP. Whether a phone number is valid is answered by libphonenumber. Being wrong against these is simply being wrong.
Other facts have no such record. Nobody can prove an address 'is a VPN' — every vendor runs heuristics over overlapping evidence. For those we compare against two independent commercial detectors and only call it an error when both disagree with us. One dissenting vendor is a difference of opinion, not a defect.
Collapsing these two into one percentage is where most published accuracy numbers go wrong.
False positives get their own column
Missing a VPN costs one signal out of several. Flagging Duke University blocks real students who have no idea why and no way to tell you. Averaging those into a single score hides the failure that actually damages people.
So the corpus deliberately contains addresses that must come back clean — consumer ISPs, universities, corporate networks, public DNS resolvers — and those are reported first and separately.
Grade the verdict, not just the flag
An early version of our benchmark scored a VPN as detected if any signal fired. That passed a set of NordVPN exit nodes 15 out of 15. The real picture:
- 12 were flagged as VPN and scored 40 — review.
- 3 were flagged only as datacenter and scored 30 — allow.
A fraudster on those three walked straight through with a clean verdict. Some signal firing is not the same as the right one, and a benchmark that cannot see the difference will report a miss as a hit.
Then we over-corrected
Tightening the rule so any expected-flag case landing on 'allow' counted as a failure promptly failed AWS, Hetzner, Cloudflare and Google — all correct behaviour. Plain datacenter is worth 30 points on purpose, because server-to-server traffic legitimately comes from AWS and scoring all of it above the allow threshold would be a false-positive machine.
The corpus now distinguishes anonymisers from plain hosting and holds only the former to 'must not be allow'. A benchmark that cries wolf on correct behaviour gets ignored, and would push real tuning the wrong way.
Verify the labels before you trust them
Every expectation in our corpus was confirmed against an independent source before being written down. This is not pedantry: our VPN list once claimed an ASN was Windscribe when it was Wake Forest University, precisely because nobody checked the label.