Identity verification accuracy rates always require context (part 2)
This is part two of a three-part series about accuracy rates during identity verification. In part one, you’ll learn why pass rates can be misleading when comparing identity verification solutions. And in part three, you’ll discover what being a bouncer can teach you about adverse scenarios during identity verification.
People regularly ask how accurate verification checks are when spotting bad actors and fake documents. But without context, the accuracy stats for identity verification checks and models can be misleading.
Below, we explore the difference between product validation accuracy and operational accuracy, and why understanding that difference is critical when evaluating identity verification systems. We also explain why real-world operational accuracy is always a best guess and moving target.
Defining pass rates and accuracy in identity verification
We covered pass rates in detail in part one, but here’s a recap of common terms and definitions:
Pass rate: The percentage of users who have completed the verification process
Accuracy rate: How well the model correctly identifies legitimate users and denies illegitimate users. You can break this down into:
True pass: Correctly verifying a legitimate user.
False pass: Incorrectly verifying a bad actor as legitimate.
True fail: Correctly denying an illegitimate user.
False fail: Incorrectly denying a legitimate user.
People also commonly refer to “false positives” and “false negatives,” but the definition of a “positive” can differ based on the context (e.g., identity verification versus fraud prevention), so it’s helpful to use more specific language. In identity verification, a false positive refers to a false pass.
The difference between product validation and operational accuracy
There’s a big difference between building for ideal conditions and operating in unpredictable ones. An identity verification model that performs well in a controlled environment might stumble when it faces off against real users.
To distinguish between the two circumstances, you can ask about product validation accuracy and operational accuracy.
Product validation accuracy reflects how well a model performs in balanced conditions used to build and test new models.
In practice, this involves evaluating models across a range of conditions. For example, you might assume one user is familiar with technology, has a modern phone, and takes a clear selfie or picture of their document in a well-lit room. The next has poor lighting, a shaky hand, and an old phone.
The resulting validation or test accuracy rate can be a benchmark when developing or updating models. It may also be helpful when comparing models based on third-party certification or evaluation programs, such as the Age Check Certification Scheme (ACCS) or NIST’s Face Recognition Technology Evaluation (FRTE).
Operational accuracy depends on how well the model performs once it's in the real world.
Measuring operational accuracy can be difficult because you don’t have a predefined data set of known true and false identities as a baseline. As a result, it’s always an educated guess. Additionally, operational accuracy can vary depending on the organization and use case. But, in general, operational accuracy is usually lower than validation accuracy.
One of the most common misconceptions is that models have a single, universal operational accuracy. In reality, accuracy varies based on the population and context.
What can affect operational accuracy rates?
Several factors can affect the operational accuracy of an identity verification model:
Adverse scenarios: We’ll explore adverse scenarios in more detail in part three, but common challenges in government ID and selfie verifications include poor lighting, blurry images, glare, and damaged documents. You can’t plan for every adverse scenario, so it’s important to understand how the model handles adversity.
Demographic variability: Even a government ID model with a 99% global accuracy rate can fall short if it underperforms in a region where your organization has a significant presence. When that region is overrepresented in your traffic, small blind spots can balloon into large discrepancies. In other words, if certain adverse scenarios appear more frequently in your data than in the global average, the model’s regional weaknesses will translate directly into operational inaccuracy and increased risk.
Your risk profile: If your company offers valuable services or products, you may attract sophisticated bad actors and require more rigorous verification methods and checks. The complexity of the attacks can result in more false passes. And although tightening security makes sense, you have carefully configure the checks to avoid increasing your false fail rate.
A few key takeaways
Remember these points when you review claims about pass rates and accuracy rates:
Pass rates aren’t the most important metric: The ratio of correct passes and failures relative to all your verifications is more valuable than the pass rate on its own.
Pass rates are not accuracy rates: Improving the accuracy of a model and reducing false fails could increase your pass rate. However, lower accuracy can also lead to higher pass rates.
Accuracy rates during production are educated guesses: You can’t be certain which users are bad actors, so accuracy rate statistics are always best guesses.
A perfectly accurate model reflects the underlying population: If your products or services attract a lot of fraud, you should expect a low pass rate if you have a high accuracy rate.
Your situation affects pass and accuracy rates: Two organizations with the same configurations will have different pass and accuracy rates depending on their user bases and the types of fraudulent attacks they encounter.
In short, asking about pass or accuracy rates won’t reveal useful information unless you understand how they were calculated.
How Persona helps businesses optimize their accuracy
Understanding the difference between validation and operational accuracy helps you ask smarter questions, set better benchmarks, and choose the right tools. But operational accuracy isn’t a fixed number.
Accuracy reflects how well your identity verification system performs in your world with your users and risks. Part of the art and science of fraud prevention is fine-tuning configurations to increase true passes and true fails without letting bad actors slip through or disrupting legitimate users.
In part three, we’ll take a closer look at how real-world adverse scenarios will affect your pass and accuracy rates — from the lens of a bouncer.
Contact us to learn more about how you can configure Persona to improve your operational accuracy.
