Enjoy complimentary entry to high concepts and insights — chosen by our editors.
VantageScore says it is constructing the longer term of credit score scoring. But based mostly on our evaluation, the inspiration it is constructing on is shaky at finest.
Earlier this month, we revealed a examine exhibiting that VantageScore 4.0’s claimed efficiency good points over Classic FICO are overstated and based mostly on flawed methodology. In response, VantageScore launched two rebuttals — neither of which immediately addresses the core issues we recognized.
Instead of partaking with the proof, VantageScore doubles down on narrative. But this is not a branding contest. It’s concerning the integrity of mortgage danger administration. And whenever you dig into VantageScore’s evaluation, the issues are too huge to disregard.
1. Apples-to-Oranges rating aggregation
VantageScore’s evaluation depends on an apples-to-oranges comparability. Its white paper evaluates VantageScore 4.0 utilizing a tri-merge common (the common of all three bureau scores which has not been authorized), whereas Classic FICO is measured utilizing the tri-merge center (the business customary utilized by the GSEs).
This issues. When we re-ran the evaluation utilizing the identical aggregation methodology for each scores — tri-merge center — the supposed efficiency benefit of VantageScore dropped from 11% to three%.
Despite our preliminary critique, VantageScore continues to tout comparisons based mostly on a rating aggregation methodology that the GSEs haven’t adopted. Unless VantageScore has entry to unannounced regulatory modifications, that is both a methodological oversight or a deliberate try and cherry-pick the very best end result.
READ MORE NMN LOANTHINK
VantageScore 4.0’s predictive energy stands as much as scrutiny
FICO is not the issue. A untimely two-score system is
Credit rating competitors reduces mortgage market danger
Pulte’s tweet palms credit score bureaus an unfair edge
2. Selection bias by design
VantageScore’s “stress testing” is a textbook case of choice bias. The mannequin was examined on loans with Classic FICO scores between 620 and 720, however the VantageScore 4.0 values have been allowed to span the total 383–850 vary. This uneven filtering offers VantageScore 4.0 extra room to rank-order danger, whereas artificially compressing the Classic FICO distribution.
When we flipped the filter—holding VantageScore 4.0 to ≤720 and permitting Classic FICO its full vary—the outcomes reversed. A mannequin that solely exhibits a bonus when the scoring vary is tilted in its favor can’t credibly declare predictive superiority.
3. Misleading headline metrics
Yet in its rebuttal, VantageScore sidestepped our core methodological issues. Instead, it repeatedly cites a +48.5% enchancment in default prediction and an 11% benefit in “head-to-head” comparisons. But each figures stem from flawed methodologies: the 48.5% from the biased stress check described above, and the 11% from the apples-to-oranges rating aggregation.
When we corrected each points, the efficiency benefit fell to simply 3% in a single metric—default seize within the backside decile. And on the opposite two of VantageScore’s most well-liked metrics – Gini coefficient and Kolmogorov–Smirnov (KS) – Classic FICO got here out forward.
As we’ve identified repeatedly, VantageScore’s efficiency benefit is finest characterised as modest, not transformational.
4. Segment-level evaluation constructed on the identical flaws
VantageScore additionally criticizes us for not replicating its segment-level findings (e.g., by rating tier or cost quantity). But these analyses undergo from the identical flawed assumptions because the headline outcomes: utilizing a tri-merge common and making use of biased filtering.
When we re-ran these breakdowns utilizing the correct methodology, the outcomes fell flat. In some circumstances, VantageScore’s claimed benefit disappeared completely. In others, Classic FICO carried out higher.
5. Mischaracterized “impartial” research
VantageScore claimed its outcomes are backed by different impartial research. But two of the 4 research cited seem to undergo from the identical methodological flaws we recognized in VantageScore’s white paper. The different two research, actually, reinforce our findings
JPMorgan’s report, for instance, discovered solely a 3% carry for VantageScore in capturing 60+ day delinquencies—an identical to our findings. Kroll Bond Rating Aagency concluded that each fashions carried out successfully, with solely “slight” benefits for VantageScore in sure segments.
This is not overwhelming proof of superiority. It’s affirmation that VantageScore’s edge—if it exists in any respect—is modest.
6. The improper repair for the true downside
Perhaps VantageScore’s most compelling argument is that it’s going to increase entry to homeownership. But the first barrier dealing with many potential homebuyers at the moment just isn’t an outdated scoring system—it’s a power scarcity of provide. Simply giving extra debtors a credit score rating would not make houses extra inexpensive. And pushing extra debtors into a decent market with looser credit score can backfire, resulting in increased costs and riskier loans
(For the extra detailed point-by-point rebuttal VantageScore’s claims, see right here.)
Proceed with warning
Ultimately, this debate is not about clinging to the previous. It’s about not dashing right into a flawed two-score regime, particularly when these flaws are hidden behind advertising and marketing spin and methodological sleight of hand.
As we famous in a latest op-ed, a rushed transfer to a dual-score regime, notably one formed by industrial pursuits, introduces critical challenges, together with complexity in pricing by new LLPA matrices, alternatives for rating procuring and mannequin gaming, and potential misallocation of credit score.
Before overhauling the mortgage credit score scoring system, FHFA should insist on rigorous, clear, and replicable evaluation—not self-serving white papers or cherry-picked comparisons.
Otherwise, we danger destabilizing the very system we’re attempting to enhance.
