“Essentially, all models are wrong, but some are useful.”
George Box wrote that with Norman Draper in 1987. I think about it more than any other sentence in my job, because almost everyone quotes it as a disclaimer and it’s actually a specification.
If your model is sometimes wrong, and every model is, then the product’s job isn’t to hide that. It’s to be precise about where the wrongness lives, and to put a person exactly there and nowhere else.
That sounds like a philosophical point. It changes what you build.
The most useful thing a model can say is “I’m not sure”
If every model is sometimes wrong, the interesting design question isn’t how to be wrong less often. It’s what the software does at the edge of its own confidence.
Our answer is three states rather than one: approve, deny, review. The two confident states exist so that nobody spends attention on them. Review exists because Box was right. It’s the model saying, in as many words, this one needs a person.
Which reframes what underwriting software is actually for. It isn’t there to make the decision. It’s there to tell a lending operation which files not to spend a human on.
You can see the difference in how lenders work. The ones running real automation concentrate their underwriters on the review queue, where judgment earns its keep. The ones without it scrutinize every file, which looks like diligence and is really uniform attention. Uniform attention is the same as no prioritization. Same headcount, far less of it pointed at anything that matters.
There’s a body of human-factors research worth knowing here, and the obvious lesson people take from it is the wrong one. Parasuraman and Manzey in “Complacency and Bias in Human Use of Automation: An Attentional Integration” reviewed decades of work on how people use automation and found that we tend to accept automated advice without independently checking it, that this happens to experts as much as to novices, and that practice doesn’t reliably fix it.
The tempting response is to tell users to be more careful. The research is fairly clear that this doesn’t work. The design response is different: don’t build a product that depends on vigilance. Route by confidence instead. If the model is sure, don’t put a person in front of it. If it isn’t, say so out loud and give that person everything they need. A product that asks people to double-check every answer is designing around a failure mode rather than removing it.
Show the reasons, and show them where someone is looking
The first consequence is one the industry agrees with and still gets wrong.
Every factor that moves the number belongs on the same screen as the number. Income stability. Overdraft and non-sufficient-funds patterns. Existing loan activity. Real debt-to-income rather than stated. Not in an appendix, not in an export, not on request.
This matters most in the review queue, because that’s where a human is actually reading. An unexplained score in a review file is worse than useless: it’s asking someone to make a judgment call while withholding the evidence. And an unexplained decision anywhere is impossible to run a business on. If a risk committee can’t reconstruct why a file was declined, they can’t defend it to an examiner, they can’t teach a new underwriter from it, and they can’t tell a good decision from a lucky one.
We publish 98% accuracy on statement analysis, measured across our model set. The other 2% is the interesting part, because that’s what your product design has to handle.
Make disagreement cheap
The second consequence is the one most vendors skip. It’s also what our last release was about.
Somewhere in every lending operation there’s an analyst with a private list of transactions they recategorize by hand every time. A deposit the model reads as income that the team knows is a family transfer. A payment that’s really a transfer between the borrower’s own accounts. They’re usually right, because they know their book in a way no model does out of the box.
Until recently that knowledge lived in a spreadsheet on someone’s desktop. Now our users can reclassify any transaction and watch the Predictive Cashflow Score™ recalculate in front of them.
That isn’t an admission that the model is unreliable. It’s what Box’s sentence looks like in software. The review queue is precisely where a person’s context beats the model’s, so correcting the model there should cost one click rather than a support ticket. A product that makes agreeing easier than disagreeing has already been decided for the user.
Decide which errors matter
The third consequence is the management half, and the unglamorous one.
Not all wrongness is equal, and treating it as if it were is the most common mistake I see in model-driven products. Being wrong about an applicant you were never going to fund costs you nothing. Being wrong on the marginal file, the one sitting right at the edge of your policy, costs you everything.
Which means ranking your error types is a product decision, not a data science one. And it’s the same decision as setting your review thresholds: every file you route to a human is a bet that the model’s uncertainty there is expensive enough to be worth a salary. If the product team doesn’t make that call deliberately, the model makes it by accident, based on whatever happened to be in the training data.
What to ask a vendor
If you’re evaluating cashflow underwriting software, the questions that separate products aren’t about accuracy figures. They’re about what happens at the boundaries.
- Does your system tell me when it isn’t confident, or does it always return a decision?
- What share of files land in review, and can I move that threshold myself?
- When your system misreads a transaction, how does my team find out and how long does fixing it take?
- Six months from now, can I reconstruct why a specific file was declined?
- Which errors does your model optimize against, and who decided that?
Any vendor can quote you a number. Fewer can tell you which files that number doesn’t cover.
The decision stays with the lender
None of this argues for less automation. It argues for automation that’s honest about its limits, because that’s the only kind a credit committee can build on. Done properly it means fewer people looking at more files, and the ones they do look at are the ones worth the time.
Cashflow underwriting sits next to the scoring models you already trust, not in place of them. It reads the borrower’s present where a score reads their history. The decision itself stays where it belongs, with your policy and your team.
All models are wrong. Building the useful ones is the whole job.
To see what an explainable cashflow read looks like next to the models you already run, book a walkthrough with our team. Bring your hardest file.
References
- George E. P. Box and Norman R. Draper, Empirical Model-Building and Response Surfaces, Wiley, 1987, p. 424. The shorter form first appeared in Box’s 1976 paper for the Journal of the American Statistical Association. Background: https://en.wikipedia.org/wiki/All_models_are_wrong
- Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration,” Human Factors, vol. 52, no. 3, 2010. https://journals.sagepub.com/doi/10.1177/0018720810376055
