Artificial intelligence has revolutionized credit decisioning. What was once a slow, manual and subjective process is becoming highly automated, and the all-important act of approving or denying credit is increasingly being turned over to highly sophisticated neural networks.

Compared with the simple logistic regression models still used by many financial institutions, AI models can provide a 15% to 25% increase in accuracy, Kevin Moss, former chief risk officer at $1.9 trillion Wells Fargo and neobank SoFi, told Bank Automation News.

AI also eliminates human biases a loan officer could hold, a promising development in a banking system that has long delivered unequal outcomes for protected groups, including women and ethnic minorities. However, this potential benefit is tempered by immense risk, as these models — if left unchecked — can absorb biases from the data they were trained on and reproduce them at scale.

A study released last year from New York University’s Stern School of Business using 11.8 million Payment Protection Program loans awarded between April 2020 and May 2021 showed that fintechs using automated decisioning processes were the fairest toward Black-owned firms. Furthermore, legacy banks that had automated their decisioning processes consistently ranked higher in fairness than their non-automated peers. 

“[When] you can make quick, data-driven decisions that also don’t flow through the potentially biased black box of the human mind, then you increase the diversity of people you’re able to lend to,” David Snitkof, an author of the study, told BAN. Snitkof is senior vice president of growth at document processing fintech Ocrolus. 

Yet lenders using AI must take proactive measures to ensure that their models do not revert to the prejudices of the past. 

“For populations whose credit data is messy, missing or wrong, the algorithms inherently regard them as riskier,” Kareem Saleh, co-founder and chief executive of FairPlay AI, a fintech that uses machine learning (ML) models to help financial institutions reduce lending bias, told BAN. “As a result, the judgment of those algorithms ought to be … more rigorously scrutinized.” 

How does AI-based lending work?

AI-based credit decisioning processes are based on neural networks — the same ML engines that power large language models (LLM) like ChatGPT, John Merrill, co-founder and chief technology officer of FairPlay, told BAN. 

 These neural networks are given a set of data inputs deemed relevant to evaluating creditworthiness and produce a “yes” or “no” decision, Merrill said. In between, the data passes through multiple layers composed of equations called nodes, which process the data according to specific, weighted parameters. 

Complex models can have upward of 50 to 100 of these layers, each containing hundreds or thousands of nodes, Merrill said. By adjusting the weights of these nodes, credit modelers can fine-tune algorithms to deliver the desired results. 

This calibration is conducted through a process called backpropagation, which compares the model’s outputs to training data and adjusts the model’s parameters so the values match, Merrill said. 

With enough tweaking, models can learn to make extremely accurate predictions from training data with only marginal error. 

Biased data, biased algorithms

Yet these algorithms are only experts in the context of the data on which they are trained. When that data reflects what FairPlay’s Saleh calls the U.S.’ “history of financial exclusion,” the algorithms learn to reproduce bias in their predictions. 

“The models are overfit to the majority population,” Saleh said, and thus ill-equipped to evaluate the creditworthiness of protected groups. 

Black borrowers, who have historically been pushed toward predatory lending products, are more likely to have lower credit scores and thus be deemed less creditworthy by an algorithm, Saleh said. 

AI-based mortgage underwriting models were 80% more likely to reject Black borrowers than their white counterparts in similar financial situations, according to a 2021 study by investigative outlet “The Markup.” 

In addition, people who have taken time off to raise children may have gaps in their credit histories, and newly arrived immigrants may have no credit history at all, Saleh said. 

The former group has been “preyed upon by the financial system,” so their risk is “overstated,” he said. And for the latter group, “there’s not enough recent data … for the conventional methods to be able to score.”  

Lenders can be held legally liable for this discrimination under the 1974 Equal Credit Opportunities Act (ECOA) if it meets one of two standards: disparate treatment or disparate impact.  

Disparate treatment entails deliberate prejudice against a borrower based on their identity, while disparate impact occurs when a lender who does not discriminate based on the appearance of the borrower still adversely impacts a protected group, according to ECOA. Exceptions are only given if a lender can prove the disparity served a “legitimate business interest” and could not be satisfied by a less discriminatory alternative. 

In 2019, the New York State Department of Financial Services opened an investigation into an Apple credit card backed by Goldman Sachs after allegations that it gave significantly lower credit limits to female borrowers. Though Apple was not found guilty of violating ECOA, the incident is an example of discrimination that flies under the radar of regulators.

Explainability and the ‘opaque box’

AI models are most at risk of displaying disparate impact when their decision-making processes are not clear to users, Hani Hagras, chief science officer at banking software provider Temenos, told BAN. 

Lack of transparency in algorithms is not new. In fact, one of the ultimate “opaque boxes” is something many take for granted as a part of credit decisioning: the FICO score. — Hani Hagras, chief science officer, Temenos

Hagras, who helped found the “explainable AI” movement more than a decade ago to advocate for transparency in ML, calls this type of algorithm an “opaque box.”

“[In] most opaque box models, there is no quality assurance,” Hagras said. “This often happens, like … rejecting females with [the] Apple credit card and saying, ‘Oops, we have done a mistake. Let’s adjust.’”

This lack of transparency is one of the reasons $208 billion Fifth Third Bank, which still uses regression modeling for credit scoring, is exercising extreme caution before adopting an ML algorithm, Jay Budzik, the bank’s senior vice president in charge of AI and ML, told BAN.

“[Among] hundreds or thousands of attributes, any one … could be a proxy for race and ethnicity and expose the bank to disparate treatment risk,” Budzik said.

One vendor approached the bank with a model that used a category called “derogatory elements” as one of its credit scoring inputs, Budzik recounted. Upon closer inspection, Budzik discovered that one of these “derogatory elements” was a history of incarceration, a metric he was not comfortable using due to its potential for discrimination.

Lack of transparency in algorithms is not new. In fact, one of the ultimate “opaque boxes” is something many take for granted as a part of credit decisioning: the FICO score, Temenos’ Hagras said.

The Fair Isaac Corp, which introduced its scoring system in 1989 to rate borrowers, lists five weighted categories, including payment history and debt, as criteria for evaluation, but does not divulge its exact formula, according to its website.

The method for determining a FICO score also exhibits bias against the poorest Americans, according to an often-cited 2015 report from Consumer Financial Protection Bureau (CFPB). About 45 million low-income Americans cannot be scored using the FICO system because they lack the necessary data, the report said. And obtaining credit is much more difficult for those without FICO scores.

Alternative data sets

Given the potential for bias in credit bureau metrics including those used by FICO, many FIs are turning to other sources of data to underwrite underserved customers.

Fifth Third now includes checking account data in its logistic regression models in considering loans for customers with thin credit histories, Budzik said.

Yet AI-based methods are far more effective than traditional approaches when it comes to modeling large amounts of alternative data, according to Moss, the former CRO at Wells Fargo and SoFi.

Fintech Plaid bases its business model on allowing users to share financial data including bank statements and rent and utility bills with FIs via open APIs, according to its website. Tech- forward lenders including SoFi and digital loan provider LendingPoint are using this cash flow data when considering loans for customers who may not qualify using traditional methods.

In the name of fairness and accuracy, other FIs are turning to more unusual metrics.

Digital banking platform Clair partners with employers to offer workers cash advances on paychecks, founder and Chief Executive Nico Simko told BAN. The company bases its decisioning process on payroll and time and attendance data collected from those partners.

“We underwrite based on, ‘Did you work or not?” said Simko, who immigrated to the U.S. from Argentina. “That has nothing to do with your credit score.”

Fairness as a service

For FIs seeking an outside perspective to help increase fairness, there is another option.

As ML technology advances, an increasing number of providers have begun offering “fairness as a service,” using proprietary algorithms to assess bias in decisioning models and offer fairer alternatives.

“We think that you can be fair and make more money,” FairPlay’s Saleh said. “Fairness is good for profits, good for people, good for progress, if done correctly.”

At the core of FairPlay’s strategy is a push to shift algorithms from a single-minded focus on minimizing defaults toward a “compound objective” that also prioritizes fairness, Saleh said. FairPlay’s algorithms treat bias as a form of “model error,” leading them to continuously adjust the weights of various parameters through backpropagation until they achieve an optimal balance of fairness and accuracy.

By combining this fine-tuning with the inclusion of alternative data sources, such as borrowers’ transaction and employment history, FairPlay says it has helped increase both fairness and revenue among its customer base, which includes fintechs such as loan refinancing provider Splash Financial and recreational vehicle lender Octane Finance. 

FairPlay’s AI-based credit model evaluation increased approval rates by 10% to 20% for one fintech lender, while simultaneously decreasing adverse impact on Black applicants by 35%, according to Saleh.

Yet, some of FairPlay’s debiasing tactics are not universally accepted in the fair lending industry.

One method, known as adversarial debiasing, uses a second, “adversary” AI model to guess the identity of a borrower based on the credit score they receive, FairPlay’s Merrill said. If the “adversary” finds that lower scores are disproportionately associated with borrowers from protected groups, it signals to the “student” model that it must adjust accordingly.

Solas AI, a FairPlay competitor, believes a more conservative approach is necessary to ensure debiasing methods pass legal muster, Solas co-founder and Chief Technology and Innovation Officer Nick Schmidt told BAN. 

“[In] the methodologies that incorporate protected class status into training … you can identify, down to an individual, someone who is harmed as a result of the fairness algorithm,” Schmidt said. “With that information, I think that courts could say that you’re engaging in illegal affirmative action.”

Solas believes that sticking to a more conventionally accepted method similar to FairPlay’s “compound objective” approach can ensure compliance and minimize legal exposure for its clients, Schmidt said. 

Exploring generative AI

As FIs and fintechs grapple with bias, another technology is bursting onto the scene with unknown implications for fair lending: generative AI.

“I think the question will be … when do we get to a point where we can trust that kind of AI, and we can explain it and we can ensure with our regulatory partners that we all trust it?” — Beth Johnson, chief experience officer, Citizens Bank

Commonly regarded as a separate branch of AI from the predictive algorithms used for underwriting, LLMs have been making headlines for their ability to produce long passages of human-sounding text and generate photos and videos from text prompts.

Experts agree that in the highly regulated banking industry, LLMs likely could never be directly involved in credit decisioning due to their unpredictability.

However, many FIs are exploring other use cases that could help improve fairness by making it easier for borrowers to access credit.

Varo Bank, a $564 million digital bank focused on lending to customers who have traditionally been underserved by the financial system, is experimenting with ways the technology could assist in “taking a customer through their financial inclusion journey,” Chief Technology Officer Sachin Shetty told BAN. This ranges from providing customers with instant customer support to designing personalized saving plans and motivating customers with personalized pep talks.

The $222 billion Citizens Bank is looking at similar use cases, though it is beginning with situations in which generative AI could help augment existing human employees, Chief Experience Officer Beth Johnson told BAN. Still, the bank and many of its peers are prioritizing caution.

“I think the question will be … when do we get to a point where we can trust that kind of AI, and we can explain it and we can ensure with our regulatory partners that we all trust it?” Johnson said.

‘A great transition’

Despite rapid advances in AI, many banks are still using “rudimentary” underwriting techniques that have the potential for human bias and yield less fair results, according to Saleh.

“Is there a lot of low-hanging fruit in terms of unfairness that we can remedy today, before we confront a lot of these existential issues that are posed by AI? One hundred percent.” — Kareem Saleh, co-founder and chief executive, FairPlay AI

So what’s stopping FIs from implementing the most sophisticated fairness technology available?

The investment necessary to overhaul fair lending practices can be a heavy lift, especially given current economic headwinds including high interest rates and falling consumer demand, according to Teddy Flo, chief legal officer at AI-powered credit model provider Zest AI.

“Replacing your underwriting system while you’re also dealing with capital challenges … it’s just a lot to ask for someone to do it all at once,” Flo told BAN.

Even so, recent statements from the CFPB indicate that regulators are in no mood to let FIs off the hook when it comes to bias in lending.

“The CFPB has been increasing its expertise in data science and analytics to ensure that we can identify fair lending violations at each stage of the credit lifecycle and hold creditors and service providers accountable for fully complying with fair lending and other federal consumer financial laws,” the agency noted in its annual report to the U.S. Congress last month.

But customers’ desire to be treated fairly may ultimately prove the strongest motivator for FIs to address their biases, Saleh said.

“We’re in the midst of a great transition … because increasingly consumers are demanding fairness from their governments, from their employers and from the brands that they patronize,” Saleh said.

“We can debate whether that transition is going to take place over two years or 10 years, but the animating belief of our business is that it will take place.”