Jaspreet Kalra
Welcome to The Buzz presented by Bank Automation News. I’m your host Jaspreet Kalra, and in this week’s edition, we dive into the discussion surrounding AI driven credit underwriting and whether it exposes customers to undue burdens. How does one minimize bias in AI models, and whether it’s too soon to hand over the underwriting reins to machine learning.

I spoke with Professor Janine Hiller, who teaches finance and business law at the pamplin College of Business at Virginia Tech. Professor halos research work focuses on the intersection of law, ethics and technology. She recently filed response to requests for public input on AI use in financial services put out in April by the Federal Reserve, OCC and three other regulatory bodies. We spoke about how legal ideas of fairness compared with mathematical notions of it for a social technical approach to regulation would look like. And what her advice to banks or credit unions considering using AI for credit underwriting would be before we get started and get into the meat of your response to the federal regulators request for information on AI using financial services, would it be possible for you to briefly introduce yourself? And give us an overview of what prompted you to respond to the request for information?

Janine Hiller
Well, first of all, thank you for having me. This is a really important topic. So I’m happy that you were are talking about it further. I’m a professor of business law at Virginia Tech. And my background is in law and technology and ethics. And we’ve been doing some work at the college and the University on AI and ethics and the decision making that’s going on with regards to AI. And so having done some research and published in areas regarded related to credit and use of AI and credit, I felt compelled to make a few comments and offer a few observations in the process.

Jaspreet Kalra
So one of the points you raise within your people are these two distinct ideas that one is the legal idea of fairness. And then there’s the mathematical idea of fairness, or like how mathematical ideas of fairness can be shaped? Could you tell us a little bit more about that? And how does one begin to sort of compare and possibly reconcile those?

Janine Hiller
Sure, that’s a really hard question to respond to. And even in the paper that I submitted, it was just an introduction to it. But briefly, you know, when we’re talking about AI, we’re talking about using mathematical constructs to make decisions. And so they are focused to a large degree in using AI and algorithms on this mathematical representation of what they believe to be fairness. And so a lot of the focus is on the results, there’s an AI model that then has results and then they look backwards to see if those results may be fair based upon either group fairness, or individual fairness, statistical methods, you know, mathematical outcomes, whether there’s, you know, the accuracy with regards to error rates, there are at least 24, as I’ve seen it in that literature, definitions of fairness, with regards to mathematics. So that’s why I guess I’m fumbling around with describing what it is concisely, because the discipline is, is fumbling with it as well. But basically, it’s that, you know, it’s an external force that’s imposed upon them. It’s not something that might be considered originally in what the algorithm is coming up for, but it is looking back on it hasn’t been fair. And legal fairness has to do with a broader concept of both process and substance. It’s more, it’s messy, as people have described it. It’s not a mathematical result. That is the numbers are fair. You know, it has to do with things of process with regards to whether things have been transparent. People know what’s going on, whether people have an opportunity to participate in this process, and whether the procedures are fair. And then the substantive fairness that we talked about in many, many different ways in the law can vary. But but the highlights of those are that if you have people with very different power levels, and people don’t have choice and they have highly unfavorable terms. To them in contracts, we call that to be the unconscionability or unfairness. And we look at the justification for the law with regards to such substantive fairness, the power over the individual and whether the the the injuries are able to be avoided in an administrative viewpoint versus the benefit. And and basically, it comes back down to the fact that in law, this messy look at what is fair recognizes that in society, individuals don’t have a lot of power in certain circumstances. So when we give the power of the government over an individual, is there legitimacy to that? Is it legitimate? power? Is it? Is it going is the individual eventually going to have trust overall in the system? That that’s a hard first question to start with. That, that some of that makes the

Jaspreet Kalra
right No, certainly, I mean, I think it builds on to this idea of complexity in AI systems as well. And how transitioning over towards these sort of statistical models that do, or at least aim to do sounds of credit underwriting also imposes these additional burdens. And that’s sort of builds on to my next question here, which is, if, you know, we see a larger shift towards AI driven credit underwriting, how do you sort of see the burden shifting towards customers? Sort of in understanding how these models work? How is their credit being granted or denied? And would it also, by extension, sort of make it harder for them to put up a case that says, I might have been discriminated against by this statistical model that I don’t really fully understand?

Janine Hiller
Yes, you’re really, you’re correct, that it’s going to be very difficult for the consumer. And my, my opinion is that it’s impossible for the consumer to really say that they have been discriminated against or even to understand it, I mean, those of us who work in the field, and we’re trying to cross disciplines, try to understand each other. So I think that consumers really have no control. They don’t have any control over the data that’s being collected about them. And, and that’s the input into these really broader AI models for credit scoring and credit underwriting. It goes beyond some financial data, it goes into personal data with regards to even social media use. And you know, these systems are opaque, they are protected as one of the things I think you will ask later about IP, about property rights, they’re protected as property rights, they’re opaque. And that’s what leads me to suggest you know, that there are consumer protection regulations that are needed, because the consumer really can’t understand the method. They don’t have a way to protect themselves, or, and they know what they don’t. It’s not as if it’s a marketplace, they can’t take their data, their business elsewhere. The choice is illusory, right? If they don’t have a choice, they can’t say, Well, I don’t like this model. So I’m going to go this model, and then the markets going to decide which model is better, and which business wins. So the that that fails in terms of a regulatory approach by the market, I believe.

Jaspreet Kalra
Right, that makes sense. And sort of jumping on to the related aspect here, which in that the impression that I’m getting from our conversation so far is that there is like, say, 24 different definitions of fairness within mathematics. And then you have this broader legal spectrum of definitions you’re in as well. So in your estimation, do you think the jump towards AI driven credit underwriting to an extent is premature right now? And is there any merit to sort of saying, you know what, maybe let’s slow down and study this further, before we actually introduce this into the marketplace?

Janine Hiller
I think there’s definitely some merit in that. But there’s also I understand and recognize truly, that there is some merit in trying new things. And one might say using sandboxes, to try new ideas out. I’d add to that I’d add to that comment is that I think it’s not just the models that are mature, but it’s our structure for under for using them in society. And therefore, how do we regulate them to also image sure because there are technology firms that don’t believe in regulated by the fair lending laws that we already have and have had for decades. And so, because and I would also add because new credit for individuals and I’m talking about individuals and consumers here, right? That is Such a doorway to all of the different aspects of of life as we know it. access to credit in our country is hugely important for individuals. And it also affects affects things like insurance or jobs. And so because of its importance to individuals, I think that some caution needs to be taken, not perhaps complete, stepping away from it, but some caution needs to be taken. And we need to really look at it carefully, and broaden our concept of who’s who’s covered by our fair lending laws.

Jaspreet Kalra
Right. And one of the ideas that you also talk about within your paper is that there needs to be a social technical approach to regulating these models wherein you accommodate for social elements and all the technical understandings of fairness, could you sort of help us understand that better? And how do you think that will work out?

Janine Hiller
Right, well, I’m going to use a very simple example that happened years ago, in a medium size city in the US, there is a river in which they needed to, if you just give me allow me a little bit of patience, because it’s not about data, in which they wanted to cross the river, this big river, right? traffic was horrible. So the engineers said, Okay, tell me where to start on this side, where to get to the other side, we want a straight bridge. But people who lived and this is a true story, people who lived on each side of the bridge said we don’t want to put a sigh, they said, We want to hear sigh B said we don’t want it on this side straight over. Because that’s going to disrupt our community, it’s going to divide this community in half. And so the engineer said, Look, guys decide tell us we want a straight edge, what they really ended up doing is, it’s this beautiful bridge where you start on one side where side A wanted it. And then there’s this gentle curve that takes it to this other side, it’s not straight across on side B. And that’s a socio technical system. It’s an engineer who says we can build this bridge, compared to AI who says we can we can give you this prediction. And yet, if you don’t involve the communities and societies were being affected by it, you’re not going to have a result that everybody wants. That means that both sides have to talk with each other, the engineers or the data, scientists might not be able to do things the way they would normally do, though. It means that in society, we realize that in order to get the benefits of this, this the great benefits that can come from it and the efficiencies that we can get in our credit system, we might have to impose some possible risks. Or it might require consumers as you implied earlier, perhaps they need to take some more steps that we can clearly set out for them to make sure it’s being done right. I hope that’s not gone too far afield. But But what it ultimately means is that this is, and this has been applied in many different situations and organizations and systems. It’s people, it’s society, it’s technology, that’s all intertwined, if you can’t make these decisions separately, for these to work well. And so the social as well as the technical have to be considered and hopefully really considered at the design point, not afterwards. But how are we designing these systems with the social goals in mind, not just going back after you have the results to see if you can change them to be acceptable?

Jaspreet Kalra
Right. And sort of one hurdle that I’ve seen emerge when it comes to engagement with Be it regulators be society is that companies want to protect some of their AI models as proprietary as, like sort of a competitive edge. How do you think that will play into when regulators want a deeper look into their model to check for fairness or for other things? Is that going to be like an uphill climb for regulators? Or do you see is the firm’s being more willing to engage and more willing to share?

Janine Hiller
Well, I think that firms are not monoliths, and certainly firms have certain firms have said we want to be ethical about what we do. And we’re going to accept ethical and proposed ethical principles going forward. But I would say that, to the extent that businesses simply put this argument forward, I think it’s a red herring. It’s not anything new. We we’ve met these objections many times before if we think about the pharmaceutical industry there Based on intellectual property, and yet, their products have to be reviewed for safety. We have methods to do that. And so I know that this is an argument that somehow but there are methods by which one’s product can be reviewed without losing the value of their, of their intention, so to speak.

Jaspreet Kalra
Right. And one last thing that I want to touch upon here is that sometimes one of the arguments that pops up is that you can either optimize for efficiency or you can optimize for fairness, do you think that’s an either or choice or a coexistence between those two is possible and doable? Well, I do have a question about that. Do you mean accuracy? Or if we’re sorry, about accuracy? Yeah, accuracy and sort of fans.

Janine Hiller
And I think you have to recognize that they’re really interrelated, there is argument within the data community that you can’t have both, you have to have one or the other, and you have to make a choice. And it’s up to the regulators to tell the data scientists where to make that choice. I would disagree with that. Number one, there are new concepts of causality, fairness, which is fairly new for that discipline that change that. And I think this also relates back to the question asked the very beginning the guards with the difference between know Fairness and Accuracy. Legally, and socially, we we feel like if it’s inaccurate, it’s unfair. So the concepts aren’t totally unrelated. And I think we should recognize that. And if one defines the goals at the beginning, versus at the end, I think that that is important.

Jaspreet Kalra
So one last thing here. So if you know there’s a financial institution or a credit union listening to this, and they’re wondering whether they should use an AI driven credit underwriting model or not, what are like a couple of things that you’d ask them to check for, as just pieces of advice from someone who’s studied this area? extensively?

Janine Hiller
I think I’d say two things. One is to do an extensive due diligence on the company and product, which they would be using.

Unknown Speaker
There are

Janine Hiller
many out there, and they approach this issue quite differently. We know from some of the congressional hearings, that that is true. So definitely due diligence separately is that you are as a bank or a credit union, you are trusting that entity. And you’re trusting that entity with your reputation, and your relationship with your customers. That’s so important. So I would be cautious, as we said before, and I think that it takes more due diligence than ever before, just a promise that the particular approach does not violate any laws and that it’s compliant is probably not enough. You need to look beyond that. And really investigate how they’re, they’re, they’re approaching this.

Jaspreet Kalra
You’ve been listening to the buzz from Bank automation news, please visit us on bank automation news.com for news and analysis of how automation is shaping financial services. You can also follow us on LinkedIn and Twitter. And please do rate this podcast on your preferred listening platform. Thank you for your time.