Loraine Lawson:
Good day. This is Loraine Lawson, associate editor with Bank Automation News. Recently, I spoke with Stephen Thomas, who heads the Analytics and AI Ecosystem in the Smith School of Business at Queen University in Ontario, Canada. The Smith School of Business works with Scotiabank and other banks to help ensure AI systems are without bias. I asked Thomas about what work his program is doing with banks and helping to reach the underbanked or the unbanked.Stephen Thomas
yeah, so there’s a few efforts going on going on at Smith and Queens right now. And I’m involved with them. One is, in the related one is just in general seen are the bank’s current lending models biased in any way. And I, and this raises a sub questions like a How do you even measure bias? Like, if there’s a lot of devil in the details once you get started. But you know, as soon as you start defining protected attributes, or defining, you know, who’s part of a minority subgroup or something like that, you can start to make some progress. So it’s like, how do you detect it? Okay, and that lets us measure how good or bad it is, then how do you fix it, and we got a bunch of kind of irons in the fire trying to figure out how to fix it. One is on the, on the data side, like, okay, that’s, you know, the bias might be in the training data, historical training data, the lending decisions in the past two is we can, we can adjust the algorithms themselves to machine learning algorithms themselves to like basically penalize them for making, you know, but what what humans consider a bias decision, and so the algorithms will not do that. Or then three, we can leave the data alone and the algorithms alone and then we can just adjust the predictions themselves and we can adjust what’s called the probability threshold of the prediction, basically making it a little bit easier for minority subgroups to get alone versus at To adjust for historical bias, and so all of these methods have their pros and cons. And probably the in solution will be a combination of all three. So it’s. So it’s kind of a whirlwind on the technical side, the bank is planning to use this in many different ways. Some of which I might not even be aware of. But basically, they want to be aware that the problem exists, and they want to fix it now that it’s changed over time they started, when they first learned about this problem, they weren’t too excited. They’re like, Well, you know, we’re not doing anything illegal. And what you’re suggesting is going to cost us a lot of money. But then we, we eventually have come around to realize this is a good thing to do, and it won’t cost that much money. And we’re gonna have to do it anyway. So,Loraine Lawson
Do you work with a lot of banks, and in terms of helping them with their AI?Stephen Thomas
We work we work most closely with Scotia, because we have the Scotiabank center for customer analytics. But we have done surveys and talks with all the banks in Canada, all the big banks, just to kind of get a survey and the lay of the land and see who’s doing what. And we realized they were all pretty much the same. I mean, not exactly the same. But there wasn’t one who was way ahead of the others are like that.Loraine Lawson
Yeah. So had that they’ve been using? Or have you seen use cases where they’re using AI to sort of try to help people get loans that might not, you know, in their credit decisioning to help people get loans that might not typically be eligible for loans? Um,Stephen Thomas
good question. I mean, that’s the goal. That’s where we’re working towards has have any of the teams at Scotiabank, or any of the other banks actually done it? Like live and deployed it? I’m not sure. I’m not sure. But I might be. I don’t want to say No, they haven’t. Because I mean, these banks are huge, and they move fast.
So I just might not know.Loraine Lawson
Well, I’m just wondering. I’m wondering, you know, in traditional credit underwriting, we look at everybody looks up Experian, or whatever, and they look at your credit rating. So some of the what I’ve heard about is banks looking at using AI through develop different models that don’t rely on your credit rating that look at other things like, do you have a steady paycheck? If you have an account with us for five years, and you’re regularly depositing money? You know, that’s that sort of thing? Have you heard anything about that?

Stephen Thomas
Exactly? Yes. Yeah. Yeah. I mean, that’s what we’re finding, as well as the typical credit reports. They’re just, I mean, they use their own model to come up with those numbers. And those models are older, and they have the same flaws that we’re talking about. So if our models are based on the output of those models, then it’s not going to be very good. I mean, we’re basically just propagating the problem kicking the can down the road, basically. So yes, so that makes getting creative and looking at more behavioral attributes, like you said, payment frequency account balances. And, and there’s, it’s very interesting, and I could we could talk all day about it, there’s some debate as to whether you should use attributes like that are tied to somebody’s identity status. For example, should you use the fact that they are female, or what their gender? And so some advocates will say, No, no, no, that that’s, that ruins the whole point. You know, we don’t want to, we don’t want the album to to be biased, we want them to be blind. But what we’re finding is actually, it’s helpful for the model to be less biased, knowing what the gender is, because otherwise, I mean, due to historical, societal prejudices, you know, the an average woman might look worse than an average male for no reason other than that she’s a woman. So if the model doesn’t know that this person’s a woman, and they think, okay, these people were the same, and this one’s a little bit worse, so they’re not going to be as credit worthy. But if we tell the I don’t know this, if we tell them, you know, what the gender is, and they can account for that difference and basis. We make them equal. So there’s some debate they’re not debate, well, there’s some studies, they’re showing that it might be actually beneficial to keep the protected attributes in. It’s hard for some human beings to, to accept that fact, like, Well, that doesn’t make sense. Anyway, there’s and stop me if I’m ranting and raving too much, you’re fine dining topic. It is a great topic, where we’re going with this. Oh, yeah, so other kinds of attributes that are helpful. Another thing is, like zip code, or postal code kind of thing. You know, historically, postal codes are very highly correlated with income and, and long story short, whether the bank will give them a loan or not. So again, the, the initial reaction was, let’s remove postal code from the model. But now we’re finding Well, it actually, let’s keep it in there. So so that they know what like the baseline expectation is, like someone from a poor postal code. Yeah, maybe they’re only depositing $50 a month. But for them, that’s really good. And it’s a good sign. And compared to a affluent postal code is deposited $500, every month, it looks horrible, but the algorithm can adjust if they know that, for that postal code, this is actually really good behavior. And this person is credit worthy, that kind of thing.

We’re letting the model understand the background of the individual. And it’s kind of like, you know, that when you’re when you’re trying to, I’m going to come up with a really bad analogy on the fly here. Let’s say you’re trying to pick a sports to sports teams, from a mix of colleagues at work. So we got some people really athletic people who aren’t got big, strong guys, you got little girls, you got everything. And if you want to make the team’s fair, you need to know as much as you can about each person, you wouldn’t put all the males together and all the six year old girls together and expect that to be a fair soccer game, that kind of thing. But that doesn’t mean the six year old soccer, girls aren’t good at soccer and aren’t gonna have a great, you know, that they need their chance to shine as well. So yeah, maybe there’s a bad analogy, basically, the thing is, just because someone come from a poor background, or their their grandpa defaulted on a loan, or people their neighbors defaulted on loan, or they don’t make as much money, just because that’s true in general doesn’t mean that that person, that individual doesn’t deserve a chance and isn’t doing well. So by kind of robbing all the things, that another way to say it is, if someone hasn’t had the advantages of life, as more affluent person, if they didn’t get, you know, their mom and dad didn’t pay for them to go to Harvard, and they didn’t, they don’t then start with the greatest job. And they didn’t start with all this. Just because that’s true doesn’t mean they’re not responsible, loan worthy individuals. So if you so we need to look at more of their behavior, but also the behavior for their upbringing or the for their background. You can’t compare all behaviors to each other, because you’re going to get a lot it’s going to be heavily weighted towards Oh, you don’t make $100,000 a year and your parents don’t don’t have a million dollars in assets, well, then you’re no good, even though that kind of thing.

Loraine Lawson
Right? It makes sense, responsibility looks different for somebody making $50,000 a year than someone making $100,000 a year if you’re just looking at raw numbers.

Stephen Thomas
Right, exactly. Yeah, that’s a good point. So we’re almost it’s true to say that we’re changing the goalposts or changing the criteria for success in a way. But saying it that way makes it sound like we’re like cheating in a way. But basically, we’re just trying to, we’re letting the algorithm be fair, it’s not fair to blind the algorithm from the background and the circumstances of the individual.

Loraine Lawson
It reminds me of the, so I used to be a teacher. And it reminds me of the conversation we would have about equity versus equality where, you know, equality is, is giving everybody seen the same platform, but maybe somebody’s trying to watch a baseball game and they actually need a two foot stand versus the one foot stand you’re giving everybody because they’re short. Graduates, so equity is giving people what they need, rather than the same thing. Exactly.

Stephen Thomas
I was the director of one of our master’s degree program a few years ago. And we came across the same thing really, which was, you know, we have a bunch of students apply. And some students have great resumes, you know, went to U of T straight A’s, got a great job, all this stuff. And other students, you know, they’re from a poor family pretty bad not having now, not a great college that they went to like, they have great reference. But first, you think, oh, let’s just let him the people the best grade from the best schools. But what were but that is unfair, because there are strong, smart, capable individuals who just need that chance to shine. And they’ve done well, given what they are given. It doesn’t look good compared to everyone else. But for them, it’s great. And you know, once they’re in, and it’s true, some of our best students have come from non traditional backgrounds. And so basically, that’s kind of what we’re doing in the banking world as well, who’s trying to learn with these models, kind of the same ideas. There’s not like a single number, like number of deposits or amount that you need to make each month that is good. depends on your background. So we have to give the background information to the model for that to them.

Loraine Lawson
So are there best practices that you’ve evolved in terms of this yet? It sounds like this is a still evolving field.

Stephen Thomas
Yeah, it’s brand new, brand new, it’s like more we go deeper, we dive into it, the more we realize we haven’t solved yet, which is good. I mean, at least we’re looking into it, best practices are. One is to just be aware that this is an issue that if you don’t, if you’re not paying attention, you give your standard training your standard training data to your standard model and make standard predictions, it’s going to be unfair, they’re going to be unfair, almost guaranteed. So that realization, number one that this and more and more organizations are having this that, okay, that’s not good enough anymore. So, best practice number two is to actively measure how biased or unfair your predictions are. And there’s some methodologies out there. But basically, you want to look at the false positive or false negative rates of your model for each protected subclass. For example, if your model was really good at predicting which affluent white males should get along, but was really bad at predicting which African women should be getting alone, that’s bad. That means it can make a lot of false positive and false negatives on the African women. So it’s going to basically be denying African women rates, loans that they should be giving out at a higher rate. So, so the model has has is wrong. So if you kind of build this in the process to measure and quantify these things for each protected subgroup, I think it Canada, there’s 13, at least 13, or maybe 17, you should be looking at, then that’s step number one, at least, you know, what, what’s going on? And then how to fix it. I don’t know if there’s any best practices that are general enough to apply to everybody? It depends on a lot of on, it depends on a lot on what data they’re using, what algorithm they’re using, and how its deployed. But there are some, you know, we’re building a body of knowledge bigger and bigger every day on how to how to address the issues on the technical side.

Loraine Lawson
So so far, no, no great success stories. We’re still working on it. Is that where we are?

Stephen Thomas
In the lab, in our like, simulations and our tests, the really good success stories, like will we’re finding 10s of 1000s of loans that were denied that should not have been, for example, and you know, this has huge potential impact to society. masses, like there’s a societal aspect of this to you know, it’s like, yes, it’s interesting. There’s many ways you can look at this, there’s, like, from the bank’s point of view, lost revenue that they’re making, or that they’re, they’re missing out on from the, from the individuals point of view, it’s, you know, customer lifetime value and all this stuff. And then from a society point of view, you know, this is great, you know, spurring the economy, it’s giving solid, responsible people that boosts in that, that boost they need for that loan that house.

Loraine Lawson
And you have a certification, right for AI. Is that right?

Stephen Thomas
Yeah, we have a exec ed program. I think that’s what you’re thinking of. called trusted AI. And it’s a partnership with I triple E, which is a leading Engineering Institute. And yeah, so we’ve delivered that a few times. And, you know, we talked about all of these issues, you know how to how to measure, you know, the whole area of ethics is, is, is really taking off. It’s, it’s, it’s a really good thing that organizations are starting to realize this. But it’s also very scary, because we realize that no one knows what they’re doing. Yes. And, you know, there’s machine learning algorithms out there running every single day, that aren’t totally guaranteed 100% biased and unethical. And people just don’t even realize it. They didn’t even know to look out for it earlier. So it’s a huge problem, but at least we know that it’s out there. But with with regulations, like GDPR, from Europe, does a huge, good first step. I know Ontario is thinking of similar things, maybe as soon as this year putting into place. And even Europe is going to tighten it even more. So there’s a lot of news, California is moving fast on this as well. So there’s a lot happening, we were living right in the middle, or maybe like towards the beginning. So things change rapidly every single day. I think in five or 10 years, we’ll look back and think, Wow, I can’t believe us do it that way. But right now, people that people are still very disorganized and trying to figure out what to do.

Loraine Lawson:
Good day. This is Loraine Lawson, associate editor with Bank Automation News. Recently, I spoke with Stephen Thomas, who heads the Analytics and AI Ecosystem in the Smith School of Business at Queen University in Ontario, Canada. The Smith School of Business works with Scotiabank and other banks to help ensure AI systems are without bias. I asked Thomas about what work his program is doing with banks and helping to reach the underbanked or the unbanked.Stephen Thomas
yeah, so there’s a few efforts going on going on at Smith and Queens right now. And I’m involved with them. One is, in the related one is just in general seen are the bank’s current lending models biased in any way. And I, and this raises a sub questions like a How do you even measure bias? Like, if there’s a lot of devil in the details once you get started. But you know, as soon as you start defining protected attributes, or defining, you know, who’s part of a minority subgroup or something like that, you can start to make some progress. So it’s like, how do you detect it? Okay, and that lets us measure how good or bad it is, then how do you fix it, and we got a bunch of kind of irons in the fire trying to figure out how to fix it. One is on the, on the data side, like, okay, that’s, you know, the bias might be in the training data, historical training data, the lending decisions in the past two is we can, we can adjust the algorithms themselves to machine learning algorithms themselves to like basically penalize them for making, you know, but what what humans consider a bias decision, and so the algorithms will not do that. Or then three, we can leave the data alone and the algorithms alone and then we can just adjust the predictions themselves and we can adjust what’s called the probability threshold of the prediction, basically making it a little bit easier for minority subgroups to get alone versus at To adjust for historical bias, and so all of these methods have their pros and cons. And probably the in solution will be a combination of all three. So it’s. So it’s kind of a whirlwind on the technical side, the bank is planning to use this in many different ways. Some of which I might not even be aware of. But basically, they want to be aware that the problem exists, and they want to fix it now that it’s changed over time they started, when they first learned about this problem, they weren’t too excited. They’re like, Well, you know, we’re not doing anything illegal. And what you’re suggesting is going to cost us a lot of money. But then we, we eventually have come around to realize this is a good thing to do, and it won’t cost that much money. And we’re gonna have to do it anyway. So,Loraine Lawson
Do you work with a lot of banks, and in terms of helping them with their AI?Stephen Thomas
We work we work most closely with Scotia, because we have the Scotiabank center for customer analytics. But we have done surveys and talks with all the banks in Canada, all the big banks, just to kind of get a survey and the lay of the land and see who’s doing what. And we realized they were all pretty much the same. I mean, not exactly the same. But there wasn’t one who was way ahead of the others are like that.Loraine Lawson
Yeah. So had that they’ve been using? Or have you seen use cases where they’re using AI to sort of try to help people get loans that might not, you know, in their credit decisioning to help people get loans that might not typically be eligible for loans? Um,Stephen Thomas
good question. I mean, that’s the goal. That’s where we’re working towards has have any of the teams at Scotiabank, or any of the other banks actually done it? Like live and deployed it? I’m not sure. I’m not sure. But I might be. I don’t want to say No, they haven’t. Because I mean, these banks are huge, and they move fast.
So I just might not know.Loraine Lawson
Well, I’m just wondering. I’m wondering, you know, in traditional credit underwriting, we look at everybody looks up Experian, or whatever, and they look at your credit rating. So some of the what I’ve heard about is banks looking at using AI through develop different models that don’t rely on your credit rating that look at other things like, do you have a steady paycheck? If you have an account with us for five years, and you’re regularly depositing money? You know, that’s that sort of thing? Have you heard anything about that?

Stephen Thomas
Exactly? Yes. Yeah. Yeah. I mean, that’s what we’re finding, as well as the typical credit reports. They’re just, I mean, they use their own model to come up with those numbers. And those models are older, and they have the same flaws that we’re talking about. So if our models are based on the output of those models, then it’s not going to be very good. I mean, we’re basically just propagating the problem kicking the can down the road, basically. So yes, so that makes getting creative and looking at more behavioral attributes, like you said, payment frequency account balances. And, and there’s, it’s very interesting, and I could we could talk all day about it, there’s some debate as to whether you should use attributes like that are tied to somebody’s identity status. For example, should you use the fact that they are female, or what their gender? And so some advocates will say, No, no, no, that that’s, that ruins the whole point. You know, we don’t want to, we don’t want the album to to be biased, we want them to be blind. But what we’re finding is actually, it’s helpful for the model to be less biased, knowing what the gender is, because otherwise, I mean, due to historical, societal prejudices, you know, the an average woman might look worse than an average male for no reason other than that she’s a woman. So if the model doesn’t know that this person’s a woman, and they think, okay, these people were the same, and this one’s a little bit worse, so they’re not going to be as credit worthy. But if we tell the I don’t know this, if we tell them, you know, what the gender is, and they can account for that difference and basis. We make them equal. So there’s some debate they’re not debate, well, there’s some studies, they’re showing that it might be actually beneficial to keep the protected attributes in. It’s hard for some human beings to, to accept that fact, like, Well, that doesn’t make sense. Anyway, there’s and stop me if I’m ranting and raving too much, you’re fine dining topic. It is a great topic, where we’re going with this. Oh, yeah, so other kinds of attributes that are helpful. Another thing is, like zip code, or postal code kind of thing. You know, historically, postal codes are very highly correlated with income and, and long story short, whether the bank will give them a loan or not. So again, the, the initial reaction was, let’s remove postal code from the model. But now we’re finding Well, it actually, let’s keep it in there. So so that they know what like the baseline expectation is, like someone from a poor postal code. Yeah, maybe they’re only depositing $50 a month. But for them, that’s really good. And it’s a good sign. And compared to a affluent postal code is deposited $500, every month, it looks horrible, but the algorithm can adjust if they know that, for that postal code, this is actually really good behavior. And this person is credit worthy, that kind of thing.

We’re letting the model understand the background of the individual. And it’s kind of like, you know, that when you’re when you’re trying to, I’m going to come up with a really bad analogy on the fly here. Let’s say you’re trying to pick a sports to sports teams, from a mix of colleagues at work. So we got some people really athletic people who aren’t got big, strong guys, you got little girls, you got everything. And if you want to make the team’s fair, you need to know as much as you can about each person, you wouldn’t put all the males together and all the six year old girls together and expect that to be a fair soccer game, that kind of thing. But that doesn’t mean the six year old soccer, girls aren’t good at soccer and aren’t gonna have a great, you know, that they need their chance to shine as well. So yeah, maybe there’s a bad analogy, basically, the thing is, just because someone come from a poor background, or their their grandpa defaulted on a loan, or people their neighbors defaulted on loan, or they don’t make as much money, just because that’s true in general doesn’t mean that that person, that individual doesn’t deserve a chance and isn’t doing well. So by kind of robbing all the things, that another way to say it is, if someone hasn’t had the advantages of life, as more affluent person, if they didn’t get, you know, their mom and dad didn’t pay for them to go to Harvard, and they didn’t, they don’t then start with the greatest job. And they didn’t start with all this. Just because that’s true doesn’t mean they’re not responsible, loan worthy individuals. So if you so we need to look at more of their behavior, but also the behavior for their upbringing or the for their background. You can’t compare all behaviors to each other, because you’re going to get a lot it’s going to be heavily weighted towards Oh, you don’t make $100,000 a year and your parents don’t don’t have a million dollars in assets, well, then you’re no good, even though that kind of thing.

Loraine Lawson
Right? It makes sense, responsibility looks different for somebody making $50,000 a year than someone making $100,000 a year if you’re just looking at raw numbers.

Stephen Thomas
Right, exactly. Yeah, that’s a good point. So we’re almost it’s true to say that we’re changing the goalposts or changing the criteria for success in a way. But saying it that way makes it sound like we’re like cheating in a way. But basically, we’re just trying to, we’re letting the algorithm be fair, it’s not fair to blind the algorithm from the background and the circumstances of the individual.

Loraine Lawson
It reminds me of the, so I used to be a teacher. And it reminds me of the conversation we would have about equity versus equality where, you know, equality is, is giving everybody seen the same platform, but maybe somebody’s trying to watch a baseball game and they actually need a two foot stand versus the one foot stand you’re giving everybody because they’re short. Graduates, so equity is giving people what they need, rather than the same thing. Exactly.

Stephen Thomas
I was the director of one of our master’s degree program a few years ago. And we came across the same thing really, which was, you know, we have a bunch of students apply. And some students have great resumes, you know, went to U of T straight A’s, got a great job, all this stuff. And other students, you know, they’re from a poor family pretty bad not having now, not a great college that they went to like, they have great reference. But first, you think, oh, let’s just let him the people the best grade from the best schools. But what were but that is unfair, because there are strong, smart, capable individuals who just need that chance to shine. And they’ve done well, given what they are given. It doesn’t look good compared to everyone else. But for them, it’s great. And you know, once they’re in, and it’s true, some of our best students have come from non traditional backgrounds. And so basically, that’s kind of what we’re doing in the banking world as well, who’s trying to learn with these models, kind of the same ideas. There’s not like a single number, like number of deposits or amount that you need to make each month that is good. depends on your background. So we have to give the background information to the model for that to them.

Loraine Lawson
So are there best practices that you’ve evolved in terms of this yet? It sounds like this is a still evolving field.

Stephen Thomas
Yeah, it’s brand new, brand new, it’s like more we go deeper, we dive into it, the more we realize we haven’t solved yet, which is good. I mean, at least we’re looking into it, best practices are. One is to just be aware that this is an issue that if you don’t, if you’re not paying attention, you give your standard training your standard training data to your standard model and make standard predictions, it’s going to be unfair, they’re going to be unfair, almost guaranteed. So that realization, number one that this and more and more organizations are having this that, okay, that’s not good enough anymore. So, best practice number two is to actively measure how biased or unfair your predictions are. And there’s some methodologies out there. But basically, you want to look at the false positive or false negative rates of your model for each protected subclass. For example, if your model was really good at predicting which affluent white males should get along, but was really bad at predicting which African women should be getting alone, that’s bad. That means it can make a lot of false positive and false negatives on the African women. So it’s going to basically be denying African women rates, loans that they should be giving out at a higher rate. So, so the model has has is wrong. So if you kind of build this in the process to measure and quantify these things for each protected subgroup, I think it Canada, there’s 13, at least 13, or maybe 17, you should be looking at, then that’s step number one, at least, you know, what, what’s going on? And then how to fix it. I don’t know if there’s any best practices that are general enough to apply to everybody? It depends on a lot of on, it depends on a lot on what data they’re using, what algorithm they’re using, and how its deployed. But there are some, you know, we’re building a body of knowledge bigger and bigger every day on how to how to address the issues on the technical side.

Loraine Lawson
So so far, no, no great success stories. We’re still working on it. Is that where we are?

Stephen Thomas
In the lab, in our like, simulations and our tests, the really good success stories, like will we’re finding 10s of 1000s of loans that were denied that should not have been, for example, and you know, this has huge potential impact to society. masses, like there’s a societal aspect of this to you know, it’s like, yes, it’s interesting. There’s many ways you can look at this, there’s, like, from the bank’s point of view, lost revenue that they’re making, or that they’re, they’re missing out on from the, from the individuals point of view, it’s, you know, customer lifetime value and all this stuff. And then from a society point of view, you know, this is great, you know, spurring the economy, it’s giving solid, responsible people that boosts in that, that boost they need for that loan that house.

Loraine Lawson
And you have a certification, right for AI. Is that right?

Stephen Thomas
Yeah, we have a exec ed program. I think that’s what you’re thinking of. called trusted AI. And it’s a partnership with I triple E, which is a leading Engineering Institute. And yeah, so we’ve delivered that a few times. And, you know, we talked about all of these issues, you know how to how to measure, you know, the whole area of ethics is, is, is really taking off. It’s, it’s, it’s a really good thing that organizations are starting to realize this. But it’s also very scary, because we realize that no one knows what they’re doing. Yes. And, you know, there’s machine learning algorithms out there running every single day, that aren’t totally guaranteed 100% biased and unethical. And people just don’t even realize it. They didn’t even know to look out for it earlier. So it’s a huge problem, but at least we know that it’s out there. But with with regulations, like GDPR, from Europe, does a huge, good first step. I know Ontario is thinking of similar things, maybe as soon as this year putting into place. And even Europe is going to tighten it even more. So there’s a lot of news, California is moving fast on this as well. So there’s a lot happening, we were living right in the middle, or maybe like towards the beginning. So things change rapidly every single day. I think in five or 10 years, we’ll look back and think, Wow, I can’t believe us do it that way. But right now, people that people are still very disorganized and trying to figure out what to do.