Loraine Lawson
Good day and welcome to The Buzz, a Bank Automation News podcast. I’m Deputy Editor Loraine Lawson. In today’s episode, we’re joined by David Van Bruwaene, who is the founder and CEO of Fairly.AI. Fairly.AI is an Artificial Intelligence governance, risk and compliance solution that automates model risk management. The company specializes in the financial services, healthcare, HR and education sectors, and recently won the UBS’ Global Future of Finance Challenge. We discussed the problems in removing bias from AI models but first I asked Van Bruwaene to explain common use cases for Fairly.AI in the banking industry.

David Van Bruwaene
So we are most of most interest right now in more customer facing application. So if you have a mortgage origination model, that can take into account a number of factors that could lead towards biased outcomes. And progressively more information is being provided on individual applicants for a mortgage or loan or anything really, where you are doing credit worthiness decisioning. Including something like his zip code, for example, can cause a model to make a prediction that’s biased against people of certain ethnicities who tend to live in that zip code, even though it’s not directly represented. Our system applies the same traditional model risk management techniques that have been in place since the boat 2011, or following the financial crisis. And the framework of champion challenger models. So you, you can be building many traditional models, and also trying to include machine learning models in the mix. And then comparing the performance and ultimately making a decision about which models to put into production. So we, we handle models at a pretty high level, but also have tooling to integrate directly with machine learning.

Loraine Lawson
Okay, so you’re looking for, for bias or potential ethical issues with the models? Are there particular mistakes that you see companies make when they’re trying to create these models or using the data or just things that you see that are common?

David Van Bruwaene
Well, the most common mistake when dealing with bias is to assume that you can just remove any of the obvious input features like race, age, gender from the model, and expect that this will lead to a model that’s not biased. It’s a way of providing some reassurance throughout management that nothing untoward is happening, how could it be biased in the first place, but nonetheless, there’s a concept of proxy values where other input features really can represent the protected class information, and then lead to biased results. And by removing this information, and not making it available to data scientists or model validators, there’s no opportunity to identify it. And there’s no opportunity to make algorithmic moves to correct for that bias, because there there’s quite a lot available in modern machine learning techniques to actually alter the decisioning process of the model. In order to lessen the effect or impact of those proxy values.

Loraine Lawson
You give an example of a proxy value that might exist, even though you’ve removed identifiers or what we think of as identifiers.

David Van Bruwaene
So I mean, a very classic one is zip code. But there’s also employment gaps, for example, that apply to women when they’re pregnant, that can be detected, and then subsequent lower income. This kind of information gets picked up very easily. That’s a very Easy pattern for a model to represent. And if you do not have a means of identifying that a particular data set has a certain representation of women that’s different from men, then you have no, no real options. To correct it. Another major problem is not at very basic is not establishing enough data to represent each of the groups that you’re legally required not to be discriminatory towards. And so the models underperform for certain groups, and sometimes it could benefit them and sometimes make it worse. But generally speaking, the the goal that is pretty well understood within the academic community, as well as legal community is that you want the most accurate performance for each and every group as a kind of baseline. And then being able to make decisions about say, giving some disadvantaged groups and additional advantage over other groups is one that needs to be made very intentionally.

Loraine Lawson
So is it better to leave identifying data in in the data so that you can

David Van Bruwaene
in certain cases, actually, leading to more to fair outcomes, there’s a huge number of papers showing this in certain cases, or that this identifying data is less significant to the models than you might think. But there’s other pieces of information that becomes more significant. So in the case of a pregnancy leave, you might say, if if you look at it, the gap could be significant. The employment gap could be significant. But the gender doesn’t turn out to be that significant when you test the model for feature sensitivity, but generally speaking, I think most, most applications would not include that protected class information. And that does raise a lot of civil liberties issues. That cause for good reason. consternation around its use, but it can still be collected and used to test the models before they’re used in production, and then spot check while they’re in production is.

Loraine Lawson
Yeah, so I guess if the data is strict of identifiers, you can’t tell them if, if things like you can’t tell the if the problems are connected to the class, right? That’s right. Okay. There’s,

David Van Bruwaene
there’s also very popular technique where it’s available, where you it’s called adversarial D biasing where you actually train a model to predict one of the input features that you don’t want to be significant to it. And then you at the same time, change the way it operates to make it unable to predict. So it’s it’s a, it’s difficult to describe algorithmically and mathematically but relatively simple concept of if you make the model, if you train the model to be bad at predicting that someone is of a certain race, then you know that it’s not using that information elsewhere.

Loraine Lawson
Interesting. So it’s pretty complex science that you’re dealing with you’re what what do you sort of advise your clients coming into the process? How do they prepare for de-biasing?

David Van Bruwaene
So it always begins with setting up the requirements that you’re going to place on your models before you’re going to allow them into production use. Coming up with the set of quantitative and qualitative reporting requirements, and then providing this information to those We’re building the model as soon as possible so that they can take the right steps from the start to build models that are ultimately gonna comply with the requirements. As models are being built, it’s a very, it’s often usually maybe always a very kind of messy, iterative process, you try something, it doesn’t work, you try something else, maybe read a paper, figure out a new kind of technique, change the data, structure a bit and see what works. So as people are, are going through this process, having a clear set, having a goalpost effectively, that’s agreed upon between teams that are doing the building in the validating, is absolutely crucial. And this can get lost in the mix. It always seems to get lost in the mix. In all of the banks that we’re working with, that we have spoken to, very kind of manual process of emails and meetings and kind of, after the fact being told to make changes to report on something. And that’s no longer available. So it causes a lot of chaos. As as these requirements become more defined, and and become used, with additional updates to models, becomes a lot easier to streamline the reporting requirements, but also to be building the models always towards very high standards.

Loraine Lawson
And those standards, are they defined per project? Or do you have standards that you sort of use across the board.

David Van Bruwaene
So there’s a very general framework that we’ve built into our system. And that’s framework around SRL 11 Seven, which is a specific requirement from the Federal Reserve, and applies to a huge range of models that you wouldn’t even think might be applicable in banking. So customer support models, for example, chat bots. So huge range, models fall under that framework, where ultimately, you, you need to apply what are very kind of reasonable processes, if you understand how they’re built, you have to be building alternative alternate models, called challenger models, select a champion amongst them, compare the performance results, provide ample support, that your models are performing up to the specifications that are required, these are normal kind of statistical measures that are now standard, but also that you’ve used enough suitable test data, and describe that test data and ultimately provide to those to a separate function in the bank, the validation function, all the materials that they need to go through that same process of building models, testing them coming to the same conclusions that you did. And then at that point, they’re able to say the one that you’ve selected, does meet our criteria and is able to go out the door. That’s very general, and and broadly applicable, the specifics of each model will differ. And so that’s where having a more technical integration into the model, building platforms, allows us to service a huge range of types of models, as well, I mean,

Loraine Lawson
sorry, what’s your technical integration?

David Van Bruwaene 19:25
So concretely, let’s say you’re building a model in Python, you’re using Kerris and its classification algorithm. And it takes binary inputs. And then you’re going to build it and have a bunch of selections about how it should operate. That you tried different kind of out of the box selections like lol or L to. Colonel it’s, it’s something that’s become very standardized, but there’s just a huge different huge array of ways you might build a model, we’re able to adapt to whichever out of a large number of standard libraries are in use, and a large number of selections and manage all of them within our service. And then the qualitative information that’s required to be reported on and maintained and have an audit trail behind it approvals and so on. That is something that we have a number of templates out of the box effectively. But also, we have an editor so that you can make whichever specific alterations you want for a particular model that lies within the line of business, or, or with a particular team.

Loraine Lawson
Okay, and you said, there are ways that banks and credit unions can stay ahead of a regulations, what do you recommend?

David Van Bruwaene
Well, I recommend, first, when you’re thinking about how you’re going to build your models, what platforms you’re going to use, how you’re ultimately going to bring them into production, you may think about it as being driven by compliance. First, with there’s already regulations in a number of sectors. And the standard ones that have applied to statistical models are things like the Fair Lending Act, there’s, there’s a lot out there, that’s not new. And we do have a pretty good idea now of what the next moves are going to be, especially in Europe, where the AI regulations proposals already been made public. And I think there’s a lot of things that are pretty already pretty well set that you will have to be able to test for bias biases, a huge one that comes up, you have to test for things like potential for financial loss, reputational risk, and so on. But with AI, the things that you need to test for and provide are now pretty standard explainability data drift was just means that your models gonna keep working, even if the data changes. There’s a pretty standard set of tools around explainability that people have been using, and would for the foreseeable future meet minimum kinds of requirements. So by integrating a system that does compliance alongside the model platforms that you’re choosing, from the start, you get a very big leg up.

Loraine Lawson
So you mentioned there are tools, standard tools for explainability. What are those? Is that something you fairly AI does or is are there separate tools,

David Van Bruwaene
we we offer a validation suite, which includes easy to use tools. So if you have a model that’s functioning as a black box, there’s ways that you can take a particular input that has a particular output, and then see what would happen if that input was different. So this allows you to say, see if the person had been male or female, or trans or whatever, whether they still would have got the loan. But doing this in a very systematic fashion, so that you can then chart the importance of each of the features as well as if you want to know why a decision was made on a individual case, also floating which were the most important reasons for the model to make that decision based upon what would have happened otherwise. There’s also a number of techniques that get a lot closer to the model itself. When you’re building the model that you can select certain algorithms where the algorithms themselves are easier to kind of read off Why they behaved as they did. So some, some of them are like decision trees that are built into the algorithm. And then you can actually look at the decisions made while the algorithm was processing the data by choosing that you have an ability to have an inherently explainable model. So, these are now pretty, pretty standard techniques amongst people in AI ml. And I’m being adopted within the banks.

Loraine Lawson
Okay, was there anything else that you think banks and credit unions should know about? I mean, there’s probably a lot they should know that you can share about eliminating bias in their algorithms and their AI ml?

David Van Bruwaene
Well, I think at at a more senior level, it needs to be understood that there’s no perfection here. And what is being sought by regulators and I am going to imagine the public is that you are actually making efforts to remove bias on a continual basis, to test for it to measure to check your models once they’re actually in use. And that, ultimately, you’re going to need to be able to prove continual efforts to improve this, this means we don’t get zero, absolute zero, bias, whatever that even means. And I think people are understanding more and more that there is a conversation to be had about how to measure it, how to make decisions when weighing between profitability, and measuring for bias. And this is definitely a conversation at the board level that is being progressively felt throughout the organization.

Loraine Lawson:
You’ve been listening to the Buzz, a Bank Automation News podcast. Thank you for your time and be sure to visit us at BankAutomationNews.com for more automation news. You can also follow us on Twitter and LinkedIn. Please don’t hesitate to rate this podcast on your podcast platform of choice.

Loraine Lawson
Good day and welcome to The Buzz, a Bank Automation News podcast. I’m Deputy Editor Loraine Lawson. In today’s episode, we’re joined by David Van Bruwaene, who is the founder and CEO of Fairly.AI. Fairly.AI is an Artificial Intelligence governance, risk and compliance solution that automates model risk management. The company specializes in the financial services, healthcare, HR and education sectors, and recently won the UBS’ Global Future of Finance Challenge. We discussed the problems in removing bias from AI models but first I asked Van Bruwaene to explain common use cases for Fairly.AI in the banking industry.

David Van Bruwaene
So we are most of most interest right now in more customer facing application. So if you have a mortgage origination model, that can take into account a number of factors that could lead towards biased outcomes. And progressively more information is being provided on individual applicants for a mortgage or loan or anything really, where you are doing credit worthiness decisioning. Including something like his zip code, for example, can cause a model to make a prediction that’s biased against people of certain ethnicities who tend to live in that zip code, even though it’s not directly represented. Our system applies the same traditional model risk management techniques that have been in place since the boat 2011, or following the financial crisis. And the framework of champion challenger models. So you, you can be building many traditional models, and also trying to include machine learning models in the mix. And then comparing the performance and ultimately making a decision about which models to put into production. So we, we handle models at a pretty high level, but also have tooling to integrate directly with machine learning.

Loraine Lawson
Okay, so you’re looking for, for bias or potential ethical issues with the models? Are there particular mistakes that you see companies make when they’re trying to create these models or using the data or just things that you see that are common?

David Van Bruwaene
Well, the most common mistake when dealing with bias is to assume that you can just remove any of the obvious input features like race, age, gender from the model, and expect that this will lead to a model that’s not biased. It’s a way of providing some reassurance throughout management that nothing untoward is happening, how could it be biased in the first place, but nonetheless, there’s a concept of proxy values where other input features really can represent the protected class information, and then lead to biased results. And by removing this information, and not making it available to data scientists or model validators, there’s no opportunity to identify it. And there’s no opportunity to make algorithmic moves to correct for that bias, because there there’s quite a lot available in modern machine learning techniques to actually alter the decisioning process of the model. In order to lessen the effect or impact of those proxy values.

Loraine Lawson
You give an example of a proxy value that might exist, even though you’ve removed identifiers or what we think of as identifiers.

David Van Bruwaene
So I mean, a very classic one is zip code. But there’s also employment gaps, for example, that apply to women when they’re pregnant, that can be detected, and then subsequent lower income. This kind of information gets picked up very easily. That’s a very Easy pattern for a model to represent. And if you do not have a means of identifying that a particular data set has a certain representation of women that’s different from men, then you have no, no real options. To correct it. Another major problem is not at very basic is not establishing enough data to represent each of the groups that you’re legally required not to be discriminatory towards. And so the models underperform for certain groups, and sometimes it could benefit them and sometimes make it worse. But generally speaking, the the goal that is pretty well understood within the academic community, as well as legal community is that you want the most accurate performance for each and every group as a kind of baseline. And then being able to make decisions about say, giving some disadvantaged groups and additional advantage over other groups is one that needs to be made very intentionally.

Loraine Lawson
So is it better to leave identifying data in in the data so that you can

David Van Bruwaene
in certain cases, actually, leading to more to fair outcomes, there’s a huge number of papers showing this in certain cases, or that this identifying data is less significant to the models than you might think. But there’s other pieces of information that becomes more significant. So in the case of a pregnancy leave, you might say, if if you look at it, the gap could be significant. The employment gap could be significant. But the gender doesn’t turn out to be that significant when you test the model for feature sensitivity, but generally speaking, I think most, most applications would not include that protected class information. And that does raise a lot of civil liberties issues. That cause for good reason. consternation around its use, but it can still be collected and used to test the models before they’re used in production, and then spot check while they’re in production is.

Loraine Lawson
Yeah, so I guess if the data is strict of identifiers, you can’t tell them if, if things like you can’t tell the if the problems are connected to the class, right? That’s right. Okay. There’s,

David Van Bruwaene
there’s also very popular technique where it’s available, where you it’s called adversarial D biasing where you actually train a model to predict one of the input features that you don’t want to be significant to it. And then you at the same time, change the way it operates to make it unable to predict. So it’s it’s a, it’s difficult to describe algorithmically and mathematically but relatively simple concept of if you make the model, if you train the model to be bad at predicting that someone is of a certain race, then you know that it’s not using that information elsewhere.

Loraine Lawson
Interesting. So it’s pretty complex science that you’re dealing with you’re what what do you sort of advise your clients coming into the process? How do they prepare for de-biasing?

David Van Bruwaene
So it always begins with setting up the requirements that you’re going to place on your models before you’re going to allow them into production use. Coming up with the set of quantitative and qualitative reporting requirements, and then providing this information to those We’re building the model as soon as possible so that they can take the right steps from the start to build models that are ultimately gonna comply with the requirements. As models are being built, it’s a very, it’s often usually maybe always a very kind of messy, iterative process, you try something, it doesn’t work, you try something else, maybe read a paper, figure out a new kind of technique, change the data, structure a bit and see what works. So as people are, are going through this process, having a clear set, having a goalpost effectively, that’s agreed upon between teams that are doing the building in the validating, is absolutely crucial. And this can get lost in the mix. It always seems to get lost in the mix. In all of the banks that we’re working with, that we have spoken to, very kind of manual process of emails and meetings and kind of, after the fact being told to make changes to report on something. And that’s no longer available. So it causes a lot of chaos. As as these requirements become more defined, and and become used, with additional updates to models, becomes a lot easier to streamline the reporting requirements, but also to be building the models always towards very high standards.

Loraine Lawson
And those standards, are they defined per project? Or do you have standards that you sort of use across the board.

David Van Bruwaene
So there’s a very general framework that we’ve built into our system. And that’s framework around SRL 11 Seven, which is a specific requirement from the Federal Reserve, and applies to a huge range of models that you wouldn’t even think might be applicable in banking. So customer support models, for example, chat bots. So huge range, models fall under that framework, where ultimately, you, you need to apply what are very kind of reasonable processes, if you understand how they’re built, you have to be building alternative alternate models, called challenger models, select a champion amongst them, compare the performance results, provide ample support, that your models are performing up to the specifications that are required, these are normal kind of statistical measures that are now standard, but also that you’ve used enough suitable test data, and describe that test data and ultimately provide to those to a separate function in the bank, the validation function, all the materials that they need to go through that same process of building models, testing them coming to the same conclusions that you did. And then at that point, they’re able to say the one that you’ve selected, does meet our criteria and is able to go out the door. That’s very general, and and broadly applicable, the specifics of each model will differ. And so that’s where having a more technical integration into the model, building platforms, allows us to service a huge range of types of models, as well, I mean,

Loraine Lawson
sorry, what’s your technical integration?

David Van Bruwaene 19:25
So concretely, let’s say you’re building a model in Python, you’re using Kerris and its classification algorithm. And it takes binary inputs. And then you’re going to build it and have a bunch of selections about how it should operate. That you tried different kind of out of the box selections like lol or L to. Colonel it’s, it’s something that’s become very standardized, but there’s just a huge different huge array of ways you might build a model, we’re able to adapt to whichever out of a large number of standard libraries are in use, and a large number of selections and manage all of them within our service. And then the qualitative information that’s required to be reported on and maintained and have an audit trail behind it approvals and so on. That is something that we have a number of templates out of the box effectively. But also, we have an editor so that you can make whichever specific alterations you want for a particular model that lies within the line of business, or, or with a particular team.

Loraine Lawson
Okay, and you said, there are ways that banks and credit unions can stay ahead of a regulations, what do you recommend?

David Van Bruwaene
Well, I recommend, first, when you’re thinking about how you’re going to build your models, what platforms you’re going to use, how you’re ultimately going to bring them into production, you may think about it as being driven by compliance. First, with there’s already regulations in a number of sectors. And the standard ones that have applied to statistical models are things like the Fair Lending Act, there’s, there’s a lot out there, that’s not new. And we do have a pretty good idea now of what the next moves are going to be, especially in Europe, where the AI regulations proposals already been made public. And I think there’s a lot of things that are pretty already pretty well set that you will have to be able to test for bias biases, a huge one that comes up, you have to test for things like potential for financial loss, reputational risk, and so on. But with AI, the things that you need to test for and provide are now pretty standard explainability data drift was just means that your models gonna keep working, even if the data changes. There’s a pretty standard set of tools around explainability that people have been using, and would for the foreseeable future meet minimum kinds of requirements. So by integrating a system that does compliance alongside the model platforms that you’re choosing, from the start, you get a very big leg up.

Loraine Lawson
So you mentioned there are tools, standard tools for explainability. What are those? Is that something you fairly AI does or is are there separate tools,

David Van Bruwaene
we we offer a validation suite, which includes easy to use tools. So if you have a model that’s functioning as a black box, there’s ways that you can take a particular input that has a particular output, and then see what would happen if that input was different. So this allows you to say, see if the person had been male or female, or trans or whatever, whether they still would have got the loan. But doing this in a very systematic fashion, so that you can then chart the importance of each of the features as well as if you want to know why a decision was made on a individual case, also floating which were the most important reasons for the model to make that decision based upon what would have happened otherwise. There’s also a number of techniques that get a lot closer to the model itself. When you’re building the model that you can select certain algorithms where the algorithms themselves are easier to kind of read off Why they behaved as they did. So some, some of them are like decision trees that are built into the algorithm. And then you can actually look at the decisions made while the algorithm was processing the data by choosing that you have an ability to have an inherently explainable model. So, these are now pretty, pretty standard techniques amongst people in AI ml. And I’m being adopted within the banks.

Loraine Lawson
Okay, was there anything else that you think banks and credit unions should know about? I mean, there’s probably a lot they should know that you can share about eliminating bias in their algorithms and their AI ml?

David Van Bruwaene
Well, I think at at a more senior level, it needs to be understood that there’s no perfection here. And what is being sought by regulators and I am going to imagine the public is that you are actually making efforts to remove bias on a continual basis, to test for it to measure to check your models once they’re actually in use. And that, ultimately, you’re going to need to be able to prove continual efforts to improve this, this means we don’t get zero, absolute zero, bias, whatever that even means. And I think people are understanding more and more that there is a conversation to be had about how to measure it, how to make decisions when weighing between profitability, and measuring for bias. And this is definitely a conversation at the board level that is being progressively felt throughout the organization.

Loraine Lawson:
You’ve been listening to the Buzz, a Bank Automation News podcast. Thank you for your time and be sure to visit us at BankAutomationNews.com for more automation news. You can also follow us on Twitter and LinkedIn. Please don’t hesitate to rate this podcast on your podcast platform of choice.