PREMIUM – Facebook built a bot called a “classifier” in order to identify credit, housing, and employment ads, in order to exclude them from the social network’s “multicultural affinity” targeting tools, but the bot failed, and the result is that Facebook may have violated multiple federal laws concerning fair lending.

This is according to new documents provide by Facebook to Congress and released yesterday on the social media company’s data and privacy practices.

When Mark Zuckerberg testified before the Senate in April there were quite a few questions he couldn’t answer. Facebook promised to provide them, and yesterday they were released, all 454 pages of them.

The documents revealed that Facebook tried and failed to prevent the targeting of vulnerable groups with potentially unfair offers of credit. In response to a written question from Senator Duckworth (D-IL), Facebook noted:

In late 2016, we began building machine learning tools (called “classifiers”) that were intended to automatically identify, at the point of creation, advertisements offering housing, employment or credit opportunities … We built these classifiers so that when we identified one of these kinds of ads, we could: (1) prevent the use of our “multicultural affinity” targeting options in connection with the ad, and (2) for the use of any other kind of targeting, require that the advertiser certify compliance with our anti-discrimination policy and applicable anti-discrimination laws.

We trained the classifiers before we launched them, including by using search terms provided by your office in January 2017. After the classifiers launched in approximately February 2017, we anticipated that, through machine learning, they would become better over time at distinguishing ads offering housing, employment, or credit opportunities from other types of ads. We also expected that we would receive feedback about the performance of the tool that would enable us to detect problems and improve the classifiers over time.

But the bots failed, the company noted, becoming “over- and under-inclusive, identifying and requiring self-certification for hundreds of thousands of ads each day that may have had nothing to do with housing, employment, or credit offers, while missing ads that may have contained such offers.”

Facebook’s machine learning failed, the company said, because its testing sample size was too small. Another reason for failure according to Facebook was the lack of feedback from users:

We also failed to fully account for the lack of feedback we would likely receive about the performance of these classifiers through other channels—feedback we typically rely on to alert us to performance issues. For example, advertisers whose ads should have been (but were not) identified through this process would have had no reason to report a problem.

To remedy this situation, Facebook is hiring more than 1,000 people to its global ads review team, as well as taking additional steps to screen ads before launch. Perhaps most importantly, Facebook has “disabled the use of multicultural affinity exclusion targeting for all ads; this prohibition is no longer limited to housing, employment and credit ads.” It noted later in the document, with a more specificity, “we disabled the use of other exclusion targeting categories that we determined, on their face, may have been misunderstood to identify a group of Facebook users based on race, color, national origin or ancestry.”

This news may put a chill on the use of bots and machine learning in sensitive areas of financial service such as underwriting.

For more coverage of the Facebook Files, click here.