Artificial intelligence is increasingly seeping into our financial lives. While the experiment started with algorithmic decisioning for social media feeds, AI systems now run chatbots, monitor for fraud, sniff out money laundering, and even underwrite loans.

While pop culture versions of AI bring up images of sentient systems trying to take over the world, currently the threats to AI are less cinematic but still damaging.

“Don’t worry as much yet about machines taking over the world,” former U.S. President Barack Obama told Wired magazine in 2016, “worry about the capacity of either nonstate actors or hostile actors to penetrate systems.”

Put simply, AI and machine learning systems use a trove of pre-existing data for training, followed by continuous improvement as they gain information from interactions with the open world. AI, in many ways, determines how much we can control, or even litigate on, its decisions.

But what if an actor, malicious or benign, could penetrate the security systems and corrupt the underlying data or change the importance of the parameters the AI system uses to make decisions? The data is said to be “poisoned” in such a case, causing the model to produce incorrect outcomes.

What is data poisoning? How risky is it?

Through 2023, up to 10% of AI training data will be poisoned by benign or malicious actors, according to a Gartner report. “Both benign and malicious actors can poison AI training data, but poisoning by benign actors is likely the [sic] more common today,” the report noted.

“Data poisoning can happen at any stage,” Avivah Litan, analyst and vice president at research firm Gartner, told Bank Automation News. She said that, for instance, if a model is being trained to detect fraud, and malicious actors introduce data that normalizes the fraud, causing the model to miss it over time, the system’s purpose can be evaded without raising any red flags.

Compared to other organizations that use AI models, “banks are way ahead because they’re regulated,” Litan said. Regulations and liability concerns often lead to banks and financial institutions investing more in rigorous testing and monitoring of such models.

Poisoned data can create unwelcome consequences for any organization. If corrupted due to human error, the model can produce skewed outcomes and dent the system’s reliability and an organization’s reputation. If poisoned with malicious intent, the corrupt data or learning model could end up helping attackers steal money or intellectual property by failing to detect an intrusion.

While such attacks can have chilling consequences, they “would likely take a lot of effort,” Bob Maley, chief security officer at automated cyber risk monitoring firm Black Kite, told BAN. He said that while banks and financial institutions could represent “high value targets” for such an attack, data poisoning typically occurs after the attacker has gained access to the internal controls.

Attackers looking to carry out such threats would also likely be highly skilled and well-funded, said Maley. Such attacks fall under the “advanced persistent threat” (APT) bucket, meaning they are likely to be carried out in stealth and can be undetected for a considerable period of time.

Attack vectors and guardrails

Data poisoning at the hands of a benign or unscrupulous actor is mostly a function of human error, and can arise out of misconfigured parameters, ingestion of biased data, or simply incorrect data entry.

On the other hand, malicious actors can attack by feeding the model incorrect inputs or repeatedly querying it in an attempt to reverse engineer the training dataset, which can then be used to produce a replica and learn how to get around it.

Poisoning data is “easier than breaking into most places that have been broken into,” said Litan, referring to recent attacks on vendors like SolarWinds and FireEye. While it’s worth noting that both these attacks were reportedly sponsored by nation-states and were highly skilled and well-funded, AI-related risk also arises when “organizations aren’t generally well-coordinated” about their usage, Litan added.

And when the monitoring buck gets passed around, it can create openings for actors looking to get into the system.

The four pillars of ensuring AI security, according to Gartner, are:

• Human focus – awareness of security measures, anti-phishing protections, and clean user interfaces for data entry;
• Enterprise security – user and entity behavior analytics, network security and proper authentication;
• AI model integrity – trustworthy AI models, validation checks; and
• AI data integrity – data poisoning detection and protection.

One challenge such data poisoning attacks throw up is that “there are very few off-the-shelf products that enterprises can use to detect” it, according to Gartner. Therefore continuous monitoring, inventory of AI models and ensuring that the decisions being made by the machine are explainable are likely to be the most suitable guardrails for now.

Bank Automation Ignite, on April 13-14, is the event for inspiring automation initiatives and investment in financial services. At the virtual event, financial services professionals can discover new use cases and technologies that are accelerating automation in banking. Learn more and register at www.BankAutomationIgnite.com.