Vetting LLMs for banks: ‘No single model can be perfect at everything’

Rapid advancements in LLMs are pressuring financial institutions to experiment with the technologies and hone their vetting process. Nearly 80% of banks with over $250 billion in assets have generative AI tools live or in the pipeline, according to a survey conducted by Hanover Research for bank software provider Temenos . The adoption rate is 75% for banks with assets between $50 billion and $250 billion, followed by 40% for banks with under $10 billion in assets. Read Part 1: LLM war heats up: Claude vs. ChatGPT Gen AI adoption by bank asset size [caption id="attachment_57384768" align="aligncenter" width="732"] (ChatGPT-AI-Generated)[/caption] The global market for generative AI in banking and finance is projected to reach $26.3 billion by 2035, up from $1.7 billion in 2025, according to data intelligence firm Precedence Research . While increasing competition between Anthropic and OpenAI is prompting some FIs to evaluate one LLM against the other, others are taking a holistic approach. Privacy, security American Heritage Credit Union is using Perplexity for its primary LLM needs, Adrian Rodriguez , senior vice president of data and innovation at the credit union, told FinAi News . Perplexity is an AI-powered search engine that combines proprietary models with options to toggle among ChatGPT, Claude, Google Gemini and other LLMs, according to the company. “Our primary considerations when selecting an LLM are privacy and security, followed by reliability." — Adrian Rodriguez, SVP of data and innovation, American Heritage Credit Union “Perplexity meets those bars for us through enterprise data boundaries that keep our prompts and responses out of foundation model training, along with citation-grounded answers that make it easier to verify the reasoning behind a response," Rodriguez said. This strategy aligns with the direction of the National Credit Union Administration 's AI resource hub and the U.S. Department of Treasury ’s AI risk management framework, he said. In direct correlation to rising gen AI adoption, 82% of 200 banks are implementing more robust security measures, 77% are conducting regular audits and 73% are increasing employee training on data security, according to professional services firm KPMG . Risk management Meanwhile, Fifth Third Bank views LLM selection, particularly advanced frontier models, from a “vendor risk management perspective” because they can only be accessed via API, Senior Vice President and Director of AI Jay Budzik , told FinAi News . “We have a whole vendor risk management process where we look at the financials, the stability of the company,” he said. Data security and rigorous testing are also crucial when considering LLMs, Budzik said. “When the AI is in a business process where the output is used directly in that process, a lot more scrutiny is applied,” he said. “We have to demonstrate that the outcomes are good, that we have the sufficient volume of testing to cover the variations in language that could be used.” For instance, Fifth Third applied strict test protocols before rolling out its AI voice agent for payment reminders , evaluating the “variations in language that could be used” in response to potential customer queries, Budzik said. This mitigates reputational risk in addition to ensuring accuracy, he added. The $294 billion bank has leaned on Microsoft Copilot as its primary LLM provider over the past two years, Chief Executive Tim Spence said March 11 during an interview on Bloomberg TV . Budzik declined to disclose the bank’s other LLM providers but said that it has a hybrid-model strategy. Finding strengths In other instances, FIs are experimenting with different LLMs to identify the strengths of each one. Thread Bank is one FI using this strategy, CEO Chris Black told FinAi News . Thread Bank employees have access to “a platform that has many dozens of models in it from all the different companies,” he said. “We approach it that way. Let people kind of have their preference and get some guidance, some guardrails set by the AI team.” The $1 billion bank’s employees use the major LLMs — such as ChatGPT, Claude and Gemini — in a “controlled environment” to draft reports, retrieve data and boost productivity overall, Chief Digital Officer Marty Miracle previously told FinAi News . [caption id="attachment_57384766" align="aligncenter" width="800"] Anthropic's Claude investigates an authorization error. (Courtesy/Bloomberg)[/caption] Hatch Bank has taken a similar approach, President Amanda Swoverland told FinAi News. “There's some things where Copilot is very, very powerful, and then plugs into OpenAI,” she said. “But there are certain things where I really like Claude . … I’ve found that for Microsoft Excel, or some of those data-rich things, that Claude is really powerful.” Hatch Bank has several “power users” of AI who share the results of their experiments, Swoverland said. “They'll talk to the company and be like, ‘Yeah, I tried to do the same thing with both [ChatGPT and Claude], and I see better results here with Claude on the analytics,’” she said. “Sometimes there's just trial and error.” Open-weight models Rather than always choosing the most advanced LLMs, FIs should consider open-weight models “that you can actually tune for very specific jobs,” Scott Weller , chief technology officer at EnFi , a provider of agentic AI for commercial lending, told FinAi News . Open-weight models provide the trained parameters and are publicly available, allowing users to download the models for free and modify them on their own infrastructure, according to AI systems developer AI21 . “If you need a model that's just excellent at reading compliance certificates, rather than go to one of the biggest models, you can tune a model on those compliance certificates,” Weller said. [caption id="attachment_57384765" align="aligncenter" width="800"] A user POV of open-weight model DeepSeek. (Courtesy/Bloomberg)[/caption] To this end, Fifth Third Bank maximizes open-weight models through “supervised fine-tuning,” Budzik said. “We teach the model to do a very specific task that we're looking for it to do and therefore constrain the kind of outputs that it will generate and the risk associated with them,” he said. Examples of open-weight models, according to IBM , include: OpenAI’s GPT-OSS; Meta ’s LLama 3; Google’s Gemma 3; Mistral AI ’s Ministral 3; and DeepSeek ’s R1. Institutions must understand that “no single model can be perfect at everything,” Weller said. “You have to bring together a series of models, and you have to route to the appropriate model with the appropriate types of questions, the appropriate harness, to make sure you're doing it the right way with context,” he said. Register here for the FinAi Lending Summit, set for Oct. 7-8 in Las Vegas.

Vetting LLMs for banks: ‘No single model can be perfect at everything’

Part 2: Market for gen AI in banking could swell to $26B by 2035

Agentic AI solution by nCino reducing credit review time by up to 70%, CPO says

Anthropic, JPMorgan execs: AI ROI hinges on cracking adoption challenges

Related Posts

Acquisitions bring AI consulting in-house

Vena grows context engine with Morpheo AI acquisition

Microsoft abandons personal AI chatbot race with Copilot reboot

Anthropic, JPMorgan execs: AI ROI hinges on cracking adoption challenges

EMERGING FINTECH DIRECTORY

FinAi Podcast

SPONSORED

Build an Antifragile Strategy to Outperform the Market

How AI and Product Experts Turn Fuzzy Requirements Into Focused Dev-ready Roadmaps

Is Your Technology Supplier There for You?

Connect

Welcome Back!

Retrieve your password

Continue Reading with FinAi News Premium - Less than $2/Day

Vetting LLMs for banks: ‘No single model can be perfect at everything’

Part 2: Market for gen AI in banking could swell to $26B by 2035

Agentic AI solution by nCino reducing credit review time by up to 70%, CPO says

Anthropic, JPMorgan execs: AI ROI hinges on cracking adoption challenges

Related Posts

FinAi Podcast

Connect

Welcome Back!

Retrieve your password

Unlock This Article

Continue Reading with FinAi News Premium - Less than $2/Day