LLMs are not your AI strategy
Why bank CEOs need a different playbook from ‘turn it on everywhere’
If you lead a bank today, you are almost certainly hearing some version of the same message from every direction:
- “We need an AI strategy.”
- “Our competitors are already using LLMs.”
- “We can automate entire functions with gen AI.”
Boards are asking about it. Regulators are asking about it. Vendors are promising it. And inside your own shop, ambitious leaders are pitching pilots in everything from complaints handling to credit operations and regulatory research.
The risk is not that you will ignore AI. The risk is that you will confuse turning on large language models with building a real, durable AI capability that supports your risk appetite, regulatory expectations and long-term profitability.
Joseph Cuanan, founder and CEO of ComplyGraph, a fully AI-driven compliance and risk company whose business is helping financial institutions design, validate and monitor LLM-enabled workflows so they are exam ready, explainable and safe to scale.
He assisted with this decision-maker’s guide to where LLMs actually fit and what you, as CEO or president, should insist on before you approve the next big AI initiative.
- An LLM is a tool. A strategy is what you do with it.
A quiet but dangerous pattern has emerged in the market: treating “we use LLMs” as synonymous with “we have an AI strategy.” Cuanan sees this every day.
“A great AI strategy for banks and fintechs is one that focuses on making their best team members even better,” he said. “This is often conflated with giving them access to chatbots and conversational experiences, which are great, but that’s just the tip of the iceberg. Start with the business problem you’re trying to solve, sit with team members whose roles most impact those problem spaces, and get creative with how LLMs can make them 10 times more effective, not just answer questions for them.”
For a CEO, that is a simple but powerful test.
- If your AI conversation starts with the question: “What can we do with the new model?” you are following a technology-first strategy.
- If it starts with “Where do our best people create disproportionate value, and how do we amplify that?” you are following a business-first strategy.
The institutions that will win with AI are not the ones that deploy the most chatbots. They are the ones that use these tools to compound the effectiveness of already excellent teams, including exam response teams, complaints resolution teams, commercial relationship managers, fraud investigators and model risk professionals.
Your first decision is to insist that every AI proposal is framed around a specific, high value business problem and a clear plan for making your best people five or 10 times more effective, not marginally faster at typing.
- LLMs are not traditional models. Your risk playbook must change.
Most banks have grown comfortable managing traditional models: scorecards, logistic regressions, credit models, fraud detection and so on. You can tune parameters, swap out training data and evaluate performance in relatively stable ways.
LLMs do not behave like that.
Cuanan puts it plainly: “Taking the time to understand how LLMs are different from traditional machine learning models will save a lot of compliance and model risk headaches. In the absence of being able to substitute training data, take models apart and assess performance with different parameters, mitigating model risk shifts to pressure testing data pipelines and evaluating outputs.”
For senior leaders, this means your oversight lens needs to move from “What data is this model trained on and how is it calibrated?” to a different set of questions:
- What can go wrong when we use this LLM in this workflow?
- Under what conditions do things go wrong?
- What guardrails do we have in place to prevent bad outputs from becoming bad outcomes?
These are not questions for data scientists alone. They are board and executive questions. When an LLM touches complaints, disclosures, account opening, collections, servicing or anything that reaches a regulator or consumer, you are in model risk territory, even if the vendor calls it “assistive AI.”
Your second decision is to demand that any LLM use case be accompanied by a clear model risk story: failure modes, conditions, controls and accountability.
- Hallucinations are not bugs. They are potential violations.
In a consumer app, an LLM that “hallucinates” is a nuisance. In a regulated bank, it is:
- A mis-explained dispute under Regulation E.
- An incorrect summary of a complaint that affects how it is coded and handled.
- A wrong statement of eligibility or fee disclosure.
In those scenarios, the model is not just wrong. It has created legal and regulatory exposure.
Managing this exposure is not about hoping the model improves, Cuanan said. It is about how you design the entire system around the model. “The most important elements when it comes to managing hallucination risk are context engineering, explainable pipelines and evaluations,” he said.
He breaks that into three concrete capabilities:
- Context engineering: “Being structured and explicit about what data the model should be using to generate outputs.”
In other words, the model should be grounded in your verified policies, procedures, disclosures, and regulatory text, not a vague prompt and the open internet.
- Explainable pipelines: “Decomposing workflows into smaller tasks, which are easier to test.”
Instead of asking the model to “resolve this complaint,” you might ask it to classify an issue, extract key facts or draft a first pass response for a human to approve. Smaller steps, easier to test.
- Evaluations: “Test cases derived from human expertise.”
These are not generic benchmarks, they are curated scenarios from your own risk and compliance experts that you run over and over to see how the system behaves.
“When the three meet,” Cuanan said, “you dramatically reduce hallucinations by requiring citations, guiding models to say ‘I don’t know’ vs. making up information and being able to repeatedly measure hallucination performance against human expertise.”
That is not magic. It is engineering, governance and disciplined testing.
Your third decision is to refuse any LLM deployment in a high-risk area that does not have all three: grounded context, decomposed workflows and a living evaluation suite owned by human experts, not vendors alone.
- Vendor concentration and model size are strategic, not technical, choices.
Most banks right now are quietly building dependence on a very small set of foundation model providers. From a distance, that looks efficient: fewer relationships, unified architecture, simple narratives for the board.
From a risk perspective, it is concentration risk.
A single provider outage, pricing change, policy change or failure can ripple across multiple critical functions if you have allowed one model to underpin everything from call center assistance to regulatory research.
Here again, the answer is not panic, it is discipline.
Cuanan points out that if you get the foundations right, you may not need to “standardize” on the largest models. “An additional benefit of investing in context engineering, explainability and evaluations is, depending on the use case, you may not even need a foundation model with billions or trillions of parameters. Well-defined and repeatable workflows and tasks can be implemented successfully with smaller, often locally deployed models. This also highlights why evaluations are so important, since they allow you to measure performance between different model sizes.”
For a CEO, that translates into two strategic moves:
- Do not let architecture decisions be driven solely by which vendor has the loudest story.
- Insist on side-by-side evaluation data that shows whether a smaller, cheaper or internal model performs just as well for a given task.
Your fourth decision is to treat model choice and vendor mix as risk and capital allocation questions, not just IT questions.
- A‘sane‘12- to 24-month LLM roadmap for a bank.
If you strip away the hype and focus on what actually creates value and survives regulatory scrutiny, you end up with a much more modest, but far more durable, roadmap.
Cuanan said: “A sane LLM roadmap starts by leveraging what an organization is already world class at. Empower subject matter experts to develop annotations and evaluations, so you have a way to understand what excellence looks like for a task or workflow. Then, leverage this data moat to experiment and identify where LLMs can deliver the most significant efficiency gains for your best team members.”
A practical roadmap for a bank CEO over the next 12 to 24 months looks something like this:
- Pick a few critical workflows, not a hundred pilots.
Think exam response, complaint classification, internal policy research or complex product support. Tie each effort to a concrete business outcome and risk metric. - Stand up an internal evaluation capability.
Ask your best compliance officers, risk managers and operations leaders to define gold-standard examples and edge cases. That “data moat” is how you prevent AI from drifting away from your standards. - Redesign workflows into small, testable steps.
Do not hand entire decisions to an LLM. Let it draft, summarize, classify and suggest. Keep human sign-off and accountability in the loop where it matters. - Benchmark models, including smaller and local options.
Use your evaluations to test multiple models and vendors. Choose what works, not what is loudest. - Align all of this with your risk appetite and supervisory expectations.
Treat every LLM deployment in a material workflow as a model risk and conduct risk question that your regulators will eventually ask about.
As Cuanan notes, “Most use cases are just a series of tasks, so there might be applications where an entire workflow shouldn’t be given to an LLM, but automating a specific component can free up human experts to do what they’re best at.”
That is where the sustainable value is: targeted automation that protects and amplifies your best people, not wholesale substitution of judgment with prediction.
The decision only you can make
LLMs are not going away. They will reshape the economics of many parts of banking. The question for CEOs and presidents is not “Are we using AI?” It is “Are we using it in a way that makes us more resilient, more compliant and more profitable over time?”
ComplyGraph, as a fully AI compliance and risk company, exists because the market has already discovered a hard truth: You cannot outsource accountability for AI. You can buy models, platforms and consulting, but you cannot buy away the obligation to understand what can go wrong, how you are controlling it and how you will explain it to your board and your regulators.
If you insist on those fundamentals now — business-first use cases, clear model risk stories, grounded and explainable workflows, real evaluations and thoughtful vendor strategy — you will not just “have an AI strategy.” You will have a bank that can use AI without betting the franchise on a hype cycle.
That is the difference between being this decade’s version of a dot com casualty and being the institution that is still standing, and thriving, when the dust settles.
Jim McCarthy is chairman for McCarthy Hatch, which provides data-driven insights for risk management. A founding member of the Consumer Financial Protection Bureau, he is a keynote speaker and fractional CRO/CCO in the financial services industry with more than three decades of experience.
