When it comes to measuring a bot’s success, the most common key performance indicator (KPI) is the total time savings it provides a full-time employee.

Calculating that is relatively simple, said Karen Reichle, the vice president of customer success engagement at Nintex, a workflow management and RPA software company based near Seattle.

Reichle gave the example of the Amarillo, Texas-based Happy State Bank, which estimated that it took 68 hours on average to build a bot, from defining the process to testing the automation. However, unlike employees, bots can work all night and all weekend; the $3.4 billion bank estimated that approximately 40 bots saved roughly 10,000 hours of employee time per year, Reichle said.

“If you even averaged it out at, on the low end, $35 an hour or $45 an hour, that’s a significant cost savings,” Reichle said.

But calculating how much employee time is saved isn’t the only way to measure bot success. The key to finding useful metrics for bot success lies in defining the business value the bot is attempting to achieve, sources told Bank Automation News.

Companies that focus on piecemeal returns on investment often struggle to scale their initiatives, said Anoop Gala, head of financial services at Orion Innovation, a digital transformation and product provider for JPMorgan Chase, S&P Global and software provider Red Hat.

“We should be looking at automation in a more holistic way,” Gala said. “It’s not just about how many bots you created, but essentially it is about this automation creating a competitive advantage for you as an enterprise and for your clients.”

Rather than simply calculating how many bots are created, Gala suggested that financial institutions examine whether the bots lower costs, increase sales or drive better financial performance.

Defining the process and measuring before deploying bots is a key strategy for establishing KPI benchmarks. Such pre-bot measurements will allow banks to determine, for example, reduced error rate from bot deployment, said Beji Varghese, who specializes in working with financial institutions as a partner with the private consultancy Guidehouse.

Documenting and measuring a process before automating it is key to creating a good KPI, Varghese said. One of his clients used a bot to automatically pull research information from six different systems, he said, adding that a useful measurement for that particular bot would include accuracy because, unlike humans, a bot will never miss a system.

While there’s no formula for measuring bot efficiency, financial institutions “have to get down into the details to be able to say ‘these are the four or five things that are going to get more efficient,” Varghese noted, adding that it is important to measure KPIs over time, because a bot may take time to show a return on investment.

Organizations see the best returns when they measure both operational and business KPIs, said Amit Kumar, vice president of financial services at RPA provider UiPath. Operational metrics — how many bots are deployed, for instance — will reveal the return on investment in a “rudimentary way,” he said. Kumar pointed to Paycheck Protection Plan bots as a model; banks made effort and time savings the focus, rather than the extra fee income the PPP bots enabled.

“The operating metric is very important, but the business metric you attach to it makes a much better value case for everyone,” Kumar said.