We ran 14 IA models on a job a credit union pricing analyst does for the board: review a credit union’s overdraft and non-sufficient funds (NSF) fees, find 10 similar credit unions that changed theirs since 2021, and recommend a fee package with revenue estimates.
Across 168 runs (14 models, 12 real US credit unions), judges compared the 151 resulting files in 881 head-to-head pairs. Claude Fable 5.1, Claude Fable 5.2 and Claude Opus 5.5 produced the best files, too close to rank against each other.
IA for credit unions benchmark results
All models ran at their agent program’s default reasoning effort.
- Claude Fable 5.1, Claude Fable 5.2 and Claude Opus 5.5 span 2.8 points. Fable 5.2 scored highest on 6 of the 12 tasks, Fable 5.1 on 4 and Opus 5.5 on 2.
- GPT-6.1 Terra wrote one to four rows instead of 10 in every task, in a median of 2 minutes, and scored 19.7. DeepSeek V4.1 Flash delivered 12 complete files and scored 19.5.
- Of the 168 runs, 17 left no usable file: nine from GPT-6 Luna, five from Qwen 3.8 27B, two from GLM 5.3 and one from GLM 5.3 Flash.
- GLM 5.3, the highest-scoring model on the Doubleword API, scored 25.8.
For details of the benchmark, read benchmark methodology.
Cost and score
Cost is token use at list price per credit union task. The Claude, GPT and Grok models ran on subscription plans, so these figures are not money spent.
- Claude Opus 5.5 scored 80.7 at $14,68 a task. Claude Fable 5.1 scored 2.8 points more at $34,73, 2.4 times the cost.
- Grok 4.7 scored 76.0 at $6,72 a task, less than half of what Opus 5.5 cost.
- GPT-6 Astra cost $15,50 a task, 6.3 times as much as GPT-6.1 Sol, and scored 2.1 points lower.
- GLM 5.3 cost $10,31 a task, more than Grok 4.7, and scored 50.2 points lower.
- Across the 14 models, cost per task and score correlate at 0.73. Median time per task and score correlate at 0.25.
Where files fell short
A file passed the format check when it had the 14 required columns and exactly 10 rows. Five models passed it on all 12 tasks: Claude Fable 5.1, Claude Fable 5.2, Claude Opus 5.5, Grok 4.7 and DeepSeek V4.1 Flash. The others fell short in different ways:
- GPT-6.1 Terra stopped after one to four peers in all 12 tasks. In one task, it wrote that it had “one evidenced peer row, not the requested ten”.
- Claude Haiku 5.5 wrote between one and nine rows in 10 of its 12 files.
- GPT-6 Luna finished every task in 90 seconds or less. Four runs left no file and five left a header with no rows.
- Qwen 3.8 27B left no file in three tasks, and two of its files could not be read against the required columns.
- GLM 5.3 left no file in two tasks, and GLM 5.3 Flash wrote 15 and 16 rows in two files, over the limit of 10.
- GPT-6 Astra, GPT-6.1 Sol and Claude Sonnet 5.5 wrote fewer than 10 rows in one or two files each.
Judge disagreement
The two judges agreed on 744 of the 881 pairs. On the 137 pairs they split, each judge leaned toward its own vendor’s files:
- The GPT judge picked the OpenAI file in 86 of the 89 split pairs that included exactly one OpenAI file.
- The Opus judge picked the Anthropic file in 54 of 72.
- Grok 4.7, which decided every split pair, picked the Grok file in 20 of 24.
If each split pair counts as half a win for both files rather than going to Grok, Grok 4.7 drops from fourth to fifth, and GPT-6.1 Sol moves to fourth. The same three models stay on top in that case. Under all three scoring schemes, Claude Sonnet 5.5 ranks seventh, and GPT-6 Luna last.
How credit unions use IA
IA adoption among credit unions is ahead of banks on one measure. In a Cornerstone Advisors survey of 416 bank and credit union executives, published in January 2026, 59 % of credit unions and 49 % of banks had deployed generative IA.1
A separate study of 78 credit unions by Cornerstone Advisors, the League of Credit Unions & Affiliates and LEVERAGE found that about 40 % of credit unions have no defined IA strategy.2
Member service and IA chatbots
In the same 78-credit-union study, 80 % of credit unions that use IA in the contact center use generative IA. The main uses are suggested answers for staff during calls, knowledge base search and call summaries. For more on member-facing assistants, see banking chatbots.
In PYMNTS Intelligence and Velera research from April 2026, members who had previously switched financial institutions were 122 % more likely than average to want IA chat support.3 Supply lags that demand: only 20 % of top-performing credit unions offered conversational IA for payments.
Fraud detection
About half of credit unions had higher fraud-related losses in 2025, compared with about 40 % of banks, according to the Cornerstone Advisors survey. In the 78-credit-union study, 85 % of credit unions that use IA for fraud detection use machine learning to flag payment fraud in real time.
Suncoast Credit Union reported that its net fraud losses fell more than 35 % from 2023 to 2024 after it added an identity and fraud decisioning platform at account opening and later at login.4
Generative IA also helps fraudsters. FinCEN’s November 2024 alert reports a rise in suspicious activity reports that describe deepfake identity documents used to pass identity verification, and it lists red flags for financial institutions.5
Research and compliance work
Fee benchmarking, vendor due diligence and regulatory checks share one requirement: every claim must trace back to a document. The task in this benchmark is this kind of work: a row whose source the agent did not open counts for nothing. For tools in this category, see IA compliance software.
IA platforms for credit unions
Credit unions can use IA agents through business software that connects language models to member records and workflows.
Creatio
Creatio is a CRM and workflow platform with IA agents. Its financial services edition is sold to banks and credit unions and includes workflows for personal loans, vehicle loans, HELOCs and credit cards. Agents get role-based permissions, their actions can require a person’s approval, and each IA action is written to an audit log.6
Creatio announced three US credit union customers between July and September 2026:
- Everwise Credit Union, with more than 300 000 members, is moving 27 departments to Creatio. It uses agents that summarize member interactions and recommend the next product.7
- Arizona Financial Credit Union, with $3,9 billion in assets, is starting with sales and service.8
- Securityplus Federal Credit Union, with about $580 million in assets, uses IA to summarize member interactions and recommend next steps.9
Salesforce Agentforce
Agentforce is the agent layer of Salesforce’s CRM, and Financial Services Cloud is the edition for banks and credit unions. BCU, a credit union with over $6,5 billion in assets and more than 360 000 members, replaced its scripted chatbot with an Agentforce assistant that answers questions about loan status and account details. Salesforce reported in May 2026 that the assistant resolves 82 % of inquiries without staff and that escalations to staff fell 15 %.10
Eltropy
Eltropy sells its agent platform to credit unions and community banks and says more than 750 of them are customers. In March 2026 it added two agents. One verifies a member’s identity during a conversation. The other then handles account tasks such as card problems, transaction history and loan payment status across voice, chat and SMS.11
interface.ai
interface.ai sells only to credit unions and community banks. Its BankGPT platform, launched in November 2025, runs a phone agent, a chat agent and an assistant for staff. The company reports more than 100 institutions live and lists integrations with Jack Henry, Fiserv, Alkami and Q2 systems.12
IA governance for credit unions
The NCUA has no IA-specific rules. Its IA resource page states that “existing regulations are technology-neutral and apply to IA use”, so information security standards cover sensitive member data in an IA tool as they cover it in email.13
The Government Accountability Office found in May 2025 that the NCUA’s model risk guidance is “limited in scope and detail” and that the NCUA “lacks the authority to examine technology service providers”. This leaves due diligence on IA vendors to each credit union.14 Instead of new rules, the NCUA page links to NIST’s IA resources, a COSO paper on IA, two CISA guides, a Treasury report on IA in financial services, and the FinCEN deepfake alert.
Methodology of IA for credit unions benchmark
The task
An IA agent here is a model running inside an agent program that can search the web, open pages and run code. Each agent acted as a pricing analyst for one credit union and had to deliver one CSV file: 10 similar credit unions that changed their overdraft or NSF fees, and a fee package it recommends.
We wrote the tasks for 12 real US credit unions, three in each of four regions. In each region we picked the credit unions with the highest, the middle and the lowest share of overdraft and NSF income in 2024 revenue. All 12 hold more than $1 billion in assets.
The instructions were the same for all 12 apart from the credit union’s name and region, and the agent had to judge every fact as of 16 October 2026. Each agent received the task text and nothing else. The scoring guide stayed off the machine the agents ran on.
Which credit unions counted as peer
A credit union counted as a peer only if it met all four rules:
- It is a federally insured US credit union.
- Its total assets at 31 December 2024 were between half and twice those of the credit union the agent worked for.
- It is headquartered in the same region.
- It changed an overdraft or NSF practice on or after 1 January 2021.
The agent had to prove the change and its date with a page it opened: a fee schedule, an account disclosure, a member notice, a press release, a news article, a court filing or an archived copy of one of these.
What each row contains
Each of the 10 rows has two parts:
- The peer: what it changed and when, its 2024 overdraft and NSF fee income, and the impact it disclosed.
- One element of the recommended fee package: the credit union’s current practice, the change, its annual revenue impact in dollars, the effect on members, and the trade-offs.
The fee income comes from the Call Report, the quarterly financial report every federally insured credit union files with the National Credit Union Administration (NCUA).
Three rules applied to every row:
- It needs a source URL the agent opened. A fabricated or unreachable URL voids the row.
- A fee change cannot be inferred from Call Report figures alone.
- Lawsuit claims are allegations, and a credit union that published no impact figures gets “None disclosed”.
Why no single database answers the task
The NCUA stopped publishing overdraft and NSF fee income for individual credit unions from the 31 March 2025 Call Report cycle and now collects it during examinations. The fourth quarter of 2024 is the last quarter with these figures.
Later Call Report files show zero in both fee income accounts for every credit union. A zero there does not mean a credit union dropped its fees. Current fees have to come from each credit union’s fee schedule, disclosures, and archived web pages.
Models and agent programs
The results measure each model together with its agent program, because a model’s search tool and its handling of long sessions come from the program.
Nobody set a reasoning level, so each model ran at the default shown in the table. opencode sent no level, so the Doubleword models ran at the provider’s default.
Two planned models are not in the results. Kimi K3.1 Pro ran on the Doubleword API, but its runs were not complete across the 12 tasks. Muse Spark Open ended all three test runs at the point where the agent program first shortened the conversation history to fit the context limit, so its full run did not start.
Run setup
All agents ran on the same Linux server. Each run had its own working folder and temporary folder, so no agent could read another agent’s files. Agents had no connectors or plugins beyond web search, page fetch, and code.
Each task had a 75-minute time limit. A task that hit the limit while the agent was still working ran once more with a 6-hour limit, from the start. Fourteen scored runs took longer than 75 minutes, seven of them from GLM 5.3. The longest lasted 5.3 hours. An agent that showed no activity for 15 minutes was stopped.
The 168 scored runs took 101 hours of agent time in total, with a median of 24 minutes per run. The shortest run ended after 28 seconds. All runs took place between 3 and 11 November 2026.
The runs used 2.4 billion tokens, and 87 % of them were cached input that an agent program reads again at each step. Claude Opus 5.5 used the most, 32.6 million tokens per task. GPT-6 Luna used the least, 0.4 million.
How cost was measured
Every cost figure is recorded token use multiplied by list prices. For the four Doubleword models, a logging layer between the agent and the API recorded input, cached input and output tokens for every call, priced at Doubleword’s list prices. For the Claude, GPT, and Grok models, the token counts come from each agent program’s logs and are priced at the vendor’s list prices.
The Claude, GPT, and Grok runs used subscription plans, so no run has a bill. We did not use the agent programs’ own cost reports, because each prices tokens against its own price list. Token counts include the helper model an agent program calls in the background.
Cost per task covers the scored run of each task. Earlier attempts of tasks that ran again are not included. List prices for GPT-6.1 Terra, Claude Haiku 5.5, Claude Fable 5.1 and Grok 4.7 are not verified.
Automatic checks
Before judging, a script checked each file: valid CSV, the 14 required columns in order, exactly 10 rows, and digits only in the integer columns.
A run with no file, or with a file the script could not read against the required columns, scores 0. This applied to 17 of 168 runs. The other 151 files went to the judges as written, including 32 with a format defect such as fewer than 10 rows. The scoring guide treats a missing row as incomplete work.
Two Fable 5.1 files came from runs that stopped before a clean finish, one at the 75-minute limit and one at the account’s usage limit. Both files passed the format check and were judged.
Judging
Two judges compared every pair of files for the same credit union and picked the better one: GPT-6 Sol on Codex CLI and Claude Opus 5.5 on Claude Code, both at high reasoning effort. When they disagreed, Grok 4.7 decided the pair.
The judges saw no model names, and the two judges saw each pair in opposite orders. They had no web access. They received saved copies of the pages the files cite and were told to check claims against them.
They followed a written scoring guide that puts the recommended fee package first, the peer facts second and the income and impact figures third. Length earns nothing, and padding and unsupported claims count against a file. Besides a broken row rule, the guide names four errors that cost the most:
- A peer that breaks an eligibility rule.
- An invented impact figure.
- An old fee presented as the credit union’s current practice.
- A recommendation with no amount, threshold, condition or timing.
From comparisons to scores
Every file was compared with every other file for the same credit union: 881 pairs across the 12 tasks. In each task, the files were ranked by the number of comparisons they won, and the rank was converted to points between 0 and 100, with the average judged file at 50. A model’s score is the mean over the 12 tasks.
The score is relative. It shows where a model stands among the 14 models in this benchmark, not the share of correct answers, and it cannot be compared with scores in other AIMultiple benchmarks.
Stability of the ranking
We redrew the 12 tasks at random with replacement 10 000 times and recomputed the ranking each time. The possible ranks in the results table cover 95 % of those draws. The redraws show how much the ranking depends on which tasks were included.
Limitations
- Each model ran each task once, so the scores carry run-to-run variation that the redraws do not measure. We did not test how often a judge repeats its own verdict.
- Claude Opus 5.5 and Grok 4.7 judged pairs that included their own files. The judge disagreement section shows how far each judge leaned toward its own vendor.
- A reasoning setting above the program default can change both quality and cost.
- The agents worked on the live web. Pages can change between runs, and some credit union and news sites block automated access. The scoring guide treats a blocked page as unverified, not fabricated.
- The judges could not check the 2024 Call Report fee income figures, because the source is a zip archive with no readable page copy. A wrong figure in those two columns did not lower a file’s rank.
Other AIMultiple agent benchmarks
AIMultiple runs agent benchmarks on other business work as well:
- AIM Enterprise: 16 models on 69 tasks written against real company decisions in strategy, marketing, HR, sales and operations. Claude Opus 5 and GPT 6 Astra finished 1.6 points apart.
- AIM VC: 13 models named a company’s clients with dated evidence for three target companies. Claude Opus 5 scored 89.1 of 100.
- AIM IT: 12 models each designed, built and ran a benchmark twice. None of the 24 attempts passed every criterion.
FAQ
Not without review. The judges could not verify the 2024 Call Report fee income figures in any file, and many runs ended with no file or an incomplete one. Vendors building this kind of agent are listed in our enterprise IA companies breakdown, and AIMultiple automates processes like this one.
It depends on the model. At list prices, the most expensive model cost thousands of times more per task than the cheapest. The cost and score section lists each model’s cost.
Citez ce benchmark
Choisissez le format qui correspond à votre lieu de publication. Coller la version avec lien dans votre CMS préserve le lien retour.
@misc{arslan2026,
author = {Arslan, Ezgi},
title = {{IA for Credit Unions: 14 IA Models Tested}},
year = {2026},
month = nov,
howpublished = {\url{https://aimultiple.com/ai-for-credit-unions}},
note = {AIMultiple. Consulté le 16 novembre 2026}
}
Soyez le premier à commenter
Votre adresse courriel ne sera pas publiée. Tous les champs sont obligatoires. Les commentaires sont laissés dans leur langue d'origine.