<!-- Canonical: https://www.aigentcy.com/case-studies/support-ai-evaluation/ -->

Customer support, online gaming

# Spend and rota set on real resolutions

We checked every conversation a vendor's support AI handled. Now the company pays for what the AI resolves, and sizes its support team for the real queue.

Client

A multi-brand online gaming group

Teams

Customer support and leadership

Scope

6 brands on 2 help desk accounts

Service

[AI testing and measurement →](https://www.aigentcy.com/services/ai-testing/)

[Book a 30-minute call](https://www.aigentcy.com/contact/)

36%of conversations resolved by the AI, against 53% claimed by the vendor

97%of our verdicts matched a person's, 288 of 297

165real customer cases tested before launch

6 brandschecked every weekday morning

## What changed for the business

### What it means for the group

-   In the first week, about 38 conversations the vendor counted as resolved still went to an agent. The company had been paying for those as AI resolutions and leaving them out of the agent rota.
-   The bill is now checked against resolutions that happened, and the rota and the rollout to new brands are planned on the checked rate.
-   The support team sees which conversations failed and why, so fixes go to real cases.

| Measure | Before | After |
| --- | --- | --- |
| What the bill is checked against | Nothing, paid as invoiced | Resolutions that happened |
| What the agent rota is sized on | The vendor's own count | The queue left for agents |
| Conversations checked | None against agreed rules | Every conversation, every weekday |

We measured these from the first week of daily checks.

## What leadership needed

Leadership wanted to roll the support AI out to more brands, and part of the vendor's bill depended on how many conversations it resolved.

Before paying for more, they needed 3 answers: what the AI resolved, what happened when it handed a customer to an agent, and whether the invoice matched the work.

### How it worked before

-   The only number was the vendor's own resolution rate. It counted every conversation the AI kept away from an agent, whether or not the customer's problem was solved.
-   Nobody checked conversations against agreed rules, so the bill, the agent rota and the rollout plan all rested on that number.

## What we built

1.  01
    
    ### Tests before launch
    
    Real customer cases replayed through the AI for each brand and scored against what a good agent would do.
    
2.  02
    
    ### Every conversation checked each morning
    
    Clear cases, such as a customer handed to an agent, are decided by fixed rules. AI judges the rest, and every verdict is stored with its evidence. Personal data is masked first.
    
3.  03
    
    ### One page for leadership
    
    The checked resolution rate next to the vendor's, with handovers to agents, repeat contacts and cost per resolved case, behind the company sign-in.
    
4.  04
    
    ### Safe access to customer data
    
    The AI reads only the account details a conversation needs, through a narrow connection that logs every lookup.
    

## How it went live

1.  **Before launch**
    
    The test cases run against each brand, and the results reviewed with the support team.
    
2.  **First week**
    
    223 conversations checked against the agreed rules, giving the first full set of numbers.
    
3.  **From then on**
    
    Checked every weekday morning across 6 brands on 2 help desk accounts. A new brand is added with a settings change.
    

> For me, this is the most major project in the whole company right now. We used to take the vendor's resolution rate on trust. Now every conversation is graded each morning, we can see what the bot really resolved, and we can check the bill against it.

**Head of Operations**Online gaming group

## Questions about this project

Did this replace the vendor?

No, the AI stays. We check it from the outside, so the same approach works with any vendor.

Why is the checked rate lower than the vendor's?

The vendor counts every conversation the AI keeps away from an agent. We count only the ones where the customer's problem was solved.

What happens to customer data?

Personal data is masked before checking, the AI reads only what a conversation needs, and every lookup is logged.

## More case studies

-   [Client support, online gaming**Staff stopped copying client emails into Jira**0 client emails staff copy into the tracker, down from every one](https://www.aigentcy.com/case-studies/service-desk-automation/)
-   [Quality assurance, online gaming**Releases no longer wait on testers**18 min for the release check, down from 2 tester-days](https://www.aigentcy.com/case-studies/release-test-automation/)
-   [Human resources**5 hours a week back for HR, payroll on time**250 hrs a year of HR time back from reports and chasing](https://www.aigentcy.com/case-studies/hr-automation/)
-   [IT services**Enquiries answered in 15 minutes**15 min median time to a reviewed reply, down from 1 to 2 days](https://www.aigentcy.com/case-studies/odoo-lead-generation-automation/)

## See what your support AI resolves and what to pay for

Send last month's numbers. We'll show you how we'd check them, and what the vendor should be paid for.

[Book a 30-minute call](https://www.aigentcy.com/contact/)
