Customer support, online gaming
Spend and rota set on real resolutions
We checked every conversation a vendor's support AI handled. Now the company pays for what the AI resolves, and sizes its support team for the real queue.
- Client
- A multi-brand online gaming group
- Teams
- Customer support and leadership
- Scope
- 6 brands on 2 help desk accounts
- Service
- AI testing and measurement →
What changed for the business
What it means for the group
- In the first week, about 38 conversations the vendor counted as resolved still went to an agent. The company had been paying for those as AI resolutions and leaving them out of the agent rota.
- The bill is now checked against resolutions that happened, and the rota and the rollout to new brands are planned on the checked rate.
- The support team sees which conversations failed and why, so fixes go to real cases.
| Measure | Before | After |
|---|---|---|
| What the bill is checked against | Nothing, paid as invoiced | Resolutions that happened |
| What the agent rota is sized on | The vendor's own count | The queue left for agents |
| Conversations checked | None against agreed rules | Every conversation, every weekday |
We measured these from the first week of daily checks.
What leadership needed
Leadership wanted to roll the support AI out to more brands, and part of the vendor's bill depended on how many conversations it resolved.
Before paying for more, they needed 3 answers: what the AI resolved, what happened when it handed a customer to an agent, and whether the invoice matched the work.
How it worked before
- The only number was the vendor's own resolution rate. It counted every conversation the AI kept away from an agent, whether or not the customer's problem was solved.
- Nobody checked conversations against agreed rules, so the bill, the agent rota and the rollout plan all rested on that number.
What we built
- 01
Tests before launch
Real customer cases replayed through the AI for each brand and scored against what a good agent would do.
- 02
Every conversation checked each morning
Clear cases, such as a customer handed to an agent, are decided by fixed rules. AI judges the rest, and every verdict is stored with its evidence. Personal data is masked first.
- 03
One page for leadership
The checked resolution rate next to the vendor's, with handovers to agents, repeat contacts and cost per resolved case, behind the company sign-in.
- 04
Safe access to customer data
The AI reads only the account details a conversation needs, through a narrow connection that logs every lookup.
How it went live
- Before launch
The test cases run against each brand, and the results reviewed with the support team.
- First week
223 conversations checked against the agreed rules, giving the first full set of numbers.
- From then on
Checked every weekday morning across 6 brands on 2 help desk accounts. A new brand is added with a settings change.
For me, this is the most major project in the whole company right now. We used to take the vendor's resolution rate on trust. Now every conversation is graded each morning, we can see what the bot really resolved, and we can check the bill against it.
Questions about this project
Did this replace the vendor?
No, the AI stays. We check it from the outside, so the same approach works with any vendor.
Why is the checked rate lower than the vendor's?
The vendor counts every conversation the AI keeps away from an agent. We count only the ones where the customer's problem was solved.
What happens to customer data?
Personal data is masked before checking, the AI reads only what a conversation needs, and every lookup is logged.
More case studies
- Client support, online gamingStaff stopped copying client emails into Jira0 client emails staff copy into the tracker, down from every one
- Quality assurance, online gamingReleases no longer wait on testers18 min for the release check, down from 2 tester-days
- Human resources5 hours a week back for HR, payroll on time250 hrs a year of HR time back from reports and chasing
- IT servicesEnquiries answered in 15 minutes15 min median time to a reviewed reply, down from 1 to 2 days
See what your support AI resolves and what to pay for
Send last month's numbers. We'll show you how we'd check them, and what the vendor should be paid for.