Customer support, online gaming

Spend and rota set on real resolutions

We checked every conversation a vendor's support AI handled. Now the company pays for what the AI resolves, and sizes its support team for the real queue.

Client
A multi-brand online gaming group
Teams
Customer support and leadership
Scope
6 brands on 2 help desk accounts
Book a 30-minute call
36%of conversations resolved by the AI, against 53% claimed by the vendor
97%of our verdicts matched a person's, 288 of 297
165real customer cases tested before launch
6 brandschecked every weekday morning

What changed for the business

What it means for the group

  • In the first week, about 38 conversations the vendor counted as resolved still went to an agent. The company had been paying for those as AI resolutions and leaving them out of the agent rota.
  • The bill is now checked against resolutions that happened, and the rota and the rollout to new brands are planned on the checked rate.
  • The support team sees which conversations failed and why, so fixes go to real cases.
MeasureBeforeAfter
What the bill is checked againstNothing, paid as invoicedResolutions that happened
What the agent rota is sized onThe vendor's own countThe queue left for agents
Conversations checkedNone against agreed rulesEvery conversation, every weekday

We measured these from the first week of daily checks.

What leadership needed

Leadership wanted to roll the support AI out to more brands, and part of the vendor's bill depended on how many conversations it resolved.

Before paying for more, they needed 3 answers: what the AI resolved, what happened when it handed a customer to an agent, and whether the invoice matched the work.

How it worked before

  • The only number was the vendor's own resolution rate. It counted every conversation the AI kept away from an agent, whether or not the customer's problem was solved.
  • Nobody checked conversations against agreed rules, so the bill, the agent rota and the rollout plan all rested on that number.

What we built

  1. 01

    Tests before launch

    Real customer cases replayed through the AI for each brand and scored against what a good agent would do.

  2. 02

    Every conversation checked each morning

    Clear cases, such as a customer handed to an agent, are decided by fixed rules. AI judges the rest, and every verdict is stored with its evidence. Personal data is masked first.

  3. 03

    One page for leadership

    The checked resolution rate next to the vendor's, with handovers to agents, repeat contacts and cost per resolved case, behind the company sign-in.

  4. 04

    Safe access to customer data

    The AI reads only the account details a conversation needs, through a narrow connection that logs every lookup.

How it went live

  1. Before launch

    The test cases run against each brand, and the results reviewed with the support team.

  2. First week

    223 conversations checked against the agreed rules, giving the first full set of numbers.

  3. From then on

    Checked every weekday morning across 6 brands on 2 help desk accounts. A new brand is added with a settings change.

For me, this is the most major project in the whole company right now. We used to take the vendor's resolution rate on trust. Now every conversation is graded each morning, we can see what the bot really resolved, and we can check the bill against it.

Head of OperationsOnline gaming group

Questions about this project

Did this replace the vendor?

No, the AI stays. We check it from the outside, so the same approach works with any vendor.

Why is the checked rate lower than the vendor's?

The vendor counts every conversation the AI keeps away from an agent. We count only the ones where the customer's problem was solved.

What happens to customer data?

Personal data is masked before checking, the AI reads only what a conversation needs, and every lookup is logged.

See what your support AI resolves and what to pay for

Send last month's numbers. We'll show you how we'd check them, and what the vendor should be paid for.

Ask your AI assistant about us

Open our summary in the assistant you use, or copy the prompt.

Read the prompt