<!-- Canonical: https://www.aigentcy.com/services/ai-testing/ -->

# Measure what your AI gets right

We test your AI on past cases before launch and check its live work every day after, whether you built it or bought it.

[Book a 30-minute call](https://www.aigentcy.com/contact/)[See the 2-week assessment](https://www.aigentcy.com/ai-readiness-assessment/)

Support AI checks, multi-brand online gaming group

53% claimed to **36%**

of support conversations resolved by the AI, checked every morning. The client stopped paying for and staffing around resolutions that hadn't happened, and sized its team for the real queue.

[Read the support AI case study](https://www.aigentcy.com/case-studies/support-ai-evaluation/)

## What independent checks found

When a vendor reports its own results, your bill and your staffing plan rest on its number. Checked results let you pay for what the AI resolves and staff for the queue that's left.

Multi-brand online gaming group**Support AI and release checks, before and after**

### Support AI, first week of daily checks

Vendor's count**53%**

Checked by us**36%**

**About 38** conversations the vendor counted as resolved still went to an agent that week.

-   **223**conversations checked in the first week
-   **288 of 297**checks that matched the verdict a person gave
-   **165**real customer cases tested before launch
-   **6**brands checked every weekday

### Release checks, at handover

Tester check**2 days**

Automated check**18 min**

**No waiting** on a tester pass before a release. Testers spend the time on new features.

-   **39 of 42**checklist items automated for brand 1
-   **28 of 37**checklist items automated for brand 2
-   **4**releases a week across the 2 brands
-   **3**device types checked: desktop, Android and iPhone

## What changes for your business

The support AI, judged on the vendor's own number before, and on our daily checks after.

| Measure | Before | After |
| --- | --- | --- |
| What the bill is based on | The vendor's own count | Resolutions that happened |
| What the agent rota is sized on | The vendor's own count | The queue left for agents |
| Conversations checked | Spot checks, with no agreed rules | Every conversation, every morning |
| Counted resolutions that hadn't happened | Unknown | Found and left out each morning |

## Testing we've delivered for customers

The support checks and the release checks are both live.

1.  ### Daily checks on a support AI
    
    Real customer cases tested before launch, then every live conversation checked each morning. Leadership sees the checked resolution rate next to the vendor's.
    
    Works with The vendor's help desk, company sign-in
    
    [The bill is now checked against real resolutions](https://www.aigentcy.com/case-studies/support-ai-evaluation/)
    
2.  ### Automated release checks
    
    Every item on the release checklist that a browser can check runs on its own in the client's build pipeline, with a pass or fail for each item.
    
    Works with Jenkins, Playwright
    
    [The client's QA team runs and extends it](https://www.aigentcy.com/case-studies/release-test-automation/)
    
3.  ### Checks on the live sites
    
    A read-only set of checks against the live sites, so a broken sign-in shows up before customers report it.
    
    Works with Jenkins, the live sites
    
    Runs every 15 minutes
    
4.  ### Tests from recordings
    
    Testers record a flow in Chrome and turn it into a test, so the QA team adds to the checks without us.
    
    Works with Chrome, Playwright
    
    Handed to the client's QA team in week 2
    

## What automated testing is worth

2 brands release twice a week, and each release waited on about 2 tester-days of checks.

-   **Week 1**first checks running in the client's pipeline

| What we count | How | Result |
| --- | --- | --- |
| Tester check per release | About 2 tester-days | 2 days |
| Releases each week | Twice a week, for each of 2 brands | 4 releases |
| Tester time on repeat checks | 4 releases at 2 days | 8 tester-days a week |
| Over a working year | 8 tester-days for 50 weeks, or 3,200 hours | 400 tester-days |
| Tester time back each year | 400 days at €350 a day | €140,000 |

## How we deliver it

1.  01
    
    ### Agreed rules first
    
    What counts as resolved, wrong or unsafe, written down with your team before checking starts.
    
2.  02
    
    ### Clear cases decided by rules
    
    A conversation that went to an agent or came back is decided by a rule. AI judges the rest.
    
3.  03
    
    ### Checked against your team
    
    Our checks are compared with cases your team already judged before any number counts.
    
4.  04
    
    ### Personal data masked
    
    Before any conversation is checked, with every access logged.
    

## Questions about AI testing

### Related reading

-   [Support staffing and spend based on real resolutions](https://www.aigentcy.com/case-studies/support-ai-evaluation/)
-   [Releases that no longer wait on testers](https://www.aigentcy.com/case-studies/release-test-automation/)
-   [Support AI resolution checker](https://www.aigentcy.com/tools/support-ai-resolution-checker/)

Does it work with AI we bought?

Yes. We check results from the outside, so it works for a vendor's bot as well as for systems you or we built.

Why was the checked rate lower than the vendor's?

The vendor counted every conversation the AI kept away from an agent. We counted only the ones where the customer's problem was solved. That's the number the bill is checked against.

Does release automation replace the testers?

No. It takes the repeat checks off them, so they spend the time on new features and the cases a script can't judge.

## Pay for the results your AI delivers

Tell us what you run and how it's billed. On a 30-minute call we'll show you how we'd check it and what the first week would tell you.

[Check your AI's numbers](https://www.aigentcy.com/contact/) [See the 2-week assessment](https://www.aigentcy.com/ai-readiness-assessment/)

## Other services

-   [**AI workflow automation**Time saved on data entry, checks and follow-ups](https://www.aigentcy.com/services/process-automation/)
-   [**AI integration services**AI in the tools your team already uses](https://www.aigentcy.com/services/ai-integration/)
-   [**Legacy system modernisation**Replace rented and outgrown systems](https://www.aigentcy.com/services/legacy-system-modernization/)
-   [**Private AI and AI cost control**One gateway, team budgets, data kept private](https://www.aigentcy.com/services/private-ai/)
-   [**AI consulting and enablement**Goals, tested tools and training for each team](https://www.aigentcy.com/services/ai-consulting/)
-   [**AI governance**Owners, permissions and records for AI in use](https://www.aigentcy.com/services/ai-governance/)
