Measuring the Return
Designing the self serve analytics that let a customer put a number on their training, and prove it pays, even without production data.
Everyone asks the same question, is it working
Customers buy Symtrain to train agents faster and better through Syms. But every training leader eventually faces the same question from the people who hold the budget, is this actually working, and is it worth what we pay for it.
Without a way to connect training to business results, that answer is a gut feeling. The goal of this work was to design self serve analytics that quantify training effectiveness and ROI, tell a manager whether an agent is ready for the floor, and crucially, do it even when a customer has limited or no production data.
One chain, from practice to profit
The whole feature hangs on a single idea. Training only matters if you can trace it all the way to money. So I anchored the design to one equation.
Every metric in the feature had to earn its place somewhere on this chain.
How the leaders attach a dollar
I studied how conversation intelligence platforms like Cresta, Observe.AI, and Level AI turn behaviour into business value, so Symtrain could learn from proven patterns rather than reinvent them.
Attribution modelling links a habit to a result, showing that an empathy habit on 80 percent of calls lifted CSAT by 23 percent.
Opportunity cost makes the gap tangible, a team missed 500 upsells and lost 12,000 dollars by not following the winning behaviour.
Inferred CSAT scores every call from its emotional trajectory, instead of relying on a handful of survey responses.
Figures above are the vendors' own published examples, studied as reference patterns, not Symtrain results.
The leaders need every live call. Most customers cannot give them one.
Conversation intelligence tools ingest one hundred percent of production call data. Many Symtrain customers do not have that, or will not share it. So the real question became, how might we prove ROI from the data Symtrain already owns, the Syms themselves.
Four groups of metrics that build to ROI
I organised the metrics into four groups, moving from simple usage all the way to financial impact, so a customer could climb from activity to outcome.
Engagement and velocity
- Syms created
- Syms started
- Syms executed
- Attempt frequency
- Abandoned rate
- Time on product
- Time to competency
- Cost per trainee
Efficiency and quality
- Time to proficiency
- Average QA score
- Case resolution rate
- Trend of customer concerns
- CSAT score
- Sentiment trajectory
Performance and sentiment
- Top performer benchmarking
- Progress across attempts
- Behavioural adoption rate
Financial impact and ROI
- Attribution modelling
- Missing revenue
- Predicted opportunity cost
Metrics, defined precisely
A metric is only useful if everyone computes it the same way. I defined the core ones from first principles, so they could be built and trusted.
Time to competency
The time from a learner's first engagement to the moment they hit a defined competency score.
Cost per trainee
What it actually costs to bring one agent to readiness.
Retention rate
How much skill a learner keeps after training, not just on the day.
Dynamic Syms cut time to competency in half
Illustrative comparison used to model the metric. Real figures vary by client and scenario.
ROI without a single production call
The heart of the work. If Symtrain cannot see live calls, it can still score behaviour inside Syms and model the financial impact through attribution. The logic runs like this.
The unlock is a simulated outcome score. Each Sym attempt gets a score for the behaviours that drive revenue or protect it. As an agent improves across attempts, that score becomes a predicted revenue impact, a number a manager can take to leadership, built entirely from training data.
Turning a Sym score into a sentence a manager can use
Illustrative model outputs that show how the feature translates a simulated score into a business claim. Not verified client results.
An answer to the budget question
Answered the hardest objection
Gave Symtrain a way to show ROI, the question that decides renewals and expansion, without depending on client call data.
A framework and defined metrics
Four KPI groups and clean definitions for time to competency, cost per trainee, and retention that the team could build against.
A real differentiator
ROI without production data is something rivals who depend on live call feeds cannot match cleanly.
As UX Researcher, I
- Framed the ROI question as a research and design problem
- Anchored the feature to the training to financial impact chain
- Studied how conversation intelligence leaders attribute value
- Separated vendor marketing claims from reusable patterns
- Defined a four group KPI framework
- Modelled time to competency, cost per trainee, and retention
- Designed the attribution logic for a no production data path
- Turned simulated scores into predicted revenue impact
- Built worked examples a manager could actually use
- Compared Dynamic Syms and manual Syms on time to competency
- Documented the model for product and leadership
- Kept every claim clearly labelled as predicted or illustrative
Training only counts when you can count it.
The job was not to prove training works in the abstract. It was to give a manager a number they can defend, from the only data they actually have.
← Back to all work