Value Realization Playbook

Measurement

Total calls every inbound call at live locations

1,264

100.0%

Not engaged no speech or keypad response; excluded from engaged-caller measures

178

14.1%

Engaged callers spoke or pressed a key

1,086

85.9%

Live agent opt-out at opening asked for a person before stating a need

9.1%

Non-self-service intent something the agent is not built to do. No completion rate is counted against it.

5.6%

By-design transfer routed to a person by design

5.7%

Self-service intent something the agent is built to do. The denominator for every self-service rate.

864

79.6%

Completed and contained

529

61.2%

Completed and transferred task done, then a second need

4.6%

Attempted and transferred transfer reason underneath

240

27.8%

Hung up during self-service

6.4%

Worked example from the synthetic population, Period B. The identities: 178 + 1,086 = 1,264; 99 + 61 + 62 + 864 = 1,086; 529 + 40 + 240 + 55 = 864; the five transfer reasons sum to 240. If the source cannot classify every engaged caller into the buckets, the mismatch is flagged; unclassified calls are never forced into a bucket to make the accounting reconcile.

Include every eligible request for the function in its denominator, including failed identification, lookup errors, incomplete attempts, and hang-ups. Document how configured business rules determine eligibility and classify required transfers.

Counting UnitCount each call once, even when it includes several tasks.

Completion and ContainmentA completed request can still transfer for a second need. Record both facts.

Function TypeConfirm and FAQ are informational. Cancel, reschedule, and schedule are transactional. Interpret results against the function’s purpose.

Transfer CodingValidate transfer reasons against transcripts and state the validation sample size.

Rate	Formula	Denominator
Engagement rate	Engaged callers divided by total calls	Total calls
Containment	Completed and contained divided by self-service intents	Self-service intents
Completion	(Completed and contained + completed and transferred) divided by self-service intents	Self-service intents
Attempted and transferred share	Attempted and transferred divided by self-service intents	Self-service intents
Transfer reason share	Calls carrying the reason divided by attempted and transferred	Attempted and transferred
Function containment and completion	The same two formulas inside the function's own intents	The function's intents
Authentication rate	Authenticated divided by callers who reached authentication	Callers who reached authentication
Estimated uplift of a fix	Calls affected divided by self-service intents, as points. A ceiling, stated as one.	Self-service intents

Completion standard80% completion is the standard for every application. It is a named standard, not a banded range and not set account by account.

Post-Launch opening expectationSet scenario by scenario from the nature of the solution, for both containment and completion. It is an opening expectation, not an overall acceptance threshold.

Platform default thresholdsUsed only when the account has set no target. Containment: below 60% is below target, 68% to 75% at target, above 82% top quartile. Authentication failures as a share of transfers: above 25% below target, 15% to 25% at target, under 10% top quartile. Business-rule transfers as a share of transfers: above 40% below target, 25% to 40% at target, under 20% top quartile. Engagement: below 80% below target, 85% to 92% at target, above 95% top quartile. A value between two named bands is read as between them.

Account targets winA documented account target replaces the default for that account and function. Nothing is read across accounts.

Investigation and Follow-Through

Plan launch and major-release reviews for day one, days two to three, and the first week. The available call volume determines which conclusions each review can support.

Day 1: delivery mechanicsRaw counts only. It answered, the task posted, the ticket carried what the caller said. No containment, no completion, no percentages: on a thin day one miss moves a rate by twenty or thirty points, so a rate quoted on day one cannot be defended.

Day 2 to 3: patternsThe same checks over two or three days of population. Defects sort into known recurrence, variation on a known defect, and new. A recurrence carries weight with engineering that a one-off does not. Mechanism statements become assertable.

Week 1 to day 10: ratesThe first point where rates survive scrutiny. Containment over live actions only; opt-out figures verified against transcripts; every count still under every rate.

During week one, review five to ten calls together in a 30–60 minute session focused on one pattern from the data.
Everyone working on an application listens to some production calls.
The project team reviews the findings log weekly from day one. Steady-state accounts get analysis ahead of the business review to find the next opportunity.

Record each finding in a shared log with its population count and supporting evidence.
Prioritize fixes by customer impact, affected volume, and expected improvement.
When the data cannot answer a question, the suggestion can be a KPI or tagging enhancement so that next month it can.
Record the measured result after each fix. Keep other customers’ results out of account-specific reports.

Each ticket records the baseline, reporting window, denominator, acceptance criteria, and verification owner. Confirm the deployment date with the owner and document changes that affect comparison.

Select the function and outcome to investigate, then compare its performance with relevant, documented reference results.
Rank the transfer reasons inside that slice.
Review calls and raw reporting fields to reconstruct the failure.
Count the issue across the eligible population before estimating its impact.
Verify hypotheses with engineering and the customer’s system or data owner.
For business-rule transfers, review downstream outcomes where authorized recordings, transcripts, and system data are available. Use the findings to inform a discussion with customer operations.
Findings land in the log, the team triages them, fixes become tickets, and the readout carries the counts.

Keep agent behavior, caller behavior, configured business rules, integrations, authentication, and data quality separate. Confirm causes with the system owner. Record unresolved causes and unassigned actions explicitly.

Analysis Formats

Play	Reach for it when	It produces
AI Agent Scorecard	A deployed agent needs its standardized performance read: a period, a release, a business review.	The funnel, the four buckets, outcomes by function, transfer reasons, a fix matrix ranked by uplift, and the estimated potential. Internal, customer, and engineering versions from one set of figures.
Post-Launch review	A launch, a release, or a production fix. Every call in the window, not a sample.	A scored workbook (1 to 5 on every call, Success or Rework on every reported field), a day readout, and a period readout with the Showcase, the Issue Spotlight, the KPI rework, and an Evidence Index.
Period comparison	What changed. Did containment or any KPI drop. A before-and-after on a release.	Every KPI in both windows with the same definitions and denominators, movements in points and counts sorted by magnitude, a mix check, and the detection limits.
Failure analysis	A function misses the standard, or a transfer reason grows.	What failed, why, how often, the business effect, and the supported next action, with the owner named only when the record names one.
Value defense	A customer questions the value, a renewal is at risk, or a champion has left and the story must be re-established.	The value headline, each lever with the data behind it and measured marked apart from estimated, the ROI view with the math shown, and the inputs still open.
ROI and break-even model	A renewal or business case needs spend and value side by side.	Monthly value by lever, net value, ROI, and payback, every input sourced and marked measured or estimated. Spend and savings are never invented; missing inputs are asked for.
Second opinion	A draft is about to go to a customer executive or to leadership.	A prioritized list of the questions the draft does not answer yet, the soft claims, and what to add or verify, read as a skeptical executive, a customer-facing reviewer, and an engineer. If the draft holds, it says so.
Prompt architect	A loose request, raw customer feedback, or an ad hoc data question that needs a clean, repeatable run.	A structured prompt: role and context, the verified data source and window, the questions, the expected output and audience, and the directives, with the instruction kept separate from pasted customer text.

Evidence and Quality Checks

SourcedDirectly supported by a transcript, source row, system output, written request, or documented configuration.

CalculatedDerived from sourced figures by reproducible arithmetic.

InferredA reasonable explanation the evidence supports but does not establish.

UnknownThe record does not establish the answer. It stays unknown; it is not converted into an assumption to finish the analysis.

A material figure cannot be traced to a source or a reproducible calculation.
The source population does not reconcile.
A percentage cannot be reproduced from its stated counts.
The denominator is unknown or changes across comparison periods.
Two source records materially conflict and the conflict is unresolved.
A required configuration fact is assumed rather than established.
A causal statement is stronger than the evidence.
A project status is inferred rather than documented.
A comparison uses materially different populations without saying so.

Reconcile totals, reproduce rates, check duplicates and exclusions, and confirm that comparison periods use consistent definitions. Resolve material mismatches before drawing conclusions.

ObservedThe record shows the result. A metric moving proves that the metric moved.

Supported explanationSeveral pieces of evidence point to a likely explanation.

Demonstrated effectThe evidence materially isolates the change from credible alternatives.

Proven causal resultThe evidence survives the strongest credible competing explanations.

Did the denominator move?A rate can change when only its denominator changed. Check volume and caller mix before attributing anything.

Did the definition or configuration change?A release, a rule change, or a tagging change inside a window splits the window at its date.

Is one segment driving it?One location, one function, one week. If the movement sits in one place, say so.

Is there a credible other explanation?An external dependency, a list change, a seasonal pattern. If it cannot be ruled out, it stays in view beside the conclusion.

Would it survive the strongest opposing example?Show the opposing exhibit next to the finding and ask whether the finding still stands.

Is the counterargument real?A counterargument is never manufactured when the record does not support one.

Show how much one call changes a rate. With 156 intents, one call changes the result by 0.6 percentage points. Small movements need more observation before they support a trend.

Comparisons use populations, not selected examples. If sampling is required, it is labeled as a sample with how it was selected, and never presented as a population finding.
Never a period average as a single-day value, never one day as a period, never a stated range that an included day violates, never mixed windows without labels, never a pre-and-post anchored to a date the record does not establish.
Every period labeled. One window per output, or every window named.

Reporting and Delivery

Internal TeamsInclude counts, failure categories, ticket references, open questions, and evidence that limits the conclusion.

CustomerLead with the customer’s outcome, business impact, and next action. Include material limitations. Keep internal debate and other customers’ information out.

ExecutiveState the result, magnitude, business impact, constraint, and decision required. Include methodology only where it affects interpretation.

Numerical ReviewProvide the source, population, exclusions, numerator, denominator, calculation, reporting window, and reconciliation checks.

Inboundengaged callers; live agent opt-out; self-service intent; non-self-service intent; by-design transfer; completed and contained; completed and transferred; attempted and transferred; hung up during self-service; transfer reason; authentication; Post-Launch.

Outbound campaignsAMD; AI AMD; Asterisk; Live Answer; Transferred; calls placed; connection rate; campaign; batch.

Match the deliverable to the request and audience: scorecard, readout, workbook, executive summary, or customer communication.
Every formal analysis ships with a calculations workbook: source rows traceable, every figure a formula, denominators visible, reconciliation checks included, assumptions explicit, samples labeled, reported figures tied to the final artifact.
Internal and customer versions are separate files. One artifact never serves incompatible audiences.
Use short headings, readable tables, consistent alignment, and space between sections.