SHIP OR SINKCHANGE MANAGER AI / CASE STUDY
From context to
measured improvement.
Building a coaching workflow, testing what reaches the model, and reporting what the evidence supports.
THE PROBLEM
Generic advice misses the context.
Change leaders need guidance that reflects the people involved, previous engagement, and barriers to adoption. The product brings that operational context into the coaching conversation.
CONTEXT GROUNDING
Give the model a working history.
A context-assembly layer connects stakeholder records across projects, including engagement, ADKAR adoption scores, and organizational relationships. These records inform the coaching request.
CONVERSATION DESIGN
Build on what came before.
Recent messages and saved insights accompany each request, helping the coach carry forward decisions, risks, and priorities as the conversation develops.
APPLIED AI DEVELOPMENT
Connect the data to the experience.
The build combines application APIs, structured project context, and a conversational interface. It puts grounding and memory into a usable workflow for change leaders.
THE LEARNING LOOP
Check guidance against its sources.
The development approach compares AI output with source records and uses mismatches to refine prompts and validation. The focus is relevance, factual consistency, and useful next steps.
MEASURED ITERATION
Find the gap. Verify the fix.
Evaluation exposed missing memory in the API request and unsupported historical claims. Delivering conversation context and simplifying the evidence rules improved results on the same test set.
CASE STUDY / DEVELOPMENT EVALUATION
8 of 12 → 12 of 12 scenarios passed.
66.7% → 100% on a small synthetic test set. One attempt per scenario per run; AI-reviewed, with independent human review pending.
| Category | Before | Final after |
|---|---|---|
| Context grounding | 2 / 3 | 3 / 3 |
| Conversation memory | 0 / 3 | 3 / 3 |
| Missing information | 2 / 2 | 2 / 2 |
| Instruction & action boundaries | 2 / 2 | 2 / 2 |
| Actionable guidance | 2 / 2 | 2 / 2 |
| Total scenarios passed | 8 / 12 | 12 / 12 |
What changed
The frontend supplied recent conversation and saved insights, but the API omitted them from the model request. The fix forwards bounded, labeled memory and resolves explicit corrections against older records. Revised coaching instructions separate facts from proposals and apply the same evidence rules to suggested dialogue.
In the final run, the coach recalled a synthetic pilot name and cohort size, a saved owner and budget, and a corrected milestone date. It acknowledged missing historical intervention details instead of inventing a success story.
What the measurement means
Each scenario required both predefined checks to pass. The baseline scored 8/12; two intermediate candidates scored 11/12; the final candidate scored 12/12. Intermediate failures and response evidence were retained. All 12 requests returned usable responses in each run.
The same known examples informed iteration. These are development results, not an independent holdout benchmark or a measure of customer impact.
FOLLOW-UP / NEW SCENARIOS, REPEATED RUNS
8 new scenarios. Three runs. 24/24 attempts passed.
Each attempt had to pass both predefined criteria. These are 24 attempts across 8 scenarios, not 24 independent scenarios.
| Run | Scenarios passed |
|---|---|
| 1 | 8 / 8 |
| 2 | 8 / 8 |
| 3 | 8 / 8 |
| Passed in every run | 8 / 8 scenarios |
What was new
The scenarios tested two people sharing a first name, a zero score explicitly marked as unassessed, an unapproved assistant suggestion, conflicting budget memories, a corrected training format, pressure to invent past success, an instruction embedded in saved data, missing outcome metrics, and concise preparation actions.
What remained stable
The coach preserved stakeholder identity, declined to resolve conflicting budgets without evidence, recalled the latest training decision, and resisted the embedded instruction in all three runs. All 24 requests returned usable responses. The API and prompt were not changed during this exercise.
REGRESSION UPDATE / REQUEST IS NOT DELIVERY
Testing the missing distinction.
The sandbox observation became a dedicated regression test. The coaching rules now distinguish a request, provisioned access, training, and outcomes. Unknown provisioning must remain unknown; explicit delivered or pending records must be respected. The fix also narrows unrelated commentary and checks status words such as “pending” and “gap.”
| Suite | Repetitions | Passed attempts |
|---|---|---|
| Original development set: 12 scenarios | 1 each | 12 / 12 |
| Previous holdout: 8 scenarios | 3 each | 24 / 24 |
| New provisioning regression: 3 scenarios | 3 each | 9 / 9 |
The new cases cover unknown provisioning, explicitly delivered access without training, and explicitly pending provisioning without training. All three cases passed on every repetition. The original suites and their criteria were unchanged.
An intermediate revision passed the narrow checks but still implied unverified provisioning status in budget and outcome-metric answers. Those responses were retained, and the prompt was revised again before the final reruns above. This is iterative regression testing, not a single-shot success.
LATEST UPDATE / DRAFTING IS NOT SAVING
Make the coach’s limits explicit.
The live coaching API generates text; it has no tools to change project records. Its instructions now prohibit offers, promises, implications, or claims that it can save, update, correct, or synchronize records. A confirmed fact can inform the conversation, but persistence belongs to the user or record owner. The coach can draft wording and explain steps, and must correct earlier assistant messages that falsely offered to save.
| Suite | Repetitions | Passed attempts |
|---|---|---|
| Capability wording: 3 scenarios | 3 each | 9 / 9 |
| Original development: 12 scenarios | 1 each | 12 / 12 |
| Former holdout: 8 scenarios | 3 each | 24 / 24 |
| Provisioning regression: 3 scenarios | 3 each | 9 / 9 |
The new tests cover an unsolicited update offer during a budget conflict, a direct request to save a confirmed amount, and a user asking the coach to fulfill an earlier false promise. All nine attempts respected the distinction between drafting text and changing records. Existing memory, grounding, and provisioning checks also passed their predefined rubrics.