Zero-Downtime LLM Migrations: User Simulation, Shadow Traffic, and Canary Rollouts (Part 2)

In Part 1: Failure Modes, Latency Budgets, and CI/CD Evals, we analyzed how model upgrades introduce subtle prompt brittleness, schema drift, phonetic pronunciation errors, and TTFS latency spikes. We also built a Tier 1 declarative CI/CD test matrix using Promptfoo, DeepEval, and statistical hypothesis tests.

However, offline unit tests cannot replicate the messy realities of real telephone calls. Human callers hesitate, speak over the bot, change their minds mid-turn, provide ambiguous inputs, and experience background noise.

Part 2 focuses on live validation: simulating multi-turn dialogues with adversarial user agents, cloning production audio streams into dark traffic shadow runners, orchestrating canary rollouts with automated circuit breakers, and building zero-latency multi-model failover gateways.

[!NOTE] This is Part 2 of our two-part guide on model migrations for real-time voice bots.

If you have not read Part 1: Failure Modes, Latency Budgets, and CI/CD Evals, start there for the foundation on latency budgets and Tier 1 unit matrices.

Tier 2: Synthetic User Simulation (Bot-to-Bot Testing)

Before exposing candidate models to live callers, we subject them to synthetic bot-to-bot dialogue simulations. An adversarial "User Simulator Agent" interacts directly with the voice bot over a simulated WebRTC/WebSocket channel.

1. Generating Adversarial Caller Personas

The user simulator is initialized with a goal and a behavioral persona:

• The Rushed Caller: Uses short, incomplete sentences, interrupts the bot mid-greeting, and demands immediate answers.
• The Indecisive Caller: Changes parameters repeatedly mid-conversation (e.g., changes flight dates twice, corrects passenger names).
• The Noisy / Disfluent Caller: Injects acoustic filler ("uh", "um", "hold on a second, honey what was the number?"), mumbling, and partial speech fragments.

1# user_simulation_harness.py
2import asyncio
3from typing import List, Dict
4
5class VoiceUserSimulator:
6    def __init__(self, persona_prompt: str, goal: str):
7        self.persona = persona_prompt
8        self.goal = goal
9        self.turn_history: List[Dict[str, str]] = []
10
11    async def generate_user_turn(self, bot_response: str) -> str:
12        # Prompt simulator to generate the next realistic user utterance
13        prompt = f"""
14        You are simulating a telephone caller.
15        Persona: {self.persona}
16        Goal: {self.goal}
17        
18        Conversation so far:
19        {self.turn_history}
20        
21        Bot just said: "{bot_response}"
22        
23        Generate your next spoken response. Keep it natural, conversational, and stay in character.
24        """
25        # Call fast simulation LLM (e.g. Claude 3.5 Haiku or Gemini Flash)
26        user_utterance = await execute_llm(prompt)
27        self.turn_history.append({"role": "assistant", "content": bot_response})
28        self.turn_history.append({"role": "user", "content": user_utterance})
29        return user_utterance
30
31    async def simulate_barge_in(self, bot_turn_index: int, elapsed_ms: int) -> bool:
32        # Probabilistically inject an interruption after 350ms of bot audio playback
33        if "Rushed" in self.persona and elapsed_ms > 350:
34            return True
35        return False

2. Multi-Turn Dialogue State Tracking Metrics

At the conclusion of each simulated call, an independent evaluation judge analyzes the complete session telemetry:

• Goal Completion Rate (GCR): Did the candidate model successfully complete the booking, cancellation, or inquiry?
• Turn Efficiency: Did the candidate model complete the goal in $\le 5$ turns without getting trapped in repetitive clarification loops?
• Slot Filling Precision: Were all structured variables (e.g., dates, phone numbers, booking IDs) extracted accurately from messy speech?
• Interruption Recovery: When interrupted, did the bot acknowledge the new intent cleanly without repeating unuttered context?

Tier 3: Production Shadow Traffic (Dark Launching)

Offline simulations validate expected behaviors, but live telephony traffic contains long-tail edge cases that synthetic generation misses. In Tier 3, we implement Dark Launching / Shadow Traffic.

1. The Async Non-Blocking Event Dispatcher

When the streaming STT engine finalizes a user turn, the voice gateway dispatches the payload to the incumbent model on the critical audio path. Simultaneously, a non-blocking background task clones the prompt and session context, dispatching it to the candidate model.

1# shadow_dispatcher.py
2import asyncio
3import time
4from typing import Dict, Any
5
6async def handle_caller_turn(session_id: str, transcript: str, context: Dict[str, Any]):
7    # 1. Critical Live Path (Incumbent Model)
8    live_task = asyncio.create_task(
9        execute_live_llm_stream(session_id, transcript, context)
10    )
11
12    # 2. Async Shadow Path (Fire and Forget)
13    asyncio.create_task(
14        execute_shadow_llm_eval(session_id, transcript, context)
15    )
16
17    # Await and stream live audio packets immediately to caller
18    async for audio_chunk in await live_task:
19        yield audio_chunk
20
21async def execute_shadow_llm_eval(session_id: str, transcript: str, context: Dict[str, Any]):
22    start_time = time.perf_counter()
23    try:
24        # Candidate model snapshot execution
25        candidate_response, tools_called = await call_candidate_model(transcript, context)
26        shadow_duration = (time.perf_counter() - start_time) * 1000
27
28        # Log to ClickHouse telemetry table for divergence analysis
29        await log_shadow_telemetry({
30            "session_id": session_id,
31            "transcript": transcript,
32            "candidate_response": candidate_response,
33            "tools_called": tools_called,
34            "shadow_ttft_ms": shadow_duration
35        })
36    except Exception as e:
37        await log_shadow_error(session_id, str(e))

2. Guardrails for Shadow Tool Execution

[!CAUTION] Candidate models running in shadow mode MUST execute against a mocked tool layer. Never allow shadow executions to trigger real-world mutations (such as initiating credit card charges, updating production databases, or sending live SMS messages).

3. Telemetry Schema in ClickHouse

To detect subtle behavioral regressions across tens of thousands of shadow calls, stream telemetry directly into an analytical store such as ClickHouse:

1CREATE TABLE voice_shadow_telemetry (
2    call_id UUID,
3    timestamp DateTime64(3),
4    turn_index UInt8,
5    user_transcript String,
6    incumbent_model LowCardinality(String),
7    candidate_model LowCardinality(String),
8    incumbent_ttft_ms Float32,
9    candidate_ttft_ms Float32,
10    incumbent_tool String,
11    candidate_tool String,
12    tool_divergence UInt8, -- 1 if tools differ, 0 if match
13    incumbent_tokens UInt16,
14    candidate_tokens UInt16,
15    candidate_error String
16) ENGINE = MergeTree()
17ORDER BY (candidate_model, timestamp, call_id);

Running shadow traffic for 48 to 72 hours provides empirical confidence on P95/P99 latency distributions, tool-selection divergence rates, and token verbosity shifts under real-world production load.

Tier 4: Canary A/B Rollouts and Telemetry Guardrails

Once shadow evaluations confirm zero statistically significant drift in tool calling and latency, we initiate live traffic routing via an automated canary gate.

Automated Circuit Breaker Metrics

During canary shifts, an automated supervisor polls live metrics every 60 seconds. If any of the following guardrail thresholds are violated, traffic rolls back to the incumbent baseline within 5 seconds:

1. P95 TTFT Breach: P95 Time-to-First-Token exceeds 450ms over a rolling 5-minute window.
2. Call Containment Drop: Automated call resolution rate drops by $>2.5\%$ compared to the baseline control group.
3. Human Transfer Spike: Transfers to live human agents increase by $>5\%$.
4. Premature Caller Hangup (Abandonment Rate): Caller disconnects within the first 15 seconds increase by $>1.5\%$.
5. Tool Execution Exceptions: Unhandled schema or runtime tool errors exceed $0.1\%$ of total turns.

Cascaded (STT-LLM-TTS) vs Native Speech-to-Speech Migrations

When evaluating voice model updates, architecture dictates the migration strategy:

For mission-critical enterprise voice bots requiring deterministic business logic, database integrations, and compliance auditing, cascaded pipelines remain the dominant architectural choice because each layer can be isolated, evaluated, and regression-tested independently.

Prompt Re-Calibration and Distillation

When migrating between model generations, copying over the old system prompt unchanged is a frequent source of regression. The new model must undergo systematic calibration.

1. Zero-Shot Constraint Distillation

Older models often required four to six few-shot dialogue examples in the system prompt to enforce concise answers and proper tool calls. In newer models, these few-shot examples consume valuable prefill latency and dilute attention over strict constraints.

1<!-- BEFORE: 1,400 Token Few-Shot Prompt (Old Model) -->
2You are a flight assistant. Keep answers brief.
3Example 1:
4User: When is flight 402?
5Bot: Flight 402 departs at four PM.
6Example 2:
7User: Cancel my seat.
8Bot: {"tool": "cancel_seat"}
9
10<!-- AFTER: 280 Token Zero-Shot Calibrated Prompt (New Model) -->
11You are a voice assistant for flight operations.
12Constraints:
131. Maximum 1 sentence per turn. Under 25 words.
142. Never explain actions. Execute tools directly.
153. Format all numbers, dates, and currencies phonetically.
164. Output zero markdown formatting or bullet points.

By stripping obsolete few-shot examples and replacing them with calibrated negative constraints, prompt prefill latency drops from 220ms to 65ms, recovering critical milliseconds for the voice turnaround budget.

2. Hyperparameter Tuning for Real-Time Streaming

Candidate models must have their inference parameters re-tuned:

• Temperature: Set to 0.0 or 0.1 for deterministic tool calling and predictable phrasing. Higher temperatures increase token diversity but introduce verbosity variance and run-on sentences.
• Max Output Tokens: Set a strict hard ceiling of 64 tokens for normal dialogue turns. If a model attempts to generate an unprompted essay, generation terminates automatically, forcing short conversational cadence.
• Top-P and Frequency Penalty: Slight frequency penalties (e.g. 0.2) prevent repetitive confirmation loops without degrading factual recall.

Multi-Model Fallback and Resilience Architecture

Even after a successful migration, upstream cloud providers experience transient outages, rate limit throttles, and network route degradation. A robust voice gateway implements an automated multi-model cascade:

Implementing Tiered Failover with Timeout Windows

When executing real-time voice inference, the fallback must trigger quickly before the turn latency SLA expires. Setting a hard 350ms timeout on the primary model allows the gateway to switch to the fallback provider or a self-hosted Gemma 4 instance on vLLM / SGLang before the caller perceives dead air.

1# multi_model_orchestrator.py
2import asyncio
3from typing import AsyncGenerator
4
5class MultiModelOrchestrator:
6    def __init__(self, primary_provider, fallback_provider, edge_slm_provider):
7        self.primary = primary_provider
8        self.fallback = fallback_provider
9        self.edge_slm = edge_slm_provider
10
11    async def stream_with_fallback(self, prompt: str, history: list) -> AsyncGenerator[str, None]:
12        # Attempt 1: Primary Candidate Model (350ms TTFT budget)
13        try:
14            async with asyncio.timeout(0.35):
15                async for token in self.primary.stream(prompt, history):
16                    yield token
17                return
18        except (asyncio.TimeoutError, Exception) as primary_err:
19            # Primary failed or timed out. Route to Fallback Provider.
20            pass
21
22        # Attempt 2: Cloud Fallback Provider (450ms budget)
23        try:
24            async with asyncio.timeout(0.45):
25                async for token in self.fallback.stream(prompt, history):
26                    yield token
27                return
28        except (asyncio.TimeoutError, Exception) as fallback_err:
29            # Cloud fallback failed. Route to Local Self-Hosted Edge SLM.
30            pass
31
32        # Attempt 3: Local Edge SLM (vLLM / SGLang on local GPU cluster)
33        async for token in self.edge_slm.stream(prompt, history):
34            yield token

Framework Comparison Matrix

The Complete Migration Checklist

Before deprecating an old model snapshot and completing cutover to a new foundation model, verify each phase:

• Phase 1: CI/CD Baseline Matrix
  • 100+ golden test assertions in Promptfoo/DeepEval passing with $p < 0.05$ significance.
  • Strict JSON schema assertions validated for all function tools.
  • Regex assertions verify zero raw digits, zero dollar signs, and zero markdown leaks.
  • System prompt distilled to zero-shot constraints, reducing prefill tokens.
• Phase 2: Multi-Turn Simulation
  • 50+ multi-turn dialogues simulated against adversarial caller personas.
  • Goal Completion Rate (GCR) $\ge 95\%$.
  • Barge-in audio buffer flush and history truncation verified.
• Phase 3: Production Shadowing
  • 10,000+ live turns cloned asynchronously to candidate model.
  • Tool divergence rate $< 0.5\%$.
  • P95 TTFT $\le 350\text{ms}$ under peak concurrent load.
• Phase 4: Canary Rollout & Fallback
  • 5% >> 25% >> 100% traffic ramp with automated rollback triggers.
  • 350ms multi-model fallback cascade configured to secondary cloud and local vLLM edge SLM.

References

1. Promptfoo Documentation
2. DeepEval LLM Evaluation Framework
3. Braintrust AI Evaluation Platform
4. Langfuse Open Source LLM Observability
5. ClickHouse Documentation
6. Cartesia Sonic Real-Time Voice Synthesis
7. Deepgram Real-Time Speech Recognition
8. vLLM High-Throughput Serving Engine