Optimizing AI-Powered Customer Service Over Telephony: Technical Stack and Performance Enhancements
Introduction
AI-powered voice assistants have the potential to transform customer service over telephony, offering businesses scalable and efficient support solutions. However, one of the biggest hurdles in achieving a seamless user experience is latency. Unlike human conversations, where responses occur within 200–600ms, naive AI-driven systems often exhibit response times of 3–5 seconds, leading to noticeable delays and a subpar user experience.
This paper examines the core technical components of an AI-powered voice assistant, identifies key latency bottlenecks, and explores strategies to optimize response times.
The Latency of a Naive AI-Powered Audio-based Service System
What's typical latency for a voice assistant?
1. Silence Detection (~500ms)
Upon detecting user speech, systems do not immediately begin processing. Instead, it first confirms that the user has completed their sentence, usually simple heuristics like such as requiring 500ms of silence before initiating processing.
2. Speech-to-Text (STT) Conversion (~300–400ms)
Once silence is detected, the spoken audio is converted into text. This process is conducted using automated speech recognition (ASR) systems such as Deepgram. These systems tend to return transcriptions within 300ms. For systems that use cloud-based STT services, this latency can might have another 100ms due to network latency.
3. Large Language Model (LLM) Processing (1.5–3s)
The transcribed text is sent to a language model, such as OpenAI's GPT-4o, to generate an appropriate response. In spot-testing over the API, GPT 4o delivers 75 tokens in around 3 seconds. Smaller, faster models like GPT-4o-mini deliver 75 tokens in around 1.5 seconds, but these can't always be relied on for sophisticated responses.
4. Text-to-Speech (TTS) (~600–700ms)
The AI-generated response must be converted into speech before being relayed to the user. While some TTS such as Cartesia Sonic and Elevenlabs Flash claim generation times of sub-100ms, realworld testing as shown that considering actual latency is closer to 600ms.
Total Latency: ~3 seconds
- Silence Detection: ~500ms
- Speech-to-Text: ~300–450ms
- LLM Processing: ~1.5–3s
- Text-to-Speech: ~600–700ms
Streaming and chunking to reduce latency
How can we use streaming and chunking to reduce voice AI latency?
The first major step to reduce latency is to stream the audio to the STT engine as soon as the user starts talking. Similarly, we'll want to stream the response with intelligent chunking to the TTS engine as the LLM generates the response.
1. Transcription Streaming (-300ms)
Instead of waiting for complete silence before transcription begins, streaming partial audio to the STT engine allows for real-time processing. By overlapping transcription with silence detection, the 300ms transcription delay can be effectively eliminated.
Potential Challenge: Context-aware STT models may lose accuracy if audio is chunked improperly, so this requires careful optimization.
2. LLM Streaming and TTS Chunking (-1000ms)
Instead of waiting for the entire response to be generated, responses can be streamed as they are produced. LLMs such as GPT-4o often return the first few tokens within 600ms. TTS processing can begin as soon as these tokens arrive, cutting down another 100–200ms.
Trade-off: Chunking responses too early can lead to unnatural speech patterns, similarly to STT, careful calibration to maintain a conversational flow is required.
Latency reduction strategies that are not streaming
What other strategies can we use to reduce latency?
1. Turn Detection Algorithms (-400ms)
Replacing fixed silence detection with dynamic turn detection can significantly improve performance. Instead of waiting for 500ms of silence, AI models can predict turn endings within ~150ms and initiate response generation earlier.
2. Audio Caching for Sentence Starters (-500ms)
Another strategy to use is to instruct your LLM to start generating responses with a set of pre-defined phrase like "Sure" "Absolutely" then cache or save these, so that you can skip the TTS step all together.
Warning: People are good at recognizing speaking patterns, so this can actually hurt UX, despite delivering a faster response.
Optimized Performance Breakdown
By integrating these improvements, response latency can be significantly reduced:
| Time (ms) | Action |
|---|---|
| 0 | User stops talking |
| 50 | Server receives final audio |
| 150 | Turn detection algorithm predicts speech end |
| 200 | LLM receives request |
| 700 | LLM streams first 10 tokens (partial response) |
| 750 | Server receives first tokens |
| 800 | Recognizes cached "starter phrase" |
| 850 | User receives cached audio |
Optimized Total: ~850ms Time-to-First-Audio. For platforms like Retell AI or Vapi, this is best-case performance. It takes a lot to get here, and while it isn't quite human-level, response times, it is impressive.
Future Optimization Opportunities: The Quest for 500ms
How can we get the fastest possible response?
Getting to super-human (500ms or faster) response times will require a combination of the following:
1. Continuous Response Generation (-150ms)
Predicting likely responses in the background before the user finishes speaking can further reduce latency. AI Coding Assistants like Cursor are trying to do this with code, but it's a little different. But this approach could shave off another 150ms.
2. Network Optimization (-100ms)
Current providers introduce significant latency by relying on separate vendors for STT, Reasoning, and TTS. This introduces at least 100ms of latency, but often more. By bringing these separate centers into the same server, we can cut this down.
3. LLM Optimization (-250ms)
Using off-the-shelf LLMs like GPT-4o is great, but running customized, on-premise LLMs can reduce latency by 250ms at least.
Putting these all together is how we get to 500ms or faster. Stay tuned to Prim Voices for more updates on how we're pushing the boundaries of voice AI latency.
Conclusion
By implementing these optimizations, AI-powered voice assistants can achieve response times as low as 500ms, potentially surpassing human response times in phone-based conversations.
Achieving human-like response latency will require further advances in streaming AI inference, improved turn detection, and speculative pre-response generation. However, even incremental improvements in these areas will substantially enhance the user experience, paving the way for truly conversational AI-powered customer service.