From 9 Seconds of Voice AI Latency to 1.5 Seconds: Building an In-House Voice AI System

Nine seconds of silence. That's how long a caller waited after asking our AI assistant a simple question like "What's the TB test process?" long enough that most people would hang up, assuming the call had dropped. We got that down to about 1.5 seconds. This is the story of how a straightforward platform-integration task turned into designing an in-house Voice AI architecture from scratch and the latency problem that became the real engineering challenge. Quick summary: What started as a VAPI integration turned into building an in-house Voice AI stack Twilio for telephony, Deepgram for speech-to-text, Cartesia for voice after customization limits made the managed platform too rigid. The naive pipeline had ~9 seconds of silence before the caller heard anything. The fix wasn't making things faster overall; it was optimizing for time-to-first-audio instead of total response time. Getting to ~1.5 seconds took conditional RAG (skip retrieval when it's not needed), sentence-level streaming to text-to-speech, warm connections, and on-device embeddings. Owning the orchestration layer yourself means you now own barge-in, turn detection, guardrails, and every edge case a managed platform used to absorb for you. It Started with VAPI The project began as a task to integrate VAPI, a Voice AI platform for building conversational phone agents, into our existing system. The goal: automate user onboarding, so callers could get answers and guidance from an AI assistant instead of waiting for a human agent. The integration went well. Then the client asked for something VAPI couldn't easily give us — a more natural, expressive voice — and one workaround led to another, until it became clear we'd eventually need to build the platform ourselves. When Requirements Changed The project took an interesting turn when the client requested additional customizations. One of the major requirements was a more natural and expressive voice capable of conveying emotions during conversations. At that time, fulfilling these requirements directly through VAPI was not as straightforward as we needed. To address this, I implemented a temporary solution. I retrieved available voices from Cartesia, stored their voice IDs in a JSON configuration, and integrated Cartesia directly during the voice selection process. This was only one example of the customizations the client wanted. As the number of customization requests increased, it became clear that relying entirely on a third-party platform would eventually limit our flexibility. Like many developers experience when working with managed platforms or pre-built solutions, customization becomes increasingly difficult as requirements become more specific. That led to a simple question: Why don't we build our own Voice AI platform? Estimating an In-House Voice AI Solution Once the idea was proposed, the first step was estimating the cost and evaluating the technologies required. The first platform that came to mind was Twilio. Whenever telephony and phone systems are discussed, Twilio is usually one of the first names that appears in the conversation. However, telephony was only one piece of the puzzle. Voice AI systems must handle challenges such as: Turn detection Interruptions (barge-ins) Conversation flow management Speech recognition Speech generation While researching these areas, I explored Deepgram, which is widely used for Speech-to-Text (STT) processing and converts raw audio into text using deep learning models. As I continued investigating, I realized that platforms such as Twilio, Deepgram, and Cartesia collectively provided many of the capabilities that VAPI offered. This led to another important decision. Should we simply use one platform? Not really. If we relied entirely on a single platform, we would be back to the same integration approach we had already implemented with VAPI. Should we combine multiple platforms? That brought us back to the original question: cost. I created a cost estimation document comparing different combinations and presented the options to the client. Interestingly, the client chose the combination that could be described as the "best of the best": Twilio for telephony Deepgram for Speech-to-Text Cartesia for Text-to-Speech and voice generation The goal was to leverage each platform for what it does best while maintaining full control over the orchestration layer ourselves. Realizing the Scale of the Project Initially, I assumed this would be another integration project. That assumption changed quickly. My lead pointed out that integrating three major platforms into a unified Voice AI system was not a simple task. The project would require designing an architecture capable of coordinating multiple real-time services while maintaining a natural conversational experience. As I explored the requirements further, one challenge stood out immediately: latency. Voice conversations are highly sensitive to delays. Even a few seconds of waiting can make an AI assistant feel slow, unnatural, or broken. The second major challenge was retrieving company-specific information efficiently through a RAG (Retrieval-Augmented Generation) pipeline. Designing the Architecture I started by designing the overall architecture. The first decision was that Twilio would never communicate directly with Deepgram or Cartesia. Instead, our in-house Voice AI platform would act as the central orchestrator for the entire conversation. For every phone call, Twilio establishes a bidirectional media stream with our Voice AI server using WebSockets. Audio flows in both directions in real time. A simple way to think about it: Twilio acts as the phone line, while our Voice AI server becomes the intelligence layer that coordinates the entire pipeline. The pipeline looks like this: Deepgram → LLM → Cartesia The system also needed the ability to: Transfer calls to real human agents during emergencies Redirect calls to different AI assistants depending on the situation Handle interruptions and conversation transitions smoothly The call flow works roughly as follows: Twilio receives the incoming call. Twilio sends a call-connected webhook to our Voice AI system. A greeting is generated and sent back through Cartesia. Twilio streams the caller's audio to our server. The audio is forwarded to Deepgram for transcription. Deepgram returns text transcripts. The transcript is processed and routed according to the user's intent. The selected context is sent to the LLM. The generated response is converted into speech using Cartesia. The audio is streamed back to Twilio and played to the caller. Once the transcript arrives, the system determines which path the conversation should follow — casual conversation, company-related questions, or appointment-related requests. For general conversation, the transcript and conversation history are sent directly to the LLM. For company-related queries, relevant information is first retrieved through our RAG service before being sent to the LLM. For appointment-related interactions, appointment-specific context is gathered and included before generating a response. Implementing RAG Since the system needed access to company knowledge, implementing a RAG pipeline became necessary. The overall process was straightforward: Company documents are divided into smaller chunks. Embeddings are generated for each chunk. The embeddings are stored in a vector database. User queries trigger a vector similarity search. Relevant information is retrieved and sent to the LLM as context. This allows the AI assistant to answer questions using company-specific information instead of relying solely on its general training data. While the implementation concept sounds simple, making retrieval fast enough for real-time voice conversations introduced its own set of challenges. And that is where the real latency optimization journey began. The Real Problem: Latency Voice conversations are different from normal applications. A web application can take a few seconds to load something, and the user might tolerate it. A phone conversation is different. When a caller stops speaking, they expect the AI to respond almost immediately. The number that matters most is time-to-first-audio: how long the caller waits before hearing the first part of the AI's response. Our initial pipeline was essentially sequential: Caller stops talking ↓ Wait for final transcript ↓ Generate query embedding ↓ Search vector database ↓ Wait for the complete LLM response ↓ Send response to TTS ↓ Start playback Each individual step seemed reasonable. Together, they made the assistant feel slow. A typical knowledge-based request looked roughly like this: Individually, none of these numbers looked terrible. Together, they could result in roughly nine seconds of silence after a question such as "What's the TB test process?" And nine seconds of silence on a phone call feels much longer than nine seconds on a webpage. That became the real challenge. Optimize for Time-to-First-Audio, Not Total Generation Time The design decision that changed the system was simple: we stopped optimizing for the total time required to generate the response and started optimizing for time-to-first-audio. Consider two scenarios. In the first, the AI generates a four-second response but starts speaking after 1.5 seconds. In the second, it generates a two-second response but doesn't start speaking until four seconds later. The first one feels significantly faster. The caller doesn't care that the AI is still generating the rest of its answer while it is already speaking. What matters is that the conversation continues naturally. This changed the way I thought about the entire pipeline. RAG and Voice Want Opposite Things RAG wants more context. Voice wants less waiting. If we retrieve information, wait for the complete LLM response, and only then generate the speech, the knowledge retrieval path becomes one of the slowest parts of the conversation. And not every conversation needs RAG. A caller saying "Hi, how are you?" shouldn't trigger a vector search. Neither should "Okay." or "Thanks, that's all." So the live pipeline became more intentional — the routing decision from earlier now determines whether retrieval even happens at all. The important idea here: RAG is a branch of the conversation, not the default path. That was both a latency optimization and a quality improvement. Stop Waiting for the Full LLM Response The first implementation treated every conversation turn almost like a batch process: Retrieve → Generate → Speak The caller wouldn't hear anything until the final step. That doesn't work well for voice. For knowledge-based questions, retrieval still needs to happen before the grounded answer can be generated. But once the relevant context is available, the LLM can start streaming its response. Instead of waiting for the entire answer, we buffer the generated text until we have a complete sentence. As soon as that sentence is available, it is sent to Cartesia through the live connection. So instead of waiting for the full answer — "The TB test is required before your onboarding process can be completed. You can complete it at…" — the system can begin speaking "The TB test is required before…" while the rest of the answer is still being generated. The pipeline therefore became something closer to: Twilio audio ↓ Deepgram ↓ Partial transcripts + end-of-turn ↓ Cheap router ↓ ┌───────────────┐ │ Small talk │ │ Knowledge │ │ Tools │ └───────────────┘ ↓ Retrieve context when required ↓ LLM streaming ↓ Sentence buffer ↓ Cartesia ↓ Twilio The "cheap router" is a fast, lightweight classifier that looks at the transcript and decides which path the conversation should take — small talk, knowledge question, or tool call — before anything expensive like retrieval or the LLM gets involved. It has to be fast and low-cost since it sits in the critical path before every single response. The three major components — Deepgram, the LLM, and Cartesia — were no longer simply running one after another. They started overlapping. That distinction made a huge difference. Reuse Connections This might sound like a small implementation detail. It wasn't. The first version was paying connection setup costs repeatedly. The LLM connection could become idle and require another connection setup. Text-to-speech was also opening a new request for individual responses. Those delays might not be obvious in a local development environment. They become much more obvious when you're having an actual phone conversation. We changed the connection strategy. Connections to the LLM stay warm across turns, allowing subsequent requests to reuse an existing session rather than repeatedly paying connection setup costs. For Cartesia, we use a persistent WebSocket for the lifetime of the call rather than creating a new HTTPS request for every sentence. The server also maintains a small pool of warm sockets so that the initial greeting doesn't have to pay the full connection setup cost while the caller is already waiting. This is not the most exciting part of building a Voice AI system. But these small delays add up. Sometimes the difference between a system that feels instant and one that feels slightly sluggish is hidden in hundreds of milliseconds that nobody notices individually. Don't Run RAG When You Don't Need It A live onboarding call contains much more than knowledge questions. A caller might say "Hi, how are you?" or "Okay." or "Can I book Thursday at 2?" None of these should automatically trigger a vector search. Only factual, company-specific questions should enter the retrieval path. This improves latency, but it also improves response quality. Running RAG for "Thanks." could retrieve unrelated company information and provide unnecessary context to the LLM. Likewise, an appointment request shouldn't make the model search through company documents when what it actually needs is live appointment information. Again, the important architectural principle is: retrieval is conditional. Hide Embedding Latency Another interesting optimization came from looking at when we generated query embeddings. Initially, embedding started only after speech-to-text confirmed that the caller had finished speaking. That meant we were already paying the end-of-turn latency before even beginning the embedding step. But during a real conversation, we receive partial transcripts while the caller is still speaking. Those partial transcripts can be useful. Once enough meaningful words are available, we can speculatively generate the embedding from the current hypothesis. By the time the end-of-turn signal arrives, the embedding may already be available. We also introduced caching for embeddings. Filler words such as "um" and "uh" don't meaningfully change the query, so they can be removed before generating the cache key. That means questions such as "um what is onboarding" and "what is onboarding" can share the same cached embedding. The vector search itself was never the biggest bottleneck. The embedding step was. For live retrieval, we eventually moved query embedding on-box using a small local model, bringing that part of the process down to roughly 10–30 milliseconds, instead of introducing another cloud round trip into the critical path. Keep the Prompt Small Voice answers should generally be concise. Sending huge document chunks into the LLM doesn't just increase the amount of information the model has to process — it can also make the assistant sound unnatural. Imagine calling a company and asking a simple question, only for the AI to start reading an entire policy document back to you. That's not a good voice experience. So we keep the retrieved context intentionally small. We used a small top-k, limit the size of retrieved chunks, and use a faster model for spoken knowledge responses where appropriate. We also used a similarity threshold — if the retrieved information isn't sufficiently relevant, we don't force it into the prompt. Sometimes "I don't have that detail. I can connect you with someone on the team." is much better than a slow or potentially incorrect answer. This is one of the biggest differences between RAG in a chatbot and RAG in a voice system. In a text interface, additional context is relatively cheap. In a phone conversation, additional context can directly affect how long the caller waits before hearing the first word. The Query Isn't a Search Box There is another problem with RAG in Voice AI that isn't immediately obvious: the query isn't typed by a user, it's produced from speech. That means the retrieval system has to deal with transcription errors and conversational context. Imagine the caller says "How long does that take?" That sentence by itself isn't very useful as a search query. But perhaps the previous conversation was about a TB test — the real question is "How long does the TB test take?" Speech recognition can introduce another problem too. A caller might say "TB test" and the transcription might produce "TV test." Sending that raw transcript directly into vector search can reduce retrieval quality. Instead of adding another expensive LLM call just to rewrite the query, we introduced a lightweight rewrite layer. It can: Fix known transcription confusions. Detect the active topic from recent conversation turns. Expand short contextual questions. Add relevant conversational context to the search query. For example, "How long does it take?" can become something closer to "How long does the TB test take?" The important part is that this happens without adding another expensive model call to the critical path. Prepare Knowledge Slowly, Answer Quickly Another architectural decision was separating knowledge ingestion from live retrieval. These are two completely different workloads. Document ingestion can take time. A phone call cannot. When a document is uploaded, we can: Divide it into meaningful chunks. Generate embeddings. Store those embeddings in Postgres using pgvector. Associate them with the appropriate AI assistant. That work happens during ingestion rather than while someone is waiting on a phone call. The live retrieval path should be much simpler: User query ↓ Query embedding ↓ Vector search ↓ Small context block ↓ LLM ↓ Streaming response The phone call should never have to wait for PDF parsing, document chunking, or other ingestion work. This separation is what makes RAG practical for a real-time voice system. Assistant scoping is important here as well. We don't search the entire company's knowledge base for every question — we search the documents associated with the specific assistant handling the call. Apart from being a correctness issue, searching unnecessary data also means doing unnecessary work. A Phone Call Isn't Request/Response Once the basic pipeline was working, several problems became apparent. Turn detection. If the AI waits too long after a caller pauses, the conversation feels slow. If it responds too quickly, it can interrupt the caller. We use conversational speech-to-text with an end-of-turn signal, along with an eager end-of-turn mechanism that allows the system to begin responding slightly earlier — saving hundreds of milliseconds. But it also introduces another problem: the same utterance can sometimes arrive more than once, with the second transcript containing additional words. The orchestrator therefore needs to understand whether the new event is the same utterance, a continuation, or a completely new question. Otherwise, the assistant can end up answering the same question twice. Barge-in. Real people interrupt. When the caller starts speaking while the AI is talking, the system needs to stop the current response. We abort the current LLM turn, cancel in-flight speech, and instruct Twilio to discard leftover audio. But we don't tear down the Cartesia connection — reconnecting the TTS connection during a call would introduce another latency problem. After an interruption, a short backoff also prevents the pipeline from constantly starting and stopping if the caller and assistant talk over each other. Backchannels. Then there are the small things people naturally say while listening — "Okay.", "Mm-hmm.", "Got it." These aren't necessarily interruptions. But "No." or "Stop." might be. The system therefore needs to distinguish between conversational backchannels and actual interruptions. These details aren't particularly impressive in an architecture diagram. But they're the difference between a Voice AI demo and something that people can actually use on a phone call. Tools Have to Cover Their Own Latency Not every interaction is a knowledge question. Appointment booking, lookups, rescheduling, cancellations, and similar actions are handled through tools connected to our existing backend. Transfers also follow routing rules — an emergency can require a human agent, while another conversation may need to be transferred to a different AI assistant. But tool calls can take time. If the caller hears complete silence while the backend checks appointment availability, the system feels like it has stopped working. So the assistant can provide a short conversational filler while the tool is executing and then continue with the actual result. The filler isn't replacing the result — it's simply covering the time required for the backend operation. Transfer intent is also handled with latency in mind. Simple phrase matching can be checked first because it is inexpensive. Semantic similarity can then be used when the wording is less obvious. Some of the data required for these decisions can also be prepared earlier in the call so that the first meaningful turn doesn't have to pay the entire cost. Guardrails Are Part of the Same Loop Voice AI isn't only about getting an answer from an LLM. The system also needs guardrails. Some instructions are handled through the system prompt. Other patterns can be detected locally to block or modify unsafe inputs and outputs. The important part is that these checks need to fit into the same real-time pipeline. A guardrail that takes long enough to create an obvious pause becomes another latency problem. So, What Does 1.5 Seconds Actually Mean? This is an important distinction. When I say we reduced latency from roughly 9 seconds to around 1.5 seconds, I'm referring to time-to-first-audio on a typical knowledge turn. It assumes a warm path where: The caller asks a genuine knowledge question. Retrieval is cached or already available. The LLM streams its response. The first complete sentence is generated. TTS can begin before the entire answer is finished. It does not mean that every possible Voice AI interaction completes in 1.5 seconds. For example, a cold appointment-booking request may require checking availability, confirming information, calling backend services, and generating multiple time slots — that's a different latency measurement. It also isn't the total duration required to speak the entire answer. Those are separate metrics. The important change was the experience the caller actually perceived. Instead of waiting in silence after asking a question, the assistant could begin responding while the remaining work continued in the background. The conversation started feeling like a conversation again. The Tradeoffs Of course, none of these optimizations came for free. Streaming the first sentence means that the system can sometimes begin speaking before the complete answer is known. We mitigate this with shorter responses and retrieval confidence thresholds rather than waiting for the entire response. Local embeddings are significantly faster, but they need to remain aligned with the embeddings generated during document ingestion. Heuristic query rewriting works extremely well for the specific domain we are dealing with, but it requires maintenance and won't necessarily generalize perfectly to every industry. And perhaps the biggest tradeoff is the one we started this entire journey with: once you stop relying entirely on a managed Voice AI platform, you own the entire orchestration layer. That means dealing with: Barge-in Turn detection Voicemail Idle re-prompts Recordings Transcripts Guardrails Transfers Backchannels Tool calls Connection management And all the strange things people do during real phone conversations That's the real cost of building your own Voice AI system. What I Learned The biggest lesson from this project wasn't really about Twilio, Deepgram, Cartesia, RAG, or even the LLM. Those components are replaceable. The real product is the orchestration loop that decides: When should I listen? When should I retrieve? When should I call a tool? When should I stop listening? When should I start talking? And most importantly: how quickly can I make the caller hear something useful? That is what ultimately determines whether a Voice AI system feels like a real conversation — or just an API pipeline talking over a phone call. If you're building or evaluating Voice AI — whether that's sticking with a managed platform or owning the orchestration layer yourself — I'd genuinely like to compare notes. Feel free to reach out or drop a comment with what you're running into. Mehar Aziz is a Software Engineer working on full-stack development including AI/ML. Find me on [LinkedIn].