Building a Voice Agent in 10 Days — My VoiceForBharat Journey

Building a Voice Agent for a Local Indian Store: 10 Days of Voice Agents — VoiceForBharat Edition Building a voice agent sounds simple at first. Listen to the user, send the text to an LLM, generate a response, and speak it back. But once you start adding real-world requirements — memory, safety guardrails, multilingual conversations, phone calls, human escalation, analytics, and specialist handoffs — it becomes a very different engineering problem. Over the last 10 days, I worked on exactly that as part of 10 Days of Voice Agents — VoiceForBharat Edition. I started with a basic voice agent and gradually turned it into a voice assistant designed for a local Indian store. Here's what I built and what I learned along the way. The Problem I Wanted to Solve For a local grocery or general store, customers often have simple questions: Is a particular product available? What is the price? What are the store timings? Are there any offers? Can I get an alternative product? What should I do if my product is damaged or incorrect? These questions don't always need a person to answer them. I wanted to build a voice-first assistant that could handle these conversations naturally, while also knowing when it should stop trying to solve the problem itself. Voice is particularly useful here because customers don't have to open an app, type their question, or navigate through menus. They can simply speak. Meet the Voice Agent My project is a voice assistant for a local Indian grocery/general store. It's designed to: Help customers with shopping-related questions Handle English and Hindi conversations Understand Hinglish-style conversations Remember returning customers Follow safety and accuracy guardrails Hand off return and refund requests to a specialist agent Make outbound phone calls Track call outcomes Provide analytics on how the agent is performing The goal wasn't just to make an AI that could talk. It was to make the conversation behave like a useful customer-service system. How the System Works The basic voice pipeline looks like this: User speaks ↓ Deepgram Speech-to-Text ↓ Google Gemini LLM ↓ Tools / Memory / Agent Logic ↓ Murf Falcon TTS ↓ LiveKit real-time audio ↓ User hears the response LiveKit handles the real-time communication between the user and the agent. Deepgram Nova-3 handles speech recognition. Google Gemini handles language understanding and response generation. Murf Falcon, using an Indian English voice, handles text-to-speech. The agent session also uses multilingual turn detection, voice activity detection, preemptive generation, and noise cancellation to make the conversation feel more natural. An Indian Voice With Murf Falcon One of the most important parts of this project was the voice itself. I used Murf Falcon with an Indian English voice and configured the agent to switch between en-IN and hi-IN depending on the conversation. The project detects Hindi/Hinglish-style input and adjusts the TTS locale accordingly, so the interaction isn't restricted to formal English — a user can speak in English, Hindi, or Hinglish, and the agent responds in the same conversational style. For this challenge, I used Murf Falcon — the fastest TTS API — as the text-to-speech layer of the agent. Memory for Returning Customers A useful assistant shouldn't treat every conversation as if it's meeting the customer for the first time. I added caller memory using SQLite. The agent has two tools: lookup_caller() save_caller_memory() When a caller is recognized, the agent looks up previously saved information. If the customer shares useful preferences, the agent asks for permission before saving them. This was an important design decision — the agent shouldn't silently collect information just because it can. The system prompt explicitly prevents the agent from inventing memories or exposing internal database information. Guardrails and Agent Personality Another important part of the project was defining what the agent should and shouldn't do. The assistant has clear objectives and guardrails. For example, it should never: Make up product prices Claim something is in stock without knowing Promise an unconfirmed discount Confirm an order that hasn't actually been confirmed Answer unrelated requests Provide harmful or illegal assistance If it doesn't know something, it's instructed to say so instead of guessing. This might seem simple, but good voice agents need clear boundaries just as much as they need good prompts. Specialist Agent Handoff One of the features I enjoyed building was the specialist handoff. The main store assistant handles normal shopping questions. But if the customer needs help with returns, refunds, damaged or defective products, wrong or missing items, or return eligibility, the main agent transfers the conversation to a dedicated Returns and Refunds Specialist. The important part is that the specialist receives the existing conversation context — the customer doesn't have to explain the problem again. The flow looks like this: Customer ↓ Main Store Assistant ↓ Return / Refund request detected ↓ Returns & Refunds Specialist ↓ Continue existing conversation This made the project feel much closer to a real customer-support system than a single chatbot. Outbound Phone Calls I also added outbound calling using LiveKit's SIP capabilities. The outbound call flow creates a unique room, dispatches the agent to that room, and creates an outbound SIP participant: Agent ↓ LiveKit Room ↓ SIP Participant ↓ Phone Network ↓ Customer This was one of the more challenging parts, since there are more moving pieces here than in a browser-based voice conversation. The system has to coordinate the LiveKit agent, the room, the SIP configuration, and the phone connection. Call Analytics Once an agent starts making calls, another question appears: how do I know whether it's actually performing well? So I added call tracking and analytics. Each call can record: Call ID Call type Duration Intent Outcome Escalation status and reason Tools used The backend exposes this data through a FastAPI service, which provides metrics like total calls, successful calls, failed calls, escalated calls, success rate, and average duration. It also supports filtering and retrieving individual call details. This changed how I looked at the project. Instead of only asking, "Does the agent talk?" I could start asking, "Did the conversation accomplish its goal?" The Hardest Parts The hardest part wasn't getting the first response from the agent — it was making all the pieces work together. Frontend ↓ LiveKit ↓ Voice Agent ├── Speech-to-Text ├── LLM ├── Text-to-Speech ├── Tools ├── Memory ├── SIP └── Analytics When something goes wrong, it isn't always obvious which layer is responsible. During the challenge, I dealt with issues around running multiple services, environment variables, database state, call handling, and getting components to communicate correctly. One lesson I learned: debugging a voice agent isn't just about debugging Python code. You also have to think about audio flow, real-time connections, API credentials, room state, SIP state, database state, and agent state. Breaking the system into smaller components and checking each layer separately made troubleshooting much easier. How You Can Build Your Own Voice Agent You don't need all of these features on day one. Start with the basic pipeline: Speech-to-Text ↓ LLM ↓ Text-to-Speech Then add real-time transport such as LiveKit. From there, gradually layer in: A system prompt Guardrails Tools Memory Multilingual support Phone calling Human escalation Analytics My project is available publicly on GitHub: Codehunter0009/murf-livekit-starter You can inspect the complete implementation, including the backend, frontend, agent logic, analytics API, database, and outbound calling code. Basic Setup The project uses Python for the backend and Node.js for the frontend. The backend uses uv for dependency management. After cloning the repository: cd backend uv sync Then configure the required environment variables in .env.local: LIVEKIT_URL LIVEKIT_API_KEY LIVEKIT_API_SECRET MURF_API_KEY DEEPGRAM_API_KEY GOOGLE_API_KEY For outbound calling, the relevant SIP configuration is also kept in environment variables. Never commit your .env.local file or API keys to GitHub. Running the Agent Start the backend in development mode: uv run python src/agent.py dev The project also includes a frontend for interacting with the voice agent through the browser. Once the backend, LiveKit, and frontend are running, open the application, allow microphone access, and start a conversation. What I Learned The biggest lesson from these 10 days is that building a voice agent is much more than connecting an LLM to a microphone. A useful voice agent needs: Conversation + Context + Tools + Safety + Observability The LLM is only one part of the system. I also learned how important it is to design failure paths: What happens when the agent doesn't know the answer? What happens when the customer asks for something outside its responsibility? What happens when a customer needs a human? What happens when the conversation needs a specialist? These are the questions that turn a demo into a complete system. What I Would Improve Next There's plenty I'd still improve: more robust multilingual support, an expanded toolset, a better analytics dashboard, additional specialist agents, and a more reliable phone experience. I'd also like to spend more time measuring latency and real-world conversation quality, rather than evaluating the system only through individual test calls. Project GitHub: Codehunter0009/murf-livekit-starter The complete source code and setup instructions are available in the repository. Final Thoughts Ten days ago, I was mainly thinking about how to make a voice agent talk. By the end of the challenge, I was thinking about something much bigger: How should a voice agent behave when it doesn't know something? How should it remember users? When should it ask for permission? When should it escalate? How do we measure whether a call was successful? And how do we make the whole system reliable enough for someone to actually use? That's what made this challenge valuable. I didn't just learn how to build a voice interface — I learned how to think about a voice agent as a complete system. 10 Days of Voice Agents — VoiceForBharat Edition completed. 🚀