Building Anisha: My 10-Day Journey to a Voice Agent for Learning & Literacy

Over the last 10 days, I built Anisha, a voice agent for Learning & Literacy that can talk with users, remember them, provide learning exercises, make outbound calls, escalate to humans, track outcomes, and hand conversations to specialist agents. 👋 Meet Anisha Anisha is built using: Python LiveKit for real-time voice Murf Falcon for text-to-speech LLM for reasoning Custom tools and memory Human escalation Call analytics Specialist handoff The core flow is: User → STT → LLM → Tools / Memory / Specialist → Murf Falcon → User She can also handle Hindi and natural Hinglish, making the experience more suitable for Indian learners. 🧠 What Did I Build? Instead of trying to build everything at once, I added capabilities step by step. Memory — Anisha can remember relevant information about returning users. Learning Tools — She can provide exercises and track successful completion. Outbound Calls — I connected the agent to an outbound calling workflow. Human Escalation — Conversations that need human help can be escalated and tracked. Call Analytics — I added outcome tracking to understand whether conversations led to successful learning interactions. Math Specialist — Anisha can hand mathematics-related conversations to a dedicated specialist agent. This turned a basic voice assistant into a small multi-capability voice AI system. 😅 The Bug That Taught Me the Most The specialist handoff looked simple: Anisha → Math Specialist But there was a problem. The Math Specialist was being triggered correctly, yet its introduction was being spoken using Anisha's voice. The issue wasn't the text. It was the interaction between the active agent, session, and TTS configuration. I changed the handoff flow so Anisha doesn't speak the specialist's introduction. The specialist takes over and uses its own configured Murf voice. That taught me: In real-time voice AI, changing the agent also means managing the audio pipeline and agent state correctly. 🔧 What I Learned The biggest lesson from this challenge was that voice AI is much more than connecting an LLM to a microphone. Memory makes it personal. Tools make it useful. Guardrails make it reliable. Human escalation makes it practical. Analytics make it measurable. Specialists make it scalable. Good TTS makes it feel human. And debugging? That's a whole feature by itself. 😄 🛠️ Want to Build One? Start small: Microphone ↓ STT ↓ LLM ↓ TTS ↓ Speaker Then add memory, tools, analytics, telephony, escalation, and specialist agents one at a time. My project is open source: GitHub: https://github.com/aashishkumawat977-debug/murf-livekit-starter Getting Started Clone the repository: git clone https://github.com/aashishkumawat977-debug/murf-livekit-starter.git cd murf-livekit-starter Configure your API keys in .env and never commit real credentials or private data to GitHub. Once the backend and frontend are running, open the application in your browser and connect to Anisha. Try: "I want to practice maths." This lets you test the voice interaction and Math Specialist handoff. 📸 Build Evidence I also shared screenshots and demos of Anisha's voice interface, learning exercises, escalation dashboard, call analytics, and Math Specialist handoff during the challenge. You can also see the project demos and build updates on my LinkedIn: https://www.linkedin.com/in/ashish-kumawat-b68239422/ These show how the project evolved from a basic voice interaction into a complete voice AI workflow. 🚀 What's Next? Anisha is only the beginning. I'd like to add: More specialist agents Better learning progress tracking Richer analytics More personalized learning experiences Final Thought 10 days ago, I wanted to learn how voice agents work. Today, I have an agent that can talk, remember, learn, call, escalate, analyze, and hand off conversations. It wasn't built perfectly. I built it by experimenting, breaking things, debugging, and trying again. 10 days. One voice agent. A lot of debugging. A lot of learning. 🚀 Huge thanks to Murf AI for the 10 Days of AI Voice Agents — VoiceForBharat Edition challenge and for giving me the opportunity to build with Murf Falcon, the fastest TTS API, LiveKit, and Python.