Building Kisan Mitra: How I Built an Ultra-Fast Voice AI for Indian Farmers in 10 Days

From zero to a full-stack, multilingual agricultural voice agent with caller memory, real-time mandi tools, outbound price alert calls, human escalation, and specialist agent handoffs — powered by Murf Falcon & LiveKit. 🌟 The Problem & The Mission In rural India, millions of farmers make critical livelihood decisions every day: When should I harvest? Will it rain before I spray pesticides? Which nearby mandi (market) is offering the best price for my cotton crop? While agricultural data exists across various portals, accessing it through complex web interfaces or text-heavy apps is challenging for farmers out in the field. Voice is the natural, frictionless interface for Bharat. A farmer standing in an orchard or driving a tractor doesn't want to type queries into a search bar; they want to speak naturally in their native language or conversational Hinglish and get instant, reliable answers. For the 10 Days of Voice Agents (VoiceForBharat Edition), I chose the Farm & Field track and built Kisan Mitra (किसान मित्र) — an empathetic, real-time AI voice assistant tailored specifically for Indian agriculture. 🏗️ Architecture & Core Components A production-grade voice agent is fundamentally different from a text chatbot. Latency is the single biggest factor in conversational realism: if the agent takes more than 1–1.5 seconds to reply, the human conversation breaks down. mermaid flowchart LR A[🎙️ Farmer Speaks] -->|Audio Stream| B(Deepgram Nova-3 STT) B -->|Transcribed Text| C(Gemini 2.5 Flash LLM) C -->|Streamed Tokens| D(Murf Falcon TTS) D -->|Real-time Audio| E(LiveKit WebRTC) E -->|Ultra-low Latency Audio| F[🔊 Farmer Hears Answer] C |Tools & Memory| G[(SQLite & External APIs)] The 4 Pillars of the Pipeline: Real-time Transport (LiveKit): Manages ultra-low-latency, bidirectional audio WebRTC streaming and turn detection. Speech-to-Text (Deepgram Nova-3): Accurately transcribes spoken Indian English and accented Hindi. LLM Brain (Google Gemini 2.5 Flash): Handles intent detection, domain reasoning, guardrails, and tool calling. Fast Text-to-Speech (Murf Falcon): The game changer. With sub-100ms time-to-first-audio, Murf Falconstreams natural, warm Indian voices (Anisha / hi-IN) without robotic pauses. 🚀 Key Features Built Across the 10 Days Authentic Indian Voice & Conversational Guardrails Using Murf Falcon's hi-IN voice with conversational styling and sentence tokenization, Kisan Mitra sounds warm, respectful, and native. The system prompt strictly enforces: Conversational, concise spoken responses (no markdown syntax or raw JSON read aloud). Absolute refusal to fabricate market rates or weather data. Live Agricultural Tools (Mandi Prices & Open-Meteo Weather) Kisan Mitra is armed with domain tools that fetch real-world data: lookup_mandi_prices(crop, district): Queries market prices across key Indian APMCs (e.g., Yavatmal, Nagpur, Lasalgaon) and explicitly cites timestamps (e.g., "As of today's Agmarknet live update..."). get_district_weather(district): Leverages Open-Meteo live satellite feeds to deliver temperature, rain probability, and actionable agronomic advice (e.g., "Rain probability is 65% today; postpone chemical spraying"). Graceful Failure & Out-Loud Transparency External APIs fail in the real world. Rather than hallucinating rates or hanging silently, Kisan Mitra catches timeouts and announces the service outage out loud to the caller: Caller Memory & Privacy-First Persistence Kisan Mitra remembers returning farmers (e.g., their land size, crops grown, district) across sessions via SQLite. Crucial guardrail: The agent never saves data without first asking: "May I save these details so I can remember you for our next call?" Outbound Telephony & Proactive Price Alerts When mandi rates cross a farmer's predefined threshold (e.g., Cotton crossing ₹7,000/quintal in Yavatmal), Kisan Mitra autonomously places a phone call via Twilio & TwiML: Compliance in the first 2 sentences: Explains who is calling, why, and how to opt out (Press 9 to unsubscribe, 1 for details). Outcome tracking: Automatically handles busy lines, no-answers (retry in 2h), and short hang-ups. Human Escalation (Krishi Vigyan Kendra Officer Support) For emergencies (severe pest attacks like Pink Bollworm or crop blight), the agent prompts the farmer for consent and logs a structured escalation ticket with sanitized PII, assigning a reference number like ESC-48291 for agricultural officer callbacks. Specialist Agent Handoffs & Analytics Dashboard Handoffs: Complex agronomic pathology queries are handed off seamlessly to a dedicated CropSpecialist sub-agent. Analytics: Complete call outcome tracking (completed, success, failed, reason) stored in SQLite and visualizable via an admin dashboard. 🛠️ The Hardest Challenges & Lessons Learned Eliminating Conversational Latency The Problem: Combining STT + LLM reasoning + TTS synthesis often introduces awkward 2–3 second silences. The Solution: Streaming tokens incrementally from Gemini into Murf Falcon using SentenceTokenizer(min_sentence_len=2) and preemptive_generation=True. Audio synthesis begins before the LLM finishes generating the full paragraph. Handling Code-Mixed Hindi & Indian English (Hinglish) The Problem: Standard VAD (Voice Activity Detection) models often cut off speakers mid-sentence when Indian language filler words ("haanji", "achha", "matlab") were used. The Solution: Integrated LiveKit's MultilingualModel turn detector paired with Silero VAD to maintain natural listening rhythms. Out-Loud Tool Failures vs. Silent Errors The Problem: When an external weather API timed out, the LLM initially tended to guess yesterday's temperature. The Solution: Enforced strict system instructions where tools return explicit FAILURE strings that instruct the LLM: "State out loud that the service is unreachable right now. Do not guess." GITHUB :- https://github.com/codebynikhil08/murf-livekit-starter