Building KisanVani (किसान वाणी): An Ultra-Low-Latency Multilingual Voice Agent for Indian Agriculture with Murf Falcon & LiveKit

Building KisanVani (किसान वाणी): An Ultra-Low-Latency Multilingual Voice Agent for Indian Agriculture with Murf Falcon & LiveKit How we built an AI Krishi Mitra for Indian farmers featuring ultra-fast streaming TTS, persistent memory, real-time weather & Mandi price tools, human escalation, outbound alert telephony, and multi-agent specialist handoffs in 10 days. 🌾 1. The Problem & The Mission Across rural India, millions of farmers rely on timely agricultural information—such as sowing advisories, weather forecasts, pest outbreak alerts, and Mandi (market) crop rates—to make daily decisions that affect their livelihoods. However, traditional text-based mobile apps fail for rural populations due to language barriers, literacy constraints, and complex navigation UI. Voice is the natural interface for Bharat. During the 10 Days of Voice Agents — #VoiceForBharat Edition challenge, we set out to build KisanVani (किसान वाणी): an empathetic, expert AI voice assistant and digital Krishi Mitra (Agriculture Friend). Built specifically for the Farm & Field track, KisanVani allows farmers to speak naturally in Hindi, Hinglish, or Indian English, receiving real-time agricultural guidance, market rates, and weather alerts with human-like responsiveness. 🏗️ 2. System Architecture & Core Stack Building a real-time voice agent requires tightly integrated speech processing, language modeling, dynamic tool invocation, and streaming synthesis. Audio latency must remain under 300ms so conversations feel natural without awkward delays. Mermaid diagram The Technology Stack: Speech-to-Text (STT): Deepgram Nova-3 (language="multi") for robust multilingual recognition. Text-to-Speech (TTS): Murf Falcon 2 (livekit-murf) — the fastest TTS API on the market, streaming audio in under 200ms with natural Indian English and Hindi pronunciation (Anisha / Easha voices). Large Language Model (LLM): Google Gemini 2.5 Flash for high-speed reasoning, function calling, and multilingual fluency. Real-time Transport & Pipeline: LiveKit Agents SDK (livekit-agents ~1.4) with Silero VAD and Multilingual Turn Detection. Persistent Memory & Storage: SQLite database for privacy-compliant caller profiles, outbound telephony logs, and human escalation tickets. ⭐ 3. Important Features Built Over 10 days of iterative development, KisanVani evolved from a basic voice pipeline into a comprehensive production-grade AI agent system: Ultra-Low Latency Indian Voice (Murf Falcon 2) Using Murf Falcon 2 TTS with sentence-level tokenization (SentenceTokenizer(min_sentence_len=2)), KisanVani achieves sub-250ms audio synthesis. The voice sounds empathetic, warm, and natural—essential for building trust with farming communities. Personality, Objectives & Safety Guardrails KisanVani operates under strict system prompt guardrails: Scope Control: Hard refusals for non-agricultural, medical, legal, or financial requests. Safety Net: Out-of-scope or unverified queries are gracefully directed to the official Kisan Toll-Free Helpline (1800-180-1551). Zero Hallucination: If tool data is unavailable, the agent speaks a polite fallback out loud rather than inventing prices or weather numbers. Multilingual Support & Native Script Enforcement The agent recognizes Hindi, Hinglish, and English. A strict rule enforces native script output for non-English languages (Devanagari script for Hindi, e.g., "नमस्ते", never romanized "namaste"), ensuring Murf Falcon synthesizes native phonemes cleanly. Opt-in Caller Memory & Privacy Protocols (SQLite) Returning callers are greeted warmly by name and past farm context (e.g., land size, crops grown, home district). Memory is bound by a Hard Privacy Consent Rule: the agent must explicitly ask permission before saving facts into SQLite (save_caller_memory) and supports a full "Forget Me" wipe protocol (forget_caller_memory). Real-Time Agricultural Tools & Tool Chaining get_weather_forecast: Queries Open-Meteo REST API for live temperature, humidity, rain probability, and spraying suitability advice. get_mandi_prices: Retrieves current market modal rates and min/max price ranges per quintal across wheat, paddy, mustard, cotton, potato, and onion markets. Tool Chaining: If a farmer asks "What is the weather today?" without mentioning their location, KisanVani automatically checks saved caller memory for their home district. Proactive Outbound Alerts & Telephony Protocols KisanVani can initiate outbound SIP calls for emergency weather or pest warnings. Outbound calls follow a strict 3-step opening protocol: Who is calling: Identify as KisanVani AI Krishi Mitra. Why calling: State the specific alert reason. How to stop: Explain how to opt out (opt_out_alerts). Human Escalation Protocol with Reference Tickets For severe crop blight epidemics or complex subsidy disputes beyond AI scope, KisanVani creates a human escalation ticket via create_escalation. After obtaining caller consent, it logs a sanitized record in SQLite, issues a reference ID (e.g. ESC-48291), and promises a 24-hour callback from a senior Krishi Officer. Multi-Agent Handoff (Assistant ↔ CropSpecialist) When questions turn to complex crop pathology, yellow rust, or pesticide dosage, the main assistant dynamically transfers the session to CropSpecialist (Fasal Visheshagya). If the caller later asks about weather or Mandi rates, the specialist hands control seamlessly back to the main agent. 💻 4. Code Snippets & Walkthrough Here is how the core pipeline and dynamic handoff are configured in Python (backend/src/agent.py): LiveKit Pipeline & Murf Falcon Configuration python from livekit.plugins import deepgram, google, murf, silero from livekit.plugins.turn_detector.multilingual import MultilingualModel from livekit.agents import AgentSession, tokenize Voice AI pipeline using Murf Falcon (Anisha voice), Gemini LLM, and Deepgram Nova-3 session = AgentSession( stt=deepgram.STT(model="nova-3", language="multi"), llm=google.LLM(model="gemini-2.5-flash"), tts=murf.TTS( voice="Anisha", style="Conversation", tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2), text_pacing=True, ), turn_detection=MultilingualModel(), vad=silero.VAD.load(), preemptive_generation=True, ) Real-Time Weather Tool with Memory Chaining & Graceful Fallback python @function_tool async def get_weather_forecast(self, ctx: RunContext, district: str = "") -> str: """Fetch live weather forecast and agricultural spraying advice.""" target_district = district.strip() # Tool Chaining: Fallback to saved caller memory if district not provided if not target_district: record = get_caller(self.user_id) if record and record.get("facts", {}).get("district"): target_district = record["facts"]["district"] else: target_district = "Karnal" try: # Fetch geocoding and live Open-Meteo forecast data... # Returns temperature, humidity, rain probability, and spraying advice return f"डेटा दिनांक {today_str} के अनुसार: {place} में तापमान {temp}°C, वर्षा की संभावना {precip_prob}% है।" except Exception: # Graceful spoken fallback return "मौसम सेवा API से संपर्क विफल रहा। कृपया किसान हेल्पलाइन 1800-180-1551 पर कॉल करें।" Dynamic Multi-Agent Specialist Handoff python @function_tool async def transfer_to_crop_specialist(self, ctx: RunContext, issue_description: str = "") -> str: """Transfer session to specialized Crop Disease & Pest Specialist.""" specialist = CropSpecialist(user_id=self.user_id) ctx.session.update_agent(specialist) return "मैं आपको हमारे फ़सल रोग और कीट विशेषज्ञ से कनेक्ट कर रहा हूँ। कृपया एक क्षण प्रतीक्षा करें।" ⚡ 5. Real Engineering Challenges & Solutions Building KisanVani wasn't without hurdles. Here are three key technical challenges we encountered and resolved: Challenge 1: Script Normalization & TTS Mispronunciations Problem: Romanized Hindi (Hinglish like "Namaste, aapka swagat hai") caused Murf Falcon to read Hindi words with an English accent, reducing audio naturalness. Solution: Implemented prompt-level native script enforcement. All Hindi outputs are strictly rendered in Devanagari script ("नमस्ते, आपका स्वागत है"). Murf Falcon's phonetic parser handles Devanagari flawlessly, producing authentic Indian accents. Challenge 2: Turn Detection in Outdoor Rural Environments Problem: Standard Voice Activity Detection (VAD) models falsely triggered background noise (tractor engine noise, wind, animal sounds) or cut off farmers during natural pauses in speech. Solution: Combined Silero VAD with LiveKit's MultilingualModel turn detector and enabled Deepgram telephony noise cancellation filters (noise_cancellation.BVC()). This allowed the agent to wait for true completion of user speech while suppressing background outdoor noise. Challenge 3: Context Preservation During Dynamic Agent Handoffs Problem: Updating the active agent (ctx.session.update_agent(specialist)) risked losing caller state and background memory mid-conversation. Solution: Passed the active user_id context directly into the initialized CropSpecialist instance, enabling the specialist to query SQLite memory and maintain seamless context across agent transfers. 🚀 6. How to Build & Run KisanVani You can easily clone, build, and run KisanVani locally! Prerequisites Python 3.10+ with uv installed (pip install uv) Node.js 18+ and pnpm API Keys for: LiveKit Cloud, Murf AI, Deepgram, and Google Gemini Step 1: Clone the Repository bash git clone https://github.com/Dharmesh-jagatiya/VoiceAgentOfBharat.git cd VoiceAgentOfBharat Step 2: Configure Environment Variables Copy .env.example to backend/.env.local and add your secret API keys: env LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_secret MURF_API_KEY=your_murf_api_key DEEPGRAM_API_KEY=your_deepgram_api_key GOOGLE_API_KEY=your_google_gemini_api_key And in frontend/.env.local: env LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_secret AGENT_NAME=kisanvani-farm-agent Step 3: Run the Backend Agent bash cd backend uv sync uv run python src/agent.py dev Step 4: Run the Frontend UI In a separate terminal: bash cd frontend pnpm install pnpm dev Open http://localhost:3000 in your browser, click "Connect to KisanVani AI", and start speaking! 🔮 7. Future Roadmap & What's Next Multilingual Regional Dialects: Support for Gujarati, Punjabi, Marathi, and Kannada audio rendering. Multimodal Plant Pathology: Allowing farmers to capture photos of diseased crop leaves via smartphone camera while talking to KisanVani for visual AI diagnosis. Offline SMS Alert Fallback: Sending SMS reference receipts automatically after every voice escalation or Mandi query. 🔗 8. Project Links & References GitHub Repository: Dharmesh-jagatiya/VoiceAgentOfBharat Murf Falcon TTS Docs: murf.ai/api/docs/text-to-speech-models/falcon-2 LiveKit Agents Documentation: docs.livekit.io/agents Challenge: #VoiceForBharat 10 Days of AI Voice Agents by Murf AI Thank you to Murf AI and LiveKit for organizing the 10 Days of Voice Agents challenge! 🌾⚡