Listen to twenty support calls from a consumer brand serving Tier 2 India and count how many stay in one language. The honest answer is close to none. A caller in Coimbatore opens in Tamil, says “order number” in English, gives the digits in Tamil, and switches to English again for “delivery date”. This is not a caller being difficult. It is how a billion people speak.
Most voice agents handle this badly because the architecture asks the wrong question at the start: press 1 for Hindi, 2 for English. That choice locks a speech recognition model for the rest of the call, and every sentence that crosses the boundary degrades.
Recognition Is Where Multilingual Breaks
Word error rate is the number that decides everything downstream. On clean English audio, modern models land around 5 to 8 percent. On Indian accented English over an 8kHz phone line, expect 12 to 18. On code-switched Hindi and English, poorly configured pipelines can exceed 30 percent, at which point intent detection is guessing.
What actually helps:
- Run a multilingual recognition model rather than switching between monolingual ones. Models trained jointly on Indic languages handle intra-sentence switching far better than a language detector routing between separate engines, which lags by a sentence and thrashes.
- Bias the vocabulary per call. Feed the recogniser the caller’s likely order IDs, product names, city names and agent names as hints. This alone can cut errors on proper nouns by half, and proper nouns are what the caller actually cares about.
- Handle numbers explicitly. Indian callers say lakh and crore, read phone numbers in pairs, and mix “double five” with “five five”. Normalise these in a post-processing layer rather than hoping the model gets it.
- Do not force a language choice up front. Greet in the language the caller’s number and history suggest, then follow the caller. Detection from the first three seconds of speech beats an IVR menu.
Generation Has To Match The Register
Getting recognition right and then replying in formal, Sanskritised Hindi is its own failure. Callers experience it as a machine that does not speak their Hindi. The register that works is the one used in everyday conversation, which for most of urban and semi-urban India means Hindi with English nouns left in English. Nobody says “sanganak” for computer, and translating “delivery” or “refund” makes the sentence harder to parse, not easier.
Text to speech quality matters more in Indic languages than in English because the market is used to bad output. A natural sounding Marathi or Bengali voice earns several seconds of patience that a robotic one does not. Latency budgets stay the same, which means the full loop from end of caller speech to first audio out should sit under 800 milliseconds, with anything past 1.2 seconds reading as a dead line and prompting the caller to repeat themselves.
Measure Per Language, Always
An aggregate containment rate of 68 percent can hide Hindi at 79 and Kannada at 41. Track recognition error, containment, and escalation separately per language, and treat a weak language as a product bug rather than an acceptable average. The fix is usually vocabulary bias and prompt examples in that language, not a model change.
At Invexa, we build voice agents that follow the caller’s language instead of asking them to pick one, because meeting people in the way they already speak is most of what makes automation feel respectful.