A caller says: "haan haan, main kal aa sakta hoon, but morning slot chahiye."
That sentence is entirely ordinary on an Indian phone line. It is also the sentence that breaks most voice deployments, and understanding why is the difference between a system that works here and one that demos well.
The structural problem
Speech recognition models are trained per language. A call is configured as Hindi or as English, and the model transcribes accordingly.
Configure it as English and the Hindi arrives as noise or as nearest-sounding English. Configure it as Hindi and the English words degrade. Automatic language detection helps less than it sounds, because it typically decides once per utterance — and the switching here happens mid-sentence, sometimes mid-clause.
Why partial failure is worse than total failure
If the system caught nothing, everyone would know immediately. Instead it catches "kal", "morning" and "slot" — enough to act. It books tomorrow morning. The caller wanted tomorrow, morning slot, which happens to be right this time.
Next call, it catches "nahi" as "haan". Now it has confirmed an appointment the caller was cancelling.
The errors are individually small, plausible, and invisible until a customer complains. That is the expensive failure mode: not a system that obviously does not work, but one that appears to work and is quietly wrong often enough to cost you.
What actually helps
Multilingual recognition in one pass. Models that transcribe mixed speech without choosing a language first. This is the single highest-impact decision and it is made at the architecture stage, not tuned in later.
Testing on your own recordings. Accent range across India is enormous, and a system tuned on one region can degrade noticeably in another. Vendor sample audio is clean, studio-recorded, and tells you almost nothing about how the system behaves on your customers over a mobile network.
Confirmation of anything consequential. Reading the booked slot back — "so that's tomorrow, 11am, correct?" — costs three seconds and catches the errors that would otherwise become missed appointments. On a bounded task this matters more than a marginal accuracy improvement.
Your actual vocabulary. Indian names, place names, your product names, and numbers spoken the local way ("do hazaar paanch sau") are where generic systems lose accuracy fastest.
The commercial argument
It is tempting to treat this as an edge case and default to English. For most Indian businesses that reverses the logic — the callers excluded by an English-only system are frequently the ones a competitor will happily take a call from.
A voice system that cannot handle code-switching is not a slightly worse version of one that can. For a large share of your callers it simply does not work, and they will not tell you why they hung up.
What to ask a vendor
- Which recognition model, and is it multilingual in a single pass?
- Can we hear it run against ten of our own recorded calls before we commit?
- What does it do when confidence is low — guess, or confirm?
- What is the measured outcome accuracy on mixed-language calls, not the word error rate on clean English?
The fourth question is the one that separates people who have deployed this in India from people who have read about it.
Questions
- Should we just ask callers to press 1 for English?
- It works, but it adds friction to every call to compensate for a limitation, and it does not help the caller who switches languages regardless of what they pressed. It is a workaround, not a solution.
- Is this a solved problem now that models are better?
- It is much better than it was and it is not solved. General multilingual capability has improved considerably; performance on compressed phone audio, in a specific regional accent, with your product names in it, still has to be tested rather than assumed.
- How many recordings do we need to test properly?
- Ten to twenty real calls will expose most problems, particularly if you deliberately include the accents and the noisy lines you would rather not think about. Testing on your easiest calls tells you nothing useful.
Want this working on your numbers rather than in an article? We scope it on one call.
Book a Consultation Call