A driver calls in and says something that starts in Amharic, names an app screen in English, gives a location in a third form entirely, and finishes in Amharic. To a person this is unremarkable. To a pipeline that decided the call was Amharic at second one, it is three errors in a row.
Where the switch actually happens
Switching is not random. It clusters around a small set of things that have no settled local form, or whose local form is not what people say out loud.
- Product and feature names, which are almost always spoken in English.
- Numbers, especially account numbers and amounts, where speakers often switch mid-string.
- Place names and landmarks, which mix borrowed and local forms freely.
- Technical or banking vocabulary that entered the language in English.
Why per-call detection fails
A conventional pipeline runs language identification on the opening audio, loads the matching acoustic and language models, and commits. Everything after that is decoded under an assumption that stopped being true two seconds in. The failure is quiet: you get plausible words in the wrong language rather than an obvious error.
Detecting more often helps, but only up to a point. Switch detection at the clause level still lags the switch itself, and re-deciding constantly introduces its own instability, where the system flips language on an ambiguous word and drags the rest of the sentence with it.
Recovery beats detection
What has worked better for us is treating the transcript as revisable. The agent does not need to have been right at the moment the word was spoken, it needs to be right by the time it acts. Holding a short window open and re-scoring it once the sentence completes fixes most switch errors, because the context that disambiguates a borrowed word usually arrives after it.
The second half is intent tolerance. If the agent is confident about what the caller wants and unsure about one token inside it, the right move is to confirm that one token rather than restart the turn. Callers accept was that account ending 4471? far better than sorry, could you repeat that?.
“Our customers finally get answers in the language they actually speak.”
What this means when you build
- Do not ask the caller to choose a language. It fixes the wrong variable and annoys them.
- Put both the English and the local form of every product name in your knowledge base.
- Confirm specific values, not whole utterances, when confidence dips.
- Read transcripts for switched tokens specifically. They are where the recoverable errors hide.
Written by
Lucy Engineering
Engineering
