How to Test a Voice AI Agent on Kenyan Speech Before Launch

Most voice AI demos sound impressive because the demo audio is easy: a clear American or British accent, one language per sentence, quiet room, studio microphone. Your Nairobi callers will give the agent none of that. The fix is not a better demo or a different vendor pitch. It is a test set built from Kenyan speech, run against your actual pipeline before a single customer call goes through.

This post is the full pre-launch testing playbook: what breaks, how to build a test set that sounds like your real callers, how to measure speech-to-text accuracy with real numbers, how to test the pipeline end to end, and what pass marks to demand before go-live.

Why the vendor demo lies to you

Global speech models post impressive accuracy numbers, and those numbers are real. They are also measured on the wrong audio. Benchmarks like LibriSpeech use clean, read, single-language English. The AfriVox-v2 benchmark, which tests speech models on unscripted, in-the-wild African audio across ten sectors, puts the gap plainly: models that score under 5 percent word error rate on standard English benchmarks degrade by a factor of five to ten on African accents and languages (AfriVox-v2, arXiv).

The Swahili numbers are starker. A community fine-tuning exercise on Hugging Face measured OpenAI Whisper large-v3 on Common Voice Swahili test audio at 51.4 percent word error rate out of the box. After fine-tuning on Swahili data the same model reached 22.6 percent (cdli model card). One in two words wrong, from the model half the voice AI stacks on the market ship by default.

None of this means voice AI cannot work in Kenya. It means the difference between a system that works and one that frustrates callers is almost entirely in whether anyone tested it on Kenyan speech before launch. That testing is a week of disciplined work, not a research project.

The four ways Kenyan speech breaks a voice agent

Before you test, know what you are testing for. There are four distinct failure modes, and each needs its own test clips.

Accent. Kenyan English has its own phonetics: non-rhotic r sounds, different vowel lengths, stress patterns that follow Bantu language rhythms. A model trained mostly on US and UK audio maps those sounds to the wrong words. This is the easiest failure to fix, because vocabulary adaptation and better models help a lot.

Code-switching. Nairobi callers do not speak Swahili or English. They speak both, often in one breath: "Nataka ku-check balance yangu, nilitumia pesa jana but haijareflect." Linguists call mixing inside a single sentence intra-sentential switching, and it is the hard case. A speech system configured for one language cannot return words from the other, so it returns the nearest in-language sound instead. The Soniox voice wiki describes the result well: a confidently wrong transcript rather than a slightly worse one, and clean-looking logs that hide the bug (Soniox voice wiki). Whole-utterance language detection does not rescue you here either. Detecting that a sentence is "mostly Swahili" still throws away every English word in it.

Names, numbers, and references. This is where errors become expensive. A misheard greeting costs nothing. A misheard M-Pesa transaction code, ID number, or account digit costs a failed payment lookup or a wrong caller. Intron reports that its Sahara-v2 model improved accuracy on African names, organizations, and locations by 68.6 percent, and on numbers, fractions, and currencies by 55.6 percent, relative to global models (TechNext). The fact that a vendor advertises specific gains on exactly these categories tells you where global models hurt most. The AfriVox-v2 benchmark includes dedicated number and entity test sets for the same reason: its Swahili portion alone has 466 number samples and 780 named-entity samples.

Telephone audio. A call over the PSTN arrives as 8 kHz mono audio, often compressed further by the carrier, with matatu noise, wind, and a handset mic. Models benchmarked on 16 kHz studio audio lose accuracy on this channel before the accent even enters the picture. If you test on clean recordings, you are not testing.

Build a test set that sounds like your callers

The test set is the whole game. A hundred clips of the right audio will tell you more than a thousand of the wrong audio. Here is the procedure.

Step 1: Collect audio from three sources

• Real calls. If you have an existing call centre or phone line, record 30 to 50 real calls. You need caller consent and a lawful basis for recording; the rules are stricter than most teams assume, and our guide to call recording consent in Kenya covers them. Strip anything that identifies the caller before the clips leave your control.
• Open datasets. Mozilla's Common Voice has a Swahili corpus, and the AfriVoices-KE dataset provides roughly 3,000 hours across five Kenyan languages from 4,777 speakers, with code-switched terms annotated. Neither will sound exactly like your customers, but both are free, consented, and useful for bulk accent coverage.
• Scripted recordings. Write 40 to 60 short scripts covering your actual call flows and pay five to ten Kenyan speakers a modest fee to record them over a real phone line, not a laptop mic. Include the exact vocabulary your agent must catch: product names, branch names, amounts in shillings, ID formats.

Step 2: Cover the failure modes deliberately

Aim for 100 to 200 clips, 5 to 30 seconds each, split roughly like this:

• 25 percent clean Kenyan English, no code-switching. Your baseline.
• 35 percent code-switched speech, with the switch happening mid-sentence, not between sentences.
• 20 percent number and entity dense: names, ID numbers, transaction codes, amounts, dates.
• 20 percent hostile audio: background noise, bad connections, fast speakers, elderly callers, strong upcountry accents.

Step 3: Transcribe by hand

Every clip needs a human reference transcript, written by someone fluent in both Swahili and Sheng-flavoured Nairobi English. This is the ground truth you score against. Do not skip the code-switched spelling conventions; decide upfront whether "nimeshare" is transcribed as written or normalised, and be consistent, because inconsistent references corrupt your measurements.

Benchmark the speech-to-text layer

With reference transcripts in hand, scoring is mechanical. Word error rate counts substitutions, insertions, and deletions against the reference, divided by the number of reference words. Lower is better. The standard tool is the open-source jiwer Python library:

from jiwer import wer

reference = "nataka ku check balance yangu ya account"
hypothesis = "nataka ku check balance young yeah account"
print(wer(reference, hypothesis))  # 0.25 -> 25% WER

That example has two substitutions in eight words: 25 percent WER. On its own the sentence still parses, and your language model would probably recover the intent. Now run the same calculation on a clip where the errors land on "EQX7K2LM9P" or "0712 345 678" and you see why overall WER is only the first number to look at.

Score three numbers, not one

• Overall WER per category. Report accent-only, code-switched, number-dense, and noisy clips separately. Averages hide the category that kills you.
• Entity accuracy. The percentage of names, codes, and numbers transcribed exactly right. For most business flows this matters more than WER; a 15 percent WER transcript with every digit correct is a working system, while a 10 percent WER transcript that garbles transaction codes is not.
• Code-switch handling. Count how often mid-sentence English words inside Swahili utterances survive transcription. This isolates the failure mode language detection cannot fix.

Test at least three recognisers

Your shortlist should include a global streaming model, an African-focused model, and an open model you can fine-tune. As of this writing:

• Global streaming. Deepgram's Nova-3 multilingual code-switching model handles ten languages, but Swahili is not among them; its Swahili path is hosted Whisper, which Deepgram offers for batch only, with no live streaming, and Swahili is absent from its language-detection list (Deepgram docs). If your agent must respond in real time to Swahili speech, check exactly which model and mode your vendor is routing to, because "we support Swahili" and "we stream Swahili at voice-agent latency" are different claims.
• African-focused. Intron's Sahara v2.5 ships bilingual models that follow a conversation across an African language and English, covering twelve language pairs including Swahili. Kenya's own MsingiAI has released Sauti STT, a Swahili model built for code-switching and real-time use, currently in early access with limited public benchmarks, so test it yourself rather than trusting the announcement.
• Open-weight. Whisper large-v3 and newer open multilingual models can be fine-tuned on your own clips. The cdli experiment quoted above is the pattern: stock model at 51 percent WER, fine-tuned model at 23 percent, on the same test audio.

Run every clip through every candidate and keep the outputs. You will reuse this harness every time a vendor ships a new model, which is now every few months.

Test the pipeline end to end, not just the transcript

A voice agent is a chain: telephony, speech-to-text, language model, tools and lookups, speech generation. Each link can pass its own test while the chain fails. End-to-end testing means playing audio into the phone number and scoring what the agent actually did.

What to score end to end

• Task completion. For each test clip with a clear intent, did the agent do the right thing? Balance read out, appointment booked, ticket logged.
• Slot capture. When the caller dictated a number or name, did the right value land in the right field? Test read-back behaviour too: a good agent repeats critical values for confirmation before acting on them.
• Interruption handling. Kenyan callers barge in. Play clips where the caller talks over the agent and check that the agent stops, listens, and recovers rather than talking through them.
• Fallback behaviour. When recognition confidence is low, does the agent ask the caller to repeat, offer to switch languages, or route to a human? Silent confident failure is the worst outcome. Give the agent deliberately unintelligible audio and verify it degrades gracefully.
• Latency under load. Kenyans will hang up on a five-second pause. Measure response time at realistic concurrency; our breakdown of where the milliseconds go on voice AI calls shows how to attribute the delay across the chain.

Run the whole set over the real phone path, through your SIP trunk or telephony provider, not over a websocket from your laptop. The trunk itself introduces failure modes; if you have not set one up before, our notes on SIP trunking for voice AI agents cover the gotchas that show up in Kenya specifically.

Test the voice that talks back

Recognition is only half the loop. The agent's own voice needs testing too, and two things matter.

First, language coverage is model-specific, not vendor-wide. ElevenLabs is a good example: its fast, cheap eleven_multilingual_v2 and flash models list around thirty languages and Swahili is not among them, while the newer v3 and v4 models advertise seventy to ninety-plus languages (ElevenLabs docs). If your cost model assumed the cheap tier, verify the language list before you commit. Intron's Sahara generates speech in thirteen African languages with code-switching support, which is currently a rare capability.

Second, run a listening test with real Kenyans. Pick three to five candidate voices, generate the same twenty agent responses with each, and have five to ten local listeners rate them on two questions: does this sound like someone I would trust on the phone, and did you catch every number and name on the first listen? Accented English that sounds natural to Nairobi ears consistently beats a polished BBC voice for trust, and a voice that swallows digits is disqualified no matter how warm it sounds.

Pass marks: a scorecard before go-live

Decide the thresholds before you look at the results, or you will rationalise whatever you get. Reasonable starting targets for a customer-facing Kenyan deployment:

• Clean Kenyan English: under 12 percent WER.
• Code-switched speech: under 20 percent WER, with mid-sentence English words surviving at least 80 percent of the time.
• Entity accuracy on numbers and names: 95 percent or higher after confirmation read-back. Below that, mandate read-back confirmation in the flow for every critical value.
• End-to-end task completion: 90 percent on your scripted flows, 85 percent on real-call replays.
• First response latency: under 1.5 seconds from the caller finishing a sentence, measured at p95, not average.
• Noise robustness: less than 5 percentage points of WER degradation between quiet and noisy versions of the same content.

Miss one category and the honest options are: switch the failing component, add a confirmation step to the affected flow, restrict launch to the call types that pass, or fine-tune. Shipping and hoping is not on the list.

When fine-tuning is the right answer

Fine-tuning is the most powerful fix and the most over-prescribed one. It makes sense when your benchmark shows a specific, persistent gap: a model that is good on clean English but poor on your code-switched clips, or one that keeps mangling your domain vocabulary after vocabulary boosting. The cdli result, stock Whisper at 51.4 percent WER down to 22.6 percent after fine-tuning on a modest Swahili dataset, shows the size of improvement on the table.

It is the wrong first move when vocabulary prompting or a better base model closes the gap, because fine-tuning buys you an ongoing obligation: you now own a model that needs retraining as products and vocabulary change, hosting that must stay under voice latency budgets, and evaluation on every update. Also note what cannot be fine-tuned: proprietary models like Anthropic's Claude or the big hosted speech APIs are closed to customer training. Fine-tuning applies to open-weight models such as Whisper-style speech models and Llama, Qwen, Mistral, or Gemma on the language side, which is exactly the kind of work our enterprise AI practice does for banks and insurers that also need the data to stay inside their own infrastructure.

Factor the operating cost honestly. The speech model is one line item in a stack that includes telephony, the language model, and synthesis; our breakdown of what a voice AI agent costs to run in Kenya puts the per-minute economics in one place.

Common mistakes

• Testing on laptop microphones. If the audio did not come through a phone line, the test is not about phone calls. Re-encode clips to 8 kHz if you must, but real PSTN recordings are better.
• Quoting a single WER. One blended number hides the code-switching and number categories where deployments die. Always report per category.
• Trusting "Swahili supported" on a pricing page. Verify the exact model, mode, and streaming behaviour behind the claim. Support in a batch API and support inside a real-time voice agent are different products.
• Skipping the confirmation read-back. For ID numbers and transaction codes, no recogniser is good enough to skip read-back. Build it into the flow and test that it triggers.
• Testing once. Models update constantly. Keep the test set and harness, and re-run the benchmark whenever a vendor ships a new model or you change any component.

Frequently asked questions

What word error rate is good enough for a Kenyan voice agent?

Under 12 percent on clean Kenyan English and under 20 percent on code-switched speech are workable targets, provided entity accuracy on names and numbers stays at 95 percent or above with read-back confirmation. Overall WER alone is a poor gate; a 10 percent WER system that garbles transaction codes is worse than a 15 percent system that gets every digit right.

Can Whisper handle Swahili-English code-switching?

Out of the box, poorly: Whisper large-v3 measured 51.4 percent WER on Common Voice Swahili test audio in a community benchmark. Fine-tuning on Swahili data cut that to 22.6 percent in the same exercise, so the open model is a viable base if you are willing to fine-tune and host it. For managed APIs, test African-focused models like Intron Sahara or MsingiAI Sauti against your own clips.

How many test clips do I need?

One hundred to two hundred clips is enough for a pre-launch benchmark if they are chosen deliberately: roughly a quarter clean English, a third code-switched, a fifth number-dense, and a fifth noisy or difficult audio. Ten clips per category tells you nothing; a hundred random easy clips tell you less than fifty hard ones.

Should I use automatic language detection for Swahili and English calls?

It helps for calls that stay in one language, but it does not fix mid-sentence code-switching, because detection assigns one language to the whole utterance and discards words from the other. Check whether your vendor supports Swahili detection at all: Deepgram's language-detection list, for example, does not include it. For Nairobi traffic, a bilingual code-switching model usually beats a detect-then-transcribe pipeline.

Is Swahili text-to-speech good enough for customer calls?

It depends on the specific model, not the vendor's brand. Coverage varies sharply even inside one provider: some ElevenLabs tiers list about thirty languages without Swahili while newer models claim ninety-plus. Run a listening test with Kenyan listeners rating trust and digit clarity on your actual agent scripts before committing.

How much does a speech evaluation like this cost?

The main costs are recording and human transcription. Expect a few thousand shillings per speaker for scripted recordings over phone lines, and one to two working days of a bilingual transcriber's time for a couple of hundred clips. API costs for benchmarking three recognisers on a few hours of audio are trivial by comparison, typically under a hundred dollars.

Where to go from here

The short version: build a test set from real Kenyan speech, score speech-to-text per category with entity accuracy as a first-class metric, test the whole chain over a real phone line, and hold the launch to thresholds you chose in advance. If you would rather have engineers who do this every week run the benchmark and the build, our custom builds team designs, tests, and ships voice agents for Kenyan and international callers, including the evaluation harness you keep afterwards.