When a customer writes "nimeget hiyo pesa but haijaingia" or "I no fit pay, the app dey hang", most published African-language benchmarks have nothing to say about whether your AI feature will cope, because they test one language at a time. The only way to know is to build your own test set from your own users' mixed speech and score it on the errors that cost you money.
Key takeaways
- Public benchmarks such as IrokoBench and AfroBench show large gaps between African languages and English, but they test single-language input, not the mixing your users produce.
- Code-switched evidence exists but is thin: one 61-hour speech benchmark, one small agriculture text set, one Kenyan sentiment study. No benchmark we found covers Sheng, Pidgin and Darija together.
- A usable test plan has four parts: a mixing inventory, slices by mixing style, error costs tied to names and numbers, and paid native-speaker review.
- Vendor claims, including new open-weight releases, are leads. Ask for results on your slices, not on theirs.
What the public benchmarks do and do not tell you
The big evaluation suites were built to compare models across languages. That is a different job from predicting how a banking assistant will handle a Nairobi teenager.
| Benchmark | What it covers | What it cannot tell you about mixed speech | |---|---|---| | IrokoBench (Adelani and 26 co-authors, NAACL 2025) | 17 African languages; inference, maths and knowledge questions (AfriXNLI, AfriMGSM, AfriMMLU), human-translated | Questions are translated into one language at a time. Reports a large gap to English and French; the best open model reached about 63% of the best proprietary one | | AfroBench (Ojo et al., ACL 2025 Findings) | 64 languages, 15 tasks, 22 datasets | Breadth over depth: one score per language and task, not per speaking style | | MasakhaNER (2021) | Named entity recognition in 10 African languages | Standard-language text; says nothing about names embedded in a mixed sentence | | FLORES-200 (NLLB paper) | Human-translated sentences for about 200 languages, used to score more than 40,000 translation directions | Sentences are translations, so each is in one language by construction |
Two further cautions. A public test set can leak into training data, so a high score may reflect memory. And the IrokoBench authors found that translating the test into English first narrowed the gap for larger English-centric models. That is useful to know if your pipeline can translate first, and a warning if you are measuring whether the model itself understands the original.
Speech has its own gap. Mozilla's Common Voice shows what community collection can reach: the Kinyarwanda effort led by Digital Umuganda gathered 1,211 hours from more than 420 contributors by 2020, under a CC0 public-domain dedication. Volunteers recorded sentences sourced with media partners. That teaches recognisers a language. It does not teach them the way a caller moves between two.
The code-switching evidence is small
Three pieces of work are worth knowing, each with a limit.
Speech. AfriSwitch, a preprint posted on arXiv on 26 August 2026 by Ashungafac, Awobade and Olatunji, offers 61.36 hours of human-transcribed, in-the-wild code-switched speech across 16 African languages and varieties. Five multilingual recognisers were tested zero-shot. The best averaged a 35.93% word error rate, and none went below 24% on any language. That is roughly one word in three wrong for the best system. The authors also report that mixing varies along two largely independent axes, how often speakers switch and how balanced the mixture is, and that Africa-targeted training predicted performance better than model scale or nominal language coverage. It is not peer reviewed yet.
Text. AfroCS-xs (ACL 2025) pairs English with Afrikaans, Sesotho, Yoruba and isiZulu in agriculture. Language models generated the sentences and native speakers validated and corrected them. Fine-tuning on this small set improved code-switched translation, including on a natural test set and on personal finance. The caveat is plain: four languages, one domain, synthetic origin.
Sentiment. RideKE, by Etori and Gini, studies Kenyan code-switched Twitter. XLM-R led on sentiment, AfriBERTa scored lowest, and every model leaned towards predicting neutral. A model built for African languages did worse than a general multilingual one on mixed text. Do not assume "African" means "better for your users".
On Sheng specifically, Sauti Halisi (PMLR volume 314, 2026) says that mismatch between standard Swahili and everyday Sheng or Swahili-English speech causes large performance gaps, and that chaining recognition and translation makes it worse. The summary we read gives no figures. A 2025 survey of 327 studies reaches a similar view across languages: models still struggle with mixed input and evaluation is biased. We found no benchmark covering Nigerian Pidgin, Darija and Sheng side by side.
A test plan you can run
Our reading: the useful unit is the mixing pattern rather than the language, and it has to come from your own traffic.
1. Build a mixing inventory. From consented, minimised logs, tag a few thousand utterances by language pair, whether the switch falls between or inside sentences, spelling habits, and whether English or French terms carry the key content. AfriSwitch's finding that switch frequency and balance vary independently is a reason to record both.
2. Cut the test set into slices. One slice per pair (Swahili-English, Hausa-English, Darija-French) and per style (full mixing, English nouns in an otherwise local sentence, romanised versus native script, noisy audio). As a rule of thumb, not a standard, give each slice enough examples that a gap between two systems would be visible to a non-statistician: hundreds, not dozens.
3. Score what costs money. Overall accuracy hides the failures that matter. Track amounts, account numbers, names, dates and negations separately. In a transfer flow, "send five hundred" misheard as "send fifteen hundred" is a worse error than ten missed filler words.
4. Run a monolingual control. Test the same content in standard Swahili or standard English, and report the gap to the mixed version. If a vendor's strong monolingual number collapses on mixed input, you have found the real product.
5. Pay bilingual reviewers from your user group. Automated metrics miss tone, register and harm. Raters should live in the speech community you serve, because Sheng and Pidgin differ by city and group. Use at least two raters per item and keep the disagreements; they show where the language itself is unsettled.
6. Write the pass mark first. Decide before testing what failure rate on money-related slices is acceptable, and what happens below it: human handover, restricted scope, no launch.
Worked example: a mobile-money support bot
Suppose a Kenyan wallet provider wants to automate first-line support in Swahili, English and Sheng. A hypothetical test set of 600 messages is cut into six slices of 100. Each message has a labelled intent and, where present, a labelled amount or transaction code. Two reviewers score the replies.
Say the bot gets the intent right in 92 of 100 standard-Swahili messages and 71 of 100 heavy-mixing ones. Those numbers are invented to show the method. The decision comes from the second figure and from the amount-extraction column, not from the headline average. A 71% slice on balance disputes might mean routing those chats to staff while you collect more data.
The strongest counterargument is cost. Building slices and paying reviewers is real money, and for low-risk uses like suggesting article links, a vendor demo and a pilot may be enough. The plan pays for itself where mistakes move money, health data or legal status.
Reading new model announcements
Vambo AI's MORENA shows how to read one. The South African startup announced on 7 October 2026 an open-weight model of 1.5 billion parameters, built from scratch, covering twelve African languages including Hausa, Kiswahili, Yoruba and Nigerian Pidgin, plus English and French. CIO Africa reports the developer's score of 1.408 bits per byte against 1.423 for Lugha-Llama-8B, and that none of it has been independently verified. Neither article states a licence.
Bits per byte measures how well a model predicts text. An analysis by Remio notes this is not factual accuracy or conversational usefulness, and that native speakers have not independently rated its answers. Vambo itself recommends fine-tuning the model as a base. So it is a promising starting point to test on your slices. It is not a finished answer for mixed-language support.
The same discipline applies to older releases. Lelapa AI's InkubaLM has 0.4 billion parameters and, its authors report, performs comparably to much larger models on several tasks. Lelapa's Vulavula describes its speech product as handling code-switching natively. Both are claims from the builder, which makes them good questions to put to a vendor, not substitutes for your own results. Community groups such as Masakhane describe participatory data creation worth asking about when a vendor says its data was "community built".
Buyer guidance
Ask for, in this order:
- Results on your language pairs and mixing styles, with the test set described and the date run. If the vendor only has monolingual scores, say so in the evaluation file.
- Error rates on names, numbers and negations, separate from overall accuracy.
- How native-speaker review was done: who, how many, how paid, how disagreement was handled.
- Training data provenance for each language. Ask whether public benchmark sets were used in training.
- What the system does when it is unsure. Confidence scores, fallback to a human, and refusal behaviour all belong in the test.
Voice and privacy. Recorded speech can identify a person. Under Nigeria's Data Protection Act 2023, as summarised by the Future of Privacy Forum, genetic and biometric data used to uniquely identify someone is sensitive personal data, section 30(1) restricts processing it, and sections 42 and 43 govern transfers abroad. Other countries differ. This is general information, not legal advice; confirm with counsel in each country where callers live. Practical steps: record the minimum, state training use in plain language, keep test audio separate from production, set deletion dates, and ask whether the vendor retains your data to improve its models. An open licence such as CC0 on a public corpus does not transfer to your own call recordings.
If you are the vendor or builder. Publish a slice-level evaluation: language pair, style, size of test set, error types, and the reviewers' background. State what the model fails at. A table of honest failures will move a careful buyer further than a leaderboard position.
Shortlisting. If you are looking for teams that work on language AI in the region, the AI and machine learning category and the directory give a starting list. Use Find Partner to describe your language pairs, then request the evidence above before any pilot. A profile tells you who to ask; it does not replace the test.
Frequently asked questions
Can I rely on a vendor's claim that it supports Swahili or Hausa? Not for mixed speech. Support usually refers to standard text in a language. Ask for results on your own slices, including a mixed-language set.
How many test examples do I need? There is no standard. Our rule of thumb is a few hundred per slice for decisions that matter, with more where a single error is costly. Smaller sets can still reveal failures.
Is Sheng or Nigerian Pidgin treated as its own language by benchmarks? Rarely. We found no benchmark covering Sheng in the text sources reviewed. Nigerian Pidgin appears in AfriSwitch for speech and in MORENA's stated language list, but coverage is not the same as tested quality.
Do open-weight models make testing easier? They let you run tests privately and fine-tune on your own data, which helps with voice data you cannot send abroad. They do not guarantee quality; check the licence and run the same tests.
Sources
- arXiv (Adelani et al.; NAACL 2025) - IrokoBench (technical)
- arXiv (Ojo et al.; ACL 2025 Findings) - AfroBench (technical)
- arXiv (Adelani, Abbott, Neubig et al.; TACL 2021) - MasakhaNER (technical)
- arXiv (NLLB Team) - No Language Left Behind (technical)
- ACL Anthology (Olaleye et al.; ACL 2025) - AfroCS-xs (technical)
- arXiv (Ashungafac, Awobade, Olatunji; preprint, 26 August 2026) - AfriSwitch (technical)
- PMLR volume 314 (O'Brian, 2026) - Sauti Halisi (technical)
- arXiv (Etori and Gini) - RideKE (technical)
- arXiv (Sheth et al.) - Survey of code-switched NLP (technical)
- arXiv (Tonja et al.) - InkubaLM (technical)
- Lelapa AI - Vulavula (primary)
- Masakhane community - Masakhane (ecosystem-data)
- Mozilla Foundation - How Rwanda is making voice tech more open (primary)
- Future of Privacy Forum - Nigeria's New Data Protection Act, Explained (institutional)
- Daba Finance - Vambo AI unveils MORENA (media)
- CIO Africa - Vambo AI releases open-weight MORENA model (media)
- Remio - MORENA analysis (media)