Study finds LLM summarization strength in English does not carry over to Bengali
Key points
- Benchmark tests eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets, zero-shot.
- GPT-5.6 Luna leads several English lexical metrics but loses that edge in Bengali.
- LLaMA 3.3 70B reportedly achieves the highest ROUGE and BLEU scores in Bengali; Gemini 2.5 Flash the highest Bengali factuality.
- Different evaluation methods produce different model rankings, so single-metric assessment is limited, according to the authors.
Why it matters for marketers: Content teams serving Bengali audiences should not assume an English-leading model will perform best in Bengali, and should test factuality alongside surface metrics.