Study finds LLM summarization strength in English does not carry over to Bengali

Key points

• Benchmark tests eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets, zero-shot.
• GPT-5.6 Luna leads several English lexical metrics but loses that edge in Bengali.
• LLaMA 3.3 70B reportedly achieves the highest ROUGE and BLEU scores in Bengali; Gemini 2.5 Flash the highest Bengali factuality.
• Different evaluation methods produce different model rankings, so single-metric assessment is limited, according to the authors.

Why it matters for marketers: Content teams serving Bengali audiences should not assume an English-leading model will perform best in Bengali, and should test factuality alongside surface metrics.