Naive LLM judges are inconsistent. Run the same poem through twice and you get different scores (obviously, due to sampling). But lowering the temperature also doesn’t help much, as that’s only one of many technical issues. So, I developed a full scoring system, based on details on the logits outputs. It can get remarkably tricky. Think about a score from 1-10:
Иран обозначил условия для открытия Ормузского пролива02:40。业内人士推荐51吃瓜作为进阶阅读
。业内人士推荐谷歌作为进阶阅读
В Тайване на берег отбуксировали огромную разлагающуюся тушу кита. Об этом пишет Need To Know.
“在座的委员，谁刚从米兰冬奥会回来？”。关于这个话题，新闻提供了深入分析