همل حسین دست گذاشته رو یه نقطه حساس و حسابی از کیفیت درپیت بنچمارک‌های عمومی هوش مصنوعی شاکی شده؛ همون اعدادی که کل این صنعت دارن چشم‌بسته روشون قسم می‌خورن

I've been looking into public benchmarks and am astounded by the poor quality vs. how much we rely on them as an industry. @xeophon pointed me to this excellent site https://epoch.ai/benchmarks/ which is an independent third party which scrutinizes benchmarks and "looks at the data" 💪 For example, here are some popular benchmarks we all rely on. Before trusting a benchmark its worth running them and inspecting examples the grader failed carefully, to get a sense of the quality. I'm continuing to look more into this, and will write something up eventually once I've collected more data, but I think this is an important subject.