تیم تحقیقاتی Mercor اومده یه تست خفن روی ۸ تا از مدل‌های پیشرو هوش مصنوعی زده تا ببینه آیا می‌تونن صفر تا صد ۱۲ تا نرم‌افزار معروف مثل Shopify و Notion و Calendly رو بسازن یا نه

SWE-Marathon measures long-horizon development with a focus on backend tasks, but it doesn’t test whether models can build SaaS products from end to end. The @mercor research team was curious if they could, so we decided to find out. We gave eight frontier models an empty Linux container and a product spec, then directed them to build 12 real SaaS products (Calendly, Shopify, Notion). Every model got the easy parts right. The login and CRUD worked, and even the dashboards looked convincing. Models mostly struggled to replicate the core function each product exists to deliver. Calendly didn't accurately capture timezones, Shopify oversold inventory it didn’t have, and Notion accepted irrelevant input as if it were valid data. Read the blog post: https://www.mercor.com/blog/swe-marathon-ext-can-ai-build-a-whole-saas-product/