الکس چیما و اَش هارت اومدن یه راهکار مشتی برای اجرای بهینه‌ی هوش مصنوعی رو ترکیب سرور و مک‌بوک رو کردن

Excited to collab with @ashxhart on this! Why does prefill/decode disaggregation work on DGX Spark + Mac ? DGX Spark: ~350 TFLOPS@fp4, 128GB@273GB/s M5 Max MacBook Pro: ~70 TFLOPS@fp4, 128GB@614GB/s Prefill is compute-bound. Prefill on Spark runs ~5x faster than M5 Max. Decode is memory-bound. Decode on M5 Max runs ~2x faster than Spark. Simple idea: run prefill on Spark, decode on M5 Max. That's faster than running on one device alone, since each phase runs on the best device for that phase. This requires sending the KV cache from the Spark to the Mac. With a 10 Gigabit connection, we can overlap fully computation and communication of the KV cache. This doesn't require low latency, and works just fine over TCP/IP. RDMA on the other hand enables low latency networking (~single digit microseconds), which is the unlock for scaling up decode with ~equal memory bw devices e.g. DGX Spark + M4 Pro / M5 Pro. Low-latency is also important for DGX Spark + RTX or Mac + RTX. Small pool of very high speed memory + large pool of slower memory is the ideal local hardware architecture. Idea is to put the attention part of the model on the faster small memory pool, and the MoE part on the slower large memory pool.