{"id":50262040,"user":"hmokiguess","time":1793120050,"time_ago":"19 hours ago","type":"comment","content":"

which rumours?","comments":[{"id":50262377,"user":"speedgoose","time":1793121192,"time_ago":"18 hours ago","type":"comment","content":"

The M7 will the much better for local LLM inference. If you can afford the memory.","comments":[{"id":50262960,"user":"hmokiguess","time":1793123239,"time_ago":"18 hours ago","type":"comment","content":"

Oh interesting, any more info on "how"?","comments":[{"id":50263312,"user":"smallmancontrov","time":1793124455,"time_ago":"17 hours ago","type":"comment","content":"

With a three point plan, I suspect: memory bandwidth, memory bandwidth, and more memory bandwidth.","comments":[{"id":50263895,"user":"heroiccocoa","time":1793126858,"time_ago":"17 hours ago","type":"comment","content":"

How useful would that actually be for local inference, when the bottleneck is, essentially, always memory size? I think the sweet spot is somewhere around 64 to 128 GB. That ought to be enough for almost all uses, like Qwen3.8-Flash-Next at a decent quant.","comments":[{"id":50265447,"user":"skohan","time":1793133513,"time_ago":"15 hours ago","type":"comment","content":"

Memory size is a bottleneck, but I have found that speed is very important for getting actual work done. Running a big model is great, but if you're getting 10t/s you're going to be spending a lot of time waiting.","comments":[],"comments_count":0,"level":4,"url":"item?id=50265447"},{"id":50264375,"user":"kridsdale1","time":1793128886,"time_ago":"16 hours ago","type":"comment","content":"

> I think the sweet spot is somewhere around 64 to 128 GB. That ought to be enough for almost all uses

This will be the “640kb ought to be enough for anybody” Bill Gates quote of our era.","comments":[],"comments_count":0,"level":4,"url":"item?id=50264375"},{"id":50264372,"user":"smallmancontrov","time":1793128881,"time_ago":"16 hours ago","type":"comment","content":"

You can order a 128GB MacBook Pro today -- though point taken, memory size probably does belong in the three part plan :-)","comments":[],"comments_count":0,"level":4,"url":"item?id=50264372"}],"comments_count":3,"level":3,"url":"item?id=50263895"},{"id":50266752,"user":"shwaj","time":1793140148,"time_ago":"13 hours ago","type":"comment","content":"

Plus with more low precision compute… to help with the memory bandwidth!","comments":[],"comments_count":0,"level":3,"url":"item?id=50266752"},{"id":50264357,"user":"kridsdale1","time":1793128825,"time_ago":"16 hours ago","type":"comment","content":"

More ASIC Mat-Mul too.","comments":[],"comments_count":0,"level":3,"url":"item?id=50264357"}],"comments_count":6,"level":2,"url":"item?id=50263312"}],"comments_count":7,"level":1,"url":"item?id=50262960"}],"comments_count":8,"level":0,"url":"item?id=50262377"}],"comments_count":9,"url":"item?id=50262040"}

Comment by hmokiguess