Comment by AmazingTurtle
14 hours ago
There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon.
At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo.
The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.
I see your point. As an oss defender myself, I agree, however, the spirit of this is to see how fast I can make it. I'm sharing this with the community, which I think is aligned with the original oss spirit.
It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.
Hey carloslfu, kudos from the other side of the internet, don't get down on people nitpicking everything here, experimenting and discovering is part of learning so keep going!, remember this is the place that said dropbox was dumb and could be replaced by a script.
I think multiple people working on the same thing is great.
Everyone comes at it from a different point of view, and some approaches work, some don't. And when people do this themselves they learn. Existing projects have their mistakes worked out already.
Maybe one of these people is going to come up with the thing that nobody else thought of because of their experience working the problem from scratch. You may not get that from someone working from an existing project, because existing projects have their approach "baked in."
What all these projects are showing so far is that it's possible to stream from disk, but that the performance isn't ideal. But I'm sure you could take this approach with smaller models and get better performance.
In addition, it's a given that when you work with large data sets performance means organizing the data to take advantage of caches, both disk and cpu. It's not clear how that would work, exactly, given that each run is a not-quite-random walk through the data. The Big Data way is to prebuild all of that as much as possible, which is probably impossible with a big model. But what about a smaller model?
true
This is one of the aspects of this year that I've been finding very grating and wasteful. Collaboration still happens among people with the ability to do so and the technical skills, but everyone else is taking their own helicopter to the top of the mountain, "putting it out there", and there's just a ton of redundant projects that do the same thing.
It's horrible. Every 20-something working on a load-bearing inference engine on GitHub.
for the record, I'm almost 35
1 reply →
Vouched especially since OP might have a perspective on this. And readers may want to look up those other repos and compare for themselves.
Thanks for the feedback! I'll create a section with a benchmark and comparisons. This will hold the project accountable and speed things up imo
Absolutely! Nice.
(The comments under the parent indicate it was improperly flagged/made dead (maybe could happen just from downvoting?) so glad I hit the Vouch.)
Why should they do that? For you? You could merge those projects and see if they get traction.
And there are `Mference` and `SwiftLM` too, I think they are doing the same use case.
AI NIH
Sorry, I don't get "NIH". what's that?
Not Invented Here.
4 replies →
> every implementation idea rediscovered five times and wrapped in a new README
That's open source since forever, unfortunately.
I agree with the sentiment, but have you seen those videos in which all men say other men are gay? This feels like the same, so much AI paranoia!
I genuinely want to contribute. And hey! I was doing oss this since 2014 so waay before AI was cool.
It's what happens when you don't do market research.
I'm sorry this makes it seem like I didn't do my research. I did a TON. To fix it I'll add a benchmark/comparison table. Also, I wouldn't call it market research since this is not commercial AT ALL.
2 replies →
Does a painter check to make sure that a portrait hasn't been painted? What a dismissive comment.
Half the people on here are using Ollama. No one is doing market research.