← Back to context

Comment by ddevnyc

8 hours ago

Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?

5 comments

ddevnyc

Reply
dannyw  8 hours ago

MoE has nothing to do with multimodality.

MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.

Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.

As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171

  • dummydummy1234  6 hours ago

    Wait I thought the router ends up specializing the experts?

    Like there is no explicit goal aside from each 'expert' getting roughly equal weight?

    And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?

    I do not know what I am talking about, this is my limited understanding...

    • ahepp  17 minutes ago

      My understanding of it is also pretty surface level, but I was not under the impression that it develops "expertise" in a particular subject matter, at least not in a way that's easy to harness. From what I've read, it develops expertise at the token level.

      Because the natural continuation of this is to say like, "Ok I want to load the bird detection expert and the navigation expert but leave the medieval European history expert behind", and my understanding is that this is not really how it works. At least at the moment.

    • juliendorra  32 minutes ago

      You are right that at training the main goal is balancing between the sections to avoid certain paths becoming the only path. In the end the inference will be routed token by token to a mixture of say 3 or 4 sections of the model. The combination can change at each turn. It’s really a statistical optimization. As for many things in neural networks, the original intuition coming from anthropomorphism once implemented becomes something very non human!