← Back to context

Comment by sachaa

18 hours ago

This is a great way to start but the perf of LLM models like Qwen are not ideal for local execution. I have adapted Laya (pure decision model) to run in a browser and I am able to get responses under 200ms. Give it a try: https://wexare-ai.github.io/browser-laya/

8 comments

sachaa

Reply
miguelspizza  7 hours ago

You can easily get under 100ms per decision in browser with a Qwen based decison model: https://alxnahas.github.io/strands-decider-web/?backend=engi... There is nothing about Qwen or Laya that make one or the other better at running in the browser. It is really just: how large is your model and how does KV cache expand as input length increases

  • catlifeonmars  7 hours ago

    I guess it depends on your application, but 100ms is pretty slow. Plus there is the model footprint (memory + cpu/gpu).

make3  17 hours ago

Laya is just ModernBert fine-tuned. It's still a language model (just a bidirectional one). Let's call it what it is instead of this weird Decision Model mysticism.

  • arendtio  16 hours ago

    I am also not a big fan of everybody calling them system one models. AFAIK, system one also refers to activities like driving a car, but I don't want to have those decision models driving cars.

    So I think I understand what is meant, but I don't like the comparison.

    • tvink  14 hours ago

      Metaphors are easy to disagree with if you don't attempt to understand what they are used to try to communicate.

      System one was an attempt to distinguish the "thinking" of thinking models (which is more akin to system two thinking with deliberate arguing/building up a decision) from fast instinctive thinking. It's a really good metaphor for the kind of trade-off or use case this applies to.

      It is not and was never implying that it is a replacement for all human system one thinking activities.

  • user43928  15 hours ago

    Is this mysticism what TypeSafe AI is valued at $7.5b for?

    Maybe the people talking about an AI bubble do have a point. I am getting the impression here that investors are throwing money at everything.

    Personally I believe AI labs have a solid business model that could soon be very profitable, but this here has me doubting now.