Comment by nopelynopington

3 days ago

I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things.

I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had better results

6 comments

nopelynopington

Reply
Me1000  3 days ago

This is a technology demo, not a model you'd want to use. Because Bitnet models are only average 1.58 bits per weight you'd expect to need the model to be much larger than your fp8/fp16 counterparts in terms of parameter count. Plus this is only a 2 billion parameter model in the first place, even fp16 2B parameter models generally perform pretty poorly.

  • nopelynopington  3 days ago

    Ok that's fair. I still think something was up with my build though, the online demo worked far better than my local build

ashirviskas  3 days ago

> I've tried other very small models that run on CPU only and had better results

Maybe you can you share some comparative examples?

  • nopelynopington  3 days ago

    sure, here's my conversation with BitNet b1.58 2B4T

    https://pastebin.com/ZZ1tADvp

    here's the same prompt given to smollm2:135m

    https://pastebin.com/SZCL5WkC

    The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet

    Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)

    https://pastebin.com/PRFqnqVF

    Edit:

    I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?