Comment by hedora

9 hours ago

RAM requirements? My current rule of thumb is “a byte per parameter”, but I doubt this runs in 1/9th that (~ 3GiB).

Also, perf speedup?

2 comments

hedora

Reply
kennywinker  4 hours ago

Without context, it should be number of params * 1.76 (the “effective bits per weight”) / 8

So for this one, 27B * 1.76 / 8 = 5.94 GB

For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:

300GB/s / 5.98 GB = 50.5t/s.

Realistically it’s probably a bit slower, but that is your theoretical maximum.