Comment by springtimesun

I’m just dipping my feet in the water of local models and I really feel this. I had a simple alignment task (align known quality transcript without timestamps with timestamped but lower quality whisper transcriptions) and I went through 12 rounds of testing across 4 generations of 3 models. The results were all over the map even across versions of the same model. Spin out the task to something as big as coding and wow.

If you have any advice/blogs on doing project specific benchmarks I’d love to hear it. I’m trying, but it’s haphazard at the moment