Comment by Scaevolus

This is a solid improvement over Needle 2, which I tried using for a tool-calling interface to a Runescape database site. Unfortunately it's still not quite capable enough for my target compared to FunctionGemma.

   Model                            Correct tool shape    Exact arguments
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━
   FunctionGemma fine-tune, BF16       209/230 (90.9%)    196/230 (85.2%)
  ───────────────────────────────  ────────────────────  ─────────────────
   Needle 3 fine-tuned W4A8             74/230 (32.2%)     47/230 (20.4%)
  ───────────────────────────────  ────────────────────  ─────────────────
   Needle 2 fine-tuned W4               59/230 (25.7%)     43/230 (18.7%)