Comment by cobolcomesback
4 days ago
Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.
If "pi install npm:pi-mcp-adapter" is preventing them from including pi in benchmarks, then they might not be competent to run trustworthy benchmarks.
There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark? And who should be the tie breaker when someone thinks they should use a different plugin? Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
It’s like you didn’t even read my comment.
It's a fair point, but for any well-established capability, the plugin with the most monthly downloads (overwhelmingly) is generally going to be "the one". For MCP, the top plugin [1] has 1M/mo, and the next one down has less than 2k/mo.
I do agree that in the case of benchmarking, it's a bit silly to put the onus on the benchmark runner to try and pick the right plugin for each capability that won't make someone point to /that/ as the reason Pi didn't do as well as expected.
[1] https://pi.dev/packages/pi-mcp-adapter?name=pi-mcp-adapter
> There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark?
The one that everyone else uses, the most popular. Exactly the one I listed.
I wrote "pi install npm:pi-mcp-adapter". It would install the mcp adapter.
> Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
That's the whole point of Pi, it doesn't have to reach checklist level parity with Strand. Just benchmark it with its default config. Add mcp adapter if your benchmark really requires it, but that's it.
> It’s like you didn’t even read my comment.
I did, but it seems you didn't read my comment, or you didn't understand that the command I gave you is an actual recipe, and not a pseudo instruction.