Comment by mrandish
2 hours ago
> I can ask each model to write a doc about something I know well and tell you which output I like better.
That's what users currently end up having to do on their own. Or they just give up in confusion, which presents in your site analytics as either bounce, buy later or buy less.
Why not have a model comparison site? Visitors choose from a dozen-ish broad categories of work (eg technical documentation, market communications, etc), then pick from a dozen sample prompts across typical tasks in that category, and see the model responses with avg time and approximate cost. Especially helpful would be having a toggle overlay with annotation highlighting relevant differences, then if they have (or create) an account, they can edit the annotations and add their own with a sharing link.
If my job was to actually help my co-workers and friends make better informed choices they feel more confident in, I'd be doing something like that. The extra credit version would help users narrow down to two models and then let them convert to a 'Test Drive' trial - same workflow but from their own prompts, side-by-side A/B response comparisons with annotation and sharing link.
As you said, the answer to "Which model is best for me?" is actually complicated, conditional and inevitably a bit confusing. When that's the reality, the right pattern isn't 'Communicate to Convert', it's 'Build credibility and confidence with maximum clarity and genuine helpfulness'.
Sounds like arena.ai: https://arena.ai/leaderboard
People vote on pairs of model responses and they aggregate everyone's votes into a leaderboard.
One thing to be aware of that the distribution of prompts on arena.ai can be pretty different than real-world usage. The people who spend time on that website casting votes tend to be more interested in entertainment than professional productivity. And turn-based voting can punish useful behavior like asking clarifying questions instead of guessing at assumptions.
Yes, I've seen Arena.ai and the reasons you cited are why I suggested something better that's more private, personalized and shareable internally without being skewed and gamed.