Comment by er4hn
4 hours ago
It's hard to try and compare these benchmarks against real world use cases. As an experiment I let Meta Muse make all major decisions on a weekend trip to Seattle ~2 weeks ago. I think this would have been using Muse Spark 1.3, which is equivalent to GPT 6 - Sol.
On the plus side it told me taking the light rail from SEA into the University district was faster and cheaper than Uber.
On the downside it would recommend restaurants that were out of business, checking out Salmon swimming through locks out of season, got the opening of Beecher's Cheese in Pikes Place wrong, and fumble it's way through most simple planning decisions. I think my favorite was when it asked me what flight I have tomorrow so it could help plan when to head to SEA. It sent me a very panicked message that the flight was leaving today! Very soon! I had to remind it this is a daily flight and I am booked on the next day before it checked and apologized.
For things that are verifiable, and code tends to be that way, these models can do all sorts of things. But I find it hard to reconcile Meta's goal of having a personal assistant bot with something that is unable to do the simplest tasks.
> checking out Salmon swimming through locks out of season
To be fair, the Coho salmon run at the Ballard locks peaks in September. It's possible you just had bad luck that day.
https://ballardlocks.org/fish-salmon-ladder.html
I run by the locks a few times a week and while I don't usually see fish, the presence of sea lions the last few weeks tells me that at least some salmon are running right now.
There were a number of sea lions hanging out there! It's entirely possible they were snacking on whatever salmon were going by. Not all was lost though, I got to enjoy watching some boats go through the locks.
I'm assuming it was out of season because in talking with the Uber driver afterwards he said we'd missed the end of the season. I'm assuming his "local model" was up to date and accurate.
ah if there's one thing I miss about Seattle it's living near the locks / discovery park, such a magical area!
but that was a different, much smaller and less capable model