Comment by Al-Khwarizmi
8 days ago
If you have some spare time, I'd be interested in knowing what kind of questions you use to test models on understanding of Chinese culture.
8 days ago
If you have some spare time, I'd be interested in knowing what kind of questions you use to test models on understanding of Chinese culture.
I'm creating hanzirama.com
I generate explanations for characters and words like so: https://hanzirama.com/character/%E6%9D%A5#explain
But I don't want to mislead learners and want to provide some cultural depth, so I have a hole sophisticated pipeline, using multiple models to generate the explanation, then multiple models look for issues in the explanation, each issue goes through the panel of judges (basically trying to squash down any hallucinations), it's fixed and it goes through such cycles a few times over.
I've been at it for some months now, so I have dozens of different probes, that I needed to evaluate prompts and method changes. Plus on some items I generated so many explanations through different means that I can tell a lot about given model just by looking at one.
Plus I'm doing some statistics, so I see how e.g. when working as judges of issues some models correlate heavily with some others... Fun fact during some testing runs basically just testing providers I stumbled upon qwen introducing himself as made by Google. And also Anhropic's Sonnet saying that it was made by OpenAI :)
At this point all my evaluations frameworks and pipelines stuff is much bigger than the site itself. I'm having lots of fun though.