Comment by mips_avatar
2 days ago
Qwen35ba3b can do a huge amount of data cleaning work on pretty modest hardware. Already have run about 100 billion tokens on it using 2x3090 gpus.
2 days ago
Qwen35ba3b can do a huge amount of data cleaning work on pretty modest hardware. Already have run about 100 billion tokens on it using 2x3090 gpus.
What exactly do you mean by data cleaning
The coolest project I’ve got this running on is improving the depicts metadata for photos on Wikipedia. A lot of times they won’t have the landmarks tagged correctly in a photo. So I will load in all the metadata that exists from each photo and the pixels of those photos and give a small qwen agent access to Wikipedia search as well as a geocoder. It does a great job of figuring out what is depicted and tagging it with the correct depicts field. Im still early on but I have been able to double the number of places that have a photo attached to them on wikimedia
Interesting project. So I guess the search tool is to crawl Wikipedia for articles? And how do you ensure that the tagging stays within the Wikidata taxonomy? How exactly are you using a geocoder? Sorry, just curious
1 reply →
are you using it differently to OpenRefine?
I’ve found it helps a lot with reconciliation tasks where tools like openrefine can’t handle it. Like I wanted to tag blog posts with links to Wikipedia articles that are relevant. But the thing is whats relevant changes a lot based on context in the blog post. So like a naive reconciler will tag the article for “sky” in the blogpost title “the sky above the Notre dame shone the morning we visited” when the element that should be tagged is the cathedral.