Comment by hedora
1 day ago
Do database dumps of it still exist?
Maybe it’s time to just download it and Wikipedia, then point an LLM / vectordb at it.
1 day ago
Do database dumps of it still exist?
Maybe it’s time to just download it and Wikipedia, then point an LLM / vectordb at it.
Yes they do, there are archives that are up to date for the month.
See if https://github.com/Tryhard-cs/reddit-download-tool works, the readme enumerates the data sources it pulls from.
Perhaps the way to solve for X is having a scraper serve the data via torrent and/or IPFS vs centralized systems like nitter and xcancel.