Comment by feconroses
1 day ago
Very cool project! Quick question: is the underlying Pushshift dataset updated with new Reddit data on any regular cadence (daily/weekly/monthly), or is this essentially a fixed historical snapshot up to a certain date? Just want to understand if self-hosters would need to periodically re-download for fresh content or if it's archival-only.
the data from 2025-12 has been released already, it is usually released every month, it just needs to be split and reprocessed for 2025 by watchful1. i will probably eventually add support for importing data from the monthly arctic shift dumps so that archives can be updated monthly.
https://github.com/ArthurHeitmann/arctic_shift/releases
Arctic Shift https://academictorrents.com/browse.php?search=RaiderBDev
Watchful1 https://academictorrents.com/browse.php?search=Watchful1
Is data web scrapped? Is reddit ok with that?..
[dead]