Comment by citizenpaul
9 hours ago
I think we hit peak AI improvement velocity sometime mid last year. The reality is all progress was made using a huge backlog of public data. There will never be 20+ years of authentic data dumped on the web again.
I've hoped against but suspected that as time goes on LLMs will become increasingly poisoned by the the well of the closed loop. I don't think most companies can resist the allure of more free data as bitter as it may taste.
Gemini has been co opted as a way to boost youtube views. It refuses to stop showing you videos no matter what you do.
> I don't think most companies can resist the allure of more free data as bitter as it may taste.
Mercor, Surge, Scale, and other data labelling firms have shown that's not true. Paid data for LLM training is in higher demand than ever for this exact reason: Model creators want to improve their models, and free data no longer cuts it.
I did read or listen on a podcast about the booming business of AI data sets late last year. I'm sure you are right.
Doesn't change my point, I still don't think they can resist pulling from the "free" data. Corps are just too greedy and next quarter focused.
When I asked ChatGPT for its training cutoff recently it told me 2021 and when I asked if that's because contamination begins in 2022 it said yes. I recall that it used to give a date in 2022 or even 2023.
Rule of thumb: never ask chatgpt about its inner working. It will lie or fabricate something. It will probably say something completely different next time
How? I just asked ChatGPT 5.2 for its training cutoff, and it said August 2025. I then tried to dig down to see if that was the cutoff date for the base model, and it said it couldn't tell me and I'd have to infer it from other means (and that it's not a fully well-formed query anymore with the way they do training).
I was double-checking because I get suspicious whenever asking an AI to confirm anything. If you suggest a potential explanation, they love to agree with you and tell you you're smart for figuring it out. (Or just agree with you, if you have ordered them to not compliment you.)
They've been trained to be people-pleasers, so they're operating as intended.
To be honest for most things probably yea. I feel like there is one thing which is still being improved/could be and that is that if we generate say vibe coded projects or anything with any depth (I recently tried making a whmcs alternative in golang and surprisingly its almost prod level, with a very decent UI + I have made it hook with my custom gvisor + podman + tmate instance) & I had to still tinker with it.
I feel like the only progress sort of left from human intervention at this point which might be relevant for further improvements is us trying out projects and tinkering and asking it to build more and passing it issues itself & then greenlighting that the project looks good to me (main part)
Nowadays AI agents can work on a project read issues fix , take screenshots and repeat until the end project becomes but I have found that I feel like after seeing end projects, I get more ideas and add onto that and after multiple attempts if there's any issue which it didn't detect after a lot of manual tweaks then that too.
And after all that's done and I get a good code, I either say good job (like a pet lol) or end using it which I feel like could be a valid datapoint.
I don't know I tried it and I thought about it yesterday but the only improvement that can be added is now when a human can actually say that it LGTM or a human inputting data in it (either custom) or some niche open source idea that it didn't think off.