Comment by chris_va
15 hours ago
I am kind of amazed how negative the comments are here, especially on HN.
Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
If he has said "to learn the math and programming skills needed to understand how to build LLMs" it'd have been much more positively received.
> I am kind of amazed how negative the comments are here, especially on HN.
I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.
Hazarding a guess, I think the moment Altman became the CEO and later during COVID, the sentiment seemed to have been shifting towards what we see today. But this is all based on hazy memory, rather than looking at the data. I'm sure there is a blog post waiting to be written about analyzing the sentiment of comments to PGs articles on HN, and you'll see a shift somewhere.
Because at some point in life everyone gets tired of fairytales. He started mending the anecdotes to his content which always rubs people the wrong way.
So, this comment of yours obviously isn't in the "knee-jerk reaction" category of comments, I suppose? What exactly from the linked tweet(s) are fairytales here? There is hardly any text at all, so strikes me as a comment about previous pg content, but then this would be one of those comments I talk about? Very confusing.
1 reply →
Imo it's a breakdown of trust of the startup ecosystem as a whole. Repeatedly startups have enshittified and it's become undeniable that the investment apparatus around startups is partly responsible. We have seen a great driver of uncreative destruction, industries undermined, small businesses undermined just to drive masses of money into few pockets - less fairness for the people working in what is now the gig economy and ultimately prices and other costs that end up as high or higher than they were before for consumers. Not to mention the whole AI/OpenAI situation which many perceive as threatening their skillset per se, essentially tearing up the social contract that existed on this site.
The sycophancy on HN is starting to break down because there is a higher proportion of users sceptical towards the outputs of the VC and wider investment world than ones who believe they're potential beneficiaries of it.
Tech industry people are becoming less interested in HN as a warm handshake into the startup world because, frequently, they're disgusted by it. And this reflects on the sentiments people post on PG's articles.
Increasingly if those at YC want the same kind of low-bar praise they got before, they will need to get it from machines.
> I can't recall or point out exactly when, but there is a stark before/after moment where the opinions of anything pg went from "Interesting and maybe true in some ways" to what we see today, lots of knee-jerk reactions and hardly any comments about the actual content.
Hard disagree. This submission is still being highly upvoted, while another recent post[1] on the harms caused by Graham’s fellows[2], with a fairly tame comment section, has been flagged. That is a constant on HN. It’s not a fluke, it’s as predictable as the sunrise and getting more pronounced.
I’m sure we’re both biased in our perceptions. Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote, together with others like Musk, until things started to really go to shit and many eyes have been opened to the effects of the unfettered greed of rich tech guys out of touch with reality.[4]
[1]: https://news.ycombinator.com/item?id=49411762
[2]: A better English word is escaping me.
[3]: That word I choose hyperbolically but deliberately. It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.
[4]: That is not “knee-jerk” but a slow realisation still ongoing.
> Mine is that HN in general (certainly more than any other website) used to worship[3] everything he wrote [...] It definitely was not “interesting and maybe true in some ways”, it was much more hardcore than that.
I guess it depends on what submission you look at, previous comment of mine solely based on memory. Now I went to https://news.ycombinator.com/item?id=19418701
Maybe you need to enable "Show Dead" or something? Pgs articles on HN definitely never was free of any critique in the HN comments, just like any article. Although I do agree with you that it used to be different than it is today, and same with Musk too, and Altman, and probably more individuals, where they were lauded before but now pretty much just mentioning them poisons the conversation.
3 replies →
I think it is more about people are a bit sick of filthy rich people giving this kind of advice. I would also not read anything he preaches.
Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
https://thinkingmachines.ai/tinker/ https://github.com/thinking-machines-lab/tinker-cookbook
(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").
TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.
One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.
There are a lot of other high level abstractions here to look at. Prime Intellect has one.
The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
His writing was overrated: It's that simple. People now see his blog posts for what they are: decent blog posts.
Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM? It seems like another thing that needs big investment to work properly, unlike those other techs where results can be shown even with little materials.
Yeah, there are retrocomputing hobbyists who mess around with sometimes-physically-large computers that were important many decades ago. I don't know if anyone is hacking on vacuum tubes of the kind that you could in principle build a computer with - there's a reason they became obsolete for digital computation almost as soon as the transistor was invented. On the other hand, I personally think it would be neat to try to build a CRT in a garage, which is of course a type of vacuum tube. I don't think this would be an easy garage project, but it does seem like might be tractable for someone who understand physical manufacturing and electronics well, has access to glassblowing equipment, etc.
There are indeed!
Vacuum tube computer: https://www.youtube.com/watch?v=KAlnJnYt5do
Making vacuum tubes in a garage: https://www.youtube.com/watch?v=-UEfqAWb3fE
> Now I'm curious, do people actually tried to hack vacuum tubes or other big servers that's barely 1MB RAM?
"Barely?"
It's insane to lump vacuum tubes together with servers with 1 MB RAM. My first PC, which I used for a decade, had 640KB RAM. And that was an upgrade from 512 KB RAM. My other PC had only 128KB RAM. None of these were considered the equivalent of (by then long dead) vacuum tubes in their day.
You can get a lot done in 1 MB.
> With LLMs and online tutorials, this isn't even a difficult suggestion.
Don't many of the commercial ones prevent you from using them to build LLMs?
I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.
> Do people think learning new tech is somehow wasted effort?
No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.
But why are people cynical? they ask.