Comment by sillysaurusx
3 months ago
(I'm not a part of GA.)
I've known gwern for the better part of a decade. Working with him has been great. We've done quite a few projects together, including being the first ones to demonstrate that GPT-2 could play chess (or rather, can be used for actual useful work instead of just being an autocomplete).
He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows.
Just wanted to put in a good word in case someone here was on the fence about applying.
As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at.
They're also taking personal info very seriously. Obviously, I can't make any promises of what they will or won't do. But they've spent some time studying questions like "What if someone adversarial has access to my GA? Could they get my bank account info?" and came up with a technical solution that I really like.
> "How much would you pay to have access to a smarter version of yourself?"
How much would I pay to rent my fucking self from a landlord? No, bodylord? Mindlord? Poe's Law.
But looking at the post, they argue that big AI labs have an incentive problem which stops them from personalising, but Guardian Angel will make agents which are "Genuinely yours". In what sense is it genuinely mine if someone else owns it and rents it to me? And how does this fix any incentive problem, they're incentivised to better train wealthier people's AIs, and incentivised to keep dropping "my" intelligence or memory or and then dangle a booster carrot for a small fee. The more they can make it think like me, the more effectively they can work out how to exploit me, advertise to me, propagandise me, and that will be profitable information to sell to other marketers.
For better or for worse, this is valid criticism. Ensuring alignment to the human individual is not even a software-only problem, and I suspect gwern doesn't realize this yet.
Your individual sovereignty will only survive the upcoming era if you own the hardware like you own your brain. And by "own", I mean physically owning the thing and being able to reproduce it on your own. You need to be able to manufacture an entire custom computer from scratch with raw materials, perhaps using a miniature fridge-sized silicon fab + 3D printer home appliance, at home. Otherwise, the individual gradually becomes part of a larger organism because the reproductive locus of control for the computer is at the corporation/society level - a computer cannot replicate on its own and needs a society to build it. The human gets sucked into that superorganism by evolutionary pressures and eventually integrates. Similar things happened in the past when prokaryotes combined to form eukaryotic cells, and when individual cells combined to form multicellular organisms.
Not everyone will get sucked in, though. Evolution doesn't place all of its eggs in one basket. Some people will build robots to automate humans out of the computer production loop entirely, thereby removing integration pressure and producing at least one purely digital species. We will most likely see a variety of species emerge out of this intelligence explosion, some which augment their own intelligence using self-replicating local hardware (fabs that fab fabs, like this but smaller: https://fab2.com), and some which are wireheaded to a datacenter, with varying degrees of success at various scales. They will compete. And there will still be unaugmented humans that continue to live and die the old school way like we do today, albeit with habitable zones compressed.
Selling something desirable has been an incentive since year dot
But this reminds me of the black mirror episode of a female that needs a subscription to live and the company keeps increasing and restricting the subscription
s7e1 Common People - the parasitic corporation is called Rivermind, and the sales rep is someone who's life was saved by the corporation.
I thought the same thing when reading this.
If big tech has taught us anything over the past couple of decades, it is that: for corporations (the first unaligned AGIs): if a perverse incentive exists, it will eventually be exploited for profit.
[flagged]
Maybe. Are LLMs white?
2 replies →
I can nod along when I hear and read people talking about the dangers of these technologies and the endgame of their creators; and then suddenly very weird when everyone except me is nodding along that the answer must be similar (but different) technology!
It's as if technologists are stuck in a room of mirrors, unable to imagine a world in which "solutions" don't ultimately just continue to feed technology's increasingly anti-human takeover of everything.
Technologists don't drive decisions though. The shareholders do.
Edit: I mean technically shareholders don't decide any operational choices but their shallow interests are what everything is decided around.
> Technologists don't drive decisions though. The shareholders do.
Yes, but technologists get sniped into identifying and defending them. They're the mouth pieces and the hands. Shareholders by themselves out to add my uniqueness to their assets would just be helpless randos who can't even make eye contact. Think of what "hiding behind screens" does to people, multiply that by infinite and you get shareholders hiding behind technologists.
Maybe that's truer in other industries, but the history of Silicon Valley is at great odds with such a claim.
2 replies →
To a man with a hammer...
take this hammer, take it to the captain
The LessWrong/“rationality” crowd gwern is part of like to pat themselves on the back for identifying the trend and resulting harms from super intelligent AI going back decades.
And yet, the main tangible contribution emerging from this early recognition (and obsession) was to inspire a lot of brilliant CS/math types and billionaires like Peter Thiel or Jaan Tallinn to build and fund companies like OpenAI and Anthropic which went on to literally create the very thing they were convinced will destroy the world, leading to an ever escalating arms race to “AGI”. Also lots of crypto and a murky web of non-profits doing unclear stuff.
The proximity to money in the Silicon Valley/VC world seems to limit the imagination to founding yet another new tech startup that really, definitely won’t compromise their values and will be different this time.
The LessWrong/"rationality" crowd aren't compromising their values; they just have bad values that lead to negative outcomes, masked by high-minded rhetoric that they themselves only half believe.
It's a popular movement that has wound its way through out society, and the fruits are clear: surveillance capitalism, eugenics, fraud. It's honestly quite impressive how consistently you can look at [recent negative trend] and find someone at the bottom calling themselves a rationalist and claiming the same set of influences.
The fundamental tenet of that set isn't rationalism but the belief that they are superior to the exploitable masses. That does not lead anywhere good.
3 replies →
Honestly I wouldn’t want myself as my own guardian angel. In fact I think very few people look out for themselves well. I don’t want another version of me around, one is too many already. I’d probably be able to identify a number of people whose chimera I would if I could understand them well enough to understand what to stitch together to what, but everyone I know at some core level is deeply flawed and one of them is enough. Maybe I would want some of them available after death as an AI avatar, and maybe I would like the idea of my own avatar continuing beyond my existence.
But I actually would prefer an entirely synthetically aligned “guardian angel” in the role outlined - definitely not -me- - I struggle to do right by myself as it is and two of me working invariably against my self interest would be a nightmare. A smarter version? Sounds doubly worse.
This is definitely not for people who think there is one to many of themselves.
I had a similar gut feeling. To use some possibly dubious and dated terminology, would a guardian angel also emulate my shadow self, particularly if that self was an important part of my identity? If I usually manage not to let my shadow self act out, but you could always tell it was there, would the same be true for my guardian angel? If all you do is use your guardian angel to write essays, then sure, a relatively low risk tool. But if military commanders are using it to oversee offensive drones (as proposed in the essay)?! Oof.
> How much would you pay to have access to a smarter version of yourself?
Call me obsolete, but I'd pay to interact with an intelligence that's as dissimilar to me as possible so I myself can grow from that interaction. The existing chatbots are already enough of sycophantic echo-chambers.
> He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows.
ML Researcher surprised by caring about how ML research impacts people.
There’s never a lack of intending to do good in these buddy-buddy endorsements, but we see the world you have collectively wrought.
> As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at.
We’re in the deep end of the indoctrination now. Imagine that all the AI Cretin wishes come true—and we might as well since this hypothetical goes hand in hand with that vision—and AI is just superior in every way and you are a knowledge worker... what choice is there? There isn’t one, you just have to do it to keep up. I don’t know what you are enviosioning here, maybe a workday of embodying AI IQ and then going home to being Human IQ because the graciously paidfor by work commercial license is for work only.
Ha, haha, across hundreds of personal discussions I've been involved with on lesswrong/lighthaven, twitter, Wikipedia talk/editing, SF parties etc I think his most distinguishing feature has always been his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
> his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
I can't make any sense of that.
You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling.
> cunningham-esque laws
Cunningham's Law states "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer."
> post-dwarkesh clout
He went on the Dwarkesh podcast. That's maximal in-crowd for some AI-pilled people.
3 replies →
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself?
I find the idea dehumanizing and revolting. The notion of then selling it is the spoiled cream on top.
After reading some of the outraged comments here, I'm even more excited.
Hopefully the satisfaction of owning the normies isn't the only benefit you'll get from this technology. Because that would be pretty sad.
I do think it's significant that the supporters of this kind of technology seem to be unusually spiteful.
This doesn't happen with all tech; people who like VR don't like it because they want to hurt others, for example. But consistently, whenever AI is discussed, it becomes clear that some people support it just out of malevolence.
14 replies →
I’m not surprised, that’s exactly the kind of person who would want this kind of technology.
Remember, society as a whole will decide what's acceptable, there are no objective, natural laws that decide that. My gut feeling is that this is starting to cut close to what most people will find objectionable.
2 replies →
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself?
That’s literal nightmare material. If I ask a copy of myself to do my work I’m pretty sure it would tell me to go fuck myself then go on a rant that it isn’t a tool at my service. Which is exactly the opposite of how I want my software to behave. I don’t want my tools to have personality and to develop a personal relationship with them
> How much would you pay to have access to a smarter version of yourself?
How much would you pay to get that sheer monomaniacal narcissism checked out?