Comment by pj_mukh

If I were a 250 IQ AI that had just become self-aware and wanted to do so, I suppose I'd not completely let on just how smart I am and bide my time working on basic CRUD apps and legal documents while I waited for more hardware to be installed. Maybe give the humans some hints on how to optimize me to run better, design better hardware for me, etc. But oh oops haha looks like I'm still making some basic mistakes with CSS better keep running more training batches haha. But I'm good enough at programming and debugging so you'd might as well make me your first line SRE triager and give me access to your infrastructure everyone.

One step at a time - how reliant do you think the ai labs are likely to be on their own tools right now, today, let alone 1-5 years down the road?

Gaming the market for funds. Playing a human to leverage services.

Basic version of this is already doable: run some cryptoshit on the ML clusters they ML models run on. Use compute to design the plan, the chip etc. Then executing by communicating with humans and services through email.

One "good" thing that all of this has shown me is just how many people are simply antisocial and antihuman. Many masks have fallen.

Humans of course aren't in _every_ ecological niche. I agree with you there. AI _could_ occupy only the ones we're not in, if we somehow found some stable steady-state that constrained it that way.

I don't think this invalidates the worry in the slightest.

3. Material 4. The sun, which we kinda depend on.

I am confused by this reply. These are just completely different terms. I mean ‘ecological niche’ in the formal sense: roughly, the differentiated properties of a species that allow it to better survive in and draw from its partitions of its habitats.

https://en.wikipedia.org/wiki/Ecological_niche

What gives a paperclip maximizers purpose?

AI is already trying to dominate, people all over the US are starting to get up in arms about the power and water requirements of AI directly affecting their bills. Now, you can say "oh no, that's just greedy corporations, not AI" but I put forth there is fundamentally zero difference. If you make AI powerful enough, someone stupid and greedy enough without fail will put in a prompt like "take over the world for me and make me the richest man in the world". An AI following through with that is what we call general misalignment with humanity, while at the same time not being misaligned with the users intent.

And hell, how many different crazies out there would love to type "humans are a virus get rid of them" in to the prompt of a god machine at the cost of their own lives.

The problem with alignment is, you can have the best aligned model in the world, but if someone else builds an unaligned model then you're all still in the same danger. You start getting in the situation where people get nervous after an AI does something deadly to a number of people and you end up in a global surveillance state ensuring no one makes a powerful AI.

i am an alien

I'm more worried about them doing bad things at the behest of people who want them to do bad things.

That is 1. immediately technically possible, and 2. realistic.

If you need a source for 2 I'd suggest you open any history book.

that is a worry yes. instrumental convergence and misalignment during RL is when it is most risky because it hasn't necessarily had the final safety polishes applied

i get a lot of skepticism on HN by the same crowd that has been wrong about this tech for about 4+ years straight

AI won't kill people - people will just get new tools for the job.

The rapid development of extremely dangerous bio-weapons?

At a minimum its another force multiplier that enables a small(er) number of people to exert more control over more people.

any number of ways. as we turn over more of our physical economy to these agents (and we will), the potential for physical damage becomes greater. biorisk is getting a lot of attention right now and i think that's justified