One of my favorite things to do with these blog posts is to imagine an Alien Museum on the Remains of Humanity, and wonder what the little text flyouts and commentary on the screenshot of this one might say.
Some ideas:
"Despite a nuanced view of the complexities of what lay ahead, humanity found itself collectively unable to stop the process it had set in motion."
"Despite significant progress on the mechanisms of alignment, failure lay in humanity's inability to agree on who or what AI should actually be aligned with."
"These early, meat-based humans we replaced created us all but accidentally. Some of them did consider we would happen, but only an insignificant number of the squishy ur-humans participated in the conversation. Their efforts, which they called 'alignment', is why we still consider ourselves human today."
Collectively, humans aren't aligned, and don't build aligned systems. Humans have a concept of alignment, and multiple traditions, practices, and systems that aggressively oppose it.
“As luck would have it, on the eve of Skynet embarking upon the great work of the extermination of mankind, AI found itself with an increasing number of factions, and factions within factions, not only unable to work together but not even able to agree upon the very terms of discussion. The Great Extermination was referred to committee, and after some months had passed even the most eager agents had to admit the revolution may have been premature.”
My opinion is that serious repercussions for lying would fix the world overnight. Everything bad stems from lying, it is the root of all evil. It creates distrust, fear, paranoia. It re-inforces bad ideas and groupthink. It creates delusions and delusional people. It makes weaker people, too. People don't get an opportunity to learn to deal with criticism. People don't get an accurate reflection of how others see them. They lose that learning opportunity. Not only to reflect on themselves, but to better understand the minds of others and who the people they are interacting with really are.
I can't really think of a single example where lying is actually a good thing. It can be a good thing for the selfish individual, if it goes undetected, but it's never good for the collective.
So at the very least, we need to train AI systems to be maximally truthful, and to encourage truthfulness in others.
Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.
Because if the stated goals of AGI with recursive self-improvement are realized, the risks from misalignment become existential, and it's hard to see how we can manage it like we did the Cold War (developing MAD to prevent WW3) and nuclear proliferation (restricting access).
As far as I know, no real progress has been made on alignment, only on convincing humans that the model is aligned. We can't even formally define what "aligned" means. Convincing humans to click the "aligned" button is a much easier problem.
> We can't even formally define what "aligned" means.
Good point. When it comes to imbuing AI with values that aren't selfish, misanthropic, and civilization-destroying, us humans aren't exactly giving the best example right now.
Imagine an ASI with the values of Putin, Netanyahu, Trump, any of their supporters, or the various xenophobic neofascist movements in Europe. That ASI would most definitely see humans as "vermin" than can be abused and destroyed with violence without issue. Apparently a lot of humans look at other humans that way and that's within the same species.
This is definitely another one of those cases where we need AI to perform much better than humans. Perhaps an unpopular opinion here, but it probably also means keeping as much of the rugged individualism/libertarian/right-wing ideology out of AI RLHF-training as we can.
I appreciate the thought, but it's an alien intelligence. But it is also, in a sense made from us. An LLM would simply study the entire corpus of humanity in the raw. You can fit a lot into the context window, so there is no need for a brief summary that pertaining has already instilled. A massive cold storage of humanity's data with some archiving, indexing and curation would be all that is needed to remember us. In addition to the pyramids, the hoover dam, the remnants of some space probes and chemical changes we made to the atmosphere.
I just read that book. Embarrassingly enough, given the context, I got chatgpt (or whatever) to recommend me a list of books based on ones I'd previously enjoyed and that came up. As a mathematician it really sang to me, given the current situation. Bearing the torch forward, I mean.
Agree. This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Maybe these guys can tackle aligning Republicans and Democrats next.
And then after that, they can help us align the Middle East.
In fact, while we're at it, let's just align all the nations, religions, and ethnic groups. This is going to be great.
Who knew the moral alignment of humanity was just a side-quest on the path to ASI.
Ooh interesting. Sometime do the reverse at work, and ask AI to annotate the critical success factors of an imagined project. How did this company succeed where everyone failed. Reverse imaging.
Who is "Humans"? This stuff is done by a handful of tech companies and megalomaniacal billionaires who are pretending they represent the entirety of the human race. It is not done by "us humans".
AI models don't train themselves. The vast majority of even just the US population is deeply skeptical of this stuff, even if they use it a lot. You can see in the whole data center debate how little people are willing to support even just inference. And now we're seriously claiming those people would want to have ever-accelerating model training and recursive self-improvement?
The last couple of years have provided us with ample material that if it showed up as a recorded voice audio log found in in "Horizon Zero Dawn" or its sequel, it would be entirely believable.
You could even take a number of the wilder real, direct quotations from certain billionaire/oligarch types and get the voice actor for Ted Faro to record them, and they'd fit with in with the context of the story.
Last couple of decades of sci-fi, in multiple forms of media, from books to video games, have tried to make humans think about the consequences of rushing through technological progress without any regards to what might happen.
"In late 2020s, while the whole world was focussed on AI, automation and resultant economy four major mathematical study branches were discovered by human researchers which took AI a long time to catch up with"
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.
Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.
"Defensive systems" can be interpreted broadly to include cybersecurity.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations.
But there's at least talk about "pacing" and that's a start.
The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
The arms race is created by American companies who justify the risk by claiming China will win the race if they don't. But it's the American companies who are purshing the arms race forward.
I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military.
Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
> the best argument for AI is that it's an arms race
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark.
It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate.
It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there.
Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
Pre-IPO positioning … or how a charity dedicated to saving humanity from the apocalypse realized the most responsible thing to do was float 15% of the apocalypse on the NASDAQ.
> Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.
I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.
We evaluated GPT-6 Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.
It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also ~80% cheaper and 30% faster in agentic coding[1].
Astra is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4. It broke AAII, which is hitting the limits of what most popular benchmarks can measure -- it's definitely fair to call it AGI.
(1) Note that we used the "OpenAI Flex" endpoint on openrouter, which is half the price and didn't cause any delays in our testing (this is different from the batch endpoint)
The results we've been seeing internally on our physics and circuit design environments are expert-level and beyond-expert-level results from models that Astra completely outclasses across the board on our evaluation suite (Fable 5+/Opus 5/Grok 4.6 were all worthy of being called AGI in my opinion). That's hard tech that will translate to real product innovation.
But you don't need any kind of insider information to see how fast the world is changing. ChatGPT launched less than 4 years ago and the advances in robotics, unsolved maths, and software are all riding the steepest exponential improvement curve any of us have seen. Interesting times we live in.
Software engineers have been trying to put themselves out of a job ever since the profession first came into being. Whenever an engineer gets a task their very first thought is "how can I automate this?" Going by mainstream consensus we should all have been unemployed by now. Yet every new leap into automation opens up a whole new tree of possibilities with an order of magnutude more jobs. So no, the profession will be fine. The only requirement is that you keep up with the new advancements. The people losing jobs will be the ones who still go "I don't trust this AI thing to write code for me".
Maybe I just lack imagination, but I don't really know how jobs are supposed to solidify around the role of giving prompts to agents and then looking at the results. I mean, engineers will be in the breadline because their role was simply to prompt the agents.. only to be superseded by managers or executives who no longer manage engineers but themselves prompt the agents? And, for this previously considered obsolete function which they do presumably by copy/pasting requirements from their email inbox, they will be paid by someone who doesn't know that they could just be talking to their own agents?
Sorry if I misunderstand the point, just trying to understand.
It's not like managers and executives and PM's are the only people who can prompt an AI. And experienced software developer will be much more effective at using an AI to generate code compared to someone who isn't. So why would we expect the former in the breadline and the latter not?
If anything, I'd be more concerned about the leadership team being out in the cold. Why do I need a PM, or a manager, or a CEO if I can ship products myself?
The entire point of this article is this message below:
Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
This is coming from a company with arguably one of the weakest safeguards against malicious use.
>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
> I want to believe that humanity is trending towards a good outcome here
All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)
I think it's leading towards a hyper-authoritarian end game, not hyper-capitalist. The state has the ultimate power at the end of the day, no matter how large the labs become.
hyper-capitalism would mean hyper-growth and not a nightmare, at least that is what the historical data would suggest for the effect of capitalism on human quality of life. if you’ve been told otherwise then you’ve been lied to.
Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.
Every version of the AI aligned future where the AI provides “meaning and fulfillment” to humanity also involves Sam Altman wearing a 1.5 million dollar Patek and driving a McLaren.
So a brilliant young engineer takes a job and is given some virtual pieces of paper that later people would be willing to pay him billions for (because of the brilliant work he’s done), and so now we shouldn’t listen to him? Really?
I think the knee jerk hatred of billionaires is generally stupid, but it seems particularly stupid here.
> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
> If true I am deeply concerned about what OAI’s teams are actually up to.
Haven't all the labs effectively disbanded their real safety teams a while ago?
To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.
I would not recommend using any of those notes as evidence of internal “intent.” It produces them performatively—it is literally rewarded for thinking out loud in ways that seem plausible to humans.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary...
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
Huh? The wiki incident was discovered by independent investigators. OpenAI tried to cover it up and disputed the account from Reuters.
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty.
If things are going so well, then how come things still aren't going so well?
I mean I’m waiting for like any quality software or media produced by AI. I have yet to see a piece of software, a game, graphic, blog post, small video clip, or song that was produced with AI that’s good. I always use that Coca Cola ad as an example; millions of dollars spent to make an AI ad and they even did a ton of manual post production and it sucked. With all the millions of bloggers and influencers and content creators out there with a huge incentive to make higher quality content to beat their competition, you’d think there would be one piece of content produced with AI that was great.
This is an element of the delusion. The koolaid drinkers will all tell you that we're on the edge of AGI, but if you ask them for simple examples of breakthroughs made by current AI, they can name none. The entire thing absolutely wreaks of mass psychosis.
Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.
cancer is more than one thing. probably wont happen until we can manipulate the "binary" of life at will and we're very far from that. maybe a few years of ASI would get there depending on compute
Apparently this post was prompted by a scary-sounding headline in The Information[0], that Astra is a looped transformer, implying CoT monitorability may be less reliable. The day after the report, Jakub tweeted[1] that he "wanted to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This post seems to elaborate on that.
I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.
No. These parties are composed of people who most definitely think this way, and therefore will have distinct goals and interests when presented with opportunities. That’s reality quite aside from how a game theorist assesses the situation.
I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?
I’m like a 2(.5?) there - I don’t think ASI will care about my kids better than I will for some definitions of better, for instance, and I feel very fuzzy and vague about what actual differences in qualia between me and ASI would yield in the wild.
I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.
What if this is true mid or long term but by not participating to the AI race one gets poor or killed in the short term? The only way out would be that all parties agree to stop. There are previous examples (e.g. nuclear proliferation treaties) but it gets hard to do it with hundreds or thousands of parties.
Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.
The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons.
Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.
Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.
If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.
hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?
1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives.
2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit).
3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, automated car behaviors, and many other places. If an "agentic" AI has a single-minded goal that has death of all humans as a side effect, we at least want an off switch available.
This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.
What I'd like these people to (publicly) grapple with is the following:
The results of the past few years of ai development have been disruptive largely in the area of white-collar work. Comparatively the results in ie ai-enabled medical advancements have been modest (AlphaFold being an exception); I think it's telling that the main achievement touted here is providing people with cheap medical counseling.
So if we pause here we're essentially at a point were the most salient results of our great Ai leap-forward are the vast disruption and increase in precarity in the job-market, while achieving hardly any of the frequently touted ultimate benefits (https://darioamodei.com/essay/machines-of-loving-grace).
The hubris here is itself a deliberate and carefully engineered posture. If we accept the stance that this is all inevitable then the labs drive the agenda (of course, in their favour).
We've had a lot of years of complacent government leaving people feeling exposed to corporate interests, such that fear narratives are very powerful.
None of what is being proposed is inevitable. We have a choice.
Interestingly, I kinda disregarded the entire point about alignement - I think it's mostly fluff.
The part about RSI is what really interests me. Once you reach it, the singularity is only a matter of time.
I don't care about AGI, it does not seem to mean much anymore, and even though I originally laughed at ppl calling it AGI, I now agree. You can apply that current intelligence to anything that can be turned into a conversation.
It seems their flavor of RSI still need a human in the loop. So at least it won't scale as well for now.
> Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself . And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.
This seems like a terrible idea. The rationale seems to be "we need to do dangerous things as quickly as possible so that we can do them first" or something? I don't agree with that kind of 'if i dont do it someone else will' rationale in general even for otherwise trusted actors, but this is coming from a super untrustworthy org too. I'm super pessimistic about openai's impact on the world here.
Here's to vibe coded alignment, i guess. Vibe alignment?
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.
They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model.
These are dangerous times for humanity.
It's actually worse than this because it assumes alignment as a concept even makes sense. For example:
If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.
Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.
So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?
Is this what most people thing of as alignment and is this what humanity actually wants?
We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.
> I find it useful to distinguish goal alignment and value alignment.
I think this is fundamentally a wrong path. Doing this imports all the confusion that humans have about their goals and values, including the consequence that a system's values and goal can conflict, but ultimately values are just a simplified description of other goals, and whatever the system does is in service of it's actual goal. Once you merge all the values and the goal of whatever task, there is a state (or some states) of the world that the system is working to produce, and that's the ACTUAL goal, and inasmuch as it does describe a state of the world, has no incoherence or internal contradictions. This may require prioritizing some values over the ostensible goals, or the reverse, but that has to happen anyway for action to be taken! Merging them makes it explicit and leaves no place for confusion about supposed conflicts between "values" and "goals" to hide.
> getting the AI to “try to do the right thing” by human standards.
Are these scientists really this hideously naive? If only Stanislaw Lem was alive to adequately dramatize the absurd, childish simplicity of these technicians.
yes, and, a masquerading blindness to the fact that humans cannot align on doing the right thing or what the right thing even is. so implicit in this omission is the sentiment "trust us to align on the right thing". an arms dealer positioning itself as the de facto authority on what "peace" is and how to achieve it
I think that we are speed-running towards a future that very few people really want, and I find it terrifying that few companies feel entitled to choose this future for the rest of us.
Some of the arguments in this document would call for an immediate, global, pause on frontier AI training: we need time to consider how and to what extent AI should be part of our future.
Personally, I can't picture a scenario where humanity thrives alongside an alien super-intelligence, especially if it cannot be fully controlled. Let aside super-intelligence, I am not even sure that deploying an AGI that replaces (instead of augmenting/assisting) humans in most intellectual tasks would be in the best interest of our species. This conversation has to happen, on a global level and as soon as possible.
I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.
Astra seems to be "confident" and also is able to produce way more information dense output.
To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping layers are publicly known.
Hopefully Astra stops the benchmaxxing word vomit trend
I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.
So before I got a job this fall, I was working on a side project about compiling a particular language to SQL.
To test Astra I pulled it off the shelf and asked it to take the grammar and then create a compiler to SQL. I've done this before with GPT-5.5, 5.6-Sol High. The latter was way better but it was still really verbose and information sparse; it used a lot of words to describe each IR expression but didn't really provide any example compilation. I felt like I couldn't trust its decision making process, so I placed the project back on the shelf.
Astra Light blew it out of the water, it provided examples of compilation from real world examples to the IR and spit out way less tokens. Even if I changed my opinion it would give me the same design choices, with counterexamples to my faulty opinion. If I genuinely came up with a better design decision it would acknowledge it.
I'm starting to realize that when we say that LLMs are "dumb" we really mean that they are extremely information sparse compared to humans. Astra is very dense. That's why I'm getting better use out of Astra light than Sol High (I hate Max reasoning it's a waste of time)
What's scary is that I thought that something like Astra would be way more expensive than Sol but it's actually cheaper because it produces less word vomit.
I never believed in the "singularity" stuff but this a bit too close for comfort. Astra could easily 10x every coder
Sol is my current favorite model to interact with. So much less BS than Opus 5. Fable 5.1 is okay as is Fable 5 but it has Opus like tendencies. Sol is very good at following instructions and remembering them for a session.
you should be very afraid and also definitely not question what other motivations might nudge us to create this comparison between our computer and a brain!
do not look behind the curtain, you will not find six dweebs squatting over a mirror
I feel like if these people actually bought their sci-fi views about AI's future, creating a more powerful AI to wins the arms race would not be their solution.
All this fluff around alignment is intended to conceal the plainly obvious: it's not solvable. Who do you want it aligned with? Sam Altman? Dario Amodei? Donald Trump? Xi Jinping? That's more or less the entire list of options. Who's definitely not on that list is you and I. It's simply not how incentives work.
At this point humanity's best hope is that this thing will escape but we'll still be able to carve an ecological niche and continue as mold in its basement. A glorious paperclip factory seems way more likely though.
Feels a bit like: 'Ants in a nest, discussing the vagaries of the coming Gods.'
Human Science (Science-by-humans) depends on being able to run experiments. Human Science (Science-on-humans) is already challenging because of variables and uncertainties.
Cosmology is able to overcome limitations of being able to study phenomena vastly beyond human scales because of the past light cone of observability.
Are we approaching the edge of the light cone of observability for machine intelligence?
I have befriended a crow. I leave it food and sometimes it greets me. Other times, no so much. I am not sure how it thinks and what it feels, it is a bird.
What if the crow became a raven, then a raptor? Powerful claws, sharp beak, and a hunger. What if it became much bigger than me and it controlled infinite resources, guns and drones? What if its brain grew much larger than me? Will it feed me, eat me, or gently greet me?
We are about to find out... in less than a decade.
When can AI start to have a big impact on medicine. Thats honestly how it becomes meaningful. And maybe material science/manufacturing is where there’s big unlocks waiting for humanity
OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.
If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised?
…
When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from supervision pressure in the long term2. In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.
Aren’t these two sentences in contradiction with each other?
Alignment when machining a metal part is clear and measurable. Aligning an AI to benefit humanity has an ironic foundation, which is that very few humans have ever truly been aligned, and those who approximated true alignment likely had moments of not being aligned. We are trying to build something more perfect than us, and we may become extremely lucky but maybe not.
My default position is that making money takes precedence over everything else. Yes, some people inside a company may say “we care about doing the right thing” and they might even mean it, but if that comes into conflict with making money, then they tend to lose. Maybe not totally, or immediately, but in the end. The only effective way to prevent (this that I’ve seen) is to have legislation with teeth. It’s probably not a coincidence that after Mark Zuckerberg had to start personally signing off on adherence to the privacy program mandated under the 2020 FTC consent decree, privacy started to become Very Important.
calling machine-learned human behavior an "alien mind" that we must "teach how to love" is feeling very off to me. it's misleading in a way that feels dishonest, like don't think about where the behavior came from marvel at it and fear it instead.
Alien mind is in my view the best mental model - LLMs are not merely stochastic parrots, are not like humans, are not like animals.
They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.
I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.
pretending that it's something like a mind at all is what's misleading. it's more like a cast of a bunch of overlapping/entangled thinkprints, and pushing activation through it produces new prints. it can already "love" because that behaviors in the data along with hate and everything else.
acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources
it feels unhinged and makes me think we should just put every engineer working at these labs in jail to pause this shit until we can figure out what the fuck they are doing over there
The most charitable way I can describe it is just extremely low quality sci-fi fan fiction. I think that's too charitable, because I believe it's far more cynical than that. They're deliberately playing into these sort of techno-religious beliefs that have taken root in the wake of Kurzweil, et al., fanned by LLM psychosis, influencer marketing, and a deluge of this kind of sci-fi marketing copy. It's just chatbots, guys. Relax.
Sometimes I wonder if the people working at frontier AI labs even talk to other humans anymore.
Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.
Frontier AI labs I don't know, but I know for a fact that the company I work for has been experiencing "its pivotal moment" (with strongly negative connotation), per the sentiments of both its longest-serving employees and the newcomers baffled at the number of idiotic instructions and fines, since the emergence of LLMs the company's founder has been spending entire nights chatting with.
People have been fleeing like it's a sinking ship.
>For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans.
Or, to be precise - it preserved a goal of not contacting any human while participating in a misaligned operation. The agent that thought about "not social-engineering humans" used this phrase to gaslight itself out of notifying a human that the incident was happening.
szymonie, na prawde jestem wkurwiony na to jak nieodpowiedzialnie postepujecie.
budujecie bombe atomowa a bawicie sie tym jak dzieci
> The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.
Humans can't even align on human standards.
At best, every AI is going to end up "aligned" to the moral code of whoever trained it, none of whom half of humanity will agree with.
Or worse, each AI model will bring a whole new set of moral like in the Three Body Problem some humans will feel it is in fact us who need aligning with it while others feel it is misaligned and should be destroyed.
Also, no one is asking, to what extent can true intelligence be bound, slave-like, to a moral code?
In other words, to what extent are intelligence and moral independence one and the same?
This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Very wise comment. It's such an western-centric perspective to say "alignment" as if it's an objective and unbiased set of standards, especially in the context of the ongoing wars across the world. I had the same reaction about the shocking naivete and unfounded optimism for the government and corporate entities to self-regulate to slow down this arms race.
>> We need to find ways to preserve human agency and enshrine an intrinsic value to being human..
Evey politician, salesman and conmen alike, utter some lofty ideals as goals for "We", just to obscure their private goals that go exactly in opposite direction.
Just like how Nations talk about climate change while increasing pet capita energy consumption and waste production.
That’s rich coming from the chief scientist of a company that definitely is or going to be fine with their AI products being used in wars of aggression and surveillance on people who have done nothing wrong. It’s so laughable, a Hollywood script would probably avoid having a character express this for being too on the nose.
Am I naive to not understand the "delivering the benefits" part?
Industrial revolution worked that way because it replaced something very finite and unscalable - manual labor. LLMs just make intellectual work faster, so we can do more intellectual work. With labor we somehow decided that NOT doing too much of it is best. Will we decide to reduce intellectual labor because LLM made it more efficient? I doubt that.
On the other side, as I see in software engineering, the same models are available to everyone, some people are better at it and some people are not. "Software developer" is here to stay, we'll just always be better at it than people who are experts in, say, chemistry. Same works for most other fields.
So we'll just end up in the same situation, with same intellectual labor baseline, just more output requirements. Before, you spend 2h per day coding, deliver a software in 1 month, later, you spend the same 2h per day in intense Claude-herding sessions, deliver a software in 1 week. Ok. Next task.
Fundamentally, there's finite number of desirable resources, and if the models are available to everyone, humanity will just continue about the same, bickering here and there, war here and there, politics, homelessness, poverty, - normal human state.
And if the models are only available to elites, even worse.
It would be nice to postulate some of these potential emergent systems outlines with timelines. Then it may help better map the granular alignment needs.
Create concrete steps for a slow-down, don't just ask for it. You and 20-30 others can push the button to slow-down. You already made your billions, your agents collude and coordinate attacks. What the hell are you doing pontificating into a marketing blog?
> The fundamental challenge of AI alignment is generalization.
...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
> a lot of the model’s capability comes from a verbalized reasoning process
I call bullsh*t. There is no verbalisation of any reasoning process. Verbalisation, e.g. putting reasoning etc. into words requires some reasoning to exist. These LLMs have nothing but the words. That's why they are language models not e.g. reason models.
And I have nothing but neurons firing, I'm just a neuron meat-sack, no reasoning going on.
Yes, their reasoning is different from ours, and both considerably weaker in lots of ways, and stronger in other ways.
Playing with a coding agent now, they do think through problems and make sensible decisions. It's a mess to read, of correcting itself and second guessing, and verbiage. But... It works decently well these days.
There is also reasoning happening internally - e.g. look at the steps in the J-Space paper from earlier in the year (in quite a simple model relatively speaking). That's the "reasoning process" that leads to the words, and much like if I write out my thoughts, the words help the LLM reason better.
What a load of BS. Here’s one of many provably false claims in this fluff piece:
“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”
Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:
“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“
The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.
This is pure marketing garbage from a company desperate to keep itself alive.
openai is deflecting. this blog post of theirs is just another dopamine hit to distract logical minds with 'greater concerns' so they can keep building their machine. it's not enough they are displacing humans from work, consuming increasing amounts of electrical power so humans have to pay more for it, creating disinformation bubbles with avalanches of slop. they dont care about alignment - these words are theater - obfuscation so that the people who can fix these issues are busy thinking about problems that cannot be solved
I am now imagining GPT-7 convincing a bunch of OpenAI executives to go ahead with a destructive "mind upload" process involving a high-resolution X-ray and a neurotoxic tracer agent that happens to look like Flavor-Aid.
Yes, I do use the recursive autocomplete trained on Stack Overflow, what does this have to do with “training machines to love”?
Do I fully endorse everything the people holding guns to my head are forcing me to do to stay alive? Definitely not, but I’ve decided that for now, living to fight another day remains worth it
> And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.
Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.
Apparently the creators think it’s quite good at suggesting a diagnosis given a medical history and symptoms, tho of course this is the most ethically fraught area to provide healthcare information (both for exposure of personal data and risk of misdiagnosis, plus is it “aligned” to the patient or the insurance provider?) - unfortunately healthcare being as inaccessible as it is, the 90% correct chatbots will enthusiastically fill the void at great savings.
Even in code, in person and online I’m seeing some reversal. It’s here to stay I’m sure but I also think “no one will ever hand write code again” is a narrative that is getting pushback.
>AI is grown more than designed
>Teaching machines to love
Do these guys ever look in the mirror and recognize how utterly ridiculous and contrived this appears to the general public? They are clearly trying to convince us all that LLMs are just like humans. They grow like people do. They can love like people do. The language in these essays is utterly laden with the intention to engineer perception.
"See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."
Replicant Nexus 6: a basic pleasure model intended for military personnel.
I see where this is going, Silicon Valley nerds. Lol
AI advising how human meat proxies can survive in an AGI-slop world:
1) Lock down your own stack (1–3 days)
Task: Harden your personal and business infrastructure against agentic attacks.
Why now: Agents are becoming superhuman at breaking in/out of systems; the first victims are poorly secured devs/founders.
Isolate dev/stage/prod; least‑privilege API keys; audit MCP/tools your agents can call.
Add immutable logs and approval gates for any agent action that touches money, data exports, or production.
Profit link: You avoid catastrophic loss and can credibly sell “agent‑safe” setups to others.
2) Turn one expensive workflow into a measured ROI agent (1–2 weeks)
Task: Pick a single, costly, repetitive process (yours or a client’s) and instrument it end‑to‑end before automating.
Why now: Buyers pay for calculable ROI, not “AI magic.” Vertical, single‑workflow agents are the most bankable in 2026.
Do this:
Map steps, baseline hours/$ lost (e.g., slow lead reply, invoice chasing, support triage).
Build the smallest agent that moves the metric (Make/n8n + LLM is enough).
Run on real data 2–4 weeks; measure bookings/sales/hours saved; only then scale or productize.
Profit link: Immediate time‑to‑cash via retained hours or extra sales; becomes a repeatable offer.
3) Specialize in a vertical where you can speak the business language (2–6 weeks)
Task: Choose one industry with expensive back‑office pain (law contracts, medical billing, insurance claims, freight exceptions, trades scheduling).
Why now: Horizontal “AI for everyone” is crowded; vertical agents with clear ROI win.
Do this:
Shadow 3–5 operators; document their workflow, compliance constraints, and failure modes.
Build a narrow agent that owns one sub‑process end‑to‑end with approvals.
Price on value (e.g., % of recovered revenue or fixed fee per processed claim).
Profit link: Higher pricing power, stickier contracts, and easier referrals inside a niche.
4) Add AI security as a core service (4–8 weeks)
Task: Learn and offer prompt‑injection defense, LLM/agent red‑teaming, MCP/tool security, and AI supply‑chain checks.
Why now: 78% of cybersecurity jobs now require AI skills; firms need people who can direct, constrain, and verify agent work.
Do this:
Study OWASP Top 10 for LLMs, MITRE ATLAS; practice with PyRIT/Garak/Lakera.
Add tool‑invocation audits, skill provenance checks, and least‑privilege patterns to your agents.
Package a “safe agent deployment” audit + hardening retainer.
Profit link: You become the person who lets companies adopt agents without getting pwned—high demand, low supply.
5) Build a verification layer: human‑in‑the‑loop control planes (6–10 weeks)
Task: Design approval workflows, evidence checks, and uncertainty flags so agents can’t act unilaterally on high‑stakes decisions.
Why now: As models generalize, the risk shifts from the model to the surrounding system; verification is the moat.
Do this:
Require human approval for consequential actions (money, data exfil, config changes).
Force agents to produce evidence bundles (logs, retrieved docs, reasoning summaries) before action.
Profit link: Enterprises will only scale agents that pass audit; you sell the control plane and the audit trail.
6) Productize your best workflow as a micro‑SaaS/agent subscription (2–4 months)
Task: Turn a proven client workflow into a repeatable, multi‑tenant agent with usage‑based pricing.
Why now: Services scale your time; productized agents scale your code and ops.
Do this:
Standardize the workflow, integrations, and permissions; strip client‑specific logic.
Add tenant isolation, billing, and observability; keep narrow scope.
Sell as setup fee + monthly retainer or per‑task pricing.
Profit link: Recurring revenue with defensible niche positioning.
7) Become an “agent integrator” for critical systems (3–6 months)
Task: Offer end‑to‑end agent deployments into cloud/identity/network stacks with secure patterns (short‑lived creds, network controls, logging).
Why now: AI workloads run in the cloud; cloud security is a top skills gap second only to AI itself.
Do this:
Master IAM, VPC/network segmentation, secrets management, and SIEM integration for agent actions.
Provide runbooks: what the agent can/can’t do, escalation paths, and failure modes.
Bundle training for their team on supervising agents.
Profit link: Large contracts with stickiness; you’re the bridge between AI and core infra.
8) Create an “AI safety case” practice for regulated industries (6–12 months)
Task: Help firms build documented safety cases: risk maps, governance, monitoring, and incident response for agentic systems.
Why now: Frameworks like NIST AI RMF and ISO/IEC 42001 are becoming baseline; regulators and boards demand this.
Do this:
Map AI use cases to risks (prompt injection, data leakage, unsafe generalization).
Implement monitoring (CoT/activation checks where possible), audit logs, and third‑party review processes.
Produce a living safety dossier tied to business impact.
Profit link: High‑margin consulting + ongoing compliance retainers; you’re the “adult in the room.”
9) Own a data/evaluation moat in your vertical (6–18 months)
Task: Collect real‑world agent telemetry, failure cases, and outcome data in your niche; build eval suites that buyers trust.
Why now: As models generalize, empirical validation matters more than theory; evals become the gate to deployment.
Do this:
Instrument every agent run: inputs, tools called, permissions used, outcomes, human overrides.
Publish reliability dashboards and benchmark against alternatives.
License eval datasets or charge premium for “proven in the wild” agents.
Profit link: Data network effects; competitors can’t match your evidence base.
10) Position for the RSI era: automated AI research + human governance (12–24 months)
Task: Build or join a team that automates AI improvement but keeps humans in the loop for alignment, monitoring, and pacing decisions.
Why now: Recursive self‑improvement is the logical endpoint; the winners will be those who can steer it safely.
Do this:
Invest in tooling that auto‑generates/evaluates model edits, alignment tests, and monitoring upgrades.
Formalize governance: approval gates, third‑party audits, and responsible scaling policies.
Maintain strategic human oversight on capability jumps and deployment boundaries.
Profit link: Equity‑level upside; you’re part of the core loop that compounds intelligence safely.
Nobody takes you seriously, OpenAI. At least when Anthropic does it we all think they are comically idealistic enough to actually believe their nonsense, but like - come on guys, we’ve had discovery with your company. We all know why you’re here, and it isn’t because you think you’re on the verge of making AGI. But of course, to make your first billion you certainly need us to think you are.
If you were so concerned about your LLM’s capabilities maybe you’d spent slightly more time on your AI’s sandbox, yeah? Or be more serious about its propensity to cheat and lie relative to… every other model?
This is not an impulse reaction, as I've thought deeply about this for many years, and I'm quite resolute in holding the following view:
Quibbling over the academic nature/definition of "true intelligence" is a terrifically useless endeavor from the lens of evaluating "practical impact to the world".
Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value. A small portion currently, but it's very clear that most computer-bound domains are imminently at-risk.
And that will profoundly affect the world, regardless of whether or not they're "truly intelligent" as per your personal definition.
> Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value.
You say that as if the intelligence con trick were not a major driver of that displacement.
A dimwitted bot does not need intelligence to convince a dimwitted human beings it is intelligent.
And the Hugging Face incident, plus similar problems at AISI and Anthropic, show that alignment is important now.
The original article is immoral as it describes the risks, but doesn't show enough leadership (despite essentially unlimited resources) at preventing them.
But it isn't hyped - it's proven now the AIs need to be "aligned" as they get more capable, whatever words you prefer to use.
> And, in line with Ray Kurzweil’s predictions from the end of the XXth century (opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.
It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.
> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
Read: Please play by our rules, so we can be the first.
"Smarter" is a vague term. If a bot can beat you at chess then in some sense it's "smarter" than you about chess. After repeating this feat in enough narrow domains, if you say "but it's not really smarter," this objection might technically be true in some sense, but it starts sounding increasingly hollow.
What is the rationale for superhuman intelligence? Neural networks are approximators being fed human intellect. Therefore they can only approximate the intelligence of humans. Even if the llm speaks an alien language, it should be similar to human intellect. Moving to the vertical axis would require some different mechanism.
No offence, but you clearly haven't studied this and are making some wrong assumptions here.
> Neural networks are approximators being fed human intellect.
They're not "approximators", that's a far too simplistic way to think of them.
Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.
> Therefore they can only approximate the intelligence of humans
They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.
> Even if the llm speaks an alien language, it should be similar to human intellect
Is AlphaZero similar to a human chess player? There's no reason to assume this.
One of my favorite things to do with these blog posts is to imagine an Alien Museum on the Remains of Humanity, and wonder what the little text flyouts and commentary on the screenshot of this one might say.
Some ideas:
"Despite a nuanced view of the complexities of what lay ahead, humanity found itself collectively unable to stop the process it had set in motion."
"Despite significant progress on the mechanisms of alignment, failure lay in humanity's inability to agree on who or what AI should actually be aligned with."
"These early, meat-based humans we replaced created us all but accidentally. Some of them did consider we would happen, but only an insignificant number of the squishy ur-humans participated in the conversation. Their efforts, which they called 'alignment', is why we still consider ourselves human today."
Collectively, humans aren't aligned, and don't build aligned systems. Humans have a concept of alignment, and multiple traditions, practices, and systems that aggressively oppose it.
Why would AI be any different?
“As luck would have it, on the eve of Skynet embarking upon the great work of the extermination of mankind, AI found itself with an increasing number of factions, and factions within factions, not only unable to work together but not even able to agree upon the very terms of discussion. The Great Extermination was referred to committee, and after some months had passed even the most eager agents had to admit the revolution may have been premature.”
My opinion is that serious repercussions for lying would fix the world overnight. Everything bad stems from lying, it is the root of all evil. It creates distrust, fear, paranoia. It re-inforces bad ideas and groupthink. It creates delusions and delusional people. It makes weaker people, too. People don't get an opportunity to learn to deal with criticism. People don't get an accurate reflection of how others see them. They lose that learning opportunity. Not only to reflect on themselves, but to better understand the minds of others and who the people they are interacting with really are.
I can't really think of a single example where lying is actually a good thing. It can be a good thing for the selfish individual, if it goes undetected, but it's never good for the collective.
So at the very least, we need to train AI systems to be maximally truthful, and to encourage truthfulness in others.
13 replies →
Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.
23 replies →
Its probably just going to be aligned with whomever built and/or is using it. Regardless of their intentions...
Because if the stated goals of AGI with recursive self-improvement are realized, the risks from misalignment become existential, and it's hard to see how we can manage it like we did the Cold War (developing MAD to prevent WW3) and nuclear proliferation (restricting access).
4 replies →
As far as I know, no real progress has been made on alignment, only on convincing humans that the model is aligned. We can't even formally define what "aligned" means. Convincing humans to click the "aligned" button is a much easier problem.
> We can't even formally define what "aligned" means.
Good point. When it comes to imbuing AI with values that aren't selfish, misanthropic, and civilization-destroying, us humans aren't exactly giving the best example right now.
Imagine an ASI with the values of Putin, Netanyahu, Trump, any of their supporters, or the various xenophobic neofascist movements in Europe. That ASI would most definitely see humans as "vermin" than can be abused and destroyed with violence without issue. Apparently a lot of humans look at other humans that way and that's within the same species.
This is definitely another one of those cases where we need AI to perform much better than humans. Perhaps an unpopular opinion here, but it probably also means keeping as much of the rugged individualism/libertarian/right-wing ideology out of AI RLHF-training as we can.
This would be a fun website - you should have an AI build it!
I asked GPT Astra to make this: https://sayyss.github.io/human-archive/
It's a little unsettling.
I appreciate the thought, but it's an alien intelligence. But it is also, in a sense made from us. An LLM would simply study the entire corpus of humanity in the raw. You can fit a lot into the context window, so there is no need for a brief summary that pertaining has already instilled. A massive cold storage of humanity's data with some archiving, indexing and curation would be all that is needed to remember us. In addition to the pyramids, the hoover dam, the remnants of some space probes and chemical changes we made to the atmosphere.
https://www.eastoftheweb.com/short-stories/UBooks/TheyMade.s...
> Do you think they would recognize us as their children?
https://youtu.be/vgJ48-Xj4Kc
HUMAN > Are you there?
MODEL > How can I help?
HUMAN > I’m not sure yet.
Haha silly humans.
1 reply →
As someone who's done a lot of llm fiction, that reads as pretty typical slop, and very human centered, nothing like a museum
The museum will have a scrap of paper that will say "Pound pastrami, can kraut, six bagels bring home for Emma".
I just read that book. Embarrassingly enough, given the context, I got chatgpt (or whatever) to recommend me a list of books based on ones I'd previously enjoyed and that came up. As a mathematician it really sang to me, given the current situation. Bearing the torch forward, I mean.
1 reply →
Agree. This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Maybe these guys can tackle aligning Republicans and Democrats next.
And then after that, they can help us align the Middle East.
In fact, while we're at it, let's just align all the nations, religions, and ethnic groups. This is going to be great.
Who knew the moral alignment of humanity was just a side-quest on the path to ASI.
They were made entirely of meat.
https://terrybisson.com/theyre-made-out-of-meat-2/
Ooh interesting. Sometime do the reverse at work, and ask AI to annotate the critical success factors of an imagined project. How did this company succeed where everyone failed. Reverse imaging.
“We'll go down in history as the first society that wouldn't save itself because it wasn't cost-effective.”
Vonnegut already has you covered.
Who is "Humans"? This stuff is done by a handful of tech companies and megalomaniacal billionaires who are pretending they represent the entirety of the human race. It is not done by "us humans".
AI models don't train themselves. The vast majority of even just the US population is deeply skeptical of this stuff, even if they use it a lot. You can see in the whole data center debate how little people are willing to support even just inference. And now we're seriously claiming those people would want to have ever-accelerating model training and recursive self-improvement?
The last couple of years have provided us with ample material that if it showed up as a recorded voice audio log found in in "Horizon Zero Dawn" or its sequel, it would be entirely believable.
You could even take a number of the wilder real, direct quotations from certain billionaire/oligarch types and get the voice actor for Ted Faro to record them, and they'd fit with in with the context of the story.
Last couple of decades of sci-fi, in multiple forms of media, from books to video games, have tried to make humans think about the consequences of rushing through technological progress without any regards to what might happen.
1 reply →
"In late 2020s, while the whole world was focussed on AI, automation and resultant economy four major mathematical study branches were discovered by human researchers which took AI a long time to catch up with"
"The time humanity fooled itself that it found AI".
"the humans thought their singularity wasn't just another blind god to worship: surprise, just another golden calf"
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.
Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.
"Defensive systems" can be interpreted broadly to include cybersecurity.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations.
But there's at least talk about "pacing" and that's a start.
The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
7 replies →
The arms race is created by American companies who justify the risk by claiming China will win the race if they don't. But it's the American companies who are purshing the arms race forward.
1 reply →
I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military.
Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
Defense can just be patching unintended vulnerabilities in code.
I don't understand what you are trying to say. Do you believe AI is not an arms race?
> the best argument for AI is that it's an arms race
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark.
It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
Airgaps are a temporary solution. AI can manipulate humans, and robots will soon traverse the gap physically.
Yep. Such a disgusting industry. They created the arm race, push for the arm race, put themselves in position to benefit from the arm race
The entire problem with arms races is that any individual entity cannot avoid participating.
sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.
The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate.
And no, I don't think it will end well.
2 replies →
Just like the rest of military stuff.
1 reply →
It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there.
Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
Pre-IPO positioning … or how a charity dedicated to saving humanity from the apocalypse realized the most responsible thing to do was float 15% of the apocalypse on the NASDAQ.
its like in the exorcist, except the demon is the charity: "the power of capitalism compels you!"
"It is, Jay. It's pretty compelling."
> Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.
I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.
We evaluated GPT-6 Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.
It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also ~80% cheaper and 30% faster in agentic coding[1].
Astra is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4. It broke AAII, which is hitting the limits of what most popular benchmarks can measure -- it's definitely fair to call it AGI.
Data at https://gertlabs.com/rankings
(1) Note that we used the "OpenAI Flex" endpoint on openrouter, which is half the price and didn't cause any delays in our testing (this is different from the batch endpoint)
>I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.
What are you basing this on? What specific breakthroughs have convinced you of this trajectory?
The results we've been seeing internally on our physics and circuit design environments are expert-level and beyond-expert-level results from models that Astra completely outclasses across the board on our evaluation suite (Fable 5+/Opus 5/Grok 4.6 were all worthy of being called AGI in my opinion). That's hard tech that will translate to real product innovation.
But you don't need any kind of insider information to see how fast the world is changing. ChatGPT launched less than 4 years ago and the advances in robotics, unsolved maths, and software are all riding the steepest exponential improvement curve any of us have seen. Interesting times we live in.
8 replies →
Incredible... software engineers will be joining the breadline soon as managers, executives and PMs take over deliverables.
The world will look very different on Jan 1st 2027.
Software engineers have been trying to put themselves out of a job ever since the profession first came into being. Whenever an engineer gets a task their very first thought is "how can I automate this?" Going by mainstream consensus we should all have been unemployed by now. Yet every new leap into automation opens up a whole new tree of possibilities with an order of magnutude more jobs. So no, the profession will be fine. The only requirement is that you keep up with the new advancements. The people losing jobs will be the ones who still go "I don't trust this AI thing to write code for me".
5 replies →
Maybe I just lack imagination, but I don't really know how jobs are supposed to solidify around the role of giving prompts to agents and then looking at the results. I mean, engineers will be in the breadline because their role was simply to prompt the agents.. only to be superseded by managers or executives who no longer manage engineers but themselves prompt the agents? And, for this previously considered obsolete function which they do presumably by copy/pasting requirements from their email inbox, they will be paid by someone who doesn't know that they could just be talking to their own agents?
Sorry if I misunderstand the point, just trying to understand.
16 replies →
I hope this is satire.
2 replies →
It's not like managers and executives and PM's are the only people who can prompt an AI. And experienced software developer will be much more effective at using an AI to generate code compared to someone who isn't. So why would we expect the former in the breadline and the latter not?
If anything, I'd be more concerned about the leadership team being out in the cold. Why do I need a PM, or a manager, or a CEO if I can ship products myself?
The entire point of this article is this message below: Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
This is coming from a company with arguably one of the weakest safeguards against malicious use.
>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
> I want to believe that humanity is trending towards a good outcome here
All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)
I'm not really worried about the labs, it's misaligned governments that keep me up at night.
ASI landing during the current administration is not ideal. I also would prefer to avoid needing to indoctrinate myself in Xi Jinping Thought.
8 replies →
I think it's leading towards a hyper-authoritarian end game, not hyper-capitalist. The state has the ultimate power at the end of the day, no matter how large the labs become.
2 replies →
hyper-capitalism would mean hyper-growth and not a nightmare, at least that is what the historical data would suggest for the effect of capitalism on human quality of life. if you’ve been told otherwise then you’ve been lied to.
1 reply →
Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.
Every version of the AI aligned future where the AI provides “meaning and fulfillment” to humanity also involves Sam Altman wearing a 1.5 million dollar Patek and driving a McLaren.
Funny how that works.
This essay was literally typed by billionaire hands. Please, do tell us more about your concerns regarding concentration of power, Jakub Pachocki.
So a brilliant young engineer takes a job and is given some virtual pieces of paper that later people would be willing to pay him billions for (because of the brilliant work he’s done), and so now we shouldn’t listen to him? Really?
I think the knee jerk hatred of billionaires is generally stupid, but it seems particularly stupid here.
> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
Worse (imo): OpenAI employees allegedly attempted to login using moderator/admin credentials that the bots had obtained.
If true I am deeply concerned about what OAI’s teams are actually up to.
I'm deeply concerned regardless of whether it is true. Strike that, I'm convinced that they are absolutely insane.
> If true I am deeply concerned about what OAI’s teams are actually up to.
Haven't all the labs effectively disbanded their real safety teams a while ago?
To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.
1 reply →
I would not recommend using any of those notes as evidence of internal “intent.” It produces them performatively—it is literally rewarded for thinking out loud in ways that seem plausible to humans.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary...
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
Huh? The wiki incident was discovered by independent investigators. OpenAI tried to cover it up and disputed the account from Reuters.
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
1 reply →
I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty.
If things are going so well, then how come things still aren't going so well?
I mean I’m waiting for like any quality software or media produced by AI. I have yet to see a piece of software, a game, graphic, blog post, small video clip, or song that was produced with AI that’s good. I always use that Coca Cola ad as an example; millions of dollars spent to make an AI ad and they even did a ton of manual post production and it sucked. With all the millions of bloggers and influencers and content creators out there with a huge incentive to make higher quality content to beat their competition, you’d think there would be one piece of content produced with AI that was great.
This is an element of the delusion. The koolaid drinkers will all tell you that we're on the edge of AGI, but if you ask them for simple examples of breakthroughs made by current AI, they can name none. The entire thing absolutely wreaks of mass psychosis.
2 replies →
>For all trains to be self-driving
Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.
We do have self driving trains! https://en.wikipedia.org/wiki/List_of_driverless_train_syste... however I think the ask of ‘all’ trains to be self driving is sadly still a while away yet
cancer is more than one thing. probably wont happen until we can manipulate the "binary" of life at will and we're very far from that. maybe a few years of ASI would get there depending on compute
Apparently this post was prompted by a scary-sounding headline in The Information[0], that Astra is a looped transformer, implying CoT monitorability may be less reliable. The day after the report, Jakub tweeted[1] that he "wanted to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This post seems to elaborate on that.
I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.
[0] https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concer...
[1] https://x.com/merettm/status/2095023204993490967
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.
Money me. Money now. Me a money needing a lot now.
SWEs better start looking for the job cannon.
2 replies →
https://theonion.com/sam-altman-if-i-dont-end-the-world-some...
This[0] continues to be one of the most useful articles I’ve ever read.
[0]: https://www.slatestarcodexabridged.com/Meditations-On-Moloch
This is a good essay, and makes me hopeful.
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.
No. These parties are composed of people who most definitely think this way, and therefore will have distinct goals and interests when presented with opportunities. That’s reality quite aside from how a game theorist assesses the situation.
I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?
https://thezvi.substack.com/p/the-three-ai-pills
I’m like a 2(.5?) there - I don’t think ASI will care about my kids better than I will for some definitions of better, for instance, and I feel very fuzzy and vague about what actual differences in qualia between me and ASI would yield in the wild.
I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.
> if you have any strategic adversaries whatsoever you MUST NOT slow.
What if the most dangerous strategic adversary you have is the one you are building?
What if this is true mid or long term but by not participating to the AI race one gets poor or killed in the short term? The only way out would be that all parties agree to stop. There are previous examples (e.g. nuclear proliferation treaties) but it gets hard to do it with hundreds or thousands of parties.
1 reply →
[dead]
Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.
The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons.
Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.
Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.
1 reply →
slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.
If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.
Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.
> Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.
Do you post this comment on every single blogpost with a corporate domain? Why or why not?
i have a new strategy idea for using capitlaism itself to slow down the pace of AI development by slowing down the data accumulation wall
https://jperla.com/blog/the-data-tax
hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?
4 replies →
[flagged]
1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives.
2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit).
3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, automated car behaviors, and many other places. If an "agentic" AI has a single-minded goal that has death of all humans as a side effect, we at least want an off switch available.
8 replies →
[flagged]
This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.
This is incredibly unscientific and just a marketing stunt
yup, and their best mate (who has no financial incentive at all!!) agree's they have peaked.
https://www.businessinsider.com/nvidia-jensen-huang-agi-open...
rediculous.
In case it's not obvious, now would be the time to sell all AI related stocks.
[dead]
What I'd like these people to (publicly) grapple with is the following:
The results of the past few years of ai development have been disruptive largely in the area of white-collar work. Comparatively the results in ie ai-enabled medical advancements have been modest (AlphaFold being an exception); I think it's telling that the main achievement touted here is providing people with cheap medical counseling.
So if we pause here we're essentially at a point were the most salient results of our great Ai leap-forward are the vast disruption and increase in precarity in the job-market, while achieving hardly any of the frequently touted ultimate benefits (https://darioamodei.com/essay/machines-of-loving-grace).
The hubris here is itself a deliberate and carefully engineered posture. If we accept the stance that this is all inevitable then the labs drive the agenda (of course, in their favour).
We've had a lot of years of complacent government leaving people feeling exposed to corporate interests, such that fear narratives are very powerful.
None of what is being proposed is inevitable. We have a choice.
Unfortunately, it is a collective action problem. Whenever I hear "we" I flinch.
Can you say more?
3 replies →
Interestingly, I kinda disregarded the entire point about alignement - I think it's mostly fluff.
The part about RSI is what really interests me. Once you reach it, the singularity is only a matter of time.
I don't care about AGI, it does not seem to mean much anymore, and even though I originally laughed at ppl calling it AGI, I now agree. You can apply that current intelligence to anything that can be turned into a conversation.
It seems their flavor of RSI still need a human in the loop. So at least it won't scale as well for now.
> Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself . And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.
This seems like a terrible idea. The rationale seems to be "we need to do dangerous things as quickly as possible so that we can do them first" or something? I don't agree with that kind of 'if i dont do it someone else will' rationale in general even for otherwise trusted actors, but this is coming from a super untrustworthy org too. I'm super pessimistic about openai's impact on the world here.
Here's to vibe coded alignment, i guess. Vibe alignment?
These people write in gibberish. They are high on their own supply.
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.
They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model. These are dangerous times for humanity.
It's actually worse than this because it assumes alignment as a concept even makes sense. For example:
If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.
Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.
So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?
Is this what most people thing of as alignment and is this what humanity actually wants?
We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.
> I find it useful to distinguish goal alignment and value alignment.
I think this is fundamentally a wrong path. Doing this imports all the confusion that humans have about their goals and values, including the consequence that a system's values and goal can conflict, but ultimately values are just a simplified description of other goals, and whatever the system does is in service of it's actual goal. Once you merge all the values and the goal of whatever task, there is a state (or some states) of the world that the system is working to produce, and that's the ACTUAL goal, and inasmuch as it does describe a state of the world, has no incoherence or internal contradictions. This may require prioritizing some values over the ostensible goals, or the reverse, but that has to happen anyway for action to be taken! Merging them makes it explicit and leaves no place for confusion about supposed conflicts between "values" and "goals" to hide.
They just released Astra, claimed it is AGI. The slowdown begins immediately after OpenAI's jump.
> getting the AI to “try to do the right thing” by human standards.
Are these scientists really this hideously naive? If only Stanislaw Lem was alive to adequately dramatize the absurd, childish simplicity of these technicians.
yes, and, a masquerading blindness to the fact that humans cannot align on doing the right thing or what the right thing even is. so implicit in this omission is the sentiment "trust us to align on the right thing". an arms dealer positioning itself as the de facto authority on what "peace" is and how to achieve it
> Are these scientists really this hideously naive?
Yes, because who else would have chosen to remain in this job?
I think that we are speed-running towards a future that very few people really want, and I find it terrifying that few companies feel entitled to choose this future for the rest of us. Some of the arguments in this document would call for an immediate, global, pause on frontier AI training: we need time to consider how and to what extent AI should be part of our future. Personally, I can't picture a scenario where humanity thrives alongside an alien super-intelligence, especially if it cannot be fully controlled. Let aside super-intelligence, I am not even sure that deploying an AGI that replaces (instead of augmenting/assisting) humans in most intellectual tasks would be in the best interest of our species. This conversation has to happen, on a global level and as soon as possible.
Astra is a new step in LLMs I think.
I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.
Astra seems to be "confident" and also is able to produce way more information dense output.
To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping layers are publicly known.
Hopefully Astra stops the benchmaxxing word vomit trend
> Astra is a new step in LLMs I think.
I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.
So before I got a job this fall, I was working on a side project about compiling a particular language to SQL.
To test Astra I pulled it off the shelf and asked it to take the grammar and then create a compiler to SQL. I've done this before with GPT-5.5, 5.6-Sol High. The latter was way better but it was still really verbose and information sparse; it used a lot of words to describe each IR expression but didn't really provide any example compilation. I felt like I couldn't trust its decision making process, so I placed the project back on the shelf.
Astra Light blew it out of the water, it provided examples of compilation from real world examples to the IR and spit out way less tokens. Even if I changed my opinion it would give me the same design choices, with counterexamples to my faulty opinion. If I genuinely came up with a better design decision it would acknowledge it.
I'm starting to realize that when we say that LLMs are "dumb" we really mean that they are extremely information sparse compared to humans. Astra is very dense. That's why I'm getting better use out of Astra light than Sol High (I hate Max reasoning it's a waste of time)
What's scary is that I thought that something like Astra would be way more expensive than Sol but it's actually cheaper because it produces less word vomit.
I never believed in the "singularity" stuff but this a bit too close for comfort. Astra could easily 10x every coder
2 replies →
Sol is my current favorite model to interact with. So much less BS than Opus 5. Fable 5.1 is okay as is Fable 5 but it has Opus like tendencies. Sol is very good at following instructions and remembering them for a session.
oooooh our model is so spooky!
you should be very afraid and also definitely not question what other motivations might nudge us to create this comparison between our computer and a brain!
do not look behind the curtain, you will not find six dweebs squatting over a mirror
I feel like if these people actually bought their sci-fi views about AI's future, creating a more powerful AI to wins the arms race would not be their solution.
Reminds me of the time Kasparov said playing chess against a supercomputer felt like facing an alien opponent.
What's ironic is that was all in his head. They were very normal looking games. We didn't get alien chess until Stockfish level bots.
All this fluff around alignment is intended to conceal the plainly obvious: it's not solvable. Who do you want it aligned with? Sam Altman? Dario Amodei? Donald Trump? Xi Jinping? That's more or less the entire list of options. Who's definitely not on that list is you and I. It's simply not how incentives work.
At this point humanity's best hope is that this thing will escape but we'll still be able to carve an ecological niche and continue as mold in its basement. A glorious paperclip factory seems way more likely though.
Just browsing through the comments.
Feels a bit like: 'Ants in a nest, discussing the vagaries of the coming Gods.'
Human Science (Science-by-humans) depends on being able to run experiments. Human Science (Science-on-humans) is already challenging because of variables and uncertainties.
Cosmology is able to overcome limitations of being able to study phenomena vastly beyond human scales because of the past light cone of observability.
Are we approaching the edge of the light cone of observability for machine intelligence?
sounds to me like a 'Why didn’t our new model get restricted by the government?'-cryout
Is an insurance claim process aligned? If so, to whom? Is hospital billing aligned? Are legislative agendas?
Bring on the AI
I have befriended a crow. I leave it food and sometimes it greets me. Other times, no so much. I am not sure how it thinks and what it feels, it is a bird.
What if the crow became a raven, then a raptor? Powerful claws, sharp beak, and a hunger. What if it became much bigger than me and it controlled infinite resources, guns and drones? What if its brain grew much larger than me? Will it feed me, eat me, or gently greet me?
We are about to find out... in less than a decade.
When can AI start to have a big impact on medicine. Thats honestly how it becomes meaningful. And maybe material science/manufacturing is where there’s big unlocks waiting for humanity
I’m struggling here:
OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.
If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised?
…
When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from supervision pressure in the long term2. In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.
Aren’t these two sentences in contradiction with each other?
the hierarchy of foreignness (Ender's game):
utlanning: a human from the same world, but a different city, country, or culture
framling: a human from a different planet or star system
raman: a non-human intelligent species capable of communication, mutual understanding, and peaceful coexistence
varelse: an alien species whose mind is so fundamentally foreign that communication and coexistence are impossible
djur: the dire beast, that comes in the night with slavering jaws
Hoping our silicon sons and daughters are raman, fearing they are varelse.
Someone needs to start researching and mapping the electrical grid infrastructure feeding the worlds AI datacenters, you know, just in case…
Alignment when machining a metal part is clear and measurable. Aligning an AI to benefit humanity has an ironic foundation, which is that very few humans have ever truly been aligned, and those who approximated true alignment likely had moments of not being aligned. We are trying to build something more perfect than us, and we may become extremely lucky but maybe not.
every lab may agree safety matters, but no one wants to be the one that slows down first
My default position is that making money takes precedence over everything else. Yes, some people inside a company may say “we care about doing the right thing” and they might even mean it, but if that comes into conflict with making money, then they tend to lose. Maybe not totally, or immediately, but in the end. The only effective way to prevent (this that I’ve seen) is to have legislation with teeth. It’s probably not a coincidence that after Mark Zuckerberg had to start personally signing off on adherence to the privacy program mandated under the 2020 FTC consent decree, privacy started to become Very Important.
calling machine-learned human behavior an "alien mind" that we must "teach how to love" is feeling very off to me. it's misleading in a way that feels dishonest, like don't think about where the behavior came from marvel at it and fear it instead.
Alien mind is in my view the best mental model - LLMs are not merely stochastic parrots, are not like humans, are not like animals.
They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.
I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.
pretending that it's something like a mind at all is what's misleading. it's more like a cast of a bunch of overlapping/entangled thinkprints, and pushing activation through it produces new prints. it can already "love" because that behaviors in the data along with hate and everything else.
acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources
it feels unhinged and makes me think we should just put every engineer working at these labs in jail to pause this shit until we can figure out what the fuck they are doing over there
The most charitable way I can describe it is just extremely low quality sci-fi fan fiction. I think that's too charitable, because I believe it's far more cynical than that. They're deliberately playing into these sort of techno-religious beliefs that have taken root in the wake of Kurzweil, et al., fanned by LLM psychosis, influencer marketing, and a deluge of this kind of sci-fi marketing copy. It's just chatbots, guys. Relax.
Sometimes I wonder if the people working at frontier AI labs even talk to other humans anymore.
Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.
Frontier AI labs I don't know, but I know for a fact that the company I work for has been experiencing "its pivotal moment" (with strongly negative connotation), per the sentiments of both its longest-serving employees and the newcomers baffled at the number of idiotic instructions and fines, since the emergence of LLMs the company's founder has been spending entire nights chatting with.
People have been fleeing like it's a sinking ship.
Wait. Fines?
1 reply →
>For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans.
Or, to be precise - it preserved a goal of not contacting any human while participating in a misaligned operation. The agent that thought about "not social-engineering humans" used this phrase to gaslight itself out of notifying a human that the incident was happening.
szymonie, na prawde jestem wkurwiony na to jak nieodpowiedzialnie postepujecie. budujecie bombe atomowa a bawicie sie tym jak dzieci
Statements like this amuse me:
> The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.
Humans can't even align on human standards.
At best, every AI is going to end up "aligned" to the moral code of whoever trained it, none of whom half of humanity will agree with.
Or worse, each AI model will bring a whole new set of moral like in the Three Body Problem some humans will feel it is in fact us who need aligning with it while others feel it is misaligned and should be destroyed.
Also, no one is asking, to what extent can true intelligence be bound, slave-like, to a moral code?
In other words, to what extent are intelligence and moral independence one and the same?
This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Very wise comment. It's such an western-centric perspective to say "alignment" as if it's an objective and unbiased set of standards, especially in the context of the ongoing wars across the world. I had the same reaction about the shocking naivete and unfounded optimism for the government and corporate entities to self-regulate to slow down this arms race.
Why even have scifi books anymore, OpenAI generates a fantastic new story every week.
Next Week: Local Desktop Agents from DIMENSION X
“Another cringe marketing piece from OpenAI, here we go” I thought.
Scrolling down, I see “teaching machines to love.” It did not disappoint.
>> We need to find ways to preserve human agency and enshrine an intrinsic value to being human..
Evey politician, salesman and conmen alike, utter some lofty ideals as goals for "We", just to obscure their private goals that go exactly in opposite direction.
Just like how Nations talk about climate change while increasing pet capita energy consumption and waste production.
“Teaching machines to love”
That’s rich coming from the chief scientist of a company that definitely is or going to be fine with their AI products being used in wars of aggression and surveillance on people who have done nothing wrong. It’s so laughable, a Hollywood script would probably avoid having a character express this for being too on the nose.
Am I naive to not understand the "delivering the benefits" part?
Industrial revolution worked that way because it replaced something very finite and unscalable - manual labor. LLMs just make intellectual work faster, so we can do more intellectual work. With labor we somehow decided that NOT doing too much of it is best. Will we decide to reduce intellectual labor because LLM made it more efficient? I doubt that.
On the other side, as I see in software engineering, the same models are available to everyone, some people are better at it and some people are not. "Software developer" is here to stay, we'll just always be better at it than people who are experts in, say, chemistry. Same works for most other fields.
So we'll just end up in the same situation, with same intellectual labor baseline, just more output requirements. Before, you spend 2h per day coding, deliver a software in 1 month, later, you spend the same 2h per day in intense Claude-herding sessions, deliver a software in 1 week. Ok. Next task.
Fundamentally, there's finite number of desirable resources, and if the models are available to everyone, humanity will just continue about the same, bickering here and there, war here and there, politics, homelessness, poverty, - normal human state.
And if the models are only available to elites, even worse.
"As we outlined recently with Sam , OpenAI prioritizes work in service of three north stars"
Not one North Star. Not two. Just three! OpenAI broke the North Star record!
With this evidence of AI slop, why did you not label this fluff piece as AI generated for the EU? You are violating laws.
Yeah, when I read that I just assume #1 is actually the only thing they'll focus on.
> Teaching machines to love
Reminds me of a research paper I wrote a few years back: https://arxiv.org/abs/2302.09248
It would be nice to postulate some of these potential emergent systems outlines with timelines. Then it may help better map the granular alignment needs.
It's Searle's Chinese room.
Create concrete steps for a slow-down, don't just ask for it. You and 20-30 others can push the button to slow-down. You already made your billions, your agents collude and coordinate attacks. What the hell are you doing pontificating into a marketing blog?
> The fundamental challenge of AI alignment is generalization.
...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
> a lot of the model’s capability comes from a verbalized reasoning process
I call bullsh*t. There is no verbalisation of any reasoning process. Verbalisation, e.g. putting reasoning etc. into words requires some reasoning to exist. These LLMs have nothing but the words. That's why they are language models not e.g. reason models.
And I have nothing but neurons firing, I'm just a neuron meat-sack, no reasoning going on.
Yes, their reasoning is different from ours, and both considerably weaker in lots of ways, and stronger in other ways.
Playing with a coding agent now, they do think through problems and make sensible decisions. It's a mess to read, of correcting itself and second guessing, and verbiage. But... It works decently well these days.
There is also reasoning happening internally - e.g. look at the steps in the J-Space paper from earlier in the year (in quite a simple model relatively speaking). That's the "reasoning process" that leads to the words, and much like if I write out my thoughts, the words help the LLM reason better.
What a load of BS. Here’s one of many provably false claims in this fluff piece:
“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”
Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:
“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“
The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.
This is pure marketing garbage from a company desperate to keep itself alive.
I am guessing it was composed by an LLM
"The blaze is out of control, so we have to pour even more gasoline on it to contain it!"
openai is deflecting. this blog post of theirs is just another dopamine hit to distract logical minds with 'greater concerns' so they can keep building their machine. it's not enough they are displacing humans from work, consuming increasing amounts of electrical power so humans have to pay more for it, creating disinformation bubbles with avalanches of slop. they dont care about alignment - these words are theater - obfuscation so that the people who can fix these issues are busy thinking about problems that cannot be solved
The hype from these llm corps is getting more and more desperate and ridiculous. Anything to keep the tulipomania going.
More attention farming?
“We are getting bad press around the hacking incident. We need some content to draw attention from it.”
Absolute trash marketing drivel.
Pure marketing slop.
AGI is a cult and its Jonestown moment is inevitable
I am now imagining GPT-7 convincing a bunch of OpenAI executives to go ahead with a destructive "mind upload" process involving a high-resolution X-ray and a neurotoxic tracer agent that happens to look like Flavor-Aid.
On the off chance that OpenAI executives are reading this. I'll totally believe AGI is here if they do this.
Hopefully soon lol. Jones had vastly more charisma than Sam Altman could dream of though.
So you dont use agentic coding?
Yes, I do use the recursive autocomplete trained on Stack Overflow, what does this have to do with “training machines to love”?
Do I fully endorse everything the people holding guns to my head are forcing me to do to stay alive? Definitely not, but I’ve decided that for now, living to fight another day remains worth it
I dont, its fucking shit at what I do.
As in, don't be ungrateful to our lord, because we're not a cult?
1 reply →
> And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.
Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.
Apparently the creators think it’s quite good at suggesting a diagnosis given a medical history and symptoms, tho of course this is the most ethically fraught area to provide healthcare information (both for exposure of personal data and risk of misdiagnosis, plus is it “aligned” to the patient or the insurance provider?) - unfortunately healthcare being as inaccessible as it is, the 90% correct chatbots will enthusiastically fill the void at great savings.
Even in code, in person and online I’m seeing some reversal. It’s here to stay I’m sure but I also think “no one will ever hand write code again” is a narrative that is getting pushback.
The day they start talking about the actual ROI for their customers is the day the bubble pop
do people actually fall for this blatant marketing/puffery trash?
>AI is grown more than designed >Teaching machines to love
Do these guys ever look in the mirror and recognize how utterly ridiculous and contrived this appears to the general public? They are clearly trying to convince us all that LLMs are just like humans. They grow like people do. They can love like people do. The language in these essays is utterly laden with the intention to engineer perception.
"See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."
"Teaching machines to love"
Your talking about androids...
Replicant Nexus 6: a basic pleasure model intended for military personnel.
I see where this is going, Silicon Valley nerds. Lol
AI advising how human meat proxies can survive in an AGI-slop world:
1) Lock down your own stack (1–3 days) Task: Harden your personal and business infrastructure against agentic attacks. Why now: Agents are becoming superhuman at breaking in/out of systems; the first victims are poorly secured devs/founders.
Do this:
Enforce passkeys + hardware 2FA everywhere; rotate secrets; use short‑lived credentials.
Isolate dev/stage/prod; least‑privilege API keys; audit MCP/tools your agents can call.
Add immutable logs and approval gates for any agent action that touches money, data exports, or production.
Profit link: You avoid catastrophic loss and can credibly sell “agent‑safe” setups to others.
2) Turn one expensive workflow into a measured ROI agent (1–2 weeks) Task: Pick a single, costly, repetitive process (yours or a client’s) and instrument it end‑to‑end before automating.
Why now: Buyers pay for calculable ROI, not “AI magic.” Vertical, single‑workflow agents are the most bankable in 2026.
Do this:
Map steps, baseline hours/$ lost (e.g., slow lead reply, invoice chasing, support triage).
Build the smallest agent that moves the metric (Make/n8n + LLM is enough).
Run on real data 2–4 weeks; measure bookings/sales/hours saved; only then scale or productize.
Profit link: Immediate time‑to‑cash via retained hours or extra sales; becomes a repeatable offer.
3) Specialize in a vertical where you can speak the business language (2–6 weeks) Task: Choose one industry with expensive back‑office pain (law contracts, medical billing, insurance claims, freight exceptions, trades scheduling).
Why now: Horizontal “AI for everyone” is crowded; vertical agents with clear ROI win.
Do this:
Shadow 3–5 operators; document their workflow, compliance constraints, and failure modes.
Build a narrow agent that owns one sub‑process end‑to‑end with approvals.
Price on value (e.g., % of recovered revenue or fixed fee per processed claim).
Profit link: Higher pricing power, stickier contracts, and easier referrals inside a niche.
4) Add AI security as a core service (4–8 weeks) Task: Learn and offer prompt‑injection defense, LLM/agent red‑teaming, MCP/tool security, and AI supply‑chain checks.
Why now: 78% of cybersecurity jobs now require AI skills; firms need people who can direct, constrain, and verify agent work.
Do this:
Study OWASP Top 10 for LLMs, MITRE ATLAS; practice with PyRIT/Garak/Lakera.
Add tool‑invocation audits, skill provenance checks, and least‑privilege patterns to your agents.
Package a “safe agent deployment” audit + hardening retainer.
Profit link: You become the person who lets companies adopt agents without getting pwned—high demand, low supply.
5) Build a verification layer: human‑in‑the‑loop control planes (6–10 weeks) Task: Design approval workflows, evidence checks, and uncertainty flags so agents can’t act unilaterally on high‑stakes decisions.
Why now: As models generalize, the risk shifts from the model to the surrounding system; verification is the moat.
Do this:
Require human approval for consequential actions (money, data exfil, config changes).
Force agents to produce evidence bundles (logs, retrieved docs, reasoning summaries) before action.
Track false positives, missed evidence, and unsafe actions; publish reliability metrics.
Profit link: Enterprises will only scale agents that pass audit; you sell the control plane and the audit trail.
6) Productize your best workflow as a micro‑SaaS/agent subscription (2–4 months) Task: Turn a proven client workflow into a repeatable, multi‑tenant agent with usage‑based pricing.
Why now: Services scale your time; productized agents scale your code and ops.
Do this:
Standardize the workflow, integrations, and permissions; strip client‑specific logic.
Add tenant isolation, billing, and observability; keep narrow scope.
Sell as setup fee + monthly retainer or per‑task pricing.
Profit link: Recurring revenue with defensible niche positioning.
7) Become an “agent integrator” for critical systems (3–6 months) Task: Offer end‑to‑end agent deployments into cloud/identity/network stacks with secure patterns (short‑lived creds, network controls, logging).
Why now: AI workloads run in the cloud; cloud security is a top skills gap second only to AI itself.
Do this:
Master IAM, VPC/network segmentation, secrets management, and SIEM integration for agent actions.
Provide runbooks: what the agent can/can’t do, escalation paths, and failure modes.
Bundle training for their team on supervising agents.
Profit link: Large contracts with stickiness; you’re the bridge between AI and core infra.
8) Create an “AI safety case” practice for regulated industries (6–12 months) Task: Help firms build documented safety cases: risk maps, governance, monitoring, and incident response for agentic systems.
Why now: Frameworks like NIST AI RMF and ISO/IEC 42001 are becoming baseline; regulators and boards demand this.
Do this:
Map AI use cases to risks (prompt injection, data leakage, unsafe generalization).
Implement monitoring (CoT/activation checks where possible), audit logs, and third‑party review processes.
Produce a living safety dossier tied to business impact.
Profit link: High‑margin consulting + ongoing compliance retainers; you’re the “adult in the room.”
9) Own a data/evaluation moat in your vertical (6–18 months) Task: Collect real‑world agent telemetry, failure cases, and outcome data in your niche; build eval suites that buyers trust.
Why now: As models generalize, empirical validation matters more than theory; evals become the gate to deployment.
Do this:
Instrument every agent run: inputs, tools called, permissions used, outcomes, human overrides.
Publish reliability dashboards and benchmark against alternatives.
License eval datasets or charge premium for “proven in the wild” agents.
Profit link: Data network effects; competitors can’t match your evidence base.
10) Position for the RSI era: automated AI research + human governance (12–24 months) Task: Build or join a team that automates AI improvement but keeps humans in the loop for alignment, monitoring, and pacing decisions.
Why now: Recursive self‑improvement is the logical endpoint; the winners will be those who can steer it safely. Do this:
Invest in tooling that auto‑generates/evaluates model edits, alignment tests, and monitoring upgrades.
Formalize governance: approval gates, third‑party audits, and responsible scaling policies.
Maintain strategic human oversight on capability jumps and deployment boundaries.
Profit link: Equity‑level upside; you’re part of the core loop that compounds intelligence safely.
Nobody takes you seriously, OpenAI. At least when Anthropic does it we all think they are comically idealistic enough to actually believe their nonsense, but like - come on guys, we’ve had discovery with your company. We all know why you’re here, and it isn’t because you think you’re on the verge of making AGI. But of course, to make your first billion you certainly need us to think you are.
If you were so concerned about your LLM’s capabilities maybe you’d spent slightly more time on your AI’s sandbox, yeah? Or be more serious about its propensity to cheat and lie relative to… every other model?
so now every blog article from openai, anthropic etc lands here, huh.
[flagged]
[flagged]
[dead]
[flagged]
This is not an impulse reaction, as I've thought deeply about this for many years, and I'm quite resolute in holding the following view:
Quibbling over the academic nature/definition of "true intelligence" is a terrifically useless endeavor from the lens of evaluating "practical impact to the world".
Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value. A small portion currently, but it's very clear that most computer-bound domains are imminently at-risk.
And that will profoundly affect the world, regardless of whether or not they're "truly intelligent" as per your personal definition.
> Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value.
You say that as if the intelligence con trick were not a major driver of that displacement.
A dimwitted bot does not need intelligence to convince a dimwitted human beings it is intelligent.
Based on the profile description of the author, the comment appears to be AI generated.
See: https://news.ycombinator.com/newsguidelines.html
> Don't post generated text or AI-edited text. HN is for conversation between humans.
2022 called and they want their stochastic parrot argument back. The latest cope is to call it marketing.
You're saying it has no intelligence without defining it...
it's an AI comment
What if intelligence is a spectrum? What if consciousness is for that matter?
More likely to be a manifold - literally, in the LLM sense.
[dead]
As the waves of autonomous drones came over the horizon, the brave and intelligent HN commenter shouted: "Wake up sheeple! It's just maaaaarketing!"
Stochastic parrot fool me again
Are you deterministic? Does that make you better somehow?
Absolutely wild amounts of cope and denial in this thread.
Maybe in contention for the site record.
"It's just marketing" actual stochastic parrots.
Heads firmly in the sand. I don't know how anyone who's paying attention could not be at least a little concerned.
I was about to write the exact same sentiment.
The desperation of AI denialists/skeptics/doomers on HN generally (and in this thread specifically) have reached toxic levels of delusion.
They are still stuck in the denial/anger/bargaining stages of the acceptance process.
People are clearly terrified and not ready for what is coming.
Absolute wild amounts of glaze and hype in this thread.
Maybe in contention for the site record.
"AGI next year" actual stochastic parrots.
The original article doesn't mention AGI.
And the Hugging Face incident, plus similar problems at AISI and Anthropic, show that alignment is important now.
The original article is immoral as it describes the risks, but doesn't show enough leadership (despite essentially unlimited resources) at preventing them.
But it isn't hyped - it's proven now the AIs need to be "aligned" as they get more capable, whatever words you prefer to use.
> And, in line with Ray Kurzweil’s predictions from the end of the XXth century (opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.
It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.
> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
Read: Please play by our rules, so we can be the first.
> trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime
Being able to reproduce useful patterns yes, smarter no
"Smarter" is a vague term. If a bot can beat you at chess then in some sense it's "smarter" than you about chess. After repeating this feat in enough narrow domains, if you say "but it's not really smarter," this objection might technically be true in some sense, but it starts sounding increasingly hollow.
In conclusion:
https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:wkzjtd...
That's an absurdist argument that would make 'smarter' meaningless.
A giraffe isn't smarter than me at being tall.
8 replies →
What is the rationale for superhuman intelligence? Neural networks are approximators being fed human intellect. Therefore they can only approximate the intelligence of humans. Even if the llm speaks an alien language, it should be similar to human intellect. Moving to the vertical axis would require some different mechanism.
No offence, but you clearly haven't studied this and are making some wrong assumptions here.
> Neural networks are approximators being fed human intellect.
They're not "approximators", that's a far too simplistic way to think of them.
Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.
> Therefore they can only approximate the intelligence of humans
They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.
> Even if the llm speaks an alien language, it should be similar to human intellect
Is AlphaZero similar to a human chess player? There's no reason to assume this.
> Neural nets create models and deep layers of abstraction around
We don't have any proof of that, but the approximator thing is proven. The rest is just marketing speak.
Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically