Comment by sethbannon
6 hours ago
The overarching message is that builders should deeply consider the impact of what they're building on civilization.
"Technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it."
Therefore builders "bear a particular ethical and spiritual responsibility" because "every design choice reflects a vision of humanity."
The questions shouldn't just be 'can we build it?' or 'will people want this?'
We need to also ask 'should we build it?' and 'will this make humanity better?'
The encyclical calls on us to “join forces in building up the common good.”
This is a message we need right now.
Not new for software and hardware industry though, practitioners have just chosen to ignore it. From the Association for Computing Machinery, which encompasses all forms of software development, the very first principle is the public good:
"Software engineers shall act consistently with the public interest. In particular, software engineers shall, as appropriate:
1.01. Accept full responsibility for their own work.
1.02. Moderate the interests of the software engineer, the employer, the client and the users with the public good.
1.03. Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good. ..."
From the IEEE, which also encompasses computer engineering, their first principle and its first few sub-items are:
"To uphold the highest standards of integrity, responsible behavior, and ethical conduct in professional activities.
1. to hold paramount the safety, health, and welfare of the public, to strive to comply with ethical design and sustainable development practices, to protect the privacy of others, and to disclose promptly factors that might endanger the public or the environment;
2. to improve the understanding by individuals and society of the capabilities and societal implications of conventional and emerging technologies, including intelligent systems; ..."
This would be more meaningful if, perhaps, we had to swear an oath to it before being able to practice. And practitioners would be treated more seriously if everyone knew we swore that oath. And the legal utility as accountability and defense would also be useful.
Of course people are going to ignore it if there's no force behind it.
Professional liability and licensure would create assurances with some teeth, but there are some major drawbacks.
1 reply →
It seems to work somewhat with medical doctors and the Hippocratic Oath.
But I would argue it is way easier there. Building software has way more grey areas.
2 replies →
Canadians do
I think not just (computer) scientists but the general population thinks to serve the common good makes sense, not last because we understand it's eventually for our own good.
It is however just that very small minority of the population with highly psychopathic/narcissistic traits - those that, in pre-historic times, would have been kicked out swiftly of hunter & gatherer / small village communities because of their parasitic nature - that in bigger civilisations seem to thrive due to abstraction (distance/time of the effect of their actions) and obfuscation (PR) and instead unfortunately seem to rise to the top (CEOs, presidents, 'thought leaders' ...) to steer the world's overall economy and mindset - and steer it in the abyss.
Sometimes I think humanity was just not made to scale, and this aspect is one very large aspect of it.
Noble ideals can be upheld, relatively consistently… only if violators are visibly punished. And to a degree roughly commensurate with the violation.
The critical point is that the violators have to be punished more consistently then the demanded consistency of the ideals.
Not only builders, the greatest takeaway for me is that everyone has a responsibility in shaping the discourse, culture, and usage of transformative technology. This "the builders will do the right thing" mentality is even (in my interpretation) explicitly called out in several places:
> It is the pursuit of the common good that gives life to a people, understood not as a mere collection of individuals, but as a living reality in which people learn to recognize that they themselves are interconnected and jointly responsible for the res publica. In this sense, every person contributes to the building up of one’s people...
> When it comes to decisions regarding economic flows and digital platforms, as well as the governance of data and algorithms, we cannot allow a handful of actors to dictate these processes on their own; instead, we must build forms of cooperation that respect the various levels of the global community and make them jointly responsible for the common good.
> We cannot be satisfied with merely calling for the moralization of machines — the so-called “alignment” of AI with human values — without also having the courage to insist on a further condition: the possibility of openly discussing the ethical frameworks involved and subjecting them to shared standards of social justice.... What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.
Agreed, I think the builders are the wrong people to ask for this self reflection. Anything can be used for good and bad. A knife can be used to prepare food, or to stab someone. A drone can be used to attack a country, or to defend a country. A wooden beam can be used to build a house, or to build a cross to crucify someone. Same way AI can be used for good and bad. And it’s up to the person using the tool to act moral. Unfortunately this world is full of people that consider power and money more important than morals. For me, religion falls in this bucket, lots of “do as I say, not do as I do”. And on top of this, there’s very little agreement on what is good, and what is bad. Morals are quite fuzzy, and flexible.
The Roman Catholic Church has a lot of things wrong with it, now and in the past, but it’s a human institution older than almost any other, and it’s composed of a lot of very intelligent people. Agree or disagree with them, but a papal encyclical is almost always worth reading and understanding.
Indeed. And because of it’s age and outlook its view is very far back and very far forward.
The casual reference to a 135 year old encyclical that dealt with the seismic shift of industrialisation took me quite aback for a number of reasons.
There are so many encyclicals, apostolic letters, etc. One could spend years reading just a fraction of them, depending on reading and comprehension speed, of course, which varies by person.
Two I recommend, from the last 40 years:
Veritatis splendor, John Paul II, 1993
Argues that Christian freedom is fulfilled, not limited, by objective moral truth: some acts are intrinsically evil regardless of intention or circumstance, conscience must be formed by divine law rather than self-authorization, and the Church must faithfully teach this moral truth as the path to authentic human flourishing in Christ.
https://www.vatican.va/content/john-paul-ii/en/encyclicals/d...
Fides et Ratio, John Paul II, 1998
Argues that faith and reason are complementary paths to truth: reason needs faith to avoid skepticism, relativism, and reductionism, while faith needs reason to express, defend, and deepen its understanding of divine revelation and the human search for meaning.
https://www.vatican.va/content/john-paul-ii/en/encyclicals/d...
The Roman Catholic Church is an institution that went to war to try to prevent people from being able to read the bible in their own languages, that executed dissenters who challenged the doctrine on which it based its power, from scientists to Protestants. If it had never lost power in western Europe, it's quite possible the enlightenment and industrial revolution would never have happened, and humanity would still be living in the dark ages.
It's an organization that is effective as an opposition party of sorts.
[dead]
The strange part is how moral responsibility somehow always lands on the builders... the people with the least leverage... while the funders get to ask the ethical questions. Weird!
No, we don't have to take the funders' money; that's what having professional standards means. Nobody would excuse a doctor performing unsafe procedures because they "needed the money". Engineers were jailed for the Volkswagen emissions tampering scandal and nobody would excuse them for needing to take funders' money.
I agree that holding only the concrete implementers responsible would be inappropriate. However, I don't believe that distinction is made. One says "I am building a house" even if they are completely contracting the job out. I'd suggest the greatest responsibility lays with the funders and that the Pope would agree.
The funders are among the principal builders in this context. This is addressing the people who have a say in what is built, and how it is built. Much of that belongs to the executive and ownership class, but not all of it.
Mark Zuckerberg is a builder in the context of the encyclical.
Yes but... No one living today knows what direction AI technology will take humanity. If we have an algorithm breakthrough then we may avoid building new data centers. If the abilities of the technology plateau then there might not be large impacts on employment. Builders need to focus on the impact of their next steps. Don't put polluting natural gas generators in neighborhoods today. Don't make unemploying folks the goal of your tool, today. Don't make decisions that harm people and the environment today.
https://www.theguardian.com/us-news/2026/may/13/utah-approve...
> The Stratos artificial intelligence datacenter footprint will cover more than 40,000 acres (62 sq miles) over three sites in Box Elder county in north-western Utah. The facility will require about 9GW of power, which is more than the entire state of Utah currently consumes
Sure hope your rosy inflection point happens real soon.
Otherwise, the direction this is taking us is pretty obvious.
9 GW is massive. About half a percent of world electricity consumption, half of what stupid Bitcoin burns.
As I keep telling my project owner at work, "the AI will get better" is not a plan.
> Scientists are actually preoccupied with accomplishment. So they are focused on whether they can do something. They never stop to ask if they should do something.
From Crichton's book Jurassic Park, which like most of his books is about the perils of technological advancements.
They used the quote in the movie, slightly tweaked.
I would replace the word scientists with engineers in that quote. People often conflate the two, but in my experience, scientists tend to be more cautious and there are built in checks and balances in the process (however flawed).
Engineers/technologists tend to have no such guardrails, and are also usually embedded into entirely profit motivated environments, whatever their own values might be.
This sounds like you don’t have much exposure to actual professional engineering disciplines. I’m sure civil, electrical, structural and mechanical PEs would be quite surprised to hear there are no guardrails on their professions.
5 replies →
I worry that companies like Anthropic, which are moving toward RSI, may prioritize speed over the timely identification of irreversible risks.
Every time I prioritize speed over risks, I, too, end up with repetitive strain injuries.
Did you mean AGI? I'd think whatever else they are doing, LLMs are reducing RSI...
[dead]
> Therefore builders "bear a particular ethical and spiritual responsibility"
> This is a message we need right now.
Feels good man. The solution found by the private parties driving technological change is sainthood. Or aiming for it. At least, better than you. They have the vision of what's good for the herd, but the more time I spend as a sheep, the more it seems the "herd" is just a way to recycle the story of their own exceptionalism stripped of any mark of individuality. A simple visit to fiftyyears.com will greet you with "We back the indispensable". I guess it's the same "we".
hard disagree. No matter what gets built, it will be used for bad things if the society is not properly constructed. We need to fix our government and then you can build anything you want. I don't like putting the blame on the builders, building is already difficult as it is. Also you can't built something in isolation, word will get out anyway. That's why this speech is either naive or he just wanted to tick a todo list: say something about AI.
> We need to fix our government and then you can build anything you want.
Are we all going to stop building things until the government is fixed and all political problems are resolved? No? Then we must think about how we should build in a world with imperfect government.
> I don’t like putting blame on the builders
Why does it have to be either/or?
From a high-level view, there are powerful cultural/political forces that nudge us towards building harmful, wasteful things without us being aware of said forces. The government, corporate greed, and billion-dollar marketing budgets are to blame.
But on the ground, from an in-the-trenches perspective, I think an engineer at Meta, for example, shares in some level of blame, too. Mainly because there is plenty of evidence showing how much harm Meta products have brought upon society. Yet Meta couldn’t be built without engineers opting in to work there in exchange for $300k salaries.
We desperately need moral clarity and courage at both the policy-level and the individual level.
Because there will always be bad actors. You cannot design a system on all actors' full cooperation.
Ofc they share the blame, but it is not solving the problem.
You can say for example: loan sharks are bad for society; so government gives anyone 0 percent credit. You just removed one problem, created another.
Just and sustaining system with individual morality is destined to fail. Only option is social regulation. Which is at government level.
It is simply not feasible.
It's not either/or in strict terms, it absolutely is either/or in practical terms. When the conscientious engineer chooses not to take the 300k job, the next one in line does. If enough choose not to, it just became a 400k job.
Can you change society in such a way that nobody would take such jobs? In strict terms, sure, in practical terms it probably entails enormous costs, both economical and societal. And then you still have other countries.
There's an entire legal code filled with things on which we can't rely on the morals of the people. We can't stop theft, rape and murder, what makes you think that stopping engineers is any more feasible?
The best thing builders can do is use their knowledge and authority to pressure the other side.
I get where you are coming from but this is the common "reduction to politics" that anyone who doesn't want to address a problem uses: think of any societal or human problem and you can have your comment with different nouns.
Sure, IF we could just go and fix our governments in some magical way then the problem would disappear. That goes from hunger, climate change, videogame addiction and AI. The problem is that what you value in life in different than what others do, so we now have a system in which sometimes you get what you want and sometimes you don't.
But back to the topic, I do think that how OpenAI and Anthropic handled the government and them asking to drop guardrails is something a company can actually and actively do without having to reinvent the universe.
Hard disagree to your disagree. Every organization is made up of people and reflects the character of its members. So with the nation itself. If people won't take responsibility for their own actions, neither will the government. I.e. self-government isn't the answer to fixing a broken people. Transformation begins within one's self, not in imposing one's will on another. First everyone much look to themselves, then the government, organizations, and projects will fix itself.
Everyone likes to talk about "fixing the government". It feels nice to understand the problem with something else that is broken. The problem is that replacing the people in charge is a no-op if you don't have a pool of good people to choose from and the will to choose them.
"That which can be destroyed by the truth should be." - P.C. Hodgell, from her novel Seeker's Mask (1994)
"That which can be automated by the AI should be."
Consciousness is the causal factor, not government. You are putting the cart before the horse to insist on any political solution.
> I don't like putting the blame on the builders
Like it or not if you knowingly build stuff that is used for evil purposes you are complicit
You can't build an orphan mulching machine and get away with just a shrug. "I don't really agree with mulching orphans but someone else paid me to build that. If I didn't build it someone else would have" just does not absolve you of your involvement in mulching orphans
When he speaks of the builders needing to take responsibility, I think he is directing his comments at tech company leadership. The Sam Altmans of the world, not the poor slobs like us who need a job for health insurance because apparently unemployed people don't deserve to live.
No, he's talking to you too. The Sam Altman's can't be a threat without useful idiots to cooperate with and doing work for them. You don't get out of your own culpability because you locked on a set of golden handcuffs.
[dead]
I have only read a few passages (and some of the excellent quotes others have shared here), but I find the underlying message here so much more compelling than those found in the various "manifestos" which come out of Silicon Valley.
I think reading this helps me imagine a version of the future I'd actually like to live in. A version where technology is used well (rather than preaching for abstinence from technology) and where values other than "intelligence" (in whatever guise) are on an equal footing.
Even writing that makes me feel naive (and to an extent I know it is) but I think it would be inconsistent for someone who cheers for humanity's efforts to solve/chip away at "impossible" problems (like LLMs were thought to be not so long ago) to shirk from the challenge of making the world better for _everyone_.
The thing is why that this feels so good future is; it is a system with no constraints. A bit like Star Trek universe in Roddenberry's imagination. This kind of utopia can only be achieved with all honest actors, but in reality systems are usually designed around bad actors.
Even with all morally good actors locally, there is no guarantees for external forces. Thinking it hypothetically, even with global coordination ( all good actors ) there is not a proven path that would lead us to better place from any starting point from past.
It's probably more predictive to model actors as being neither good nor bad but constrained by various collective dilemmas, such as prisoners dilemma, the security spiral, tragedy of the commons, race dynamics, collective action or first mover problems, information asymmetries, the commitment problem, among others. Those are the hardest problems to solve because they're pathologies that result from the global, largely amoral structure rather than consequences of the individual exercise of morality.
In the AI case, each firm is in an arms race, and nobody can slow down without effectively collapsing due to positive gross margins only being viable with a frontier model that attracts marginal demand. An appeal to morality might have an impact but more effective action would be to address the structure that the AI companies are situated in that causes this dynamic in the first place. In practice, thats going to be a global agreement to slow down, and global regulations.
1 reply →
This sort of rationalization of evil is a core of technocratic support for Trumpism, I find, and has parallels to the evangelical prosperity gospel. Choice tenets:
It’s always cloaked in a veil of realism, but it’s just the classic 14-year-old-boy-just-got-introduced-to-the-prisoners-dilemma situation. There’s nothing philosophically interesting about it.
Ironically, these are often the same people denouncing multiculturalism, yet the culture they strive for is completely morally bankrupt.
4 replies →
To get from here to Roddenberry's communism, according to Roddenberry's lore, we passed through the Eugenics Wars, the Second Civil War, and then fifty years of World War Three and the 'post-atomic horror' before coming to our senses.
> I think reading this helps me imagine a version of the future I'd actually like to live in. A version where technology is used well (rather than preaching for abstinence from technology)
I believe the Amish figured this out over a century ago.
> I believe the Amish figured this out over a century ago.
The Amish rather came to a different conclusion (which I don't want to judge on, but on which I nevertheless have a different opinion than the Amish).
1 reply →
The church, the message comes from, literlay makes the world worse for so many people on the planet.
Woman have less value under the church than man. Alternative sexual views are evil.
Church and any kind of believe system hurts our society and divides us.
> Church and any kind of believe system hurts our society and divides us.
Any belief system? And yet I bet you value freedom over slavery, wisdom over ignorance and compassion over brutality. That’s a belief system, despite not being a religion.
3 replies →
"Church"
ok
"and any kind of believe system hurts our society and divides us."
People shouldn't believe anything?
Disagreement and conflict are natural. How we handle these disagreements while striving for widespread peace and prosperity is the question.
1 reply →
Does that mean that nothing of value can come from the church, and we should ignore all ideas they put forth out of some kind of spite?
4 replies →
Whenever I hear these "tech overlords", I am always baffled at the total lack of culture, the absence of taste, the empty visions and the implied complete subjugation of humans to ideals of "efficiency" or "quick and easy". Maybe they would have been more interesting people if they had been brought up in beautiful towns and cities, if they had lived in a rich cultural environment instead of being raised as consumer of cheap and flashy pop culture. Maybe we should tax bad architecture, it gives me headaches but others might incur heavier damage.
As an aside, at least Trump is drawn to the grandeur of high culture from historical times, but he also doesn't understand a jota about aesthetics, and so the White House gets turned into a tacky gypsy-style abomination with one dollar ornaments.
We lost the “liberal education” (not the political one, but the “freeing” classical one) and it’s starting to show.
When you compare the robber barons to Google and Meta it’s kind of embarrassing- they build massive empires of iron horses screaming across the world and covered cities in magnificent buildings (stations, libraries, etc). G&M built an empire of advertising and … not much else?
3 replies →
Cool, inspires me to think, "should we regulate away the toxic side effects which we realize COULD have come, if we built it less considerately?"
Science and Engineering is morally detached.
Ask a scientist or engineer what philosophy or theology has taught us about the source of morality and their education, training and experiences havent prepared them to answer that question.
This didnt matter so much in the past because their activities never had the scale it does today. For basic training in philosophy if you are mid or upper level exec whose decisions are going to effect a whole lot of people, go to open yale courses and take the intro to Philosophy classes. It will help develop your answer to the question - why do you do what you do if you are going to die anyway tomorrow.
This is a ridiculous comment. Science papers always have sections on impact. It's the running with scissors industry types simply chasing the bigger paycheck that don't stop to think.
Far too many of us (particularly younger people, but not only them) undervalue or are dismissive of philosophy. I once was like this, partly because at the time I'd been brought up - like most humans - to believe my parents' religion held all the answers philosophy might address.
I quickly learned as an adult that whether you're a person of faith or not, it's not pointless at all. It's the foundation of everything. Philosophy is how you explain the deeper reasons behind why you follow whatever religion you do, or adhere more meaningfully to whatever kind of agnosticism, atheism and/or 'spirituality' (with or without woo) you espouse.
[dead]
This can only be true if you believe in a certain mad-scientist version of these things. We as a society have systems (however flawed) to hold scientists and engineers to ethical standards: e.g. peer review and in many cases--e.g. civil engineering, architecture, medicine--even legal frameworks to enforce ethical standards. Science and engineering are human endeavors, they are not divorced from the human condition, they cannot be separated from humanity and human rights.
Maybe you mean mathematics is amoral and are committing the common conflation of "engineering is just applied science and science is just applied mathematics," which is a really bad case of missing the forest for the trees.
Thanks for this!
Not from a church which indocrinates young people and has a massive child fucking scandal in the running for the last 15 years.
>for the last 15 years.
The scandal broke out in worldwide media starting in 2002 (not coincidentally, the same year South Park released Red Hot Catholic Love). This reminds me of when Contrapoints off-hand commented about how things were different in the 70s but that was "30 years ago".
I would also recommend God and Golem, Inc., A Comment on Certain Points Where Cybernetics Impinges on Religion:
https://monoskop.org/images/1/1f/Wiener_Norbert_God_and_Gole...
Norbert Weiner was an atheist but he talks about three areas religion is the only thing to have really examined that relate to capable AI: omniscience, omnipotence, and worship (gadget worship). It has very prescient stuff on blackbox learning/distillation, reinforcement learning/reward hacking, alignment through human feedback.
His The Human Use of Human beings and Cybernetics are extremely good too and have more of a mash of the themes between Rerum Novarum and Magnifca Humanitas, and more near-term automation.
Awesome, thx for posting. I now have my new next book to read. Been wanting to read more of the original cybernetics stuff.
Would you also recommend "The Human Use of Human Beings"?
Yes, it starts slow with a lot of history of Cybernetics stuff through Leibniz and stuff (kind of prescient given chain rule -> back propagation, and control theory's relevance to optimizers). It is about twice as long as God & Golem and covers a lot of the same plus more examination of automation and human augmentation. I think either it or Cybernetics also goes into mass communication with some relevance to how social media played out.
God & Golem is the most succinct and up to date though, probably a 2hr read.
The book Cybernetics is a lot of math and ergodic theory stuff that went beyond me, but is the longest and you still get a lot out of it skimming over that stuff if you don't have the background for it. The last revision of it in the 50s added some of the same blackbox function copying/imitation learning/distillation stuff, reinforcement learning with reward hacking concerns, and superintelligence as genie/monkey's paw.
I would read the three in reverse order of publication.
He also foresaw another big area of potential existential danger, Wiener filter for guidance and control of missiles (later superceded with Kalman filter bringing the nuclear hard targets era with 15min retaliation windows) and refused to work on it or share prior work, and he also had bioweapons delivery concerns before the bioweapons treaties, publishing this open letter in The Atlantic in 1947:
https://archive.ph/D7BPt
"Technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it."
My comment under the post on Omarchy here got downvoted and flagged by the "technology is neutral" crowd because I dared to say that it is unethical to use software produced by a white supremacist.
[flagged]
[flagged]
Under this logic, would the Pope have built a Hammer? a Knife?
I am hoping the Pope has a cleaner view of AI quietly automating the drudgery in the back offices and not just robot dogs with machine guns. And with that view, should we build it?
It's too simplistic to imagine the tension is between robot patrol dogs vs automating drudgery. If automating drudgery suddenly puts 30% of people out of work, it has huge broad negative impact on people who are currently alive and working in the current system. Innovate, but do it with awareness.
Than an even better question for the pope is:
If a technology existed that reduced the cost of producing a critical thing (think food, housing, medical care) down to near zero, however, it made the humans currently building the thing redundant, should we build it? Would it be okay to use the hyper-optimization power of Capitalism to build such a technology faster?
1 reply →
This is also the message I got from "the wind rises"! Though from talking with other people that takeaway doesn't seem universal -- which IMO is one of the ways to tell it's a great film :)