I have read and enjoyed each of Yuval Noah Harari’s previous books. Sapiens: A Brief History of Humankind; Homo Deus: A Brief History of Tomorrow; and 21 Lessons for the 21st Century. As the titles suggest, he writes about history and persuaded this reader that is the only context to fully understand what is happening in the world.
This book scared the shit out of me. I grew up during the early days of the Cold War, watching B-52 bomber packed with nukes flying overhead. As a teenager, I held my breath with the rest of the world during the Cuban Missile Crises. But Harari makes a compelling case for AI (assuming we fuck it up and we will) as a greater existential threat.
Like all of Harari’s books, this one (about 400 pages) got a loft of highlighter. More excerpts after the jump. Or you can watch this 40 minute discussion with Sam Harris.
Never summon empowers you cannot control
Human power is never the outcome of individual initiative. Power always stems from cooperation between large numbers of humans.
While each individual human is typically interested in knowing the truth about themselves in the world, large networks bind members and create order by relying on fictions and fantasies.
We should not assume that delusional networks are doomed to failure.
What we usually think of as ideological and political conflicts often turn out to be clashes between opposing types of information networks.
Information is increasingly seen by many philosophers and biologists, and even some physicists, as the most basic building block of reality, more elementary than matter and energy.
The “naive view” – information is an attempt to represent reality, and when this attempt succeeds, we call it truth.
Errors, lies, fantasies, and fictions are information, too. Information has no essential link to truth. pg 12
People think they connect to the person, but in fact they connect to the story told about the person. pg 21
After his death, Jesus became the subject of one of the most remarkable branding campaigns in history. […] The story of Jesus managed to have a much bigger impact on history than the person of Jesus.
While most Christians were not physically present at the Last Supper, they have heard the story so many times, and they have seen so many images of the event, that they “remember” it more vividly than they remember most of the family dinners in which they actually participated.
Telling the truth about the universe is hardly the most efficient way to produce order among large numbers of humans. […] What holds human networks together tends to be fictional stories, especially stories about intersubjective things like gods, money, and nations. When it comes to uniting people, fiction enjoys two inherent advantages over the truth. First, fiction can be made as simple as we like, whereas the truth tends to be complicated, because the reality it is supposed to represent is complicated.
An uncompromising adherence to the truth is essential for scientific progress, and it is also an admirable spiritual practice, but it is not a winning political strategy.
Religions always claim to be an objective and eternal truth rather than a fictional story invented by humans.
Though neuroscientists have made some progress in the study of memory, nobody yet understands what memories are, or how exactly they are stored and retrieved. pg 49
Mythology and bureaucracy are the twin pillars of every large-scale society.
It takes a minute to tweet allegations of bias, fraud, or corruption, and many weeks of arduous work to prove or disprove them.
The Bible as a single holy book didn’t exist in biblical times. King David and the prophet Isaiah never saw a copy of the Bible. […] no two ancient Bibles were identical.
Catholic theology accepted that Jesus told us to love our enemies, but explained that burning heretics was an act of love, because it deterred additional people from adopting heretical views, thereby saving them from the flames of hell.
The first rule of changing church teachings is that you never admit to changing church teachings.
Elections establish what the majority of people desire, rather than what the truth is.
Prior to the development of modern information technology, there are no examples of large-scale democracies anywhere.
In 1960, about seventy million Americans (39 percent of the total popula-tion), dispersed over the North American continent and beyond, watched the Nixon-Kennedy presidential debates live on television, with millions more listening on the radio.
Like the Catholic Church, the Bolshevik party was convinced that though its individual members might err, the party itself was always right.
In recent years we have been inundated with so many groundbreaking inventions that it is difficult to determine what is driving this (information) revolution. Is it the internet? Smartphones? Social media? Blockchain? Algorithms? AI? […] The seed of the current revolution is the computer. Everything else from the internet to Al—is a by-product.
A computer is a machine that can potentially do two remarkable things: it can make decision decisions by itself, and it can create new ideas by itself.
Since we don’t understand how consciousness emerges in carbon-based life-forms, we cannot foretell whether it could emerge in nonorganic entities. Perhaps consciousness has no essential link to organic biochemistry, in which case conscious computers might be just around the corner.
Just as airplanes fly faster than birds without ever developing feathers, so computers may come to solve problems much better than humans without ever developing feelings.
The emergence of computers capable of pursuing goals and making decisions by themselves changes the fundamental structure of our information network.
Computer-to-computer chains can now function without humans in the loop. For example, one computer might generate a fake news story and post it on a social media feed. A second computer might identify this as fake news and not just delete it but also warn other computers to block it. Meanwhile, a third computer analyzing this activity might deduce that this indicates the beginning of a political crisis, and immediately sell risky stocks and buy safer government bonds. Other computers monitoring financial transactions may react by selling more stocks, triggering a financial downturn. All this could happen within seconds, before any human can notice and decipher what all these computers are doing.
Religions throughout history claimed a nonhuman source for their holy books; soon that might be a reality. Attractive and powerful religions might emerge whose scriptures are composed by AI. Pg 209
For thousands of years prophets, poets, and politicians have used language to manipulate and reshape society. Now computers are learning how to do it. And they won’t need to send killer robots to shoot us. They could manipulate human beings to pull the trigger.
The Bible had a profound effect on billions of people, even though it was a mute document. Now try to imagine the effect of a holy book that not only can talk and listen but can get to know your deepest fears and hopes and constantly mold them.
AI isn’t progressing toward human level intelligence. It is evolving an entirely different type of intelligence.
I prefer to talk about the computer network in the singular, in order to contrast it to the human network it is superseding.
When we write computer code, we aren’t just designing a product. We are redesigning politics, society, and culture, and so we had better have a good grasp of politics, society, and culture. We also need to take responsibility for what we are doing.
Altogether, in 2023 more than one billion CCTV cameras were operative, globally, which is about one camera per eight people.
“In the past, stepping into a taxi or barbershop meant stepping into someone’s private space. Now, when customers come into your taxi or barbershop, they bring cameras, microphones, a surveillance network, and thousands of potential viewers with them. This is the foundation of a nongovernmental peer-to-peer surveillance network.”
(A) social credit system seeks to give people points for everything and produce an overall personal score that will influence everything. […] One way to think of the social credit system is as a new kind of money. […] A comprehensive social credit system will annihilate privacy and effectively turn life into a never-ending job interview. Anything you do, anytime, anywhere, might affect your chances of getting a job, a bank loan, a husband, or a prison sentence.
In quantum mechanics the act of observing subatomic particles changes their behavior; it is the same with the act of observing humans. The more powerful our tools of observation, the greater the potential impact.
We have reached a turning point in history in which major historical processes are partly caused by the decisions of non-human intelligence.
As we have seen again and again throughout history, in a completely free information fight, truth tends to lose.
The problem with computers isn’t that they are particularly evil but that they are particularly powerful.
The humans who currently engineer computers need to accept that they are not manufacturing new tools. They are unleashing new kinds of independent agents, and potentially even new kinds of gods.
AI doesn’t have any emotions of its own, but it can nevertheless learn to recognize these patterns in humans.
We regard entities as conscious, not because we have proof of it, but because we develop intimate relationships with them and become attached to them.
(AlphaGo) uncovered ideas that hadn’t occurred to the most brilliant players in thousands of years.
Al is expected to add $15.7 trillion to the global economy by 2030. But if current trends continue, it is projected that China and North America-the two leading Al superpowers-will together take home 70 percent of that money.