Author: Ben

Since the ancestors of homo sapiens first made tools out of rocks and sticks and grass, society has been transformed by the development of ever more powerful technologies, from stone axes to the steam engine to the GPS satellite. Computers and the internet, the most important technologies of recent decades, are the latest step in this long, long process.

How important are these technologies? How much have they changed society? Compared to other events in living memory, they have been revolutionary. The world’s most valuable corporations are now mostly internet software companies. The internet has been responsible for the rise and fall of heads of government, and sometimes of entire governments. Computers and the internet play a large role in the daily life and experience of billions of people. I am writing this on a computer right now to share it over the internet.

Commentators have therefore described computers and their effects as among the greatest transformations to ever strike human society, or occasionally even the single most important transition in the history of the species. We see this in the use of terms like “Information Age” or “Digital Revolution”. According to Wikipedia’s article on the Information Age, “During rare times in human history, there have been periods of innovation that have transformed human life. The Neolithic Age, the Scientific Age and the Industrial Age all, ultimately, induced discontinuous and irreversible changes in the economic, social and cultural elements of the daily life of most people. Traditionally, these epochs have taken place over hundreds, or in the case of the Neolithic Revolution, thousands of years, whereas the Information Age swept to all parts of the globe in just a few years.” Or in the “Fourth Industrial Revolution” framework popularized by the World Economic Forum, human history has seen four distinct industrial revolutions, and fully half of them are the result of computer technology.

From a historical perspective, however, the social changes caused by computers do not live up to these claims. There are many earlier technologies that have also transformed the structure of society and the landscape of power: bronzeworking, electricity, antibiotics, the horse collar, the radio, gunpowder, the railroad, the atom bomb, contraception, the printing press… any student of history could keep the list going. To put computers into historical perspective, we cannot look only at the effects of computers. We also need to compare these effects to those of other major technologies.

There is no definitive comparison, but one rough categorization scheme is below:

1. Utterly transformative. The difference between societies with and without this technology was on par with the difference between societies of different hominid species. Examples: Agriculture, writing, fire.
2. Civilization-scale. This technology was sufficient to force a complete reorganization of one of a civilization’s most basic functions, such as economic production or political legitimacy. Examples: Centralized irrigation, printing press, steam engine.
3. Transformative. While this did not force a major reorganization of how the civilization’s core institutions related to each other, the individual institutions in the relevant fields were forced to reorganize themselves to adapt to the new technology, or else were supplanted by those that did. Examples: Railroads, automobiles, broadcast radio, muskets.
4. Decisive. While the civilization’s core institutions could adopt the new technology without major reorganization, details of the technology’s powers and limits were a major factor in the specific balance of power between institutions and a source of local advantage. Many particular institutions rose and fell as a result of the new technology—companies went bankrupt, militaries lost wars, governments lost elections. Examples: Artillery, airplanes, television.

A comparison like this cannot be perfectly objective. You might argue over exactly where to place different technologies. Perhaps the railroads should be at level 4 rather than level 3, depending on how much of a role you assign them in the development of megacorporations (or “trusts”, as they were called at the time). Or you might use a different ranking system that emphasizes something else, e.g. whether a technology changes the total number of people a society can support, or how much of an average person’s time is spent interacting with the technology—this scale focuses on how much a technology affects the structure of society only because that’s the question we’re asking right now.

Still, we can get a very rough ranking of how different technologies stack up against each other. It is safe to say that writing transformed society more than the telegraph, or that the printing press transformed society more than the airplane, or the steam engine transformed society more than artillery. Different rankings will disagree on edge cases, but big differences should be consistent. Keeping in mind that this process is inherently very fuzzy, let’s run with this scale for now.

Where do computers and the internet fall on this scale?

These technologies have allowed many individual institutions to dominate the competitive landscape. In business there are companies like Microsoft, Google, and many more. In politics there are the campaigns of candidates like Barack Obama or Donald Trump. Advertising and political education which had previously been done via other media has shifted more and more to the internet. The transformation in journalism, academia, and intellectual discourse more broadly has been especially dramatic.

Economically, computers have had their strongest impact on record-keeping and administration, where they have been truly revolutionary. More and more commerce is being done over the internet. Industrially, these technologies’ effects have also been substantial, but hardly overwhelming. Computerized tools such as articulated robot arms or CNC machine tools occupy very important niches. Thanks to tools like these as well as incremental efficiency gains from computerized administration, computers have helped industrial productivity to continue its long-term growth, albeit at rates slower than those seen in the first half of the 20th century. But the industrial effects of computers are not nearly as deep or as widespread as the industrial effects of famous earlier innovations such as interchangeable parts or Bessemer steel.

There are also the arguments that computers should be considered revolutionary because of predicted future changes. Mass automation will transform industrial production as deeply as the Industrial Revolution of old. The blockchain will usher in a new financial age. Swarms of autonomous combat drones will make infantry obsolete. Chatbots will put the laptop class out of work. Unfriendly artificial general intelligence will disassemble humanity for parts. Perhaps reality will someday catch up to the think tank whitepapers and science fiction—it wouldn’t be the first time. Or perhaps belief in these hypothetical technologies will peter out like we’ve seen in quests for the holy grails of previous generations, like moon bases, fusion power, or the cure for cancer. Predicting the future of society is hard, and predicting the future of technology is even harder. Narratives which try to claim a special place for computers on the basis of predicted future technologies, which could arrive in ten years or a hundred years or never, cannot play a role in our assessment of what computers have achieved so far.

Taking all this together, in terms of the four-level scale above, I would rank computers at level 3. This judgment is necessarily a bit subjective, and I can imagine events developing so that in the year 2080 we look back and move the classification up or down a notch. Still, this lets us put some bounds on the computer’s importance relative to other technologies. Computers have changed our economy, but not nearly as much as the steam engine did. The internet has changed our intellectual environment and structures of political legitimacy somewhat more than the transition from radio to television, but much less than the printing press. In short, computers are a big deal, and very probably the most important technological development in living memory. But from the perspective of human history as a whole, the computer doesn’t stand out. There have been dozens of technologies that were at least as important, although probably fewer than a hundred.

Then why do so many people think computers are the most important transformation ever? Because many of these commentators aren’t trying to place the development of computers within a firm understanding of the grand sweep of history. They’re trying to explain their own direct experience of the world. In their own lifetimes, computers were indeed the most revolutionary new technology, and compared to the things that they experience firsthand or spend time thinking about, nothing else comes close. It’s tempting to attribute big changes to the most powerful force you’ve experienced, and it’s tempting to believe that the changes happening in your own lifetime are the most important in human history. Or sometimes people are trying to hype up new projects and products, which of course calls for boosting the present technology and gliding past the previous cases. But in either case, the reason their viewers and readers let them get away with it is that most people just don’t know the history they’re implicitly comparing to.

To take one personal pet peeve as an example, I have heard dozens and dozens of people describe how the internet has created never-before-seen problems in journalism, completely unaware that the unprecedented problems they describe were just as bad or worse in the journalism of, say, the late 1800s. Of course it’s perfectly fine to think about these things without knowing all the history, but making historical comparisons without knowing the history is dangerous. You don’t need to know who Charles Sumner was before you say that news media is more polarized today than it was in your childhood. However, if you want to say that social media has made the discourse more fragmented and conspiracy-prone than ever before, you do need to know who Alfred Dreyfus was.

Comparisons like these are necessary to thinking about how dramatically computers have changed our world. Media and intellectual discourse have been transformed many many times in modern history, and evaluating the scale of the internet’s changes requires a memory that goes back further than American media’s golden age of the late 20th century. While anyone can and should observe the very clear fact that computers are changing the economy, figuring out the significance of a “Fourth Industrial Revolution” requires a comfortable knowledge of the First. When you hear a claim that computers have caused some gigantic change, you should ask yourself “Compared to what?”

Plenty of common arguments cast computers and the internet as far more transformative than they actually are, often through ignorance of history or through wildly overconfident predictions about the near future. Yet there’s no need to exaggerate. When taking the long view and properly comparing computers to other technologies, they are still a pretty big deal. We can think about their effects on society even if they haven’t caused a fundamental transformation.

Several Rationalists objected to my recent “Against AGI Timelines” post, in which I argued that “the arguments advanced for particular timelines [to AGI]—long or short—are weak”. This disagreement is broader than predicting the arrival of one speculative technology. It illustrates a general point about when thinking in explicit probabilities is productive and illuminating vs when it’s misleading, confusing, or a big waste of effort.

These critics claim that a lack of good arguments is no obstacle to using AGI timelines, so long as those timelines are expressed as a probability distribution rather than a single date. See e.g. Scott Alexander on Reddit, Roko Mijic on Twitter, and multiple commenters on LessWrong.¹

And yes, if you must have AGI timelines, then having a probability distribution is better than just saying “2033!” and calling it a day, but even then your probability distribution is still crap and no one should use it for anything. Expressing yourself in terms of probabilities does not absolve you of the necessity of having reasons for things. These critics don’t claim to have good arguments for any particular AGI timeline. As far as I can tell, they agree with my post’s central claim, which is that there’s no solid reasoning behind any of the estimates that get thrown around.

You can use bad arguments to guess at a median date, and you will end up with noise and nonsense like “2033!”. Or you can use bad arguments to build up a probability distribution… and you will end up with noise and nonsense expressed in neat rows and figures. The output will never be better than the arguments that go into it!²

As an aside, it seems wrong to insist that I engage with people’s AGI timelines as though they represent probability distributions, when for every person who has actually sat down and thought through their 5%/25%/50%/75%/95% thresholds and spot-checked this against their beliefs about particular date ranges and etc etc in order to produce a coherent distribution of probability mass, there are dozens of people who just espouse timelines like “2033!”.

Lots of people, Rationalists especially, want the epistemic credit for moves that they could conceivably make in principle but actually have not done. This is bullshit. Despite his objection above, even Alexander—who is a lot more rigorous than most—is still perfectly happy to use single-date timelines in his arguments, and to treat others’ probability distributions as interchangeable with their median dates:

“For example, last year Metaculus thought human-like AI would arrive in 2040, and superintelligence around 2043 … If you think [AGI arrives in] 2043, the people who work on this question (“alignment researchers”) have twenty years to learn to control AI.”

Elsewhere he repeats this conflation and also claims he discards the rest of the probability distribution [emphasis mine]:

“I should end up with a distribution somewhere in between my prior and this new evidence. But where?

I . . . don’t actually care? I think Metaculus says 2040-something, Grace says 2060-something, and Ajeya [Cotra] says 2050-something, so this is basically just the average thing I already believed. Probably each of those distributions has some kind of complicated shape, but who actually manages to keep the shape of their probability distribution in their head while reasoning? Not me.”

Once you’ve established that you ignore the bulk of the probability distribution, you don’t get to fall back on it when critiqued. But if Alexander doesn’t actually have a probability distribution, then plausibly one of my other critics might, and Cotra certainly does. Some people do the real thing, so let’s end this aside about the many who gesture vaguely at “probability distributions” without putting in the legwork to use one. If this method actually works, then we only need to pay attention to the few who follow through, and I’ll return to the main argument to address that. 

Does it work? Should we use their probability distributions to guide our actions, or put in the work to develop probability distributions of our own?

Suppose we ask an insurance company to give “death of Ben Landau-Taylor timelines”. They will be able to give their answer as a probability distribution, with strong reasons and actuarial tables in support of it. This can bear a lot of weight, and is therefore used as a guide to making consequential decisions—not just insurance pricing, but I’d also use this to evaluate e.g. whether I should go ahead with a risky surgery, and you bet your ass I’d “keep the shape of the probability distribution in my head while reasoning” for something like that. Or if we ask a physicist for “radioactive decay of a carbon-14 atom timelines”, they can give a probability distribution with even firmer justification, and so we can build very robust arguments on this foundation. This is what having a probability distribution looks like when people know things—which is rarer than I’d like, but great when you can get it.

Suppose we ask a well-calibrated general or historian for “end of the Russia-Ukraine war timelines” as a probability distribution.³ Most would answer based on their judgment and experience. A few might make a database of past wars and sample from that, or something. Whatever the approach, they’ll be able to give comprehensible reasons for their position, even if it won’t be as well-justified and widely-agreed-upon as an actuarial table. People like Ukrainian refugees or American arms manufacturers would do well to put some weight on a distribution like this, while maintaining substantial skepticism and uncertainty rather than taking the numbers 100% literally. This is what having a probability distribution looks like when people have informed plausible guesses, which is a very common situation.

Suppose we ask the world’s most renowned experts for timelines to peak global population. They can indeed give you a probability distribution, but the result won’t be very reliable at all—the world’s most celebrated experts have been getting this one embarrassingly wrong for two hundred years, from Thomas Malthus to Paul Ehrlich. Their successors today are now producing timelines with probabilistic prediction intervals showing when they expect the growth of the world population to turn negative.⁴ If this were done with care then arguably it might possibly be worth putting some weight on the result, but no matter how well you do it, this would be a completely different type of object from a carbon-14 decay table, even if both can be expressed as probability distributions. The arguments just aren’t there.

The timing of breakthrough technologies like AGI are even less amenable to quantification than the peak of world population. A lot less. Again, the critics I’m addressing don’t actually dispute that we have no good arguments for this, the only people who argued with this point were advancing (bad) arguments for specific short timelines. The few people who have any probability distributions at all give reasons which are extremely weak at best, if not outright refutable, or sometimes even explicitly deny the need to have a justification.

This is not what having a probability distribution looks like when people know things! This is not what having a probability distribution looks like when people have informed plausible guesses! This is just noise! If you put weight on it then the ground will give way under your feet! Or worse, it might be quicksand, sticking you to an unjustified—but legible!—nonsense answer that’s easy to think about yet unconnected to evidence or reality.

The world is not obligated to give you a probability distribution which is better or more informative than a resigned shrug. Sometimes we have justified views, and when we do, sometimes probabilities are a good way of expressing those views and the strength of our justification. Sometimes we don’t have justified views and can’t get them. Which sucks! I hate it! But slapping unjustified numbers on raw ignorance does not actually make you less ignorant.

---

[1] While I am arguing against several individual Rationalists here, this is certainly not the position of all Rationalists. Others have agreed with my post. In 2021 ur-Rationalist Eliezer Yudkowsky wrote:

“I feel like you should probably have nearer-term bold predictions if your model [of AGI timelines] is supposedly so solid, so concentrated as a flow of uncertainty, that it’s coming up to you and whispering numbers like “2050” even as the median of a broad distribution. I mean, if you have a model that can actually, like, calculate stuff like that, and is actually bound to the world as a truth.

If you are an aspiring Bayesian, perhaps, you may try to reckon your uncertainty into the form of a probability distribution … But if you are a wise aspiring Bayesian, you will admit that whatever probabilities you are using, they are, in a sense, intuitive, and you just don’t expect them to be all that good.

…

I have refrained from trying to translate my brain’s native intuitions about this into probabilities, for fear that my verbalized probabilities will be stupider than my intuitions if I try to put weight on them.”

Separately, “Against AGI Timelines” got a couple other Rationalist critics who do claim to have good arguments for short timelines. I’m not persuaded but they are at least not making the particular mistake that I’m arguing against here. ⮐

[2] It’s not a priori impossible that there could ever be a good argument for a strong claim about AGI timelines. I’ve never found one and I’ve looked pretty damn hard, but there are lots of things that I don’t know. However, if you want to make strong claims—and “I think AGI will probably (>80%) come in the next 10 years” is definitely a strong claim—then you need to have strong reasons. ⮐

[3] The Good Judgment Project will sell you their probability distribution on the subject. If I were making big decisions about the war then I would probably buy it, and use it as one of many inputs into my thinking. ⮐

[4] I’m sure every Rationalist can explain at a glance why the UN’s 95% confidence range here is hot garbage. Consider this a parable about the dangers of applying probabilistic mathwashing to locally-popular weakly-justified assumptions. ⮐

Some of my friends have strong views on how long it will be until AGI is created. The best arguments on the subject establish that creating a superintelligent AGI is possible, and that such an AGI would by default be “unfriendly”, which is a lot worse than it sounds. So far as speculative engineering goes, this is on relatively solid ground. It’s quite possible that as research continues, we’ll learn more about what sorts of intelligence are possible and discover some reason that an AGI can’t actually be built—such discoveries have happened before in the history of science and technology—but at this point, a discovery like that would be a surprise.

The loudest voices warning of AGI also make additional claims about when AGI is coming.¹ A large contingent argue for “short timelines”, i.e., for AGI in about 5-10 years. These claims are much shakier.

Of course, short timelines don’t follow automatically from AGI being possible. There is often a very long time between when someone figures out that a technology ought to work in principle, and when it is built in reality. After Leonardo da Vinci sketched a speculative flying machine based on aerodynamic principles similar to a modern helicopter, it took about 400 years before the first flight of a powered helicopter, and AGI could well be just as far from us. Since the discovery of nuclear physics and the construction of particle accelerators about a century ago, we have known in principle how to transmute lead into gold, but this has never actually been done. Establishing timelines to AGI requires additional arguments beyond its mere possibility, and the arguments advanced for particular timelines—long or short—are weak.

Whenever I bring this up, people like to switch to the topic of what to do about AI development. That’s not what I’m discussing here. For now I’m just arguing about what we know (or don’t know) about AI development. I plan to write about the implications for action in a future post.

1.

The most common explanation² I hear for short timelines (e.g. here) goes roughly like this: Because AI tech is getting better quickly, AGI will arrive soon. Now, software capabilities are certainly getting better, but the argument is clearly incomplete. To know when you’ll arrive somewhere, you have to know not just how fast you’re moving, but also the distance. A speed of 200 miles per hour might be impressive for a journey from Shanghai to Beijing (high-speed rail is a wonder) but it’s very slow for a journey from Earth orbit to the Moon. To be valid, this argument would also need an estimate of the distance to AGI, and no one has ever provided a good one.

Some people, like in the earlier example, essentially argue “The distance is short because I personally can’t think of obstacles”. This is unpersuasive (even ignoring the commenters responding with “Well I can think of obstacles”) because the creation of essentially every technology in the history of the world is replete with unforeseen obstacles which crop up when people try to actually build the designs they’ve imagined. This is most of the reason that engineering is even hard.

Somewhat more defensibly, I’ve heard many people argue for short timelines on the basis of expert intuition. Even if most AI experts shared this intuition—which they do not, there is nothing close to a consensus in the field—this is not a reliable guide to technological progress. An engineer’s intuition might be a pretty good guide when it comes to predicting incremental improvements, like efficiency gains in batteries or the cost of photovoltaic cells. When it comes to breakthrough technologies with new capabilities, however, the track record of expert intuition is dismal. The history of artificial intelligence specifically is famously littered with experts predicting major short-term breakthroughs based on optimistic intuition, followed by widespread disappointment when those promises aren’t met. The field’s own term for this, “AI winter”, is older than I am.

It’s worth a look at the 1972 Lighthill Report, which helped usher in the first AI winter half a century ago (emphasis added): 

“Some workers in the field freely admit that originally they had very naive ideas about the potentialities of intelligent robots, but claim to recognise now what sort of research is realistic. In these circumstances it might be thought appropriate to judge the field by what has actually been achieved than by comparison with early expectations. On the other hand, some such comparison is probably justified by the fact that in some quarters wild predictions regarding the future of robot development are still being made.

When able and respected scientists write in letters to the present author that AI, the major goal of computing science, represents another step in the general process of evolution; that possibilities in the nineteen-eighties include an all-purpose intelligence on a human-scale knowledge base; that awe-inspiring possibilities suggest themselves based on machine intelligence exceeding human intelligence by the year 2000; when such predictions are made in 1972 one may be wise to compare the predictions of the past against performance as well as considering prospects for the realisation of today’s predictions in the future.”

While there has been tremendous progress in software capabilities since the Lighthill Report was written, many of the experts’ “wild predictions” for the next 20-30 years have not yet come to pass after 50. The intuition of these “able and respected scientists” is not a good guide to the pace of progress towards intelligent software.

Attempts to aggregate these predictions, in the hopes that the wisdom of crowds can extract signal from the noise of  individual predictions, are worth even less. Garbage in, garbage out. There has been a great deal of research on what criteria must be met for forecasting aggregations to be useful, and as Karger, Atanasov, and Tetlock argue, predictions of events such as the arrival of AGI are a very long way from fulfilling them. “Forecasting tournaments are misaligned with the goal of producing actionable forecasts of existential risk”.

Some people argue for short AGI timelines on the basis of secret information. I hear this occasionally from Berkeley rationalists when I see them in person. I’m pretty sure this secret information is just private reports of unreleased chatbot prototypes before they’re publicly released about 2-6 weeks later.³ I sometimes get such reports myself, as does everyone else who’s friends with engineers working on chatbot projects, and it’s easy to imagine how a game of Telephone could exaggerate this into false rumors of a new argument for short timelines rather than just one more piece of evidence for the overdetermined “the rate of progress is substantial” argument.

2.

It’s worth a step back from AGI in particular to ask how well this type of speculation about future technology can ever work. Predicting the future is always hard. Predicting the future of technology is especially hard. There are lots of well-publicized, famous failures. Can this approach ever do better than chance?

When arguing for the validity of speculative engineering, short timeline proponents frequently point to the track record of the early 20^th century speculative engineers of spaceflight technology like Tsiolkovsky. This group has many very impressive successes—too many to be explained by a few lucky guesses. Before any of the technology could actually be built, they figured out a great deal of the important principles: fundamental constraints like the rocket equation, designs for spacecraft propulsion which were later used more-or-less according to the early speculative designs, etc. This does indeed prove that speculative engineering is a fruitful pursuit whose designs should be taken as serious possibilities.

However, the speculative engineers of spaceflight also produced many other possible designs which have not actually been built. According to everything we know about physics and engineering, it is perfectly feasible to build a moon base, or even an O’Neill cylinder. A space elevator should work as designed if the materials science challenges can be solved, and those challenges in turn have speculative solutions which fit the laws of physics. A mass driver should be able to launch payloads into orbit (smaller versions are even now being prototyped as naval guns). But just because one of these projects is possible under the laws of physics, it does not automatically follow that humanity will ever build one, much less that we will build one soon.

After the great advances in spaceflight of the mid-20^th century, most of the best futurists believed that progress would continue at a similar pace for generations to come. Many predicted moon bases, orbital space docks, and manned Mars missions by the early 2000s, followed by smooth progress to colonies on the moons of outer planets and city-sized orbital habitats. From our vantage today, none of this looks on track. Wernher von Braun would weep to see that since his death in 1977 we have advanced no further than Mars rovers, cheaper communication satellites, and somewhat larger manned orbital laboratories.

On the other hand, technological advances in some fields, such as materials science or agriculture, have continued steadily for generation on generation. Writers throughout the 1800s and 1900s spoke of the marvels that future progress would bring in these fields, and those expectations have mostly been realized or exceeded. If we could bring Henry Bessemer to see our alloys and plastics or Luther Burbank to see our crop yields, they would be thrilled to see achievements in line with our century-old hopes. There is no known law to tell which fields will stall out and which will continue forward.

If today’s AI prognosticators were sent back in time to the 1700s, would their “steam engine timelines” be any good? With the intellectual tools they use, they would certainly notice that steam pump technology was improving, and it’s plausible that their speculation might envision many of steam power’s eventual uses. But the intellectual tools they use to estimate the time to success—deference to prestigious theorists, listing of unsolved difficulties, the intuition of theorists and practitioners⁴—would have given the same “timelines” in 1710 as in 1770. These methods would not pick out the difference ahead of time between steam engines like Savery’s (1698) and Newcomen’s (1712), which ultimately proved to be niche curiosities of limited economic value, and the Watt steam engine (1776), which launched the biggest economic transformation since agriculture.

3.

Where does this leave us? While short AGI timelines may be popular in some circles, the arguments given are unsound. The strongest is the argument from expert intuition, and this one fails because expert intuition has an incredibly bad track record at predicting the time to breakthrough technology improvements.

This does not mean that AGI is definitely far away. Any argument for long timelines runs into the same problems as the arguments for short timelines. We simply are not in a position to know how far away AGI is. Can existing RLHF techniques with much more compute suffice to build a recursively self-improving agent which bootstraps to AGI? Is there some single breakthrough that a lone genius could make that would unlock AGI on existing machines? Does AGI require two dozen different paradigm-shifting insights in software and hardware which would take centuries to unlock, like da Vinci’s helicopter? Is AGI so fragile and difficult to create as to be effectively out of human reach? Many of these possibilities seem very unlikely, but none of them can be ruled out entirely. We just don’t know.

Addendum: Several people object that none of this presents a problem if you give timelines probabilistically rather than as a single date. See Probability Is Not A Substitute For Reasoning for my response.

---

[1] I’ve even heard a secondhand report of one-year timelines, as of February 2023. ⮐

[2] Okay, the most common explanation people give me for short timelines is that they’re deferring to subculture orthodoxy or to a handful of prestigious insiders. But this essay is about the arguments, and these are the arguments of the people they defer to. ⮐

[3] On the other hand, if the Berkeley AI alignment organizations had more substantial secret information or arguments guiding their strategy, I expect I would’ve heard it. They’ve told me a number of their organizations’ other secrets, sometimes deliberately, sometimes accidentally or casually, and on one occasion after I specifically warned my interlocutor that he was telling me sensitive information and should stop but he kept spilling the beans anyway. I very much doubt that they could keep a secret like this from me when I’m actually trying to learn it, if it were really determining their strategy. ⮐

[4] The theorists and practitioners of the 1700s didn’t predict accurate “steam engine timelines”. Watt himself thought his machine would be useful for pumping water out of mines more efficiently than the Newcomen engine, and did not expect its widespread revolutionary use. ⮐

I’ve been behind on posting my work to this site. Some recent material:

New Industries Come From Crazy People at Palladium. This is a look at how different cultures interface with the antisocial weirdos like Thomas Edison and Steve Jobs who drive industrial breakthroughs. Most societies don’t tolerate them. The ones that do, drive economic progress.

Narratives Podcast with Will and David Jarvis. We talk about some of the ideas in the Palladium essay, how state capacity has changed over the last century, why I’m optimistic about America’s future, and more.

I recently concluded an in-depth case study on the machine tool industry with Oberon Dixon-Luinenburg. We just published two summaries of our research.

1. Machine Tools: A Case Study In Advanced Manufacturing, with Bismarck Analysis. This report covers where machine tools are built, why they’re built in those places, and what this tells us about industrial policy more broadly.
2. How State Capacity Drives Industrialization, with Palladium. This article describes the history of industrialization, and why it doesn’t happen without guidance from the state.

Please do comment or email me your thoughts.

When people talk about a “market economy”, what exactly does that mean? You might assume it means a society where all or most economic decisions are made through markets. But this does not describe our own economy, or anything else that gets called a “market economy”. Most economic decisions are made inside particular organizations, which make their internal decisions according to hierarchical structures and individual judgment, taking markets into account as one factor among many. If two workers both want to avoid a Monday morning shift, or two department heads are arguing over hiring policy, or two engineers are debating which design to adopt, no one expects the people involved to start setting prices and bidding against each other like they would in a market. And yet markets are nevertheless crucial to our economic organization, in ways that are notably different from non-“market economies”. How, then, does the market fit in?

What markets make possible is smooth interaction between independent units. Organizations have hierarchical control over their own assets, including very complex social processes like medical schools or aircraft control towers, but they also need to coordinate with other parts of society external to the organization itself, like customers and suppliers. Markets make this easy and reliable. The market is a thin layer of crucial but simple social technology that lets an organization ignore the deep complexity of the people and organizations it interacts with. This is helpful because the market interface an organization presents to the outside world is so much simpler than the non-market-based coordination it uses internally.

In a way, the market is like the docking module between the Apollo and Soyuz rockets. It’s a crucial component, and without it the entire system couldn’t possibly function as a unified whole, but it’s still a relatively small fraction of the system’s mass and complexity. Understanding how it works won’t tell you all that much about how the Soyuz itself works.

Of course organizations don’t always relate to each other according to the rules of the “free market”. Perfect markets don’t and can’t exist, and large swathes of the current economy don’t even approximate markets (see e.g. Patrick McKenzie noting how success in venture capital is determined by “access”). Nevertheless, such exceptions are uncommon. Most of our economy is mostly market-based. This means that organizations and individuals can usually use the market as a reasonably accurate guide to how they should interact with the rest of society—and nothing more.

Normal Angell’s The Great Illusion: A Study of the Relation of Military Power to National Advantage, published in 1910, is the author’s attempt to prevent World War I. At the time, tensions between Germany and Britain were running high, inflamed in part by rhetoric on both sides claiming that a nation’s prosperity—and even its ability to feed its population—depended on maintaining military supremacy. Angell dismantles those arguments, desperately trying to show that there is no economic reason for war before the shells start flying. The book is a strong analysis of the relationship between military power and wealth, and while it obviously did not keep Britain from going to war with Germany, Angell’s model predicted the war’s economic consequences with Cassandra-like accuracy. As a result, Angell was awarded the Nobel Peace Prize in 1933. However, Angell’s model utterly fails to describe the economic consequences of World War II, for reasons which expose the limits and assumptions of Angell’s theory.

Angell’s foes are a host of reputable journalists and academics who claim that trade and wealth are ephemeral without the backing of a strong military. Any nation that lacks a strong military, they argue, will lose the commercial war of all against all to nations willing to throw their weight around. Because international trade could be halted by blockade, or wealth could be seized by a conqueror, such events inevitably will happen to Britain unless she maintains military superiority over other nations in general and Germany in particular. (The warmongers Angell quotes speak with exactly the mix of bombast and pique that I’ve seen in cheerleaders for American wars like Iraq, Libya, and Vietnam, along with the preference for generalities and hypotheticals over concrete cases that is the hallmark of political hacks everywhere. Anyone born after 1995 or so should read Chapter 2 just to get a feel for the immortal rhetoric of jingoism.)

Angell’s main counterargument is that, as the world’s economy has financialized, wealth has become harder to seize. Rather than gold and jewels which can be carted away, wealth has come to consist of companies, shares, debts, and a host of complex financial instruments. The value of these instruments is tied up in the arcane web of international obligations and business ties, and in ordinary investors’ faith in market mechanisms. While it’s certainly possible to seize these financial instruments by force or threat of force, Angell argues, this would cause local market crashes, which would spread and become international market crashes, and the “winners” would lose more money in the crash than they gained in the seizure. So, while a conqueror can certainly destroy wealth, Angell denies that they can gain wealth by force.

Angell also notes that the ongoing prosperity and economic security of small states like Switzerland, Norway, and the Netherlands is a crushing counterexample to the jingoist economic model. These states were doing well in Angell’s time, and the century since then has only reinforced his argument that their people’s wealth is not threatened by their military inferiority, whether from raw plunder and extortion or from more complex international trade deals subtly weighted by military might. Since the book was published, some of these countries have in fact been conquered and occupied, and while this was important politically, it has had relatively little impact economically.

The aftermath of World War I would vindicate Angell’s claims more broadly. The Treaty of Versailles imposed heavy reparations on Germany, the main result of which was hyperinflation and the temporary ruin of the German economy. There was no corresponding boom in the victorious nations, and no British or French profiteers getting rich off of markets and businesses that had once been German. The French occupation of the Ruhr led to strikes and the shutdown of more German industry. Little wealth was transferred and much was destroyed. (Edit 2023: There were also some French chemical firms that extorted and reverse-engineered intellectual property from German factories in the occupied Ruhr, establishing French production of ammonia and of improved dyes. I expect similar behavior in other advanced industries. But this was relatively minor at the time.) Many in France were primarily concerned with weakening Germany and so were content with this situation, but American diplomats and financiers tried to prevent the crash.

However, it only happened this way because the victorious nations limited their confiscations to the realm of finance. Note that Angell’s analysis focuses on financial wealth (e.g. gold, money, stocks) and not on the factors of production (e.g. farmland, labor, machine tools).

Angell is correct in noting that financial wealth became much harder to seize after the value of stocks and bonds eclipsed the value of gold and silver. However, military seizure of financial wealth has never been very important in determining the wealth of nations, even in raiding cultures like the Vikings. At most it allowed a few personal fortunes to be made and personal legends to be built. This sort of plunder has often been popular among the sort of risk-seeking young men who today dream of becoming star athletes or founding billion-dollar startups, but it has never been the economic foundation of a society.

The factors of production are a very different story. Military seizure of the factors of production has always been rare and difficult, and success at this task can reshape entire regions. Before industrialization, by far the most important factors of production were productive land and semiskilled labor to work it, so seizing them could only mean settling land (e.g. the Volkerwanderung or the colonization of America), capturing masses of slaves and integrating them into one’s social system (e.g. the Comanche or the Roman Republic), or conquering a productive agricultural society, leaving it mostly intact, and inserting one’s own people into the elite positions (e.g. the Normans or the Manchus). All of these approaches require far more effort and skill than simple plunder.

Since industrialization, the most important factors of production have been heavy machinery, skilled labor, and groundbreaking technologists. World War II shows that, when they disregard claims of legal ownership, modern regimes are no worse at seizing the factors of production than medieval regimes. The Soviets were notorious for shipping heavy machinery out of captured areas, and sending skilled technicians and engineers to work in slave labor camps. The Nazis transferred factories and shops from Slavs and Jews to ethnic Germans. (The best-known beneficiary of this policy was Oskar Schindler, who received a factory seized from Polish Jews and spent most of its profits on bribing the Nazi death machine to spare the Jewish slaves who worked his plundered machines.) Contrary to Angell’s fears, German financial markets did quite well when foreign capital was being seized. Moreover, Schindler and his peers produced vast quantities of useful goods, and that is the true test of economic value, not piles of gold or numbers in a stock exchange. As the Nazi government collapsed, American industrialists looted German patent files, setting research and development forward by years without concern for intellectual property. More importantly, the American military acquired Werner von Braun and other German rocket scientists, who would design the first ICBMs and the rockets that took us to the Moon. This transferred more wealth than all the plundered machinery put together.

Seizing the factors of production at scale is always a bloody affair. It requires mass graves, tearing parents away from children, kicking in doors at 3 AM, and shooting 13-year-old saboteurs in the back of the head. Most people hate doing this, which is a big part of why it’s so rare. Even the architects of the Treaty of Versailles, who are not known for their soft hearts, refused to contemplate something so extreme. But sometimes it happens anyway.

The Franklin Effect is a concept in pop psychology which asserts that, if Alice does a favor for Bob, then Alice will be more inclined to do more things for Bob in the future. While I’ve observed a real effect like this, I think it’s different from the usual story in subtle but important ways.

The Franklin Effect takes its name from a passage in Benjamin Franklin’s autobiography:

I therefore did not like the opposition of this new member, who was a gentleman of fortune and education, with talents that were likely to give him, in time, great influence in the House, which, indeed, afterwards happened. I did not, however, aim at gaining his favour by paying any servile respect to him, but, after some time, took this other method. Having heard that he had in his library a certain very scarce and curious book, I wrote a note to him, expressing my desire of perusing that book, and requesting he would do me the favour of lending it to me for a few days. He sent it immediately, and I return’d it in about a week with another note, expressing strongly my sense of the favour. When we next met in the House, he spoke to me (which he had never done before), and with great civility; and he ever after manifested a readiness to serve me on all occasions, so that we became great friends, and our friendship continued to his death. This is another instance of the truth of an old maxim I had learned, which says, “He that has once done you a kindness will be more ready to do you another, than he whom you yourself have obliged.” And it shows how much more profitable it is prudently to remove, than to resent, return, and continue inimical proceedings.

The usual explanation for this effect is that it works by changing the giver’s self-concept. Once you do someone a favor, the story goes, you automatically think of yourself as the recipient’s friend, and you’ll be more likely to do friendly things for them going forward.

By now I’ve given and received quite a few favors, and my experience doesn’t quite match this explanation.

As a recipient of favors, I’ve found that it matters a lot what I do with what I’m given. When I make good use of the favor and demonstrate this to the giver, the effect works more or less as described, albeit more weakly than one would assume from a naive reading of Franklin’s text. When I do this *and also* make myself helpful to the giver, this is often the start a deep, ongoing relationship. On the other hand, when I don’t do much with the favor (or on one occasion when the favor was useful but I stupidly neglected to demonstrate this to the giver), the effect is small or negative.

As a giver of favors, I find that I feel very differently depending on whether I think it led anywhere. I do favors for people when I want to help them for whatever reason, and I watch to see which favors result in actual help vs. which seem to go nowhere. (Relatedly, I’ll often give an opportunity to someone I don’t know well as a way of evaluating what they can do.) If it helped, I’m inclined to do more for the person, because I feel confident my future efforts will also be helpful. If not, then I become skeptical of my ability to help the person. I’m wary of pouring too much effort into a project that’s unlikely to work, and I’ll usually cut my losses after two or three failures if there are no other considerations. From watching others, I think this is a very common pattern, even if I’m more explicit about it than most.

Here’s what I think is actually going on: Requesting a favor from a stranger or acquaintance has two important components. There’s a request for charity, and also an overture towards partnership. People often want to dispense limited charity on the basis of magnanimity, civic responsibility, ego, or some such. People also want to partner with good allies or useful coalition members. The Franklin Effect relies on the overture towards partnership. The charity component can help facilitate the early steps of the process, but is otherwise irrelevant as far as turning one-off favors into ongoing relationships.

When you request a favor, some people will consciously evaluate your suitability as a coalition partner, and many more will do so subconsciously. Everyone will notice, consciously or subconsciously, whether you subsequently act like a good coalition partner. This includes social things like showing due appreciation, and also material things like doing favors in return. The latter will work better for people who have real value to offer each other, like Ben Franklin and his rival-turned-friend in the Pennsylvania Assembly.

I sometimes see my friends try to use the self-concept model of the Franklin Effect to get support from influential patrons. They’ll try to get a favor or endorsement from a big name, not because the favor is useful in itself, but because it represents a step forward in the ongoing project of catching the patron’s eye. I rarely see this turn into anything lasting in the way my friend wants. Because they lack the fundamentals that would allow them to become a good friend or ally, my friend can only ask for charity, which by itself is not a foundation for an ongoing relationship.

All this is to say that the Franklin Effect is not a hack to beguile people into helping you. Rather, it is an audition that gives you a chance to demonstrate your worth.

At the opening of the American Revolution, the state militias were a formidable force. They went toe-to-toe with the British army, then the most powerful in the world, and came off reasonably well. However, by the time of the American Civil War, the militias of those same states had become a joke. Their first serious battle was a litany of logistical and tactical blunders, where “Stonewall” Jackson earned his famous nickname because his men did not immediately run away.

This is unsurprising. The militias of 1776 were sharpened by frequent conflicts against the Native Americans, and had learned from both British officers and native allies. The militias of 1861 were far from the frontier and were led by local politicians and businessmen with no military experience to speak of. No wonder one was more competent than the other. A militia (or university, or retail chain, or…) often functions the way it does because of skilled people in key roles. If those people leave or lose motivation or get reassigned, then the organization will become very different. Whether it’s a restaurant losing its manager, or a militia losing its veteran commander, the dynamics are the same.

If new skilled people arise or gain power, this will also change the organization. After Paul Graham retired and Sam Altman took over Y Combinator, the company expanded from its previous role as a startup incubator and investor. It is now also a central industry hub, as well as an interface between the startup scene and adjacent sectors. This is mostly due to Altman’s skill at networking and making durable alliances.

Skill isn’t the only dimension on which organizations can change. They can also change their purpose. For example, in 1800 the United States military was built for frontier defense, but by 1900 it was built to project power externally. Also, in 1963 the counterculture movement was about reforming a corrupt society via moral and legal pressure, but by 1973 it was about seceding from a corrupt society via sex, drugs, and rock and roll. If an organization’s mission changes drastically, then it may lose the skills needed to carry out the previous mission as the old generation dies off.

All this may seem like a banal point, and in a way it is. Organizations change. Everyone knows this, at least in the abstract. Nevertheless I see a persistent error where many, many people make predictions based on stories of an organization’s previous form in a way that is utterly incompatible with the realities of its current form.

Among the most striking examples of this phenomenon is labor unions. Contemporary unions rest their claim to auctoritas on the victories of unions from the late 1800s and early 1900s. At the time, unions fought—literally fought, with truncheons and bombs—against factory owners who used private mercenaries and sometimes public police forces to violently suppress any independent power bases among the working class. The unions succeeded, establishing an organized power base and using this to negotiate for higher standards of living, then enshrining their victories (the 40-hour 5-day workweek, the concepts of minimum wage and overtime pay, etc) in law and custom.

Today’s unions present their activities as the continuation of this struggle. They claim that the mission of “get higher wages and better working conditions for our members” has remained constant, and that any differences in their tactics are a sensible response to changing circumstances. There’s a narrow sense in which it’s true that both groups would endorse that mission as a subset of their goals, but from the broader perspective of trying to predict how an organization will affect things we care about—which is the only reason it’s worthwhile to bother with this sort of analysis—this is as spectacularly unhelpful as drawing structural parallels between the engineers who designed the Minuteman missile on the one hand, and the actual Minutemen who Paul Revere rallied on the other, because there’s a line of institutional descent between them that maintained the mission “defend America”.

When looking at the structural factors that determine an organization’s capabilities and its effects on society, contemporary unions are nearly as different from their historical predecessors as the military-industrial complex is different from a citizen militia.  Historical unions fought the state; contemporary unions are a branch of the state. The main negotiating tool of historical unions was illegal or legally-gray strikes; the main negotiating tool of contemporary unions is the bureaucratic application of legal privileges. Historical unions were mostly made up of the industrial proletariat, and fought on their behalf; contemporary unions are mostly made up of government service workers, and fight on their behalf. Historical unions were ideologically committed to expanding their membership into new areas in order to carry forward the class struggle; contemporary unions are uninterested in class struggle because their leaders are no longer working class, and are uninterested in dramatic expansion because they lack the ideologue’s soaring ambition.

This is all fairly abstract, so let me say it more directly. If the John L. Lewises of the world were around today, they would be organizing workers to illegally seize warehouses from Walmart and Amazon. From their perspective, capital still controls the world, and labor is less organized than in the unions’ heydey, so similar action would be justified. Their self-styled heirs would never do anything remotely this extreme. Even if they wanted to, they don’t know how. This means that their role in society is utterly different. It may be that contemporary unions are a force for good, but if so then it’s good as conceived of by Teddy Roosevelt rather than by Emma Goldman. The only thing today’s unions have in common with their historical predecessors is a brand.

This is an unusually stark case, but similar situations are common. Organizational continuity does not come about because of some metaphysical property of the organization itself. It does not come merely from people’s duty to the stated mission, or from adherence to procedures and org charts, or from new members automatically assimilating to the culture. Rather, continuity comes from an organization’s current members using selection, training, and internal structure to shape the organization’s future membership. There is continuity only to the extent that these mechanisms produce it. You can evaluate this by comparing an organization’s past and present activities, or its past and present people and culture, or even by looking at the mechanisms of continuity themselves.

While many organizations live up to their old ideals, many others will mislead you by claiming to be more like their predecessors than they actually are. If you don’t want to be fooled, you need to be able to look beyond an organization’s brand—the restaurant down the street, the AFL-CIO, the Catholic Church, the company offering you a job, the New York Times, or whatever else—and see its people and institutions as they exist today.

The historical consensus holds that the invention of the stirrup was a major development in military history. By permitting the horseman to keep his seat, the simplified story goes, the stirrup changed the dominant strategy from the infantry-based armies of antiquity to the shock cavalry-based armies that came to dominate in the middle ages.

This story seems to make sense. The change in the composition of European armies is real and needs to be explained. (Infantry remained the numerical majority of most armies, but heavy cavalry became more important in determining the outcome of battles.) Horsemen without stirrups used different equipment in different ways than the stirrup-using knights we’re familiar with. The cavalry charge against massed infantry is almost unheard of in antiquity but becomes an extremely important tactic from the early middle ages until well after the ubiquity of firearms.

However, there is no historical consensus on when the stirrup became important in Europe. I’ve seen serious claims ranging from the late 300s to the late 700s. There’s sharp disagreement over very basic claims, like “Was the Battle of Adrianople a triumph of cavalry over infantry?” or “Did the Carolingian military use stirrups?” (I haven’t checked whether these questions were resolved by recent archeological work, but if the answers weren’t obvious 40 years ago, that’s still a notable fact.) The history of the stirrup before it reached Europe, e.g. in India or central Asia, is no clearer.

This is super weird. If the stirrup was such a huge deal, shouldn’t we be able to see its effects? If a historian in the year 3000 were trying to date the advent of the machine gun, and only had fragments of secondary sources and doubtful archeological scraps, it would still be possible because the machine gun so greatly transformed strategy, tactics, and the experience of individual soldiers. (The American Civil War is the only case I can think of where a smart scholar might get the wrong answer.) This is what we see for other massive shifts in historical weapons, such as chariots, castles, and artillery. If the stirrup were anywhere near this important, its effects should be similarly visible.

I’ve read all these historians arguing about the minutiae of manuscripts and archeological finds to set dates on when the stirrup was used where, but if their basic claim about the importance of the stirrup is true, then there should be much simpler avenues to answering the question.

At this point, I’m inclined to think the stirrup was not as overpowering as is commonly asserted. Important, yes, but important on the scale of chainmail or the rifled barrel, not on the level of the phalanx or the nuclear bomb. Not important enough to explain the transition from armies dominated by infantry to armies dominated by cavalry. If it were, its history would be more apparent.

If true, this raises two questions. The first, why so many historians have overstated its importance, is relatively easy to answer. For one thing, contemporary prejudices favor explaining large-scale trends as the natural consequence of technological development. More importantly, historians are like anyone else in that they are biased towards simple and compelling explanations for things. The story of the stirrup transforming combat has enough truth to it to lay the foundation for such a narrative. It fits very well from a purely local perspective. In contrast, broad sociological outside-view checks like the one I’m running here seem, if not rare, then at least uncommon.

The more difficult question is why Europe transitioned from infantry-based armies to cavalry-based armies, if not for stirrups. I’m not sure. It could be a combination of technological factors: larger horses, improved saddles, better armorsmithing, horseshoes, and the temporary loss of the composite bow, together with stirrups, producing a combined effect greater than the sum of its parts. It’s possible, but I don’t trust this type of explanation. Strategic considerations are usually Pareto-distributed in importance, and one major factor tends to overwhelm many medium-size factors.

It could be a matter of economic and social organization: the sharp division between landholding knights extracting wealth from their tenants on the one hand, and peasant farmers with little capital on the other, led to a combination of arms that was perhaps inefficient from a purely military standpoint but crucial with regard to internal coherence, thus leading Carolingian-style feudalism to succeed and spread in spite of some necessary overemphasis on heavy cavalry. This strikes me as plausible but far from certain. The institutional and cultural prominence of knighthood in Europe is consistent with this story, at the very least.

A more exotic version of the prior hypothesis is that Europe transitioned to cavalry not because its cavalry was strong, but because its infantry was weak. If feudalism made it institutionally and ideologically difficult to raise large masses of competent, well-equipped infantry, then perhaps this explains the shift.

I’m not confident in any of these explanations. The more I look into this, the more I think the dominance of cavalry in medieval Europe is a mystery that still needs to be explained.

Edit: Stephen Morillo can explain the dominance of cavalry. See his comment and full article. Turns out the “Infantry was weak” hypothesis is correct.