Against AGI Timelines

Some of my friends have strong views on how long it will be until AGI is created. The best arguments on the subject establish that creating a superintelligent AGI is possible, and that such an AGI would by default be “unfriendly”, which is a lot worse than it sounds. So far as speculative engineering goes, this is on relatively solid ground. It’s quite possible that as research continues, we’ll learn more about what sorts of intelligence are possible and discover some reason that an AGI can’t actually be built—such discoveries have happened before in the history of science and technology—but at this point, a discovery like that would be a surprise.

The loudest voices warning of AGI also make additional claims about when AGI is coming. A large contingent argue for “short timelines”, i.e., for AGI in about 5-10 years.¹ These claims are much shakier.

Of course, short timelines don’t follow automatically from AGI being possible. There is often a very long time between when someone figures out that a technology ought to work in principle, and when it is built in reality. After Leonardo da Vinci sketched a speculative flying machine based on aerodynamic principles similar to a modern helicopter, it took about 400 years before the first flight of a powered helicopter, and AGI could well be just as far from us. Since the discovery of nuclear physics and the construction of particle accelerators about a century ago, we have known in principle how to transmute lead into gold, but this has never actually been done. Establishing timelines to AGI requires additional arguments beyond its mere possibility, and the arguments advanced for particular timelines—long or short—are weak.

Whenever I bring this up, people like to switch to the topic of what to do about AI development. That’s not what I’m discussing here. For now I’m just arguing about what we know (or don’t know) about AI development. I plan to write about the implications for action in a future post.

1.

The most common explanation² I hear for short timelines (e.g. here) goes roughly like this: Because AI tech is getting better quickly, AGI will arrive soon. Now, software capabilities are certainly getting better, but the argument is clearly incomplete. To know when you’ll arrive somewhere, you have to know not just how fast you’re moving, but also the distance. A speed of 200 miles per hour might be impressive for a journey from Shanghai to Beijing (high-speed rail is a wonder) but it’s very slow for a journey from Earth orbit to the Moon. To be valid, this argument would also need an estimate of the distance to AGI, and no one has ever provided a good one.

Some people, like in the earlier example, essentially argue “The distance is short because I personally can’t think of obstacles”. This is unpersuasive (even ignoring the commenters responding with “Well I can think of obstacles”) because the creation of essentially every technology in the history of the world is replete with unforeseen obstacles which crop up when people try to actually build the designs they’ve imagined. This is most of the reason that engineering is even hard.

Somewhat more defensibly, I’ve heard many people argue for short timelines on the basis of expert intuition. Even if most AI experts shared this intuition—which they do not, there is nothing close to a consensus in the field—this is not a reliable guide to technological progress. An engineer’s intuition might be a pretty good guide when it comes to predicting incremental improvements, like efficiency gains in batteries or the cost of photovoltaic cells. When it comes to breakthrough technologies with new capabilities, however, the track record of expert intuition is dismal. The history of artificial intelligence specifically is famously littered with experts predicting major short-term breakthroughs based on optimistic intuition, followed by widespread disappointment when those promises aren’t met. The field’s own term for this, “AI winter”, is older than I am.

It’s worth a look at the 1972 Lighthill Report, which helped usher in the first AI winter half a century ago (emphasis added): 

“Some workers in the field freely admit that originally they had very naive ideas about the potentialities of intelligent robots, but claim to recognise now what sort of research is realistic. In these circumstances it might be thought appropriate to judge the field by what has actually been achieved than by comparison with early expectations. On the other hand, some such comparison is probably justified by the fact that in some quarters wild predictions regarding the future of robot development are still being made.

When able and respected scientists write in letters to the present author that AI, the major goal of computing science, represents another step in the general process of evolution; that possibilities in the nineteen-eighties include an all-purpose intelligence on a human-scale knowledge base; that awe-inspiring possibilities suggest themselves based on machine intelligence exceeding human intelligence by the year 2000; when such predictions are made in 1972 one may be wise to compare the predictions of the past against performance as well as considering prospects for the realisation of today’s predictions in the future.”

While there has been tremendous progress in software capabilities since the Lighthill Report was written, many of the experts’ “wild predictions” for the next 20-30 years have not yet come to pass after 50. The intuition of these “able and respected scientists” is not a good guide to the pace of progress towards intelligent software.

Attempts to aggregate these predictions, in the hopes that the wisdom of crowds can extract signal from the noise of  individual predictions, are worth even less. Garbage in, garbage out. There has been a great deal of research on what criteria must be met for forecasting aggregations to be useful, and as Karger, Atanasov, and Tetlock argue, predictions of events such as the arrival of AGI are a very long way from fulfilling them. “Forecasting tournaments are misaligned with the goal of producing actionable forecasts of existential risk”.

Some people argue for short AGI timelines on the basis of secret information. I hear this occasionally from Berkeley rationalists when I see them in person. I’m pretty sure this secret information is just private reports of unreleased chatbot prototypes before they’re publicly released about 2-6 weeks later.³ I sometimes get such reports myself, as does everyone else who’s friends with engineers working on chatbot projects, and it’s easy to imagine how a game of Telephone could exaggerate this into false rumors of a new argument for short timelines rather than just one more piece of evidence for the overdetermined “the rate of progress is substantial” argument.

Edit: Twitter commenter Interstice reminds me of an additional argument, “that intelligence is mostly a matter of compute and that we should expect AGI soon since we are approaching compute levels comparable to the human brain”, and links Karnofsky’s argument. Yudkowsky’s refutation of this approach is correct.

2.

It’s worth a step back from AGI in particular to ask how well this type of speculation about future technology can ever work. Predicting the future is always hard. Predicting the future of technology is especially hard. There are lots of well-publicized, famous failures. Can this approach ever do better than chance?

When arguing for the validity of speculative engineering, short timeline proponents frequently point to the track record of the early 20^th century speculative engineers of spaceflight technology like Tsiolkovsky. This group has many very impressive successes—too many to be explained by a few lucky guesses. Before any of the technology could actually be built, they figured out a great deal of the important principles: fundamental constraints like the rocket equation, designs for spacecraft propulsion which were later used more-or-less according to the early speculative designs, etc. This does indeed prove that speculative engineering is a fruitful pursuit whose designs should be taken as serious possibilities.

However, the speculative engineers of spaceflight also produced many other possible designs which have not actually been built. According to everything we know about physics and engineering, it is perfectly feasible to build a moon base, or even an O’Neill cylinder. A space elevator should work as designed if the materials science challenges can be solved, and those challenges in turn have speculative solutions which fit the laws of physics. A mass driver should be able to launch payloads into orbit (smaller versions are even now being prototyped as naval guns). But just because one of these projects is possible under the laws of physics, it does not automatically follow that humanity will ever build one, much less that we will build one soon.

After the great advances in spaceflight of the mid-20^th century, most of the best futurists believed that progress would continue at a similar pace for generations to come. Many predicted moon bases, orbital space docks, and manned Mars missions by the early 2000s, followed by smooth progress to colonies on the moons of outer planets and city-sized orbital habitats. From our vantage today, none of this looks on track. Wernher von Braun would weep to see that since his death in 1977 we have advanced no further than Mars rovers, cheaper communication satellites, and somewhat larger manned orbital laboratories.

On the other hand, technological advances in some fields, such as materials science or agriculture, have continued steadily for generation on generation. Writers throughout the 1800s and 1900s spoke of the marvels that future progress would bring in these fields, and those expectations have mostly been realized or exceeded. If we could bring Henry Bessemer to see our alloys and plastics or Luther Burbank to see our crop yields, they would be thrilled to see achievements in line with our century-old hopes. There is no known law to tell which fields will stall out and which will continue forward.

If today’s AI prognosticators were sent back in time to the 1700s, would their “steam engine timelines” be any good? With the intellectual tools they use, they would certainly notice that steam pump technology was improving, and it’s plausible that their speculation might envision many of steam power’s eventual uses. But the intellectual tools they use to estimate the time to success—deference to prestigious theorists, listing of unsolved difficulties, the intuition of theorists and practitioners⁴—would have given the same “timelines” in 1710 as in 1770. These methods would not pick out the difference ahead of time between steam engines like Savery’s (1698) and Newcomen’s (1712), which ultimately proved to be niche curiosities of limited economic value, and the Watt steam engine (1776), which launched the biggest economic transformation since agriculture.

3.

Where does this leave us? While short AGI timelines may be popular in some circles, the arguments given are unsound. The strongest is the argument from expert intuition, and this one fails because expert intuition has an incredibly bad track record at predicting the time to breakthrough technology improvements.

This does not mean that AGI is definitely far away. Any argument for long timelines runs into the same problems as the arguments for short timelines. We simply are not in a position to know how far away AGI is. Can existing RLHF techniques with much more compute suffice to build a recursively self-improving agent which bootstraps to AGI? Is there some single breakthrough that a lone genius could make that would unlock AGI on existing machines? Does AGI require two dozen different paradigm-shifting insights in software and hardware which would take centuries to unlock, like da Vinci’s helicopter? Is AGI so fragile and difficult to create as to be effectively out of human reach? Many of these possibilities seem very unlikely, but none of them can be ruled out entirely. We just don’t know.

Addendum: Several people object that none of this presents a problem if you give timelines probabilistically rather than as a single date. See Probability Is Not A Substitute For Reasoning for my response.

This essay was originally posted on March 12, 2023.

1. I’ve even heard a secondhand report of one-year timelines, in February 2023.

2. Okay, the most common explanation people give me for short timelines is that they’re deferring to subculture orthodoxy or to a handful of prestigious insiders. But this essay is about the arguments, and these are the arguments of the people they defer to.

3. On the other hand, if the Berkeley AI alignment organizations had more substantial secret information or arguments guiding their strategy, I expect I would’ve heard it. They’ve told me a number of their organizations’ other secrets, sometimes deliberately, sometimes accidentally or casually, and on one occasion after I specifically warned my interlocutor that he was telling me sensitive information and should stop but he kept spilling the beans anyway. I very much doubt that they could keep a secret like this from me when I’m actually trying to learn it, if it were really determining their strategy.

4. The theorists and practitioners of the 1700s didn’t predict accurate “steam engine timelines”. Watt himself thought his machine would be useful for pumping water out of mines more efficiently than the Newcomen engine, and did not expect its widespread revolutionary use.