I think it's worth observing that at the time of Nick Bostrom's 2014 book Superintelligence the success of NN algorithms had not yet broken out in the form of Google's DeepDream. Eliezer Yudkowsky claims this as a success i.e. "progress in GAI has occurred even faster than I have anticipated" but I think it goes to your point that making bets on technology is not reliable. Even after we narrowed it down to NN (or appear to have!), the impact of just scaling up was not appreciated until OpenAI's ChatGPT.
So what it will take to make AI general, what it will take to overcome data limitations, what it will take to make AI agentic, and what it will take to control it I think are all bets that I don't think we can make reliably.
I think it's worth observing that at the time of Nick Bostrom's 2014 book Superintelligence the success of NN algorithms had not yet broken out in the form of Google's DeepDream. Eliezer Yudkowsky claims this as a success i.e. "progress in GAI has occurred even faster than I have anticipated" but I think it goes to your point that making bets on technology is not reliable. Even after we narrowed it down to NN (or appear to have!), the impact of just scaling up was not appreciated until OpenAI's ChatGPT.
So what it will take to make AI general, what it will take to overcome data limitations, what it will take to make AI agentic, and what it will take to control it I think are all bets that I don't think we can make reliably.