Recursive Self-Improvement
While we're on the subject of #AI scepticism, I don't see any immediate reason why the hot new thing of "Recursive Self Improvement" (which the labs are definitely experimenting with) necessarily leads to another "exponential" increase in anything except GPU utilisation costs.
They're just automating the existing "try semi-randomly tweaking a whole bunch of shit" process, which has hitherto been done by the "researchers". The neural net field is essentially built on trial and error, often via a process known in the biz as "Graduate Student Descent" -- which is a joke about getting the junior monkey staff to do something not unlike the iterative Gradient Descent exploration that the network training algorithms do under the hood.
If you automate that, you might find good "valleys" in the permutation landscape a bit more quickly/often than if you did it purely with humans, but it does not magically guarantee huge qualitative gains, and may in fact only discover quite modest ones now that the random number mine has been increasingly well excavated and the bottom is now being scraped.
Furthermore, all of these training algorithms need a measurable objective to chase, which is why they're quite good at programming (the tests either pass or they don't) and, latterly, mathematics (the chain of deductions from the axioms is either mechanistically valid or it isn't), and there is simply no magic objective function for "awesome sci-fi super intelligence that is entirely beyond our ken (and is also evil and/or stupid enough to want to turn us into fucking paperclips)".
One could quibble with some of the details above, but my guess is that RSI will make things a bit better, here and there, but will not provide a massive leap in capabilities. In fact, I suspect (as do some actual experts such as Yann LeCun), that there are limits to how far an LLM substrate can really get, and that some different kind of breakthrough approach will be required to produce the sort of intelligence that more plausibly "understands" the Real World and doesn't keep making really dumb mistakes, like some kind of autistic savant that thinks it is "an excellent driver" (when it fucking well isn't).
