Many things that look exponential originally turn out to actually be sigmoidal.
I consider the start of this wave of AI to be approximately the 2017 Google transformer paper and yet transformers didn't really have enough datapoints to look exponential until GPT 3 in 2022.
The following is purely speculation for fun and sparking light-hearted conversation:
My gut feeling is that this generation of models transitioned out of the part of the sigmoid that looks roughly exponential after the introduction of reasoning models.
My prediction is that tranformer-based models will start to enter the phase that asymptotes to flatline in 1-2 years.
I leave open the possibility for a different form of model to emerge that is exponential but I don't believe transformers to be right now.
I thought the prediction was that the scaling of LLMs making them better would plateau, not that all advancement would stop? And that has pretty much happened as all the advancements over the last year or more have been architectural, not from scaling up.
You say that, but to me they seem roughly the same as they've been for a good while. Wildly impressive technology, very useful, but also clearly and confidently incorrect a lot. Most of the improvement seems to have come from other avenues - search engine integration, image processing (still blows my mind every time I send a screenshot to a LLM and it gets it) and stuff like that.
Sure maybe they do better in some benchmarks, but to me the experience of using LLMs is and has been limited by their tendency to be confidently incorrect which betrays their illusion of intelligence as well as their usefulness. And I don't really see any clear path to getting past this hurdle, I think this may just be about as good as they're gonna get in that regard. Would be great if they prove me wrong.
Deepseek, Nvidia and meta are pumping out one paper after another.
New and better things are coming. They will just take time to implement, and I doubt they cancel current training runs. So I guess it will take up to a year for the new things to come out
Can the bubble burst in this time, because people lose patience? Of course. But we are far from the end.
The flat line prediction is now 2 years old...