Because the babble problem isn't solved, people will learn not to trust the output of an LLM. Simple, raw factual errors will be caught often enough to keep people on their toes.
It will put cheap copywriters out of a job, but will never be good enough for research.
We will reach Peak Training Data in the next five years, where you can't improve the model by feeding it more training data because you're already using everything worth using.
We will reach a point of diminishing returns on increasing parameters within the next 20 years, where the cost of hardware to increase parameter counts isn't worth the increase in value you get from the model.
We won't ever hit Peak Parameters, because a new paradigm will appear and draw people away from LLMs before we do.
In 30 years, LLMs will be used for short text generation in products that aren't considered to be AI anymore.
Reminder that you shouldn't listen to me about anything. I'm a dilettante and my knowledge is a mile wide an an inch deep.
@WomanCorn This feels quite true to me. (Where "new paradigm" could also just be "better activation function found").
@WomanCorn
Hm. This feels too pessimistic ("pessimistic") to me.
I guess if I take LLM very narrowly, then yes, we're running out of training data. But we have much much video data {{cn}} and can much more easily generate more, *and* I have an inkling that there's some alpha left in generating training true training data+doing RLHF with real-world prediction.
I guess I think we can probably reduce the confabulation problem enough so that it doesn't matter *as much*.
@WomanCorn
Especially if we get good mechanistic interpretability, there'd be some nice boundary conditions to use during training ("oh, this model clearly still has circuit xyz, maybe show it datapoint 67559438 a couple more times so that it learns geography better", or even directly editing networks).
@niplav I'm not sure how much of the magic of LLM is that the input and output are both text.
If we can get something that learns from videos, they may be more value in that.
I expect that the text -> art bots will have similar limitations, but probably decoupled from the text -> text ones.
The babble problem will not be solved. Effectively ever. It cannot be solved without a major change in architecture.