Are the smartest systems today (brains/AI) near the limit of how much a world-model can be compressed, or are we hitting the limit? Given all the data we observe, how many intrinsic dimensions is it distributed over?
Follow
One reason to expect brains to merely be scratching the surface is the deep inevitability of pleiotropy and build-up of technical debt (increased refactoring-cost) for all hill-climbing algorithms. "Consciousness" (i.e. global neuronal workspace) is the only adaptation on the planet that even marginally addresses it. But it too is powerless against the tide.
So if we can compress the world as much as we already have, using tools (brains & GNW) for which there we have strong reasons to expect are very limited, that suggests to me that there's a wealth of untapped simplicity beyond the horizon. But it can only be accessed by top-down non-myopic deliberate practice.