gavin leech @gl@schelling.pt
context maximiser @Arb
Joined Dec 2022
RIP Ryuichi Sakamoto. Pretty hard to summarise
Includes an open review, bravely solicited from the grumpiest man in the neighbourhood
https://epochai.org/files/direct-approach-review-nuno-sempere.pdf
'we can interpret a model’s cross-entropy loss as “how distinguishable” the model is from its training distribution, and use that to upper bound the difficulty of training a model to perform reliable, high-quality reasoning over long sequences.'
I also only now got the joke of his blog TLD:
.vc references Vapnik-Chervonenkis, not venture capital.
"the fact we can't describe it doesn't mean unreasonably helpful inductive biases can't be there. evidence is mounting that they are.
As intellectually unsatisfying as this is, the LLM approach works, but most likely not for any of the reasons we know. We may be surprised again"
"[Previously] I said if your objective function doesn't reflect the task, no amount of engineering or hacks will help you bridge that gap...
I have now abandoned this argument as well... we have barely a clue what inductive biases SGD on a model like GPT-3 has..."
"in 2018... I was shown a preview of... key GPT results: the ability to solve problems it wasn't explicitly trained [to do]... My immediate reaction was that this can't possibly work... this approach will never be even nearly competitive with specialised solutions"
" 'theory predicts deep learning shouldn't work, but it does, therefore our theory is insufficient.' This seems almost trivial now, but it represented a massive shift... the theory needed fixing, not deep learning... It may have been alchemy, but some actual gold was produced."
Characteristically open-minded + grounded post from @fhuszar on the deep learning shock to learning theory and the looming possibility of an LM shock
"[But] how *could* we be capable of disenchanting the world, when every day our laboratories and our
factories populate the world with hundreds of hybrids stranger than those of the day before? Is Boyle's air pump any less strange than the Arapesh spirit houses?"
- Bruno Latour
"Haven't we felt sorry enough for the consumer who leaves his car only to move to the sofa in the TV room, manipulated by the powers of the media and postindustrial society?! How we love to wear the hairshirt of the absurd, what greater pleasure we take in postmodern nonsense!"
"Haven't we shivered enough before the spectacle of the mechanized proletarian who is subject to the absolute domination of a mechanized capitalism and a Kafkaesque bureaucracy, abandoned smack in the middle of language games, lost in cement and formica? "
"Haven't we shed enough tears over the disenchantment of the world? Haven't we frightened ourselves enough with the poor European thrust into a cold soulless cosmos, wandering on an inert planet in a world devoid of meaning?"
whoa, the Mormons made a giant people database
"over 1.3 billion individuals in the tree and... over 5.7 bn digital images... the Vatican [called] baptism for the dead "erroneous" and directing its dioceses to keep parish records from Latter-day Saints"
(I still don't use it there for aesthetic reasons but maybe I should)
context maximiser @Arb
Joined Dec 2022
