Unbelievably simple recent ideas in ML, often top-conference fodder:
To detect if text comes from LM X, randomly modify it and get X's logprobs of the original and the mod.
If p(original) > p(mod), classify as LM generated.
"to increase performance by 10% absolute, just take the majority-vote answer of several LM answers"
Follow
"to reduce resource use by 50%(!), use a large model to do rejection sampling of small models' output"
I guess chain of thought is itself one of these.
"to find hyperparams about twice as fast, start a bunch of networks training and after a while copy the weights of the one improving fastest. repeat"
https://www.deepmind.com/blog/population-based-training-of-neural-networks
Neural networks have shown great success in everything from playing Go and Atari games to image recognition and language translation. But often overlooked is that…
www.deepmind.com