When they release GPT-4 API access, I want to make a serious effort on a "language model overlay".
The models themselves require you to put some effort in preparing a context, previous patterns, etc. But if you were willing to invoke GPT-3 hundreds or thousands of times to answer a prompt:
"Write me a business plan for an ice cream store"
-> "Produce a high-level template"
-> "Given this template, which section should you work on first?"
-> "What data sources are needed for this section?"
So the overlay could iteratively refine some high-level plan, store context into persistent buffers(maybe a stack of contexts?), recursively invoke itself, generate requests to invoke a limited set of tools(such as fetching financial ratios for any public ice cream companies), and any requests without a specific tool could be bubbled back up to the user as part of the interactive prompt.
ChatGPT's model at least seems capable enough, though I'm not sure I value the RLHF they've done.
It's not going to replace your lawyer, but your lawyer might say "Here's the facts of the case. Can you review every legal filing in the county for the past 5 years and generate a list of the most obvious defenses?"
"I'm studying Genetic Programming. Can you review every single CS paper published, and if summarize any obvious relevance to my project?"
"3000 public companies and we have to keep up with all their filings. Anything somewhat related that we're missing?"
etc.
Sometimes you hear a news story like "The Republicans dumped 1800 pages of dense legal text written by their lobbyists on us and said 'you have 24 hours to read it before we vote'".
GPT-3 can read the whole thing once for $20. Add in a 10x or 100x multiplier to reread or summarize or to give it context on what things you value or want to pay attention to, and that's still only $2000.
Mergers & acquisitions, discovery proceedings, etc. often involve "dump too much information for them to read".
@mira
Interesting case study of attack/defense ratio here: My mind immediately went to "But GPT-N can also *generate* boilerplate legalese, so they'll just add more."
But confabulation is a problem here! You don't want confabulated text in your laws, and adding 20k pages is problematic. But they could strategy-steal: Let GPT-N legalese it out, *check* via GPT-N whether it works, and then dump it on the opposition.
4. I think it actually is difficult to write text that is meaningful past a certain point. Laws are supposed to constrain people's actions, and there's only so many bits of policy selection someone can meaningfully want.
So I would expect very large amounts of text to be compressible closer to some constant size, if it was expanded from a policy of constant size.
And once expansion becomes a common thing, there will start to be GPTs trained for compression and I think they would win.
@niplav
1. OpenAI lets you finetune models via their API, or finetuning can be negotiated to be done offsite for especially sensitive cases. So you can't rely on accessing the model of your opposition.
2. The prompt is important too, and you might not have access to your opponent's prompts.
3. For scaling up the quantity, I think "1800 pages" might be plausible if undesirable, but "1 million pages" starts to get truly ridiculous and politicians will have to "just say no" to some upper bound.