If I were a Shard Theory person, I'd say that constitutional AI is a next step in training AIs in the similar way that humans are trained: Reinforcement learning from interacting with other agents, starting with a simple set of values

@niplav I dunno what shard theory is, but I agree with this notion.

@Paradox They claim that human learning is a lot like current AI training: A lot of self-supervised pre-training+some fine-tuning+a little bit of RL (and in this view then multi-agent RL on top)

@niplav @Paradox on my view, there are "base drives" and "verbal values". the former are selected for producing effective behaviour, and the latter are selected for producing effective words. (somewhat tracking near/far mode of human behaviour.)

and since humans hv the ability to do hypocrisy (aka value-action gap, rationalisation, memetic-immune-system), it enables our verbal values to evolve independently of what makes effective behaviour. this is crucial, and (i think) extremely lucky, bc no brain cud possibly evolve cosmopolitan values if it had to actually implement it in its behaviour.

"effective altruism" is the v rare mutation where a brain starts to break down its own rationalisation/hypocrisy-barriers, and instead of then becoming consistently selfish, it generalises the other way, such that verbal values start to influence actual behaviour. humans can do this bc we are v prone to overgeneralising our learned proxies.

@rime love this explanation! Explains some tension: if some parts generalize twd altruism and others twd selfishness you have to find the equilibrium

Follow

@rime wouldn't go as far as Ngo to say all of alignment risk comes from here but seems like a rather large source

Sign in to participate in the conversation
Mastodon

a Schelling point for those who seek one