In reply to Venkatesh Rao's Blyg — Venkatesh Rao
One of the points I’ve wanted to develop in my AI philosophy is a proper account of goals. Ironically, despite their strongly consequentialist stance, the rationalist crowd has no meaningful account of them and don’t seem to realize they need one for a complete theory. That’s why they find themselves resorting to thought experiments like paperclip maximizers.
Yes I agree with this 100%, I've kind of been hammering on this point for a decade. Except it's slightly worse, they actually have an anti-theory of goals (the orthogonality thesis) which ensures they will never have a decent theory of goal-directed behavior. See LWMap/Being a Robust Agent and [LWMap/Agency: Introduction .
it is pretty clear that future goals emerge from current behaviors through a kind of bootstrapping from innate drives (feeding, sex, energy minimization, play).
Freud is out of fashion but this is mostly what he was trying to investigate (focusing on the pathologies of the process of course, because they are easier to see than the normal cases). Rationalism is pretty pathological in its denial of desire, which also probably explains why it has morphed into a sex cult via enantiodromia.
One of the points I’ve wanted to develop in my AI philosophy is a proper account of goals. Ironically, despite their strongly consequentialist stance, the rationalist crowd has no meaningful account of them and don’t seem to realize they need one for a complete theory. That’s why they find themselves resorting to thought experiments like paperclip maximizers. Critics attack the explicit optimization frame, but that doesn’t bother me much. Treating goal seeking in constrained optimization terms is reasonable. What’s actually ridiculous is an account of goals that’s forced to make up absurdities like paperclips because it has no idea where goals come from.
The thing is, in the only examples we have of goal-directedness, the behavior of complex life that can think more than one step ahead, it is pretty clear that future goals emerge from current behaviors through a kind of bootstrapping from innate drives (feeding, sex, energy minimization, play). The ontogeny of goals involves a rich behavioral ecology. The idea of an entity going from “I think, therefore I am” to “I need to maximize paperclips” sounds absurd because it is. The phenomenology of goal emergence is much richer than that, and significantly more constrained by the path-dependent emergence process than we realize. Goals don’t emerge arbitrarily or get pursued indefinitely regardless of environmental effects and behavioral history. Our goals for today come from our behaviors and experiences yesterday. So what AIs will want in the future will be a strong function of what we ask them to do today, which is a strong function of what humans want and do. Even for AIs we let loose with minimal human-biased pretrainings in alien environments aren’t and cannot be blank slates. A protein-folding AI let loose in the mind of a Europa-exploring space probe is not a blank slates. It won’t suddenly decide to paperclip maximize.
This doesn’t mean AIs can’t set and pursue goals that might be dangerous to humans. But it does mean the goals won’t be arbitrary or disconnected from the environments of evolutionary adaptation in which they evolve. There will be stories that unfold along the way, in which our own behaviors will feature meaningfully.