There's a story I've heard a few times in agent foundations, that involves starting with a very idealized theory of agency, like AIXI, IB, or UDT, in which we can achieve strong, provable safety properties,[1] and gradually making it more realistic, until it can eventually be implemented, or at least used to model real agents accurately. I'll call this process "de-idealization".[2] In this case, it means creating a tractable approximation of the theory that is also embedded, self-modifying, hier