Decision Theory: Overview
Why decision theory?
Sufficiently powerful AI systems are, in an important sense, consequentialist: they reliably steer the world into a narrow region of configurations — those that satisfy their objectives — regardless of the initial conditions they start from. This is expected for two mutually reinforcing reasons. Economically, designing an AI to achieve our goals means that we no longer need to manually specify how to act given each possible state of the world; the system itself learns to funnel a broad distribution of initial conditions toward the target configuration. Competitively, an agent that can reliably steer toward its goals regardless of what other agents do will win out over agents whose behavior is more contingent. The result is that powerful AI is naturally modeled as an agent with a decision procedure — a way of predicting the future consequences of its actions and selecting among them. One informal gloss: decision theory is agency as time travel, reasoning backward from desired future states to present actions.
This modeling assumption matters practically even if a deployed system does not internally contain a cleanly factored “decision theory module.” What matters for safety is whether the system’s behavior is well-approximated by some decision procedure, not whether that procedure is explicitly represented. Safety arguments typically take the form: if the system has properties Y and Z, then it will satisfy safety desideratum X. One of those assumed properties is often a particular decision theory — how the agent selects actions to achieve its goals. If the actual decision theory differs from the assumed one (or if the agent self-modifies from one to another), the proof breaks down. Understanding decision theory is therefore prerequisite to knowing which assumptions about agent behavior are load-bearing in a safety argument, and which are fragile.
Beyond modeling, decision theory also bears on the design of agents. Understanding what the optimal decision procedure looks like informs how to architect systems with a cleanly factored world model and planning algorithm, rather than ones in which goal-pursuit and belief-updating are entangled in ways that are hard to analyze. And decision theory is central to two further problems: reflective stability (will an agent self-modify to use a better decision theory, and what makes a decision theory “better” by its own lights?) and multiagent reasoning (how should an agent behave when other agents can model it, and vice versa?).
Session materials
Reading list
The readings are arranged roughly chronologically. FDT is the primary technical reference; read it first. The UDT sequence develops the updateless perspective independently and surfaces open problems that FDT does not fully resolve.
- Functional decision theory: a new theory of instrumental rationality — read Chapters 1–5
- Towards a new decision theory — read entirely
- Updateless decision theory — read entirely
- Conceptual problems with UDT and policy selection — read entirely
- Pitfalls of building UDT agents — read entirely
Discussion prompt
Consider two agents A1 and A2. Suppose both agents’ code will at time T be perfectly transparent to each other — A1 can verify A2’s preferences, decision algorithm, and anything else inspectable from A2’s state at time T, and vice versa. (If the scope of “transparent” or some other term is ambiguous to you, state your interpretation and note how your answers might depend on it.)
Under these conditions, when might A1 and A2 end up in a Pareto-inefficient outcome? If there are cases where inefficiency seems plausible but a non-obvious argument shows otherwise, you are encouraged to mention that argument.