C.3 Agent Foundations: Overview

Participant guide for the Agent Foundations session — readings, exercises, and discussion material on embedded agency, decision theory, and descriptive agent foundations.

Why agent foundations?

Prerequisites

If you want a single read-ahead document, see the shared prerequisites refresher.

Probability theory refresher

The main probability facts used here are:

  • conditional probability and Bayes’ theorem;
  • marginalization, i.e. summing out variables from a joint distribution;
  • independence and conditional independence;
  • the chain rule for probabilities.

For this session, the most important practical skill is being comfortable moving between joint, marginal, and conditional distributions.

For a fuller version, see the shared prerequisites refresher.

Formal logic refresher

The formal-logic prerequisites are mostly lightweight:

  • implication, negation, contradiction, and quantifiers;
  • direct proof, contrapositive, and contradiction;
  • the meaning of \(L \vdash \phi\), i.e. provability inside a formal system.

The self-reference material later in the guide depends heavily on the distinction between proving \(\phi\) and proving that \(\phi\) is provable.

For a fuller version, see the shared prerequisites refresher.

Computability theory refresher

You do not need a full theory course here. The main background is:

  • what it means for a program or Turing machine to halt;
  • the difference between decidable and recursively enumerable sets;
  • the halting problem as the canonical undecidable problem;
  • diagonalization and self-reference as proof techniques.

This is the background behind both Gödel-style incompleteness results and reflective self-reference problems.

For a fuller version, see the shared prerequisites refresher.

Information theory, causality, and statistical mechanics refresher

For the descriptive-agent-foundations material, the main ingredients are:

  • entropy, KL divergence, and mutual information;
  • Bayesian networks, d-separation, and the difference between conditioning and intervention;
  • the high-level thermodynamic picture that low-entropy states are atypical and require explanation.

You do not need heavy statistical mechanics machinery to follow the session. The important conceptual link is that optimization can often be understood as steering toward low-entropy outcomes, and that steering is constrained by information.

For a fuller version, see the shared prerequisites refresher.