Abstractions & Latents: Overview

Why abstractions?

To build an AI that robustly does what we want, we need at least three things: to know which goals to encode, to encode them faithfully in a language a computer understands, and to build a system that actually acts on them. We don’t know how to do any of these. We don’t even know, mathematically, what a human concept is. Natural abstractions research attacks the middle problem: it looks for a faithful map between human concepts and structure an AI could also represent.

The reason a map might exist at all is that abstraction throws away detail and keeps what predicts. An agent with limited memory relative to its sense data will never see the same input twice, so it cannot use a lookup table; it needs a short program that compresses its data by orders of magnitude while preserving what determines its actions. Concepts are the components of that program — and we don’t want a single lossy compression, we want many modular, reusable functions that each capture a different part of the structure.

This matters for alignment because of the pointers problem. Human goals are stated in terms of concepts — don’t harm people, be honest — and words like “harm” and “people” are pointers to real-world structure. If the AI’s latent structure differs from ours, the same words point to different things. The empirical pathologies of interpretability (feature splitting, absorption, non-identifiability, “features” that match no human concept) are exactly what we should expect when we lack a mathematical signature for the thing we are looking for. Translation between unlike agents is only tractable if they converge on the same abstractions. So: do different observers, with different sensors and histories, get forced onto the same abstractions?

Many human concepts feel non-arbitrary, but it is hard to tell whether that is a genuine property or just entrenchment. The conventionalist reads concepts as artifacts of language and history, arbitrarily revisable, with no guarantee that translation across unlike agents is possible; and from a technical angle, concepts really are mathematically underdetermined — many models fit any given data, and a simplicity prior need not pin a unique ontology, especially for bounded agents. The most natural first attempt at “carving at the joints” is geometric — data as a vector space, concepts as clusters or directions — but it can’t ground objectivity, because a cluster tight in one basis can be scattered by a nonlinear change of coordinates. Naturalness grounded in clustering inherits the arbitrariness of the basis.

The way out used here is information theory. Entropy, mutual information, and total correlation depend only on which states a variable distinguishes from which — they are invariant under bijective relabeling of values, so they never see the basis. We model the world as a base variable \(D\) over a finite state space; an observable is a partition of that space (the distinctions it makes, up to relabeling); and a latent \(Z\) is a function of \(D\). A latent is a full explanation of \(D\) when conditioning on it makes every finite family of observables independent — total correlation drops to zero. The finest latent (\(Z = D\)) explains everything trivially, so the interesting theory comes from constraining which latents count.

Two conditions do that work. Mediation is the explanatory side: \(Z\) is an \(\varepsilon_{\mathrm{med}}\)-mediator of \(X = (X_1,\dots,X_n)\) if \(\mathrm{TC}(X \mid Z) \le \varepsilon_{\mathrm{med}}\) — conditioning on \(Z\) screens off the dependence among the observables. But mediation alone is too weak: the entire input \(D\) mediates trivially. Redundancy is the reuse / nontriviality side: \(Z\) is an \(\varepsilon_{\mathrm{red}}\)-redund over \(X\) if \(H(Z \mid X_i) \le \varepsilon_{\mathrm{red}}\) for every \(i\) — it is recoverable from any single observation alone. Redundancy alone is also too weak: the constant latent is a perfect redund and explains nothing. A natural latent sits at the intersection — a mediator that is also a redund. When one exists, it captures the structure common across observations rather than the accidental detail of any one of them.

The payoff is a pair of theorems. Mediator Determines Redund says any mediator already determines any redund: \(H(Z_{\mathrm{red}} \mid Z_{\mathrm{med}}) \le \varepsilon_{\mathrm{med}} + 2\varepsilon_{\mathrm{red}}\). From this one inequality, near-uniqueness falls out (a natural latent is minimal among mediators and maximal among redunds). Applied across two agents who share an observable family, it becomes Guaranteed Translatability: if Alice and Bob model the same world and agree only on some shared observables, then each agent’s natural latent is a function of the other’s, up to the tolerances. Convergence is forced by the structure of the world, not negotiated by the agents. Condensation then scales this picture from one latent to a whole organized system of latents — and recovers the same agreement payoff.

Schedule

Time Block Format
10:00 – 11:00 Lecture: motivation & the natural-latents picture Lecture (slides)
11:00 – 1:00 Mediation, redundancy & Mediator-Determines-Redund Reading + discussion + exercises
1:00 – 2:00 Lunch
2:00 – 3:30/3:45 Minimal mediation, guaranteed translatability & coarsening Reading + discussion + exercises
3:30/3:45 – 4:00 Break (15–30 min, depending on pace)
4:00 – 5:30 Condensation & a survey of open problems Readings + discussion; exercises optional
5:30 – 6:00 Quiz & feedback Wrap-up

Session materials

Slides — Abstractions & Latents

Readings

The lecture previews the whole day; the working sessions then go deep. Readings are grouped by topic below, in the order the day follows.

Pre-reading. Big-picture motivation for the natural-abstractions program; read before the session if you can.

Mediation, redundancy, and Mediator-Determines-Redund.

Minimal mediation, translatability, and coarsening. No reading required. The originals below are optional, and use John Wentworth’s diagrammatic Bayes-net notation, which I avoid.

Condensation.

Exercises

The exercises are collected by topic, each with setup, statements, and staged hints:

Open problems

Open Problems in Natural Abstractions Research

Wrap-up

Prerequisites

If you want a single read-ahead document, see the shared prerequisites refresher.

Information theory refresher

The exercises lean heavily on a short list of information-theoretic identities:

  • entropy \(H(X)\) and conditional entropy \(H(X \mid Y)\);
  • mutual information \(I(X;Y)\) and its entropy expansions;
  • total correlation \(\mathrm{TC}(X_1,\dots,X_n) = \sum_i H(X_i) - H(X_1,\dots,X_n)\), and its conditional version;
  • conditional independence and screening off;
  • the fact that if \(Y = f(X)\) deterministically, then \(H(Y \mid X) = 0\) and \(I(X;Y) = H(Y)\);
  • the invariance of all of these under bijective relabeling of values.

If these feel rusty, review them before diving into the exercises; they do real work in almost every proof. For a fuller version, see the shared prerequisites refresher.

Bayesian networks refresher

The main graphical ideas you need are:

  • a Bayesian network factorizes a joint distribution according to a DAG;
  • chain, fork, and collider are the three local patterns to remember;
  • d-separation is the criterion for when conditioning blocks or opens information flow.

For this session, the most important practical point is understanding when a latent variable screens off observables and how conditional independence shows up in graph structure. The “algebra of Bayes nets” reading turns this into a diagrammatic calculus that underlies the graphical proof of Mediator-Determines-Redund.

For a fuller version, see the shared prerequisites refresher.

Category theory: universal properties (optional)

This is optional. The only intuition worth keeping in mind is that a universal property characterizes an object by the maps into or out of it, together with a uniqueness condition. If you do not already know category theory, you can safely skip this on a first pass.