C.5 Abstractions: Overview
Why do we care about abstractions?
The pointers problem
Open questions about natural abstractions
Prerequisites
If you want a single read-ahead document, see the shared prerequisites refresher.
Information theory refresher
The abstractions exercises lean heavily on a short list of information-theoretic identities:
- entropy \(H(X)\) and conditional entropy \(H(X \mid Y)\);
- mutual information \(I(X;Y)\) and its entropy expansions;
- KL divergence \(D_{\mathrm{KL}}(P \| Q)\);
- the fact that if \(Y = f(X)\) deterministically, then \(H(Y \mid X) = 0\) and \(I(X;Y) = H(Y)\).
If these feel rusty, review them before diving into the natural latents exercises; they do real work in almost every proof.
For a fuller version, see the shared prerequisites refresher.
Bayesian networks refresher
The main graphical ideas you need are:
- a Bayesian network factorizes a joint distribution according to a DAG;
- chain, fork, and collider are the three local patterns to remember;
- d-separation is the criterion for when conditioning blocks or opens information flow.
For this session, the most important practical point is understanding when a latent variable screens off observables and how conditional independence shows up in graph structure.
For a fuller version, see the shared prerequisites refresher.
Category theory: universal properties (optional)
This is optional. The only intuition worth keeping in mind is that a universal property characterizes an object by the maps into or out of it, together with a uniqueness condition. If you do not already know category theory, you can safely skip this on a first pass.