Outline of MDL

Developed by Jorma Rissanen in 1978; ongoing MDL research; a tutorial by Peter Grünwald
An information-theoretic approach to machine learning (also connections to Bayesian inference and Kolmogorov complexity)
A formalization of Occam's Razor
Learning as data compression, "squeezing out as much regularity as possible", separating structure from noise
The problem: given some data embodying regularities, what is the most efficient way to describe it?
An encoding of some data is based on a theory (hypothesis, model, code) that represents a way of compressing the data
The success of an encoding depends on the complexity of the theory and how well the theory covers the data
Description length of a hypothesis H for describing some data D
`D(H) = L(H) + L(D|H)`
`L(H)` is the length in bits of the description of the hypothesis
`L(D|H)` is the length in bits of the description of the data when encoded with the help of the hypothesis
(Crude) MDL picks the H that minimizes `D(H)`.
MDL maximizes information compression:
`L(D) - L(H) - L(D|H)`
Protects against overfitting; a hypothesis with too many parameters scores too high on the `L(H)` term
`L(D|H)` as a function of the probabilities of the data points given the hypothesis

Goldsmith (2001): an example (1)

Corpus:
bake, bakes, baked, baking, vote, votes, voted, voting, list, lists, listed, listing, salt, salts, salted, salting, wait, waits, waited, waiting, mark, marks, marked, marking, pull, pulls, pulled, pulling

Length of the model	List lengths		`lambda(T) + lambda(F) + lambda(Sigma)`	`log 28 = 4.80`
	Stems	bake bakes baked baking vote votes voted voting list lists listed listing salt salts salted salting wait waits waited waiting mark marks marked marking pull pulls pulled pulling	`sum_t (log26 * text(length)(t) + log ( [[W]] / [[t]] ) )`	`150 * log 26 + 28 * log 28` `= 839.67`
	Suffixes	NONE	`sum_(f) (log26 * text(length)(f) + log ( [[W_A]] / [[f]] ))`	`0`
	Signatures	{ptrs to all stems} {NONE}	`sum_sigma log ( [[W]] / [[sigma]] ) +` `sum_sigma [ lambda(t_sigma) + lambda(f_sigma) + sum_t^(t_sigma) log ( [[W]] / [[t]] ) + sum_f^(f_sigma) log ( [[sigma]] / [[f in sigma]] )]`	`29 * log 28 = 139.41`
				`983.89`
Length of the compressed corpus			`sum_{w in W} [w] [log([[W]]/[[sigma_w]]) + log([[sigma_w]] / [[t_w]]) + log([[sigma_w]] / [[f_w in sigma_w]])]`	`28 * log 28 = 134.61`

Goldsmith (2001): an example (2)

Length of the model

List lengths

`lambda(T) + lambda(F) + lambda(Sigma)`

`log 7 + log 6 + log 2 = 6.39`

Stems

bak vot
list salt wait
mark pull

`sum_t (log26 * text(length)(t) + log ( [[W]] / [[t]] ) )`

`26 * log 26 + 7 * log 7 = 141.86`

Suffixes

NONE
e es s
ed ing

`sum_(f) (log26 * text(length)(f) + log ( [[W_A]] / [[f]] ))`

`9 * log26 + 2 * log 10.3 + 2 * log 3.4 + log 4.6`
`= 54.76`

Signatures

bak vot	e es ed ing
list salt wait mark *pull	NONE s ed *ing

`sum_sigma log ( [[W]] / [[sigma]] ) +`
`sum_sigma [ lambda(t_sigma) + lambda(f_sigma) + sum_t^(t_sigma) log ( [[W]] / [[t]] ) + sum_f^(f_sigma) log ( [[sigma]] / [[f in sigma]] )]`

`log 14 + log 5.6 +`
`log 2 + log 4 + 2 * log 3.5 + 4 * log 4 +`
`log 5 + log 4 + 5 * log 1.4 + 4 * log 4`
`= 35.66`

`233.13`

Length of the compressed corpus

`sum_{w in W} [w] [log([[W]]/[[sigma_w]]) + log([[sigma_w]] / [[t_w]]) + log([[sigma_w]] / [[f_w in sigma_w]])]`

`8 * (log 3.5 + log 2 + log 4) +`
`20 * (log 1.4 + log 5 + log 4)`
`= 134.61`