I recently gave a talk to the excellent research team at Signal Media. And got asked the question I didn’t want to be asked. It was what is the BFMI in PyMC3?
The way I largely think of it is, in a practical level – which is – if the BFMI metric is below the threshold of 0.2 your model is bad. However I understand that this isn’t a deep understanding of it, and could be irritating to some.
Let me use the illustration from a paper by Michael Betancourt.
From a practical point of view if you had a model that returned (b) in the picture above, you’d know something is wrong. And you’d know the model is misspecified. This doesn’t tell you how to fix your model, but model building and tweaking in Probabilistic Programming is tricky, and involves intuition/ exploration.
What does it mean technically?
Hamiltonian Monte Carlo proceeds in two phases — the algorithm first simulates a Hamiltonian trajectory that rapidly explores a slice of the target parameter space before resampling the auxiliary momenta to allow the next trajectory to explore another slice of the target parameter space. Unfortunately, the jumps between these slices induced by the momenta resamplings can be short, which often leads to slow exploration.
This slow exploration is a problem, because ideally you want your Hamiltonian Monte Carlo to explore the space as quickly as possible. Otherwise in practical time – i.e. the time you have to run a model – your model won’t give good estimates.
Alternatively as my friend Junpeng Lao said – BFMI is about momentum… I understand it as: if you have a heavy tail distribution, you need large momentum to go to the tail, and Gaussian momentum proposal is not enough to give you that. The reason we us the Gaussian distribution for the proposal is that it’s easy to compute, and often in Hamiltonian Monte Carlo we use that. So we sample the momentum and compute the energy step. It is the difference between the momentum and energy step that results in the BFMI (Bayesian Fraction of Missing Information).
If that doesn’t satisfy you, you can look here for more details.