Probabilistic Programming Languages

1. Probabilistic Programming Languages#

Graphical models#

The original lecture is about Bayesian models as graphical models. The professor talks a little bit about why they’re a useful way to think about the relationships between random variables, the Markovian property, and independence of nodes. A related concept is plate notation.

He cites Machine Learning: A Probabilistic Perspective, which is the old version of Probabilistic Machine Learning: An Introduction mentioned in the other recommended resources section.

Starting with this unit, we will be using Probabilistic Programming Languages (PPLs) to run our models. Check out Chapter 10 of Bayesian Modeling and Computation in Python for a look at what goes into creating a PPL and the components involved.

Missing example#

In the original unit 6 code archive, there’s a file called DeMere.odc that doesn’t seem to have anything to do with the rest of the unit. You can find a Python implementation here (right-click and Save Link As…).

BUGS#

Warning

Since Fall 2025, we use PyMC as our only PPL. I’m leaving this information here for reference, but don’t use BUGS for the class.

The course lecture examples use OpenBUGS from this point forward. I’ve mirrored the last version of OpenBUGS on this site:

I don’t recommend attempting to get BUGS working on a Mac.

If you do want to use BUGS-style models, I recommend NIMBLE or MultiBUGS. These can both handle most lecture models without modification. However, 95% of students successfully use PyMC to complete our course at this point.

PyMC#

Some history#

In 2003, Chris Fonnesbeck started writing PyMC (Wiecki et al. [2023]) as a graduate student at the University of Georgia—partly out of frustration with WinBUGS. Eventually, PyMC3 became popular as a Python-based alternative to Stan. Both used implementations of the NUTS algorithm for sampling (Hoffman and Gelman [2011]). PyMC3 was based on Theano, a tensor library for doing machine learning-related math. In 2016, PyMC was sponsored by NumFOCUS, an organization started in part by the authors of NumPy, Matplotlib, and IPython and others to promote open scientific computing.

Aaron started converting the course examples to PyMC3 in 2022. Later that year, PyMC version 4.0 was released. The backend switched to Aesara, a rewrite of Theano based on JAX. Towards the end of 2022, the PyMC devs forked Aesara to create Pytensor over some governance conflicts, and PyMC v5 came out. The current version is now v6, which uses PyTensor 3 and is compatible with Arviz 1.0. This is the first version to use Numba compilation by default, which is much faster and easier to install.

We will copy the practice of using the watermark extension as in the PyMC Example Gallery to show the exact versions each example was last run on.

Installation#

Warning

We are in the process of updating the site to PyMC 6 and most of our old troubleshooting steps are no longer relevant. Please post in the PyMC troubleshooting thread on Ed Discussion if you have issues with any particular model.

This site is currently using PyMC version 6. Please follow the installation instructions on the PyMC website. We do recommend installing with nutpie.

Virtual environments#

It’s probably a good idea to create a dedicated Python virtual environment for this course and use it throughout the rest of the semester. A virtual environment keeps PyMC and its dependencies separate from other Python projects, reducing the likelihood of package conflicts.

There are several tools for managing Python environments, including conda, uv, and Python’s built-in venv. Any of these is fine, so use whichever you prefer. If you’re already comfortable with one, there’s no reason to switch.

Once you’ve created your environment and installed PyMC, make sure your Jupyter notebooks are configured to use the Python interpreter associated with that environment.

Debugging tips#

  • Try running a minimal model first, like the Taste of Cheese example to make sure your installation is working okay.

  • Use Google Colab to see if it’s your model or your installation. You can open any of the notebooks on this site directly in Colab using the pop-up link from the rocket ship icon on the top of the page. Colab will likely be somewhat slower than your machine, but it works reliably well for newer versions of PyMC.

  • In the lecture examples, the professor often uses 100,000 or more samples. Don’t do that in PyMC! The NUTS sampler, which is PyMC’s default, can explore the posterior much more efficiently. Start with 1,000 or fewer when first testing out your model, then increase to a comfortable number for your final results.