The evolution of CHI-FRO

Three principles from an engineering-heavy partnership to build out tools for probabilistic programming

How does a child learn about the world so much, from so little, so quickly? Why does it take so much more data to train today’s leading AI models? How do you grow a mind?

One answer might come from probabilistic programming.

Probabilistic programs are generative models written as code. Probabilistic programming systems allow users to specify how (sometimes very noisy) observations could have been produced, then infer what hidden causes of those observations are the most plausible. They do this by developing and then leveraging domain knowledge and rules about the world, like an understanding of physics or biology or economics. The models and programs produced by probabilistic learning can also be refined by “metaprograms” that can transfer their learning from one domain to another. This ability to specify structured models is meant to allow probabilistic programs to be significantly more energy and data-efficient, compared to unstructured deep learning models.

Deep learning, by contrast, often learns its “rules” mostly through gradient-descent based self-supervised learning: a model is presented with massive piles of examples (often strings of tokenized text), and the model adjusts its weights until its outputs match the training signal. This method of developing artificial intelligence has worked astonishingly well by leveraging newfound scales — of data, compute, and training modes. And it has produced a kind of superhuman performance, where a single model can achieve scores on tests and benchmarks across fields that no single person could reasonably master in their lifetime.

But this scale has a price. Even as mechanistic interpretability work helps us glimpse the vast latent structure these models discover, today’s systems often spend huge amounts of compute learning regularities that humans seem to acquire with a few examples and a much more modest metabolic budget. The brain runs on roughly as much power as a dim light bulb — yet it can learn quickly, adapt fluidly, and generalize in ways we still don’t fully know how to engineer.

The quest to develop better algorithms and implementations of probabilistic programming is, in part, an attempt to close that gap and answer the question of how the human brain became such an efficient learner. How can we build models that discover the world’s structure from scratch? Can we sustain those models with energy or compute scales that are closer to the human brain than a data center?

Here’s Vikash Mansinghka, a Principal Research Scientist in the Department of Brain and Cognitive Sciences at MIT - describing this vision for probabilistic programming in a TEDx talk:

Around the same time as Vikash gave this talk, he was also gearing up to co-found and lead CHI-FRO, a FRO we’ve just added to our ecosystem page.

CHI-FRO (pronounced “kai-fro”, like the Greek letter χ) launched with a mission to develop an open source platform that shows how probabilistic programming can scale cheaply and effectively. As with any fast-moving and ambitious effort, the design decisions made at the outset of an FRO shape the arc of its growth and impact.

In the case of CHI-FRO, we believe that three of those choices really stand out and contributed to the product development successes and remarkably well-fitting early transition that the organization saw during its time at Convergent. (After support from Convergent from 2023 to 2025, CHI-FRO transitioned to a home with another non-profit.)

In addition to covering some of the technical vision and origins of the FRO, we hope that the reusable institutional lessons we share can help the ecosystem — especially in light of the upcoming Tech Labs RFP from NSF and other opportunities in the metascience space.

1. Consider the Counterfactual

How did the CHI-FRO come to be?

We are in the middle of a wave of new technologies driven by deep learning and GPUs. These two technologies took off together: GPUs had already been heavily engineered as graphics cards for computer games and video-intensive workflows. Because they enabled matrix calculations (like those done by the transformer neural network architecture) to be done in a massively parallel way, they became a natural hardware platform for a new software paradigm that was beginning to show a lot of promise.

But what if the path of technology had been different?

Sara Hooker has a great piece called “The Hardware Lottery” about the history of computer science. Per Hooker, a hardware lottery is when “a research idea wins because it is compatible with available software and hardware and not because the idea is superior to alternative research directions.”

Nearly a decade before CHI-FRO, Adam was intrigued when Vikash posed a deceptively simple question. Consider the counterfactual: what could we learn if we applied a similar engineering mindset to probabilistic programming as the one driving deep learning? What if we could get probabilistic programming running ultra-efficiently on powerful hardware, and make it ultra-accessible with languages as usable and widespread as TensorFlow or PyTorch were when deep learning took off?

We didn’t know the answers, but we knew one way to start pursuing them: build better tools for researchers working on probabilistic programming. And in this case, that scope of ambition and technical roadmap indicated an FRO was needed, in addition to and in support of academic research.

2. Build like a Startup, not a Side Project

Engineering projects often need the right organizational infrastructure to become exceptionally productive. Take as one example the productivity of the Lean FRO as measured by its annual roadmaps after it was able to be structured more like a software start-up and less like an ambitious side-project.

When CHI-FRO started, there had already been numerous intellectual achievements in the probabilistic programming literature. Scaling the promise of those early papers to tackling modern, high-dimensional problems required a particular kind of engineering team structure and labor force. In particular, a robust, well-maintained programming language can provide that kind of needed scale and act as a real force multiplier for a field.

And that’s exactly what CHI-FRO’s engineering team worked on, enabling the continuing development of the Gen language and its ecosystem of open-source libraries, which has been expanded to a number of domains.

These include “3D-scene perception that learns to perceive new objects and scenes on one GPU in real time” (Gen3D), “trustworthy conversational AI that gives grounded, auditable answers, and aims to be more accurate than GPT4 in data-driven domains, using probabilistic programs built and fine-tuned on one GPU” (GenLM), and “the first GPU-accelerated probabilistic programming stack with programmable inference, enabling rational inference with probabilistic programs to scale via GPUs” (GenJAX, see this paper).

Together, these achievements pave the way for technologies that can leverage probabilistic program inference using hundreds or a thousand times less computation than deep learning for similar performance.

3. Engineer for (and with) the Ecosystem

Creating essential technology for probabilistic programming means helping answer some very big questions for humans and machines alike. As Josh Tenenbaum, Vikash’s co-founder at CHI-FRO, expressed in the conclusion of his seminal paper from 2011:

The key research questions are as follows: What approximate algorithms does the mind use, how do they relate to engineering approximations in probabilistic AI, and how are they implemented in neural circuits?

These questions have a notable structure that comes from the late British cognitive scientist David Marr. Marr famously articulated three connected levels at which one can analyze cognitive systems: the computational level (which describes what the system does and why); the algorithmic level (which describes how the system does those computations and what representations it uses); and the implementational level (which is how the system physically manifests those computations and representations).

In the case of probabilistic programming, Vikash and Josh wanted to advance all three levels. They already were: Josh’s research program had long asked what kinds of structured generative models could underwrite human learning and common sense. In a more applied way, Vikash was driving his efforts towards making probabilistic programming work on modern, high-dimensional tasks. And hovering in the background was a still deeper question: how could these models be implemented efficiently in neural circuits?

Because each of Marr’s levels tend to demand a different kind of labor, these levels are helpful not only for intellectually categorizing research questions, but also point towards how science and engineering can be organized institutionally. Vikash, Josh, and their colleagues at MIT and other universities were already pushing the bleeding edge of this field in their theoretical and computational papers, building models and running experiments with subjects in their labs. At the same time, they were building some very promising proofs-of-concept through the ambitious Missions of the MIT Siegel Family Quest for Intelligence.

Complementary to these efforts, CHI-FRO’s structure as a Focused Research Organization allowed it to organize a software team to attack the engineering and scalability bottlenecks facing the researchers at MIT and elsewhere. Their contributions sat in the critical algorithmic and implementational layers, helping turn ideas about probabilistic programming into functions, libraries, and language contributions that could be leveraged by the wider research community.

This critical partnership, as part of a broader Project CHI, was patterned into CHI-FRO’s roadmap and organizational structure at the outset, ensuring that what CHI-FRO made was responsive to and coordinated with the research community that could benefit from it.

At the outset of each FRO, we also urge our founders to think about their post-FRO transitions. A helpful framing for thinking about this can be choosing to moving your tools and work into a “.org” (a perpetual foundation), a “.gov” (becoming a state-supported public good), or a “.com” (commercializing your work and spinning it out), or some mix of the three. It was another critical part of the ecosystem — the Probabilistic Computing Foundation — which gave CHI-FRO the chance to pursue a “.org” transition, moving from an incubation period under Convergent into a new home that is even more closely co-integrated with the pursuit of scientific questions. There, CHI-FRO continues its work, and the team’s focus has shifted to building field-wide standard machine-executable models of the human mind that explain the workings of the brain, rooted in the tenets of probabilistic programming.

---

We are grateful to the CHI-FRO team for all their hard work, and are very proud of what they accomplished in building critical infrastructure just two years under the Convergent umbrella. And we continue to be quite excited about their future, and the continued accomplishments that we believe will emerge in this exciting intersection of computer science, cognitive science, neuroscience, and much else!