Advancing Foundational Cyber-resilience

AI technology could spur a resilience renaissance; we just need to mind the gaps

AI capabilities are growing quickly, spurring us to grow our AI resilience efforts.

Evan Miyazono joined us earlier this year as a research fellow at Convergent. Evan is a physicist, metascientist and founder of Atlas Computing, which has been building the field of AI-based formal verification and hardware-enabled governance mechanisms.

Good Numbers, Bad Numbers

How will science and technology change with abundant intelligence? How will people tap into a “country of geniuses in a data center” to bring about new technologies and cures? These are among questions that motivate us in our work.

But what’s to prevent the people tapping into the data center — or the “geniuses” themselves — from doing intentional or accidental harm?

Computer systems innervate every aspect of our day-to-day lives, undergirding the core infrastructure that supports society. Beneath all our digital possessions — e-mails, smartphone apps, bank accounts, electronic health records — is a world of real things: electrical grids, oil and gas pipelines, buildings and their access controls, hospitals and medical equipment, water treatment plants and air traffic control systems.

Somewhere, in each of those pieces of infrastructure, are deeply embedded cyberphysical systems. To put it overly simply, computers take in data, run calculations, and produce numbers. Trains don’t run into each other, and planes land — for the most part — when and where they should because the computers are good at outputting the right numbers at the right times.

When we converted phone lines into the internet, we made it almost as easy to hack into a computer as it was to make a prank phone call. Even though the systems have evolved significantly since, we still exist in the midst of an arms race between the information security experts who help develop products and systems and malicious actors who want to exploit holes in them. For some of us, that world may only reach us when a video from DEFCON or James Mickens makes its way into our YouTube queues. For some of us, our (or our loved ones’) access to medical care, energy, flights, online content, or financial security have already been affected by mistakes, hacks and breaches. Either way, it is always there, humming along as we tap our cards to check out or get on a train, as we check our e-mail or play a song on our phone, as we visit a loved one in the hospital or turn on our water faucet.

Today, trillions of dollars and thousands of companies are working to build technology that can serve as a drop-in replacement for people, whether they are call center employees, truck drivers, or junior software developers. These machines are increasingly trusted to write code and agentically interact with the world. We’re entering a new stage of vulnerability to the electronic systems that make our fast-moving, deeply interconnected lives possible, with many humans moving onto or even out of the loop.

By creating synthetic labor at scale, labor that can interact with the real world in countless ways, we are creating a new category of vulnerability and attack vectors.

As AI develops, we have a critical window to steer its use so that it promotes resilience rather than increasing vulnerability.

Raising Resilience

Some AI capabilities are growing exponentially. As this trajectory continues, it becomes increasingly valuable to rigorously map what breakthrough technologies and organizations are needed to increase our resilience and ensure that the things we care about remain secure and robust.

After we map the gaps — the missing capabilities — we have the opportunity to decide which should be bridged and crossed. Our principle here is to use the steering wheel, not just the pedals:

We believe that we have a relatively narrow window to steer technological progress in beneficial directions - not just in AI itself, but in the surrounding fields. We believe right action during this critical window will shape many of the enduring impacts of this technology. This means that advancing “defensive” technologies faster today could spell the difference between stability and chaos in an accelerated world. We’ve previously made reference to “differential technological development” and “defensive (or decentralization, or differential) acceleration”; this principle says that we must consider whether the technologies we are accelerating are defense-dominant or not, and that we ought to steer new technology development away from equilibria in which technology systems that impinge upon human wellbeing negatively can proliferate unchecked.

We’ve been talking about this as “AI Resilience”: an approach of identifying features of a world where humans are flourishing with powerful AI, and identifying what needs to be built to move toward it. We believe this approach can be pursued by differentially accelerating defense-dominant technologies.

Critically, some approaches that might seem defense-dominant, like using AI systems to find and fix vulnerabilities, can actually quickly become dual-use, because it can be far easier to find and exploit a vulnerability with powerful AI tools than to widely deploy a patch that AI helped us find.

We’ve already begun to roadmap using this approach:

Earlier this year, we supported a report on this topic by Nora Ammann and Eddie Kembery, which you can read at airesilience.net. In it, they frame a set of technologies and practices that could create defense-dominant scenarios for each of cybersecurity, biosecurity, and epistemic security, which combines protection of mental health and defense against mis/disinformation.

Atlas Computing also recently completed an AI Resilience Gap Map exercise (building on the Convergent Research Fundamental Development Gap Map) with dozens of ideas for potential organizations that the team intends to explore by hiring a cohort of field strategists to design and source founding teams for the orgs. This map provides a framework for thinking about opportunities, but there are many that still need to be identified, so much more work is needed.

Atlas and Convergent also collaborated on a piece for the Institute for Progress on securing what is commonly referred to as the US AI tech stack. Another piece of ours in that collection considers the value of brain connectomic data for developing better-aligned AIs, and several other pieces bear on key opportunities in the AI resilience gap map.

This roadmapping work suggests some clearly useful paths forward.

Provably Secure

Even precious gems, fossils, and deep stone weather over millennia. But logic holds perfectly over time; once you mathematically prove that something is true for a given set of assumptions, it’s true, given those assumptions, forever. Proofs persist. This permanence offers a foundation for building for cyberphysical resilience.

This strategy builds upon our capacity to find, formalize, and verify proofs. We support this at one of our existing Focused Research Organizations, called Lean FRO: the Lean FRO is developing an open source language that allows anyone (human or AI) to guarantee absolute correctness in their mathematical proofs by boiling down mathematical abstractions into watertight chains of logic that a computer can decisively check.

Ultimately, every circuit, program, and chip that does calculations can be represented by mathematical rules. Because we can mathematically prove facts about what these rules can and cannot produce, we can know durable truths about each of those objects and the calculations they do. We can start from reasonable assumptions and prove things about how those combined hardware and software systems work. This could range from a proof that the total amount of money being moved in a transaction remains constant at every step in its execution from start to finish, or that a particular power plant will never blow a transformer as long as no more than one other component fails. When you can begin layering these proven security elements on top of each other, you can have a more robust and resilient system overall.

Even in a world in which the outputs of LLMs are the results of complex neuronal activations and training parameters that we don’t fully understand, code produced by an LLM can be checked and verified relative to a safety or performance specification all the same – a process called formal verification. Even if that code is longer than any human or group of humans can parse, if a fact is proved about it, that fact is true. And because proofs are much, much easier to verify than find, humans can still verify these proofs.

In this way, we can have deterministic knowledge about code. And these proofs constrain not just human actions, but also potential actions of a superintelligent future AI system; if we’ve proven that a program will never crash for any input, then no AI system can find an input that will make it crash, no matter how intelligent the AI is.

In a world that is increasingly dependent on code that no human can review and in which hardware pervades every system we depend on, we can create digital systems that are provably secure.

Here’s Adam Marblestone discussing this on Ashlee Vance’s podcast, Core Memory :

So far at Convergent, Evan has helped mentor a team working on flexHEG-adjacent technology for compute governance through our UK FRO residency, powered by ARIA, and is helping to design and recruit for a FRO to generate formal specifications, which we’ll talk about below. These are just two early examples of efforts that we believe can meaningfully advance cyberphysical system security.

But there are many more.

A Resilience Renaissance

AI agents are going to open many gates to futures that we thought were locked or didn’t know existed. Even as frontier labs work to harden their systems against misuse, and even as leading companies use frontier models to develop better security against fraud, terrorism, and acts of cyberwar, there will still be poorly protected systems that will crack under the pressure. Market demand and national security might address most of these, but each of these are typically governed on timescales slower than AI actions.

But AI tools can also give us new capabilities in this arms race. As Eric Drexler writes:

Not all gates matter equally. Some unlock applications, for better or worse. Some unlock tool rooms, and some tools open more gates. Sequences matter. Whether interpretability precedes capabilities, whether steering methods precede autonomy, whether knowledge integration pulls ahead of epistemic collapse — these differentials in technology development can shape outcomes for the world.

We can live in a vastly more safe and resilient future, supported by the ballast of defense-dominant technologies. AI-enabled developments could introduce suites of technologies that make the world dramatically safer, healthier, richer, and stronger.

Beyond some sequence of those gates could be the tools that usher in a renaissance of resilience. In our next post, we describe the most promising gates to begin with.

Let’s get knocking.

Read the follow up to this post here:

Thanks to Mary Wang and Christina Agapakis for their comments on drafts of this piece.