AI 2040 Plan A — Verification SITREP
Grading the verification work that the AI Futures Project's “Plan A” depends on — where we're on track, and where almost nothing has started.
This page covers Plan A, the AI Futures Project’s proposal for a positive vision for AI development. In AI 2040 (Plan A), they propose a verified slowdown in AI development. This is one example of a use for AI verification, but it is far from the only approach the AI verification field is working on. Put simply, the AI verification field works on a toolbox of mechanisms which allow AI developers to make trustworthy claims about the systems they are developing and deploying. Sometimes the claims are useful for international deals; sometimes they support consumer or regulator trust.
Plan A is the AI Futures Project’s proposal for a positive vision for AI development. Under their proposal, in 2029, the US President announces that the US will pursue international cooperation to avoid an imminent intelligence explosion.
The US and China want to not do an intelligence explosion in the near future. In order for an agreement along these lines to be trustless, they need verification infrastructure in place to make sure that AI development isn't proceeding too quickly. They arrive at a simple solution for enforcing this: both sides agree to a temporary pause on new training runs.
A pause on training can be enforced by ensuring that compute is being spent on inference, which can be enforced by retrofitting existing data centres with inference-only devices. (Additional layers of enforcement include software-only privacy-preserving monitoring, and of course, old-fashioned human auditors.)
This description of 2029 requires AI verification technologies. Systems that provide sufficient assurance to both the US and China that defection by the other would either be caught or inconsequential.
In this piece, we break down the AI 2040 verification plans and describe the current state of play of different research projects.
Inference-only verification
The US and China want to avoid an imminent intelligence explosion. In order for an agreement along these lines to be trustless, they need some way to verify that neither side is unilaterally training dangerous AI. Distinguishing dangerous AIs from safe ones will require more time and understanding, so for now they go with a simple solution: a temporary pause on all new training runs. Both sides can still use datacenters for inference (i.e. running the AIs that already exist), but they will retrofit each other's datacenters with devices that will let them verify that they aren't being used for new large training runs.
The clear goal: develop solutions that allow datacentres to be used for inference, but block datacentres from being used for new large training runs. This is what we've been working on at Amodo, so how does AIFP's plan stack up compared to our work, and where are their important research gaps to achieve the AI 2040 Plan A in time?
Active effort: There is at least one active R&D effort underway. This doesn't mean it'll succeed, and in most cases there is only an initial R&D POC being developed. You should not rely on any of the points happening successfully yet.
Not started: There is not yet an active R&D effort, but we don't think the work requires too much R&D effort.
Not on track: There is not yet an active R&D effort and we are concerned that it is a research problem that won't be easily solved without considerable effort.
Add in simple, passive network taps on the frontend network (also known as north-south networking) that redirect copies of all input/output traffic to a recomputation server.
These network taps on frontend networks have multiple possible implementations. We currently think they should probably be completely passive and split optical signals into two because this is probably easiest to trust. Separately there should be a path from the storage bank back to the inference units.
Passive optical TAPs
Passive optical TAPs already work at slower speeds (400G), and whilst we haven't seen demos at 800 and 1600G (speeds that will be, or will soon be, deployed in data centres), some TAP companies have told us this is possible. We're not yet certain it'll be easy, but we're not too concerned. We have written about some of the technical hurdles here.
Recomputation servers (capture)
Recomputation servers may need to be able to capture all of the front-end traffic; in the worst case, this would require a 1:1 mapping of inference server front-end NICs to recomputation server NICs. This can work without new R&D, but it will require a lot of infrastructure. This will benefit from further work to define possible network topologies and sampling techniques.
New TAP types & bandwidth limits
New R&D projects might pursue different TAP types (e.g. sampling or packet hashing TAPs), ways to limit the bandwidth required on the front-end network of the recomputation servers, or dedicated packet capture systems.
Path from the storage bank to the inference units
Assuming that model weights cannot be transported on the front end network, and that memory wiping is used, there might need to be a separate network from a storage bank to the inference units. We have not seen any work on this. It is likely to require clear network partitioning and network lines with data diodes. An alternative approach would be to use the existing front-end network to transfer the model weights to the inference server. These packets must be verified as approved model weight/server image packets.
Force all AI workloads (inference, agentic tool use, etc.) to be organized in discrete and reproducible packets, which are transparent to the recomputation server (because they pass through the network taps).
Reproducibility is required to enable correctness proofs on the packets. By default, inference workloads are not necessarily reproducible. Achieving reproducibility may involve some cost overhead in order to maintain reproducibility but we think it is a solvable problem (see here for example of work in this direction).
Inference reproducibility workarounds
There are already demos of inference recomputation systems that work around problems of non-determinism (see our work implementing DiFR). This is different from making the workload truly reproducible.
Reproducible inference stack
Non-determinism isn't inherent; fundamentally, computers are deterministic, but the ML stack has allowed for (intentional and unintentional) non-determinism to creep in. We expect it is possible to write a reproducible inference stack, but that there is a reasonable amount of software and tooling to be written for this.
Network reproducibility · could be hard
The networking stack adds further complexity. Optimisations throughout the stack mean that packets are not individually reproducible by default. As described above, computers are ultimately deterministic, and it should be possible to design a stack with reproducible packets. But this might require considerable software, firmware, and even hardware work. Alternatively, it is conceivable that we could design schemes that do not require exact packet replication.
Perform partial recomputation (random samples of the packets) on the recomputation server to check the correctness of the packets.
The integrity of the recomputation server is an extremely important aspect of this step, but assuming its computations and outputs can be trusted, then partial recomputation through random sampling should scale extremely favourably to verify the correctness of all the outputs with extremely high probability while only using a small fraction of the original compute. See calculations in the appendix.
Recomputation algorithms
Initial inference recomputation schemes (e.g. TOPLOC and DiFR) are well described and are now being tested on relevant hardware and models.
Frontier recomputation algorithms
We expect recomputation algorithms will need to evolve as model architectures and inference techniques change. Existing algorithms are an early sign that this kind of recomputation approach can work, but we want to see model developers creating recomputation systems in lock-step with new model developments to ensure recomputation remains possible.
Recomputation red-teaming
Most algorithm development remains primarily academic in nature. Robust red-teaming is needed to bring techniques up to scratch.
Have strong physical security properties to help ensure integrity of the verification measures from e.g. tampering attacks.
The integrity of the verification process requires there to be sufficient physical security measures to avoid tampering, and other attacks that could subvert the verification process.
Recomputation server security
Whilst simpler than developing a whole SL5 cluster, the recomputation server needs to sit within the prover's data centre, possibly under their physical control. Significant research effort is needed to harden a recomputation server from all forms of integrity attacks.
TAP installation and network links
In the first instance, TAPs might be installed in 100-1000s of locations in a data centre (assuming pods of ~1000 H100 equivalent accelerators). Ultimately, high assurance network inspection might require 10s-100s of thousands of cables to be tapped. Whether 100s, 1000s, or 100,000s are being installed, checking correct installation, and monitoring that TAPs remain in place is not a solved problem. R&D might include radically simplified network architectures to make it easier to verify that TAPs are correctly installed (ensuring it's 100s, not 100,000s of TAPs), or other processes/schemes that allow the verifier to be sure all of the TAPs are correctly fitted.
Verification reporting
An extension of the integrity of the recomputation server is the integrity of the reporting that server does. Different schemes might allow for different verification reports to be released (e.g. from only saying “compliant/non-compliant” to sharing more detailed information with the verifier). The integrity of this report has not been explored.
Physical security and audits
Plan A described inspectors who will, firstly, count compute supplies, and secondly, be part of monitoring that verification systems are correctly installed and not tampered with. There has been little work on exactly how one inspects compute, inspects compute at scale, or inspects a verification system.
Verify completeness by bounding the maximum side channel communication, and using periodic memory wipes (probably through forced memorization).
The completeness of this verification proposal, i.e., that the only outputs from the cluster are verified inference tokens, requires side channel mitigations and memory wiping as described. We do not verify that spare compute isn't used for unapproved workloads (because this seems very challenging) but try to ensure that the only persisting results in the form of outputs are verified inference outputs. This requires strong side channel mitigations and periodic memory wiping on the inference units (which we think can be done through forced memorization).
Memory wipes
Existing algorithms (e.g. PoSE) may be suitable. We have started to test this on relevant hardware and expect to push this to the point where it could be quickly implemented in an emergency.
Side channel mitigation (shielding + noise)
There does not yet exist a plan for how to quickly scale side-channel defences on a frontier AI cluster. Early work includes pieces from Naci Cankaya and Amodo, but these are only early theoretical pieces.
Side channel wardens
In addition to suppressing side-channels, it would be useful to scan/observe side-channels to detect covert communication.
In this section, we described just the state of play of the engineering R&D. We want to see more work on all of these problems. Even problem areas described as “Active effort” need more attention. At a minimum, existing work needs to be extended, including:
- Turning proof-of-concept papers into reliable prototypes.
- Red-teaming prototypes, including with NatSec teams.
- Preparing plans for quick and secure scaled-up manufacturing.
- Stockpiling crucial components, including ready-to-implement TAPs and recomputation servers.
- Creating an installation playbook and guidance for governments.
- Collaborating with AI developers, data centres, and governments on which bits of the plan are likely to work, or fall apart.
Further verification research
This inference-only retrofit is just the start of AI verification. As the deal progresses, more advanced verification features need to be developed; ideally, R&D for these should start now. In this section, we copy some of the verification projects that the AI Futures Project describe in Plan A, so they’re in an easy-to-read list:
Secure network gateways
One potential upgrade to this proposal would be to build more complex 'secure network gateways' that have more functionality than the network taps, and do random sampling and organize and randomly sample the logs of reproduction packets to send to the recomputation server in a robust way.
Cryptographic/ZKP verification
On top of this key effort to build the network-tap and recomputation centered approach, there is also significant funding (order of $100M) mobilized into R&D for other approaches, including software-only and cryptography-centric (e.g., ZKPs) approaches, and a similar pool for physical security R&D (order of $100M) that supplements the inference-only retrofitting solution.
Workload verification.
Our view is that workload verification is solvable in a similar way to what we discussed previously (evidence collection via network taps, and then partial recomputation of randomly selected reproducible packets)
Workload reproducibility mandate
Our guess that, in order to reach extremely high levels of assurance, which will be increasingly necessary in Plan A (more on this later), a reproducibility mandate, combined with complete (and increasingly granular) on-path evidence collection, paired with a partial recomputation-based verification algorithm, will be a viable approach.
Low-latency challenges and validity checks.
That being said, we are not confident that another approach (e.g., using low-latency challenges and validity checks) might not be able to reach the same levels of assurance at lower costs or higher levels of assurance at equal costs.
Packet-Based Verification: Increasingly granular on-path evidence collection
By default, computational steps execute with randomness, but if they can be forced to be reproducible (e.g., seeded randomness is fine), then we can have a predictable relationship between the granularity of the evidence collection and the assurance level of the verification algorithm. This is because partial recomputation with random sampling with some small percentage of the workload being recomputed has favourable scaling of detection probability as you decrease the size of the individual workload packets that you check the correctness of. The concrete regime we imagine for the first few years of the deal is to still use network taps for evidence collection, and partial recomputation of randomly selected reproducible packets to verify correctness. The same high level approach as inference-only, with smaller and smaller packet sizes over time by moving the network taps to lower levels of the backend network.
Workload approval.
Workload approval will be closely tied to the form of AI R&D rules that are in place (e.g., are they just experiment compute caps? Are there specific banned techniques? Are there specific required techniques?) and our overall baseline proposal for approval is for this to just be manually carried out by teams of auditors in the R&D clusters.
Anti-steganography measures on the declared workloads and data
Auditor compilation into gpu instructions, ideally with increasingly robust and secure programming languages.
AI assisted data filtering and checks
Other parts of a workload verification scheme.
Running evaluations that also get verified for correctness
Final model approval for release
leads to the model weights being downloaded from the opaque internal database and transported to the inference-only clusters. This process is extremely high stakes from a model weights theft perspective, so would need to be carefully designed and executed.
SL5 inference-only clusters
First SL5 inference-only clusters are online and start serving the first post-deal models that have been approved from the R&D clusters for broad public deployment. These clusters have the same basic verification approach as the retrofitted ones (network taps and partial recomputation) but with improved security properties. By the end of the year, around 40% of world compute (500M of the global 1.2B H100e) is in these new post-deal SL5 inference-only clusters.
Security and verification improvements
By 2033 we expect significant hardware security, and more general security properties of the data centers, verification regime and capabilities to have improved significantly.
New compute production facilities (fabs)
built from scratch with mutual monitoring measures and being colocated with supply chain and potentially even with final centralized datacenters they will fill to minimize attack surface.
Increases to the logging / evidence collection granularity
of the verification regime, especially of the R&D datacenters, with logging technology being compatible with extremely high bandwidth data paths on backend networking to reduce overhead.
Improved TEEs, secure boot, encryption.
Generally improved hardware security properties with more robust trusted execution, secure boot, encryption, reproducibility, etc., which are especially important for recomputation servers.
Inference only chips.
Special hardware designs for automatic on-device verification of certain workloads, in particular, hardcoded weights on some chips to verify their inference-only.
Fixed set.
Another more specific area for hardware improvements is to help with verification robustness, such as extremely high fidelity reproducibility, support for increasingly granular logging, and other more specific solutions like fixed set hardware designs (e.g., hardcoded weights might be a nice way to upgrade the inference-only verification).
Offline licensing.
Another nice property that could also help with mutually assured compute destruction (purely through cryptography) could be offline licensing.