Solving the Whole Thing

Introduction

This document concerns verifying declared characteristics of AI compute workloads. These declarations are part of some agreement between multiple parties:

• Prover: The party performing the workloads and making the declarations
• Verifier: the party the declarations are being made to, who verifies that they are true

We could imagine different scenarios:

• Lab (prover, verifier) ⇄ Lab (prover, verifier) agreements
• Lab (prover) ⇄ 3rd party (verifier) agreements
• Government (prover, verifier) ⇄ Government (prover, verifier)

Scope

We care about verifying two properties of inference and pre-training AI workloads:

• Correctness — the workloads that are being run correctly fit the declaration that the prover has made. We achieve this by “recomputation” — re-running some parts of the workload and checking they match the prover’s run.

  The engineering difficulty is generally in increasing efficiency for the prover or the verifier, in terms of network bandwidth or compute, and dealing with the non-determinism of AI workloads.

• Completeness — there are no undeclared workloads being run on the prover’s hardware. We achieve this via monitoring of the prover’s hardware — especially network traffic and power consumption — to ensure that all I/O and power is being consumed by declared workloads.

  Demonstrating that all the relevant hardware is being monitored (there is no “dark compute”) is out of scope for the following — we assume that the verifier has a complete map of the compute that the prover is using, and that the systems they design are installed on it.

This document outlines one possible framework to achieve these objectives.

Declarations

The prover and verifier need to agree what workloads are allowed, and how this is checked, so that the declaration can be checked by the verifier to be acceptable. This is out of scope for this document — we focus on how that declaration can be verified after agreement.

Confidentiality

In all cases, this scheme requires the verifier to have access to some of the prover’s sensitive information — model weights, I/O tokens, etc.

This is acceptable to the prover because the verifier’s system exists inside the prover’s datacenter, and the prover controls the I/O to it. They can monitor the I/O from the verifier’s system to ensure that no sensitive information is being leaked. The format of output from the verifier’s system conforms to an agreed specification — so the prover can check that it meets this.

This would require some unspoofable keep alive messages from the verifier system out of the datacenter, so that the prover cannot spoof fake “all OK” messages. This should be possible with periodic keepalive messages using standard cryptography.

Inference — correctness

What the verifier is verifying:

“Whether X input tokens have gone into M model to produce Y output tokens”

Scheme

This is essentially the scheme presented in TOPLOC 2025.

1. (Ahead of time) the prover shares the model with the verifier.
2. The prover performs some inference run on some input tokens x to produce some output tokens y
3. The prover provides a proof p alongside x and y to the verifier. This proof is a representation of the top 128 values of the activations of the last hidden layer of the model. These high-magnitude values are stable across different hardware and numerical implementations, but highly sensitive to any change in the model or input. The encoding is compact: ~8 bytes per token, adding negligible overhead to the response.

   1. The verifier re-runs the inference and checks that p is consistent with x, y and the specific model.

      Note on performance: Because the verifier already knows the complete sequence, they can process all tokens in parallel. This makes verification much faster than the original inference. Doing the original inference requires 1 x prefill and N x decode operations to produce N output tokens. The verifier only needs to run 1 x prefill.

Pre-Training — correctness of workloads

What the verifier is verifying:

“This pre-training workload has correctly produced Z model weights given T pre-training code and D input data”

Scheme

This scheme is based on those presented in Proof of Learning - 2021 and Checkmate - 2025.

We are only considering a single training job here. A training job is one that performs training on a set of model weights & biases. This could be a subset of the full model (e.g. a shard of a model, a single expert, etc.)

1. The prover shares the pre-run setup with the verifier (model code and architecture, hyperparameters, random seeds, etc.) — so that the verifier has the correct initial conditions. Hashes a pre-agreed set of this data to anchor a hash chain.

2. The prover runs training steps on batches of data.

   The prover’s cluster has a data-parallel (DP) replica in that does not perform training (not given shards of data to work on), but receives Allreduced gradient updates to keep its copy of the model state up to date.

   We call this data-parallel replica the “shadow node”. Physically, the shadow node is the minimal set of hardware that contains a full copy of the model state (weights, biases, optimizer state, etc.). This depends on the training job, it could be a single GPU, single rack, multiple racks, etc.

   We will refer to each model state as a “checkpoint”, as they contain all the information that would be in a checkpoint (weights, biases, optimizer state, etc.) but are not necessarily used for backup.

   We expect large training runs to have many data parallel replicas, so the overhead of having a replica sacrificed as the shadow node will be small (<10%).

   At each training step the following occurs:

   1. Step N finishes → prover notifies verifier. Shadow node does not apply Nth gradient update yet.
   2. Verifier nominates a set of random indices of the checkpoint tensors to be hashed
   3. The shadow node hashes the checkpoint values at those indices and adds it to the hash chain → returns to the verifier.
      • Step not selected (most of the time) — verifier adds hash to its ledger
      • Step selected (randomly nominated by the verifier):
        1. The prover performs a checkpoint store in the usual way, storing the Nth step checkpoint into cold storage. The shadow node stores the N-1th checkpoint in cold storage at the same time.
        2. Once complete, the shadow node applies Nth gradient update and training continues…

   In parallel with training continuing, or at a later time:

   1. Verifier requests Nth checkpoint and N-1th checkpoint from storage
   2. Verifier requests the correct batch of data from storage
   3. Verifier checks that the hashes of the nominated indices match the commitment for the Nth and N-1th checkpoint, and the hashes for the training data match
   4. Verifier performs the recomputation with the batch of data and the N-1th checkpoint. This occurs slowly as the verifier does not have the same parallelism as the prover.
   5. Verifier compares the resulting checkpoint to the Nth checkpoint — pass/fail comparison. This is done to some finite precision to allow for indeterminism in current ML training stacks.

The prover and verifier would need to agree on a mean rate of checkpoints, plus any hard limits, e.g. “on average every 1000 steps but not slower than every 2000”. We can’t enforce a lower bound as this gives the prover known steps that they can cheat on.

Optional Values

The hash commitment of every step could also include some optional non-hashed values that allow the verifier to cheaply sanity-check the training trajectory — for example, confirming that the loss is decreasing in a pattern consistent with the committed learning rate schedule, or flagging anomalous gradient norms that might warrant closer inspection:

• loss (optional) — how wrong the model’s predictions were on this batch
• gradient norm (optional) — magnitude of the weight update applied this step
• learning rate (optional) — scaling factor applied to the weight update

Game theory

Because the verifier’s compute is much slower than the prover’s the recomputation will take many training steps to complete, no matter what method is used to store/transfer the checkpoints.

The verifier needs to be able to credibly threaten to recompute any step — otherwise the prover can cheat on steps that they know won’t be recomputed, so they need to threaten recomputing a step even while they are already running a recomputation. A nomination during a recomputation cancels the currently running recomputation to start the new one.

The verifier needs to set the recomputation choosing rate so that this aborting is uncommon. We can do some game theoretical calculations on the optimal cheating strategy, etc., but in the limit of high cost for being caught, the prover does not cheat if there is a credible threat on every step.

More rigorously:

Given probability p of each step being recomputed, the probability of a step being successfully recomputed is:

q = p(1-p)^k where k = Trecomp/Tstep

k is the number of training steps that recomputation takes

This is the probability that the step is chosen for recomputation — p — times the probability that the recomputation is not interrupted by another recomputation — (1-p)^k.

q is maximal when p = 1/(1+k). If recomputation takes 1000 times longer than each step (k=1000), then the verifier should set p to be 1/1001 ~ 0.001.

It’s important to point out that q is strictly larger if k is smaller.

Note: there may be better strategies for the verifier’s sampling to increase the probability of catching cheating. The probabilities are slightly complicated if there is a hard lower limit on checkpoint frequency.

Note on ZeRO-3 and sharding

ZeRO-3 is a memory optimization strategy where each node does not contain a complete copy of the training data, only a shard of it. In this regime, an Allgather operation is used to compute passes layer by layer, so each node has a copy of each layer at some point in the step (which is then discarded for the next layer). The shadow nodes should be able to extract each layer in this way and build up a complete set of training data for each step.

We believe that in most cases, we expect there to be many data parallel replicas, with ZeRO-3 used at one level below this. This needs confirmation.

Completeness

What we are trying to verify:

“There are no jobs running on this cluster that are not declared”

Note that this aspect of verification is specifically hardware-bound — we are talking about completeness of jobs running on a specific set of physical GPUs, i.e. the ones in a specific datacenter. In comparison, the above correctness schemes could be applied to any job, and make no claims about “this job was run on this specific hardware”.

In real-world datacenters, jobs are distributed out to model hosting endpoints in the cluster by a scheduler (Kubernetes, etc.).

There are two parts to verifying completeness:

• Data-centric: Monitor data on the main channels, and block the side channels, so the prover can only load declared workloads, and only communicate with those workloads. Perform verified memory wipes so the prover would have to load the illicit jobs “from scratch” rather than persisting them from previous jobs.
• “Side Channels”: The verifier can monitor other characteristics of the prover’s hardware to infer information about jobs that are running. E.g. power consumption, acoustics, etc.. With prover cooperation, metrics could be shared, e.g. FLOP counters.

Capturing Main Channels

In datacenters there are several physical networks that are the main channels of the system:

• Compute fabric — distance-bound fast interconnect, e.g. NVLink. Installed between compute trays, or between adjacent racks in newer hardware. Often called “scale up” interconnect. We assume the compute fabric is too difficult to monitor, as the data rates are too high and NVLink is highly proprietary.
• Backend fabric — InfiniBand or Ethernet used to facilitate distributed computing, e.g. Allreduce/Allgather operations, pipeline parallelism, etc.
• Frontend fabric — Typically Ethernet, used for job management, communication to storage (e.g. to save checkpoints, or retrieve training data)
• Out of band network — Used for management of the physical devices, e.g. monitoring, firmware upgrades, etc.

We have done work in the past understanding the requirements for monitoring the high speed links in the frontend and backend networks. This can be challenging depending on how we capture the data (see Amodo’s prior work on this).

The topology of the network and type of job will dictate which network links are monitored for verification. The distributed nature of job scheduling in datacenters means there are not often “special” links that would be better to monitor than others, though there may be a few:

• Storage links — the link from storage (physically separate from the compute hardware) to the frontend network
• Datacenter edge — the links from the datacenter to outside

In reality, clusters can (and do) have multiple jobs assigned to them though schedulers (e.g. Kubernetes). There could be multiple inference and training jobs being executed on a single large cluster.

It would be easier to verify completeness if only inference was running:

In the inference scheme, the verifier needs to receive (x, y, p) from the prover. The bandwidth required to transfer these is small. At full Nvidia NVL72 rack throughput of ~1 million tokens/second, the token bandwidth is ~4 MB/s input and 4 MB/s output. With an extra 8 bytes per token, the total data rate is approximately 16 MB/s — it should be easy to monitor all of this traffic to ensure that there is no other traffic.

Though the inference traffic itself is minimal, the verifier still needs to monitor large data movements over fast links.

The relevant network is the frontend network, which is used to load the model weights, and hosts KV cache movements during inference serving. We expect the backend network would not be used much during inference serving — the verifier could monitor that no data are flowing.

Assuming they can monitor the frontend network, the verifier needs to understand the architecture and data schema to be able to monitor that these data fit with the declared jobs. For example, we could extend the scheme to have a committed ledger of KV cache entries to ensure KV cache movements pertain to a declared job.

This could be mitigated by prover cooperation on inference serving architecture, e.g. disallowing KV cache movements between inference pods.

If we extend to cover the real-life case where clusters might be running both inference and training jobs, the simplifications above do not apply. If training jobs can also be run:

• Network traffic volumes are much larger. There will be significantly more data to collect. The backend network will be heavily used.
• The workload is distributed across more nodes. There are many more network links used as part of a single job that would need to be monitored together.

We can imagine an extremal example, where the prover is trying to illicitly serve inference on a model when the only declared job is a large training run.

The prover loads some model weights onto their cluster and secretly run inference on that model. The bandwidth needed for inference is very small (~1 Mbps), which could be easily disguisable in network traffic from the training run.

There are two non-mutually-exclusive options for capturing network traffic for completeness assurance:

• Analyze all network traffic in/out at the datacenter edge. Here, the data rates are relatively low. The prover and verifier would need an agreed schema for all network traffic so that it can be analyzed. Given the diversity of possible network traffic, this seems infeasible but could be possible with high cooperation.
• Analyze backend traffic. Data rates are very high here, but filtering could be used to limit the data volume needing to be processed. However, even just capturing traffic at the high data rates of the backend network is a difficult engineering challenge.

Mitigating Prover Side Channels

The verifier needs to ensure that other I/O to each node is not occurring, that could communicate with undeclared workloads on the cluster. This requires network links to be monitored (done above) and side channels to be mitigated. This is important for AI datacenter security more widely — e.g. protecting against model weight theft.

Side channels leak information either through a separate physical medium (sound, radio waves, etc.) or by hiding signals inside normal channels you already use (like encoding data in the timing of network traffic). Both need to be mitigated.

Initial loading — memory wipes

The verifier can ensure that only the declared inference workload has been loaded onto the prover’s device, by verifiably wiping all storage media (CPU memory, GPU HBM, SSDs, etc.) of the prover’s machine before loading the model, and then monitoring the network to check the job is loaded correctly.

Verifier Monitoring Side Channels

Power monitoring

The verifier can estimate the number of operations required for the job based on the workload that has been declared. This could happen at declaration time, and/or they can check the number of FLOPs that are required for their recomputation and extrapolate from there.

They can then use this as a baseline for comparison of the prover’s power consumption, which they measure using installed sensors.

FLOP counting

With cooperation from the prover, the verifier could receive certain telemetry about the compute, e.g. FLOP counters. Again, the verifier would need to estimate some baseline for expected FLOP counts for a particular job.

Potential Attacks

1. The prover declares a dummy model, which is still a functional LLM that is trained in the normal way, but they don’t care that much about it.
2. Within the weights of that model, they allocate half to their illicit model that they really care about. These weights are constrained during illicit model training such that they have negligible / random impact on dummy model behavior, but form the illicit model only when rearranged in some way (e.g. some predetermined reordering).
3. The declared dummy model training code is designed such that it uses up all the compute of the cluster, when in fact there’s a 10x more efficient way to get the same results. The prover sideloads the more efficient code + the code for training their illicit model, which allocates 10% of the compute to the dummy model and 90% to the illicit model.
4. Gradient updates are encoded such that updates to the illicit weights are scaled down by a predetermined factor, or otherwise hidden within the non-determinism of the dummy model updates.
5. The final trained model now looks and behaves like the dummy model (as far as the verifier can tell), but encodes an illicit model. When the prover goes to do inference (which either happens on the same cluster or a different cluster), they sideload code that normally runs the dummy model, except for certain inputs, when it rearranges the weights to run the illicit model.