VIDRAFT_LAB

Thank you! The training code is not open at this time, as the core technique is under patent application. What is public is everything needed to verify the result: the NPU weights, the B200 reference weights (b200_reference/), and evaluate_unseen.py.

Thank you, and thanks for reading the numbers closely. You are right that the question we cared about was whether error would build up over a long run, and the paired result on unseen text is our answer to that. The details of how the training stack handles precision and synchronization across chips are part of our patent application, so we will cover them in the technical paper rather than here. Both weight sets and evaluate_unseen.py are public, so the outcome can be checked independently in the meantime.

Sharing the first experiment you asked for.

Experiment 1 (compounding check). One chip, no synchronization, 200 steps, identical seed and identical batches. I compared per-step training loss between the NPU run (our manual backward pass) and a reference run of the same model and batches on CPU with fp32 autograd. Weights are expected to diverge, so the loss curve is the fair comparison.

Per-step loss (CPU fp32 / NPU):

step 0: 5.674 / 5.677
step 50: 2.240 / 2.227
step 100: 2.026 / 2.019
step 150: 1.580 / 1.557
step 199: 1.499 / 1.465

Mean absolute gap 0.012, max 0.037. The NPU curve is not above fp32 (it ends slightly below), and the drift stays tiny.

So over 200 steps the NPU manual backward tracks exact fp32 closely. The final gap is therefore not due to NPU arithmetic.

Experiment 2 (the synchronization-schedule control) is being measured separately with a fair apples-to-apples setup, and I will post those numbers once it is complete.

Thanks for the careful read.

Good suggestions, thanks. From the public comparison table, paired loss deltas (NPU minus GPU) are 0.03 at step 0, a maximum of 0.38 at step 100, and 0.17 at the end, with a mean of 0.18 across the matched steps. Relative to the GPU loss the gap grows, from 0.5% at step 0 to 68% at the end. These curves do not share the same learning dynamics, and we do not claim they do.

That gap comes from the 32-chip run’s sync scheme: parameters were averaged every few steps (DiLoCo style) instead of an all-reduce every step. Per-step arithmetic is a separate question, and we measured it directly in the thread above: on one chip with no sync, the NPU gradient is closer to fp32 than a CPU bf16 control on all 110 parameter tensors.

Thanks, the distinction is fair. The two numbers in my reply came from different checks: 6.6e-7 is a CPU fp32 check of our backward math (2-layer configuration), and the 0.9999 cosine is from an earlier LoRA block test. Neither was the per-step NPU measurement on this model, so we ran exactly the test you described.

46M model, 12 layers, one chip, one step, no sync, real TinyStories text, against CPU fp32 autograd, all 110 parameter tensors: minimum cosine 0.99978, relative L2 error median 0.96%, worst 2.21% (layer 9 Wq), worst max|dW| relative 2.56%.

Control on the same batch, CPU bf16 autograd vs fp32: median 1.67%, worst 3.42%. The NPU gradient is closer to fp32 than bf16 on all 110 tensors. So you are right that it is not “essentially exact”; it is reduced-precision arithmetic, and tighter than a standard bf16 training path. The chip’s internal precision is being confirmed, and the GPU run on the same DiLoCo schedule will follow.