← All case studies

Correcting a severe hidden scaling defect in 4-bit quantized AI inference

4-bit quantization can reduce memory use. We found missing scale factors that severely distorted an expert-network calculation. In a controlled component test, a host-side correction brought the output close to the runtime’s quantized reference.

4-bit correctness and speed
Repair illustration · * Earlier OCR test

One plot starts with an active gray line exactly over a fixed gray 16-bit baseline. With the label 16-bit baseline, the active line first moves away while still gray, then gradually turns red as its distance from the baseline grows. As the line turns red, the label crossfades to 4-bit quantized Before fix using the same color progress. The before label stays while the line contracts in red. As the line turns blue near the baseline, Before fix fades out and After fix fades in using the same progress. Each text transition finishes with its matching line color. The 4-bit quantized prefix stays in place and remains blue in both states. The background gray reference stays fixed throughout. Path and sample points move together without disappearing or flashing between states. The loop also transitions smoothly from corrected blue back to the initial gray baseline. Normal quantization differences remain in the corrected result. All curves and transitions come from one simplified numerical example, not model measurements. The lower Performance comparison stays static. Number accuracy displays numeric/code F1 to one decimal place: 100.0% for 16-bit and 99.9% for 4-bit. The original scores are 100.00% and 99.89%. The asterisk on 4-bit after fix identifies scores and timing from a separate, earlier OCR test of the BF16 and NVFP4 base models on the same locked 30-page holdout, not measurements after this component repair. Mean processing time was 16.73 seconds per page for 16-bit and 6.00 seconds for 4-bit. The ten-second loop gives each main movement about two seconds. Pause, offscreen state, and page visibility freeze both curve motion and text fades. Reduced motion shows the final corrected output with the 16-bit baseline.

Project
Efficient local AI inference
Key outcomes
  • Isolated a scaling defect in a quantized expert-network computation.
  • Reduced relative L2 error to below 1% against the runtime’s quantized reference in the component test.
  • Demonstrated a host-side repair and specified proposed kernel changes for both prompt processing and token generation.

Background

Bringing a capable AI model onto local hardware means working within memory and response-time limits. Quantization can reduce memory use by representing values at lower precision. Some information can be lost in that conversion. This case concerns a separate problem: the runtime did not carry out the intended 4-bit calculation correctly.

A separate OCR comparison showed the practical value of 4-bit inference. On the same locked 30-page holdout, the Qwen3.6-35B-A3B base models scored 100.00% numeric/code F1 with BF16 and 99.89% with NVFP4. The homepage labels this Number accuracy and shows 100.0% and 99.9%, rounded to one decimal place. Mean processing time was 16.73 seconds per page and 6.00 seconds per page, respectively. These measurements used vLLM on NVIDIA DGX Spark; they were not measurements of the component repair described below. This score compares text items that contain digits, such as amounts, dates, and product codes. It accounts for missing and extra items, but does not check their position. The score is not the percentage of pages read without errors.

During our OCR work, we investigated a mixture-of-experts model, which routes token representations through selected expert networks. The affected expert-network path used four-bit floating-point values with higher-precision scale factors. Its runtime combined several operations in a kernel to execute that work efficiently.

Challenge

The optimized expert-network output differed sharply from the runtime’s quantized reference calculation. We needed to establish whether this was expected numerical variation or a defect in how the runtime executed the quantized operations.

The cause was hidden in how scale factors passed between operations. Activation quantization used a global factor for each expert, but a later fused calculation omitted that factor. Tests with all activation scale factors set to one could not expose the omission. An intermediate nonlinear operation then acted on values at the wrong scale. Multiplying the final output by a correction factor could not recover the intended calculation. The repair had to restore the scale at the correct point inside the execution path.

Solution

We followed the scale values through model conversion, the host interface, activation packing, and the fused calculation. A separate execution path that retained the factors helped establish the expected behavior.

We then compared the original path, an instrumented control, and a correction that folded the missing factors into coefficients supplied by the host. The comparison held the recorded inputs, expert routing, and source scale arrays constant. All measured paths used the same unmodified kernel archive, and independent processes repeated the experiment.

This experiment covered the first prompt-processing call of one layer’s routed expert computation in a four-layer model derivative. It did not measure whole-model OCR accuracy or validate every layer and token-generation step.

The corrected output was checked against the runtime’s quantized reference. We also built a reference that deliberately omitted the same factors. The original output closely matched this deliberately altered calculation. Together with a separate activation-packing check, these controls tied the error to the missing scales.

Result

In the controlled component test, the corrected path reached a cosine similarity of at least 0.99995 against the runtime’s quantized reference, with relative L2 error no greater than 0.96%. The original path’s output had an L2 norm approximately 17,810 times that of the reference and a cosine similarity of approximately 0.371.

The demonstrated host-side correction restored close numerical agreement in the affected expert calculation. It provided a precise repair target for an execution fault that could distort later conclusions about OCR model quality.

We also specified proposed kernel patches for prompt processing and token generation, with explicit rules to prevent applying the scale twice. The measured component repair gave that work a defined target: restore the intended quantized calculation before drawing conclusions about the model’s recognition quality.

What would you like to achieve?

Tell us about your goal or the challenge in your way.

Let’s build a solution