Nunchux AI’s VC-Attention: Revolutionizing Video Diffusion Speed and Quality

Unlocking Faster, High-Fidelity Video Generation with VC-Attention

At a glance, The landscape of AI-powered video generation is rapidly evolving, with Diffusion Transformers (DiTs) leading the charge in creating stunning visual content. However, the computational demands of these models, especially for high-resolution and longer clips, have presented a significant bottleneck. Nunchux AI is addressing this challenge head-on with the introduction of VC-Attention, a groundbreaking, training-free, low-bit attention kernel designed to dramatically accelerate video DiTs while simultaneously enhancing output quality.

The Bottleneck: Why Video DiTs Are So Slow

Meanwhile, At the heart of modern video DiTs lies the self-attention mechanism, a powerful component that enables models to understand relationships between different parts of a video. To process video, these models flatten a clip into a long sequence of spatiotemporal tokens, then apply full self-attention at every layer.

This process is incredibly resource-intensive. For instance, a mere 5-second 720p video clip can generate around 70,000 tokens, with attention consuming over 64% of the total generation time on high-end GPUs like the RTX 5090.

While low-bit Tensor Cores have helped speed up the initial matrix products (QK and PV), two critical obstacles have remained:

1. Value Quantization Error: Previous methods often struggled with value outliers, leading to significant output errors.
2. Slow Softmax Stage: The softmax operation, vital for attention weights, typically runs in FP32 precision, becoming the longest pipeline stage on powerful accelerators like the B200 and H200 due to its exponential calculation and subsequent FP8 cast.

Introducing VC-Attention: A Dual Solution

In practical terms, VC-Attention directly targets these two persistent problems with a novel, two-pronged approach:

• V-Smooth: A mechanism to mitigate value quantization errors by effectively handling outliers.
• ExpCast-FP8: A method to bypass the slow FP32 softmax, speeding up the attention process.

Crucially, VC-Attention is a training-free solution, meaning it can be integrated into existing DiT architectures without requiring extensive retraining, making it highly practical for immediate adoption.

How VC-Attention Works: Two Key Innovations

V-Smooth: Taming Value Outliers

For example, Value outliers, concentrated in a few tokens and shifting across different heads, layers, and steps, are a major source of error in low-bit quantization. V-Smooth takes an innovative approach to manage these:

• Group: It employs an online k-means clustering technique to group value tokens per batch and head. This permutation ensures that non-causal attention output remains unaffected.
• Demean: Within each 128-token hardware block, the mean is subtracted. Only the residual values are then quantized using efficient per-channel E4M3 (8 bits) or NVFP4 (4 bits) formats.
• Restore: The subtracted mean is added back without the need for a second pass or an extra buffer, cleverly utilizing the row sum already maintained by online softmax.

This process significantly reduces block energy and quantization error, with grouping adding only a minor overhead (3-4% of attention time) when averaged over the full denoising schedule.

ExpCast-FP8: Accelerating the Softmax Bottleneck

That said, The FP32 softmax operation has long been a performance bottleneck. ExpCast-FP8 offers a direct and efficient alternative:

• Instead of performing a full FP32 exponential calculation followed by a cast, ExpCast-FP8 writes the FP8 byte directly from the log-domain score using a single fused multiply-add operation.
• This direct path achieves remarkable accuracy, matching the FP32 exponent-then-cast path on nearly 80% of values and landing just one code away elsewhere.

While ExpCast-FP8 currently applies to the 8-bit kernel, its impact on reducing the softmax latency is substantial, with custom CUDA fusion further cutting processing times dramatically.

Impressive Performance Benchmarks

Interestingly, Nunchux AI’s VC-Attention has demonstrated significant performance gains across a range of open-weight video DiTs, including Wan2.2-T2V-A14B, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3. Fidelity was meticulously measured against BF16 FlashAttention-4 outputs over 100 prompts.

• Attention Speedup: On an RTX 5090, VC-Attention achieved a remarkable 3.58x speedup for 4-bit attention. On the B200, 8-bit attention saw a 1.59x speedup.
• End-to-End Speedup: These attention improvements translated to substantial overall generation speedups, reaching 1.70x on the RTX 5090 and 1.19x on the B200.
• Superiority Over Competitors: VC-Attention proved significantly faster than prior methods like SageAttention2 (e.g., 6.02x faster on B200) while also delivering superior fidelity.

Beyond speed, VC-Attention consistently delivered improved fidelity. At 8 bits, V-Smooth added 2.3 dB PSNR over SageAttention2 on Wan2.2 and 2.8 dB on HunyuanVideo-1.5. Even with ExpCast-FP8, it still outperformed SageAttention2 across all four models. At 4 bits, V-Smooth surpassed SageAttention3 by 2.9 dB on Wan2.2 and 3.6 dB on LongCat-Video.

“VC-Attention is not just about speed; it’s about achieving higher quality video generation with unprecedented efficiency, all without the need for model retraining.”

Future Prospects and Availability

However, A key advantage of VC-Attention is its composability. Because it only alters per-interaction cost, it can seamlessly integrate with other advanced techniques such as sparse attention (e.g., Sparse VideoGen, Radial Attention), distillation methods, and multi-GPU execution setups. This ensures its compatibility with future innovations in AI video.

While the core VC-Attention technology operates as a proprietary extension within Nunchux AI’s stack, the company has announced that free access to MiniMax-H3 powered by this technology will soon be available through its Modelverse waitlist, signaling an exciting future for faster, higher-quality AI video generation.

Expert Perspective

From an industry angle, the clearest signal around VC-Attention is how it may influence attention. The story reads less like a one-day spike and more like a marker of broader movement.

The next phase will depend on how quickly teams, regulators, or customers react. In practice, that gives VC-Attention room to reshape expectations across video over the near term.

For readers focused on practical impact, the best next step is to watch what changes around softmax once attention turns into execution.

Frequently Asked Questions

Why does VC-Attention matter right now?

Unlocking Faster, High-Fidelity Video Generation with VC-AttentionAt a glance, The landscape of AI-powered video generation is rapidly evolving, with Diffusion Transformers (DiTs) leading the charge in creating stunning visual content.

What broader change could VC-Attention signal?

However, the computational demands of these models, especially for high-resolution and longer clips, have presented a significant bottleneck.

What should the market watch next around VC-Attention?

Nunchux AI is addressing this challenge head-on with the introduction of VC-Attention, a groundbreaking, training-free, low-bit attention kernel designed to dramatically accelerate video DiTs while simultaneously enhancing output quality.The Bottleneck: Why Video DiTs Are So SlowMeanwhile, At the heart of modern video DiTs lies the self-attention mechanism, a powerful component that enables models to understand relationships between different parts of a video.

Source: https://www.marktechpost.com/2026/09/16/nunchux-ai-introduces-vc-attention-a-training-free-low-bit-attention-kernel-that-speeds-up-video-diffusion-transformers/