NVIDIA Audex: A Breakthrough Unified LLM for Audio and Text Without the “Text Tax”

NVIDIA Audex: A Breakthrough Unified LLM for Audio and Text Without the “Text Tax”

The bigger takeaway is simple: The world of large language models (LLMs) is rapidly expanding, with a growing demand for models that can understand and generate more than just text. While many multimodal LLMs emerge, they often come with a hidden cost: a ‘text tax’ where adding audio or visual capabilities degrades their core text intelligence.

NVIDIA, however, is challenging this norm with its latest release: Audex (Nemotron-Labs-Audex-30B-A3B). This innovative unified audio-text LLM is engineered to seamlessly integrate audio capabilities while preserving, and in some cases even enhancing, its foundational text understanding.

What is Audex?

Meanwhile, Audex is a sophisticated 30-billion-parameter Mixture-of-Experts (MoE) Transformer decoder, activating 3 billion parameters per token. Its powerful foundation is the Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM known for its hybrid Mamba-Transformer architecture.

What makes Audex particularly remarkable is its ability to handle both audio input and output, treating them uniformly within its system. This design philosophy is intentionally straightforward, aiming to avoid the common pitfalls of complex, multi-component multimodal architectures.

How Audex Achieves Unified Intelligence

The core strength of Audex lies in its elegant and simple design, which allows it to operate on standard LLM stacks like Megatron-LM for training and vLLM for inference, supporting an impressive context length of 1 million tokens. This unified approach is facilitated by three key components:

• Advanced Audio Encoder: Audex utilizes AF-Whisper, derived from Audio Flamingo 3, to process 16kHz sound inputs. This encoder effectively translates raw audio into a format the model can understand.
• Seamless Feature Mapping: Two-layer MLP adapters are used to project these audio features directly into the model’s existing text embedding space. This ensures audio and text are represented consistently.
• Expanded Vocabulary: To accommodate discrete audio output tokens, Audex’s vocabulary has been significantly extended from 131,072 to 205,312 tokens.

In practical terms, For audio generation, Audex employs distinct codecs: X-Codec2 for speech, operating at 50 tokens per second, and X-Codec for non-speech sounds, which processes more complex audio at 200 tokens per second, providing a larger token budget for rich soundscapes.

A Smart Training Curriculum

NVIDIA’s research team developed a unique multi-stage training curriculum that is crucial to Audex’s success in avoiding text degradation. Instead of pre-training with audio, Audex begins from a text-only SFT (Supervised Fine-Tuning) checkpoint. The training then progresses through specific stages:

1. Text SFT
2. Audio Warmup (with frozen text token embeddings to prevent quality loss)
3. Audio Generation
4. Audio Understanding

For example, This careful, sequential approach, combined with text-only Cascade Reinforcement Learning (RL) and multi-domain on-policy distillation (MOPD), ensures that text intelligence is maintained and even improved, while audio capabilities are robustly integrated. The training data itself is extensive, combining 157.4 billion audio tokens with 320.5 billion text tokens, covering a wide array of tasks from speech recognition to text-to-audio generation.

Benchmark Performance: A Glimpse at Audex’s Prowess

Text Intelligence: Matching the Backbone

Remarkably, Audex closely mirrors its text-only backbone in performance, scoring 86.4 on MMLU-Redux compared to the backbone’s 86.3. It even surpasses its backbone on IMO AnswerBench (81.1 vs.

79.3) and consistently outperforms comparably sized models like Qwen3.5-35B-A3B on several reasoning and alignment benchmarks. This demonstrates its success in sidestepping the typical ‘text tax’ seen in multimodal models.

Audio Capabilities: Leading and Unique

That said, On speech recognition, Audex leads other open models, achieving an average word error rate of 6.82 on the OpenASR leaderboard. While its performance on audio understanding benchmarks like MMAR and MMSU shows some gaps compared to the strongest audio LLMs, Audex holds a significant advantage: it is one of the few open models capable of generating general audio beyond just speech, opening up new possibilities for sound design and creative applications.

Practical Applications: Where Audex Shines

The unified nature of Audex unlocks a diverse range of practical applications:

• Multilingual Call Centers: Imagine transcribing a German call and instantly translating it into English, all through a single model. Audex can provide source language, transcript, and translation seamlessly.
• Enhanced Accessibility Tools: Developers can integrate high-quality, fixed-voice text-to-speech into reading applications, offering a low word error rate for clear audio output.
• Innovative Sound Design: From a simple text prompt like ‘birds chirping in a forest,’ Audex can generate a 10-second audio clip, revolutionizing prototyping and creative soundscapes.
• Advanced Voice Assistants: Audex can serve as a powerful backbone for voice assistants, handling speech-to-speech interactions efficiently, even if it operates in a cascaded manner for this specific task.

Strengths and Considerations

Key Strengths:

• No Text Regression: Successfully maintains or improves text intelligence compared to its text-only backbone.
• Unified & Compatible: A single model architecture compatible with standard LLM training and inference stacks.
• General Audio Generation: A unique capability among open models to generate diverse non-speech sounds.
• Strong Reasoning: Outperforms other open models on various reasoning and alignment tasks.

Points to Consider:

• Noncommercial License: Currently, Audex is released under an NVIDIA OneWay Noncommercial License, limiting its use in commercial projects.
• Audio Understanding Gaps: Shows some performance gaps in specific audio understanding benchmarks (MMAR, MMSU) compared to leading audio LLMs.
• Cascaded Speech-to-Speech: The speech-to-speech functionality operates in a cascaded manner rather than being natively full-duplex.
• Future RL Work: Reinforcement learning is currently text-only, with audio-text RL noted as future development.

Interestingly, NVIDIA’s Audex represents a significant leap forward in the quest for truly unified multimodal AI, demonstrating that advanced audio capabilities don’t have to come at the expense of robust text intelligence.

Expert Perspective

A practical read on NVIDIA Audex starts with audio. That is where the earliest effects are likely to show up if this development keeps building.

What happens next will come down to adoption speed, policy response, and execution quality. That combination could make NVIDIA Audex a meaningful reference point across audex.

For decision-makers, the useful lens is not the headline alone but how text changes priorities once organizations have to respond.

Frequently Asked Questions

Why is NVIDIA Audex important?

NVIDIA Audex: A Breakthrough Unified LLM for Audio and Text Without the “Text Tax”The bigger takeaway is simple: The world of large language models (LLMs) is rapidly expanding, with a growing demand for models that can understand and generate more than just text.

What impact could NVIDIA Audex have?

While many multimodal LLMs emerge, they often come with a hidden cost: a ‘text tax’ where adding audio or visual capabilities degrades their core text intelligence.NVIDIA, however, is challenging this norm with its latest release: Audex (Nemotron-Labs-Audex-30B-A3B).

What should readers watch next with NVIDIA Audex?

This innovative unified audio-text LLM is engineered to seamlessly integrate audio capabilities while preserving, and in some cases even enhancing, its foundational text understanding.What is Audex?Meanwhile, Audex is a sophisticated 30-billion-parameter Mixture-of-Experts (MoE) Transformer decoder, activating 3 billion parameters per token.

How does this relate to audio?

It connects because the article frames audio as one of the clearest areas where the topic may be felt in practice.

Source: https://www.marktechpost.com/2026/07/07/nvidia-releases-audex-nemotron-labs-audex-30b-a3b-a-unified-audio-text-llm-that-preserves-the-text-intelligence-of-its-backbone/