H Company Unveils NeoMME: A New Era for Efficient Multimodal Document Retrieval

For readers tracking the shift, In the rapidly evolving landscape of AI, multimodal models that can understand both text and images are crucial. However, many existing visual document retrieval systems rely on repurposed generative vision-language models, leading to inefficiencies like redundant vision towers and unused causal decoders. These ‘hand-me-down’ architectures consume significant parameters and compute for tasks that primarily require robust representations.

Meanwhile, Enter NeoMME, a groundbreaking family of multimodal encoders released by H Company. Available in 260 million and 800 million parameter versions, NeoMME is designed from the ground up to be lean, efficient, and highly effective for visual document retrieval, fundamentally streamlining how AI processes complex documents.

Revolutionizing Multimodal Encoding with NeoMME

A Single-Tower Approach for Dual Modalities

The core innovation behind NeoMME lies in its unique single-tower architecture. Unlike its predecessors, NeoMME eliminates the need for a separately pretrained vision tower and a causal decoder. Instead, a single Transformer processes both multilingual text tokens and raw 32×32 RGB image patches through the same layers. This unified approach, trained from random initialization, drastically reduces parameter and compute overhead, making it far more efficient for tasks focused on representation generation rather than token generation.

Under the Hood: NeoMME’s Technical Prowess

NeoMME’s sophisticated design incorporates several advanced features:

• Text Processing: Text inputs are handled via an ALBERT-style factorized embedding, projected to the model’s width.
• Image Processing: Images are divided into non-overlapping 32×32 patches, then projected using a 2-layer MLP trained from scratch, avoiding complex patch-merging modules or separate vision towers like SigLIP2.
• Context and Attention: Both NeoMME models support an extensive 16,384-token context, sufficient for processing two standard 4K UHD images after patching. The architecture employs symmetric sliding-window attention for most layers, with global attention in every sixth and the final layer. Further enhancements include grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs.
• Efficient Tokenization: A newly developed, whitespace-unconstrained BPE tokenizer with a 131,072-entry vocabulary significantly reduces token count. On the FLORES-200 devtest across 14 target languages, it emits 44.4% fewer tokens than ModernBERT.

The precise parameter counts for the two models are 262,937,906 and 793,715,032 respectively.

Training for True Multimodal Understanding

For example, NeoMME’s pretraining strategy is crucial to its success. It’s trained as a masked diffusion denoiser over text, with the option to condition on visible image patches.

For multimodal segments, a higher corruption rate (0.30 to 1) is applied, compelling the model to genuinely interpret the visual content rather than relying solely on language cues. Cross-modal ablation probes confirm the effectiveness of this approach, showing significant improvements in masked-token accuracy when visible page patches are present.

Benchmarking Performance and Deployment Readiness

Impressive Retrieval Capabilities

The fine-tuned retrieval variant, NeoMME-Retriever, demonstrates remarkable performance. The 260M parameter model achieves 0.523 nDCG@10 on ViDoRe v3, a score that outperforms all other evaluated models below 800M parameters. Notably, it comes within 0.002 points of the 3.75 billion-parameter ColQwen2.5-v0.2, achieving comparable results with a model 14.4 times smaller! The 800M model further boosts performance to 0.556 nDCG@10.

However, the models exhibit weaker performance in text-only retrieval tasks on BEIR-15, which the authors attribute to differences in supervision scale during training.

Designed for Real-World Deployment

NeoMME is built with deployment in mind:

• Open Source: Every checkpoint is released under the Apache 2.0 license, ensuring accessibility and flexibility for developers.
• Hugging Face Integration: Day-zero support in Hugging Face Transformers means immediate integration into existing ML workflows.
• Exceptional Efficiency: The 260M model can index 51.3 pages per second on a single NVIDIA L40S GPU, nearly twice as fast as ColModernVBERT (26.0 pages/sec). Query encoding is also swift, taking only 78.3 ms on a CPU-only host.

Optimizing Storage for Large Indexes

Interestingly, While late-interaction indexes can be large (around 1.5 MB per ViDoRe v3 document in float32), NeoMME addresses this with innovative compression techniques:

• Hierarchical Token Pooling: Combined with int8 queries and documents, a pooling factor of 10 reduces storage to 39.0 kB per page (a 39.4x reduction) while retaining 99.16% of baseline nDCG@10.
• Binary Documents: A pooling factor of 8 with int8 queries and binary documents achieves an astonishing 6.0 kB per page (a 255.5x reduction), retaining 95.19% of the baseline performance.

These optimizations make NeoMME highly practical for large-scale document retrieval systems.

Key Takeaways from NeoMME

• NeoMME introduces a single, bidirectional Transformer for both text and raw image patches, eliminating separate vision towers and causal decoders.
• The 260M parameter NeoMME-Retriever achieves a competitive 0.523 nDCG@10 on ViDoRe v3, outperforming smaller models and matching much larger ones.
• Innovative token pooling and asymmetric quantization dramatically reduce index size from 1.5 MB to just 6 kB per page.
• While excelling in multimodal retrieval, text-only retrieval and frozen natural-image transfer are identified as areas for future improvement.

However, NeoMME represents a significant leap forward in efficient multimodal understanding, offering a powerful, deployable solution for visual document retrieval.

Expert Perspective

From an industry angle, the clearest signal around NeoMME Multimodal Encoder is how it may influence neomme. The story reads less like a one-day spike and more like a marker of broader movement.

The next phase will depend on how quickly teams, regulators, or customers react. In practice, that gives NeoMME Multimodal Encoder room to reshape expectations across models over the near term.

For readers focused on practical impact, the best next step is to watch what changes around text once attention turns into execution.

Frequently Asked Questions

Why does NeoMME Multimodal Encoder matter right now?

For readers tracking the shift, In the rapidly evolving landscape of AI, multimodal models that can understand both text and images are crucial.

What broader change could NeoMME Multimodal Encoder signal?

However, many existing visual document retrieval systems rely on repurposed generative vision-language models, leading to inefficiencies like redundant vision towers and unused causal decoders.

What should the market watch next around NeoMME Multimodal Encoder?

These ‘hand-me-down’ architectures consume significant parameters and compute for tasks that primarily require robust representations.Meanwhile, Enter NeoMME, a groundbreaking family of multimodal encoders released by H Company.

Source: https://www.marktechpost.com/2026/09/06/h-company-releases-neomme-a-family-of-260m-and-800m-single-tower-multimodal-encoders-that-drop-the-vision-tower-and-causal-decoder/