Beyond the Lab: Liquid AI’s Pipette Benchmarks Real-World On-Device AI Performance

The Challenge of On-Device AI: Why Server Benchmarks Fall Short

The central development is this: In the rapidly evolving world of artificial intelligence, foundation models are becoming ubiquitous. However, a significant challenge arises when these powerful models are deployed from high-performance server environments onto everyday edge devices like smartphones, smart home gadgets, or automotive systems. Traditional model benchmarks, often conducted under ideal, full-precision server conditions, rarely provide an accurate prediction of how a model will truly perform in the real world on a constrained device.

Meanwhile, This discrepancy can lead to costly development cycles, unexpected performance bottlenecks, and ultimately, a subpar user experience. Recognizing this critical gap, Liquid AI has stepped forward with an innovative solution.

Introducing Pipette: A Holistic Approach to Edge AI Benchmarking

This week, Liquid AI proudly announced the open-sourcing of Pipette, a groundbreaking platform designed to accurately benchmark foundation models on edge devices. Developed in partnership with Artificial Analysis, an independent methodology validator, Pipette redefines performance measurement by treating on-device behavior not as an isolated model characteristic, but as a property of the entire deployed system.

In practical terms, Unlike conventional tools, Pipette’s fundamental unit of measurement is a “full configuration,” encompassing the model, its quantization, the specific runtime environment, and the actual device hardware. This holistic perspective is crucial for understanding true performance in real-world scenarios where all these elements interact.

Unpacking Pipette’s Comprehensive Benchmarking Suite

Pipette arrives with an impressive launch dataset, covering five critical on-device performance metrics across more than 1,000 distinct configurations. This extensive dataset spans over 30 foundation models, various quantization formats, and llama.cpp builds tailored for macOS, iOS, Windows, and Android. It also explores a wide range of context lengths, from 256 up to 8,192 tokens.

For example, Initial verified results provide concrete insights from popular consumer devices, including a MacBook Pro with M5 Max, an iPhone 17 Pro, and a Galaxy S26 Ultra. Future results are anticipated to include devices featuring AMD Ryzen AI Max+ 395 and Radeon 8060S.

It’s important to note how Pipette handles model quality. While performance metrics are measured directly on-device, quality evaluations are conducted separately on high-performance NVIDIA H100 80GB reference systems using benchmarks like IFBench, GPQA Diamond, and MATH-500. These quality scores are then carefully matched to the on-device performance runs sharing the same model and quantization, ensuring a comprehensive view.

Why Deployment Context is a Game Changer: Key Insights

That said, Liquid AI’s initial findings powerfully demonstrate how the deployment context dramatically influences performance, often leading to counter-intuitive results:

1. Context Scaling Divergence

• Two 350M models, Granite-4.0-H-350M and Granite-4.0-350M, showed vastly different decode throughput retention on a Galaxy S26 Ultra at Q4_K_M quantization. While Granite-4.0-H-350M maintained 78.4% of its throughput when scaling from 256 to 4,096 input tokens, Granite-4.0-350M retained only 33.8%. This highlights how seemingly similar models can behave drastically differently under varying context loads.

2. Speed vs. Memory in Sparse Activation

• The LFM2.5-8B-A1B model, despite activating only 1.5B of its 8.5B parameters per token, decoded 2.4x faster than Qwen3.5-4B and 2.6x faster than Ministral-3-3B-Instruct-2512 on the same phone at 2,048 input tokens. However, it still peaked at 5.29 GiB of memory usage because all expert weights need to be loaded. This illustrates that sparse activation boosts speed but doesn’t necessarily reduce peak memory footprint.

3. The Speed-Quality Trade-off

• On an iPhone 17 Pro with Q4_K_M, MiniCPM5-1B completed a specific workload 15.8% faster than LFM2.5-1.2B-Instruct. Yet, on the same artifacts, LFM scored 9.0 points higher on the MATH-500 quality benchmark. This emphasizes that optimizing for speed may come at the cost of quality, and vice-versa.

4. Task-Level Reversals Despite Similar Profiles

• Granite-4.1-8B and Ministral-3-8B-Instruct-2512 showed only minor differences (2.4% in decode throughput, 1.2% in peak RAM) on an M5 Max at Q4_K_M and 2,048 input tokens. However, Granite led IFBench by 7.3 points, while Ministral led GPQA Diamond by 14.0 points. This shows that near-identical system profiles can mask significant performance differences on specific tasks.

Ensuring Reproducible and Reliable Measurements

Pipette’s strength lies in its meticulous and transparent measurement methodology. Performance runs adhere to a published protocol, featuring fixed token shapes, greedy decoding, a discarded warm-up phase, and five measured repetitions. Crucially, before each timed repetition, a platform-specific check verifies thermal and load conditions, ensuring that only valid, uncompromised data is published.

Interestingly, Every submitted benchmark record includes comprehensive details: benchmark version, token shape, model artifact, quantization, runtime version and settings, and the specific device hardware and operating system. This level of detail guarantees reproducibility and allows for deep analysis of results.

Who Benefits from Pipette?

Pipette is designed to serve a broad spectrum of users across various industries:

• Solo Developers & Seed-Stage Startups: Can leverage the hosted dashboard and native apps without needing to set up complex infrastructure.
• Mid-Market Product Teams: Can run Pipette clients across their internal device fleets for robust testing.
• Large OEMs, Chip Vendors & Enterprises: Have the option to operate the entire Pipette pipeline behind their own firewalls for maximum control and security.

However, Industries poised to benefit include consumer electronics, smartphone OEMs, automotive, industrial and robotics, healthcare devices, financial services, and defense – essentially, any sector where latency, privacy, or connectivity demands on-device inference.

Practical Applications:

• Model & Quantization Selection: Make informed decisions before committing to a development sprint.
• SoC & Hardware Procurement: Validate vendor performance claims and optimize hardware choices.
• Regression Testing: Identify performance degradation when runtime, OS, or driver updates occur.
• Context-Length Capacity Planning: Accurately plan for how much context your device can handle.
• Independent Verification: Confidently verify performance claims from third-party vendors.

Accessibility and the Future of Edge AI

Liquid AI has released Pipette as Apache 2.0 licensed infrastructure, making its components (pipette-mgmt, pipette-clients, pipette-scores) freely available. This includes a public results dataset, a hosted dashboard, and native iOS and Android benchmark apps – all immediately accessible without waitlists. While the publication of community-submitted results is currently in beta, this open approach fosters collaboration and accelerates innovation in the edge AI ecosystem.

Meanwhile, By providing a transparent, reproducible, and holistic benchmarking suite, Pipette is set to become an indispensable tool for anyone developing or deploying AI models on edge devices. It empowers developers and enterprises to move beyond theoretical performance numbers and truly understand how their AI will perform in the hands of users.

Expert Perspective

From an industry angle, the clearest signal around on-device AI benchmarking is how it may influence performance. The story reads less like a one-day spike and more like a marker of broader movement.

The next phase will depend on how quickly teams, regulators, or customers react. In practice, that gives on-device AI benchmarking room to reshape expectations across device over the near term.

For readers focused on practical impact, the best next step is to watch what changes around model once attention turns into execution.

Frequently Asked Questions

Why does on-device AI benchmarking matter right now?

The Challenge of On-Device AI: Why Server Benchmarks Fall ShortThe central development is this: In the rapidly evolving world of artificial intelligence, foundation models are becoming ubiquitous.

What broader change could on-device AI benchmarking signal?

However, a significant challenge arises when these powerful models are deployed from high-performance server environments onto everyday edge devices like smartphones, smart home gadgets, or automotive systems.

What should the market watch next around on-device AI benchmarking?

Traditional model benchmarks, often conducted under ideal, full-precision server conditions, rarely provide an accurate prediction of how a model will truly perform in the real world on a constrained device.Meanwhile, This discrepancy can lead to costly development cycles, unexpected performance bottlenecks, and ultimately, a subpar user experience.

Source: https://www.marktechpost.com/2026/08/25/liquid-ai-open-sources-pipette-a-reproducible-benchmarking-suite-that-measures-on-device-models-quantization-runtime-and-hardware-together/