Introduction: Revolutionizing Local AI Inference on Apple Silicon
For readers tracking the shift, In the rapidly evolving landscape of artificial intelligence, the ability to run powerful language models efficiently on local hardware is becoming increasingly crucial. Perplexity, a leader in AI innovation, has taken a significant leap forward by open-sourcing Lily, a highly specialized inference engine designed specifically for Apple Silicon. This new engine promises to unlock unprecedented performance for the Qwen3.6-35B-A3B model, demonstrating the power of tailored software and hardware integration.
Table of Contents
- Introduction: Revolutionizing Local AI Inference on Apple Silicon
- What is Lily? A Deep Dive into its Architecture
- The Strategic Advantage of Specialization
- Understanding the Qwen3.6-35B-A3B Model
- Engineering for Peak Performance: Prefill and Decode
- Impressive Performance Benchmarks
- Deployment and System Requirements
- Expert Perspective
- Frequently Asked Questions
- Key Takeaways
- Optimizing Prefill Operations
- Streamlining Decode Efficiency
- Why does Apple Silicon AI inference matter right now?
- What broader change could Apple Silicon AI inference signal?
- What should the market watch next around Apple Silicon AI inference?
What is Lily? A Deep Dive into its Architecture
Meanwhile, Lily is not just another inference engine; it’s a meticulously crafted, single-process runtime that forms the backbone of Perplexity’s Hybrid Compute. Built with a Rust layer for checkpoint loading and generation loop management, and featuring hand-written Metal kernels for model execution, Lily stands apart. Notably, it bypasses traditional frameworks like PyTorch and MLX in its execution path, allowing for maximum optimization.
Its design philosophy is rooted in narrow specialization: Lily is exclusively optimized for the Qwen3.6-35B-A3B model on Apple Silicon hardware. This deliberate focus is the cornerstone of its impressive performance argument, enabling it to extract every ounce of computational efficiency from the underlying hardware.
The Strategic Advantage of Specialization
In practical terms, One might wonder, why specialize when general-purpose stacks like MLX (which already offers Qwen implementations with advanced features) exist? The answer lies in performance.
While MLX aims for reusability across diverse architectures, Lily gives up this broad compatibility in favor of deep optimization. By integrating model structure, execution plans, and kernel selection directly into a single runtime, Lily eliminates overheads and unlocks efficiencies that general-purpose frameworks simply cannot achieve.
Understanding the Qwen3.6-35B-A3B Model
The Qwen3.6-35B-A3B model is a complex beast, housing 35 billion parameters but activating roughly 3 billion per token. Its sophisticated architecture includes:
- A router that scores 256 experts and selects eight, alongside one shared expert.
- A mix of 10 full-attention layers utilizing grouped-query attention (16 query heads, two KV heads).
- 30 Gated DeltaNet layers.
For example, This structure leads to three distinct workload patterns: uneven expert groups, attention over a growing KV cache, and a fixed-size recurrence, each requiring careful optimization.
Engineering for Peak Performance: Prefill and Decode
Lily’s performance gains stem from highly optimized strategies for both the prefill and decode phases of text generation.
Optimizing Prefill Operations
That said, The prefill phase, where the model processes the initial prompt, sees significant enhancements:
- Efficient Quantization Handling: The 4-bit checkpoint, originally 70 GB of bfloat16 weights, is compressed to 19.4 GB. Lily reconstructs these weights one tile at a time within the grouped GEMM, preventing the expanded array from flooding unified memory. This fusion alone boosted end-to-end prefill by 77.4% for a 512-token prompt.
- GPU-Resident Routing: Keeping the entire routing histogram, prefix scan, scatter, and block map within a single GPU command buffer eliminates CPU synchronization, adding an 89% performance gain for 512-token prompts in MoE layers.
- Enhanced Tile and Scan Operations: Moving to 32-row tiles and optimizing Gated DeltaNet scans further improved prefill speeds. Expert GEMMs account for approximately 90% of prefill time, making these optimizations critical.
Streamlining Decode Efficiency
The decode phase, where the model generates tokens one by one, focuses on minimizing data movement and maximizing parallel execution:
- Reduced Data Movement: Lily minimizes bytes moved per token, recognizing that bandwidth is the primary bottleneck in batch-1 decode.
- Concurrent Metal Passes: Real dependencies are recorded in concurrent Metal passes, allowing independent kernels to overlap and reducing the 555 sequential stages to launch 795 kernels efficiently.
- CPU Round-Trip Elimination: The selected token is written directly into the next step’s GPU-resident input slot, removing a costly per-token CPU round trip.
- Cache Optimizations: Coalesced cache reads significantly lifted key and value bandwidth. Grouped-Query Attention (GQA) packing improved decode by 23.8% at 32K context, while a fixed-block attention layout provided up to a 40.2% improvement at 128K context.
Impressive Performance Benchmarks
Interestingly, Benchmarked against MLX-LM on a 40-core, 128 GB M5 Max, Lily demonstrated substantial superiority:
- Prefill Speed: Lily averaged 4,156 prefill tokens/s compared to MLX-LM’s 3,388 tokens/s, representing a 1.23x speedup.
- Decode Speed: For decode, Lily achieved 170.0 tokens/s versus 126.4 tokens/s for MLX-LM, a 1.35x speedup.
At a 4K prompt and 4K context, Lily reached 5,749.9 prefill tokens/s and 186.6 decode tokens/s, outperforming MLX-LM’s 4,737.5 and 140.9 tokens/s respectively. Across various prompt lengths, Lily consistently delivered 1.12–1.42x faster prefill and 1.31–1.37x faster decode. Importantly, a teacher-forced check confirmed Lily’s high accuracy, with a perplexity only 0.04% higher and 96.35% identical top-ranked tokens.
Deployment and System Requirements
However, For those eager to experiment, Lily is deployable. A standalone demo is publicly available in the pplx-garden repository, offering greedy text generation via a minimal OpenAI-compatible HTTP API. Given the 19.4 GB 4-bit checkpoint, an Apple Silicon Mac with 32 GB or more of unified memory is recommended, though Perplexity’s Hybrid Compute product lists macOS 15+ with a minimum of 24 GB and 32 GB for optimal results.
Expert Perspective
From an industry angle, the clearest signal around Apple Silicon AI inference is how it may influence lily. The story reads less like a one-day spike and more like a marker of broader movement.
The next phase will depend on how quickly teams, regulators, or customers react. In practice, that gives Apple Silicon AI inference room to reshape expectations across prefill over the near term.
For readers focused on practical impact, the best next step is to watch what changes around decode once attention turns into execution.
Frequently Asked Questions
Why does Apple Silicon AI inference matter right now?
Introduction: Revolutionizing Local AI Inference on Apple SiliconFor readers tracking the shift, In the rapidly evolving landscape of artificial intelligence, the ability to run powerful language models efficiently on local hardware is becoming increasingly crucial.
What broader change could Apple Silicon AI inference signal?
Perplexity, a leader in AI innovation, has taken a significant leap forward by open-sourcing Lily, a highly specialized inference engine designed specifically for Apple Silicon.
What should the market watch next around Apple Silicon AI inference?
This new engine promises to unlock unprecedented performance for the Qwen3.6-35B-A3B model, demonstrating the power of tailored software and hardware integration.What is Lily?
Key Takeaways
- Lily is a Rust + Metal inference engine, purpose-built for the Qwen3.6-35B-A3B model on Apple Silicon, entirely bypassing PyTorch or MLX in its execution path.
- It achieves remarkable performance gains, averaging 1.23x faster prefill and 1.35x faster decode compared to MLX-LM on a 40-core, 128 GB M5 Max.
- Key prefill optimizations include GPU-resident expert routing (+89%) and dequantization fused into the grouped GEMM (+77.4%).
- Major decode improvements come from GQA packing (+23.8% at 32K) and a fixed-block attention layout (+40.2% at 128K).
- It is deployable with a public demo, requiring an Apple Silicon Mac with at least 24 GB (preferably 32 GB+) of unified memory.
