Unlocking Physical Understanding: MirroS’s Code-as-World AI Transforms Videos into Executable Physics Simulations

Bridging the Gap Between Pixels and Physical Reality

For readers tracking the shift, In the rapidly evolving world of artificial intelligence, video models have made incredible strides in predicting future frames or generating realistic footage. However, a fundamental challenge persists: truly understanding the underlying physics of a scene – the mass, contact, and gravity that govern interactions. MirroS is tackling this head-on with its groundbreaking ‘Code-as-World’ paradigm, a novel approach that transforms real-world videos into editable, executable physics programs.

Meanwhile, Instead of merely processing visual data, Code-as-World aims to recover the intrinsic mechanisms of a scene, creating a rich, verifiable representation that can be simulated, edited, and even used to generate invaluable training data with precise physical labels.

The Core Philosophy: Beyond Pixels

MirroS’s core argument is elegantly simple yet profound: pixels are evidence of a physical scene, not its fundamental ontology. Traditional video models might predict plausible frames, but they often lack a deep comprehension of the physical laws at play. They see the effect, not the cause.

In practical terms, Code-as-World shifts this perspective by representing a scene not as a collection of pixels, latent features, or descriptive captions, but as executable code. Specifically, it generates a `scene.json` file compatible with the MuJoCo physics engine. This allows an agent to verify the simulated world against the original video, and critically, enables anyone to modify and re-simulate the scene, opening up new avenues for research and development.

What is an Executable World Representation (EWR)?

At the heart of Code-as-World is the concept of an Executable World Representation (EWR), which is defined as a triple p = (C, E, A):

• Composition (C): This encompasses all the static and dynamic physical entities within a scene. It includes objects, their geometry, metric dimensions, mass, friction coefficients, and the omnipresent force of gravity. Floors and walls, for instance, are treated as static physical entities capable of supporting other objects and participating in collisions.
• Evolution (E): This component describes how the scene unfolds over time. It specifies initial states of objects, applied forces, contact points, collision events, termination conditions, and the overall duration of the simulation. Executing the EWR expands the composition into a full trajectory of states, showing how objects move and interact dynamically.
• Appearance (A): This covers all the visual aspects of rendering the scene, such as camera parameters, lighting conditions, material properties, background elements, and the desired frame rate. Crucially, changes to the appearance never alter the underlying physics of the scene, ensuring a clear separation between visual presentation and physical reality.

For example, In the current implementation, this triple is compiled into a `scene.json` file, which can then be executed by MuJoCo, leveraging either an animation engine for kinematic poses or a physics engine for forces and contacts.

The Agentic Discovery Loop: Learning from Reality

Recovering an EWR from a raw video is an intricate inverse problem. To solve this, MirroS employs an ingenious agentic loop, framing the process as an abductive search. This loop involves up to five iterative rounds, following a sequence of steps:

1. Propose: An initial hypothesis for the scene’s physical representation is generated.
2. Instantiate: This hypothesis is translated into a concrete, executable EWR.
3. Execute: The EWR is run within the MuJoCo physics engine, simulating the proposed scene.
4. Render: The simulated scene is rendered back into a visual format, generating synthetic video frames.
5. Verify: The rendered output is then rigorously compared against the original input video at key frames, scrutinizing RGB values, depth information, object masks, and trajectories.

That said, Discrepancies identified during the verification stage are aggregated into structured feedback (Δ), which intelligently guides the next revision of the hypothesis. If the allocated budget of five rounds is exhausted without an acceptable match, the hypothesis is rejected. This iterative, self-correcting process has demonstrated superior performance, outperforming traditional ‘Best-of-5’ independent sampling methods across various metrics, including visual alignment and object Intersection over Union (IoU).

Revolutionizing Training Data and Performance Benchmarks

One of the most significant benefits of Code-as-World is its ability to generate high-quality training data. The verified worlds produced by the agentic loop come with exact physical labels – information that real-world video inherently lacks. This enables a new form of supervised learning.

Interestingly, MirroS trained its models in two phases: initially on extensive image-space QA pairs, followed by a second phase applying GRPO (Guided Reinforcement Policy Optimization) to world-space VQA (Visual Question Answering) derived from over 2,500 executable worlds. This world-space reinforcement learning proved critical.

The Code-as-World-VL-9B model achieved an impressive 55.4 MRA (Macro-averaged Relative Accuracy) on the QuantiPhy-validation benchmark. This performance surpasses Gemini-3.1 Flash, which scored 54.8, and significantly outstrips other strong open-weight baselines. The ablation studies further highlight the impact of world-space learning: adding both world-space sources boosted the 9B model’s score from 50.9 to 55.4 MRA, demonstrating the profound value of physically grounded supervision.

Accessibility and Future Outlook

However, MirroS has made Code-as-World accessible to the research community, releasing the GitHub repository and two checkpoints – Code-as-World-VL-4B and Code-as-World-VL-9B – under the permissive Apache 2.0 license. These models are fine-tuned from Qwen3.5-4B and Qwen3.5-9B, respectively, and are deployable at the research and internal-prototype tier.

While Code-as-World represents a monumental leap, it currently focuses on rigid-body dynamics. Furthermore, the model itself does not learn the discovery loop, relying on the external agentic process. Nevertheless, this paradigm marks a crucial step towards equipping AI with a deeper, more actionable understanding of our physical world, paving the way for more intelligent and interactive systems.

Expert Perspective

A practical read on Code-as-World AI starts with scene. That is where the earliest effects are likely to show up if this development keeps building.

What happens next will come down to adoption speed, policy response, and execution quality. That combination could make Code-as-World AI a meaningful reference point across world.

For decision-makers, the useful lens is not the headline alone but how physical changes priorities once organizations have to respond.

Frequently Asked Questions

Why is Code-as-World AI important?

Bridging the Gap Between Pixels and Physical RealityFor readers tracking the shift, In the rapidly evolving world of artificial intelligence, video models have made incredible strides in predicting future frames or generating realistic footage.

What impact could Code-as-World AI have?

However, a fundamental challenge persists: truly understanding the underlying physics of a scene – the mass, contact, and gravity that govern interactions.

What should readers watch next with Code-as-World AI?

MirroS is tackling this head-on with its groundbreaking ‘Code-as-World’ paradigm, a novel approach that transforms real-world videos into editable, executable physics programs.Meanwhile, Instead of merely processing visual data, Code-as-World aims to recover the intrinsic mechanisms of a scene, creating a rich, verifiable representation that can be simulated, edited, and even used to generate invaluable training data with precise physical labels.The Core Philosophy: Beyond PixelsMirroS’s core argument is elegantly simple yet profound: pixels are evidence of a physical scene, not its fundamental ontology.

How does this relate to scene?

It connects because the article frames scene as one of the clearest areas where the topic may be felt in practice.

Source: https://www.marktechpost.com/2026/08/29/mirros-code-as-world-executable-world-representations/