Inside ByteDance's Video Models: Building a Generative MLLM Planner from Scratch

1. Connecting the Dots: From 200B Rumors to Latent Semantic Planning

When ByteDance recently showcased their video generation models (Seedance / Seaweed), the community was quick to notice their impressive physics consistency and prompt adherence. Naturally, rumors started swirling—with some speculating that ByteDance might be running massive models scaling up to ~200 Billion parameters.

While 200B remains an unverified rumor, it sparks a fascinating technical question: If a team actually wanted to scale visual generation to that magnitude, how would they do it without diffusion models becoming unstable or compute-impossible?

A close look at ByteDance’s published research—specifically Bernini (Latent Semantic Planning for Video Diffusion via Multimodal LLMs)—suggests a plausible answer. Rather than scaling an all-in-one monolithic pixel diffusion model, they explore decoupling the thinking from the painting: using an MLLM as a high-level semantic planner and a lightweight flow model as the renderer.

Intrigued by this concept, I wanted to see if this architecture holds up in practice. I built Bernini-MNIST—a minimal, toy reproduction to understand and demonstrate how this 2-stage generative paradigm works from the ground up.

Weights & Code: 🤗 Hugging Face Hub · GitHub Repository

2. The Core Idea: Split Planning from Painting

Traditional diffusion models try to do everything in a single neural network: understand text, plan the scene layout, simulate physics, and paint high-res pixels.

Bernini splits the task into two specialized stages:

1. The Planner MLLM: A Large Language Model (Qwen3-0.6B) works in abstract visual space. Given a prompt (e.g. digit class 8), it plans a sequence of 16 continuous visual tokens representing a $4 \times 4$ spatial grid.
2. The Diffusion Renderer: A lightweight convolutional denoiser (ConvResnet) takes those 16 planned tokens and paints the final clean pixels.

3. How the Planner Works: ViT + Qwen Attention

Instead of discrete tokens (like VQ-VAE codebooks that round values and cause visual glitches), Bernini uses continuous visual vectors extracted from a Vision Transformer (ViT):

• A pretrained Vision Transformer splits an image into a $4 \times 4$ grid ($16$ continuous vectors).
• The sequence [Class Prompt, Visual Tokens 1..16] is passed into Qwen’s self-attention layers.
• Qwen handles all the heavy lifting: learning how the patches relate to each other in 2D space.
• The output head on top of Qwen is just a lightweight MLP that predicts velocities for continuous diffusion.

4. How Inference Works: Step-by-Step Masked Refinement

When generating a new image from scratch, the system uses an iterative refinement loop based on MaskGIT:

1. Step 1 (All Masked): All 16 token slots start as [MASK]. Qwen looks at the class label and predicts a rough draft for all patches.
2. Step 2 (Commit Core Strokes): The model commits the high-confidence center tokens (e.g. the solid body of the digit) and keeps peripheral patches masked.
3. Step 3 (Context-Aware Boundary Polish): The committed center tokens are fed back into Qwen’s attention. Because Qwen can see the committed center, it easily predicts the matching boundary curves.
4. Step 4 (Final Painting): Once all 16 continuous tokens are unmasked, they are sent to the 2D ConvFlow Denoiser, which denoises random noise into clean pixels in 30 steps.

5. Visual Results

Let’s look at how the generated samples compare to real digits:

Real vs. Reconstructed vs. Generated

• Row 2 (Reconstruction): Proves that 16 continuous tokens ($4 \times 4$ grid) carry all the geometric detail needed to paint clean, solid digits.
• Row 3 (Pure Generation): Shows that Qwen successfully generates smooth, unbroken handwritten digits from scratch with zero background noise.

The 10×10 Generation Matrix

• Notice the natural diversity across samples: different stroke slants, loops, and handwriting curvatures for every digit class.

Continuous Denoising Trajectory Gallery (Classes 0–9)

To see the continuous flow matching process in action, here is the step-by-step denoising trajectory for every digit class from 0 to 9 across 30 Euler ODE steps (from pure Gaussian static at $t=0.0$ on the left to clean digits at $t=1.0$ on the right):

6. Takeaways

This 2-stage generative paradigm shows why ByteDance’s vision stack is so compelling:

1. Planning vs. Painting: Letting LLMs handle structure and reasoning while letting diffusion handle pixels makes scaling to hundreds of billions of parameters feasible.
2. Continuous Latents: Working in continuous float feature space avoids codebook quantization artifacts.
3. Simplicity: The LLM handles cross-token attention, while the flow head and renderer stay lightweight and fast.

Check out the full repository and open-source checkpoints:

• GitHub Repository: https://github.com/ruwwww/bernini-mnist
• Hugging Face Model Hub: https://huggingface.co/ruwwww/bernini-mnist

References

• Bernini: Latent Semantic Planning for Video Diffusion via Multimodal Large Language Models (ByteDance Research) · Official Code
• MaskGIT: Masked Generative Image Transformer (Chang et al., CVPR 2022)
• Flow Matching: Flow Matching for Generative Modeling (Lipman et al., ICLR 2023)