Native multimodal models: How an AI learns to see and read in one network
Native multimodal models train on text and images together in one network, rather than attaching a separate vision module. Here is how early fusion works and where the approach falls short.

A native multimodal model is an AI system trained from the beginning on several kinds of data, such as text and images, inside one network. It differs from a text model that later gets a separate vision module attached. The design is often called early fusion, because the different inputs are merged before the main processing starts.
The idea came up again when Alibaba’s Qwen team released Qwen4-Max, which it describes as having native vision (our news report).
Learning a language and seeing at the same time
Think of two ways to train a translator who must also describe photographs. In the first, you hire a language expert and a photo expert, then teach them to pass notes to each other. In the second, one person studies words and pictures together from day one.
The first approach is the older and more common one, usually called late fusion. A language model is trained on text. A separate vision encoder, a network that turns an image into numbers, is trained on pictures. A small connecting layer then translates between them, and the combined system is fine-tuned.
The second approach is the native one. There is no separate expert, only a single model that sees both kinds of data throughout training.
Turning pictures into the same pieces as words

Language models read tokens, which are small chunks of text converted to numbers. For a model to treat an image like text, the image must also become a sequence of pieces.
The standard trick comes from the Vision Transformer, introduced by Alexey Dosovitskiy and colleagues at Google in 2020 in the paper “An Image is Worth 16×16 Words”. It treats an image as a sequence of small patches, the way a sentence is a sequence of words.
In an early-fusion model, those patches, or discrete image tokens, are placed in the same sequence as the text tokens. One transformer, the network design behind most modern language models, reads the whole sequence. It does not need to know in advance which pieces are words and which are picture fragments.
From bolted-on eyes to single-network designs
Many well-known vision-language systems of the early 2020s used the late-fusion recipe, pairing a pretrained vision encoder with a language model. It was practical, because both parts already existed and could be reused.
Early fusion at scale drew attention with Meta’s Chameleon in May 2024. The Chameleon team described mixed-modal early-fusion models that process and generate images and text in any order, using a single token-based architecture. In the paper they reported competitive results against Mixtral 8x7B and Gemini-Pro, and strong human-evaluation results on mixed image-and-text generation.
A scientific comparison followed in 2025. Mustafa Shukor and colleagues at Apple and Sorbonne University trained 457 models with different designs and data mixes, in a paper titled “Scaling Laws for Native Multimodal Models” (ICCV 2025). Their finding was that late fusion has no inherent advantage. Early-fusion models, which skip the image encoder, performed better at lower parameter counts, and the authors describe them as cheaper to train and easier to deploy.
Who builds this way today
The term now appears in model announcements. Alibaba lists Qwen4-Max as a model with native vision, built on a mixture-of-experts design and a 1M-token context window. The announcement does not publish architectural detail, so how its vision pathway is built, and how it compares with a late-fusion design, is not public from that post.
The same Apple paper offers a hint of why the two ideas travel together. The authors report that adding mixture-of-experts layers let native models learn modality-specific weights, which improved performance significantly. In plain terms, different parts of the network can specialize in pictures or in words while still being trained as one system.
Other labs use the phrase for their own models as well. It is worth checking each vendor’s own documentation, because “native” is not a standardized label.
Limits and what to check next
Early fusion is not free. Higher-resolution images produce many more image tokens, which raises the cost of every request. The Apple paper treats this as an open problem.
The design also removes the safety net of a pretrained encoder. A late-fusion system can start from a vision model that already works well. A native model must learn vision during its own, much larger, training run, which demands large amounts of paired image and text data.
The findings come from controlled experiments at research scale. They do not prove that every native model beats every late-fusion one, and the Apple authors report the advantage is strongest at smaller sizes and lower compute budgets.
Several things are worth watching:
- Whether vendors publish architecture details, rather than only the word “native”.
- Independent tests of image reading, such as charts, documents and screenshots, rather than internal evaluations.
- How labs handle the token cost of high-resolution images.
- Whether native models extend to audio and video in the same network, which is the stated direction of work such as Chameleon.
For readers choosing a model, the practical advice is simple. A native design is a claim about how the model was built, not a guarantee of quality. Test it on your own images and documents.
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Daniel ReedAI Research, Safety & Ethics AnalystDaniel Reed currently works as an AI Research, Safety & Ethics Analyst at Aiholics, writing about how changes in artificial intelligence are affecting and will affect scholarship,…
You may also like

Apple’s humanoid runs Qwen on a Mac mini and clears 91% of real-room trials

ByteDance opens 17 GB sparse autoencoders to probe what Qwen3’s MoE experts do

GPT-6 Astra lifted a tiny Qwen model’s math score sixfold by editing its weights
