# Multimodal tokenization of images and audio explained

[Skip to content](#lm-inhoud)Network/[NL](/en/multimodale-tokenisatie-uitgelegd)EN[Hubhub.llmnet.nlCompare models on task, language, cost and license.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organization, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-tokenisatie-uitgelegd&text=Multimodal%20tokenization%20of%20images%20and%20audio%20explained)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-tokenisatie-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-tokenisatie-uitgelegd&title=Multimodal%20tokenization%20of%20images%20and%20audio%20explained)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-tokenisatie-uitgelegd&text=Multimodal%20tokenization%20of%20images%20and%20audio%20explained)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-tokenisatie-uitgelegd)[](https://www.reddit.com/submit?url=https%3A%2F%2Fleren.llmnet.nl%2Fen%2Fmultimodale-tokenisatie-uitgelegd&title=Multimodal%20tokenization%20of%20images%20and%20audio%20explained)[](#)

# Multimodal tokenization of images and audio explained

By Ivo Donker - 6 August 2026

Large language models were originally designed to process sequences of text characters. To make these models work with visual and auditory information, images and sounds must be translated into exactly the same abstraction: discrete or continuous sequences of tokens. Multimodal tokenization forms the bridge between analog or two-dimensional data and the sequential processing architecture of a neural network.

Where another article covers the fundamentals of [text tokenization](https://leren.llmnet.nl/en/tokenisatie-uitgelegd) via algorithms such as Byte-Pair Encoding in depth, this guide focuses specifically on how raw pixels and acoustic waveforms are converted. What mechanisms make it possible for a model to simultaneously read text, analyze images, and understand spoken language?

## Why multimodal signal conversion is necessary

A neural network such as the transformer processes information in the form of vectors that exchange information with each other via a specific series of steps (such as the attention mechanism). Textual input lends itself well to this because human language is inherently discrete: words and letters can be broken down into separate units. As soon as you work with images or audio, however, you're dealing with a continuous stream of data.

An image consists of a grid of pixels, where each pixel has specific color values (such as red, green, and blue). An audio file is a constant measurement of air pressure changes over time, measured thousands of times per second. If a neural network had to process every individual pixel or every individual audio sample as a separate element, its computational capacity would be overwhelmed immediately. An image of 1024 by 1024 pixels already contains more than a million data points.

To make these data streams manageable, the raw input must be compressed and structured. The goal of multimodal tokenization is not just data reduction, but also extracting meaningful features. Instead of recording that pixel (100, 200) has the color code RGB(240, 120, 50), the system learns to recognize a group of pixels as part of an edge, a texture, or an object.

Note: Multimodal tokenization does not replace the underlying information with text. It converts the source data into mathematical representations that are mathematically equivalent to how characters are processed. The model does not "see" letters when it reads an image, but processes vectors that represent the visual properties.

## Image tokenization: from two-dimensional pixels to vector sequences

Converting a two-dimensional image into a one-dimensional sequence of tokens follows a structured process. Where traditional computer vision used intricate hand-crafted filters, modern multimodal architectures often use an approach based on the vision transformer (ViT) or quantization networks.

### Patch embedding

The first step in processing an image is breaking down the entire image into a grid of small, uniform blocks, also called 'patches.' A common size for such a patch is, for example, 14 by 14 or 16 by 16 pixels.

After the image has been divided into a series of patches, these parts are flattened into a single vector of pixel values. This vector is then passed through a linear projection layer. This layer transforms the raw pixel values of the patch into a high-dimensional embedding vector that summarizes the visual information of that specific area. Because the spatial position of the patches is crucial for the context of the image (a patch of an eye must sit above the patch of a mouth), a positional embedding is added to each patch vector.

### Quantization and the codebook (VQ-VAE)

In some architectures, particularly models that also need to be able to generate images or store them very compactly, continuous patch embeddings are converted into discrete tokens. This process often uses a technique called Vector Quantized Variational Autoencoder (VQ-VAE).

- Encoder: A neural network converts the patches into a series of continuous vectors.

- Codebook (discrete vocabulary): The model has a pretrained collection of a fixed number of unique 'visual building blocks' (for example, 8,192 unique vectors).

- Quantization: For each generated continuous vector, the quantizer looks for the closest matching vector in the codebook. The index of that codebook vector becomes the final token.

In this way, a visual fragment is converted into a discrete number, similar to how a word in a text dictionary is assigned a specific number. Besides discrete tokens, many modern [multimodal models](https://hub.llmnet.nl/en/multimodale-modellen-overzicht) nowadays instead use continuous embeddings, where the quantization step is skipped to prevent information loss. This is closely related to how [embeddings](https://leren.llmnet.nl/en/embeddings-uitgelegd) can capture visual relationships directly in a continuous vector space.

## Audio tokenization: from sound wave to acoustic codes

Audio brings a unique challenge to tokenization: the factor of time. A standard CD-quality audio stream contains 44,100 samples per second. An audio recording of just ten seconds therefore already yields almost half a million data points. Without effective compression and tokenization, it's impossible to process long audio fragments within a context window.

### From time domain to frequency domain

Raw audio in the time domain (the waveform) is typically first transformed into a representation that better matches human perception of sound. This is often done by creating a spectrogram via a Fourier transform. Here, the signal is split into short time frames (for example, 25 milliseconds), and for each frame, the frequencies present are analyzed.

This spectrogram is in effect a two-dimensional image of sound (with time on the x-axis and frequency on the y-axis). From that point on, a similar patch-based approach can be applied as with image tokenization.

### Neural audio codecs and speech encoders

For direct processing of speech and music, modern systems use neural audio codecs or specially trained speech encoders. These networks drastically reduce the size of the data:

- Framing: The continuous audio stream is split into very short time blocks.

- Feature extraction: A convolutional network learns to filter important patterns such as pitch, timbral characteristics, and phonemes from the raw audio.

- Residual Vector Quantization (RVQ): To preserve audio quality and speech clarity without using enormous amounts of data, neural codecs often apply hierarchical quantization. A first token captures the coarse structure of the sound, while subsequent tokens add increasingly finer detail.

By reducing these steps, a second of complex audio can be reduced to a few dozen tokens, depending on the chosen compression level. This is of great importance for applications in [local speech-to-text processing](https://gids.llmnet.nl/en/spraak-naar-tekst-lokaal) and real-time spoken interaction.

## Comparison of tokenization methods per modality

The table below provides an overview of how the different modalities compare in terms of source data, the type of conversion, and the final representation for the model.

Modality | 
Source data | 
Conversion mechanism | 
Final representation | 

Text | 
Character strings / subwords | 
Byte-Pair Encoding (BPE) / WordPiece | 
Discrete vocabulary index | 

Image | 
Pixel grid (RGB) | 
Patch extraction + linear projection or VQ-VAE | 
Continuous patch embedding or codebook index | 

Audio | 
Waveform / spectrogram | 
Neural codecs / Fourier + RVQ | 
Sequence of acoustic/semantic frame tokens | 

## How different modalities come together in one model

At its core, a transformer model only processes vectors of a fixed dimension. Once images and audio have been converted into tokens, they must be presented to the neural network in a uniform way.

### Projection layers and alignment

Because the encoder of a vision system or an audio network often works with a different internal dimension than the central language transformer, projection layers are used. These layers (often simple linear transformations or small neural networks) project the tokens from the image or sound directly into the same vector space as the text tokens.

To ensure the model understands that the written text "dog" and a visual patch token of a dog refer to the same concept, alignment takes place during the training process. Models learn, through contrastive learning or joint language modeling, to establish meaningful relationships between the vectors of different modalities. Consult the explanation of [how a transformer works](https://leren.llmnet.nl/en/wat-is-een-transformer) for more background on how these internal attention layers process information.

### Shared versus separate vocabulary

There are two main architectures for setting this up:

In a separate vocabulary the text model has its own tokens, while an external visual or audio encoder converts the source data into continuous vectors. These vectors are added directly to the context window as 'soft tokens,' without taking the form of a discrete number from the text dictionary.

In a integrated (shared) vocabulary the model uses one large table of tokens in which both textual subwords and visual and auditory codebook indices are included. This allows the model to generate image and audio in exactly the same way as text, simply by predicting the next token index in the sequence.

## Impact on the context window and efficiency

Converting images and sound into tokens carries an important practical consequence: the amount of context space required grows enormously fast compared to text.

A written paragraph generally spans a few dozen to a few hundred text tokens. A single image processed via fixed patches, by contrast, can take up hundreds to even thousands of tokens, regardless of how much information the image actually contains. A uniform blue image at high resolution costs just as many tokens in basic processing as a complex diagram full of detailed data.

The same principle applies to audio. Although neural audio codecs achieve significant data reduction, every second of spoken language counts as a series of frame tokens. A spoken conversation of a few minutes can thus fill a model's context window just as quickly as an entire book chapter of text. Managing this is an important part of [context engineering](https://leren.llmnet.nl/en/context-engineering-uitgelegd) when designing multimodal applications.

## Practical implications and optimizations

Because processing visual and acoustic tokens directly affects a model's memory usage and response time, various strategies have been developed to handle this efficiently.

### Dynamic resolution and tiling

Instead of scaling every image to a fixed size (which can lead to distortion or loss of small details such as text on a document), modern visual systems often use dynamic tiling. A large image is divided into multiple tiles of a standard size, plus an extra low-resolution overview tile of the entire image.

If a user provides an image with a very high resolution, the system generates more tiles and therefore more tokens. This explains why processing costs and latency increase directly as the input resolution rises. To save tokens, it's often wise for applications to downscale images beforehand to the minimum resolution needed to preserve essential details.

### Downsampling and region selection

When processing longer audio streams or video footage, techniques are applied to reduce the token stream:

- Temporal downsampling: Skipping consecutive audio frames or video frames when little change occurs in the signal.

- Spatial pooling: Merging neighboring visual tokens when they represent repetitive or low-information areas (such as a uniform background).

- Focal attention: Having the model first scan an overview at low resolution, then process only specific regions at a higher token density.

By understanding these mechanisms, developers and users can better prepare their data. Reducing background noise in audio or cropping unnecessary margins on images directly reduces the number of tokens that contribute no relevant information, resulting in more efficient and accurate processing by the network.

## Further reading

- [Text tokenization explained: BPE and subword algorithms](https://leren.llmnet.nl/en/tokenisatie-uitgelegd)

- [Embeddings explained: from words to vector spaces](https://leren.llmnet.nl/en/embeddings-uitgelegd)

- [What is a transformer? The architecture behind modern AI](https://leren.llmnet.nl/en/wat-is-een-transformer)

- [Context engineering: effectively managing the context window](https://leren.llmnet.nl/en/context-engineering-uitgelegd)

- [Multimodal models: an overview of the latest architectures](https://hub.llmnet.nl/en/multimodale-modellen-overzicht)

- [Multimodal evaluation: how models perform on image and sound](https://benchmark.llmnet.nl/en/multimodale-evaluatie)

llmnet.nl - Course & education platform