Text, images, audio, and video rarely appear alone. Multimodal learning trains models on more than one type of data at once, letting them connect a caption to a photo or a spoken command to a visual scene.

Alignment is the hard part. The model must learn a shared representation where a picture of a dog and the word dog land near each other. Contrastive objectives like CLIP pull matching pairs together and push mismatched ones apart.

Common multimodal tasks

• Image captioning
• Visual question answering
• Text-to-image generation
• Speech-to-text with visual context
• Video understanding

Data collection is expensive. Paired datasets are scarcer than single-modality ones. Web-scraped image-text pairs help, but they carry noise and bias. Building a robust multimodal system often requires substantial curation.