# Multimodal Input

> Combines different data types (images, text, tabular) into a unified model

- **Category**: Machine Learning
- **Subcategory**: Data and Feature Representation
- **Canonical URL**: https://designpattern.fyi/machine_learning/multimodal-input/

---

## Description
**Intent**: Handle prediction problems that mix genuinely different data types where useful signal is spread across all modalities, not contained in any single one.

**Context**: Real-world problems often involve images, free text, structured/tabular fields, and categorical metadata simultaneously. The predictive signal is distributed across these different modalities.

**Solution**: Each modality gets its own suitable representation (embedding for text/categorical, convolutional or pretrained backbone for images, normalization for numeric fields). These representations are then fused at some point: early (concatenated near input), late (combined near output), or hybrid. Also handles metadata about the data itself (device, process) as additional input features.



## Use Cases
- Product recommendation using images, descriptions, and metadata
- Medical diagnosis combining imaging, clinical notes, and lab results
- Content moderation using text, images, and user metadata
- Any task where predictive signal spans multiple data types






## Trade-offs


### Advantages

- Leverages signal from multiple data types

- More accurate than single-modality approaches

- Flexible fusion strategies (early, late, or hybrid)

- Can handle missing modalities with learned defaults




### Considerations & Drawbacks

- Adds architectural complexity

- Needs more data to train well

- Harder to determine which modality drives predictions

- Risk of one modality numerically dominating others







---
**Reference**: [Original Source](https://github.com/GoogleCloudPlatform/ml-design-patterns)