SKILL.md
343 lines · 1,596 words
Multimodal AI Builder
Overview
Multimodal AI systems process and generate content across multiple data types -- text, images, audio, and video -- within unified architectures. Building production multimodal pipelines requires understanding modality-specific preprocessing, fusion strategies, model selection trade-offs, and serving infrastructure.
Multimodal Architecture Patterns
Pattern 1: Late Fusion (Separate Encoders, Combined Output)
+----------+ +-----------+
| Text | --> | Text Enc | --+
+----------+ +-----------+ | +----------+ +--------+
+----------+ +-----------+ +--> | Fusion | --> | Output |
| Image | --> | Vision Enc| --+ | Layer | | Head |
+----------+ +-----------+ | +----------+ +--------+
+----------+ +-----------+ |
| Audio | --> | Audio Enc | --+
+----------+ +-----------+
Pattern 2: Early Fusion (Shared Tokenization)
All modalities -> Unified Tokenizer -> Shared Transformer -> Output
Pattern 3: Cross-Attention Fusion
Text Encoder <--cross-attention-- Vision Encoder features