Beyond Text — Multimodal AI

Most of this course has treated AI as a text machine: prompt in, tokens out. Production systems rarely stay that tidy. Users paste screenshots, upload scanned PDFs, leave voicemails, and expect the system to cope. Multimodal AI is the umbrella term for models and services that work across modalities — text, images, audio, and increasingly video — and it's core AI-102 exam territory.

What multimodal LLMs can actually do

Modern frontier models (GPT-4o-class on Azure OpenAI, and comparable models elsewhere) accept vision input natively: you send an image alongside your text prompt and the model reasons about both together. This is not classic OCR bolted onto a chatbot — the model understands layout, charts, handwriting, and the relationship between visual elements and your question.

For an architect, the everyday use cases are surprisingly close to home:

• Reviewing architecture diagrams — paste a whiteboard photo and ask "where's the single point of failure?" The model reads boxes, arrows, and labels as a system, not as pixels.
• Reading dashboards and error screenshots — a support engineer forwards a screenshot; the model extracts the error code, the failing service, and the timestamp in one call.
• Interpreting documents with visual structure — tables, forms, and stamped scans where meaning lives in position, not just words.

Audio is the other production-grade modality: models can take speech in and produce speech out, either through dedicated speech services (Lesson 3) or through natively speech-capable models.

Multimodal LLM vs specialized service

The exam — and real budgets — turn on one decision: when do you use a general multimodal LLM, and when do you reach for a specialized cognitive service like Azure AI Vision or Azure AI Document Intelligence?

Use the multimodal LLM when the task needs reasoning over the image: answering open questions, comparing a diagram to a stated requirement, summarizing a chart. Use the specialized service when the task is extraction at scale with predictable structure: OCR on ten thousand invoices, reading the same form fields every time, or when you need confidence scores, field-level outputs, and per-page pricing that's an order of magnitude cheaper than LLM tokens. Specialized services are also easier to validate — a deterministic schema out, not free-form prose.

A robust enterprise pattern combines them: a document service extracts structured fields cheaply, and the LLM reasons over the extracted text.

Where LyraLearn fits

LyraLearn is already multimodal in one direction: every lesson you're reading is narrated by a TTS model — a real speech pipeline in this very platform, covered in Lesson 3. The same interface-first architecture applies: a vision or document capability would land behind its own service interface, keeping the choice between an LLM call and a cognitive service a configuration decision, not a rewrite.