Integration Patterns

An AI feature is only as good as the content behind it, and that content lives in other systems — a CMS, SharePoint, a partner API, your own transactional tables. The architect's job is to keep the AI's knowledge fresh, consistent, and cheap to maintain as those sources change. That means choosing the right integration pattern for each kind of change, rather than re-embedding the world every night and hoping.

Syncing external content: webhooks plus reconciliation

There are two ways to learn that source content changed, and a robust system uses both:

• Webhooks (push) — when the source supports it, subscribe to change events. A lesson is edited in the CMS, the CMS fires a webhook, and LyraLearn enqueues a re-index of just that lesson. This is low-latency and surgical.
• Nightly reconciliation (pull) — webhooks get missed: deliveries fail, services are down, events are dropped. A scheduled job walks the source, compares a content hash or LastModified timestamp against what's stored, and queues anything that drifted. This is the safety net that guarantees eventual consistency.

Webhooks keep you current minute-to-minute; reconciliation guarantees you're never quietly stale. Neither alone is enough for an enterprise.

Durable async work: the outbox and a queue

Re-indexing is slow and failure-prone — it chunks text, calls an embedding model, and writes vectors. You must never do that inline on a web request, and you must never lose the work if a process restarts mid-batch. The pattern is the transactional outbox:

1. The webhook handler (or reconciliation job) writes an outbox row — ReindexLesson(lessonId) — in the same SQL Server transaction as any business update. The intent to re-index is now as durable as the data itself.
2. A background ingestion worker polls the outbox (or a queue fed from it), processes each message, and marks it done only after the embeddings are committed.
3. Failures retry with backoff; poison messages move to a dead-letter table for a human.

Because the outbox commits atomically with the data, you can never end up with updated content and a lost re-index, or vice versa. The pipeline is at-least-once and idempotent — running a message twice produces the same vectors.

Re-process only what changed

The cardinal efficiency rule: embed deltas, not everything. Embedding is the expensive step, so before re-indexing, compare the new content hash to the stored one — if it matches, skip the work entirely. Chunk content stably (by heading or paragraph) so an edit to one section only invalidates that section's chunks, leaving the rest untouched.

In LyraLearn, editing a single lesson's markdown fires one webhook, writes one outbox row, re-embeds only that lesson's changed chunks, and leaves the other thousands of vectors alone. Costs stay proportional to change, not to catalog size — which is exactly what makes the system affordable to keep current.