Queues, Webhooks, and Background Work

The scenario: "When a transcript review completes, three things must happen — notify the candidate, update the search index, and post to a partner agency's system. Right now it's all in the controller action and the request takes 12 seconds. Fix it." This question tests one instinct: the HTTP request should do the minimum durable thing and return — everything else is background work.

Webhook vs polling vs queue — pick with reasons

When systems need to know something happened, there are three shapes. Polling: simple, consumer-controlled, but wasteful and laggy — fine for low-frequency checks. Webhooks: push, near-real-time, but now you own delivery problems — the receiver may be down, so you need retries, and receivers must verify signatures (HMAC over the payload) and handle duplicate delivery, because at-least-once is the only honest webhook contract. Queues: durable, absorb bursts, decouple producer from consumer speed — the right answer whenever the work must not be lost. The interview move is matching the tool to the requirement out loud: "The partner post must survive our process restarting, so it goes through a queue; the search index can rebuild, so it could tolerate less."

BackgroundService and its sharp edges

In-process background work in ASP.NET Core means BackgroundService (hosted services). Know the mechanics — a long-running ExecuteAsync loop honoring the CancellationToken for graceful shutdown — and the two classic pitfalls: it's a singleton, so you create a scope to resolve a DbContext (IServiceScopeFactory); and an unhandled exception in the loop kills the worker silently, so you catch-log-continue inside the loop. Then draw the boundary that shows seniority: in-process work dies with the process. App-pool recycle, deployment, scale-in — anything queued only in memory is gone. If losing it matters, it needs durable storage: a real queue (Azure Service Bus / Storage Queues) or a database table a worker drains.

The outbox pattern, interview-style

The killer follow-up: "You save the review to SQL Server, then publish to the queue — what if the publish fails after the commit?" You can't have a distributed transaction across a database and a message broker. The outbox pattern: in the same database transaction as the business change, insert a row into an Outbox table describing the message. A background publisher reads unsent rows, publishes them, marks them sent. The state change and the intent to notify commit atomically; delivery becomes at-least-once, so consumers must be idempotent — dedupe by message id. Saying "atomic commit, at-least-once delivery, idempotent consumers" as one connected thought is the whole answer.

Answers that fall flat

• Task.Run fire-and-forget from a controller, presented as the solution.
• Not knowing in-memory work is lost on recycle, or that webhooks deliver duplicates.
• Describing the outbox but missing why it exists (the dual-write problem).

Practice prompts

• Redesign the 12-second controller action aloud: what stays in the request, what queues, why.
• Explain the outbox pattern on a whiteboard in four boxes.
• You're receiving a webhook from a payment provider: list your first four implementation concerns.