Calling LLMs from .NET

Sooner or later the interviewer drops out of architecture and into code: "Walk me through how you'd actually call a model from our ASP.NET Core app." This is a chance to show that an LLM endpoint is, to a .NET developer, a slow, expensive, occasionally flaky HTTP dependency — and that you already know how to treat those.

The shape of a good answer

Start with the client: a typed HttpClient registered via AddHttpClient<ITranscriptAiClient, TranscriptAiClient>(), pointed at an OpenAI-compatible or Azure OpenAI endpoint. That buys you DI, HttpClientFactory handler pooling, and one place to attach resilience policies. Then hit the operational beats out loud:

• Keys stay server-side. The API key lives in configuration (Key Vault or environment via IConfiguration), attached in the handler. Nothing model-related is ever called from the browser; Razor views talk to your controller, your controller talks to the model.
• Timeouts and retries. LLM calls can take tens of seconds — set an explicit per-request timeout, add retry with backoff for 429/5xx (Polly or AddStandardResilienceHandler), and make retried operations idempotent so a duplicate call can't double-write an analysis.
• Streaming. For chat-style UX, request server-sent events and forward tokens to the client as they arrive (IAsyncEnumerable<string> flows naturally through a controller). For batch transcript analysis, skip streaming — you want the whole structured result or nothing.
• Structured outputs. Ask the model for JSON constrained by a JSON schema (most providers support this), then deserialize into a C# record and validate it — required fields present, enums in range, scores between 0 and 1. If parsing fails, that's a retry-or-refuse path, never a "best effort" path.

The follow-ups they will ask

"What if it's slow?" — Separate interactive from batch. Interactive calls stream and show progress; transcript batches run in a background service (hosted service or queue) so a web request never waits minutes. Cache repeated prompts where results are deterministic enough, and measure latency percentiles, not averages.

"How do you test it?" — Two layers. Unit tests mock the typed client interface — controllers and services are tested with canned responses, including malformed JSON and timeouts. Model quality is tested separately with an evaluation set: real prompts, expected outputs, scored offline on every prompt or model change. Never let "the model is nondeterministic" become an excuse for zero tests.

Red flags

• Calling the provider from JavaScript with the key in the page.
• new HttpClient() per request, no timeout, no retry policy.
• Parsing free-text model output with string splitting instead of schema-constrained JSON.
• "We'll test it manually" — no mocked failure cases, no eval set.

Practice prompts

1. Sketch the TranscriptAiClient interface and its registration in Program.cs on a whiteboard.
2. The model starts returning 429s during a submission surge — narrate your mitigation, in order.
3. Explain to a reviewer why your integration tests don't call the real model — and what does.