LLM APIs: Remote Model Inference & Production Architecture

In previous modules, we ran lightweight models locally using Hugging Face Transformers. However, modern frontier models (such as GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash) contain hundreds of billions of parameters requiring distributed GPU clusters.

Instead of managing multi-million-dollar hardware clusters, AI Engineers consume models as an on-demand utility service over HTTPS:

The Restaurant Kitchen Analogy: You don't build a commercial kitchen in your house to eat dinner; you visit a restaurant. You hand the waiter an order slip (the API Request), the kitchen prepares the meal (the GPU Inference), and the waiter brings your food back to your table (the API Response).

An AI Engineer should never assume that all models reside within one vendor. Here is how the four major providers compare architecturally:

When generating a 300-word response, waiting for the entire text to complete creates an awkward 5-second silence. Server-Sent Events (SSE) keeps the HTTP connection open with Transfer-Encoding: chunked, emitting a continuous stream of text chunks as the GPU generates them.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

data: {"choices": [{"delta": {"content": "Server"}}]}

data: {"choices": [{"delta": {"content": "-Sent"}}]}

data: {"choices": [{"delta": {"content": " Events"}}]}

data: [DONE]

At each generation step, the model computes logit scores for every token in its vocabulary. Generation controls determine how those logits are sampled:

• temperature: Divides raw logits before the softmax pass. T = 0.0 is greedy argmax (strictly deterministic). As T → 1.5, lower-probability tokens are sampled more frequently.
• top_p (Nucleus Sampling): Constrains sampling to the smallest pool of tokens whose cumulative probability exceeds p (e.g. 0.9 = top 90% mass).
• max_tokens / max_completion_tokens: Hard ceiling on how many output tokens the model may generate before terminating with finish_reason: "length".
• stop: Array of strings that cause the model to halt generation immediately upon emission (e.g. ["User:", "\\n\\n"]).

When an LLM powers a database ingestion pipeline or an internal microservice, receiving conversational commentary like "Here is your extracted customer: {"name": "Alice"}" breaks JSON parsers.

Modern providers support Strict Structured Outputs:

from pydantic import BaseModel
from openai import OpenAI

class CustomerProfile(BaseModel):
    name: str
    age: int
    is_vip: bool
    preferred_language: str

client = OpenAI()

completion = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "user", "content": "Extract: Rahul Verma, 34 years old, speaks Hindi, platinum member."}
    ],
    response_format=CustomerProfile  # Guarantees 100% compliant schema
)

profile = completion.choices[0].message.parsed
print(profile.name)  # "Rahul Verma"
print(profile.age)   # 34

When an API returns a 429 Rate Limit or 503 Service Unavailable, hammering the endpoint every 10ms only prolongs the outage. Production systems use Exponential Backoff with Full Jitter:

Providers enforce rate limits using the Token Bucket Algorithm. Every organization tier is allotted a maximum bucket capacity (e.g. 500 RPM and 100,000 TPM) refilling at a constant rate.

API operational cost is modeled as: