Pratik Dhanave — Blog
Articles about Pratik Dhanave — Blog
Risk Management and Building Models in Practice
Risk management is the purpose every model served. Value at Risk and its tail-hiding flaw (so expected shortfall), fat tails, hedging with the Greeks, market microstructure as a systems problem, and the overriding lesson: models are approximations that will eventually be wrong — so respect the tails and stay humble. (Educational, not financial advice.)
Operating in Production — Cost, Latency, Incidents, and the Tightening Loop
Running an LLM app means treating cost and latency as first-class constraints (model routing, token hygiene, caching, streaming), preparing for incidents as certainties (provider failures, instant rollback, blameless postmortems), and the closing idea: operating is continuous improvement — a tightening loop, because an LLM app is never done.
Observability and Tracing — Seeing Inside an LLM System
'The model messed up' is never a useful diagnosis. Why ordinary monitoring isn't enough, tracing as the spine (spans capturing the rendered prompt, retrieved context, tool calls, raw output, tokens/cost), and turning aggregated traces into metrics, quality trends, and drift detection.
Randomness — Stochastic Models and Monte Carlo
Prices are random, so finance is a probabilistic discipline. Brownian motion and geometric Brownian motion model how a price wanders; Monte Carlo simulation turns that model into a universal engine for pricing and risk — the computational core of modern quant finance, and territory engineers find familiar. (Educational, not financial advice.)
Option Pricing — Black-Scholes and the Greeks
Black-Scholes turned option pricing into engineering: price an option by the cost of continuously replicating it, so no-arbitrage pins its value and the underlying's expected return drops out entirely. Risk-neutral valuation, what the model assumes, implied volatility, and the Greeks traders actually hedge. (Educational, not financial advice.)
Guardrails and Safety in Production — Containing Open-Ended Output
An LLM can say anything, which is the problem. The new failure modes — hallucination, prompt injection, data leakage, harmful output — and layered, fail-closed guardrails (input, output, action) with humans in the loop, treated as part of the lifecycle and red-teamed.
Online Evaluation and Feedback — Production Is the Real Test
Offline evals say a change is safe to ship; online evals say whether it worked. The intrinsic offline–online gap, deploying to measure (canary, A/B, shadow), capturing explicit and implicit feedback, and the pipeline that turns production behavior into continuous improvement.
Derivatives and the No-Arbitrage Principle
A derivative's value derives from something else — and it's priced not by prediction but by the no-arbitrage principle: if a portfolio replicates a payoff, it must cost the same. Forwards, futures, options, payoff shapes, and the replication argument that prices a forward and links calls to puts. (Educational, not financial advice.)
Risk and Return
Higher expected return requires more risk — and the surprising twist is that diversification reduces risk for free when assets are imperfectly correlated. Volatility, covariance, the efficient frontier, systematic vs. idiosyncratic risk, and how CAPM prices risk through beta. (Educational, not financial advice.)
Eval-Driven Development — You Can't Ship What You Can't Measure
The practice that separates improving an LLM app from merely changing it. The curated eval set as the core asset, how to score open-ended output (deterministic checks, reference metrics, LLM-as-judge, human review), and how to gate every change on evals and compare B vs A.
Bonds and the Yield Curve
A bond is the simplest real application of discounting: fixed future cash flows, priced by present value. From there come yield, the inverse price-yield relationship, duration and convexity as interest-rate risk, and the yield curve as the market's price of time. (Educational, not financial advice.)
Prompt and Version Management — Treat Prompts Like Code
In an LLM app the prompt is the source code — where behavior is defined and most bugs happen. Treat prompts like code: version the whole configuration (template + model + parameters + context strategy), decouple them from app code, and route every change through review, eval, rollout, and rollback.
The LLMOps Lifecycle — The Loop an LLM App Lives In
The LLMOps lifecycle — develop, evaluate, deploy, observe, improve — as a loop an LLM app lives inside, not a line it runs once. Why evaluation is a first-class stage appearing twice, why production feedback flows backward, and why the loop turns whether you act or not.
The Time Value of Money
A dollar today is worth more than a dollar tomorrow — and that single fact, made precise, is how everything in finance gets valued. Present value, future value, discounting, and the discounted-cash-flow method that prices anything with a future payoff. (Educational, not financial advice.)
Building a Forward Deployment Function
A practical blueprint for founders: when to start forward deployment, who to hire first, how to scope engagements, what to measure, and how to keep it from eating your margins — plus the four failure modes (agency, hero, science-project, over-serving) and a 90-day start.
What LLMOps Is — and Why It’s Not Just MLOps
Getting an LLM to work in a demo is easy; keeping it working, safe, and affordable for a year is a discipline. What LLMOps is, how it differs from MLOps, and the five properties of LLMs — opaque external models, prompt-defined behavior, fuzzy correctness, non-determinism, new failures/costs — that force a different playbook.
The Forward Deployed Leader
Forward deployment that isn't run as a disciplined function becomes an unscalable custom-dev shop. The Forward Deployed Leader owns the motion as a business — team design, scoping discipline, unit economics, and the decision of what becomes product — and names the invisible trap: it feels like success right up until the margins invert.
What Financial Engineering Is
Financial engineering applies math, statistics, and computation to money — and under the jargon it rests on three ideas an engineer will recognize: the time value of money, the trade-off between risk and return, and no-arbitrage. This series builds quant finance from those foundations. (Educational, not financial advice.)
The Forward Deployed Architect
The forward deployed engineer builds a win inside one customer; the Forward Deployed Architect makes it survive the next ten — reference architecture, security and governance that passes review, and the reuse decision that separates product from one-off. The missing layer between heroics and product.
Profiling and the Optimization Workflow — Measure, Don't Guess
You can't optimize what you haven't measured. The optimization loop — baseline, profile, find the dominant limit, apply the matching lever, repeat — plus knowing when to stop, leaning on the ecosystem, and the durable mental model behind it all.
The Forward Deployment Stack
The forward deployed engineer gets all the attention, but forward deployment is bigger than one role — it's a go-to-market operating model with three layers: build (engineer), scale (architect), run (leader). Seeing the whole stack is how AI companies turn embedded engineering from a heroic cost center into a repeatable engine.
Memory Management for LLM Inference — The KV Cache Problem
The KV cache often exceeds the model weights in HBM, and naive contiguous allocation wrecks concurrency with fragmentation. PagedAttention applies operating-system paging to fix it — a core inference bottleneck solved by borrowing a classic systems idea.
The Attention Bottleneck — and How FlashAttention Fixed It
Naive attention materializes an N×N matrix in HBM — memory-bound and quadratic. FlashAttention fixes it with pure roofline reasoning: tile into shared memory, use an online softmax, fuse the kernel, never touch HBM. The archetype of all inference optimization.
Numeric Precision — Why Lower Precision Makes Inference Faster
FP32, FP16, BF16, and INT8/INT4 — how a float splits bits between range and precision, why bf16 won, and why fewer bits is a double win: less data to stream from HBM and faster tensor-core math. Quantization and why LLMs tolerate it.
From Bespoke AI to Product
Every AI FDE builds one-offs — a bespoke deployment for one customer's data, workflow, and trust. The ones who create lasting value turn those one-offs into product: the patterns that repeat become a platform, the platform makes the next deployment faster, and field learnings flow back to shape what gets built. The flywheel that turns bespoke AI work into a compounding asset — and the AI FDE career arc.
The CUDA Execution Model — Threads, Warps, and Occupancy
Threads, warps, blocks, and grids; SIMT and branch divergence; and the cleverest trick — hiding slow memory by oversubscribing warps (occupancy). Why a GPU can be underutilized without being either compute- or memory-bound.
The Roofline Model — Are You Compute-Bound or Memory-Bound?
The roofline model answers the one question that governs every optimization — compute-bound or memory-bound? — from arithmetic intensity (FLOPs per byte). Why LLM prefill is compute-bound and decode is memory-bound, and why batching is the master throughput lever.
Productionizing and Handing Over AI
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — then handing it over so the customer operates it without you. The FDE's goal, in the end, is to make themselves unnecessary. Cost, latency, drift, observability, and a handover that transfers the eval discipline, not just the code.
Integrating AI into Real Workflows
A technically excellent AI system that nobody uses has delivered zero value. The last mile of an AI deployment is not the model — it's fitting the system into how real people actually do their jobs, designing an interface that handles uncertainty honestly, and managing the human change of introducing AI into someone's work. Human-in-the-loop, calibrated reliance, autonomy levels, and change management.
The Memory Hierarchy — Why Bandwidth Rules Inference
The GPU memory hierarchy from registers to HBM, the difference between capacity and bandwidth limits, and the defining fact of LLM generation: it's memory-bandwidth-bound because every token reads the whole model from HBM — which is why batching and quantization work.
Production Recommenders
Building a good model is half the work; running a recommender in production is the other half. Real systems must serve in milliseconds, stay fresh as the catalog and tastes change, handle cold start gracefully, resist the filter bubbles their own optimization creates, and be monitored like any critical service. This closing post assembles everything — and gives a practical path to building one.
Why GPUs Power AI — The Hardware Underneath Inference
Every LLM runs on a GPU, but the spec-sheet FLOPS number is a near-fiction in practice. Why neural nets map onto GPUs (matrix multiply, SMs, tensor cores), and the governing intuition of the series: compute is cheap, data movement is expensive.
Evaluation and Trust for Deployed AI
You cannot ship AI you cannot measure, and no enterprise grants a probabilistic system authority over real work on faith. Both problems have the same answer: evaluation. Building the customer's own eval set — real examples, their definition of correct — turns 'it seemed good in the demo' into a reliability number, and that number is how trust gets earned. Offline/online eval, safe failure, and trust.
Evaluating Recommenders
A recommender that scores well offline can flop in production, and a metric that looks like success can quietly harm the product. Evaluation is genuinely hard: offline metrics only approximate real behavior, the only ground truth is a live A/B test, and the very act of recommending shapes the data you learn from next. Offline metrics, online testing, the offline-online gap, and feedback loops.
Grounding AI in the Customer's Data
A frontier model knows the public internet and nothing about the customer. All the value of an AI deployment comes from the opposite: making the model reason over the customer's own documents, records, and knowledge. Grounding — retrieval-augmented generation over messy, permissioned, incomplete data — is the technical heart of the AI FDE's job. With an interactive reference-architecture diagram.
Deep Learning Recommenders
Modern large-scale recommenders are built on neural networks — not replacing the core ideas (embeddings, two stages) but supercharging them. This post covers the two workhorses: two-tower models for retrieval (the neural evolution of matrix factorization, built for fast ANN search) and rich neural ranking models that score the shortlist with cross-features, sequences, and multiple objectives.
Operating Knowledge Graphs in Practice — Governance, Evolution, and When to Build One
A knowledge graph is a system you run, not a project you finish. Governing the ontology as it evolves, keeping the populated graph correct (freshness, re-resolution, provenance), and the honest question: given the cost, when is a knowledge graph the right tool — and when is a table or plain vector RAG cheaper and enough?
The Two-Stage Architecture
You can't run your best, most expensive model on millions of items per request. The near-universal answer is to split recommendation into two stages: a cheap, fast net that narrows millions to a few hundred, then a precise, heavier model that ranks those few. This retrieve-then-rank structure is the single most important architectural pattern in production recommenders — with an interactive pipeline diagram.
The Semantic Layer — The Enterprise Ontology as Shared Meaning
The enterprise ontology made operational: a governed layer that defines the business's entities, relationships, and metrics once, so dashboards, analysts, and AI agents all inherit the same definitions. Why 'revenue' means three things, and why AI made the semantic layer urgent — an agent is only as correct as its grasp of your terms.
From Demo to Pilot
An AI demo is the easiest impressive thing to build and the most misleading: it runs on cherry-picked inputs, in a clean environment, with the failures edited out — and convinces everyone the problem is nearly solved when the real work has barely begun. Use the demo to win belief, then walk the customer honestly across the chasm to a pilot that survives real data.
Matrix Factorization and Embeddings
Represent every user and item as a short vector of latent factors, and a user's affinity for an item becomes just the dot product of their vectors. Matrix factorization turned recommendation into learning good embeddings — the technique that won the Netflix Prize and the conceptual bridge from classical collaborative filtering to today's deep-learning systems and vector search.
The eBPF Ecosystem and Building With It
You rarely write raw eBPF bytecode by hand — a rich ecosystem of libraries, languages, and platforms sits on top, and one breakthrough (CO-RE, Compile Once Run Everywhere) solved the portability problem that once made eBPF programs fragile across kernel versions. This closing post surveys how you actually build with eBPF, and where it's heading.
Scoping an AI Use Case
The most important decision an AI FDE makes happens before any code: which problem to point the model at. Choose a problem AI is genuinely suited for, with real value and a clear way to measure it, and the engagement can succeed. Choose AI theater — impressive-sounding but ill-fit — and no engineering saves it. Fit vs value, resisting AI theater, augmentation over automation, and picking the wedge.
GraphRAG — Grounding an LLM in a Knowledge Graph
The post the series built toward. Plain vector RAG retrieves disconnected chunks and misses relationships; GraphRAG retrieves a subgraph by traversing the knowledge graph, so multi-hop answers come from walked edges, not guessed connections — typed, verifiable, citable. When to use it, and why hybrid often wins.
Building and Productionizing Voice Agents
A voice demo that works in a quiet room with a good headset is a long way from a voice agent that survives a noisy phone call from a real customer. Production voice AI has to handle bad audio, unpredictable humans, failures at every stage, and the peculiar demands of telephony — and be evaluated in ways text systems never require. Making a voice agent real.
Building a Knowledge Graph — Construction and Entity Resolution
The hard part isn't the ontology or the queries — it's construction. The pipeline from heterogeneous sources, and its make-or-break challenge: entity resolution (deciding when two records are the same thing). Under-merging fragments; over-merging asserts false connections; connectivity amplifies every error.
Content-Based Filtering and the Cold-Start Problem
Collaborative filtering has one crippling blind spot: it knows nothing about brand-new users or items. Content-based filtering fills that gap by recommending based on what items *are* rather than who interacted with them. Understanding the cold-start problem — and how each approach handles it — is key to building a recommender that works from day one, which is why most real systems are hybrids.
The Rise of the AI Forward Deployed Engineer
The forward deployed engineer was born at Palantir to bridge powerful software and messy customer reality. In the AI era the role exploded, because frontier models widened that gap: a model that dazzles in a demo is a long way from a system that works inside one company's data, workflows, and trust constraints. This series is the technical playbook for the engineer who closes that gap.
EBPF for Security
Security is eBPF's most natural fit: the kernel sees every syscall, process, file access, and network connection, so an eBPF program in the kernel is perfectly positioned to watch for and stop malicious behavior in real time. This is why modern runtime-security tools — detecting and blocking threats on live systems — are increasingly built on eBPF.
Building and Operating an AI Gateway
You've seen what an AI gateway does; the last question is how to get one — build it, adopt an open-source proxy, or use a managed service — and how to run it once you have it. This closing post assembles the full architecture, weighs build-versus-buy honestly, and covers operating the gateway as the critical infrastructure it becomes.
Speech-to-Speech and the New Realtime Architectures
The cascaded pipeline — ASR, then LLM, then TTS — has powered voice agents for years, but it has an inherent ceiling: every stage adds latency and every conversion loses information. A newer architecture collapses the three models into one that goes from speech directly to speech, promising lower latency and richer understanding. Comparing the cascade with the emerging end-to-end approach.
EBPF for Networking
Networking is where eBPF delivers its most dramatic performance wins. By running packet-processing logic in the kernel — at the earliest moment a packet arrives, before the kernel builds its usual data structures — eBPF can filter, route, and load-balance at speeds user-space networking can't touch. It's why modern cloud-native networking is increasingly eBPF underneath.
Querying a Knowledge Graph — SPARQL and Graph Patterns
Querying a graph is pattern matching, not joining: you describe a shape of nodes and edges and the engine finds every match. SPARQL (and Cypher for property graphs), path expressions for arbitrary-depth connection that SQL can't express, and inference-aware queries over the reasoned graph.
Collaborative Filtering
The most powerful idea in recommendation is also the simplest: recommend things based purely on the behavior of people like you, without knowing anything about the items themselves. Collaborative filtering turns 'people who liked what you liked also liked X' into an algorithm, in user-based and item-based flavors — the backbone of the whole field, and where it breaks (cold start, sparsity).
The Recommendation Problem
Every time a service suggests what to watch, buy, or read next, a recommender system is at work — one of the most economically important and technically rich problems in applied ML: from a catalog of millions, pick the handful a specific person will want, right now. This series builds recommender systems from the ground up, starting with the problem and the two-stage architecture that structures nearly all of them.
EBPF for Observability
Observability is where eBPF first went mainstream: it lets you see almost anything the kernel sees — syscalls, function calls, network events, disk I/O — with very low overhead and, crucially, without changing the code you're observing. That combination broke a long-standing trade-off in tracing and profiling, and it's why modern observability tools are increasingly built on eBPF.
Turn-Taking, Interruption, and Barge-In
The difference between a voice agent that feels like a conversation and one that feels like a walkie-talkie is turn-taking: knowing when to listen, when to speak, and — hardest of all — gracefully handling being interrupted. Humans do this effortlessly and unconsciously; making a machine do it is one of the subtlest problems in voice AI. The dynamics that make an agent feel alive.
Observability and Governance
You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it and govern it: what was called, what it cost, how long it took, whether it was allowed. Turning the gateway into your AI system's source of truth.
Latency: The Make-or-Break Constraint
Everything about voice AI comes down to one number: how long the user waits to hear a reply. Get it under the threshold where conversation feels natural and the agent is a delight; miss it and no intelligence saves the experience. The latency budget — where the milliseconds go, and how streaming the entire pipeline turns an additive delay into something that feels instant. With an interactive turn diagram.
The Verifier and Safety
The verifier is the reason eBPF exists in its modern form. Running arbitrary code in the kernel would be reckless; running code the kernel has proven safe is not. The verifier is the static analyzer that stands between your program and the kernel, mathematically checking it can't crash, hang, or misbehave — and it explains both eBPF's safety and its constraints.
Rate Limiting, Quotas, and Budgets
Nothing concentrates the mind like a surprise five-figure AI bill from one runaway loop, or one team's spike exhausting the shared rate limit. Because every model call flows through the gateway, it's the one place you can enforce limits and budgets that actually hold — protecting your spend, your providers' rate limits, and fairness across teams. Spending control as a first-class capability.
Caching at the Gateway
Model calls are slow and expensive, and a surprising fraction are repeats or near-repeats. Caching at the gateway turns those into instant, free responses — and because the gateway sees all traffic, it's the one place a cache benefits every app at once. Beyond exact-match caching, semantic caching catches queries that mean the same thing in different words, where the real savings live.
How eBPF Works: Programs, Hooks, Maps
An eBPF program's journey — from code you write to logic running in the kernel — has a few distinct pieces: the program, the hook it attaches to, the maps it uses to share data, and the load path that gets it safely into the kernel. Once you can name these four things and see how they fit, eBPF stops being magic. With an interactive architecture diagram.
Text-to-Speech: Giving the Agent a Voice
The last stage of the pipeline is where the agent finally speaks — and where an interaction either sounds human or sounds like a robot reading a menu. Modern TTS is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and pronounce the messy real world correctly. TTS for real-time voice.
Reliability: Fallback, Retries, and Circuit Breakers
Model providers go down, rate-limit you, and time out — regularly. If your application calls one provider directly, its reliability is capped at that provider's. An AI gateway breaks that ceiling: because it can route across providers, a failure on one becomes a transparent retry on another. Retries, fallback, and circuit breakers — with an interactive request-flow sequence diagram.
Operating Pipelines and Measuring Delivery
A pipeline isn't a build-it-once artifact; it's a production system that needs operating. It degrades — tests get flaky, builds get slow, rollbacks get rusty — and if you can't measure your delivery, you can't improve it. This closing post covers keeping a pipeline healthy, fast recovery over rare failure, observability, and using the DORA metrics to know whether your CI/CD actually works.
The LLM Turn: The Brain in the Loop
The language model is where a voice agent stops being a transcription toy and becomes something you can talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming so the user isn't left waiting. The LLM stage, adapted for voice.
The Kernel/User-Space Boundary
To understand why eBPF is a big deal, you have to understand the wall it lets you cross: the boundary between user space and the kernel. That boundary exists for good reasons — safety and isolation — but it meant extending the kernel was a choice between two bad options: a dangerous kernel module or a limited user-space workaround. eBPF is the third option nobody had before.
Routing and Load Balancing
Once every model call flows through one place, that place can make an intelligent decision on every request: which model should serve this, and through which of your capacity? Routing picks the right model for the task; load balancing spreads traffic across providers and keys so no single limit or outage bottlenecks you. Together they turn the gateway into a control plane.
Speech-to-Text: Hearing the User
The first real stage of a voice agent is turning sound into words, and it's harder than it looks. Modern ASR is astonishingly good on clean audio, but real conversations are full of noise, accents, and disfluencies — and the transcript has to arrive fast and incrementally while the model figures out when the user has actually stopped talking. ASR for real-time voice.
What eBPF Is, and Why It Matters
eBPF lets you run your own sandboxed programs inside the Linux kernel, safely, without changing kernel source or loading a risky module — quietly unlocking a new generation of observability, networking, and security tools. It's one of the most consequential systems technologies of the last decade. This series builds eBPF up from the problem it solves.
The Unified API
The first thing an AI gateway gives you is one interface to every model. Instead of your applications learning each provider's SDK, request format, and quirks, they speak a single API and the gateway translates. That translation layer is what decouples your code from any one vendor — and makes model-swapping a config change instead of a rewrite.
The Anatomy of a Voice Agent
Talking to a computer feels simple — you speak, it answers — but under that simplicity is a real-time cascade of models racing a stopwatch. Audio becomes text, text becomes a response, the response becomes audio, fast enough to feel like conversation. This series builds voice AI from the ground up, starting with the pipeline that makes a voice agent work — with an interactive diagram.
Pipeline as Code
A pipeline is only trustworthy if it's defined the same way your application is: as version-controlled code, reviewed and reproducible. Pipeline as code turns the path to production from clicked-together settings in a web UI into a file in your repository — bringing the whole discipline of software engineering to bear on how you ship software. The declarative model and principles for doing it well.
Efficient Attention
The two costs of long context each have a family of solutions, and together they're why modern models handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. The techniques that made long context practical, and which cost each attacks.
The Long-Context Problem
MoE scales a model's parameters cheaply, but there's a second axis that matters just as much: context length. Attention's compute grows with the square of the sequence, and the KV-cache memory grows linearly and relentlessly — two distinct costs, often confused, that make long context hard. Understanding both is the setup for the efficiency techniques that solved them.
Why You Need an AI Gateway
The moment your application talks to more than one model — or one model but seriously — you accumulate cross-cutting concerns: differing provider APIs, outages, costs, logging. An AI gateway is the single control point that handles all of it, sitting between your applications and every model provider. This series builds one from first principles, with interactive architecture diagrams.