Self-Hosted LLM Inference

Sriram Panyam has spent over 20 years building large-scale distributed systems, cloud and AI platforms and the data analytics that run on them, at Google, LinkedIn, Amazon, several startups and now General Motors. At Google Cloud he led engineering teams working on petabyte-scale workloads and control planes on Cloud Dataflow. At LinkedIn he built and managed identity and analytics platforms serving more than a billion members. He has mentored a number of senior technical leaders, and writes about systems engineering and AI infrastructure.

His recent work is on AI platform infrastructure and the standards beneath it, including the Model Context Protocol (MCP). Most of that effort goes into conformance, the test suites that decide whether an implementation matches the specification rather than merely looking like it does, and into the reference implementations that show what a correct one looks like.

He holds engineering degrees from the University of Melbourne.

- PART I: FOUNDATIONS.- Chapter 1: Introduction to Self-Hosted Inference.- Chapter 2: How LLMs Work.- Chapter 3: Hardware Fundamentals.- Chapter 4: Model Formats and Quantization.- Chapter 5: Inference Engines.- Chapter 6: Building the Control Plane (v0.1).- PART II: SERVING REAL WORKLOADS.- Chapter 7: Routing, Queueing, and the Control Plane in the Data Path.- Chapter 8: Stateful Routing and Prefix-Cache Reuse.- Chapter 9: Scaling to 30B+ Models.- PART III: DISTRIBUTED INFERENCE.- Chapter 10: Multi-GPU and Distributed Inference.- Chapter 11: Capacity Across Providers.- Chapter 12: 70B Deployment.- PART IV: LET’S GO BIG.- Chapter 13: Serving a Frontier Model, and What the Market Will Sell You.- Chapter 14: What It Costs, and Why Batching Stops Helping.- Chapter 15: The Ceiling, and What Comes Next.