5phases
7projects
8months
Phase 1C++ systems foundationWeeks 1–8
Learn
Modern C++ (C++17/20): memory ownership, RAII, smart pointers
Resource: "A Tour of C++" by Bjarne Stroustrup. 2 chapters/week.
CMake build system with sanitizers: ASan, TSan, UBSan baked into every build
Enable sanitizers from day 1. They catch memory corruption, data races, and undefined behavior before they become production nightmares.
Multithreading: std::thread, mutexes, condition variables, atomics
Everything in inference is concurrent. This is non-negotiable.
Non-blocking I/O: epoll (Linux), kqueue (macOS), io_uring basics
A thread-per-connection model dies at scale. Event-driven I/O is how real inference servers handle thousands of SSE/WebSocket streams.
Custom allocators and arena allocation patterns
KV cache memory management uses block allocation. Understanding allocators now pays off in Phase 2 when you build a cache block allocator.
Sockets and HTTP from scratch: BSD socket API in C++
Don't use a library yet. Write raw send/recv. Understand the protocol at the byte level.
Project 1: event-driven HTTP server in C++
tokoro: HTTP/1.1 server in C++
Build an event-driven HTTP server using epoll/kqueue that serves static files and handles concurrent connections. No libraries: raw POSIX sockets. Implement: non-blocking accept loop with I/O multiplexing, HTTP parser, thread pool for compute work, keep-alive, and graceful shutdown. All builds run with ASan + TSan enabled in CI.
Why this matters: every inference server is a networked C++ process using event-driven I/O. This teaches you what's below frameworks like NestJS, and the project is immediately legible to AI infrastructure engineers. The event-driven architecture directly maps to how vLLM and TensorRT-LLM handle thousands of concurrent streaming connections.
Milestones by week 8
Server handles 1000+ concurrent connections via epoll/kqueue
Event-driven I/O benchmark vs naive thread-pool: measured throughput difference
Zero ASan/TSan warnings across entire codebase
HTTP parser handles chunked encoding
README with architecture diagram and benchmark results
Published to GitHub with CI (sanitizers + tests) via GitHub Actions
Phase 2AI inference engineWeeks 7–18
Learn
How LLM inference works: tokenization, attention, KV cache, batching
Read the llama.cpp source. Understand every struct. Don't just run it.
GGUF format: quantization math — scale/zero-point, per-channel vs per-tensor, not just file parsing
Open a .gguf file in a hex editor. Map the header. Then understand how INT4/INT8 quantization preserves model quality. This is a numerics problem, not just a file format.
Continuous batching & PagedAttention: the core of modern inference serving
Read the vLLM paper. Understand iteration-level batching (insert new requests each decode step) and paged KV memory allocation. This is the single most interview-relevant topic for inference-serving roles.
GPU memory model & CUDA/Triton fundamentals
Global memory vs shared memory (SRAM) vs registers. Memory coalescing and warp divergence. Arithmetic intensity and the roofline model. Why LLM decode is memory-bandwidth bound while prefill is compute bound. Write a toy fused kernel in OpenAI Triton.
Profiling C++: perf, Valgrind, gprof, flamegraphs, cache miss analysis
You can't optimize what you haven't measured. This skill separates real systems engineers. Learn to generate and read flamegraphs.
SIMD basics: SSE2/AVX2 intrinsics for vectorized float math
llama.cpp uses SIMD heavily. Even reading the intrinsic calls builds intuition for vectorized compute.
Python ↔ C++ interop: pybind11 / nanobind
Production AI infra is hybrid: Python control plane, C++/CUDA execution engine. Knowing how to expose a fast C++ module as an importable Python library is standard practice.
Speculative decoding: draft-model verification, Medusa, EAGLE
Increasingly standard in production inference. A small draft model proposes multiple tokens, the target model verifies in a single forward pass. Read the SpecInfer and Medusa papers, then implement a toy speculative decoder in C++ using vahan as the target engine. Strong interview differentiator for Anthropic/DeepMind-tier roles.
Project 2: inference engine with continuous batching
vahan: LLM inference engine in C++
Not a wrapper around llama.cpp's high-level API. Build an inference server that implements a continuous batching scheduler and a KV cache block allocator. Load GGUF models via llama.cpp as a library, but manage request scheduling yourself: dynamically insert new requests at each decode iteration, allocate/free KV cache pages, handle prompt prefill vs token decode separately. Expose /v1/completions (OpenAI-compatible), /generate, and /health endpoints with SSE streaming. Expose a pybind11 bridge so the engine can be imported from Python. Add per-request latency metrics with Prometheus-compatible output. Implement backpressure (reject with HTTP 429 when the request queue is full), graceful shutdown (drain in-flight requests before exit), and crash recovery (checkpoint KV cache allocation state to enable fast restart without re-loading the full model).
Project 3: inference benchmarking CLI
drishti: inference benchmark & analysis CLI
A CLI tool in C++ or Python that stress-tests any OpenAI-compatible inference endpoint. Measures TTFT, ITL (inter-token latency), prefill vs decode throughput (tokens/sec separated by phase), p50/p95/p99 latency, prefix cache hit rate, stream jitter, and concurrent load scaling. Outputs structured JSON reports with embedded flamegraph data. Includes a roofline model analysis mode that estimates MFU and memory bandwidth utilization. Genuinely useful to the community.
Milestones by week 18
vahan streams Llama 3.2 3B locally with continuous batching — dynamic request insertion verified
KV cache block allocator handles allocation/deallocation without memory leaks (ASan-clean)
Backpressure tested: vahan returns 429 under overload, graceful shutdown drains in-flight requests
pybind11 bridge: import vahan from Python, run inference in 3 lines
drishti reports ITL, prefill/decode throughput, and prefix cache metrics
Toy Triton kernel written and benchmarked (fused softmax or activation)
Blog post: "What I learned reading the llama.cpp source" with flamegraph analysis
Phase 3ML systems & education AIWeeks 15–24
This phase overlaps with late Phase 2. The goal: build the ML systems layer (RAG, evals, guardrails) and ship a thin vertical product slice — proving the education AI works end-to-end on a third-party API before swapping in your own inference engine.
Learn
RAG architecture: chunking strategies, embedding models, vector stores (pgvector, Qdrant)
Biggest single gap. Without RAG, the education app can't ground answers in a specific syllabus or textbook. Start with naive chunking, then implement semantic chunking with overlap.
Evaluation frameworks: correctness evals, not just performance benchmarks
Without evals, auto-grading a real exam is genuinely risky. Build a test harness that compares model output against human-graded reference answers. Track accuracy, consistency, and format compliance.
Structured output & function calling: JSON mode, constrained generation, output parsers
Question papers and grading rubrics need parseable, consistent formats — not freeform text. Learn JSON schema enforcement and function-calling patterns.
Guardrails & content safety: prevent hallucinated facts, enforce age-appropriate content
Especially critical for an education product used by students. Implement output validation, fact-checking against source material, and content filtering.
OCR, document ingestion & multi-modal grading: Tesseract, PaddleOCR, vision-language models
Answer sheets are images/PDFs/handwriting — this is a whole subsystem. Start with printed text (Tesseract), then tackle handwriting recognition with vision models. Also evaluate direct multi-modal LLM input (GPT-4o, Claude) — sending answer sheet images directly to a vision-language model for grading without an OCR intermediate step. This approach is rapidly improving and may outperform traditional OCR pipelines for handwritten content.
Prompt engineering patterns: chain-of-thought, few-shot grading, rubric-based evaluation
The grading system's accuracy depends heavily on prompt design. Build a prompt library with versioning and A/B testing support.
Caching strategies: prompt cache, KV cache reuse, semantic caching, TTS output caching
Underrated by most people building on LLMs. Reusing KV cache for shared system prompts and caching common TTS outputs dramatically cuts cost and latency at scale.
Fine-tuning fundamentals: LoRA, QLoRA, when to fine-tune vs prompt-engineer vs RAG
Sequence matters: prompting first, then RAG, then fine-tuning as the last resort. Fine-tuning is relevant for grading consistency (training on rubric-scored examples to reduce prompt sensitivity) and voice agent language adaptation (accent/dialect tuning). Learn LoRA/QLoRA for parameter-efficient fine-tuning. Don't reach for fine-tuning before exhausting prompting and RAG — but know how to do it when you need it.
Project 4: education AI backend (thin vertical slice)
vidya: education AI platform backend
RAG-powered education backend. Starts on a third-party API (OpenAI/Anthropic) to validate the product fast, then swap in vahan underneath. Three core features: (1) Syllabus-grounded chat — ingest syllabus/textbook via document pipeline, chunk and embed into vector store, answer student queries with source citations. (2) Question paper generation — structured JSON output with configurable difficulty, topic distribution, and question types. (3) Answer sheet grading — OCR + vision model ingestion, rubric-based evaluation with chain-of-thought scoring, per-question feedback. Includes a 50+ test case eval suite for grading accuracy. User feedback loop: teacher corrections improve grading prompts over time.
Milestones by week 24
RAG pipeline answers syllabus questions with source citations and <2s latency
Question paper generator produces valid structured JSON for 3 different exam formats
OCR + grading pipeline scores a sample answer sheet within 5% of human grading
Multi-modal grading tested: vision-LLM vs OCR pipeline accuracy comparison documented
Eval suite: 50+ test cases covering grading accuracy, format compliance, hallucination rate
Guardrails pass: zero age-inappropriate outputs in 1000-query stress test
Blog post: "Building RAG for education: what chunking strategy actually matters"
Phase 4Voice AI pipelineWeeks 21–28
Overlaps with late Phase 3. This phase builds the real-time voice agent pipeline — directly targeting the Sarvam AI-style voice product goal.
Learn
Audio chunking & VAD: Silero VAD, energy-based endpointing, silence detection
Voice AI is won or lost on endpointing accuracy. Silero VAD is the industry standard. Understand frame sizes, speech probabilities, and how to tune sensitivity.
WebRTC/WebSocket real-time protocols: full-duplex audio streaming
Voice agents need bidirectional audio. WebSocket for server-controlled flows, WebRTC for peer-to-peer with TURN/STUN for NAT traversal.
TTS models: Piper, Coqui XTTS, edge TTS options
Compare latency vs quality vs language support. Indian language TTS is a hard problem — evaluate Sarvam's approach vs open-source alternatives.
Barge-in & interruption handling: speculative TTS cancellation
When a user interrupts, you must cancel the current TTS stream, flush the audio buffer, and restart the STT pipeline — all within 200ms. This requires speculative cancellation patterns.
Distributed/multi-GPU serving concepts: tensor parallelism, pipeline parallelism
Real inference infra teams use multi-GPU setups. Even reading the Megatron-LM or DeepSpeed inference papers builds understanding of how models are sharded across GPUs.
Multilingual & code-switching: Indian language STT/TTS, Hindi-English mixed speech
Core to the Sarvam AI-style voice agent goal. Whisper handles many languages but accuracy varies significantly for Indian languages. Hindi-English code-switching (mixing languages mid-sentence) is extremely common in India and breaks most STT pipelines. Evaluate Sarvam's multilingual models, IndicWhisper, and Bhashini. Also consider multilingual RAG — syllabi in Hindi, Gujarati, or regional languages need embedding models that handle non-English text.
Project 5: real-time voice AI pipeline
shabda: real-time voice AI engine
End-to-end: microphone input → Silero VAD endpointing → Whisper STT → LLM (via vahan or API) → TTS → speaker output. WebSocket server in C++. Full duplex with barge-in interruption handling via speculative TTS cancellation. Per-stage latency budget: VAD <50ms, STT <300ms, LLM <350ms, TTS <100ms. Audio chunk streaming (not wait-for-complete). Backpressure handling: reject new sessions with HTTP 503 when the pipeline is saturated, with a configurable max-concurrent-sessions limit. Graceful shutdown: drain active audio streams before exit, sending end-of-stream markers to connected clients. Prometheus metrics dashboard for per-stage latency. Deployed as a demo anyone can try via a web interface.
Milestones by week 28
End-to-end voice pipeline latency <800ms (VAD to speaker output)
Barge-in interruption works within 200ms of user speech detection
Hindi-English code-switching tested: STT accuracy measured on mixed-language samples
Per-stage flamegraph published: identify bottleneck stage
Prometheus dashboard live with per-stage latency percentiles
Web demo deployed: anyone can test the voice agent from a browser
Phase 5Ship, signal & hardenWeeks 25–32
The final phase turns demos into deployable products and raw code into public credibility. This is where Layer 3 (product) gets hardened.
Product hardening
Auth & user management: NextAuth, role-based access (student/teacher/admin)
Multi-user product needs accounts, roles, and permissions. Students see their own work; teachers see their class; admins see everything.
File upload & storage pipeline: S3/R2 for answer sheets, syllabi, question papers
Answer sheets, syllabi, and question papers need to be uploaded, stored, versioned, and retrieved. Design the storage schema for multi-tenant isolation.
Full observability: distributed tracing across STT → LLM → TTS, error dashboards, cost tracking
Not just perf metrics. Full request tracing with OpenTelemetry, error rate dashboards, and per-request cost tracking. Grading at scale gets expensive fast.
Conversation & dialogue design: multi-turn syllabus chat with memory
"Chat that solves queries based on syllabus" is a UX and prompting design problem, not just a backend one. Design conversation flows, handle context windows, implement conversation history.
User feedback loops: teacher corrections improve grading, student feedback improves chat
This is the product's flywheel. When a teacher corrects a grade, that correction feeds back into the prompt library. Track correction rate as a key metric.
Multi-tenant deployment: hosting strategy for concurrent students/teachers
A demo vs a hosted product serving many concurrent users are different engineering problems. Plan for rate limiting, queue management, and graceful degradation under load.
Cost modeling: per-request cost tracking, budget alerts, usage dashboards
Running voice agents and grading at scale gets expensive. Track token usage, API costs, and compute time per request. Set up budget alerts before you get a surprise bill.
Data privacy & compliance: student data protection, parental consent, secure storage
An education product handling student grades, answer sheets, and personal data is legally sensitive. In India: comply with DPDPA 2023 (parental consent for minors, data minimization, purpose limitation). If targeting US markets: FERPA and COPPA apply. Implement: encrypted storage for student PII, data retention policies, audit logging for all data access, and a clear privacy policy. Do not treat this as an afterthought — it can block your entire launch.
Project 6: full education AI product
vidya-app: deployed education AI platform
Full product deployment of vidya with a Next.js frontend. Auth via NextAuth (student/teacher roles). File upload for answer sheets and syllabi via S3/R2. Chat UI with conversation history and source citations. Question paper generator with template selection. Grading dashboard showing per-student, per-question scores with teacher override capability. Feedback collection that improves grading prompts. Cost tracking per user/school. Deployed and usable by real test users.
Open source contribution
Contribute a real PR to llama.cpp, vLLM, sglang, or whisper.cpp
Not a docs fix. A bug fix, a perf improvement, or a missing feature. Target Good First Issues, quantization additions, or architecture support. One merged PR > 10 side projects.
Contingency: if the PR isn't merged by week 32 (maintainer timelines are outside your control)
Publish the PR as a detailed RFC + standalone benchmark report. Document the problem, your approach, benchmark results, and community discussion. An unmerged PR with a thorough technical writeup is still a strong signal.
Signal the work publicly
Write 1 technical post/week on LinkedIn about what you're building
Not summaries of articles. Your own findings, failures, benchmarks. "I ran X and found Y."
Publish 2 long-form blog posts on your personal site
"How I built a continuous-batching inference server in C++" and "Benchmarking open-source LLM inference: ITL, prefill/decode throughput, and the metrics that actually matter."
All 7 projects documented, tested, and CI-passing on GitHub
Recruiters and staff engineers look at GitHub before your resume. Clean READMEs, architecture diagrams, reproducible benchmark scripts, and passing CI make it easy for them.
Milestones by week 32
1 merged PR in a major open-source inference project — or published RFC with community engagement
1 deep technical benchmark report with flamegraphs and roofline model analysis
vidya-app deployed and usable by 5+ real test users (students/teachers)
shabda voice demo publicly accessible from browser
Inference backend swapped: vidya runs on vahan instead of third-party API
Data privacy audit passed: encrypted PII storage, audit logging, DPDPA-compliant consent flow
Resume updated: AI inference infrastructure + shipped education product as lead skills
The three layers
Layer 1 — INFRA (Phases 1–2): The engine. C++ systems, inference internals, continuous batching, GPU fundamentals, speculative decoding. This is what gets you the infrastructure engineering interview.
Layer 2 — ML SYSTEMS (Phases 2–4): The transmission. RAG, evals, guardrails, structured output, OCR, multi-modal grading, fine-tuning, caching, voice pipeline, multilingual handling, observability. This is what turns "I can run an inference server" into "I have infra reliable enough to power a real product."
Layer 3 — PRODUCT (Phases 3–5): The car. Auth, UI, storage, dialogue design, deployment, feedback loops, cost tracking, data privacy compliance. This is what proves the infra actually works under real user load — and that you can ship.