Articles

Attention has no sense of order. Group theory says every linear relative positional encoding is a matrix exponential, and the complete list is turn, fade, and slide: RoPE, decay, and ALiBi.

After writing about attention on paper, I wanted to feel the full stack: BPE tokens, causal MHA, Pre-LN blocks, AdamW, and a model that actually spits out Romeo dialogue on a Colab T4.

Attention sees a bag of tokens, not a sequence. We try crude position hacks, watch them fail, then arrive at sinusoidal encodings and why the heatmap gets calm as depth increases.

GQA shrinks the KV cache by sharing heads. MLA takes a different route: compress keys and values into a learned latent, cache that instead, and keep RoPE on a separate track so the math still works.

One head looking at the whole sentence is not enough, and it is not what production models do. Causal masking, multi-head attention, and the two tricks that make long context affordable at inference time.

The dot product told us how to compare two vectors. Now we build self-attention on top of it why a static embedding isn't enough, what Q, K and V actually do, and what the whole thing looks like in code.

Before Q, K, and V feel magical, you need one idea from linear algebra: the dot product measures how aligned two vectors are and that alignment is the heart of attention.

Before attention, transformers, or embeddings, every LLM turns your text into token IDs. Here's how tokenization works—and why BPE became the default compromise.

Routing in LLM systems isn't one idea—it can mean choosing a model, an agent, a workflow, or a tool, and those problems are not the same.

Graph, loop, and harness engineering aren't new buzzwords—they're layered patterns for designing agentic systems around a model.

Architectural patterns for lowering token usage in production AI systems—beyond max_tokens limits—to cut costs, reduce latency, and scale better.