Attention Archive 2017 → 2026

TRANSFORMERS (2017) ➔ FLASHATTENTION (2022–24) ➔ BLACKWELL / RDNA 4 (2026)

PROCEEDINGS:

NeurIPS 2017 (Long Beach) 2017

Attention Is All You Need

Vaswani et al. · Scaled Dot-Product Attention & Self-Attention

Isolate 2017 research ➔

ICLR 2022 · NeurIPS 2022 2022

FlashAttention-1 & Chinchilla Scaling

Dao et al. · Hoffmann, Kaplan et al. · Tiled SRAM Matrix Loops

Isolate 2022 research ➔

ASPLOS 2024 · ICML 2024 2024

FlashAttention-3 & DeepSeek-V2 MLA

Shah, Dao et al. · Hopper TMA, WGMMA & Latent Compression

Isolate 2024 research ➔

IEEE Micro 2025–2026 · Hot Chips 2026

Blackwell NVFP4 & Agent Micro-VMs

NVIDIA, AMD RDNA 4 & Industrial Agent Protocols

Isolate 2026 research ➔

| SYSTEMS & ARCHITECTURE ARCHIVE

The Architecture of
Attention

The mathematical, algorithmic, and micro-architectural lineage of modern LLMs: from Scaled Dot-Product Attention in 2017 to FlashAttention asynchronous SRAM tiling, Multi-Head Latent Compression (MLA), and Blackwell NVFP4 / RDNA 4 WMMA silicon powering autonomous software engineering agents.

NOC-DIE // CANONICAL ATTENTION OPERATOR ROOFLINE INTENSITY: O(N) IO · O(N²) FLOP

NeurIPS 2017 · 2022 · 2023

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

IO-aware asynchronous SRAM tiling & online softmax (Tri Dao et al.)

HIGH-SPEED ON-CHIP CROSSBAR // 128-BIT ASYNC INTERCONNECT

3.2 TB/S HBM3e ⟷ 256 KB L1/SRAM PER SM

BUS_0: TMA_CTX

BUS_1: SRAM_TILE

BUS_2: WARP_ALU

BUS_3: MLA_LATENT

Effective Context

1M+ Tokens

Up from 512 tokens (2017) · >99.8% Needle-in-a-Haystack retrieval

RoPE + RingAttention

HBM Memory Traffic

O(N) Linear IO

Tiled SRAM eliminates intermediate O(N²) quadratic HBM writes

FlashAttention-1/2/3

Tensor Core Efficiency

73% A100/H100

Warp-specialized asynchronous TMA copy & ping-pong GEMM

FP8 / NVFP4 Tensor Cores

KV-Cache Latent VRAM

-92% Footprint

Low-rank joint key-value compression via latent vector c_t^KV ∈ ℝ^512

Multi-Head Latent Attn

The 5-Pillar Autonomous SEA Stack (Click a layer to filter)

Interactive Layer Selector

Year:

/

2017: Attention Revolution & Matrix Tensor Cores

⚡

⚡ ATTENTION ARCHITECTURE Foundation Context

Jun 2017 Context Layer

Scaled Dot-Product Attention & The Transformer

Replacing sequential recurrent bottlenecks with parallelizable global receptive fields

Vaswani et al. (Google Brain / Google Research)

⚡ CANONICAL ATTENTION

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

From: Recurrent Networks (RNN/LSTM) requiring O(N) sequential propagation steps

To: All-to-all attention matrix with O(1) path length between any two program tokens

Max Path Length: O(1) (down from O(N))

Layer Complexity: O(N^2 * d) (parallelizable)

2021: Relative Position (RoPE), Code CLM & BF16

⚡

⚡ ATTENTION ARCHITECTURE Foundation Context

Apr 2021 Context Layer

Rotary Position Embedding (RoPE)

Encoding relative token distances through complex coordinate rotations

Su et al.

⚡ CANONICAL ATTENTION

R_{\Theta, m}^d = \text{diag}\left( R_{\theta_1, m}, \dots, R_{\theta_{d/2}, m} \right)

From: Additive absolute positional embeddings (fixed vocabulary coordinates)

To: Multiplicative complex rotations preserving relative distance invariants under translation

Positional Window: 128k - 1M+ (extrapolatable)

Inner Product Decay: Bounded (preserves lexical order)

Jul 2021 Training & RL

Causal LM on Code & Functional pass@k Metric

Shifting evaluation from cosmetic n-gram similarity to executable correctness

Chen et al. (OpenAI Codex) & Li et al. (StarCoder)

∑ Causal Language Modeling Objective on Code

\mathcal{L}_{\text{CLM}}(\theta) = -\sum_{t=1}^T \log P(x_t \mid x_{<t}; \theta)

From: Cosmetic text-overlap scoring (BLEU, ROUGE, chrF) on code snippets

To: Functional execution harnesses running sandboxed Python test suites

Evaluation Paradigm: Unit Tests (replaces BLEU)

Codex HumanEval pass@1: 28.8% (baseline established)

2021 Hardware

Hardware Numerical Standardization: BF16 Precision

Eliminating underflow/overflow during multi-trillion token code pretraining

NVIDIA (Ampere Architecture) & Google Cloud TPU

∑ Brain Floating Point 16 Bit Specification

x_{\text{BF16}} = (-1)^s \cdot 2^{e - 127} \cdot \left(1 + \frac{m}{128}\right)

From: FP16 training with complex loss scalers and sudden vanishing gradients

To: BF16 hardware-native mixed precision across all training clusters

Exponent Range: 8 bits (identical to FP32)

Memory Savings: 50% (vs FP32 baseline)

2022: Compute-Optimal Scaling, FlashAttention & The ReAct Cycle

Mar 2022 Scaling

Chinchilla Compute-Optimal Scaling Laws

Deriving power-law trade-offs between parameters and training tokens

Hoffmann et al. (DeepMind) & Kaplan et al. (OpenAI)

∑ Compute-Optimal Loss Power Law

L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}

From: Under-trained giant models (e.g. 175B parameters trained on only 300B tokens)

To: Dense, compute-optimal models (e.g. 7B-70B parameters trained on 2T-15T+ tokens)

Token/Param Ratio: ~20 : 1 (up from 1.7 : 1)

Inference Efficiency: 4x faster (smaller parameter footprints)

⚡

⚡ FLASHATTENTION ENGINE Peak IO-Aware Silicon

Jun 2022 Hardware

FlashAttention: IO-Aware Exact Attention Optimization

Eliminating quadratic HBM read/write bottlenecks via SRAM tiling and online softmax

Dao et al. (Stanford University)

⚡ FLASHATTENTION CORE

\begin{aligned} m_{\text{new}} &= \max(m_{\text{prev}}, x) \\[3pt] d_{\text{new}} &= d_{\text{prev}} e^{m_{\text{prev}} - m_{\text{new}}} + e^{x - m_{\text{new}}} \end{aligned}

From: Materializing intermediate N x N attention matrices in slow GPU HBM memory (O(N^2) memory traffic)

To: Tiling Q, K, V blocks directly in ultra-fast on-chip SRAM registers with online softmax (O(N) memory traffic)

HBM IO Complexity: O(N) (down from O(N^2))

Speedup: 2x - 4x (zero approximation error)

Jul 2022 Training & RL

Fill-in-the-Middle (FIM) Pre-Training

Enabling bidirectional infilling and in-editor refactoring in autoregressive models

Bavarian et al. (OpenAI)

∑ FIM Permutation Data Transformation

x_{\text{FIM}} = \langle\text{PRE}\rangle x_{\text{pre}} \langle\text{SUF}\rangle x_{\text{suf}} \langle\text{MID}\rangle x_{\text{mid}}

From: Unidirectional left-to-right code generation unable to condition on following context

To: Arbitrary middle synthesis conditioned on both preceding file headers and trailing call sites

Infilling Capability: Native (without dual encoders)

Pre-training Compute Overhead: 0% (data permutation only)

Oct 2022 Agent Loops

The ReAct Pattern: Reasoning + Acting

Establishing the closed-loop agent control cycle across thoughts, tools, and observations

Yao et al. (Princeton / Google Research)

∑ Interleaved Agent Trajectory

\tau = (o_0, t_1, a_1, o_1, \dots, t_n, a_n, o_n)

From: Static single-turn text completion without environment awareness

To: Dynamic multi-step feedback loop iterating until external assertions are satisfied

Control Loop: Closed-Loop (replaces open-loop gen)

Hallucination Reduction: >50% (grounded in tool stdout)

Dec 2022 Training & RL

RLHF & Verifiable Automated Code Rewards

Augmenting human preference modeling with objective compiler and test suite rewards

Schulman et al. (PPO) & Code Alignment Researchers

∑ Standard PPO with Reference Policy Regularizer

\max_{\theta} \mathbb{E}_{(x, y)} \left[ R(x, y) - \beta \, \mathbb{D}_{\text{KL}}\left(\pi_\theta(y \mid x) \,\parallel\, \pi_{\text{ref}}(y \mid x)\right) \right]

From: Subjective human ratings prone to superficial bias (e.g. favoring nicely commented broken code)

To: Deterministic execution metrics enforcing zero compilation errors and high test pass rates

Reward Precision: 100% Deterministic (no human rater variance)

Compilation Rate: >95% (after RL alignment)

2023: llama.cpp, Ollama, FlashAttention-2 & Deterministic Diffs

Mar 2023 Hardware

The Local Inference Revolution: llama.cpp & GGUF Quantization

Pure C/C++ inference, 4-bit k-quants, and unified memory acceleration across consumer CPUs and Apple Silicon

Georgi Gerganov & Open-Source Contributors

∑ Block Quantization & Dequantization with Scale Factor

w_i = d \cdot q_i + m, \quad q_i \in \{-8, \dots, 7\}

From: Bulky Python/PyTorch inference stacks strictly tethered to expensive datacenter GPUs via CUDA

To: Lightweight, single-binary C/C++ tensor runtime executing on consumer CPUs, unified memory APUs, and mobile silicon

Memory Reduction: -75% (FP16 to 4-bit GGUF)

Loading Latency: Instant (via zero-copy mmap)

⚡

⚡ FLASHATTENTION ENGINE Peak IO-Aware Silicon

Jul 2023 Hardware

FlashAttention-2 & Kernel Fusion

Achieving 73% of theoretical peak GPU throughput via sequence-query outer loops

Tri Dao

⚡ FLASHATTENTION CORE

O_i = \text{softmax}\left(\frac{Q_i K^T}{\sqrt{d_k}}\right) V

From: Outer loop over keys/values requiring synchronization across thread blocks

To: Outer loop over queries eliminating inter-thread block atomics and maximizing warp occupancy

A100 Peak Utilization: 73% (2x faster than FlashAttn-1)

Max Context Feasible: 64k - 128k (at interactive speeds)

Aug 2023 Agent Loops

Local Agent Tooling: Ollama, LM Studio & Air-Gapped Coding Daemons

Containerizing local models with Modelfiles, OpenAI-compatible local APIs, and zero-cost private agentic loops

Jeffrey Morgan (Ollama), LM Studio Team & Local Agent Ecosystem

∑ GPU Layer Offloading Memory Split

V_{\text{VRAM}} \ge \frac{N_{\text{offload}}}{N_{\text{layers}}} V_{\text{model}} + V_{\text{KV}}

From: Manual CLI flag management, fragmented weight downloads, and brittle Python wrapper servers

To: Unified local background daemons with automated GPU layer offloading, Modelfile versioning, and OpenAI-standard REST endpoints

Token API Bill: $0.00 / mo (100% Free Local)

Localhost RTT: <1 ms (Zero cloud latency)

Nov 2023 Agent Loops

Deterministic Patching, Edit Primitives & AST Skeleton Maps

Replacing full-file hallucination with surgical string search-and-replace & unified diffs

Aider, SWE-bench Team, Tree-sitter Developers

∑ Turn Step Complexity with KV Caching

T_{\text{turn}} = O(N^2) \;\longrightarrow\; O(N)

From: Emitting entire 1,000-line source files to modify a single line (high hallucination & latency)

To: Surgical search-and-replace blocks and unified git diffs validated by AST parsers

Token Consumption per Edit: -85% (surgical patches)

Patch Application Success: >92% (up from 45%)

2024: FlashAttention-3, Blackwell NVFP4 & Latent Context (MLA)

Early 2024 Hardware

FP8 Precision & Low-Latency Serving Engines (TensorRT-LLM)

In-flight continuous batching, paged KV caching, and kernel fusion

NVIDIA (Hopper Architecture / TensorRT-LLM)

∑ Dual FP8 Numerical Precision Formats

x_{\text{FP8}} = (-1)^s \cdot 2^{e - \text{bias}} \cdot \left(1 + \frac{m}{2^M}\right)

From: Static batching and 16-bit weight storage requiring multi-GPU node clusters for 70B models

To: Single-GPU 8-bit serving with dynamic token-level continuous batching and zero-fragmentation paged memory

Param Memory Footprint: 1 Byte / Param (half of BF16)

Throughput: 2.5x higher (on H100 clusters)

⚡

⚡ ATTENTION ARCHITECTURE Foundation Context

May 2024 Context Layer

Multi-Head Latent Attention (MLA) & Context Economy

Compressing KV cache by 85–93% via low-rank latent vector projection with decoupled RoPE

DeepSeek AI (DeepSeek-V2 / DeepSeek-V3)

⚡ CANONICAL ATTENTION

\begin{aligned} \mathbf{c}_t^{KV} &= W^{DKV} \mathbf{h}_t \\[3pt] \mathbf{k}_{t, i}^C &= W^{UK}_i \mathbf{c}_t^{KV}, \quad \mathbf{v}_{t, i}^C = W^{UV}_i \mathbf{c}_t^{KV} \end{aligned}

From: Caching full multi-head Key and Value tensors for all layers (huge VRAM footprint stalling long-context agents)

To: Caching a compact low-rank latent vector with matrix absorption into query projections during inference

KV Cache Footprint: -85% to -93% (vs MHA baseline)

Concurrent Agent Sessions: 5x - 8x (per GPU)

⚡

⚡ FLASHATTENTION ENGINE Peak IO-Aware Silicon

Jul 2024 Hardware

FlashAttention-3: Asynchrony, TMA & Low-Precision FP8 Overlap

Unlocking 750–850 TFLOPs/s on Hopper H100 via Tensor Memory Accelerator, warp specialization, and incoherent processing

Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao (Stanford / Together AI / Princeton)

⚡ FLASHATTENTION CORE

\text{TMA}(K_j, V_j) \longrightarrow \text{WGMMA}(Q_i K_j^T) \parallel \text{softmax}(S_{ij})

From: Synchronous warp execution in FlashAttention-2 where threads stall on shared memory barriers and register-allocated tile loads

To: Decoupled asynchronous execution with 1 Producer Warp Group and 2 Consumer Warp Groups overlapping TMA loads, WGMMA multiplies, and softmax

Hopper H100 Throughput: 740–850 TFLOPs (1.5x–2.0x vs FA-2)

Peak Hardware Efficiency: 78% Peak (Highest on GH100)

Aug 2024 Training & RL

Multi-Token Prediction (MTP)

Enforcing long-range syntax tree lookahead and zero-overhead speculative decoding

Gloeckle et al. (Meta) & DeepSeek AI

∑ Multi-Token Prediction Training Loss

\mathcal{L}_{\text{MTP}} = -\frac{1}{T} \sum_{t=1}^T \sum_{k=1}^K \lambda_k \log P\left(x_{t+k} \mid \mathbf{h}_t^{(k)}; \theta\right)

From: Greedy single-token prediction vulnerable to short-sighted syntactic traps (e.g. premature bracket closes)

To: Parallel horizon prediction forcing representations to plan entire statement blocks and function signatures

Speculative Speedup: 2x - 3x (lossless verification)

Code Reasoning Accuracy: +3.5% (on HumanEval/SWE)

⚡

⚡ ATTENTION ARCHITECTURE Foundation Context

Oct 2024 Context Layer

Cognitive Neuropsychology of Attention & Pharmacodynamics

Prefrontal catecholamines (DA, NE, ACh), ADHD pharmacology (Elvanse, Ritalin), and transcending inattention in autonomous agents

Arnsten, Posner, Barkley & Modern AI Neuro-Homology

⚡ CANONICAL ATTENTION

\begin{aligned} \text{SNR}_{\text{PFC}} &= \frac{\mu_{\text{signal}}(D_1, \alpha_{2A})}{\sigma_{\text{noise}}^2} \\[4pt] P_{ij} &= \text{softmax}\left(\frac{Q_i K_j^T}{\tau \sqrt{d_k}}\right) \end{aligned}

From: Treating attention purely as heuristic matrix multiplication without biological grounding in working memory limits, noise gating, or impulse control

To: Unified bi-directional homology between PFC catecholaminergic gain control (Arnsten inverted-U) and artificial attention temperature/caching

Prefrontal SNR: +14.8 dB (Arnsten Inverted-U Peak)

Distractor Suppression: 94.2% (-68% agentic tool thrashing)

Late 2024 Hardware

NVIDIA Blackwell: 2nd Gen Transformer Engine, NVFP4 Microscaling & NVLink-5

Dual-die 10 TB/s NV-HBI, 20 PFLOPS FP4 microscaling, and 1.8 TB/s NVLink-5 exascale clusters for trillion-parameter coding agents

NVIDIA Architecture & Systems Engineering

∑ Block Microscaled Floating Point (OCP MXFP4 Standard)

X_{i} = S_{\text{block}} \times x_{i}^{\text{NVFP4}}, \quad S_{\text{block}} \in \text{E8M0}, \; x_i \in \text{E2M1}

From: Monolithic single-die GPUs bound by reticle size limits (Hopper GH100) running FP8 with per-tensor quantization

To: Dual-die 208B-transistor chiplet packaging with fine-grained per-vector block microscaling (NVFP4) and hardware decompression engines

Compute Throughput: 20 PFLOPS FP4 (5x vs Hopper FP8)

NVLink-5 Bandwidth: 1.8 TB/s/GPU (2x vs Hopper)

Dec 2024 Training & RL

Critic-Free RL: Group Relative Policy Optimization (GRPO)

Eliminating value network overhead via group sample statistics and verifiable test execution

DeepSeek AI (DeepSeek-R1-Zero / DeepSeek-R1)

∑ Group Relative Advantage Normalization

A_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\})}

From: Actor-Critic PPO requiring an extra 70B+ critic model in GPU memory prone to value divergence

To: Critic-free group advantage calculated from compiler passes and deterministic test verification

Critic VRAM Overhead: 0 GB (-50% training GPU memory)

SWE-bench Verification: Surged (powers DeepSeek-R1)

2025: AMD RDNA 4 WMMA, Test-Time Scaling & Critic-Free RL

Jan 2025 Scaling

Test-Time Compute & Search Scaling Laws

Scaling inference compute log-linearly via MCTS, PRMs, and sandbox rollouts

OpenAI (o1, o3) & DeepSeek AI (R1)

∑ Inference Reasoning Decomposition

P(Y \mid X) = \sum_{Z} P(Z \mid X) P(Y \mid X, Z)

From: Greedy single-pass forward generation expecting flawless code on first attempt

To: System-2 deliberative search: hypothesizing, branching in ephemeral sandboxes, pruning failing branches

SWE-bench Verified: >70% (up from 13% in 2023)

Scaling Dimension: C_test (bypasses pretrain plateau)

Early 2025 Hardware

AMD RDNA 4: 2nd Gen AI Accelerators, FP8 WMMA & Client Agent Silicon

Democratizing local agent execution with native FP8 matrix throughput, Wave32 matrix execution, and unified ROCm 6

AMD GPU Architecture & ROCm Core Systems

∑ Dual-Issue Wave Matrix Multiply-Accumulate (WMMA)

D = A \cdot B + C, \quad A, B \in \mathbb{R}^{16 \times 16}_{\text{FP8/INT4}}

From: Client GPUs relying on generic SIMD ALUs with limited 1st gen WMMA requiring cloud API calls for code generation

To: Dedicated 2nd Gen AI Accelerators with double-rate native FP8/FP4 WMMA instructions integrated into consumer workstations

WMMA AI Throughput: 2x vs RDNA 3 (Native FP8 & INT4)

Local Agent Speed: >100 tps (On-device Qwen 7B)

2026: Industrial MCP, Micro-VM Sandboxes & Autonomous SEA

2025–2026 Agent Loops

Industrial Agent Protocols & Micro-VM Sandboxes

Model Context Protocol (MCP), Code Graph PageRank, and FP4 Blackwell Microscaling

Anthropic (MCP), Cloud Native Agent Alliances, NVIDIA (Blackwell)

∑ Code Graph Personalized PageRank Retrieval

\mathbf{p} = (1 - d)\mathbf{v}_{\text{query}} + d \cdot \mathbf{A}_{\text{norm}}^T \mathbf{p}

From: Ad-hoc text scraping, naive vector RAG, and dangerous unisolated local shell execution

To: Standardized MCP RPCs, static-analysis graph PageRank, and micro-VM snapshots with instant rollback

SWE-bench Verified Pass Rate: >70% (autonomous production-grade)

Context Retrieval Precision: >94% (via Call-Graph PageRank)

Hardware Footprint: 4x density (NVFP4 on Blackwell)