From-scratch GPU LLM training.

Most LLM training stacks are vendored — PyTorch + Megatron + DeepSpeed + a cloud abstraction layer. Each link is a black box from the math perspective and the deployment story is "rent a hyperscaler".

The KeiLab stack is written bottom-up: six gradient kernels, hand-rolled, finite-difference verified, parity-checked against a CPU reference layer-backward.

• Rented NVIDIA datacenter GPUs — a single-GPU tier for parameter-efficient fine-tunes, and a multi-GPU tier used only when wall-clock pressure justifies the cost.
• Local unified-memory ARM GPU workstation (NVIDIA GB10) — where these kernels were developed and measured: inference, KV-streaming, GPU kernel development, and experiments outside the cloud cost budget.

• Six gradient kernels on GPU — matrix-multiply, RMSNorm, SiLU / SwiGLU, RoPE, softmax cross-entropy, scaled-dot-product attention.
• GPU layer-backward orchestrating all six — in our tests it matched the CPU reference on 10 of 10 tensors.
• GPU forward path with paged key-value cache, saved-activations for backward, and FlashAttention-2 wired for long context.
• GPU lm-head — softmax cross-entropy + argmax fused in a single launch.
• Layer-streaming forward — full 64-layer Qwen3-32B runs by holding one layer at a time. Depth-independent peak RAM (about 4.4 GB in our tests).

Every cloud launch goes through a pre-flight memory and cost gate before the meter starts — measured per-GPU budget, dry-run loss, single-GPU-first sizing — under the lab's standing rules. Launches are orchestrated end-to-end (spawn → upload → train → poll → download → terminate) so there are no orphan billed pods.

Training artifacts stay on the lab's own machines. The audit trail is preserved per run.

The stack is aimed at production-scale training at full depth, which needs streaming offload and grad-clip discipline. Partners can run the kernels in their own environment to verify them before committing.