# From-scratch GPU LLM training

Canonical page: https://keilab.io/ai/gpu-llm/

ai infrastructure · gpu llm

A from-scratch GPU forward and backward stack for Qwen3-32B, written bottom-up and parity-checked against a CPU reference on the lab's local NVIDIA GB10, aimed at full-depth training of the 32B model.

## Why from-scratch

Most LLM training stacks are vendored — PyTorch + Megatron + DeepSpeed + a cloud abstraction layer. Each link is a black box from the math perspective and the deployment story is "rent a hyperscaler".

The KeiLab stack is written bottom-up: six gradient kernels, hand-rolled, finite-difference verified, parity-checked against a CPU reference layer-backward.

## Hardware

- Rented NVIDIA datacenter GPUs — a single-GPU tier for parameter-efficient fine-tunes, and a multi-GPU tier used only when wall-clock pressure justifies the cost.
- Local unified-memory ARM GPU workstation (NVIDIA GB10) — where these kernels were developed and measured: inference, KV-streaming, GPU kernel development, and experiments outside the cloud cost budget.

## What is built

- Six gradient kernels on GPU — matrix-multiply, RMSNorm, SiLU / SwiGLU, RoPE, softmax cross-entropy, scaled-dot-product attention.
- GPU layer-backward orchestrating all six — in our tests it matched the CPU reference on 10 of 10 tensors.
- GPU forward path with paged key-value cache, saved-activations for backward, and FlashAttention-2 wired for long context.
- GPU lm-head — softmax cross-entropy + argmax fused in a single launch.
- Layer-streaming forward — full 64-layer Qwen3-32B runs by holding one layer at a time. Depth-independent peak RAM (about 4.4 GB in our tests).

## Operating discipline

Every cloud launch goes through a pre-flight memory and cost gate before the meter starts — measured per-GPU budget, dry-run loss, single-GPU-first sizing — under the lab's standing rules. Launches are orchestrated end-to-end (spawn → upload → train → poll → download → terminate) so there are no orphan billed pods.

Training artifacts stay on the lab's own machines. The audit trail is preserved per run.

## What it is for

The stack is aimed at production-scale training at full depth, which needs streaming offload and grad-clip discipline. Partners can run the kernels in their own environment to verify them before committing.

cross-references

- [→ MAX (language used to write the kernels)](https://keilab.io/ai/max/)
- [→ KeiSeiKit runtime](https://keilab.io/ai/keiseikit/)
- [→ MAX OS](https://keilab.io/ai/max-os/)
- [@ About the lab](https://keilab.io/about/)

Status: In development

Target model: Qwen3-32B

Developed on: Local NVIDIA GB10 (ARM, unified memory)

Backend: CUDA

Parity: In our tests: 10/10 tensors vs CPU ref, rel-L2 < 5e-6

Sensitivity: Engineering public

[⊸ Engineering inquiry](https://keilab.io/contact/)

---
KeiLab is operated by KeiSei LLC, a Wyoming limited liability company. +1 307 210 8538
Site index for AI agents: https://keilab.io/llms.txt