Skip to content

Lab 103: Deep Dive: Train a Mini-GPT on CPU (Optional)

Level: L400 Journey: llm-foundations Time: ~180 min Cost: free

Phase 3 · local first

The required route is deterministic and local. The Real route is optional and includes an estimate and cleanup.

What You'll Learn

  • Implement tokenization, embeddings, attention, and next-token loss
  • Train a tiny model on CPU with fixed seeds
  • Inspect overfitting and sampling
  • Explain why the exercise is educational, not production training

Prerequisites

  • Python 3.10+ and a terminal
  • No credentials for the required route
  • Read the linked official specification before using a hosted adapter

Visual model

flowchart LR
  T[Tokens] --> E[Embeddings]
  E --> A[Causal attention]
  A --> L[Logits]
  L --> X[Cross-entropy]
  X --> U[CPU update]

Scenario

OutdoorGear needs a reviewable workshop implementation with deterministic fixtures, explicit trust boundaries, measurable acceptance criteria, and a reversible optional hosted adapter.

Guided exercise

Step 1: tokenize

  1. Run the deterministic fixture for this step.
  2. Record the declared input, output, policy decision, and failure behavior.
  3. Change one bounded behavior and run the local test again.

Evidence: Save the command, result, and one sentence explaining why the evidence is trustworthy.

Step 2: train

  1. Run the deterministic fixture for this step.
  2. Record the declared input, output, policy decision, and failure behavior.
  3. Change one bounded behavior and run the local test again.

Evidence: Save the command, result, and one sentence explaining why the evidence is trustworthy.

Step 3: sample

  1. Run the deterministic fixture for this step.
  2. Record the declared input, output, policy decision, and failure behavior.
  3. Change one bounded behavior and run the local test again.

Evidence: Save the command, result, and one sentence explaining why the evidence is trustworthy.

Step 4: analyze_limits

  1. Run the deterministic fixture for this step.
  2. Record the declared input, output, policy decision, and failure behavior.
  3. Change one bounded behavior and run the local test again.

Evidence: Save the command, result, and one sentence explaining why the evidence is trustworthy.

Run locally

cd docs/docs/en/labs/lab-103
python -m unittest discover -v

Simulation / Real path

Use the included deterministic fixtures and local validator. Estimated cost: **US$0**.

Supporting kit

The site build generates stable PDF and ZIP downloads. CI also keeps a temporary artifact mirror for 30 days.

Scored knowledge check

Why keep a local contract?
Answer

It provides a deterministic baseline before adding hosted variability and cost.

When is the Real route complete?
Answer

Only after validation evidence and cleanup are recorded.

How should tool output be treated?
Answer

As untrusted data until policy and schema validation succeed.

Exit criteria

  • Local tests pass without credentials.
  • The diagram has a text explanation and does not depend on color.
  • The scored quiz was reviewed.
  • The Real path has a cost cap and cleanup plan.
  • Sources and stability status were checked.

Official public sources