Tutorials

Every tutorial below has a one-click Launch in Modal Notebook button. The button opens the .ipynb in a fresh Modal Notebook — the first code cell is a ! pip install git+https://github.com/modal-projects/training-gym.git@joy/initial-setup that installs modal-training-gym into the notebook kernel, so the rest of the cells run as-is.

The Difficulty column is a rough self-assessed signal for where to start: Beginner tutorials are single-node and introduce one framework concept; Intermediate tutorials span 1–2 nodes or wire up something non-default (custom reward, external script); Advanced tutorials run on ≥2 nodes with non-trivial parallelism (tensor-parallel, colocated RL, long context) and assume familiarity with the underlying framework.

Tutorials

Intro

• quickstart — Shared concepts: config containers, framework factories, volume layout, running the pipeline. Beginner · — (concepts) · — ·

RL

• slime_gsm8k — Qwen3-4B GRPO on GSM8K (colocated). Advanced · slime · 4 × 8×H200 ·
• slime_haiku — Qwen3-4B GRPO on haiku poems — structure score + LLM judge. Intermediate · slime · 1 × 8×H200 ·
• verl_qwen3_32b_gsm8k — Qwen3-32B GRPO on GSM8K (Megatron + vLLM). Advanced · verl · 4 × 8×H100 ·

SFT

• ms_swift_glm_4_7_gsm8k — GLM-4.7 LoRA SFT on GSM8K (Megatron). Advanced · ms_swift · 4 × 8×B200 ·
• ms_swift_custom_hf — Custom HuggingFace model (SmolLM2-135M) LoRA SFT — inline ModelConfiguration subclass, no catalog entry. Beginner · ms_swift · 1 × 1×H100 ·
• megatron_glm_4_7_longmit128k — GLM-4.7 LoRA on LongMIT-128K (NeMo bridge). Advanced · megatron · 4 × 8×B200 ·
• starcoder_llama2_7b — Llama-2-7B SFT on Go + Rust (FSDP). Intermediate · torchrun + hf_accelerate · 2 × 8×H100 ·

Misc

• nanogpt_owt — GPT-2 124M on OpenWebText. Intermediate · — (clones karpathy/nanoGPT) · 2 × 8×H100 ·
• lightning_fabric_demo — Transformer on WikiText2 (Fabric DDP). Beginner · lightning · 2 × 8×H100 ·
• resnet50_imagenet — ResNet50 on ImageNet (DDP). Intermediate · — (clones pytorch/vision) · 4 × 8×H100 ·
• nccl_benchmark — All-reduce bandwidth benchmark (500000×2000 fp32). Beginner · — (NCCL utility) · 2 × 8×H100 ·
• ray_slime_standalone — Ray-on-Modal pattern demo. Intermediate · — (raw ModalRayCluster) · 2 × 8×H100 ·

Running from the CLI instead

Every tutorial is also a plain .py file runnable via modal run. See the top-level README.md for the usage pattern.

Authoring a new tutorial

Tutorials are generated from a Python source file under tutorial_generator/. The generator AST-walks each source and emits tutorials/<bucket>/<name>/<name>.py + tutorials/<bucket>/<name>/<name>.ipynb. Edit the source, not the generated files — the pre-commit hook (.pre-commit-config.yaml) regenerates on commit, so hand-edits to the .py / .ipynb will be overwritten.

Regenerate manually after editing a source file:

uv run python tutorials/generate_tutorial.py

Cell decorators

Top-level functions in the source file produce cells; function names and argument lists don't matter, only decorator + body. Cells appear in source order.

• @markdown — the function's docstring becomes one markdown cell (.ipynb) or # comments (.py).
• @code — the function's body (dedented) becomes one code cell.
• @shell("…") — the string argument is emitted verbatim as a code cell (supports ! pip install … shell magic for notebook installs).
• @py_only / @notebook_only — restrict a cell to one output format; stacks on top of @markdown / @code / @shell.

TUTORIAL_METADATA

Every source file declares a module-level TUTORIAL_METADATA dict that drives the catalog table above. Fields:

Key	Required	Purpose
framework	yes	Backtick-wrapped framework name (e.g., '`slime`') or '— (…)' for standalone tutorials
cluster_shape	yes	Human-readable shape ('4 × 8×H200') — must match what the config actually launches
summary	yes	One-line description of what the tutorial trains
difficulty	recommended	One of 'Beginner', 'Intermediate', 'Advanced'. Falls back to — if omitted. See the column description at the top of this page for what each tier means.
order	yes	Integer controlling row order in the catalog; lower appears earlier

Writing style

Treat the notebook as the tutorial's home — the root README and this index are maps, the .ipynb is the walkthrough. Aim for roughly:

• An intro cell naming what it trains, why someone would run it, and where to watch progress (W&B project/group, Modal dashboard).
• A markdown cell before each @code block that names the non-obvious choices (why these hyperparams, why this cluster shape, why this chat template). Don't repeat what the code itself makes obvious.
• Custom pieces (reward functions, dataset preprocessing, model wrappers) get their own explanation cell.
• A "Run it" section splitting CLI invocation (@py_only) from cell-by-cell interactive invocation (@notebook_only).
• Optional: a "Serve / evaluate / next step" tail, where relevant — see slime_haiku for the shape.

slime_gsm8k and slime_haiku are the reference examples for tutorial narration depth.