mnist-nn-scratch / 2025
2025 · Machine learning · Experiments
mnist-nn-scratch
Backprop by hand, then checked honestly against PyTorch.
Overview
A deliberately small neural network that proves the fundamentals still matter when the framework layer is removed.
- Role
- Machine learning · from scratch
- Year
- 2025
- Stack
- NumPy · Python · PyTorch · MNIST
- Source
- GitHub ↗
mnist-nn-scratch — hero visual
image slot — asset coming later
Reproducing the mechanics by hand
I wanted to understand the mechanics of training well enough to reproduce them by hand, then compare the result against a PyTorch baseline without hand-waving the math. Small on purpose: a two-layer network where every shape bug is visible and every gradient can be checked.
Keeping the comparison fair
Forward pass, backpropagation, and stochastic gradient descent in NumPy, with seed, data split, and architecture controlled so the PyTorch baseline comparison stays honest. The model converged close to the baseline — and the shape bugs along the way built sharper intuition for how training actually fails than any tutorial could.
Story visual — Keeping the comparison fair
image slot — asset coming later
Structure
- 01
NumPy forward pass
- 02
Hand-derived backprop
- 03
SGD
- 04
PyTorch baseline
The result
The model converged close to the PyTorch baseline and turned shape bugs into a much sharper intuition for how training actually fails.
- 2-layer MLP
- trained on MNIST
- baseline parity with PyTorch
If I built it again
Frameworks hide exactly the failure modes you need to recognize in production. Next step would be extending the scratch implementation to convolutions to feel weight sharing the same way.
Further reading + source