I implemented a minimalist Llama-2 language model from scratch in PyTorch, including self-attention, Rotary Positional Embeddings (RoPE), RMSNorm, SwiGLU activation functions, and core transformer blocks.
I pretrained and evaluated an 8-layer, roughly 42M-parameter model on TinyStories, then tested it on text completion, zero-shot sentiment classification, and task-specific finetuning.