Research Papers

AlexNet

• AlexNet Convolution LayerEasy
• Data AugmentationMedium
• Dropout RegularizationEasy
• Local Response NormalizationMedium
• Overlapping Max PoolingEasy
• ReLU Activation FunctionEasy

Arcee Trinity

Arcee Trinity Large Technical Report · Official GitHub · Hugging Face

• Coarse-Grained MoE FFN (Plus)HardPlus
• Full Forward Pass (Plus)HardPlus
• Gated Attention (Plus)MediumPlus
• Interleaved RoPE + NoPE (Plus)EasyPlus
• NoPE Layer (Plus)MediumPlus
• Sandwich Norm (Plus)EasyPlus
• Sigmoid MoE Router (Plus)EasyPlus
• SMEBU Load Balancing (Plus)EasyPlus
• Transformer Block (Plus)HardPlus

BERT

• BERT PoolerEasy
• Fine-tuning ArchitectureHard
• Masked Language ModelingMedium
• Next Sentence PredictionHard
• Segment EmbeddingsEasy
• WordPiece TokenizationHard

DDPM

• DDPM SamplingMedium
• DDPM Training LossMedium
• Forward Diffusion ProcessMedium
• Noise ScheduleEasy
• Reverse Diffusion ProcessMedium

DeepSeek-V3

DeepSeek-V3 Technical Report · Official GitHub

• Decoupled RoPE (Plus)MediumPlus
• Dense Prefix Layers (Plus)EasyPlus
• Full Forward Pass (Plus)HardPlus
• KV Compression (Plus)EasyPlus
• KV Reconstruction (Plus)EasyPlus
• Load Balancing (Plus)MediumPlus
• MoE Router (Plus)MediumPlus
• Multi-head Latent Attention (Plus)HardPlus
• Multi-Token Prediction (Plus)EasyPlus
• Shared Expert (Plus)EasyPlus
• Sparse MoE FFN (Plus)HardPlus
• Transformer Block (Plus)HardPlus

DenseNet

• Bottleneck Layer (DenseNet-B)Medium
• Channel Growth and CompressionEasy
• Composite Layer (BN-ReLU-Conv)Medium
• Dense Block (Concatenative Connectivity)Hard
• Full DenseNet Forward PassHard
• Transition LayerMedium

GAN

• Complete GAN SystemHard
• GAN DiscriminatorMedium
• GAN GeneratorEasy
• GAN Loss FunctionsMedium
• GAN Training LoopHard
• Mode Collapse DetectionEasy

Gemma 3

Gemma 3 Technical Report · Official GitHub

• Full Gemma 3 Block (Plus)HardPlus
• Gemma 3 Attention Block (Plus)HardPlus
• Global vs Local Layer Routing (Plus)EasyPlus
• QK-Norm (Plus)EasyPlus
• Sliding Window Attention (Plus)MediumPlus

GLM-4.5

GLM-4.5 Technical Report · Official GitHub

• Dense SwiGLU (Plus)EasyPlus
• Full Forward (Plus)HardPlus
• GQA Attention (Plus)MediumPlus
• Group Router (Plus)MediumPlus
• MoE Forward (Plus)HardPlus
• MTP Head (Plus)MediumPlus
• Partial RoPE (Plus)EasyPlus
• QK Norm (Plus)EasyPlus
• RMSNorm (Plus)EasyPlus
• RoPE Freqs (Plus)EasyPlus
• Shared Expert (Plus)EasyPlus
• Transformer Block (Plus)HardPlus

GPT-2

Language Models are Unsupervised Multitask Learners · Official GitHub

• BPE Encode/Decode (Plus)MediumPlus
• BPE Training (Plus)HardPlus
• Causal Masked Attention (Plus)MediumPlus
• Decoder Block (Plus)HardPlus
• Feed-Forward Network (Plus)EasyPlus
• Full Forward Pass (Plus)HardPlus
• GELU Activation (Plus)EasyPlus
• Greedy Decoding (Plus)MediumPlus
• Layer Normalization (Plus)EasyPlus
• Multi-Head Attention (Plus)HardPlus
• Residual Weight Scaling (Plus)MediumPlus
• Scaled Dot-Product Attention (Plus)MediumPlus
• Token + Position Embedding (Plus)EasyPlus
• Top-k Sampling (Plus)MediumPlus

gpt-oss

gpt-oss model card · Official GitHub

• Attention Sinks (Plus)MediumPlus
• Full Forward (Plus)HardPlus
• GQA Attention (Plus)MediumPlus
• MoE Forward (Plus)HardPlus
• MoE Router (Plus)MediumPlus
• MXFP4 Dequant (Plus)MediumPlus
• Sliding Window (Plus)EasyPlus
• Transformer Block (Plus)HardPlus
• YaRN RoPE (Plus)MediumPlus

GRU

• Candidate Hidden StateEasy
• Complete GRU CellEasy
• Complete GRU NetworkHard
• Hidden State UpdateEasy
• Reset GateEasy
• Update GateEasy

Kimi K3

• Block Attention ResidualsHard
• Full Attention ResidualsMedium
• Gated MLAHard
• KDA Context ParallelismHard
• KDA RecurrenceHard
• Multi-Teacher On-Policy DistillationMedium
• MXFP4 Routed Expert LinearHard
• Per-Head MuonMedium
• Quantile BalancingHard
• SiTU-GLUEasy
• Stable LatentMoEHard

LLaMA

The Llama 3 Herd of Models · Official GitHub

• Full Forward Pass (Plus)HardPlus
• Grouped Query Attention (Plus)MediumPlus
• KV Head Repeat (Plus)EasyPlus
• RMSNorm (Plus)EasyPlus
• RoPE Frequency Table (Plus)EasyPlus
• Rotary Positional Embeddings (Plus)MediumPlus
• SwiGLU FFN (Plus)EasyPlus
• Transformer Block (Plus)HardPlus

LSTM

• Cell State UpdateEasy
• Complete LSTM CellMedium
• Complete LSTM NetworkHard
• Forget GateEasy
• Input GateEasy
• Output GateEasy

ResNet

• BatchNorm in ResNetMedium
• Bottleneck BlockMedium
• Convolutional BlockEasy
• Full ResNet AssemblyHard
• Identity BlockEasy
• Skip Connection AnalysisMedium

RNN

• Backpropagation Through TimeEasy
• Complete Vanilla RNNHard
• Forward Through SequenceMedium
• Hidden StateEasy
• RNN CellEasy
• Vanishing GradientsEasy

Transformer

• Embedding LayerEasy
• Encoder BlockMedium
• Feed-Forward NetworkEasy
• Layer NormalizationEasy
• Multi-Head AttentionHard
• Positional EncodingMedium
• Scaled Dot-Product AttentionMedium
• TokenizationMedium

U-Net

• Complete U-Net NetworkHard
• U-Net BottleneckEasy
• U-Net Decoder BlockMedium
• U-Net Encoder BlockEasy
• U-Net Output LayerEasy
• U-Net Skip ConnectionsEasy

VAE

• Complete VAE NetworkEasy
• ELBO Loss FunctionEasy
• KL Divergence LossEasy
• Reparameterization TrickEasy
• VAE DecoderEasy
• VAE EncoderEasy

VGG

• Complete VGG NetworkEasy
• VGG Classifier HeadEasy
• VGG ConfigurationEasy
• VGG Conv BlockEasy
• VGG Feature ExtractorMedium
• VGG Max PoolingEasy

ViT

• Class Token [CLS]Easy
• Classification HeadEasy
• Complete Vision TransformerHard
• Patch EmbeddingEasy
• Position EmbeddingEasy
• ViT Encoder BlockMedium

word2vec

• CBOW Forward PassMedium
• Frequent-Word SubsamplingEasy
• Negative Sampling DistributionEasy
• SGNS Gradient StepMedium
• Skip-gram Negative Sampling LossMedium
• Skip-gram Pair GenerationEasy

Pages

• Transformer Lab