Continuously Updated · Machine Learning & AI Engineering

Architectures, Mathematical Foundations & Production ML Systems

Welcome to Will's personal engineering log. Exploring rigorous statistical learning theory, Transformer mechanics, LLM fine-tuning & alignment, optimization dynamics, and high-performance machine learning workflows.

Core Topic Series

25+

In-depth Articles

100%

Code & Mathematical Derivations

2026-06-18 • 15 min read • Deep Learning

Demystifying Transformer Architectures: From Scaled Dot-Product Attention to FlashAttention-3 GPU Kernel Optimization

A rigorous mathematical derivation of multi-head self-attention, query-key-value geometric projection, and a low-level dive into SRAM tiling strategies eliminating High Bandwidth Memory (HBM) IO bottlenecks.

#Transformer #FlashAttention #CUDA #PyTorch

2026-05-24 • 18 min read • LLMs

Modern LLM Fine-Tuning & Alignment: Low-Rank Adaptation (LoRA/QLoRA) and Direct Preference Optimization (DPO)

Bridging full-parameter fine-tuning constraints with parameter-efficient fine-tuning (PEFT), followed by a comparative analysis of RLHF reward modeling versus DPO closed-form loss formulations.

#LLM #LoRA #DPO #Alignment

2026-04-12 • 12 min read • Statistical ML

Advanced Gradient Boosted Decision Trees: Architectural Dissection of XGBoost, LightGBM, and CatBoost

Benchmarking second-order Taylor expansions, histogram-based split finders, Gradient-based One-Side Sampling (GOSS), Exclusive Feature Bundling (EFB), and ordered target encoding on tabular structures.

#XGBoost #LightGBM #CatBoost #TabularML

2026-03-05 • 10 min read • Optimization & Systems

Evolution of Neural Optimizers: Mathematical Convergence of AdamW, Lion, and Non-Convex Landscape Traversal

Deconstructing first- and second-moment estimations, the mathematical justification for decoupling L2 regularization from weight decay, and escaping saddle points via cosine annealing schedules.

#AdamW #Optimizers #NumericalMethods #Convergence

2026-02-14 • 20 min read • Deep Learning

Mathematical Foundations of Diffusion Probabilistic Models: From DDPM to Continuous Score-Based SDEs

A rigorous journey through forward Markovian noising, Evidence Lower Bound (ELBO) reparameterization, and the unified framework of stochastic differential equations and reverse-time drift estimation.

#Diffusion #DDPM #GenerativeAI #SDE

2026-01-20 • 11 min read • Statistical ML

Modeling Extreme Class Imbalance: Penalized Likelihood (Firth), Probability Calibration & Production Metrics

Addressing severe class sparsity in high-stakes environments: implementing Firth's bias reduction, Platt scaling, Isotonic regression, and selecting between ROC-AUC and PR-AUC for reliable decision thresholds.

#Calibration #ImbalancedData #Statistics #Metrics