00: Overview

How this module fits into QTM 447

In QTM 447, we treat modern machine learning models as optimization problems over high-dimensional parameter spaces.
Neural networks, variational models, and diffusion models may look complex on the surface, but at their core they all rely on the same machinery:

gradients
Hessians (explicit or implicit)
iterative optimization algorithms

This module is designed to offload that machinery from live class time.

Rather than learning optimization at the same time as new models, you will use familiar models—binary and multinomial logistic regression—to build a deep, mechanical understanding of how gradients, Hessians, and optimization algorithms actually work. When we later introduce SGD, backpropagation, and neural networks in class, you should recognize them immediately as direct generalizations of what you do here.

Think of this module as learning the grammar of optimization so that class time can focus on ideas.

Learning Goals

By the end of this module, you will be able to:

Implement Gradient Descent and Newton’s Method from scratch for
- binary logistic regression
- multinomial (softmax) logistic regression
Derive gradients and Hessians using
- matrix calculus rules
- computational graphs and the chain rule
Interpret the gradient for multinomial logistic regression as a
tensor-shaped object (a matrix of class-specific gradients), and connect this directly to how gradients flow through a neural network linear layer.
Diagnose optimization behavior by understanding
- step size (learning rate)
- conditioning
- why some methods converge slowly while others converge in a few steps

Why logistic regression?

Logistic regression is not the point of this module—it is the vehicle.

You have already seen logistic regression before. That familiarity is intentional. It allows us to focus on how optimization works, rather than what the model means.

Everything you do here reappears later in the course:

The gradient for multinomial logistic regression has the same structure as the gradient of a neural network’s final linear layer.
Newton’s method clarifies what second-order information buys you—and why it becomes computationally infeasible in deep learning.
Gradient descent paths, step sizes, and conditioning directly motivate stochastic optimization methods like SGD and Adam.

If you understand this module well, backpropagation and neural network training will feel mechanical rather than mysterious.

Prerequisites

You are expected to be comfortable with:

Linear algebra
Matrix multiplication, transposes, basic inverses.
Multivariable calculus
Partial derivatives and gradients (formal proofs are not required).
Python programming
Especially numpy and basic plotting with matplotlib.
Logistic regression concepts
Sigmoid/softmax functions and likelihood-based loss functions.

This module will review and reinforce these ideas, but it will not re-teach them from scratch.

How to use these notes

These notes are designed to be worked through actively, not skimmed.

Run the code cells if you are working locally.
Study the figures showing loss surfaces, gradients, and optimization paths.
Pay close attention to how gradients are built step-by-step using computational graphs.

Above all, we emphasize shape tracking.

Shape Tracking

Throughout these notes, you will see annotations such as X \in \mathbb{R}^{N \times P}

W \in \mathbb{R}^{P \times K}

or code comments like W.shape == (d, K).

This is not cosmetic.

Most optimization and backpropagation errors come from losing track of shapes.

If you can always say what object lives in what space, the algebra and code become straightforward.

How this will show up in class

You are responsible for understanding this material before we cover:

stochastic gradient descent and optimization dynamics
backpropagation in neural networks
training deep and generative models

In lecture, we will assume you know:

what a gradient and Hessian are,
how they are computed for logistic-type models,
and why iterative optimization is necessary.

Class time will focus on why we use certain architectures and algorithms—not on re-deriving these mechanics.

--- title: "00: Overview" --- ## How this module fits into QTM 447 In QTM 447, we treat modern machine learning models as **optimization problems over high-dimensional parameter spaces**. Neural networks, variational models, and diffusion models may look complex on the surface, but at their core they all rely on the same machinery: - gradients - Hessians (explicit or implicit) - iterative optimization algorithms This module is designed to **offload that machinery from live class time**. Rather than learning optimization *at the same time* as new models, you will use familiar models—**binary and multinomial logistic regression**—to build a deep, mechanical understanding of how gradients, Hessians, and optimization algorithms actually work. When we later introduce SGD, backpropagation, and neural networks in class, you should recognize them immediately as **direct generalizations of what you do here**. Think of this module as learning the *grammar* of optimization so that class time can focus on *ideas*. --- ## Learning Goals By the end of this module, you will be able to: 1. **Implement** Gradient Descent and Newton’s Method *from scratch* for - binary logistic regression - multinomial (softmax) logistic regression 2. **Derive** gradients and Hessians using - matrix calculus rules - computational graphs and the chain rule 3. **Interpret** the gradient for multinomial logistic regression as a **tensor-shaped object** (a matrix of class-specific gradients), and connect this directly to how gradients flow through a neural network linear layer. 4. **Diagnose** optimization behavior by understanding - step size (learning rate) - conditioning - why some methods converge slowly while others converge in a few steps --- ## Why logistic regression? Logistic regression is not the point of this module—it is the *vehicle*. You have already seen logistic regression before. That familiarity is intentional. It allows us to focus on **how optimization works**, rather than *what the model means*. Everything you do here reappears later in the course: - The gradient for multinomial logistic regression has the **same structure** as the gradient of a neural network’s final linear layer. - Newton’s method clarifies what second-order information buys you—and why it becomes computationally infeasible in deep learning. - Gradient descent paths, step sizes, and conditioning directly motivate stochastic optimization methods like SGD and Adam. If you understand this module well, backpropagation and neural network training will feel *mechanical rather than mysterious*. --- ## Prerequisites You are expected to be comfortable with: - **Linear algebra** Matrix multiplication, transposes, basic inverses. - **Multivariable calculus** Partial derivatives and gradients (formal proofs are *not* required). - **Python programming** Especially `numpy` and basic plotting with `matplotlib`. - **Logistic regression concepts** Sigmoid/softmax functions and likelihood-based loss functions. This module will *review and reinforce* these ideas, but it will not re-teach them from scratch. --- ## How to use these notes These notes are designed to be **worked through actively**, not skimmed. - Run the code cells if you are working locally. - Study the figures showing loss surfaces, gradients, and optimization paths. - Pay close attention to how gradients are built step-by-step using computational graphs. Above all, we emphasize **shape tracking**. ::: {.callout-tip} ## Shape Tracking Throughout these notes, you will see annotations such as $$X \in \mathbb{R}^{N \times P}$$ or $$W \in \mathbb{R}^{P \times K}$$ or code comments like `W.shape == (d, K)`. This is not cosmetic. Most optimization and backpropagation errors come from losing track of shapes. If you can always say *what object lives in what space*, the algebra and code become straightforward. ::: --- ## How this will show up in class You are responsible for understanding this material **before** we cover: - stochastic gradient descent and optimization dynamics - backpropagation in neural networks - training deep and generative models In lecture, we will **assume** you know: - what a gradient and Hessian are, - how they are computed for logistic-type models, - and why iterative optimization is necessary. Class time will focus on *why* we use certain architectures and algorithms—not on re-deriving these mechanics.