04: Binary Logistic Derivatives

Why this matters

To train logistic regression, we need the gradient.
To train it efficiently, we need the Hessian.

Rather than treating these as mysterious formulas, we will derive both by walking backward through the computational graph you just learned.

This section is the payoff:

once you understand this derivation, backpropagation will never feel abstract again.

Part I: Analytical Derivation (with Simplification)

In this section, we derive the gradient and Hessian analytically by walking backward through the computational graph and simplifying algebraically whenever possible.

This approach gives us insight into:

why the residual p_i - y_i appears,
why logistic regression is convex,
and why Newton’s method works so well here.

Later, we will deliberately turn off these simplifications to mirror how automatic differentiation actually works.

Recall the per-observation computation pipeline:

\mathbf x_i \;\longrightarrow\; z_i = \mathbf x_i^\top \mathbf w + b \;\longrightarrow\; p_i = \sigma(z_i) \;\longrightarrow\; L_i(p_i, y_i).

Each arrow corresponds to a node in the graph. Backpropagation means taking derivatives one node at a time, then chaining them together.

Step 1: Loss derivative (output node)

The binary cross-entropy loss for a single observation is

L_i = -\big(y_i \log p_i + (1 - y_i)\log(1 - p_i)\big).

Differentiating with respect to the probability node p_i:

\frac{\partial L_i}{\partial p_i} = -\frac{y_i}{p_i} + \frac{1-y_i}{1-p_i} = \frac{p_i - y_i}{p_i(1-p_i)}.

This derivative tells us how sensitive the loss is to small changes in the predicted probability.

Step 2: Sigmoid derivative (activation node)

The sigmoid node maps logits to probabilities:

p_i = \sigma(z_i).

Its derivative is

\frac{\partial p_i}{\partial z_i} = p_i(1 - p_i).

This factor appears everywhere in logistic regression and controls how strongly gradients propagate through confident predictions.

Step 3: Chain rule at the logit node

Now we combine the previous two steps:

\frac{\partial L_i}{\partial z_i} = \frac{\partial L_i}{\partial p_i} \frac{\partial p_i}{\partial z_i}.

Substituting:

\frac{\partial L_i}{\partial z_i} = \frac{p_i - y_i}{p_i(1-p_i)} \cdot p_i(1-p_i) = p_i - y_i.

This simplification is the key structural feature of logistic regression.

Learning is driven entirely by the residual p_i - y_i.

Everything else in the model just routes this error signal back to the parameters.

Step 4: Gradient with respect to the weights

The linear node is

z_i = \mathbf x_i^\top \mathbf w + b.

Its derivative with respect to the weight vector is

\frac{\partial z_i}{\partial \mathbf w} = \mathbf x_i.

Applying the chain rule one final time:

\frac{\partial L_i}{\partial \mathbf w} = \frac{\partial L_i}{\partial z_i} \frac{\partial z_i}{\partial \mathbf w} = (p_i - y_i)\mathbf x_i.

For the full dataset, the total loss is

L = \frac{1}{n}\sum_{i=1}^n L_i.

Because gradients of sums are sums of gradients:

\nabla_{\mathbf w} L = \frac{1}{n}\sum_{i=1}^n (p_i - y_i)\mathbf x_i = \frac{1}{n}\mathbf X^\top(\mathbf p - \mathbf y).

Shape check

[d \times n] \cdot [n \times 1] = [d \times 1].

The gradient has the same shape as the parameter it updates.

Gradient with respect to the intercept

The intercept enters the model through the same linear node:

z_i = \mathbf x_i^\top \mathbf w + b.

The derivative of the logit with respect to the intercept is

\frac{\partial z_i}{\partial b} = 1.

Applying the chain rule:

\frac{\partial L_i}{\partial b} = \frac{\partial L_i}{\partial z_i} \frac{\partial z_i}{\partial b} = (p_i - y_i).

For the full dataset:

\frac{\partial L}{\partial b} = \frac{1}{n} \sum_{i=1}^n (p_i - y_i).

Key observation

The gradients for \mathbf w and b share the same upstream term (p_i - y_i). Only the local derivative of the linear node differs.

Hessian derivation

The Hessian describes the curvature of the loss surface and is the key object used by Newton’s method.

Start from the gradient:

\nabla_{\mathbf w} L = \frac{1}{n}\mathbf X^\top(\mathbf p - \mathbf y).

Only \mathbf p depends on \mathbf w. Taking another derivative:

\mathbf H = \frac{1}{n}\frac{\partial}{\partial \mathbf w} \big(\mathbf X^\top \mathbf p\big) = \frac{1}{n}\mathbf X^\top \frac{\partial \mathbf p}{\partial \mathbf w}.

Jacobian of the probability vector

Each probability depends only on its own logit:

\frac{\partial p_i}{\partial z_i} = p_i(1-p_i).

Define a diagonal matrix

\mathbf S = \mathrm{diag}\big(p_1(1-p_1), \dots, p_n(1-p_n)\big).

Then

\frac{\partial \mathbf p}{\partial \mathbf z} = \mathbf S, \qquad \frac{\partial \mathbf z}{\partial \mathbf w} = \mathbf X.

Combining these pieces:

\mathbf H = \frac{1}{n}\mathbf X^\top \mathbf S \mathbf X.

Shape check

[d \times n]\,[n \times n]\,[n \times d] = [d \times d].

Interpretation

\mathbf S weights each observation by its uncertainty
confident predictions (p \approx 0 or 1) contribute little curvature
uncertain predictions contribute more curvature

Because p_i(1-p_i) \ge 0 for all i:

\mathbf S is positive semidefinite
\mathbf H = \frac{1}{n}\mathbf X^\top \mathbf S \mathbf X is positive semidefinite

Logistic regression is a convex optimization problem.

Part II: Gradient as a Product of Local Derivatives

The previous derivation relied on algebraic simplifications that make logistic regression especially clean.

However, automatic differentiation does not simplify expressions symbolically. Instead, it computes gradients by multiplying local derivatives node-by-node along the graph.

Let’s now re-derive the gradient without canceling terms.

Explicit chain rule (single observation)

From the graph:

\mathbf x_i \;\rightarrow\; z_i \;\rightarrow\; p_i \;\rightarrow\; L_i

the chain rule gives

\frac{\partial L_i}{\partial \mathbf w} = \frac{\partial L_i}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial \mathbf w}.

Substituting the local derivatives:

Loss node: \frac{\partial L_i}{\partial p_i} = \frac{p_i - y_i}{p_i(1-p_i)}
Sigmoid node: \frac{\partial p_i}{\partial z_i} = p_i(1-p_i)
Linear node: \frac{\partial z_i}{\partial \mathbf w} = \mathbf x_i

Putting everything together:

\frac{\partial L_i}{\partial \mathbf w} = \left( \frac{p_i - y_i}{p_i(1-p_i)} \right) \cdot \left( p_i(1-p_i) \right) \cdot \mathbf x_i.

Only after multiplying do the middle terms cancel, yielding

(p_i - y_i)\mathbf x_i.

Explicit chain rule for the intercept

The intercept uses the same computational graph and the same upstream derivatives.

The chain rule gives

\frac{\partial L_i}{\partial b} = \frac{\partial L_i}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial b}.

The only new local derivative is

\frac{\partial z_i}{\partial b} = 1.

Substituting:

\frac{\partial L_i}{\partial b} = \left( \frac{p_i - y_i}{p_i(1-p_i)} \right) \cdot \left( p_i(1-p_i) \right) \cdot 1.

After multiplication:

\frac{\partial L_i}{\partial b} = p_i - y_i.

For the full dataset:

\frac{\partial L}{\partial b} = \sum_{i=1}^n (p_i - y_i).

This is the clearest demonstration that backpropagation reuses upstream gradients.

Why this matters

Automatic differentiation systems (PyTorch, JAX, TensorFlow):

never “know” that cancellation will happen,
store intermediate values during the forward pass,
and multiply local derivatives mechanically during the backward pass.

The simplification we observed earlier is a property of the model, not the algorithm.

Backpropagation works even when no simplification is possible.

Part III: Swapping the Loss Function (Modularity in Action)

One of the main advantages of computational graphs is modularity.

To change the objective, we only need to change the loss node. All upstream derivatives remain unchanged.

Let’s demonstrate this by replacing the binary cross-entropy loss with the Brier score.

Brier score loss

For a single observation, the Brier score is

L_i^{\text{Brier}} = (p_i - y_i)^2.

This loss penalizes squared probability errors rather than log-likelihood errors.

New loss derivative

The only derivative that changes is the one at the loss node:

\frac{\partial L_i^{\text{Brier}}}{\partial p_i} = 2(p_i - y_i).

Everything else in the graph is identical.

Backpropagating through the same graph

Applying the chain rule:

\frac{\partial L_i^{\text{Brier}}}{\partial \mathbf w} = \frac{\partial L_i^{\text{Brier}}}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial \mathbf w}.

Substitute the local derivatives:

= 2(p_i - y_i) \cdot p_i(1-p_i) \cdot \mathbf x_i.

So the gradient becomes

\frac{\partial L}{\partial \mathbf w} = \frac{1}{n}\sum_{i=1}^n 2(p_i - y_i)p_i(1-p_i)\mathbf x_i.

Intercept gradient under the Brier score

The intercept gradient follows the same logic.

Applying the chain rule:

\frac{\partial L_i^{\text{Brier}}}{\partial b} = \frac{\partial L_i^{\text{Brier}}}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial b}.

Substituting the local derivatives:

= 2(p_i - y_i) \cdot p_i(1-p_i) \cdot 1.

So for the full dataset:

\frac{\partial L}{\partial b} = \frac{1}{n}\sum_{i=1}^n 2(p_i - y_i)p_i(1-p_i).

Once again, only the loss node derivative changed. Everything upstream is reused.

What changed — and what didn’t

✅ The linear node is unchanged
✅ The sigmoid node is unchanged
❌ Only the loss derivative changed

This is exactly how modern ML frameworks operate:

Change the loss → change one node → reuse the rest of the graph.

This is why computational graphs scale to: - new objectives, - new architectures, - and new learning problems.

Key takeaways

At this point, we have derived gradients three ways:

analytically with simplification,
mechanically via Jacobian products,
and modularly by swapping loss functions.

All three perspectives describe the same computation.

Each derivative corresponds to a node in the computational graph
The gradient is
\nabla_{\mathbf w} L = \frac{1}{n}\mathbf X^\top(\mathbf p - \mathbf y)
The Hessian is
\mathbf H = \frac{1}{n}\mathbf X^\top \mathbf S \mathbf X
Convexity guarantees a unique global minimum
Gradients for \mathbf w and b share the same upstream error signal; only local derivatives differ

Self-check questions

Why does the residual p_i - y_i appear naturally in the gradient?
Why does the Hessian down-weight observations with confident predictions?
What part of this derivation changes for multinomial logistic regression?

--- title: "04: Binary Logistic Derivatives" format: html: code-fold: true jupyter: python3 --- # Why this matters To *train* logistic regression, we need the **gradient**. To train it *efficiently*, we need the **Hessian**. Rather than treating these as mysterious formulas, we will derive both by **walking backward through the computational graph** you just learned. This section is the payoff: > once you understand this derivation, backpropagation will never feel abstract again. # Part I: Analytical Derivation (with Simplification) In this section, we derive the gradient and Hessian *analytically* by walking backward through the computational graph **and simplifying algebraically whenever possible**. This approach gives us insight into: - why the residual $p_i - y_i$ appears, - why logistic regression is convex, - and why Newton’s method works so well here. Later, we will deliberately *turn off* these simplifications to mirror how automatic differentiation actually works. Recall the per-observation computation pipeline: $$ \mathbf x_i \;\longrightarrow\; z_i = \mathbf x_i^\top \mathbf w + b \;\longrightarrow\; p_i = \sigma(z_i) \;\longrightarrow\; L_i(p_i, y_i). $$ Each arrow corresponds to a **node** in the graph. Backpropagation means taking derivatives *one node at a time*, then chaining them together. ## Step 1: Loss derivative (output node) The binary cross-entropy loss for a single observation is $$ L_i = -\big(y_i \log p_i + (1 - y_i)\log(1 - p_i)\big). $$ Differentiating with respect to the probability node $p_i$: $$ \frac{\partial L_i}{\partial p_i} = -\frac{y_i}{p_i} + \frac{1-y_i}{1-p_i} = \frac{p_i - y_i}{p_i(1-p_i)}. $$ This derivative tells us how sensitive the loss is to small changes in the predicted probability. ## Step 2: Sigmoid derivative (activation node) The sigmoid node maps logits to probabilities: $$ p_i = \sigma(z_i). $$ Its derivative is $$ \frac{\partial p_i}{\partial z_i} = p_i(1 - p_i). $$ This factor appears everywhere in logistic regression and controls how strongly gradients propagate through confident predictions. ## Step 3: Chain rule at the logit node Now we combine the previous two steps: $$ \frac{\partial L_i}{\partial z_i} = \frac{\partial L_i}{\partial p_i} \frac{\partial p_i}{\partial z_i}. $$ Substituting: $$ \frac{\partial L_i}{\partial z_i} = \frac{p_i - y_i}{p_i(1-p_i)} \cdot p_i(1-p_i) = p_i - y_i. $$ This simplification is the key structural feature of logistic regression. > **Learning is driven entirely by the residual $p_i - y_i$.** Everything else in the model just routes this error signal back to the parameters. ## Step 4: Gradient with respect to the weights The linear node is $$ z_i = \mathbf x_i^\top \mathbf w + b. $$ Its derivative with respect to the weight vector is $$ \frac{\partial z_i}{\partial \mathbf w} = \mathbf x_i. $$ Applying the chain rule one final time: $$ \frac{\partial L_i}{\partial \mathbf w} = \frac{\partial L_i}{\partial z_i} \frac{\partial z_i}{\partial \mathbf w} = (p_i - y_i)\mathbf x_i. $$ For the full dataset, the total loss is $$ L = \frac{1}{n}\sum_{i=1}^n L_i. $$ Because gradients of sums are sums of gradients: $$ \nabla_{\mathbf w} L = \frac{1}{n}\sum_{i=1}^n (p_i - y_i)\mathbf x_i = \frac{1}{n}\mathbf X^\top(\mathbf p - \mathbf y). $$ **Shape check** $$ [d \times n] \cdot [n \times 1] = [d \times 1]. $$ The gradient has the same shape as the parameter it updates. ### Gradient with respect to the intercept The intercept enters the model through the same linear node: $$ z_i = \mathbf x_i^\top \mathbf w + b. $$ The derivative of the logit with respect to the intercept is $$ \frac{\partial z_i}{\partial b} = 1. $$ Applying the chain rule: $$ \frac{\partial L_i}{\partial b} = \frac{\partial L_i}{\partial z_i} \frac{\partial z_i}{\partial b} = (p_i - y_i). $$ For the full dataset: $$ \frac{\partial L}{\partial b} = \frac{1}{n} \sum_{i=1}^n (p_i - y_i). $$ **Key observation** The gradients for $\mathbf w$ and $b$ share the same upstream term $(p_i - y_i)$. Only the *local derivative* of the linear node differs. ## Hessian derivation The Hessian describes the **curvature** of the loss surface and is the key object used by Newton’s method. Start from the gradient: $$ \nabla_{\mathbf w} L = \frac{1}{n}\mathbf X^\top(\mathbf p - \mathbf y). $$ Only $\mathbf p$ depends on $\mathbf w$. Taking another derivative: $$ \mathbf H = \frac{1}{n}\frac{\partial}{\partial \mathbf w} \big(\mathbf X^\top \mathbf p\big) = \frac{1}{n}\mathbf X^\top \frac{\partial \mathbf p}{\partial \mathbf w}. $$ ### Jacobian of the probability vector Each probability depends only on its own logit: $$ \frac{\partial p_i}{\partial z_i} = p_i(1-p_i). $$ Define a diagonal matrix $$ \mathbf S = \mathrm{diag}\big(p_1(1-p_1), \dots, p_n(1-p_n)\big). $$ Then $$ \frac{\partial \mathbf p}{\partial \mathbf z} = \mathbf S, \qquad \frac{\partial \mathbf z}{\partial \mathbf w} = \mathbf X. $$ Combining these pieces: $$ \mathbf H = \frac{1}{n}\mathbf X^\top \mathbf S \mathbf X. $$ **Shape check** $$ [d \times n]\,[n \times n]\,[n \times d] = [d \times d]. $$ ### Interpretation - $\mathbf S$ weights each observation by its uncertainty - confident predictions ($p \approx 0$ or $1$) contribute little curvature - uncertain predictions contribute more curvature Because $p_i(1-p_i) \ge 0$ for all $i$: - $\mathbf S$ is positive semidefinite - $\mathbf H = \frac{1}{n}\mathbf X^\top \mathbf S \mathbf X$ is positive semidefinite **Logistic regression is a convex optimization problem.** # Part II: Gradient as a Product of Local Derivatives The previous derivation relied on algebraic simplifications that make logistic regression especially clean. However, **automatic differentiation does not simplify expressions symbolically**. Instead, it computes gradients by multiplying **local derivatives node-by-node** along the graph. Let’s now re-derive the gradient *without* canceling terms. --- ## Explicit chain rule (single observation) From the graph: $$ \mathbf x_i \;\rightarrow\; z_i \;\rightarrow\; p_i \;\rightarrow\; L_i $$ the chain rule gives $$ \frac{\partial L_i}{\partial \mathbf w} = \frac{\partial L_i}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial \mathbf w}. $$ Substituting the local derivatives: - Loss node: $$ \frac{\partial L_i}{\partial p_i} = \frac{p_i - y_i}{p_i(1-p_i)} $$ - Sigmoid node: $$ \frac{\partial p_i}{\partial z_i} = p_i(1-p_i) $$ - Linear node: $$ \frac{\partial z_i}{\partial \mathbf w} = \mathbf x_i $$ Putting everything together: $$ \frac{\partial L_i}{\partial \mathbf w} = \left( \frac{p_i - y_i}{p_i(1-p_i)} \right) \cdot \left( p_i(1-p_i) \right) \cdot \mathbf x_i. $$ Only *after* multiplying do the middle terms cancel, yielding $$ (p_i - y_i)\mathbf x_i. $$ ### Explicit chain rule for the intercept The intercept uses the *same* computational graph and the *same* upstream derivatives. The chain rule gives $$ \frac{\partial L_i}{\partial b} = \frac{\partial L_i}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial b}. $$ The only new local derivative is $$ \frac{\partial z_i}{\partial b} = 1. $$ Substituting: $$ \frac{\partial L_i}{\partial b} = \left( \frac{p_i - y_i}{p_i(1-p_i)} \right) \cdot \left( p_i(1-p_i) \right) \cdot 1. $$ After multiplication: $$ \frac{\partial L_i}{\partial b} = p_i - y_i. $$ For the full dataset: $$ \frac{\partial L}{\partial b} = \sum_{i=1}^n (p_i - y_i). $$ This is the clearest demonstration that **backpropagation reuses upstream gradients**. --- ## Why this matters Automatic differentiation systems (PyTorch, JAX, TensorFlow): - never “know” that cancellation will happen, - store intermediate values during the forward pass, - and multiply local derivatives mechanically during the backward pass. The simplification we observed earlier is a **property of the model**, not the algorithm. > Backpropagation works even when no simplification is possible. # Part III: Swapping the Loss Function (Modularity in Action) One of the main advantages of computational graphs is **modularity**. To change the objective, we only need to change the **loss node**. All upstream derivatives remain unchanged. Let’s demonstrate this by replacing the binary cross-entropy loss with the **Brier score**. --- ## Brier score loss For a single observation, the Brier score is $$ L_i^{\text{Brier}} = (p_i - y_i)^2. $$ This loss penalizes squared probability errors rather than log-likelihood errors. --- ## New loss derivative The only derivative that changes is the one at the loss node: $$ \frac{\partial L_i^{\text{Brier}}}{\partial p_i} = 2(p_i - y_i). $$ Everything else in the graph is identical. --- ## Backpropagating through the same graph Applying the chain rule: $$ \frac{\partial L_i^{\text{Brier}}}{\partial \mathbf w} = \frac{\partial L_i^{\text{Brier}}}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial \mathbf w}. $$ Substitute the local derivatives: $$ = 2(p_i - y_i) \cdot p_i(1-p_i) \cdot \mathbf x_i. $$ So the gradient becomes $$ \frac{\partial L}{\partial \mathbf w} = \frac{1}{n}\sum_{i=1}^n 2(p_i - y_i)p_i(1-p_i)\mathbf x_i. $$ ### Intercept gradient under the Brier score The intercept gradient follows the same logic. Applying the chain rule: $$ \frac{\partial L_i^{\text{Brier}}}{\partial b} = \frac{\partial L_i^{\text{Brier}}}{\partial p_i} \cdot \frac{\partial p_i}{\partial z_i} \cdot \frac{\partial z_i}{\partial b}. $$ Substituting the local derivatives: $$ = 2(p_i - y_i) \cdot p_i(1-p_i) \cdot 1. $$ So for the full dataset: $$ \frac{\partial L}{\partial b} = \frac{1}{n}\sum_{i=1}^n 2(p_i - y_i)p_i(1-p_i). $$ Once again, only the **loss node derivative** changed. Everything upstream is reused. --- ## What changed — and what didn’t - ✅ The **linear node** is unchanged - ✅ The **sigmoid node** is unchanged - ❌ Only the **loss derivative** changed This is exactly how modern ML frameworks operate: > Change the loss → change one node → reuse the rest of the graph. This is why computational graphs scale to: - new objectives, - new architectures, - and new learning problems. # Key takeaways At this point, we have derived gradients three ways: - analytically with simplification, - mechanically via Jacobian products, - and modularly by swapping loss functions. All three perspectives describe the *same computation*. - Each derivative corresponds to a **node in the computational graph** - The gradient is $$ \nabla_{\mathbf w} L = \frac{1}{n}\mathbf X^\top(\mathbf p - \mathbf y) $$ - The Hessian is $$ \mathbf H = \frac{1}{n}\mathbf X^\top \mathbf S \mathbf X $$ - Convexity guarantees a unique global minimum - Gradients for $\mathbf w$ and $b$ share the same upstream error signal; only local derivatives differ # Self-check questions 1. Why does the residual $p_i - y_i$ appear naturally in the gradient? 2. Why does the Hessian down-weight observations with confident predictions? 3. What part of this derivation changes for multinomial logistic regression?