Concept 02: Cross-Entropy & Classification Loss

In Concept 20, we used Mean Squared Error (MSE) to measure errors for continuous numbers (like meters or flywheel RPM). But what if our robot is classifying objects—like determining whether a game piece in front of its intake is a Note, a Coral, or an Algae?

For category probabilities, we use Cross-Entropy Loss.

Open the interactive demo below to adjust the model’s confidence for each object class and watch how Cross-Entropy imposes severe penalties when the model is confidently wrong.

The Everyday Robot Problem

Your autonomous vision system detects an object on the carpet. The neural network outputs a probability distribution across 3 possible game piece classes:

# Model predictions (must add up to 1.0 / 100%)
predicted_probs = {"Note": 0.85, "Coral": 0.10, "Algae": 0.05}

The human referee confirms the object is indeed a Note (True Label = 100% Note).

How should we score the model’s prediction?

• If the model said 99% Note, it was confident and correct. It should receive nearly 0.00 loss.
• If the model said 50% Note, it was uncertain. It should receive a moderate loss penalty.
• If the model said 1% Note (and was 99% convinced it was Coral!), it was confidently wrong. It should receive a massive loss penalty!

1. The Natural Logarithm as a Surprise Meter

To heavily penalize confident mistakes, we take the negative natural logarithm (-ln(p)) of the probability assigned to the correct class:

• Loss = -ln(1.00) = 0.00 (Zero loss when 100% confident and correct)
• Loss = -ln(0.85) = 0.16 (Low loss)
• Loss = -ln(0.50) = 0.69 (Moderate loss)
• Loss = -ln(0.10) = 2.30 (High loss)
• Loss = -ln(0.01) = 4.60 (Extreme penalty!)
• Loss = -ln(0.0001) = 9.21 (Approaches infinity as confidence in the truth approaches 0)

2. Solving It in Code (Java)

First-Principles Java: Cross-Entropy Loss

public class CrossEntropy {
    public static void main(String[] args) {
        // True One-Hot Label: [Note, Coral, Algae] (True object is Note at index 0)
        double[] yTrue = {1.0, 0.0, 0.0};

        // Model Predicted Probabilities (from Softmax)
        double[] yPred = {0.85, 0.10, 0.05};

        // Cross-Entropy: -sum(yTrue * ln(yPred))
        double loss = 0.0;
        for (int i = 0; i < yTrue.length; i++) {
            if (yTrue[i] > 0.0) {
                loss -= yTrue[i] * Math.log(Math.max(yPred[i], 1e-15));
            }
        }

        System.out.printf("Cross-Entropy Loss: %.4f%n", loss);
    }
}

3. Math! Translation Sidebar

Here is how Cross-Entropy is written formally:

L_CE = - ∑ yᵢ · ln(pᵢ)

For a single correct class c, this simplifies directly to:

L_CE = -ln(p_c)

How to Read This Out Loud:

• L_CE: Cross-Entropy Loss.
• yᵢ (“y sub i”): The true probability for class i (either 1.0 for the true class, or 0.0 for all other classes).
• pᵢ (“p sub i”): The model’s predicted probability for class i (between 0.0 and 1.0).
• ln(pᵢ): The natural logarithm of the predicted probability.

Why Not Just Use MSE for Classification?

If the model predicts 0.01 instead of 1.0, MSE produces an error of (0.01 - 1.0)² = 0.98. That is a small, bounded number that doesn’t push the network hard enough to fix dangerous mistakes. Cross-Entropy produces an error of 4.60+, providing huge gradient slopes that rapidly steer the weights away from confident errors.

4. Bridge to Machine Learning & LLMs

• YOLO & Vision Classifiers: When an object detector locates a bounding box, it uses MSE for the bounding box corners [x, y, w, h] and Cross-Entropy for the class probabilities (Note vs Robot vs Reef).
• Large Language Models (LLMs): When ChatGPT or Gemini generates text, it produces a probability distribution over 50,000 vocabulary tokens. During training, the loss for every single generated word is Cross-Entropy: -ln(P(correct_next_word))!