Concept 03: Discrete Distributions & Softmax

▶ Interactive Demo: Softmax & Temperature Visualizer

Open the interactive demo below to adjust the raw class logits and drag the Temperature slider to observe how temperature turns sharp greedy argmax into smooth probability distributions.

1. The Real-World Problem: Turning Raw Scores into Confidences

When a robot’s computer vision neural network inspects an object on the field, the raw output layer generates unconstrained real numbers called logits:

• Game Piece (Note): +3.2
• Field Element (Reef): +1.1
• Opponent Robot: -0.8

Logits have two major problems:

1. They can be negative (e.g. -0.8), but probabilities can never be negative.
2. Their sum is not 1.0 (3.2 + 1.1 - 0.8 = 3.5 ≠ 1.0).

How do we convert raw numbers into clean, calibrated probabilities that sum to 100%?

2. Solving It in Code (Java & WPILib)

First-Principles Java: Softmax Probabilities

// Raw neuron outputs (logits)
double[] logits = {2.5, 1.0, 0.2}; // [Note, Coral, Algae]

// 1. Exponentiate each logit
double expSum = 0.0;
double[] expValues = new double[logits.length];
for (int i = 0; i < logits.length; i++) {
    expValues[i] = Math.exp(logits[i]);
    expSum += expValues[i];
}

// 2. Normalize to sum to 1.0 (100%)
double[] probabilities = new double[logits.length];
for (int i = 0; i < logits.length; i++) {
    probabilities[i] = expValues[i] / expSum;
}

System.out.printf("P(Note): %.1f%%, P(Coral): %.1f%%, P(Algae): %.1f%%%n",
    probabilities[0] * 100, probabilities[1] * 100, probabilities[2] * 100);

3. Bridge to Machine Learning: LLM Token Generation

In Large Language Models (like ChatGPT, Gemini, and Claude):

• The AI computes logits across its entire 32,000-word dictionary for the next token.
• Temperature Sampling:
  • When coding or doing math, the user sets Temperature = 0.2 (predictable, deterministic answers).
  • When writing poetry or creative stories, the user sets Temperature = 0.8 (sampling diverse tokens).

4. Review Checkpoints

Checkpoint 1

A classifier outputs two logits: z₁ = 0.0 and z₂ = 0.0. What are the resulting probabilities?

Solution:

• e⁰ = 1.0, e⁰ = 1.0.
• P₁ = 1.0 / (1.0 + 1.0) = 0.50 (50%).
• P₂ = 1.0 / (1.0 + 1.0) = 0.50 (50%). Equal logits always yield equal probabilities!

Checkpoint 2

Why can’t we simply divide logits by their sum (i.e. zᵢ / ∑ z) instead of using exponentials?

Solution: Because negative logits (e.g. -0.8) would produce invalid negative probabilities, and if the sum of logits happened to equal zero (∑ z = 0), the formula would crash from division by zero! Exponentiation guarantees every term is strictly positive (e^z > 0).