Classification Diagnostics: Confusion Matrix, Precision, Recall & F1-Score

💻 Open-Source Code: The Python simulation scripts, vectorized confusion matrix implementations, multi-class aggregators, and optimal threshold search routines are open-sourced on GitHub: myhlow/pocket-data-science (05-mathematical-foundations).

💡 Core Principle: Accuracy hides mistakes; the confusion matrix exposes them. Decoupling False Positives (Type I errors) from False Negatives (Type II errors) is essential because in real-world systems and competitive benchmarks, the operational and statistical costs of these two failure modes are almost never symmetric.

1  ·  The Confusion Matrix Anatomy & The Four Fundamental Outcomes

A Confusion Matrix is a cross-tabulated 2×2 contingency table that compares the ground truth labels against the predictions produced by a binary classifier. Every prediction in the evaluation set maps onto exactly one of four mutually exclusive quadrants:

Layout Convention Note: In machine learning frameworks like Scikit-Learn (sklearn.metrics.confusion_matrix), rows represent the ground truth classes while columns represent the predicted classes: row 0 corresponds to actual negatives, row 1 to actual positives. Be vigilant when reading academic papers or medical literature, where axes are sometimes reversed.

2  ·  The Accuracy Paradox & The Perils of Class Imbalance

Overall classification accuracy is computed as the proportion of all correct decisions divided by the total number of evaluations:

While intuitive, accuracy suffers from the Accuracy Paradox when class distributions are imbalanced. Consider a disease screening cohort of N = 1,000 patients, where only 50 patients (5.0%) have the condition, and 950 (95.0%) are healthy:

Relying on accuracy would lead an automated model selection pipeline to discard the trained machine learning model in favor of the completely useless dummy classifier. This is why competitive Kaggle benchmarks and real-world production systems discard raw accuracy in favor of decoupled diagnostic metrics.

3  ·  Precision vs. Recall: The Fundamental Engineering Trade-Off

To resolve the accuracy paradox, we decompose performance into two complementary perspectives: quality of alerts (Precision) versus completeness of coverage (Recall).

The Precision-Recall Seesaw

Precision and Recall exist in intrinsic opposition. If you adjust your model to be more conservative (demanding 95% probability before flagging an instance as positive), False Positives drop to near zero, causing Precision to surge. However, many borderline positive cases will be classified as negative, increasing False Negatives and causing Recall to plummet.

Conversely, lowering the threshold to catch every marginal positive case boosts Recall toward 100%, but sweeps in hundreds of benign false alarms, cratering Precision.

4  ·  The Harmonic Mean: Mathematical Formulation of F1 & F-Beta

To optimize and compare models with a single scalar score, we require a balanced blend of Precision and Recall. Taking the standard arithmetic mean fails: a model with Precision = 1.0 and Recall = 0.0 would score (1.0 + 0.0) / 2 = 0.50, creating an illusion of acceptable performance for a model that catches zero positive instances.

Instead, machine learning relies on the Harmonic Mean, which punishes extreme imbalances by being dominated by the smaller of the two terms:

Notice that True Negatives (TN) do not appear anywhere in the F1 formula. This property makes F1 completely invariant to massive floods of easy negatives, preserving its diagnostic sensitivity under extreme class skew.

The Generalized F-Beta (β) Family

In applications where one error type is objectively more damaging than the other, the generalized F-Beta score introduces a parameter β that governs the relative importance of Recall versus Precision:

5  ·  Multi-Class Aggregations: Macro, Micro, and Weighted Formulations

When extending classification from binary to C ≥ 3 classes (such as 3-class LLM preference modeling in Chatbot Arena or 10-class MNIST digit recognition), we construct a C × C confusion matrix. For each individual class k, we compute its One-vs-Rest (OvR) metrics:

• TP_k: Diagonal element M[k, k].
• FP_k: Sum of column k minus M[k, k] (instances falsely labeled as class k).
• FN_k: Sum of row k minus M[k, k] (actual class k instances assigned to other classes).

To summarize the entire model with a single diagnostic score, we choose among three aggregation strategies:

Worked Numerical Example: 3-Class Animal Classifier (100 Samples)

Consider an edge-deployed animal vision classifier evaluated on N = 100 test images across three categories: Cat (30 samples), Dog (50 samples), and Bird (20 samples). The resulting confusion matrix is:

Computing the per-class diagnostics:

• Cat (Class 0): TP = 25, FP = 9, FN = 5 → Precision = 25/34 = 0.7353, Recall = 25/30 = 0.8333 → F1 = 0.7812
• Dog (Class 1): TP = 42, FP = 5, FN = 8 → Precision = 42/47 = 0.8936, Recall = 42/50 = 0.8400 → F1 = 0.8660
• Bird (Class 2): TP = 14, FP = 5, FN = 6 → Precision = 14/19 = 0.7368, Recall = 14/20 = 0.7000 → F1 = 0.7179

Aggregating the overall metrics across the three classes:

• Macro F1: (0.7812 + 0.8660 + 0.7179) / 3 = 0.7884
• Weighted F1: (30×0.7812 + 50×0.8660 + 20×0.7179) / 100 = 0.8110
• Micro F1 (Accuracy): (25 + 42 + 14) / 100 = 0.8100

Notice that Macro F1 is significantly lower (0.7884) than Weighted F1 (0.8110). Because Bird is the minority class (only 20 samples) and suffered the lowest F1 score (0.7179), the unweighted Macro calculation penalizes the model fairly for poor minority performance, whereas Weighted F1 and Accuracy let the dominant Dog class mask the deficiency.

6  ·  Probability Threshold Tuning & PR Curves on Mobile RAM Budgets

Most classification algorithms (LightGBM, XGBoost, CatBoost, Logistic Regression, Neural Networks) do not output discrete 0 or 1 labels; they output continuous predicted probabilities p = P(y = 1 | x) ∈ [0.0, 1.0]. By default, virtually all frameworks convert these probabilities into labels using an arbitrary threshold τ = 0.50:

Under class imbalance, τ = 0.50 is almost never optimal. By performing a vectorized grid search over τ ∈ [0.01, 0.99] across Out-of-Fold (OOF) cross-validation predictions, you can locate the exact decision boundary that maximizes your evaluation metric.

Empirical Demonstration: Threshold Optimization on Imbalanced Data

Evaluating 500 samples containing a 10% positive rate under a calibrated GBDT model:

PR Curve vs. ROC Curve Under Severe Skew

Why do experienced Kaggle practitioners plot the Precision-Recall (PR) Curve rather than the standard Receiver Operating Characteristic (ROC) curve on imbalanced datasets?

• The ROC-AUC Flaw: ROC plots True Positive Rate vs. False Positive Rate (FPR = FP / (FP + TN)). When True Negatives are vast (e.g., 99,000 negatives and 1,000 positives), adding 500 false alarms barely moves FPR (from 0 to 0.005), giving the illusion that the model is near-flawless with an ROC-AUC of 0.98.
• The PR-AUC Clarity: The PR curve plots Precision (TP / (TP + FP)) vs. Recall (TP / (TP + FN)). Those exact same 500 false positives cut Precision in half (from 100% to 66%), immediately exposing the degradation on the PR-AUC plot.

7  ·  Robust Vectorized Python Implementation (Pure NumPy)

On mobile edge hardware (such as Android Termux running within limited memory envelopes), loading heavy scientific suites like full Scikit-Learn pipelines for metric evaluation introduces unnecessary memory overhead. Here is a production-grade, zero-dependency, vectorized NumPy diagnostic suite:

import numpy as np

def compute_confusion_matrix(y_true, y_pred, num_classes=None):
    \"\"\"Compute C x C confusion matrix using vectorized indexing.\"\"\"
    if num_classes is None:
        num_classes = max(np.max(y_true), np.max(y_pred)) + 1
    
    # Vectorized 2D bincount: row = true, col = pred
    indices = y_true * num_classes + y_pred
    cm = np.bincount(indices, minlength=num_classes**2).reshape(num_classes, num_classes)
    return cm

def classification_diagnostics(y_true, y_pred, average=\"macro\", beta=1.0):
    \"\"\"Vectorized computation of Precision, Recall, and F-Beta metrics.\"\"\"
    cm = compute_confusion_matrix(y_true, y_pred)
    num_classes = cm.shape[0]
    
    tp = np.diag(cm).astype(float)
    fp = cm.sum(axis=0) - tp
    fn = cm.sum(axis=1) - tp
    support = cm.sum(axis=1)
    total_samples = support.sum()
    
    # Per-class metrics with zero-division handling
    precision_per_class = np.divide(tp, tp + fp, out=np.zeros_like(tp), where=(tp + fp) != 0)
    recall_per_class = np.divide(tp, tp + fn, out=np.zeros_like(tp), where=(tp + fn) != 0)
    
    beta_sq = beta ** 2
    denom = (beta_sq * precision_per_class) + recall_per_class
    f_beta_per_class = np.divide((1 + beta_sq) * (precision_per_class * recall_per_class),
                                 denom, out=np.zeros_like(tp), where=denom != 0)
    
    # Aggregation
    if average == \"macro\":
        agg_precision = float(np.mean(precision_per_class))
        agg_recall = float(np.mean(recall_per_class))
        agg_f_beta = float(np.mean(f_beta_per_class))
    elif average == \"weighted\":
        agg_precision = float(np.sum(precision_per_class * support) / total_samples)
        agg_recall = float(np.sum(recall_per_class * support) / total_samples)
        agg_f_beta = float(np.sum(f_beta_per_class * support) / total_samples)
    elif average == \"micro\":
        total_tp = tp.sum()
        total_fp = fp.sum()
        total_fn = fn.sum()
        agg_precision = float(total_tp / (total_tp + total_fp)) if (total_tp + total_fp) > 0 else 0.0
        agg_recall = float(total_tp / (total_tp + total_fn)) if (total_tp + total_fn) > 0 else 0.0
        agg_f_beta = float(2 * agg_precision * agg_recall / (agg_precision + agg_recall)) if (agg_precision + agg_recall) > 0 else 0.0
    
    accuracy = float(np.trace(cm) / total_samples)
    return {
        \"confusion_matrix\": cm,
        \"accuracy\": accuracy,
        \"precision\": agg_precision,
        \"recall\": agg_recall,
        \"f_beta\": agg_f_beta,
        \"per_class\": {
            \"precision\": precision_per_class,
            \"recall\": recall_per_class,
            \"f_beta\": f_beta_per_class,
            \"support\": support
        }
    }

def optimize_binary_threshold(y_true, y_prob, metric=\"f1\", steps=100):
    \"\"\"Vectorized search to locate optimal probability decision threshold.\"\"\"
    thresholds = np.linspace(0.01, 0.99, steps)
    best_score, best_thresh = -1.0, 0.50
    
    for t in thresholds:
        pred = (y_prob >= t).astype(int)
        tp = np.sum((y_true == 1) & (pred == 1))
        fp = np.sum((y_true == 0) & (pred == 1))
        fn = np.sum((y_true == 1) & (pred == 0))
        
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        score = 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0.0
        
        if score > best_score:
            best_score = score
            best_thresh = float(t)
            
    return best_thresh, best_score

8  ·  Practical Takeaways for Machine Learning Competitions & Edge Systems

When designing machine learning models for edge deployment or competing on Kaggle leaderboards, apply these four battle-tested diagnostic principles:

• Rule 1: Always Normalize the Confusion Matrix: Raw counts can deceive. Plotting the row-normalized confusion matrix (where rows sum to 1.0) displays recall per class directly, immediately revealing which minority classes are being silently absorbed by the majority.
• Rule 2: Tune Thresholds Strictly Out-of-Fold (OOF): Never optimize your decision threshold on the entire training set. Threshold searching on in-sample predictions leads to severe overfitting. Tune your optimal threshold τ* across validation folds, then average or ensemble the fold thresholds before applying to test data.
• Rule 3: Match Loss Functions to Diagnostic Objectives: When evaluating on F1 or F-Beta under heavy skew, standard binary cross-entropy loss can pull probabilities toward zero. Adopting Focal Loss (downweighting easy negatives) or class-weighted log loss during training produces better probability distributions for threshold tuning.
• Rule 4: Beware the Multi-Class Micro-F1 Trap: If a competition specifies Micro-F1 as the evaluation metric on single-label data, recognize that it is mathematically identical to Accuracy. Spend your modeling effort maximizing raw classification accuracy rather than balancing per-class precision and recall.

Recommended Machine Learning Reading

Explore related hands-on engineering case studies in our mobile data science series:

• Master Curriculum: Pocket Data Science: Complete Learning Path & 6-Track Roadmap.
• Foundational Theory: Multi-Class Log Loss Explained: Math, MLE & Python Code (Module 0).
• Part I: Titanic 10-Fold Ensemble: Training on Android (Top 3% Global).
• Part II: Spaceship Titanic: Domain Deductions & Native CatBoost (Top 6% Global).
• Part V: NLP Disaster Tweets: Text Vectorization & Regularized Baselines.
• Part X: LMSYS Chatbot Arena: Mobile GBDT Machine Learning & Dual Ensemble.

Continue the Pocket Data Science Series