Pocket Data Science IV: Kaggle MNIST Subspace SVM Guide

💻 Open-Source Code: All 10 experiment scripts, pipelines, and reproduction guides are available on GitHub: github.com/myhlow/kaggle-digit-recognizer-mnist (and in the master monorepo: myhlow/pocket-data-science).

1  ·  The Mobile ML Environment

All data loading, model fitting, validation, and Kaggle submissions were executed entirely on a consumer smartphone:

• Host Architecture: Android 14 running Termux with a Debian userspace via PRoot Distro on 64-bit ARM (aarch64).
• Agentic CLI: Google Antigravity CLI (agy) pair-programming in bash, managing background tasks, and verifying data pipelines.
• Environment: Python 3.14 with pandas, numpy, scikit-learn, and the official kaggle CLI.
• Constraints: Operating within smartphone RAM and thermal envelopes requires algorithmic efficiency (e.g., avoiding raw O(N²) distance calculations across 42k × 28k × 784 floating-point matrices).

2  ·  Progression: 10-Tier Ladder (Day 1 & Day 2 Results)

Across Day 1, we utilized all 5 available daily submissions to establish clear empirical milestones against the 862 active teams on the public leaderboard:

3  ·  Tier 1: Zero-Training Baselines & “Ghost Templates”

Before training parameterized models, baseline tests define the floor of predictability:

• Empirical Random Guessing (10.14%): Drawing randomly according to class frequencies in the training set matches the theoretical expectation for 10 balanced classes (~10%).
• Constant Majority Class (11.40%): Digit 1 is slightly more frequent than others (4,684 of 42,000 samples, or 11.15%). Predicting 1 for all 28,000 test images yielded 11.40% on Kaggle.
• Nearest Centroid Classifier (81.45%): By simply calculating the element-wise arithmetic mean of all 4,200 training images per digit class, we obtain 10 average “ghost templates”. Classifying each test image to the template with highest cosine similarity requires zero parameter tuning or gradient updates, yet hits 81.45% accuracy, demonstrating the geometric separation already inherent in raw pixel alignments.

4  ·  Tier 2: The Linear Boundary & Non-Linear Latent Ensembles

Moving to machine learning algorithms revealed the threshold between linear and non-linear classification in 784 dimensions:

• Multinomial Softmax Regression (92.09%): Training a linear multinomial logistic regression classifier via L-BFGS sets 10 separating hyperplanes. It achieves 92.09%, illustrating the limit of linear separability without feature expansion or spatial convolutions.
• Dimensionality Reduction via PCA: Full k-Nearest Neighbors on raw pixels is prohibitive on phone CPUs (over 1.1 billion pairwise Euclidean distance calculations). By applying Principal Component Analysis (PCA) to compress 784 pixels to 55 components (preserving 84.0% variance), inference dropped to just 8 seconds.
• Neighborhood + Tree Ensembling: Combining distance-weighted k-NN (k = 5) with an Extra Trees Classifier (150 trees) on raw pixel partitions raised validation accuracy to 97.96% and public leaderboard accuracy to 97.58%, vaulting past 235 teams to Rank #571.

5  ·  Exp 06: Deep MLP & Empirical Computational Profiling on Mobile ARM64

To push beyond classical decision boundaries without exceeding smartphone hardware constraints, we designed Experiment 06: a tri-blend ensemble pairing a 235k-parameter Deep Multi-Layer Perceptron (MLP(256, 128)) with our distance-weighted k-NN(5) and ExtraTrees(150) models.

On a 5,000-sample stratified holdout, the Tri-Blend achieved 98.34% accuracy (+0.38% over Exp 05), cutting classification error by 18.6%. Here is the exact training and inference computational profile measured directly on our 8-core ARM64 Android device:

• The Dual-Axis 126,000-Sample Breakthrough: In Experiment 10, tripling the training partition to 126,000 samples by combining both horizontal (±1px on the X-axis) and vertical (±1px on the Y-axis) translational jitter provided complete 2D shift-invariance. Fitting the entire 126,000-sample pipeline (PCA + RBF-SVM) took 3.4 minutes on phone CPU and drove our Kaggle score to 0.98792 (Rank #406 / Top 47.05%, beating 457 teams)!
• The Latent Space Regularization Win: Projecting images onto a 55-dimensional orthogonal subspace did not just accelerate training—it acted as an optimal low-pass filter against boundary pixel noise, achieving 98.60% validation accuracy versus 98.24% on uncompressed 784 pixels while fitting 17× faster (7.2s vs 148.6s).
• The Dual Meta-Ensemble Synergy: In Experiment 08, weighting the calibrated RBF-SVM at 90% and Deep MLP at 10% demonstrated how soft neural representations resolve ambiguous edge cases without disrupting primary maximum-margin decision planes.

6  ·  Day 2 Milestone Achieved: Top 47.05% on Mobile Hardware (Rank #406 / 863)

With Experiment 10 scoring 0.98792, we officially breached the Top 50% cutoff (0.98739, Rank #431) to claim Rank #406 out of 863 active teams worldwide, outperforming 457 competitive teams.

• Zero GPU Resources: The entire progression—from empirical random guessing (0.10139) to classical ensembles (0.97578) and 126k dual-axis augmented subspace SVM (0.98792)—was engineered, trained, and submitted entirely on an Android smartphone CPU via Termux and Antigravity CLI.
• Looking Ahead to Day 3: With all 5 daily submissions successfully exhausted for Day 2, our next frontier is custom lightweight Convolutional Neural Networks (LeNet-5 & Modern Compact ResNets) built directly on PyTorch ARM64 to break into the Top 20%.

Field Tests & Applications: The Pocket Data Science Series

See this Termux + Antigravity CLI setup in action across competitive machine learning benchmarks on mobile hardware:

• Setup Guide: Running Google Antigravity CLI on Android with Termux PRoot.
• Part I: Training a 10-Fold Blended Ensemble on Titanic (Top 3%).
• Part II: Spaceship Titanic with Domain Deductions & CatBoost (Top 6%).
• Part III: House Prices Regression Baselines, Metric Alignment, and Ensembling (Top 19%).
• Part IV: Kaggle MNIST on Android: From Zero-Parameter Baselines to Subspace SVM (Top 47%).

Continue the Pocket Data Science Series