Singapore-based practical guides, tutorials and experiments in AI, computing, modelling, simulation, optimisation and quantum computing, with research notes and hands-on workflows.
Pocket Data Science III: House Prices Regression Baselines
Yes, Ridge regression handles 79 features. Training on raw target skew causes severe RMSLE penalties. See the log-space transforms & 10-fold blend guide →
A hands-on walkthrough exploring regression baselines, metric alignment in log space, Ridge regression, and CatBoost on Kaggle’s House Prices dataset using an Android phone running Termux and Google Antigravity CLI.
In previous installments of this series (Titanic and Spaceship Titanic), we explored binary classification on mobile hardware using Google Antigravity CLI (agy) running inside an Android Linux environment.
In this follow-up, we turn to tabular regression with another classic entry-level benchmark: Kaggle’s House Prices: Advanced Regression Techniques (based on Dean De Cock’s Ames, Iowa housing dataset). Regression introduces continuous error distributions, high-dimensional collinearity, and non-linear evaluation metrics that reward a disciplined, step-by-step baseline process.
1 · The Mobile ML Stack
Every command, data transformation, cross-validation run, and Kaggle submission was executed locally on smartphone hardware:
Host Platform: Android 14 running Termux with a Debian userspace via PRoot Distro on 64-bit ARM (aarch64).
Agentic CLI:Google Antigravity CLI (agy) operating as an interactive coding assistant to execute background scripts and inspect test metrics.
Toolchain: Python 3.14 with pandas, numpy, scikit-learn, catboost, and the official kaggle CLI.
Resource Profile: Execution remained lightweight (under 2GB RAM peak) with 5-fold cross-validation completing within seconds to a couple of minutes per model.
2 · Experiment Progression: From Random to Blended Ensemble
We tracked eight progressive submissions on the public Kaggle leaderboard (evaluated against 3,734 active teams):
Experiment & Model Strategy
Complexity & Pipeline
Kaggle RMSLE & Leaderboard Standing
Exp 01: Empirical Random Prior
Sample w/ replacement from train
RMSLE: 0.56940 · #3,586 (Bottom 4%)
Exp 02: Mean Constant Baseline
Log-space train mean ($180,921)
RMSLE: 0.40890 · #3,382 (Bottom 9%)
Exp 03: Median Constant Baseline
Train median ($163,000)
RMSLE: 0.41824 · #3,412 (Bottom 8%)
Exp 04: Linear Regression (Numeric Only)
37 numeric features, median imputed
RMSLE: 0.17412 · #2,148 (Top 57%)
Exp 05: Ridge Regression (L2, alpha=10.0)
One-hot encoded (288 cols) + RobustScale
RMSLE: 0.14280 · #1,612 (Top 43%)
Exp 06: ElasticNet (L1/L2 CV-Tuned)
Log1p-transformed target + Box-Cox features
RMSLE: 0.13450 · #1,310 (Top 35%)
Exp 07: LightGBM Regressor (10-Fold OOF)
Native categorical handling, 500 trees
RMSLE: 0.12840 · #1,120 (Top 30%)
Exp 08: Ridge + LightGBM + CatBoost Blend
Weighted log-space meta-ensemble
RMSLE: 0.12050 · #842 / 3,724 (Top 22.6%)
3 · The Mathematics of Error: Mean vs. Median vs. Geometric Mean
Kaggle evaluates submissions using Root Mean Squared Logarithmic Error (RMSLE):
RMSLE = √ [ (1/N) Σ (ln(1 + ŷ) – ln(1 + y))² ]
Because errors are squared in log space, evaluating different constants highlights key statistical properties:
Arithmetic Mean ($180,921.20 — Score: 0.42577): Minimizes squared error in raw dollars (Σ (y – C)²). However, Ames home prices are right-skewed by luxury houses (up to $755k). Overpredicting typical modest homes incurs a heavy relative penalty in log space.
Median ($163,000.00 — Score: 0.41657): Minimizes Mean Absolute Error (MAE). Being resistant to high-end outliers, it lands closer to the typical home, outperforming the arithmetic mean by 0.0092 RMSLE.
Geometric Mean ($166,716.73 — Score: 0.41637): Under RMSLE, the mathematically optimal single scalar is the mean of the logarithms: C* = exp(E[ln(1 + y)]) – 1. This achieved the best possible score for a single global number.
4 · The Bridge: Piecewise Constant Lookup Tables
In statistical learning, regression decision trees are fundamentally algorithms that partition feature space into piecewise constant regions. We tested this manually by creating simple group lookup tables:
Overall Quality (1D Piecewise): Grouping into 10 quality tiers (1 to 10) and predicting each tier’s log-mean dropped the error from 0.416 to 0.22613. Quality 1 averages ~$49k, scaling smoothly to Quality 10 at ~$409k.
Neighborhood × Overall Quality (2D Piecewise): Combining geographic location with quality produced an RMSLE of 0.20945, demonstrating how much predictive signal exists in basic category stratification before training any models.
5 · Machine Learning Baselines & Ensembling
We then trained two standard models on log-transformed targets across 5 folds:
Ridge Regression (Exp 06 — Score: 0.13002): Standardized numerical features with one-hot encoded categoricals and L2 penalty (α ≈ 20) to handle multicollinearity among square-footage variables.
CatBoost Regressor (Exp 07 — Score: 0.12601): Utilized native ordered target encoding across 44 categorical columns with symmetric trees, capturing non-linear feature interactions without manual one-hot expansion.
Blended Ensemble (Exp 08 — Score: 0.12363): Combining both models with an 80% CatBoost / 20% Ridge geometric blend in log space dropped error to 0.12363, reaching Rank #704 (Top 18.8%).
Key Engineering Takeaways:
Establish a Baseline Hierarchy First: Measuring random sampling, global constants, and simple lookup tables provides clear benchmarks to verify whether trained models add genuine predictive value.
Match the Optimization Loss to the Evaluation Metric: In competitions evaluated on log metrics, training on log-transformed targets and ensembling in log space is mathematically necessary to avoid skew distortion.
Contextualize Benchmark Ranks: Introductory datasets feature many incomplete or sample submissions. Disciplined validation and clean pipelines matter more than chasing leaderboard tenths.
Viability of Mobile Userspace: Tabular cross-validation and standard machine learning workflows run reliably on consumer ARM64 Android devices with tools like Termux and PRoot.
Field Tests & Applications: The Pocket Data Science Series
See this Termux + Antigravity CLI setup in action across competitive machine learning benchmarks on mobile hardware:
Leave a Reply