Pocket Data Science VIII: Kaggle Store Sales Quad-Ensemble

The progression across our second wave of experiments is shown below. Scores marked LB are Kaggle public leaderboard results; scores marked Val are from our local 16-day holdout (2017-08-01 to 2017-08-15).

Before achieving our breakthrough, we suffered two sharp regressions on the public leaderboard. Understanding why these intuitive ideas failed is more instructive than the eventual improvement itself.

The Blanket Multiplier Trap (Experiment 6: 0.41709)

In Ecuador, the Sierra region experiences a strong back-to-school surge in late August. On local validation holdout, multiplying SCHOOL AND OFFICE SUPPLIES sales by a calibrated factor of 2.6x in Sierra stores cut that family’s holdout error from 0.63841 to 0.44847.

Confident in this domain discovery, we submitted Experiment 6 to Kaggle. The result was a regression from 0.41072 to 0.41709. Why? The Root Mean Squared Logarithmic Error (RMSLE) measures error in log space, so it scores relative error rather than absolute error:

RMSLE = sqrt( 1/N * sum( (log(pred_i + 1) - log(actual_i + 1))^2 ) )

Predicting 50 units when actual demand is 10 costs log(51/11) = 1.534, while a 40-unit miss on a 1,000-unit series costs only log(1041/1001) = 0.039. Back-to-school purchasing across Ecuador’s 54 stores is highly heterogeneous, and several smaller Sierra stores sell only a handful of school supplies per day, or none at all. A uniform 2.6x multiplier is harmless on high-volume series, but on low-volume and near-zero series it turns small absolute errors into large ratio errors, and those rows dominate the score. Our holdout gain came from a few large stores; the test period punished the many small ones.

The Annual Lag Pollution (Experiment 7: 0.41901)

In Experiment 7, we attempted to capture year-over-year seasonality by introducing a 52-week (364-day) lag (lag_364) alongside a split of the 16-day forecast horizon. The model regressed further to 0.41901.

Because Experiment 7 changed two things at once, we cannot attribute the regression to the lag alone. Our working hypothesis is that lag_364 for the August 2017 test window points back to August 2016, when sales were still recovering from the disruption of the April 2016 Ecuador earthquake. With only four years of history, a single annual lag carries one noisy observation per date, which gradient boosting can easily overfit. Isolating the two changes is on our list for a future experiment.

CatBoost had been our primary modeling engine throughout Parts II, V, and VII. While exceptional at preventing categorical target leakage, CatBoost builds symmetric (oblivious) decision trees:

• Oblivious Tree Property: At depth d, every decision node across the entire tree is forced to split on the exact same feature and threshold.
• The Retail Sparsity Problem: In retail forecasting, high promotional activity is rare and deeply localized. A large promotional surge (e.g. onpromotion > 25) only triggers demand spikes for specific product families (such as GROCERY I or BEVERAGES) in high-volume stores (Type A).

Because oblivious trees must apply the same split condition across every branch at a given level, isolating a sparse interaction among 1,782 series requires deeper trees (depth 8 to 10). On mobile CPUs, deeper trees sharply increase training time and tend to overfit non-promoted rows.

We realized we needed an asymmetric, best-first tree architecture: LightGBM.

Conventional wisdom holds that running multi-library gradient boosting on mobile hardware is either unsupported or prohibitively slow. Inside our Android Termux Debian PRoot environment, we installed official pre-built Linux aarch64 wheels with no compilation required:

pip install lightgbm xgboost --break-system-packages

Both libraries ran without issues on the phone’s 8-core ARM Cortex processor. By keeping our dataset memory-optimized using int8, float32, and Pandas category dtypes, our total memory footprint remained under 500 MB RAM:

Unlike depth-wise or oblivious tree growth, LightGBM uses leaf-wise (best-first) tree splitting. It expands the leaf that produces the largest decrease in loss, allowing the tree to grow asymmetric branches that focus on complex promotional sub-populations:

# Leaf-wise LightGBM configuration
m_lgb = lgb.LGBMRegressor(
    n_estimators=650,
    learning_rate=0.06,
    num_leaves=63,
    subsample=0.8,
    colsample_bytree=0.8,
    min_child_samples=40,
    random_state=42,
    n_jobs=4,
    verbose=-1
)

Evaluating standalone LightGBM on our 16-day holdout (2017-08-01 to 2017-08-15) gave 0.40810 RMSLE, outperforming standalone CatBoost (0.41110) by 0.00300 in less than one-eighth of the training time.

Rather than replacing our existing models entirely, we recognized that our Part VII hybrid base (Experiment 5: domain prior plus CatBoost residual correction) provided a well-regularized anchor for weekly store-level velocity. We constructed a linear blend in logarithmic space:

# Experiment 8 Blend (75% Base + 25% Leaf-Wise LightGBM)
log_pred_exp8 = 0.75 * np.log1p(sales_exp5) + 0.25 * pred_lgb
sales_exp8 = np.clip(np.expm1(log_pred_exp8), 0, None)

We submitted Experiment 8 to Kaggle. The score dropped from 0.41072 to 0.40403, moving us 12 positions up to Rank #104 globally.

To test sensitivity to the blend weight, we submitted Experiment 8.1 with LightGBM at 40% (60% Base + 40% LightGBM). It scored 0.40408, showing that the blend had plateaued around 0.404 and that further gains would need a different kind of model rather than a different weight.

Ensembles generalize best when their members make different kinds of errors. Our Quad-Ensemble combines four modeling paradigms in three blend components, since the Experiment 5 anchor already contains two of them:

• Hybrid Anchor (70%): The Experiment 5 base, made of (1) a domain heuristic prior capturing day-of-week seasonality and paydays, corrected by (2) a symmetric-tree CatBoost residual model.
• (3) Leaf-Wise LightGBM (20%): Captures localized promotional effects via best-first splits.
• (4) Depth-Wise XGBoost (10%): Adds level-wise histogram trees with different regularization and split dynamics.

# Quad-Ensemble (domain prior + CatBoost inside log_exp5)
log_ensemble = 0.70 * log_exp5 + 0.20 * log_lgb + 0.10 * log_xgb
sales_ensemble = np.clip(np.expm1(log_ensemble), 0, None)

Upon submitting Experiment 9 to Kaggle, our score dropped to 0.40336 RMSLE, breaking the 0.404 plateau:

With our daily Kaggle submission quota exhausted, we turned to local optimization to see whether data-driven blend weights could beat the hand-set 70/20/10 split of Experiment 9.

SLSQP Constrained Optimization

Using scipy.optimize.minimize with Sequential Least Squares Programming (SLSQP), we solved for blending weights bounded between [0, 1] subject to sum(w) == 1.0 on our holdout validation set:

Optimal Stacking Weights:
Domain Prior Base: 0.60
LightGBM Regressor: 0.25
XGBoost Regressor:  0.10
CatBoost Regressor: 0.05
Holdout Optimized RMSLE: 0.40182

The Leaderboard Verdict

On the holdout, the optimized blend looked like our best model yet. On the public leaderboard, Experiment 10 scored 0.40426: worse than the hand-set Quad-Ensemble (0.40336) and 0.00244 behind its own validation estimate.

The cause is selection bias. The weights were fitted on the same holdout window used to score them, so the optimizer tuned itself to the quirks of early August 2017 rather than to patterns that carry into the test window. The coarse 70/20/10 split absorbed less of that noise and generalized better.

Dead-Series Guardrail with Promotional Gating

Experiment 10 also applied a guardrail that is independent of the blend weights. In retail data, certain product families are discontinued or inactive. If a store has zero sales for a product family over the preceding 14 days and has zero scheduled promotions (onpromotion == 0), tree models often predict a residual trickle of 0.1 to 0.4 units. Because RMSLE penalizes non-zero predictions on true zero rows, we identified 144 dead store-family pairs and enforced zero-sales masks across 2,278 test rows.

Our progression from a simple non-ML baseline to a Global Top 100 ensemble yielded five practical principles:

• Ensemble Divergent Paradigms: Blending models from the same algorithmic family (e.g. multiple CatBoost seeds) yields marginal gains. Combining a domain statistical prior, symmetric CatBoost, leaf-wise LightGBM, and depth-wise XGBoost improved our leaderboard score from 0.41072 to 0.40336.
• Tree Architecture Shapes What Is Learned: Symmetric oblivious trees excel at global categorical stability but struggle with sparse localized interactions. Leaf-wise splitting handles promotional retail spikes more naturally.
• Know What Your Metric Rewards: RMSLE scores relative error, so mistakes on low-volume and near-zero series weigh as much as mistakes on the biggest sellers. Blanket manual multipliers should be replaced with learned, conditioned interactions.
• Simple Blend Weights Generalize Better: SLSQP-optimized weights improved our holdout score to 0.40182 but scored 0.40426 on the leaderboard, losing to a hand-set 70/20/10 blend. Weights fitted on the scoring holdout are biased toward that window.
• Edge Hardware is Sufficient: Competitive time series forecasting does not require server farms or GPUs. With disciplined memory management, a three-library GBDT ensemble trains in under 2.5 minutes on a mobile CPU.

The pipeline script below trains all three engines and writes the Experiment 10 stacked submission. To reproduce our best-scoring Experiment 9 result instead, apply the 70/20/10 blend from Section 7 to the same model outputs.

# Navigate to competition workspace
cd ~/kaggle_store_sales

# Install native ARM64 gradient boosting engines
pip install lightgbm xgboost catboost --break-system-packages

# Train all engines and write the Experiment 10 stacked submission
python3 exp10_grandmaster_sub40_ready.py

# Verify submission integrity
head -n 5 submission_exp10_grandmaster_sub40_ready.csv