Global bike sales 2013–2023 on Databricks
End-to-end capstone: PySpark exploration, business hypotheses and regression
Problem
Run a full data science project («50 steps») on bike sales: frame business hypotheses, prepare the data and predict quantity per order (Order_Quantity).
Data
Global Bike Sales Dataset 2013–2023 (Kaggle): costs, prices, revenue, shipping, customer age, gender and country. Tables in workspace.default (bike_sales_order, bike_sales_complete).
Methods
- Exploration with Spark DataFrames and Databricks visualisations
- 3 business hypotheses (revenue by region, units per order, revenue by age group)
- Excluded Profit/Cost/Revenue to avoid target leakage
- Linear regression evaluated with R², MAE and RMSE (the notebook uses Spark ML and scikit-learn)
- Residual analysis and error severity by percentiles
- Enrichment with Product, Sub_Category and Unit_Price + Random Forest
- K-means (course exercise)
Results
- Linear regression on Order_Quantity (as delivered): MAE 5.9154, RMSE 7.6347, R² 0.3594 on test.
- Random Forest with Product, Sub_Category and Unit_Price added: R² 0.8083 on test (0.9728 on train) and MAE 2.84 units vs 5.92 for linear regression.
Corrected after submission: the notebook attributed R² 0.8083 to linear regression; the cell ran after the Random Forest had replaced the `model` variable. Local re-run (scikit-learn, no Spark): Random Forest R² 0.8088 on test and 0.9728 on train, MAE 2.84; linear regression R² 0.358. Published figures are the delivered ones, with corrected attribution.