Master Tree Ensemble Models

Learn how decision trees and tree-based ensembles work under the hood, and how to train and tune them in practice. From single decision trees and random forests, to AdaBoost, gradient boosting, XGBoost, LightGBM and CatBoost.

This repository contains the practical notebooks for the Master Tree Ensemble Models course. The examples cover classification and regression, using scikit-learn, XGBoost, LightGBM, and CatBoost.

Course launch: November 2026

Status: Actively maintained

What you will learn

• Understand how decision trees find splits, when to stop growing them, and how they produce predictions.
• Build ensembles with bagging and random forests, and understand why decorrelating trees works.
• Understand boosting, from AdaBoost to gradient boosting, and implement both from scratch.
• Learn what makes XGBoost, LightGBM and CatBoost fast and accurate: regularized losses, split-finding algorithms, histogram-based learning, GOSS, EFB, ordered boosting and more.
• Tune the key hyperparameters of each library and train models for classification and regression.
• Explore advanced topics like DART and gradient boosting with imbalanced data.

Course contents

1. Decision Trees — notebooks

   • How decision trees work
   • Tree induction: selecting features and split values
   • Pruning and stopping tree growth
   • Training classification and regression trees

2. Bagging and Random Forests — notebooks

   • Foundations of ensemble models
   • Bagging
   • Random forests and decorrelating the trees
   • Training classification and regression random forests

3. Boosting — notebooks

   • AdaBoost: intuition, algorithm and exponential loss
   • Gradient boosting: steepest descent, residuals and the algorithm
   • Training GBMs for classification and regression

4. XGBoost — notebooks

   • Regularized loss function and split gain
   • Split finding: exact greedy and approximate algorithms
   • Weighted quantile sketch and sparsity-aware split finding
   • Feature importance
   • Hyperparameters
   • Training XGBoost for classification and regression

5. LightGBM — notebooks

   • Histogram-based split algorithm
   • Gradient-based One-Side Sampling (GOSS)
   • Exclusive Feature Bundling (EFB)
   • Leaf-wise tree growth
   • Hyperparameters
   • Training LightGBM for classification and regression

6. CatBoost — notebooks

   • Target statistics and conditional shift
   • Ordered target statistics
   • Oblivious trees
   • Prediction shift and ordered boosting
   • Feature combination
   • Hyperparameters
   • Training CatBoost for classification and regression

Getting started

Clone the repository (Python 3.12+ required), then set up a dedicated environment with either of the options below.

Option 1: venv + pip

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt
jupyter notebook

Option 2: uv

uv sync
uv run jupyter notebook

Open the notebooks in numerical order.

Course

For lectures, explanations, and the complete learning path, visit the online course.