Sample Scenario

Let’s imagine a simple situation. Typically, we expect that the more time a student dedicates to studying, the higher their exam scores will be. Picture a group of elementary school students surveyed about their weekly study hours, resulting in a dataset that correlates their study time with exam scores.

While this dataset may seem idealized—since exam results can be influenced by factors beyond just study time—it serves well for illustrating the regression tree model.

Building the Tree: Making Branches

The essence of constructing a regression tree lies in determining how to split the nodes—or, in simpler terms, how to create branches.

For instance, if we decide to split our data based on the condition "Study Hours < 3.1", this value represents a threshold between the 7th and 8th data points.

This split creates two regions:

• The left region includes students who study less than 3.1 hours, with an average score of 15.96.
• The right region encompasses those studying 3.1 hours or more, averaging 87.65.

Thus, we can predict:

• Students studying under 3.1 hours will likely score around 15.96.
• Those studying 3.1 hours or more can expect around 87.65.

However, how do we decide on the split conditions? Essentially, we want to minimize the error in our predictions.

Error Measurement

To minimize error, we need to define how to measure it. For instance, if a student studies for 0.5 hours and scores 1.4, but our regression tree predicts 15.96, the error is the difference between these two values: 15.96 - 1.4 = 14.56.

To find the optimal split, we will evaluate every possible split condition, aiming for the one that results in the least error.

Continuing the Splitting Process

While the initial condition "Study Hours < 3.1" is a good starting point, it’s not fully optimized, as the total error remains substantial. Therefore, we must continue splitting the tree to enhance its predictive accuracy.

After the first split, we have two branches. We apply the same logic to these branches, seeking the best split for both left and right samples.

By refining our regression tree further, we can improve predictions. For example, if a student studies only 0.5 hours, our revised prediction might be 4.82 instead of the earlier 15.96.

The first video titled "Hands-on Machine Learning -- Decision Trees" offers practical insights into decision trees, showcasing how they function in real scenarios.

Identifying the Overfitting Issue

Having established how to construct a regression tree, we must also recognize a potential pitfall—overfitting.

When we continue splitting branches excessively, we may create a tree that perfectly fits our training data, but fails in real-world applications. For instance, it may suggest that students studying slightly more hours end up with lower scores, which contradicts common sense.

Overfitting occurs when a model is overly complex, accurately predicting training data but lacking generalizability.

Strategies to Mitigate Overfitting

Although we cannot completely eliminate overfitting, we can employ strategies to lessen its impact. Two common approaches include:

1. Limiting the tree's depth: By setting a maximum depth for the tree, we can prevent excessive complexity. For instance, if we cap the depth at 2, further splits will cease when we reach this limit.
2. Restricting the number of samples on leaf nodes: By ensuring that leaf nodes contain a minimum number of samples (e.g., more than 5), we can maintain the model's robustness.

Implementing both strategies can create a more balanced regression tree that performs well in practice.

Conclusion

In this article, I aimed to clarify the regression tree model in machine learning without relying on complex formulas or scientific expressions. Understanding how to construct a regression tree and the importance of minimizing overfitting is crucial for effective predictive modeling.

If you found this article helpful, consider supporting me and other writers on Medium!

The second video, "Machine Learning Lecture 29: Decision Trees / Regression Trees," provides a comprehensive overview of decision trees, ideal for those looking to deepen their understanding of this topic.