Front Matter Page European Department
I. Introducing Optimal Pooling with Machine Learning
II. A Two-Step Method for Optimal Pooling
III. Applying the Method
II. IV. Conclusions
1. The Bias-VarianceTradeoff
1. Step 1—Proximity
2. Step 2—Optimal Pool
3. Turkey-Most Proximate Countries
4. Turkey-Least Proximate Countries
5. Turkey—Relative Forecast Errors for Different Pools
6. Other Countries—Relative Forecast Error of Different Pools
I. Machine Learning and Cross
We leverage insights from machine learning to optimize the tradeoff between bias and variance when estimating economic models using pooled datasets. Specifically, we develop a simple algorithm that estimates the similarity of economic structures across countries and selects the optimal pool of countries to maximize out-of-sample prediction accuracy of a model. We apply the new alogrithm by nowcasting output growth with a panel of 102 countries and are able to significantly improve forecast accuracy relative to alternative pools. The algortihm improves nowcast performance for advanced economies, as well as emerging market and developing economies, suggesting that machine learning techniques using pooled data could be an important macro tool for many countries.
. There is a tradeoff between bias and variance in expanding a dataset to include data from additional countries. Adding countries with a similar economic structure (i.e., data-generating process) may reduce variance and improve forecasts, but adding countries with a dissimilar economic structure may introduce bias to the forecasts. Optimal pooling amounts to solving a version of the bias-variancetradeoff for which machine learning methods have specifically been developed. The aim of this paper is to use insights from machine learning to provide a method for selecting
Front Matter Page European Department
I. INTRODUCING MACHINE LEARNING
II. THE BASICS OF FORECASTING—A BIAS-VARIANCETRADEOFF
A. Shortcomings of OLS-Based Forecasting Methods
B. The Advantages of Machine Learning Methods
III. A FRAMEWORK FOR MACRO FORECASTING WITH MACHINE LEARNING
A. Limiting Preselection
B. Identifying Complementary Algorithms
C. Evaluating Performance and Interpreting Results
IV. RESULTS—MORE ACCURATE FORECASTS
I. The Bias-VarianceTradeoff
We develop a framework to nowcast (and forecast) economic variables with machine learning techniques. We explain how machine learning methods can address common shortcomings of traditional OLS-based models and use several machine learning models to predict real output growth with lower forecast errors than traditional models. By combining multiple machine learning models into ensembles, we lower forecast errors even further. We also identify measures of variable importance to help improve the transparency of machine learning-based forecasts. Applying the framework to Turkey reduces forecast errors by at least 30 percent relative to traditional models. The framework also better predicts economic volatility, suggesting that machine learning techniques could be an important part of the macro forecasting toolkit of many countries.
All forecasting methods aim to minimize expected forecast errors. Forecasting consists of selecting a function that maps indicator data to a forecast while minimizing a particular loss function. Suppose a researcher wants to forecast a variable y t (e.g., real GDP growth) using K predictor variables summarized in the K x 1 vector X t , with the h-step ahead forecast of y t denoted as y t+h :
where ε t+h is an
I regress real GDP growth rates on the IMF’s growth forecasts and find that IMF forecasts behave similarly to those generated by overfitted models, placing too much weight on observable predictors and underestimating the forces of mean reversion. I identify several such variables that explain forecasts well but are not predictors of actual growth. I show that, at long horizons, IMF forecasts are little better than a forecasting rule that uses no information other than the historical global sample average growth rate (i.e., a constant). Given the large noise component in forecasts, particularly at longer horizons, the paper calls into question the usefulness of judgment-based medium and long-run forecasts for policy analysis, including for debt sustainability assessments, and points to statistical methods to improve forecast accuracy by taking into account the risk of overfitting.
after controlling for a large number of candidate predictors, WEO forecasts contain valuable information that would otherwise not be captured by a linear prediction model – though less so for developing countries. But the gains in accuracy from reducing the weight of judgement in the IMF’s forecasts appear to be significant.
The rest of the paper is organized as follows: First, I revisit the bias-variancetrade-off. Then I apply Copas’ approach to detect overfitting in IMF forecasts. Section 4 benchmarks the accuracy of forecasts against a naive prediction rule and
El Bachir Boukherouaa, Khaled AlAjmi, Jose Deodoro, Aquiles Farias, and Rangachary Ravikumar
. Overfitting is one of the main challenges in ML and it occurs when the model performs very well on the training data set, but poorly on new, unseen data (test data). The better a model performs on the training data, the less biased it is. However, its performance can change dramatically on a set of unseen data. In this case, in order to achieve robustness in performance, the researcher may opt to choose a model that does not perform as well on the training set but maintains its performance on the test set. This situation is known as the bias-variancetrade-off
estimates. On the downside, by discarding past information, the variance of parameter estimates increases, which is reflected in larger forecast errors. In sum, when estimating model parameters under structural change, there is a balance between how much information one should use and how much of it should be discarded. In other words, there is a bias-variancetradeoff that can be exploited.
18. If the sample size is large enough, as it is typically in financial applications or in macroeconomic data available for advanced countries, one way to exploit the bias-variance