Title Page

TECHNICAL NOTES AND MANUALS

How to Assess Country Risk: The Vulnerability Exercise Approach Using Machine Learning

Approved by Sanjaya Panth

Strategy, Policy, and Review Department

INTERNATIONAL MONETARY FUND

Title Page

INTERNATIONAL MONETARY FUND

How to Assess Country Risk: The Vulnerability Exercise Approach Using Machine Learning

Approved by Sanjaya Panth

Prepared by an interdepartmental team led by Kevin Wiseman, and comprising Pranav Gupta and Andrew Tiffin (AFR), Andrew Swiston (APD), Juliana Gamboa Arbelaez, Klaus Hellwig, Andrew Hodge, Paulo Medas, Marialuz Moreno Badia, Roberto Perrelli, and Yuan Xiang (all FAD), Maksym Ivanyna (ICD), Sean Simpson (iLab), André Leitão Botelho and Wan Li (ITD), Silvia Iorgova (MCM), Suman Basu (RES), Chuqiao Bi, Jorge Chan-Lau, Sandile Hlatshwayo, Chengyu Huang, Lamya Kejji, Agustin Roitman, Weining Xin, Harry Zhao, and Yunhui Zhao (all SPR), Le Xu (SPR Summer Intern), and Daria Ulybina (STA); under the supervision of Daria Zakharova and Wojciech Maliszewski (SPR). Sharon Eccles (SPR) provided excellent administrative assistance.

ACKNOWLEDGMENTS

We are grateful for the support of this project by the iLab through the AI/ML Innovation Challenge and Catalyst series. We would also like to gratefully acknowledge the support from and the discussions with Michal Andrle, Alberto Behar, Angana Banerji, Fabian Bornhorst, Eugenio Cerutti, Marcos Chamon, Kirpal Chauhan, Qianying Chen, Mali Chivakul, Federico Diaz Kalan, Florence Dotsey, Aquiles Farias, Vikram Haksar, Yuko Hashimoto, Niko Alfred Hobdari, Plamen Iossifov, Tetsuya Konuki, Miguel Lanza, Emilia Magdalena Jurzyk, Maxym Kryshko, Nan Li, Sandra Lizarazo, Albert Touna Mama, Jimmy McHugh, Alexis Meyer Cirkel, Chifundo Moya, Nkunde Mwase, Rajesh Nilawar, Liam O’Sullivan, Marijn Otte, Mamoon Saeed, Jasmin Sin, Shannon Staley, Fabian Valencia, Tristan Walker, Hans Weisfeld, Jason Weiss, and Weijia Yao.

Copyright Page

©2021 International Monetary Fund

Cataloging-in-Publication Data

Joint Bank-Fund Library

Names: International Monetary Fund.

Title: How to Assess Country Risk: The Vulnerability Exercise Approach Using Machine Learning

Other titles: Technical Notes and Manuals (International Monetary Fund)

Series/volume #: TNM/21/03

Description: Washington, DC : International Monetary Fund | Periodic | Some issues also have thematic titles.

Classification: LCC HC10.W79

HC10.80

ISBN: 978–1-51357–421-9

DISCLAIMER: The views expressed in this paper are those of the author(s) and do not necessarily represent the views of the IMF, its Executive Board, or IMF management. The opinions contained in this document are the sole responsibility of the authors.

article image

Publication orders may be placed online, by fax, or through the mail:

International Monetary Fund, Publication Services

P.O. Box 92780, Washington, DC 20090, U.S.A.

Tel. (202) 623–7430 Fax: (202) 623–7201

E-mail: publications@imf.org

www.imfbookstore.org

www.elibrary.imf.org

Contents

  • Executive Summary

  • Introduction

  • Machine Learning and Crisis Forecasting

  • Models and Estimation Strategy

  • Communicating Results

  • External Sector Model

  • Fiscal Sector Model

  • Financial Sector Model

  • Real Sector Model

  • Conclusion and Next Steps

  • References

  • Annexes

    • I. Signal Extraction Method

    • II. From Decision Trees to a Random Forest

    • III. Beyond Random Forests: Balanced Forest and Boosting

    • IV. Predicting Out-of-Sample Performance: Cross Validation

    • V. Comparing Classifiers: Area Under the Curve (AUC)

    • VI. Assigning the Blame: Shapley Values

    • VII. Filling in the Gaps: Dealing with Missing Data

    • VIII. Fiscal Sector Explanatory Variables

  • Figures

    • 1. Risk Architecture at the IMF

    • 2. Fiscal Crisis Risk – Country A, 2019

    • 3. Contribution to Risk Index

    • 4. Distribution of Variable with Largest Risk Contributions

    • 5. Countries with a Similar Financial Sector Risk Profile with Ireland in 2005

    • 6. Frequency of Sudden Stops

    • 7. Frequency of EMP Events

    • 8. External Sector Model Performance: SSGI

    • 9. External Sector Model Performance: EMP Events

    • 10. SSGI Model Variable Importance

    • 11. EMP Model Variable Importance

    • 12. Historical Risk Indices Over Time

    • 13. External Risk Interactions

    • 14. External Sudden Stop Index and the Asian Financial Crisis, Selected Countries

    • 15. Countries with Fiscal Crises, 1980–2017

    • 16. Fiscal Sector Model Performance

    • 17. Fiscal Model Variable Importance

    • 18. Fiscal Risk in Greece, 2009

    • 19. Bank Crisis History and Frequency

    • 20. Financial Sector Model Performance

    • 21. Financial Model Variable Importance

    • 22. Historical Evolution of Average Risk Index

    • 23. Global-Local Variable Interaction in Financial Sector Model

    • 24. Financial Crisis Risk Index, Iceland and Ireland, 2005 and 2007

    • 25. Real Sector Crisis History

    • 26. Real Sector Model Performance

    • 27. Real Model Variable Importance

    • 28. Nonlinear Interactions in Real Sector Model

    • 29. Crisis Risk Indices in Four Sectors, Ethiopia and Greece

  • Tables

    • 1. External Crisis: Explanatory Variables

    • 2. Definitions

    • 3. Financial Crisis: Explanatory Variables

    • 4. Real Crisis: Explanatory Variables

Executive Summary

The IMF’s Vulnerability Exercise (VE) is a cross-country exercise that identifies country-specific near-term macroeconomic risks. As a key element of the Fund’s broader risk architecture, the VE is a bottom-up, multi-sectoral approach to risk assessments for all IMF member countries. Assessments reflect the judgement of country teams informed by consistent, cross-country quantitative models as well as country-specific context.

The VE modeling toolkit is regularly updated in response to global economic developments and the latest modeling innovations. Earlier models evolved organically, assessing advanced economies, emerging markets, and low-income countries separately and looking at different types of risks. The new generation of models presented here closes gaps in risk and country coverages from previous models, while improving consistency and comparability of risk assessments across countries.

The new generation of VE models presented here leverages machine-learning (ML) algorithms. Macroeconomic risk assessment is a challenging task: crises are infrequent and almost always involve some elements of surprise. They tend to feature interactions between different parts of the economy and non-linear relationships that are not well measured in “normal times.” ML tools can often better capture these relationships. They can also be more robust to outliers, noise, and the diversity of experiences across countries.

The performance of machine-learning-based models is evaluated against more conventional models in a horse-race format. The models assess the near-term risk of a crisis in the external, financial, fiscal, and real sectors. In each sector, rigorous performance metrics are used to compare new tools against traditional approaches. It turns out that random forest-based models, which are popular modern ML methods that average over many decision trees, outperform other options in most cases. In other cases, the signal extraction approach, a robust non-parametric method designed for macro-crisis detection, performs best. These winning models represent a new generation of models at the core of the VE.

The paper also presents direct, transparent methods for communicating model results. ML techniques can sometimes appear to be a black box due to their complexity and infrequent (though rapidly growing) use in economics. Communication tools, developed to inform country teams about the model assessments, help take the last step from predicting to informing.

How to Assess Country Risk: The Vulnerability Exercise Approach Using Machine Learning
Author: International Monetary Fund. Strategy, Policy, & Review Department