With a high predictive power and a low tendency to overfit, Gradient Boosting Decision Trees (GBDT) are very popular, and they are often used in winning solutions to machine learning competitions, as well as industry applications. After presenting Gradient Boosting as a special kind of gradient descent, we will describe various tricks and techniques that can speed up a GBDT implementation by multiple orders of magnitude. We will also present PyGBM, a fast pure-Python implementation of GBDTs inspired by LightGBM, that leverages Numba's JIT compilation. We will also provide feedback about our extensive use of Numba.