In this talk we will discuss a common and highly effective model ensemble technique known as stacking and how it can be used for classification to predict rare target events. We will start with the business problem, predicting which users will respond to online advertising and creating a list of these users called an “audience” to be used in ad serving. We will then describe stacking and explain the advantages, from reducing generalization bias to the practical implications of parallelization of model development amongst developers. Finally we will describe how we optimized a stacked model ensemble to create audiences.