Due to current advances in computer vision, social scientists increasingly rely on images as data. One area of application is the detection of political events from these images, for example political protest. While existing classifiers reach high levels of accuracy, it is difficult to systematically assess what visual features they base their classification on. We present a new two-level classification method that remedies this problem. At the first stage, our method uses image segmentation to detect the objects present on the image. This information is then represented as a (non-visual) feature vector that can be used in the second stage by standard machine learning classifiers. We apply our method to a new dataset of protest images. We show that in addition to predicting the final outcome (protest), this method produces much more transparent insights into the data. For example, we demonstrate how it can be used to study cross-national differences in the tactics of protest, which is an important question for comparative research.