This book focuses on the study of visual vocabularies for category-level object recognition. Our aim is not just to obtain more discriminative and more compact visual codebooks, but to bridge the gap between visual features and semantic concepts. A novel approach for obtaining class representative visual words is presented. It is based on a maximisation procedure, i.e. the Cluster Precision Maximisation, of a novel cluster precision criterion, and on an adaptive threshold refinement scheme for agglomerative clustering algorithms based on correlation clustering techniques. A novel clustering aggregation based approach for building effective visual vocabularies is described too. It consist of a novel framework for incorporating meaningful spatial coherency among the local features into the visual codebook construction. We also propose an efficient high-dimensional data clustering algorithm, the Fast Reciprocal Nearest Neighbours. Finally, we release a new database of images called Image Collection of Annotated Real-world Objects (ICARO), which is especially designed for evaluating category-level object recognition systems.