SBIRC — Style Based Image Retrieval and Classification

The automatic classification of images according to stylistic and aesthetic criteria would allow: image browsers and search engines to take into account the user’s aesthetic preferences; on-line artwork sites to tailor their offer to match the implicit preferences revealed by the previous purchases of a specific user; online museums to reorganize their exhibitions according to user preferences, thus offering personalized virtual tours; digital cameras to make suggestions regarding photographic composition. Additionally, this type of system could be used to automatically index image databases or even, if coupled with an image generation system, to create images of a particular style or possessing certain aesthetic qualities.

As the title indicates, the main goal of this project is the development of techniques for stylistic and aesthetic based image classification and retrieval. To pursue this goal we assembled a multifaceted team of researchers with considerable experience in fields such as: Computational Aesthetics, Evolutionary Computation (EC) , Artificial Neural Networks (ANNs), Music Information Retrieval (MIR), Creative Systems and Computational Art. The contribution of these different competences and backgrounds to the overall project becomes clear when we consider the steps involved.

We begun by studying, identifying and developing a series of features that are thought to be aesthetically and/or stylistically relevant. To this end we rely on the team’s experience in computational aesthetic and on its previous research in the field. We will give particular importance to image complexity estimates, Zipf law based metrics, Fractal Dimension (FD) estimates, wavelets and color vicinity relations.

The use of these type of feature in the context of content based image retrieval (CBIR) is one of the innovative aspects of our proposal, contrasting with current CBIR approaches that tend to rely on techniques such as histogram analysis, shape analysis, face detection and identification, color matching, etc. This reflects a core difference in objectives between our proposals, which aims to retrieve and classify images according to aesthetic and stylistic properties, and most CBIR systems that tend to focus on tasks such as retrieving images of a given subject matter (e.g. of a given person or place), filter out images of a given nature (e.g. pornographic or violent), or roughly match a given outline or sketch drawn by the user.

In previous research the team used an ANN based classifier that took as input the results provided by the feature extractor (FE). The study and comparison of alternative classifier approaches, namely ANNs Support Vector Machines (SVM) and evolutionary computation based classifier systems, is an integral part of this project. The developed classifiers will be tested in series of tasks including: author and style identification; aesthetic judgment (using as reference the collective assessment performed by a set of users, a single user, and an expert in the field); image balance and composition analysis. These experiments serve three purposes: determining the appropriateness of different classifier approaches, ascertaining the aesthetic relevance of the considered features, and identifying the most relevant ones.

Once trained in this sort of task the classifier systems will also be used to guide NEvAr, an EC-based image generation system. The goal of the EC is to evolve images that the classifier identifies as belonging to a given style or author. In addition to the obvious image generation goal, his serves two purposes: testing the capabilities of the EC generator and of the image classifier. As previous results show the ability of the EC system to exploit the shortcomings of the classifier make this a powerful validation technique. Additionally, the classifiers will also be submitted psychological tests designed for humans that aim to assess how people identify and react to several principles of aesthetics.

Finally, using the set of most relevant features identified in the course of the experiments, the team will develop a stylistic based image retrieval prototype, which will be accessible through a web interface and will retrieve images that are stylistically and/or aesthetically similar to a target image provided by the user.

Articles

J. Romero, P. Machado, A. Carballal, and A. Santos, “Using complexity estimates in aesthetic image classification,” Journal of Mathematics and the Arts, vol. 6, iss. 2-3, pp. 125-136, 2012.
- Bibtex
- |
- Link
- |
- PDF

In Collections

J. Romero, P. Machado, A. Carballal, and J. Correia, “Computing Aesthetics with Image Judgement Systems,” in Computers and Creativity, J. McCormack and M. d’Inverno, Eds., Springer Berlin Heidelberg, 2012, pp. 295-322.
- Bibtex
- |
- Link
- |
- PDF
P. Machado, J. Romero, and B. Manaris, “Experiments in Computational Aesthetics: An Iterative Approach to Stylistic Change in Evolutionary Art,” in The Art of Artificial Evolution: A Handbook on Evolutionary Art and Music, J. Romero and P. Machado, Eds., Springer Berlin Heidelberg, 2007, pp. 381-415.
- Bibtex
- |
- Link
- |
- PDF

In Proceedings

J. Correia, P. Machado, J. Romero, and A. Carballal, “Feature Selection and Novelty in Computational Aesthetics,” in Evolutionary and Biologically Inspired Music, Sound, Art and Design – Second International Conference, EvoMUSART 2013, Vienna, Austria, April 3-5, 2013. Proceedings, 2013, pp. 133-144.
- Bibtex
- |
- PDF
J. Romero, P. Machado, A. Carballal, and O. Osorio, “Aesthetic Classification and Sorting Based on Image Compression,” in Applications of Evolutionary Computation – EvoApplications 2011: EvoCOMNET, EvoFIN, EvoHOT, EvoMUSART, EvoSTIM, and EvoTRANSLOG, Torino, Italy, April 27-29, 2011, Proceedings, Part II, 2011, pp. 394-403.
- Bibtex
- |
- PDF

Author

Penousal Machado

João Nuno Correia

Juan Romero

Adrian Carballal

Amílcar Cardoso

Date

28/02/2013

Acknowledgements

PTDC/EIA-EIA/115667/2009